Skip to content

Renovate Operations

Renovate is the dependency-update bot. A self-hosted Mend Renovate CE worker runs in the cluster and serves three GitHub accounts. This page states what exists, the constraints that bind an edit to the repository configuration, and the procedures an operator runs. kubectl commands assume --context fzymgc-house.

Property Value
Namespace renovate
Workload deploy/renovate-ce
Chart and pin mend-renovate-ce, targetRevision in argocd/cluster-app/templates/renovate.yaml
Accounts discovered fzymgc-house/*, seanb4t/*, holomush/*, except fzymgc-house/node-bootstraps (mendRnvAutoDiscoverFilter)
Account-wide configuration config.js, embedded in the same manifest
Repository configuration .github/renovate.json
Schedule * * * * 0,1,5,6 (Friday through Monday), America/New_York
Cache PVC renovate-ce-cache-hi, 32Gi, longhorn-highinode, mounted at /tmp/renovate
Cache root that survives cleanup /tmp/renovate/cache
State PVC renovate-ce-db, 1Gi, mounted at /db (mendRnvSqliteFilePath)
Cache prune CronJob renovate-cache-prune, Sundays 03:00, deletes cache files not read in 14 days
GitHub App credential Vault fzymgc-house/cluster/renovate through ExternalSecret renovate-app
GHCR credential Vault fzymgc-house/cluster/ghcr/pull-secret through ExternalSecret renovate-ghcr
Manifests argocd/app-configs/renovate/
Admin API System and jobs APIs, Bearer mendRnvServerApiSecret from Secret renovate-app. Reach it with kubectl port-forward only

The worker cleans /tmp/renovate/repos and /home/ubuntu after every run (mendRnvWorkerCleanup and mendRnvWorkerCleanupDirs). It keeps /tmp/renovate/cache. That is the purpose of the PVC, and it is also why a corrupt cache entry outlives every run.

Two configurations resolve, and only one is editable from this repository

Section titled “Two configurations resolve, and only one is editable from this repository”

The account-wide config.js in argocd/cluster-app/templates/renovate.yaml contributes the first packageRules entries in the effective list. Every repository the worker serves reads it, and a change to it restarts the pod for three accounts. Do not edit it for a rule that concerns this repository alone.

Two consequences bind every rule in .github/renovate.json:

  • The account-wide all-minor-patch rollup is asserted for every minor and patch update before any repository rule runs. A repository rule can re-group or override it. It cannot remove it. This is why the blast-radius split needs a catch-all bucket.
  • The account-wide root sets automerge: true, platformAutomerge: true and reviewers: []. An attended rule must set both automerge keys to false and name a reviewer. One key alone leaves the other in force.

The postures below are what the committed rules resolve to. Resolve the rules last-match-wins per update class when you need the posture of one dependency. Do not read the array in order.

Update class Posture Rule
Any major attended, 7-day floor, reviewer seanb4t, branch prefix major- the rule with matchUpdateTypes: ["major"] and no manager matcher, after every automerge: true rule
Any minor of a 0.y.z pin attended, 7-day floor, reviewer seanb4t, own branch pre-1-0-minor the one rule with matchCurrentVersion
Minor, patch and digest elsewhere automerge, floor from the matching group, one branch per tree the five blast-radius buckets and the catch-all
Dependencies of the buf CLI in apps/dns-control/tools/go.mod disabled, except github.com/bufbuild/buf and google.golang.org/protobuf the rule with matchFileNames: ["apps/dns-control/tools/go.mod"]
mend-renovate-ce chart attended, 7-day floor the mend-renovate-ce guard
Annotated Ansible pins attended, 7-day floor the custom.regex plus ansible/** guard
k3s (spec.version of the two system-upgrade-controller Plans) attended, 7-day floor, one group k3s for both Plans the k3s rule (docs/operations/k3s.md)
pnpm (the three packageManager fields and the mise.toml pin) attended, 7-day floor, one group pnpm. The reviewer regenerates the lock files the pnpm rule (pnpm updates)
Lock file maintenance of the three pnpm-lock.yaml files attended. The reviewer restores the pnpm version record the rule with matchUpdateTypes: ["lockFileMaintenance"] (pnpm updates)
LiteLLM, ClickStack, hl-assets encoders attended, 7 or 30-day floor, own group the attended tail rules
temporal-worker-controller and temporal-worker-controller-crds charts attended, 7-day floor, one group for both charts, chart versions only (no v image tags) the temporal worker controller rule in the attended tail
valkey-operator chart attended, 7-day floor, own group valkey-operator the valkey operator rule in the attended tail (Holds and separate groups)
ghcr.io/seanb4t/charts/engram, every update type automerge, no floor, own group engram chart the engram chart rule
Charts that a kustomization inflates through helmCharts:, such as external-dns automerge, 3-day floor, group helm charts, as the charts of the argocd manager the rule after helm charts that matches the kustomize manager and the helm datasource. The ClickStack rule attends its own chart
cloudflare/cloudflare Terraform provider held below 5.26.0 the cloudflare hold (Holds and separate groups)
Terraform CLI (terraform in mise.toml and terraform_version in tf/hcp-terraform/workspaces.tf) one group terraform-cli-lockstep for both pins the terraform cli (lockstep) rule
velero/velero-plugin-for-aws disabled its own rule
Python runtime and Node.js majors ignored or LTS only their own rules
Vulnerability alerts open at once, no automerge, no floor vulnerabilityAlerts at the top level

Six customManagers entries reach pins that no stock manager reads:

Files Pin shape Notes
*.sh # renovate: line above a _VERSION= assignment no Makefiles exist in this repository
ansible/roles/*/defaults/main.yml, ansible/inventory/{group_vars,host_vars}/*.yml # renovate: line above a YAML key default versioning docker
apps/*/Dockerfile # renovate: line above an ARG datasource deb, versioning deb, one registry URL
.github/workflows/*.yml # renovate: line above an env: key the stock github-actions manager reads uses: lines only
argocd/app-configs/system-upgrade/plan-*.yaml # renovate: line above version: the k3s version of the cluster. The kubernetes manager skips the untagged rancher/k3s-upgrade images
argocd/app-configs/clickstack-chart/values.yaml # renovate: line above tag: the collector image tag, which has no sibling repository for helm-values to read. The ClickStack rule groups it with the HyperDX tag. The annotation names the Docker Hub repository, because docker.clickhouse.com gives no release dates and the release-age floor then lets the two tags drift apart

The # renovate: line must sit directly above the key it annotates. The regex requires adjacency. A blank line or another comment between them un-annotates the pin, and the configuration still validates. The optional fields follow the order versioning= then registryUrl=, because that is the order the regex accepts.

A pin that cannot be annotated carries # renovate-exempt: <reason> on the line above it. Absence of governance and absence of updates are different claims, and the exemption keeps them apart.

The string matchManagers sees for a custom manager is custom.regex, with the prefix. The name-anchored groups do not reach annotated pins, because the annotations emit names such as cilium or k3s-io/k3s that no matchPackageNames entry matches.

Key Committed value Default
prCreation not-pending immediate
internalChecksAsSuccess true false
internalChecksFilter unset strict
prNotPendingHours unset 25

internalChecksFilter is unset because its default is already strict. A key written at its own default asserts nothing.

Under strict, Renovate still selects the highest candidate release. It flags the update pendingChecks when every candidate sits inside its minimumReleaseAge window. Renovate creates no branch for a flagged update, and it does not update an existing branch. This key withholds a pending release, not prCreation. Renovate writes the renovate/stability-days status on each branch. The status is green when the release age elapsed before Renovate looked.

prCreation: not-pending opens a pull request only when the branch status is green. No CI runs on a renovate/* branch before its pull request exists. Every push: trigger in .github/workflows/ is scoped to main or to tags, and the pull_request: workflows need a pull request. The only status on the branch is renovate/stability-days.

internalChecksAsSuccess: true makes Renovate count its own green renovate/* statuses as a green branch. Without it, Renovate reads such a branch as yellow. After the release age elapsed, a yellow branch never gets a pull request, and prNotPendingHours does not apply. Outside the update window, Renovate opens no pull request for an existing branch. The branch then stays without a pull request and counts toward branchConcurrentLimit until someone deletes it.

internalChecksAsSuccess changes how Renovate reads the branch, not what GitHub requires before a merge. The ruleset on main requires a pull request and the pr-gates and Octopus Review checks. It requires no approving review. No rule sets automergeType: branch, so every automerge goes through a pull request and its checks.

prNotPendingHours is unset. It applies only while the release-age status is yellow, and strict keeps a branch in that state from being created.

The floor a reviewer reads is minimumReleaseAge on the rule that matches the dependency. The headline floors are 7 days on majors and on the attended tail rules, 2 days on the LiteLLM rule, and 3 days on the rollup.

If a workflow ever pushes a commit status to a renovate/* branch, re-derive this section. Do the same if internalChecksAsSuccess changes or if a rule sets automergeType: branch. Nothing in CI reports that change.

prCreation changes when a pull request appears, not whether it automerges. It does not override automerge, automergeType or platformAutomerge.

Key Committed value Default
prConcurrentLimit 10 10
branchConcurrentLimit unset the value of prConcurrentLimit
prHourlyLimit 0, which means no limit 4 in config.js

Renovate opens new pull requests for this repository only inside the update window, Friday through Monday in America/New_York. The worker visits the repository about every two hours, so the window gets about 48 runs. prHourlyLimit is 0, and prConcurrentLimit bounds the CI load instead.

An open pull request holds one of the ten slots until it merges or closes. An attended pull request holds its slot until a person merges it. A pull request with a red required check holds its slot until someone repairs it. When all ten slots are full, Renovate opens no more pull requests in that run. Outside the window, Renovate opens no pull request for a branch that already exists. A held update therefore waits for the next window.

Renovate sorts digest updates before all other updates. For this reason, the five buckets and the catch-all also take digest updates. Without them, each digest takes a slot of its own, and the digests fill the slots before any grouped update.

branchConcurrentLimit counts every renovate/* branch, also a branch without a pull request. When the count reaches the limit, Renovate creates no new branch. The Dependency Dashboard then lists the held updates under “Rate-Limited”. A vulnerability remediation ignores this limit.

If the dashboard lists many updates under “Rate-Limited” or “Awaiting Schedule” after a window, count the branches and the open pull requests before you raise a limit:

Terminal window
git ls-remote --heads origin 'renovate/*'
gh pr list --state open --author app/fzymgc-renovate --json headRefName --jq '.[].headRefName'

If ten pull requests are open, the pull request limit held the rest. Merge or close the open pull requests, and repair the ones with a red required check. If fewer than ten are open, read the gate in “Cooling-off: what gates a pull request”.

Renovate injects vulnerabilityAlerts last, as a forced block, so no packageRules entry outranks it by position. The only lever is the value. .vulnerabilityAlerts.automerge is false, so a remediation pull request opens at once and waits for a human. The block sets reviewers to seanb4t, so every remediation requests a review, also in a tree that names no reviewer. The block sets no platformAutomerge, so the attended rules keep theirs. It forces minimumReleaseAge to null, so no release-age floor applies to a remediation.

The account-wide config.js sets vulnerabilityAlerts.automerge to true. The repository value false overrides it. If you remove the repository key, the account-wide true applies again.

Keep .vulnerabilityAlerts.automerge at false. A true there lets a remediation for LiteLLM automerge past the attended lane. Nothing asserts this value, so review a change to it as a change to the LiteLLM posture. A narrower control scoped to a few dependency names needs force, which is a global-only option and leaves a permanent configuration error on the dashboard.

The ClickStack alert Renovate cache volume inode exhaustion fires when the cache volume has fewer than 1,000,000 free inodes for 30 minutes.

Alert value Cause Action
Falling toward 1,000,000 Real inode growth Run the renovate-cache-prune Job (see Prune the cache)
0 Series missing. A FailedMount event says no Pending workload pods Delete finished prune pods (see Prune the cache)

A value of 0 follows a kubelet restart while a finished prune pod still uses the volume. Renovate keeps working. The kubelet retries the mount after the pod is gone, and the series returns.

Constraints when editing .github/renovate.json

Section titled “Constraints when editing .github/renovate.json”

The rule description fields are one or two sentences each. The mechanics a description cannot carry, and that an edit can break, are here.

Renovate merges later matches over earlier ones, so a rule’s index is part of what it does. Nothing asserts the order. This table is the record of it.

Rule Position Because
terraform core after terraform providers it peels hashicorp/terraform back out of the providers group
the major guard (matchUpdateTypes: ["major"], no manager matcher) after every rule that sets automerge: true, the last being python dependencies, and before the mend-renovate-ce guard and the attended tail every automerging group declares no matchUpdateTypes, so without this guard last-match-wins hands each of them the major lane. A later rule that weakens automerge, platformAutomerge, minimumReleaseAge, reviewers or additionalBranchPrefix reopens the lane
the pre-1.0 guard (the one rule with matchCurrentVersion, matchUpdateTypes: ["minor"], no manager matcher) after the five buckets and the catch-all, after every automerge: true rule, before the mend-renovate-ce guard and the attended tail for a 0.y.z pin the minor is the breaking bump. Sitting after the buckets makes its own group the last one to decide the branch. A bucket placed after it overwrites its group
mend-renovate-ce guard after the container-image and chart groups all of those automerge that chart on a short floor
the five blast-radius buckets and the catch-all before the attended tail rules the buckets decide which branch a dependency lands in. The attended rules decide its posture
the k3s rule (matchFileNames on the system-upgrade Plans) after the argocd/** bucket and the pre-1.0 guard the bucket groups the Plans into argocd-minor-patch, which automerges. An earlier position hands the k3s version to that lane
the pnpm rule (matchDepNames: ["pnpm"]) after the five buckets and the catch-all the buckets put each pnpm pin into the branch of its tree. A pnpm update without new lock files turns that whole branch red
the pnpm lock file maintenance rule (matchManagers: ["npm"], matchUpdateTypes: ["lockFileMaintenance"]) anywhere no other rule sets a merge key for lock file maintenance of the npm manager
the custom.regex plus ansible/** guard after the buckets, before the attended tail it overrides the grouping-only ansible/** bucket and must not overwrite an attended posture
the attended tail (key-affecting encoders, clickstack observability plane, temporal worker controller, valkey operator, litellm data plane) after every group and bucket they override an earlier position is silently overridden. Only the Image Updater handover rule follows them, and it matches no path of an attended rule
the four-pin extractVersion rule anywhere it sets no posture key. Its scope is load-bearing: it is matched by matchDepNames only, and a manager or file matcher on it strips the prefix from pins that keep theirs
vault agent (matchDepNames: ["hashicorp/vault"] under ansible/**) after the Ansible guard and the five buckets it sets only groupName and groupSlug. A bucket placed after it puts the Vault pin back into ansible-minor-patch
lightningstream (lockstep) anywhere it sets the same posture keys as the Ansible guard, and no attended tail rule names the dependency
terraform cli (lockstep) after the five buckets and the catch-all it sets only groupName and groupSlug. A bucket placed after it puts the two pins on different branches again
the two changie fragment rules (matchFileNames on one app, with postUpgradeTasks) anywhere they set only postUpgradeTasks, and no other rule sets it
the buf CLI rule (matchFileNames: ["apps/dns-control/tools/go.mod"], enabled: false) after the gomod rule with matchDepTypes: ["indirect"] that rule sets enabled: true for every indirect gomod dependency. An earlier position lets it enable the dependencies of buf again
vite toolchain anywhere it sets only groupName and groupSlug, only for major updates, and no other rule in the file sets a group for a major npm update
the typescript hold (allowedVersions: "<7") anywhere it sets only allowedVersions, and no other rule sets it for typescript
the cloudflare hold (allowedVersions: "<5.26.0") anywhere it sets only allowedVersions, and no other rule sets it for cloudflare/cloudflare
the ansible label rule (matchFileNames: ["ansible/**"], addLabels) anywhere addLabels merges with the labels of the other rules, so no position overrides it
the engram chart rule (an exact matchPackageNames) after the major guard, the pre-1.0 guard and the attended tail, and before the Image Updater handover rule it lifts the floor and the review of both guards for the first-party engram chart alone. An earlier position hands the chart back to a guard
the Image Updater handover rule (matchDatasources: ["docker"], an exact matchPackageNames list, enabled: false) last Argo CD Image Updater owns these first-party images (Argo CD Image Updater). A later rule that matches one of them can enable it again, and then two updaters move one pin

Renovate merges branches, not dependencies

Section titled “Renovate merges branches, not dependencies”

A branch automerges only when every upgrade in it automerges, so one attended upgrade turns automerge off for the whole branch. A branch takes its reviewers from one upgrade, so a co-grouped unattended upgrade can decide who is asked to look. See the Renovate documentation on grouping for the current behaviour.

Two rules exist because of this:

  • The pre-1.0 guard carries its own groupName and groupSlug, so its updates never share a branch with an unattended bucket.
  • The major guard carries additionalBranchPrefix: "major-". A branch name is branchPrefix + additionalBranchPrefix + branchTopic, and for a grouped update the topic is the groupSlug. The constant prefix separates the attended major lane from every other lane while each tree keeps its own slug. Keep the value a literal, not a template, so the split is readable from the file.

The major guard does not get a group of its own. A group would collapse every attended major in the estate into one branch and discard the per-tree split.

managerFilePatterns is a glob unless it is /-delimited

Section titled “managerFilePatterns is a glob unless it is /-delimited”

An entry is a minimatch glob unless it is wrapped in /.../, in which case it is a regex. A bare argocd/**/*.ya?ml$ is a glob whose trailing $ is a literal character, and it matches nothing. renovate-config-validator accepts a pattern that matches nothing.

Assert a pattern by file count against git ls-files, never by a validator exit code:

Terminal window
git ls-files | rg -c '^argocd/.*\.ya?ml$' # the count the argocd and kubernetes managers reach

matchFileNames tests the package file first and then the dependency’s lock files. That is how tf/cloudflare/.terraform.lock.hcl reaches the Terraform bucket. A path matcher survives a provider rename or a new registry. A name matcher does not, and Renovate rewrites a bare provider name to hashicorp/<name>, which makes a /^hashicorp// matcher look wider than it is.

matchRegexOrGlobList ORs positive patterns and ANDs !-prefixed ones. That is what makes the catch-all bucket read as any path in none of the five named buckets.

postUpdateOptions stays inside packageRules

Section titled “postUpdateOptions stays inside packageRules”

postUpdateOptions is mergeable and array-typed. A top-level entry concatenates into every manager’s resolved configuration. It is written inside packageRules, scoped to matchManagers: ["gomod"]. Do not hoist it.

gomodTidy needs no allowedCommands entry. That allowlist governs postUpgradeTasks only. gomodTidyE is rejected: its -e flag reports a partial write as a completed one.

The two gomod rules stay two rules. Folding matchDepTypes: ["indirect"] into the tidy rule narrows the tidy to indirect updates only.

Two rules give each update under apps/dns-control/ and apps/hl-assets-uploader/ a postUpgradeTasks command. The command writes a changie fragment of kind Dependencies to .changes/unreleased/, and fileFilters adds the file to the commit. The pr-gates member changelog-fragment fails an app change that has no fragment. docs/operations/releases.md describes the fragments.

The worker runs the command only when the rendered command matches an anchored entry of allowedCommands in argocd/cluster-app/templates/renovate.yaml. Renovate renders the templates first, so the entry sees the real dependency name and version. Its character classes refuse a name or a version that can leave the quotes of bash -c or of the printf argument. A refused command puts an artifact error on the pull request, and the fragment member then fails it.

  • Change the command in .github/renovate.json and the entry in the cluster template in the same pull request. After the merge, make sure that ArgoCD synced the renovate application before the next Renovate run. Until the sync, the worker refuses the new command.
  • The fragment has no time: line, and its file name comes from the dependency name and the version. A new run of the branch therefore writes the same file, and Renovate finds no change to push.
  • A lock file maintenance update has no dependency name. For it, the command writes Updated the lock file of <package file>. to the file <app>-Dependencies-lock-file-maintenance.yaml.
  • A digest update has no newVersion. For it, the command names newDigestShort, the first seven characters of the new digest. Without that value, the command renders Updated node to ., and the cluster entry refuses it.
  • For a dependency with a range, such as ^8.0.0, the fragment names Renovate’s newVersion. The lock file can resolve a later version. Before a release, make sure that the fragment names the version in the lock file.

To make sure that the cluster entry allows a command, render the template and test the command. Replace <command> with a rendered command, for example one from the worker log:

Terminal window
helm template argocd/cluster-app --show-only templates/renovate.yaml \
| yq '.spec.sources[0].helm.valuesObject.renovate.config' > /tmp/renovate-config.js
CMD='<command>' node -e '
const m = new (require("module"))();
m._compile(require("fs").readFileSync("/tmp/renovate-config.js", "utf8"), "config.js");
console.log(m.exports.allowedCommands.some((p) => new RegExp(p).test(process.env.CMD)));'

The command prints true when an entry allows the command.

Some packages accept only one major version of another package. Each major of @sveltejs/vite-plugin-svelte accepts one major of vite. Version 6 accepts vite ^6.3.0 || ^7.0.0, and version 7 accepts vite ^8.0.0. Renovate does not read the peer ranges of other packages, so it opens a separate branch for each major. Each of those branches is broken alone.

The rule vite toolchain puts the major updates of the two packages into one branch, renovate/major-major-vite-toolchain. Renovate adds major- to the slug of a grouped major update, and the major guard adds the second one. A group holds only the releases that exist at the time of the run. If a vite major ships before the plugin major that accepts it, the branch holds vite alone. When the plugin major ships, Renovate adds it to the same branch.

Both web apps set strictPeerDependencies: true in web/pnpm-workspace.yaml. pnpm checks peer ranges only when it resolves, so the setting fails the lock file update of a Renovate branch that leaves a peer unmet, and Renovate reports an artifact error on the pull request. A frozen install, such as the one in each app Dockerfile, does not resolve and does not check peers. Without the setting, a split branch passes every check, because pnpm only prints a warning.

To check the peers of a web app by hand, run this command in its web/ directory:

Terminal window
pnpm peers check

If another pair of packages moves in lockstep, add both names to matchPackageNames of the vite toolchain rule, or add a rule of the same shape with its own groupName.

A group helps only when a release of the other package accepts the new major. typescript 7 is the native compiler, and its package has no JavaScript compiler API. svelte-check, the type checker that both web apps run, works inside that API, so it accepts only typescript ^5.0.0 || ^6.0.0. In both web apps, svelte-check is the only package that loads typescript. Vite and Vitest remove types without it.

The typescript hold sets allowedVersions: "<7" for apps/*/web/**, so Renovate proposes typescript 6 at most. Remove the hold only when both of these are true:

  1. A typescript 7 release has a stable compiler API. The TypeScript team plans it for 7.1.
  2. A svelte-check release accepts typescript 7 in its peer range.

Then add typescript and svelte-check to a lockstep group. A person closed the typescript 7 pull request #2847 without a merge, so Renovate ignores typescript 7. After you remove the hold, give pull request #2847 a new title to clear that ignore, as the Renovate documentation describes.

svelte-check also has a --tsgo mode that needs typescript 6 and an npm alias for typescript 7. It uses an experimental API, so the web apps do not use it.

From 5.26.0, the cloudflare/cloudflare provider adds the attribute include_shadow_metadata to cloudflare_dns_record. Every record in state then plans an in-place update. The update fails on the records that Email Routing manages, so each later apply of main-cluster-cloudflare fails too. The hold sets allowedVersions: "<5.26.0". Remove it when the upstream issues cloudflare/terraform-provider-cloudflare#7396 and #7387 close. Before you merge the first update past the hold, read the speculative plan of main-cluster-cloudflare, because that workspace applies on merge.

The valkey-operator chart has its own attended group, valkey-operator, outside the pre-1.0 group. Between chart 0.3.0 and 0.7.0 the CRD moved the TLS setting from spec.tls to spec.networking.tls. ArgoCD applies the new CRDs and the new operator in one sync. The operator then reads no TLS setting and restarts the Valkey nodes without TLS, and oauth2-proxy loses its session store. To upgrade, stop the operator first: set its replicaCount to 0, move the chart and the ValkeyCluster resources to the new schema, and then start the operator again.

ignorePaths also holds .planning/spikes/** and .claude/skills/spike-findings-selfhosted-cluster/sources/**. These files are the sources that the spikes ran, and their pins are part of what each spike recorded. A version update there makes a source disagree with its own findings.

The pnpm version appears in packageManager in apps/dns-control/web/package.json, apps/hl-assets-uploader/web/package.json and website/package.json, and in mise.toml and mise.lock. The pnpm rule puts all of these on one attended branch, renovate/pnpm. The app Dockerfiles have no pnpm version. They run corepack install, which reads packageManager.

From pnpm 12, each pnpm-lock.yaml records the pnpm version under packageManagerDependencies. pnpm install --frozen-lockfile fails when that record does not match packageManager. The worker keeps the record when it updates a lock file for a dependency. It does not change the record for a pnpm update, and it leaves the record out when lock file maintenance rebuilds a lock file. So the renovate/pnpm branch and the renovate/lock-file-maintenance branch arrive with red checks, and both are attended. Before you merge either of them, run this on its branch from the repository root:

Terminal window
for d in apps/dns-control/web apps/hl-assets-uploader/web website; do
(cd "$d" && mise x -- pnpm install --lockfile-only)
done

Make sure that each lock file diff changes only the packageManagerDependencies block and the pnpm packages under it. On the lock file maintenance branch, the diff only adds lines. Then commit the three lock files. After your push, Renovate no longer updates the branch, so merge it soon.

The deb datasource: one ARG, three apt pins, one dependency

Section titled “The deb datasource: one ARG, three apt pins, one dependency”

apps/hl-assets-uploader/Dockerfile pins libheif to an exact Debian package revision through ARG LIBHEIF_VERSION. The annotation sits above the one declaration that carries a value. The re-declarations that carry the ARG into later stages are not annotated, because annotating them reports four pins where there is one. The one value feeds libheif-dev in the builder stage and libheif1 plus libheif-plugin-libde265 in the runtime stage, so all three move in one commit.

The registry URL is https://deb.debian.org/debian?suite=trixie&components=main&binaryArch=amd64. Each part is load-bearing:

Decision Why the alternative is wrong
suite=trixie golang:1.26 and debian:trixie-slim are both Debian 13. Another suite proposes a different package line, which can be a downgrade that validates clean
components=main only the suite’s Packages index lands on the cache PVC. Other components add index bytes for packages this repository does not install
one binaryArch registryStrategy: 'merge' unions version sets across registry URLs, so a second architecture widens the candidate set
versioning=deb Debian revisions such as 1.19.8-1+deb13u1 are not semver. The default docker versioning mis-orders the -N+debXuY suffix

A bump here is a key-affecting encoder change. The key-affecting encoders (hl-assets) rule brakes it, and it matches only because custom.regex is in that rule’s matchManagers. apps/hl-assets-uploader/testdata/golden.webp.sha256 and golden.key are stop-the-line artifacts. A bump that moves a golden hash is stopped and deferred with the hash delta as evidence. It is never re-baselined to make a check pass. See hl-assets Uploader.

hl-assets-uploader stores each photo under the object key HMAC(salt, the stored WebP bytes). The encoders make those bytes: github.com/chai2010/webp, github.com/strukturag/libheif and the libheif1 Debian package. If a bump changes one output byte, the same photo gets a new key. The old URL keeps working, but deduplication stops for that photo.

The encoders rule attends every bump and holds a 30-day release-age floor. Before you merge a bump, re-baseline apps/hl-assets-uploader/testdata/golden.webp.sha256 and golden.key. Record the new baseline as an accepted decision that names what changed, not as a routine test fix.

One ARG LIBHEIF_VERSION pins libheif-dev, libheif1 and libheif-plugin-libde265. Only libheif1 carries the annotation, so Renovate reports only libheif1.

Annotated Ansible pins: the hazard is the minor lane

Section titled “Annotated Ansible pins: the hazard is the minor lane”

An annotated Ansible pin is extracted by the custom.regex manager and by nothing else. The major lane is already attended by the major guard. The minor lane is the exposure: a cilium_version minor is a minor under both docker and semver versioning, and without a guard it rides the account-wide rollup at automerge: true with nobody named. An Ansible-only diff carries no Go and no lockfile, so pr-gates passes it.

The guard is scoped by manager and by path. matchManagers: ["custom.regex"] alone also catches the shell _VERSION pins and the Dockerfile deb pin, which the encoders rule guards instead.

.github/workflows/build-lightningstream.yml pins LIGHTNINGSTREAM_VERSION, and the Ansible role pins lightningstream_version. The two are one release train. The fourth custom manager reads the workflow pin, and the lightningstream (lockstep) rule moves both in one pull request.

The workflow pin holds no leading v. The workflow prepends one at each use site. The four-pin extractVersion rule strips the prefix and reaches this dependency by name, so the bare form keeps the two pins equal as strings. GOLANG_BUILDER_IMAGE in the same block carries # renovate-exempt: because it must follow upstream’s Dockerfile at the pinned tag.

Forward coverage: rules that read as guards but match nothing today

Section titled “Forward coverage: rules that read as guards but match nothing today”

Each is listed so that the rule reaches the missing half on the day it appears. Do not describe any of them as guarded.

Listed Reached by no manager because Detector until then
helmv3 in the helm charts rule argocd/cluster-app/Chart.yaml declares no dependencies: block none
pip_requirements in python dependencies no tracked requirements.txt exists. The three Python projects declare dependencies in pyproject.toml, which is the pep621 manager none
/^renovate$/ in ci/cd components the self-hosted chart’s depName is mend-renovate-ce the attended mend-renovate-ce rule
libheif-plugin-libde265 in the encoders rule only libheif1 carries the annotation the golden-file test, plus the attended libheif1 bump the other pins move with
argocd/app-configs/litellm/** in the LiteLLM rule that directory carries no version pin today none

ci/cd components is not repointed at mend-renovate-ce. That rule automerges, and repointing it overwrites the attended guard.

Pins under argocd/app-configs/clickhouse-mcp/ sit outside the clickstack-chart/** glob and resolve to the argocd manifests group. They are client pins, and they are excluded from the attended ClickStack rule on purpose. The ClickHouse client image in argocd/app-configs/clickstack/kustomization.yaml is the exception. It must equal the server tag, so the attended rule lists that file.

  1. Put # renovate: datasource=<ds> depName=<name> on the line directly above the key. Add versioning=<v> and then registryUrl=<url> when the datasource needs them, in that order.
  2. If the pin cannot be annotated, put # renovate-exempt: <reason> above it instead.
  3. Make sure that the pin appears on the Dependency Dashboard issue after the next run, or run the worker log query below and look for the depName.
  1. Decide the posture keys the rule sets. An attended rule sets automerge: false, platformAutomerge: false, a minimumReleaseAge and a reviewers list.

  2. Place it by the ordering table above. A rule that sets posture sits after every rule it must override.

  3. Resolve the committed rules last-match-wins for one real pin in each lane the rule touches, and make sure that no other lane changed.

  4. Validate the file:

    Terminal window
    npx --yes --package renovate renovate-config-validator --no-global .github/renovate.json

    Without --no-global, the validator checks the file as the global configuration of the worker. The validator catches a malformed edit. It does not prove a pattern matches anything.

Confirm a repository-configuration key reaches the worker

Section titled “Confirm a repository-configuration key reaches the worker”

The worker logs at LOG_LEVEL=info, so the resolved-configuration dump is not ingested. The resolved configuration still reaches default.otel_logs inside the payload of some error lines, so this reading is opportunistic. Use a key whose committed value differs from its default, for example prCreation:

Terminal window
kubectl exec -n clickstack "$(kubectl get pod -n clickstack \
-l clickhouse.com/role=clickhouse-server -o name | head -1 | cut -d/ -f2)" \
-- clickhouse-client --query "
SELECT Timestamp,
JSONExtractString(Body, 'logContext') AS logContext,
extract(Body, '\"prCreation\":\"[a-z-]+\"') AS prCreation
FROM default.otel_logs
WHERE ResourceAttributes['k8s.pod.name'] LIKE 'renovate-ce-%'
AND JSONExtractString(Body, 'repository') = 'fzymgc-house/selfhosted-cluster'
AND Body ILIKE '%prCreation%'
AND Timestamp > now() - INTERVAL 7 DAY
ORDER BY Timestamp DESC LIMIT 20"

The timestamp column is Timestamp, never TimestampTime.

Result Reading
"prCreation":"not-pending" The key reached the worker
"prCreation":"immediate" The wrapper dropped the key, or the row predates the change. Compare the row’s Timestamp with the merge
No rows No configuration-bearing line was logged in the window. Widen the window or wait for another cycle
Terminal window
kubectl -n renovate logs deploy/renovate-ce --since=24h | rg 'fzymgc-house/selfhosted-cluster'

For history beyond the pod, query default.otel_logs with ResourceAttributes['k8s.pod.name'] LIKE 'renovate-ce-%' and JSONExtractString(Body, 'repository') = 'fzymgc-house/selfhosted-cluster'.

The server has an admin API. The public tunnel forwards only /webhook, and the NetworkPolicy admits only cloudflared, so reach the API through a port-forward. Use it when a Dependency Dashboard tick or a webhook did not start a run.

  1. Open the port-forward and read the secret:

    Terminal window
    kubectl -n renovate port-forward svc/renovate-ce 18080:80 &
    TOKEN=$(kubectl -n renovate get secret renovate-app -o jsonpath='{.data.mendRnvServerApiSecret}' | base64 -d)
  2. Read the job queue:

    Terminal window
    curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:18080/system/v1/jobs/queue | jq
  3. Read the recent jobs of one repository. The repository slug is not URL-encoded:

    Terminal window
    curl -s -H "Authorization: Bearer $TOKEN" \
    'http://127.0.0.1:18080/api/v1/repos/fzymgc-house/selfhosted-cluster/-/jobs?limit=10' | jq
  4. If no job is queued for the repository, queue a run:

    Terminal window
    curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
    -d '{}' http://127.0.0.1:18080/api/v1/repos/fzymgc-house/selfhosted-cluster/-/jobs/run
  5. Stop the port-forward with kill %1.

The CronJob runs on Sunday. To run it now:

Terminal window
kubectl -n renovate create job --from=cronjob/renovate-cache-prune renovate-cache-prune-manual
kubectl -n renovate logs job/renovate-cache-prune-manual -f # prints df -i before and after

To clear a missing inode series after a kubelet restart:

Terminal window
kubectl -n renovate delete pod \
-l app.kubernetes.io/name=mend-renovate-ce-cache-prune --field-selector status.phase=Succeeded

Read the free inodes on the box at any time:

Terminal window
kubectl -n renovate exec deploy/renovate-ce -- df -i /tmp/renovate

This is cache maintenance inside a PVC. No kubectl apply is involved and none is permitted. The PVC is shared by every repository the worker serves, so one rm against the wrong path degrades the cache for all of them.

Symptom: a Renovate-authored gomod pull request arrives with a go.sum that does not verify, and the worker log names stdlib packages as missing:

context: package context is not in std (/tmp/renovate/cache/others/go/pkg/mod/golang.org/toolchain@v0.0.1-go<version>.linux-arm64/src/context)

Cause: a toolchain directory under /tmp/renovate/cache/others/go/pkg/mod/golang.org/ has no src/ tree, and Go still treats it as complete. .ziphash is Go’s downloaded-and-verified marker. Go reads it and never re-extracts, so a half-empty extraction is trusted permanently. The worker cleanup keeps /tmp/renovate/cache, so the entry survives every run.

Judge intactness by the src/ tree, never by the .zip. Go deletes the archive after a successful extraction, so a healthy toolchain has no .zip either. This is not a missing gomodTidy step. go get -t ./... already runs and fails on the broken toolchain.

  1. Read every toolchain entry and its src/ state:

    Terminal window
    kubectl -n renovate exec deploy/renovate-ce -- sh -c '
    for d in /tmp/renovate/cache/others/go/pkg/mod/golang.org/toolchain@*; do
    if [ -d "$d/src" ]; then echo "$d src: present ($(ls "$d/src" | wc -l) entries)"; else echo "$d src: ABSENT"; fi
    done
    ls -l /tmp/renovate/cache/others/go/pkg/mod/cache/download/golang.org/toolchain/@v/
    '
  2. Continue only if one entry reads ABSENT and the toolchain that apps/hl-assets-uploader/go.mod names reads present. If the named toolchain is absent too, the purge scope is wider than this procedure. Stop.

  3. Remove the poisoned extraction and its marker, and those two paths only. Write the version out in full. A glob such as toolchain@v0.0.1-go1.26.* reaches the intact entries.

    Terminal window
    V=<version, for example 1.26.5>
    kubectl -n renovate exec deploy/renovate-ce -- sh -c "
    rm -rf /tmp/renovate/cache/others/go/pkg/mod/golang.org/toolchain@v0.0.1-go$V.linux-arm64
    rm -f /tmp/renovate/cache/others/go/pkg/mod/cache/download/golang.org/toolchain/@v/v0.0.1-go$V.linux-arm64.ziphash
    "
  4. Re-run step 1. The removed entry is gone and every other entry still reads present.

  5. On the next Renovate-authored gomod pull request, check out its head and run go mod verify. Then re-run module resolution and make sure that go.sum is unchanged.

Running the procedure twice is safe. Step 2 finds no absent entry and the procedure stops.

apps/hl-assets-uploader/go.mod carries a go directive and no toolchain directive. The go line names the toolchain Go must fetch. A toolchain directive is for a toolchain newer than the go line requires, and it is a second version-bearing line that can drift. Renovate tracks the go directive as a dependency (datasource: golang-version), so bumps to it are ordinary updates.

Compare three refs before concluding anything: the branch tip, origin/main, and refs/pull/N/merge. The pull_request event builds the merge of head into base, so a pull request with no Go change goes red when it inherits a broken go.sum from main.

The pin is targetRevision in argocd/cluster-app/templates/renovate.yaml. This chart carries the pod that does the proposing, so a bad values-key move takes out the mechanism that proposes its own repair.

  1. Make sure that the pull request is Renovate-authored. A human-authored bump proves nothing about the manager.
  2. Render both sides with helm template at the current pin and at the proposed one, and read the diff for moved or renamed values keys.
  3. Merge attended, under the mend-renovate-ce guard’s named reviewer. Never by an agent.