Renovate Operations
Renovate is the dependency-update bot. A self-hosted Mend Renovate CE worker runs in the cluster
and serves three GitHub accounts. This page states what exists, the constraints that bind an edit
to the repository configuration, and the procedures an operator runs. kubectl commands assume
--context fzymgc-house.
What exists
Section titled “What exists”Quick reference
Section titled “Quick reference”| Property | Value |
|---|---|
| Namespace | renovate |
| Workload | deploy/renovate-ce |
| Chart and pin | mend-renovate-ce, targetRevision in argocd/cluster-app/templates/renovate.yaml |
| Accounts discovered | fzymgc-house/*, seanb4t/*, holomush/*, except fzymgc-house/node-bootstraps (mendRnvAutoDiscoverFilter) |
| Account-wide configuration | config.js, embedded in the same manifest |
| Repository configuration | .github/renovate.json |
| Schedule | * * * * 0,1,5,6 (Friday through Monday), America/New_York |
| Cache PVC | renovate-ce-cache-hi, 32Gi, longhorn-highinode, mounted at /tmp/renovate |
| Cache root that survives cleanup | /tmp/renovate/cache |
| State PVC | renovate-ce-db, 1Gi, mounted at /db (mendRnvSqliteFilePath) |
| Cache prune | CronJob renovate-cache-prune, Sundays 03:00, deletes cache files not read in 14 days |
| GitHub App credential | Vault fzymgc-house/cluster/renovate through ExternalSecret renovate-app |
| GHCR credential | Vault fzymgc-house/cluster/ghcr/pull-secret through ExternalSecret renovate-ghcr |
| Manifests | argocd/app-configs/renovate/ |
| Admin API | System and jobs APIs, Bearer mendRnvServerApiSecret from Secret renovate-app. Reach it with kubectl port-forward only |
The worker cleans /tmp/renovate/repos and /home/ubuntu after every run (mendRnvWorkerCleanup
and mendRnvWorkerCleanupDirs). It keeps /tmp/renovate/cache. That is the purpose of the PVC,
and it is also why a corrupt cache entry outlives every run.
Two configurations resolve, and only one is editable from this repository
Section titled “Two configurations resolve, and only one is editable from this repository”The account-wide config.js in argocd/cluster-app/templates/renovate.yaml contributes the first
packageRules entries in the effective list. Every repository the worker serves reads it, and a
change to it restarts the pod for three accounts. Do not edit it for a rule that concerns this
repository alone.
Two consequences bind every rule in .github/renovate.json:
- The account-wide
all-minor-patchrollup is asserted for every minor and patch update before any repository rule runs. A repository rule can re-group or override it. It cannot remove it. This is why the blast-radius split needs a catch-all bucket. - The account-wide root sets
automerge: true,platformAutomerge: trueandreviewers: []. An attended rule must set both automerge keys tofalseand name a reviewer. One key alone leaves the other in force.
What the repository configuration decides
Section titled “What the repository configuration decides”The postures below are what the committed rules resolve to. Resolve the rules last-match-wins per update class when you need the posture of one dependency. Do not read the array in order.
| Update class | Posture | Rule |
|---|---|---|
| Any major | attended, 7-day floor, reviewer seanb4t, branch prefix major- |
the rule with matchUpdateTypes: ["major"] and no manager matcher, after every automerge: true rule |
Any minor of a 0.y.z pin |
attended, 7-day floor, reviewer seanb4t, own branch pre-1-0-minor |
the one rule with matchCurrentVersion |
| Minor, patch and digest elsewhere | automerge, floor from the matching group, one branch per tree | the five blast-radius buckets and the catch-all |
Dependencies of the buf CLI in apps/dns-control/tools/go.mod |
disabled, except github.com/bufbuild/buf and google.golang.org/protobuf |
the rule with matchFileNames: ["apps/dns-control/tools/go.mod"] |
mend-renovate-ce chart |
attended, 7-day floor | the mend-renovate-ce guard |
| Annotated Ansible pins | attended, 7-day floor | the custom.regex plus ansible/** guard |
k3s (spec.version of the two system-upgrade-controller Plans) |
attended, 7-day floor, one group k3s for both Plans |
the k3s rule (docs/operations/k3s.md) |
pnpm (the three packageManager fields and the mise.toml pin) |
attended, 7-day floor, one group pnpm. The reviewer regenerates the lock files |
the pnpm rule (pnpm updates) |
Lock file maintenance of the three pnpm-lock.yaml files |
attended. The reviewer restores the pnpm version record | the rule with matchUpdateTypes: ["lockFileMaintenance"] (pnpm updates) |
| LiteLLM, ClickStack, hl-assets encoders | attended, 7 or 30-day floor, own group | the attended tail rules |
temporal-worker-controller and temporal-worker-controller-crds charts |
attended, 7-day floor, one group for both charts, chart versions only (no v image tags) |
the temporal worker controller rule in the attended tail |
valkey-operator chart |
attended, 7-day floor, own group valkey-operator |
the valkey operator rule in the attended tail (Holds and separate groups) |
ghcr.io/seanb4t/charts/engram, every update type |
automerge, no floor, own group engram chart |
the engram chart rule |
Charts that a kustomization inflates through helmCharts:, such as external-dns |
automerge, 3-day floor, group helm charts, as the charts of the argocd manager |
the rule after helm charts that matches the kustomize manager and the helm datasource. The ClickStack rule attends its own chart |
cloudflare/cloudflare Terraform provider |
held below 5.26.0 | the cloudflare hold (Holds and separate groups) |
Terraform CLI (terraform in mise.toml and terraform_version in tf/hcp-terraform/workspaces.tf) |
one group terraform-cli-lockstep for both pins |
the terraform cli (lockstep) rule |
velero/velero-plugin-for-aws |
disabled | its own rule |
| Python runtime and Node.js majors | ignored or LTS only | their own rules |
| Vulnerability alerts | open at once, no automerge, no floor | vulnerabilityAlerts at the top level |
Custom managers
Section titled “Custom managers”Six customManagers entries reach pins that no stock manager reads:
| Files | Pin shape | Notes |
|---|---|---|
*.sh |
# renovate: line above a _VERSION= assignment |
no Makefiles exist in this repository |
ansible/roles/*/defaults/main.yml, ansible/inventory/{group_vars,host_vars}/*.yml |
# renovate: line above a YAML key |
default versioning docker |
apps/*/Dockerfile |
# renovate: line above an ARG |
datasource deb, versioning deb, one registry URL |
.github/workflows/*.yml |
# renovate: line above an env: key |
the stock github-actions manager reads uses: lines only |
argocd/app-configs/system-upgrade/plan-*.yaml |
# renovate: line above version: |
the k3s version of the cluster. The kubernetes manager skips the untagged rancher/k3s-upgrade images |
argocd/app-configs/clickstack-chart/values.yaml |
# renovate: line above tag: |
the collector image tag, which has no sibling repository for helm-values to read. The ClickStack rule groups it with the HyperDX tag. The annotation names the Docker Hub repository, because docker.clickhouse.com gives no release dates and the release-age floor then lets the two tags drift apart |
The # renovate: line must sit directly above the key it annotates. The regex requires adjacency.
A blank line or another comment between them un-annotates the pin, and the configuration still
validates. The optional fields follow the order versioning= then registryUrl=, because that is
the order the regex accepts.
A pin that cannot be annotated carries # renovate-exempt: <reason> on the line above it. Absence
of governance and absence of updates are different claims, and the exemption keeps them apart.
The string matchManagers sees for a custom manager is custom.regex, with the prefix. The
name-anchored groups do not reach annotated pins, because the annotations emit names such as
cilium or k3s-io/k3s that no matchPackageNames entry matches.
Cooling-off: what gates a pull request
Section titled “Cooling-off: what gates a pull request”| Key | Committed value | Default |
|---|---|---|
prCreation |
not-pending |
immediate |
internalChecksAsSuccess |
true |
false |
internalChecksFilter |
unset | strict |
prNotPendingHours |
unset | 25 |
internalChecksFilter is unset because its default is already strict. A key written at its own
default asserts nothing.
Under strict, Renovate still selects the highest candidate release. It flags the update
pendingChecks when every candidate sits inside its minimumReleaseAge window. Renovate creates no
branch for a flagged update, and it does not update an existing branch. This key withholds a pending
release, not prCreation. Renovate writes the renovate/stability-days status on each branch. The
status is green when the release age elapsed before Renovate looked.
prCreation: not-pending opens a pull request only when the branch status is green. No CI runs on a
renovate/* branch before its pull request exists. Every push: trigger in .github/workflows/ is
scoped to main or to tags, and the pull_request: workflows need a pull request. The only status
on the branch is renovate/stability-days.
internalChecksAsSuccess: true makes Renovate count its own green renovate/* statuses as a green
branch. Without it, Renovate reads such a branch as yellow. After the release age elapsed, a yellow
branch never gets a pull request, and prNotPendingHours does not apply. Outside the update window,
Renovate opens no pull request for an existing branch. The branch then stays without a pull request
and counts toward branchConcurrentLimit until someone deletes it.
internalChecksAsSuccess changes how Renovate reads the branch, not what GitHub requires before a
merge. The ruleset on main requires a pull request and the pr-gates and Octopus Review checks. It
requires no approving review. No rule sets
automergeType: branch, so every automerge goes through a pull request and its checks.
prNotPendingHours is unset. It applies only while the release-age status is yellow, and strict
keeps a branch in that state from being created.
The floor a reviewer reads is minimumReleaseAge on the rule that matches the dependency. The
headline floors are 7 days on majors and on the attended tail rules, 2 days on the LiteLLM rule, and
3 days on the rollup.
If a workflow ever pushes a commit status to a renovate/* branch, re-derive this section. Do the
same if internalChecksAsSuccess changes or if a rule sets automergeType: branch. Nothing in CI
reports that change.
prCreation changes when a pull request appears, not whether it automerges. It does not override
automerge, automergeType or platformAutomerge.
Branch and pull request limits
Section titled “Branch and pull request limits”| Key | Committed value | Default |
|---|---|---|
prConcurrentLimit |
10 |
10 |
branchConcurrentLimit |
unset | the value of prConcurrentLimit |
prHourlyLimit |
0, which means no limit |
4 in config.js |
Renovate opens new pull requests for this repository only inside the update window, Friday through
Monday in America/New_York. The worker visits the repository about every two hours, so the window
gets about 48 runs. prHourlyLimit is 0, and prConcurrentLimit bounds the CI load instead.
An open pull request holds one of the ten slots until it merges or closes. An attended pull request holds its slot until a person merges it. A pull request with a red required check holds its slot until someone repairs it. When all ten slots are full, Renovate opens no more pull requests in that run. Outside the window, Renovate opens no pull request for a branch that already exists. A held update therefore waits for the next window.
Renovate sorts digest updates before all other updates. For this reason, the five buckets and the catch-all also take digest updates. Without them, each digest takes a slot of its own, and the digests fill the slots before any grouped update.
branchConcurrentLimit counts every renovate/* branch, also a branch without a pull request. When
the count reaches the limit, Renovate creates no new branch. The Dependency Dashboard then lists the
held updates under “Rate-Limited”. A vulnerability remediation ignores this limit.
If the dashboard lists many updates under “Rate-Limited” or “Awaiting Schedule” after a window, count the branches and the open pull requests before you raise a limit:
git ls-remote --heads origin 'renovate/*'gh pr list --state open --author app/fzymgc-renovate --json headRefName --jq '.[].headRefName'If ten pull requests are open, the pull request limit held the rest. Merge or close the open pull requests, and repair the ones with a red required check. If fewer than ten are open, read the gate in “Cooling-off: what gates a pull request”.
The vulnerability path
Section titled “The vulnerability path”Renovate injects vulnerabilityAlerts last, as a forced block, so no packageRules entry outranks
it by position. The only lever is the value. .vulnerabilityAlerts.automerge is false, so a
remediation pull request opens at once and waits for a human. The block sets reviewers to
seanb4t, so every remediation requests a review, also in a tree that names no reviewer. The block
sets no platformAutomerge, so the attended rules keep theirs. It forces minimumReleaseAge to
null, so no release-age floor applies to a remediation.
The account-wide config.js sets vulnerabilityAlerts.automerge to true. The repository value
false overrides it. If you remove the repository key, the account-wide true applies again.
Keep .vulnerabilityAlerts.automerge at false. A true there lets a remediation for LiteLLM
automerge past the attended lane. Nothing asserts this value, so review a change to it as a change
to the LiteLLM posture. A narrower control scoped to a few dependency names needs force, which is
a global-only option and leaves a permanent configuration error on the dashboard.
The cache inode alert
Section titled “The cache inode alert”The ClickStack alert Renovate cache volume inode exhaustion fires when the cache volume has fewer
than 1,000,000 free inodes for 30 minutes.
| Alert value | Cause | Action |
|---|---|---|
| Falling toward 1,000,000 | Real inode growth | Run the renovate-cache-prune Job (see Prune the cache) |
| 0 | Series missing. A FailedMount event says no Pending workload pods |
Delete finished prune pods (see Prune the cache) |
A value of 0 follows a kubelet restart while a finished prune pod still uses the volume. Renovate keeps working. The kubelet retries the mount after the pod is gone, and the series returns.
Constraints when editing .github/renovate.json
Section titled “Constraints when editing .github/renovate.json”The rule description fields are one or two sentences each. The mechanics a description cannot
carry, and that an edit can break, are here.
Order is behaviour
Section titled “Order is behaviour”Renovate merges later matches over earlier ones, so a rule’s index is part of what it does. Nothing asserts the order. This table is the record of it.
| Rule | Position | Because |
|---|---|---|
terraform core |
after terraform providers |
it peels hashicorp/terraform back out of the providers group |
the major guard (matchUpdateTypes: ["major"], no manager matcher) |
after every rule that sets automerge: true, the last being python dependencies, and before the mend-renovate-ce guard and the attended tail |
every automerging group declares no matchUpdateTypes, so without this guard last-match-wins hands each of them the major lane. A later rule that weakens automerge, platformAutomerge, minimumReleaseAge, reviewers or additionalBranchPrefix reopens the lane |
the pre-1.0 guard (the one rule with matchCurrentVersion, matchUpdateTypes: ["minor"], no manager matcher) |
after the five buckets and the catch-all, after every automerge: true rule, before the mend-renovate-ce guard and the attended tail |
for a 0.y.z pin the minor is the breaking bump. Sitting after the buckets makes its own group the last one to decide the branch. A bucket placed after it overwrites its group |
mend-renovate-ce guard |
after the container-image and chart groups | all of those automerge that chart on a short floor |
| the five blast-radius buckets and the catch-all | before the attended tail rules | the buckets decide which branch a dependency lands in. The attended rules decide its posture |
the k3s rule (matchFileNames on the system-upgrade Plans) |
after the argocd/** bucket and the pre-1.0 guard |
the bucket groups the Plans into argocd-minor-patch, which automerges. An earlier position hands the k3s version to that lane |
the pnpm rule (matchDepNames: ["pnpm"]) |
after the five buckets and the catch-all | the buckets put each pnpm pin into the branch of its tree. A pnpm update without new lock files turns that whole branch red |
the pnpm lock file maintenance rule (matchManagers: ["npm"], matchUpdateTypes: ["lockFileMaintenance"]) |
anywhere | no other rule sets a merge key for lock file maintenance of the npm manager |
the custom.regex plus ansible/** guard |
after the buckets, before the attended tail | it overrides the grouping-only ansible/** bucket and must not overwrite an attended posture |
the attended tail (key-affecting encoders, clickstack observability plane, temporal worker controller, valkey operator, litellm data plane) |
after every group and bucket they override | an earlier position is silently overridden. Only the Image Updater handover rule follows them, and it matches no path of an attended rule |
the four-pin extractVersion rule |
anywhere | it sets no posture key. Its scope is load-bearing: it is matched by matchDepNames only, and a manager or file matcher on it strips the prefix from pins that keep theirs |
vault agent (matchDepNames: ["hashicorp/vault"] under ansible/**) |
after the Ansible guard and the five buckets | it sets only groupName and groupSlug. A bucket placed after it puts the Vault pin back into ansible-minor-patch |
lightningstream (lockstep) |
anywhere | it sets the same posture keys as the Ansible guard, and no attended tail rule names the dependency |
terraform cli (lockstep) |
after the five buckets and the catch-all | it sets only groupName and groupSlug. A bucket placed after it puts the two pins on different branches again |
the two changie fragment rules (matchFileNames on one app, with postUpgradeTasks) |
anywhere | they set only postUpgradeTasks, and no other rule sets it |
the buf CLI rule (matchFileNames: ["apps/dns-control/tools/go.mod"], enabled: false) |
after the gomod rule with matchDepTypes: ["indirect"] |
that rule sets enabled: true for every indirect gomod dependency. An earlier position lets it enable the dependencies of buf again |
vite toolchain |
anywhere | it sets only groupName and groupSlug, only for major updates, and no other rule in the file sets a group for a major npm update |
the typescript hold (allowedVersions: "<7") |
anywhere | it sets only allowedVersions, and no other rule sets it for typescript |
the cloudflare hold (allowedVersions: "<5.26.0") |
anywhere | it sets only allowedVersions, and no other rule sets it for cloudflare/cloudflare |
the ansible label rule (matchFileNames: ["ansible/**"], addLabels) |
anywhere | addLabels merges with the labels of the other rules, so no position overrides it |
the engram chart rule (an exact matchPackageNames) |
after the major guard, the pre-1.0 guard and the attended tail, and before the Image Updater handover rule | it lifts the floor and the review of both guards for the first-party engram chart alone. An earlier position hands the chart back to a guard |
the Image Updater handover rule (matchDatasources: ["docker"], an exact matchPackageNames list, enabled: false) |
last | Argo CD Image Updater owns these first-party images (Argo CD Image Updater). A later rule that matches one of them can enable it again, and then two updaters move one pin |
Renovate merges branches, not dependencies
Section titled “Renovate merges branches, not dependencies”A branch automerges only when every upgrade in it automerges, so one attended upgrade turns
automerge off for the whole branch. A branch takes its reviewers from one upgrade, so a co-grouped
unattended upgrade can decide who is asked to look. See the Renovate documentation on grouping for
the current behaviour.
Two rules exist because of this:
- The pre-1.0 guard carries its own
groupNameandgroupSlug, so its updates never share a branch with an unattended bucket. - The major guard carries
additionalBranchPrefix: "major-". A branch name isbranchPrefix + additionalBranchPrefix + branchTopic, and for a grouped update the topic is thegroupSlug. The constant prefix separates the attended major lane from every other lane while each tree keeps its own slug. Keep the value a literal, not a template, so the split is readable from the file.
The major guard does not get a group of its own. A group would collapse every attended major in the estate into one branch and discard the per-tree split.
managerFilePatterns is a glob unless it is /-delimited
Section titled “managerFilePatterns is a glob unless it is /-delimited”An entry is a minimatch glob unless it is wrapped in /.../, in which case it is a regex. A bare
argocd/**/*.ya?ml$ is a glob whose trailing $ is a literal character, and it matches nothing.
renovate-config-validator accepts a pattern that matches nothing.
Assert a pattern by file count against git ls-files, never by a validator exit code:
git ls-files | rg -c '^argocd/.*\.ya?ml$' # the count the argocd and kubernetes managers reachPath matchers over name matchers
Section titled “Path matchers over name matchers”matchFileNames tests the package file first and then the dependency’s lock files. That is how
tf/cloudflare/.terraform.lock.hcl reaches the Terraform bucket. A path matcher survives a provider
rename or a new registry. A name matcher does not, and Renovate rewrites a bare provider name to
hashicorp/<name>, which makes a /^hashicorp// matcher look wider than it is.
matchRegexOrGlobList ORs positive patterns and ANDs !-prefixed ones. That is what makes the
catch-all bucket read as any path in none of the five named buckets.
postUpdateOptions stays inside packageRules
Section titled “postUpdateOptions stays inside packageRules”postUpdateOptions is mergeable and array-typed. A top-level entry concatenates into every
manager’s resolved configuration. It is written inside packageRules, scoped to
matchManagers: ["gomod"]. Do not hoist it.
gomodTidy needs no allowedCommands entry. That allowlist governs postUpgradeTasks only.
gomodTidyE is rejected: its -e flag reports a partial write as a completed one.
The two gomod rules stay two rules. Folding matchDepTypes: ["indirect"] into the tidy rule
narrows the tidy to indirect updates only.
Changelog fragments for app updates
Section titled “Changelog fragments for app updates”Two rules give each update under apps/dns-control/ and apps/hl-assets-uploader/ a
postUpgradeTasks command. The command writes a changie fragment of kind Dependencies to
.changes/unreleased/, and fileFilters adds the file to the commit. The pr-gates member
changelog-fragment fails an app change that has no fragment. docs/operations/releases.md
describes the fragments.
The worker runs the command only when the rendered command matches an anchored entry of
allowedCommands in argocd/cluster-app/templates/renovate.yaml. Renovate renders the templates
first, so the entry sees the real dependency name and version. Its character classes refuse a name
or a version that can leave the quotes of bash -c or of the printf argument. A refused command
puts an artifact error on the pull request, and the fragment member then fails it.
- Change the command in
.github/renovate.jsonand the entry in the cluster template in the same pull request. After the merge, make sure that ArgoCD synced therenovateapplication before the next Renovate run. Until the sync, the worker refuses the new command. - The fragment has no
time:line, and its file name comes from the dependency name and the version. A new run of the branch therefore writes the same file, and Renovate finds no change to push. - A lock file maintenance update has no dependency name. For it, the command writes
Updated the lock file of <package file>.to the file<app>-Dependencies-lock-file-maintenance.yaml. - A digest update has no
newVersion. For it, the command namesnewDigestShort, the first seven characters of the new digest. Without that value, the command rendersUpdated node to ., and the cluster entry refuses it. - For a dependency with a range, such as
^8.0.0, the fragment names Renovate’snewVersion. The lock file can resolve a later version. Before a release, make sure that the fragment names the version in the lock file.
To make sure that the cluster entry allows a command, render the template and test the command.
Replace <command> with a rendered command, for example one from the worker log:
helm template argocd/cluster-app --show-only templates/renovate.yaml \ | yq '.spec.sources[0].helm.valuesObject.renovate.config' > /tmp/renovate-config.jsCMD='<command>' node -e ' const m = new (require("module"))(); m._compile(require("fs").readFileSync("/tmp/renovate-config.js", "utf8"), "config.js"); console.log(m.exports.allowedCommands.some((p) => new RegExp(p).test(process.env.CMD)));'The command prints true when an entry allows the command.
Lockstep majors
Section titled “Lockstep majors”Some packages accept only one major version of another package. Each major of
@sveltejs/vite-plugin-svelte accepts one major of vite. Version 6 accepts vite ^6.3.0 || ^7.0.0,
and version 7 accepts vite ^8.0.0. Renovate does not read the peer ranges of other packages, so it
opens a separate branch for each major. Each of those branches is broken alone.
The rule vite toolchain puts the major updates of the two packages into one branch,
renovate/major-major-vite-toolchain. Renovate adds major- to the slug of a grouped major update,
and the major guard adds the second one. A group holds only the releases that exist at the time of
the run.
If a vite major ships before the plugin major that accepts it, the branch holds vite alone. When the
plugin major ships, Renovate adds it to the same branch.
Both web apps set strictPeerDependencies: true in web/pnpm-workspace.yaml. pnpm checks peer
ranges only when it resolves, so the setting fails the lock file update of a Renovate branch that
leaves a peer unmet, and Renovate reports an artifact error on the pull request. A frozen install,
such as the one in each app Dockerfile, does not resolve and does not check peers. Without the
setting, a split branch passes every check, because pnpm only prints a warning.
To check the peers of a web app by hand, run this command in its web/ directory:
pnpm peers checkIf another pair of packages moves in lockstep, add both names to matchPackageNames of the
vite toolchain rule, or add a rule of the same shape with its own groupName.
A group helps only when a release of the other package accepts the new major. typescript 7 is the
native compiler, and its package has no JavaScript compiler API. svelte-check, the type checker that
both web apps run, works inside that API, so it accepts only typescript ^5.0.0 || ^6.0.0. In both
web apps, svelte-check is the only package that loads typescript. Vite and Vitest remove types
without it.
The typescript hold sets allowedVersions: "<7" for apps/*/web/**, so Renovate proposes
typescript 6 at most. Remove the hold only when both of these are true:
- A typescript 7 release has a stable compiler API. The TypeScript team plans it for 7.1.
- A svelte-check release accepts typescript 7 in its peer range.
Then add typescript and svelte-check to a lockstep group. A person closed the typescript 7 pull
request #2847 without a merge, so Renovate ignores typescript 7. After you remove the hold, give
pull request #2847 a new title to clear that ignore, as the Renovate documentation describes.
svelte-check also has a --tsgo mode that needs typescript 6 and an npm alias for typescript 7. It
uses an experimental API, so the web apps do not use it.
Holds and separate groups
Section titled “Holds and separate groups”The cloudflare provider hold
Section titled “The cloudflare provider hold”From 5.26.0, the cloudflare/cloudflare provider adds the attribute include_shadow_metadata to
cloudflare_dns_record. Every record in state then plans an in-place update. The update fails on
the records that Email Routing manages, so each later apply of main-cluster-cloudflare fails too.
The hold sets allowedVersions: "<5.26.0". Remove it when the upstream issues
cloudflare/terraform-provider-cloudflare#7396 and #7387 close. Before you merge the first update
past the hold, read the speculative plan of main-cluster-cloudflare, because that workspace
applies on merge.
valkey-operator
Section titled “valkey-operator”The valkey-operator chart has its own attended group, valkey-operator, outside the pre-1.0 group.
Between chart 0.3.0 and 0.7.0 the CRD moved the TLS setting from spec.tls to
spec.networking.tls. ArgoCD applies the new CRDs and the new operator in one sync. The operator
then reads no TLS setting and restarts the Valkey nodes without TLS, and oauth2-proxy loses its
session store. To upgrade, stop the operator first: set its replicaCount to 0, move the chart and
the ValkeyCluster resources to the new schema, and then start the operator again.
Paths that Renovate does not read
Section titled “Paths that Renovate does not read”ignorePaths also holds .planning/spikes/** and
.claude/skills/spike-findings-selfhosted-cluster/sources/**. These files are the sources that the
spikes ran, and their pins are part of what each spike recorded. A version update there makes a
source disagree with its own findings.
pnpm updates
Section titled “pnpm updates”The pnpm version appears in packageManager in apps/dns-control/web/package.json,
apps/hl-assets-uploader/web/package.json and website/package.json, and in mise.toml and
mise.lock. The pnpm rule puts all of these on one attended branch, renovate/pnpm. The app
Dockerfiles have no pnpm version. They run corepack install, which reads packageManager.
From pnpm 12, each pnpm-lock.yaml records the pnpm version under packageManagerDependencies.
pnpm install --frozen-lockfile fails when that record does not match packageManager. The worker
keeps the record when it updates a lock file for a dependency. It does not change the record for a
pnpm update, and it leaves the record out when lock file maintenance rebuilds a lock file. So the
renovate/pnpm branch and the renovate/lock-file-maintenance branch arrive with red checks, and
both are attended. Before you merge either of them, run this on its branch from the repository root:
for d in apps/dns-control/web apps/hl-assets-uploader/web website; do (cd "$d" && mise x -- pnpm install --lockfile-only)doneMake sure that each lock file diff changes only the packageManagerDependencies block and the pnpm
packages under it. On the lock file maintenance branch, the diff only adds lines. Then commit the
three lock files. After your push, Renovate no longer updates the branch, so merge it soon.
The deb datasource: one ARG, three apt pins, one dependency
Section titled “The deb datasource: one ARG, three apt pins, one dependency”apps/hl-assets-uploader/Dockerfile pins libheif to an exact Debian package revision through
ARG LIBHEIF_VERSION. The annotation sits above the one declaration that carries a value. The
re-declarations that carry the ARG into later stages are not annotated, because annotating them
reports four pins where there is one. The one value feeds libheif-dev in the builder stage and
libheif1 plus libheif-plugin-libde265 in the runtime stage, so all three move in one commit.
The registry URL is https://deb.debian.org/debian?suite=trixie&components=main&binaryArch=amd64.
Each part is load-bearing:
| Decision | Why the alternative is wrong |
|---|---|
suite=trixie |
golang:1.26 and debian:trixie-slim are both Debian 13. Another suite proposes a different package line, which can be a downgrade that validates clean |
components=main only |
the suite’s Packages index lands on the cache PVC. Other components add index bytes for packages this repository does not install |
one binaryArch |
registryStrategy: 'merge' unions version sets across registry URLs, so a second architecture widens the candidate set |
versioning=deb |
Debian revisions such as 1.19.8-1+deb13u1 are not semver. The default docker versioning mis-orders the -N+debXuY suffix |
A bump here is a key-affecting encoder change. The key-affecting encoders (hl-assets) rule brakes
it, and it matches only because custom.regex is in that rule’s matchManagers.
apps/hl-assets-uploader/testdata/golden.webp.sha256 and golden.key are stop-the-line artifacts.
A bump that moves a golden hash is stopped and deferred with the hash delta as evidence. It is never
re-baselined to make a check pass. See hl-assets Uploader.
Key-affecting encoders
Section titled “Key-affecting encoders”hl-assets-uploader stores each photo under the object key HMAC(salt, the stored WebP bytes). The
encoders make those bytes: github.com/chai2010/webp, github.com/strukturag/libheif and the
libheif1 Debian package. If a bump changes one output byte, the same photo gets a new key. The old
URL keeps working, but deduplication stops for that photo.
The encoders rule attends every bump and holds a 30-day release-age floor. Before you merge a bump,
re-baseline apps/hl-assets-uploader/testdata/golden.webp.sha256 and golden.key. Record the new
baseline as an accepted decision that names what changed, not as a routine test fix.
One ARG LIBHEIF_VERSION pins libheif-dev, libheif1 and libheif-plugin-libde265. Only
libheif1 carries the annotation, so Renovate reports only libheif1.
Annotated Ansible pins: the hazard is the minor lane
Section titled “Annotated Ansible pins: the hazard is the minor lane”An annotated Ansible pin is extracted by the custom.regex manager and by nothing else. The major
lane is already attended by the major guard. The minor lane is the exposure: a cilium_version
minor is a minor under both docker and semver versioning, and without a guard it rides the
account-wide rollup at automerge: true with nobody named. An Ansible-only diff carries no Go and
no lockfile, so pr-gates passes it.
The guard is scoped by manager and by path. matchManagers: ["custom.regex"] alone also catches the
shell _VERSION pins and the Dockerfile deb pin, which the encoders rule guards instead.
Workflow env: pins
Section titled “Workflow env: pins”.github/workflows/build-lightningstream.yml pins LIGHTNINGSTREAM_VERSION, and the Ansible role
pins lightningstream_version. The two are one release train. The fourth custom manager reads the
workflow pin, and the lightningstream (lockstep) rule moves both in one pull request.
The workflow pin holds no leading v. The workflow prepends one at each use site. The four-pin
extractVersion rule strips the prefix and reaches this dependency by name, so the bare form keeps
the two pins equal as strings. GOLANG_BUILDER_IMAGE in the same block carries # renovate-exempt:
because it must follow upstream’s Dockerfile at the pinned tag.
Forward coverage: rules that read as guards but match nothing today
Section titled “Forward coverage: rules that read as guards but match nothing today”Each is listed so that the rule reaches the missing half on the day it appears. Do not describe any of them as guarded.
| Listed | Reached by no manager because | Detector until then |
|---|---|---|
helmv3 in the helm charts rule |
argocd/cluster-app/Chart.yaml declares no dependencies: block |
none |
pip_requirements in python dependencies |
no tracked requirements.txt exists. The three Python projects declare dependencies in pyproject.toml, which is the pep621 manager |
none |
/^renovate$/ in ci/cd components |
the self-hosted chart’s depName is mend-renovate-ce |
the attended mend-renovate-ce rule |
libheif-plugin-libde265 in the encoders rule |
only libheif1 carries the annotation |
the golden-file test, plus the attended libheif1 bump the other pins move with |
argocd/app-configs/litellm/** in the LiteLLM rule |
that directory carries no version pin today | none |
ci/cd components is not repointed at mend-renovate-ce. That rule automerges, and repointing it
overwrites the attended guard.
Pins under argocd/app-configs/clickhouse-mcp/ sit outside the clickstack-chart/** glob and
resolve to the argocd manifests group. They are client pins, and they are excluded from the
attended ClickStack rule on purpose. The ClickHouse client image in
argocd/app-configs/clickstack/kustomization.yaml is the exception. It must equal the server tag,
so the attended rule lists that file.
Procedures
Section titled “Procedures”Annotate a pin
Section titled “Annotate a pin”- Put
# renovate: datasource=<ds> depName=<name>on the line directly above the key. Addversioning=<v>and thenregistryUrl=<url>when the datasource needs them, in that order. - If the pin cannot be annotated, put
# renovate-exempt: <reason>above it instead. - Make sure that the pin appears on the Dependency Dashboard issue after the next run, or run the worker log query below and look for the depName.
Add or change a package rule
Section titled “Add or change a package rule”-
Decide the posture keys the rule sets. An attended rule sets
automerge: false,platformAutomerge: false, aminimumReleaseAgeand areviewerslist. -
Place it by the ordering table above. A rule that sets posture sits after every rule it must override.
-
Resolve the committed rules last-match-wins for one real pin in each lane the rule touches, and make sure that no other lane changed.
-
Validate the file:
Terminal window npx --yes --package renovate renovate-config-validator --no-global .github/renovate.jsonWithout
--no-global, the validator checks the file as the global configuration of the worker. The validator catches a malformed edit. It does not prove a pattern matches anything.
Confirm a repository-configuration key reaches the worker
Section titled “Confirm a repository-configuration key reaches the worker”The worker logs at LOG_LEVEL=info, so the resolved-configuration dump is not ingested. The resolved
configuration still reaches default.otel_logs inside the payload of some error lines, so this
reading is opportunistic. Use a key whose committed value differs from its default, for example
prCreation:
kubectl exec -n clickstack "$(kubectl get pod -n clickstack \ -l clickhouse.com/role=clickhouse-server -o name | head -1 | cut -d/ -f2)" \ -- clickhouse-client --query " SELECT Timestamp, JSONExtractString(Body, 'logContext') AS logContext, extract(Body, '\"prCreation\":\"[a-z-]+\"') AS prCreation FROM default.otel_logs WHERE ResourceAttributes['k8s.pod.name'] LIKE 'renovate-ce-%' AND JSONExtractString(Body, 'repository') = 'fzymgc-house/selfhosted-cluster' AND Body ILIKE '%prCreation%' AND Timestamp > now() - INTERVAL 7 DAY ORDER BY Timestamp DESC LIMIT 20"The timestamp column is Timestamp, never TimestampTime.
| Result | Reading |
|---|---|
"prCreation":"not-pending" |
The key reached the worker |
"prCreation":"immediate" |
The wrapper dropped the key, or the row predates the change. Compare the row’s Timestamp with the merge |
| No rows | No configuration-bearing line was logged in the window. Widen the window or wait for another cycle |
Read the worker log
Section titled “Read the worker log”kubectl -n renovate logs deploy/renovate-ce --since=24h | rg 'fzymgc-house/selfhosted-cluster'For history beyond the pod, query default.otel_logs with
ResourceAttributes['k8s.pod.name'] LIKE 'renovate-ce-%' and
JSONExtractString(Body, 'repository') = 'fzymgc-house/selfhosted-cluster'.
Read the job queue, or queue a run
Section titled “Read the job queue, or queue a run”The server has an admin API. The public tunnel forwards only /webhook, and the NetworkPolicy
admits only cloudflared, so reach the API through a port-forward. Use it when a Dependency
Dashboard tick or a webhook did not start a run.
-
Open the port-forward and read the secret:
Terminal window kubectl -n renovate port-forward svc/renovate-ce 18080:80 &TOKEN=$(kubectl -n renovate get secret renovate-app -o jsonpath='{.data.mendRnvServerApiSecret}' | base64 -d) -
Read the job queue:
Terminal window curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:18080/system/v1/jobs/queue | jq -
Read the recent jobs of one repository. The repository slug is not URL-encoded:
Terminal window curl -s -H "Authorization: Bearer $TOKEN" \'http://127.0.0.1:18080/api/v1/repos/fzymgc-house/selfhosted-cluster/-/jobs?limit=10' | jq -
If no job is queued for the repository, queue a run:
Terminal window curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \-d '{}' http://127.0.0.1:18080/api/v1/repos/fzymgc-house/selfhosted-cluster/-/jobs/run -
Stop the port-forward with
kill %1.
Prune the cache
Section titled “Prune the cache”The CronJob runs on Sunday. To run it now:
kubectl -n renovate create job --from=cronjob/renovate-cache-prune renovate-cache-prune-manualkubectl -n renovate logs job/renovate-cache-prune-manual -f # prints df -i before and afterTo clear a missing inode series after a kubelet restart:
kubectl -n renovate delete pod \ -l app.kubernetes.io/name=mend-renovate-ce-cache-prune --field-selector status.phase=SucceededRead the free inodes on the box at any time:
kubectl -n renovate exec deploy/renovate-ce -- df -i /tmp/renovatePurge a poisoned Go toolchain extraction
Section titled “Purge a poisoned Go toolchain extraction”This is cache maintenance inside a PVC. No kubectl apply is involved and none is permitted. The
PVC is shared by every repository the worker serves, so one rm against the wrong path degrades the
cache for all of them.
Symptom: a Renovate-authored gomod pull request arrives with a go.sum that does not verify, and
the worker log names stdlib packages as missing:
context: package context is not in std (/tmp/renovate/cache/others/go/pkg/mod/golang.org/toolchain@v0.0.1-go<version>.linux-arm64/src/context)Cause: a toolchain directory under /tmp/renovate/cache/others/go/pkg/mod/golang.org/ has no
src/ tree, and Go still treats it as complete. .ziphash is Go’s downloaded-and-verified marker.
Go reads it and never re-extracts, so a half-empty extraction is trusted permanently. The worker
cleanup keeps /tmp/renovate/cache, so the entry survives every run.
Judge intactness by the src/ tree, never by the .zip. Go deletes the archive after a successful
extraction, so a healthy toolchain has no .zip either. This is not a missing gomodTidy step.
go get -t ./... already runs and fails on the broken toolchain.
-
Read every toolchain entry and its
src/state:Terminal window kubectl -n renovate exec deploy/renovate-ce -- sh -c 'for d in /tmp/renovate/cache/others/go/pkg/mod/golang.org/toolchain@*; doif [ -d "$d/src" ]; then echo "$d src: present ($(ls "$d/src" | wc -l) entries)"; else echo "$d src: ABSENT"; fidonels -l /tmp/renovate/cache/others/go/pkg/mod/cache/download/golang.org/toolchain/@v/' -
Continue only if one entry reads
ABSENTand the toolchain thatapps/hl-assets-uploader/go.modnames readspresent. If the named toolchain is absent too, the purge scope is wider than this procedure. Stop. -
Remove the poisoned extraction and its marker, and those two paths only. Write the version out in full. A glob such as
toolchain@v0.0.1-go1.26.*reaches the intact entries.Terminal window V=<version, for example 1.26.5>kubectl -n renovate exec deploy/renovate-ce -- sh -c "rm -rf /tmp/renovate/cache/others/go/pkg/mod/golang.org/toolchain@v0.0.1-go$V.linux-arm64rm -f /tmp/renovate/cache/others/go/pkg/mod/cache/download/golang.org/toolchain/@v/v0.0.1-go$V.linux-arm64.ziphash" -
Re-run step 1. The removed entry is gone and every other entry still reads
present. -
On the next Renovate-authored
gomodpull request, check out its head and rungo mod verify. Then re-run module resolution and make sure thatgo.sumis unchanged.
Running the procedure twice is safe. Step 2 finds no absent entry and the procedure stops.
apps/hl-assets-uploader/go.mod carries a go directive and no toolchain directive. The go
line names the toolchain Go must fetch. A toolchain directive is for a toolchain newer than the
go line requires, and it is a second version-bearing line that can drift. Renovate tracks the
go directive as a dependency (datasource: golang-version), so bumps to it are ordinary updates.
Diagnose a red go.sum check
Section titled “Diagnose a red go.sum check”Compare three refs before concluding anything: the branch tip, origin/main, and
refs/pull/N/merge. The pull_request event builds the merge of head into base, so a pull request
with no Go change goes red when it inherits a broken go.sum from main.
Review a mend-renovate-ce bump
Section titled “Review a mend-renovate-ce bump”The pin is targetRevision in argocd/cluster-app/templates/renovate.yaml. This chart carries the
pod that does the proposing, so a bad values-key move takes out the mechanism that proposes its own
repair.
- Make sure that the pull request is Renovate-authored. A human-authored bump proves nothing about the manager.
- Render both sides with
helm templateat the current pin and at the proposed one, and read the diff for moved or renamed values keys. - Merge attended, under the
mend-renovate-ceguard’s named reviewer. Never by an agent.
Related
Section titled “Related”- ArgoCD Operations, how the Renovate CE workload is deployed
- hl-assets Uploader, the Go module the encoder rule protects