Design: first-party image updater
Context
Section titled “Context”See openspec/changes/first-party-image-updater/proposal.md, section “Why”, for the delay that
Renovate adds to first-party images.
ArgoCD v3.2.1 runs from the argo-cd chart 9.1.6, which tf/cluster-bootstrap/argocd.tf installs.
The Application cluster-app renders every other Application from argocd/cluster-app/templates/,
with automated sync, prune and selfHeal. A first-party image is pinned today as
<tag>@sha256:<index digest> on the image: line of a manifest, or in the inline valuesObject of
an Application.
Argo CD Image Updater (version 1.3, read from the source of v1.3.1) works like this:
every 2 minutes, for each ImageUpdater resource | v baseline = status.summary.images of the Application (live images, not git) | v registry tags --> newest tag for the update strategy | v override in the Application spec, in memory kustomize source --> spec.source.kustomize.images helm source --> spec.source.helm.parameters | v write-back method argocd --> patch of the live Application git, default target --> new file .argocd-source-<app>.yaml git, kustomization:<dir> --> upsert of one entry in the images: list git, helmvalues:<file> --> rewrite of the configured keys in that fileOnly the digest strategy records a digest. The semver, alphabetical and newest-build strategies write a tag with no digest. For an Application with several sources, Image Updater uses the first Helm, Kustomize or plugin source. It takes the default target path and the git repository from that source.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- One updater and one pin for each first-party image in scope.
- A pull request within 5 minutes of a publish, and a merge with no person for the apps on the auto-merge list.
Non-goals:
- Updates of charts. Renovate keeps every chart, the first-party charts included.
- The
dns_node_worker_versionpin in Ansible. - The images that the proposal puts out of scope: octopus,
headroom-otel,k8s-utils,actions-runnerandnas-otel. - Address changes. This change adds no address, VIP or route, and
docs/reference/network.md, the addressing authority, stays as it is.
Decisions
Section titled “Decisions”1. Image Updater instead of a faster Renovate
Section titled “1. Image Updater instead of a faster Renovate”Image Updater reads the registry every two minutes. It has no schedule window, no pull request limit and no classes of update type.
Alternatives:
- A release workflow calls the Renovate jobs API. A run of this repository takes 313 seconds at the median, the jobs API answers only through a port-forward, and Mend drops an event while a job for the repository runs. The rule traps of Renovate stay.
- A faster Mend scheduler. The scheduler runs all 35 repositories of three accounts, so a shorter interval multiplies that load.
- Each publisher opens the pin pull request itself. That is new code in each repository, and the publisher needs to know the pin layout of this repository.
2. An ArgoCD Application in the project core-services
Section titled “2. An ArgoCD Application in the project core-services”The Application argocd-image-updater installs the argocd-image-updater chart of argo-helm,
version 1.3.1, which ships app v1.3.0. Pull request mode and verified commits arrived in v1.3.0. A
second source, argocd/app-configs/argocd-image-updater/, holds the ExternalSecrets, the registry
configuration and the ImageUpdater resources. The ImageUpdater resources carry
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true, because the chart brings their
CRD in the same sync.
Terraform installs ArgoCD because ArgoCD starts the cluster. Image Updater is an ordinary controller, like Renovate and Reloader, so it is an Application. Renovate keeps its chart version current.
3. Git write-back with pull requests against main
Section titled “3. Git write-back with pull requests against main”Each ImageUpdater resource has one entry in applicationRefs. The CRD accepts spec.writeBackConfig
as the default for every entry, and the writeBackConfig of an entry overrides it. With one entry,
the two places act the same. The writeBackConfig of the entry sets these values:
method: git:secret:argocd/argocd-image-updater-gitgitConfig.repository: https://github.com/fzymgc-house/selfhosted-clustergitConfig.branch: maingitConfig.pullRequest.github: {}
The repository is explicit. For temporal-workers and agent-memory, the first source is a chart
registry, and Image Updater takes the repository from that source. The controller
runs with the commit method api, so GitHub signs each commit as Verified.
The argocd method is not an option. cluster-app renders each Application with selfHeal, so it
reverts the patch, and git does not hold the running version.
4. A GitHub App for the updater, and the GHCR credential that exists
Section titled “4. A GitHub App for the updater, and the GHCR credential that exists”A new GitHub App has write access to Contents and Pull requests and read access to Metadata. It is
installed on fzymgc-house/selfhosted-cluster only. Its credential goes to Vault at
fzymgc-house/cluster/argocd-image-updater, with the fields github_app_id,
github_app_installation_id and github_app_private_key. An ExternalSecret writes the Secret
argocd-image-updater-git in argocd with the keys githubAppID, githubAppInstallationID and
githubAppPrivateKey, which Image Updater reads in pkg/argocd/gitcreds.go.
The App of ArgoCD (fzymgc-house-argocd-app) has write access to Contents and Repository hooks,
and no access to Pull requests. It cannot open a pull request, and the ArgoCD repo server never
holds the credential of the updater.
The GHCR credential is fzymgc-house/cluster/ghcr/pull-secret, which already pulls both
ghcr.io/fzymgc-house/* and ghcr.io/seanb4t/*. An ExternalSecret writes it as the Secret
argocd-image-updater-ghcr in argocd. Image Updater refuses a pull secret from a namespace other
than the namespace of the Application, and every Application is in argocd.
5. The pin is the place where Image Updater writes
Section titled “5. The pin is the place where Image Updater writes”| App | Image | Write-back target | Strategy |
|---|---|---|---|
life-agent |
life-agent-sandbox, life-agent-smokescreen |
kustomization |
digest of tag main |
life-agent-gateway |
life-agent-hermes |
kustomization |
alphabetical |
hl-assets |
hl-assets-uploader |
kustomization |
semver |
firewalla-mcp |
firewalla-mcp |
kustomization |
semver |
dns-control |
dns-control |
kustomization:/argocd/app-configs/dns-control-image |
semver |
temporal-workers |
temporal-workers/terraform |
helmvalues:/argocd/app-configs/temporal-workers/values.yaml |
semver |
agent-memory |
engram |
helmvalues:/argocd/app-configs/agent-memory/values.yaml |
semver |
The app external-dns runs the image of dns-control as its webhook, with no ImageUpdater resource
of its own. Decision 12 gives that image one pin for both apps.
The allowTags of each image matches its release tags only, and never latest or main-<sha>. A
task lists the tags of each image with crane ls before it writes the expression. The date tags of
life-agent-hermes have a fixed width, so an alphabetical sort orders them, and Image Updater reads
no image metadata for them.
For a kustomize app, the manifests name the image without a tag, and the images: entry of
kustomization.yaml holds newTag, plus digest for the two main images. Kustomize v5 renders an
entry with both fields as <name>:main@sha256:<digest>, so these two render as today. A test with
kubectl v1.33.9 and Kustomize v5.6.0 printed that form.
For a chart app, the content of valuesObject moves to a values file in git. The Application reads
it through a ref: values source and $values/... in valueFiles, as external-dns did before
decision 12.
manifestTargets.helm names the repository key and the tag key of each image. The chart of the
temporal workers builds each worker image as
<global.image.registry>/<global.image.repository>/<worker>:<Chart.appVersion>. Up to chart 3.2.0,
no value overrides that reference. The chart change in fzymgc-house/temporal-workers#197 adds
workers.<worker>.image.repository and workers.<worker>.image.tag. The values file sets both, with
the full image name, so Image Updater writes the long form that it reads.
The default target .argocd-source-<app>.yaml is rejected. It is a second file that overrides the
manifest, and a reader of the manifest does not see it. ArgoCD also reads it only from a git
directory, so it never applies to a chart from a registry.
6. Tag-only pins, guarded at the publisher
Section titled “6. Tag-only pins, guarded at the publisher”On 2026-10-09 Sean accepted pins with a tag and no digest for first-party images. GHCR has no setting
that makes a tag immutable. GitHub immutable releases lock release assets and git tags, not
container tags. Each publisher therefore refuses to push a version tag that exists, as
.github/workflows/build-hl-assets-uploader.yaml does. If docker buildx imagetools inspect <image>:<version> succeeds, the job fails. If the answer is “not found”, the job continues. Any
other answer fails the job.
.github/workflows/release.yml runs the same step for dns-control before GoReleaser. The
publishers in seanb4t/life-agent, seanb4t/engram, fzymgc-house/temporal-workers and
fzymgc-house/firewalla-mcp get the same step. The tag main of the life agent moves by design, so
those two pins keep their digest.
7. The changed files and lines decide the merge posture
Section titled “7. The changed files and lines decide the merge posture”The workflow .github/workflows/image-updater-automerge.yml runs on pull_request (opened,
reopened, synchronize). It acts only when the author is the bot user of the updater App and the
head branch starts with image-updater-. If every changed file is a modified pin file of an app on
the auto-merge list, and every changed line is a tag:, newTag: or digest: line, it turns on
auto-merge with the squash method (gh pr merge --auto --squash). Otherwise it requests a review
from seanb4t (gh pr edit --add-reviewer seanb4t). The workflow never merges a pull request
itself. GitHub merges it after the required checks pass.
Image Updater v1.3.1 cannot label a pull request. Labels exist only on its unreleased main branch. The files and their diffs are facts of each pull request, so the workflow uses them.
A check of the files alone trusts the whole content of a pin file. A kustomization.yaml can add a
resource, and a Helm values file can set any key of its chart. For the temporal workers, that file
also holds the guarded-apply configuration and the Vault roles. The line check limits auto-merge to
a version change. The updater App then can at most choose another tag of the same repository, and
the tag guard of decision 6 keeps each tag fixed.
The default auto-merge list is life-agent, life-agent-gateway, hl-assets, firewalla-mcp and
temporal-workers. The attended apps are dns-control and agent-memory, because dns-control is
the only writer of pdns and an engram release can need a schema migration. The pin of dns-control
also moves the webhook of external-dns, as decision 12 says.
The workflow uses GITHUB_TOKEN. A merge through that token starts no workflow on push. No such
workflow reads argocd/app-configs/, and ArgoCD still gets the webhook of the push.
8. One Renovate rule hands each image over
Section titled “8. One Renovate rule hands each image over”One rule in .github/renovate.json sets enabled: false for matchDatasources: ["docker"] and an
exact matchPackageNames list of the images that Image Updater owns. Its description names Image
Updater as the owner. It is the last package rule, so no later rule enables these images again. Each
pull request that gives an image to Image Updater adds that image to the list. Pull requests 3 and 4
remove the two life agent rules.
9. A hold goes through ignoreTags
Section titled “9. A hold goes through ignoreTags”To hold a bad version, a person adds it to ignoreTags of that image in its ImageUpdater resource,
through a pull request. Image Updater then opens a pull request for the newest tag that is not held.
GetNewestVersionFromTags picks the newest allowed tag and does not compare it with the running
tag, so that pull request moves the pin back.
A revert of an updater pull request is not a hold. The live image is then older than the newest
tag, and the next poll opens the same update again. For the two main images, a hold removes the
image from its ImageUpdater resource, and a person sets the digest, as
docs/operations/life-agent.md, section “Roll back an image”, describes.
10. engram and the temporal workers: the image moves alone
Section titled “10. engram and the temporal workers: the image moves alone”Image Updater moves the image tag, and Renovate moves the chart. A first-party chart therefore accepts an image that was released after the chart. The worker chart accepts one only from the release of fzymgc-house/temporal-workers#197, so pull request 6 first moves the chart pin to that release.
The engram updater pull request is attended. If an engram release needs a newer chart, Sean adds the chart bump to the same pull request before the merge. The summarize CronJob moves into the engram chart, so one value drives every engram workload that ArgoCD runs. The three Jobs that a person runs by hand get no pin. Their runbook reads the running tag from the engram Deployment at run time. Today all engram pins move in one pull request, and this decision replaces that practice.
On 2026-10-11 Sean decided that Renovate merges an engram chart release with no release-age floor
and no review, for every update type. A first-party release deploys at once, and Argo CD rolls a bad
one back. A Qdrant downgrade stays a snapshot restore. The rule engram chart in
.github/renovate.json lifts the major guard and the pre-1.0 guard for that one chart.
The summarize CronJob was already in the chart, behind memory.summarize.cronjob.enabled, but it ran
/engram serve with the arguments of the sweep and failed at once. seanb4t/engram#808 (release
0.24.1) makes it run summarize-missing with the security context, limits and deadline of the
hand-written CronJob.
11. Polling, no webhook
Section titled “11. Polling, no webhook”Image Updater accepts the package webhook of GitHub. That needs a public route and a GitHub App
that subscribes to package events. Two minutes of delay is enough, so this change uses polling only.
12. One pin for an image that two apps run
Section titled “12. One pin for an image that two apps run”dns-control runs twice: as the Deployment of the app dns-control, and as the webhook in the pod of
external-dns. Up to 0.22.0, each app had its own pin and its own ImageUpdater resource. Image
Updater names the head branch of a pull request image-updater-<targetKey>-<sha256>. <targetKey>
hashes the repository, the branch and the target path, and <sha256> hashes the update. Two target
files gave two pull requests for one release. The releases 0.20.3, 0.21.0 and 0.22.0 each opened
two, with one <sha256> and two values of <targetKey>. On 2026-10-11 Sean asked for one.
The pin moves to a kustomize Component, argocd/app-configs/dns-control-image/kustomization.yaml,
which holds the one images: entry. The kustomization of dns-control includes it in
components:. external-dns moves to the pattern of argocd/app-configs/clickstack-chart: one git
source whose kustomization inflates the chart through helmCharts: and includes the same
Component. Its values file sets the tag of the webhook image to set-by-kustomize, and the
Component replaces that tag. Only the ImageUpdater resource of dns-control remains, and it writes
the Component, so a release opens one pull request.
Image Updater edits only the images node of the target file, with kyaml, and keeps the rest of the
file, so the file stays kind: Component. On 2026-10-11, kustomize build --enable-helm with
Kustomize v5.8.3 rendered the new external-dns as the same nine resources as the Helm source plus
the kustomization of today, with the webhook image from the Component.
The two pods now roll together, so dns-control no longer moves before its webhook. external-dns retries a failed call of the webhook at its next interval of one minute. A dns-control release therefore keeps the routes that the webhook of the previous version calls, and the minute between the two restarts costs at most one retry.
The chart version of external-dns moves from the Application to helmCharts:. The Renovate rule
“Helm charts” matches the managers helm-values, helmv3 and argocd only, so a new rule after it
gives the manager kustomize and the datasource helm the same settings. The ClickStack rule comes
later and still overrides it for that chart.
Alternatives:
- Two pull requests, one for each pin. Sean asked for one on 2026-10-11.
- Fold the Deployment of external-dns into the app
dns-control. This also gives one pin, but it moves the live resources of external-dns from one Argo CD app to another. - Auto-merge the pull request of
external-dnsafter the one ofdns-control. That is still two pull requests.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [A tag-only pin trusts the publisher. A re-pushed tag changes the content on the next pull, and each node caches its own copy under the same tag.] → Each publisher gets the guard of decision 6. Pull request 1 adds it here, and the cross-repository tasks add it in the other repositories.
- [Image Updater v1.3 builds against the ArgoCD 3.5 API types, and the cluster runs ArgoCD v3.2.1.] → Pull request 3 uses the life agent stateless images first, which carry the least risk.
- [Image Updater rewrites
kustomization.yamland can drop comments in it.] → Theimages:entries carry no comments. The first updater pull requests get a review against the requirement “An updater pull request changes only pin files”. - [Each pull request that changes a pin from
<tag>@sha256:<digest>to<tag>changes the image string. The pods restart once with the same content.] → Pull request 5 restarts dns-control and the webhook of external-dns, so it merges in a quiet window. - [An image that no pod runs is not in
status.summary.images, so Image Updater skips it.] → None of the images in scope scales to zero. - [dns-control and the webhook of external-dns restart together after decision 12.] → external-dns retries at its next interval, and a release keeps the routes of the previous webhook.
- [Image Updater writes a file of kind Component for the first time.] → The first updater pull request after pull request 8 gets a review against the requirement “An updater pull request changes only pin files”.
- [The updater App can write to the repository.] → It is installed on one repository, and branch
protection still requires a pull request with
pr-gatesandOctopus Review. - [An engram image can need a newer engram chart.] → The engram pull request is attended, as decision 10 says.
- [After a worker release, the old worker version runs until each workflow pinned to it ends, and
then for one more hour (
scaledownDelay: 1h). ArgoCD sorts the live images by name, and Image Updater takes the first match, which can be the old tag.] → Image Updater then writes the pin that is already there, and GitHub answers422each cycle until the old pods stop. No pull request comes from it. The runbook names this error as expected for up to five hours after a worker release, and a longer run as a workflow that does not end.
Migration Plan
Section titled “Migration Plan”The pull requests merge in the order of proposal.md, section “Pull request split”. Each one has a
rollback. An agent opens the revert pull request, and Sean merges it:
git switch -c revert/<name> origin/maingit revert -m 1 <merge-sha>| Step | Rollback |
|---|---|
| 1. The tag guard | The revert pull request. Nothing in the cluster changes. |
| 2. The updater | The revert pull request. ArgoCD prunes the Application. The CRD stays if the chart ships it in crds/. |
| 3 to 7. The handover of each app | The revert pull request restores the old pin lines, the Renovate entries and the ImageUpdater resource. The pods restart once. |
| 8. One pin for dns-control | The revert pull request restores the two pins and the ImageUpdater resource of external-dns. Nothing restarts, because both renders stay the same. |
| Any merge in flight | gh workflow disable image-updater-automerge.yml --repo fzymgc-house/selfhosted-cluster stops every new auto-merge at once. |
| A bad image | The hold of decision 9. |
After a rollback of pull request 5, this command waits until dns-control runs again:
kubectl --context fzymgc-house -n dns-control rollout status deployment/dns-control --timeout=5mOpen Questions
Section titled “Open Questions”Each question has a default, and the tasks follow that default.
- engram: does decision 10 replace the practice that all engram pins move in one pull request? The default is yes, with the attended pull request and the chart bump that Sean adds when a release needs it.
- The auto-merge list. The default is the list in decision 7. A change moves an app between the list and the attended set, in the workflow and in the runbook only.
- GitHub immutable releases on
fzymgc-house/selfhosted-cluster. They protect thedns-control/v<version>tags and the worker tarball. The release workflow makes a draft first and publishes it after a test, so the setting needs a test against that flow first. The default is to leave it off in this change. - Does Renovate move the engram chart? In August 2026 the GHCR lookup of Renovate for the private engram packages failed with 403, and the dependency dashboard now lists the chart. Pull request 7 finds out. If Renovate cannot move it, Sean moves the chart by hand, as now. Answer: yes. The dependency dashboard detects the chart with no lookup warning. Renovate never opened a chart pull request, because Sean moved each release within an hour and the chart rules waited 3 or 7 days.
- An alert on the errors of Image Updater. The default is no alert in this change. The controller log reaches ClickStack like every pod log.
- octopus and
headroom-oteljoin later, after octopus has a tag to follow andheadroom-otelhas one tag for each build.