Skip to content

Design: first-party image updater

See openspec/changes/first-party-image-updater/proposal.md, section “Why”, for the delay that Renovate adds to first-party images.

ArgoCD v3.2.1 runs from the argo-cd chart 9.1.6, which tf/cluster-bootstrap/argocd.tf installs. The Application cluster-app renders every other Application from argocd/cluster-app/templates/, with automated sync, prune and selfHeal. A first-party image is pinned today as <tag>@sha256:<index digest> on the image: line of a manifest, or in the inline valuesObject of an Application.

Argo CD Image Updater (version 1.3, read from the source of v1.3.1) works like this:

every 2 minutes, for each ImageUpdater resource
|
v
baseline = status.summary.images of the Application (live images, not git)
|
v
registry tags --> newest tag for the update strategy
|
v
override in the Application spec, in memory
kustomize source --> spec.source.kustomize.images
helm source --> spec.source.helm.parameters
|
v
write-back
method argocd --> patch of the live Application
git, default target --> new file .argocd-source-<app>.yaml
git, kustomization:<dir> --> upsert of one entry in the images: list
git, helmvalues:<file> --> rewrite of the configured keys in that file

Only the digest strategy records a digest. The semver, alphabetical and newest-build strategies write a tag with no digest. For an Application with several sources, Image Updater uses the first Helm, Kustomize or plugin source. It takes the default target path and the git repository from that source.

Goals:

  • One updater and one pin for each first-party image in scope.
  • A pull request within 5 minutes of a publish, and a merge with no person for the apps on the auto-merge list.

Non-goals:

  • Updates of charts. Renovate keeps every chart, the first-party charts included.
  • The dns_node_worker_version pin in Ansible.
  • The images that the proposal puts out of scope: octopus, headroom-otel, k8s-utils, actions-runner and nas-otel.
  • Address changes. This change adds no address, VIP or route, and docs/reference/network.md, the addressing authority, stays as it is.

1. Image Updater instead of a faster Renovate

Section titled “1. Image Updater instead of a faster Renovate”

Image Updater reads the registry every two minutes. It has no schedule window, no pull request limit and no classes of update type.

Alternatives:

  • A release workflow calls the Renovate jobs API. A run of this repository takes 313 seconds at the median, the jobs API answers only through a port-forward, and Mend drops an event while a job for the repository runs. The rule traps of Renovate stay.
  • A faster Mend scheduler. The scheduler runs all 35 repositories of three accounts, so a shorter interval multiplies that load.
  • Each publisher opens the pin pull request itself. That is new code in each repository, and the publisher needs to know the pin layout of this repository.

2. An ArgoCD Application in the project core-services

Section titled “2. An ArgoCD Application in the project core-services”

The Application argocd-image-updater installs the argocd-image-updater chart of argo-helm, version 1.3.1, which ships app v1.3.0. Pull request mode and verified commits arrived in v1.3.0. A second source, argocd/app-configs/argocd-image-updater/, holds the ExternalSecrets, the registry configuration and the ImageUpdater resources. The ImageUpdater resources carry argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true, because the chart brings their CRD in the same sync.

Terraform installs ArgoCD because ArgoCD starts the cluster. Image Updater is an ordinary controller, like Renovate and Reloader, so it is an Application. Renovate keeps its chart version current.

3. Git write-back with pull requests against main

Section titled “3. Git write-back with pull requests against main”

Each ImageUpdater resource has one entry in applicationRefs. The CRD accepts spec.writeBackConfig as the default for every entry, and the writeBackConfig of an entry overrides it. With one entry, the two places act the same. The writeBackConfig of the entry sets these values:

  • method: git:secret:argocd/argocd-image-updater-git
  • gitConfig.repository: https://github.com/fzymgc-house/selfhosted-cluster
  • gitConfig.branch: main
  • gitConfig.pullRequest.github: {}

The repository is explicit. For temporal-workers and agent-memory, the first source is a chart registry, and Image Updater takes the repository from that source. The controller runs with the commit method api, so GitHub signs each commit as Verified.

The argocd method is not an option. cluster-app renders each Application with selfHeal, so it reverts the patch, and git does not hold the running version.

4. A GitHub App for the updater, and the GHCR credential that exists

Section titled “4. A GitHub App for the updater, and the GHCR credential that exists”

A new GitHub App has write access to Contents and Pull requests and read access to Metadata. It is installed on fzymgc-house/selfhosted-cluster only. Its credential goes to Vault at fzymgc-house/cluster/argocd-image-updater, with the fields github_app_id, github_app_installation_id and github_app_private_key. An ExternalSecret writes the Secret argocd-image-updater-git in argocd with the keys githubAppID, githubAppInstallationID and githubAppPrivateKey, which Image Updater reads in pkg/argocd/gitcreds.go.

The App of ArgoCD (fzymgc-house-argocd-app) has write access to Contents and Repository hooks, and no access to Pull requests. It cannot open a pull request, and the ArgoCD repo server never holds the credential of the updater.

The GHCR credential is fzymgc-house/cluster/ghcr/pull-secret, which already pulls both ghcr.io/fzymgc-house/* and ghcr.io/seanb4t/*. An ExternalSecret writes it as the Secret argocd-image-updater-ghcr in argocd. Image Updater refuses a pull secret from a namespace other than the namespace of the Application, and every Application is in argocd.

5. The pin is the place where Image Updater writes

Section titled “5. The pin is the place where Image Updater writes”
App Image Write-back target Strategy
life-agent life-agent-sandbox, life-agent-smokescreen kustomization digest of tag main
life-agent-gateway life-agent-hermes kustomization alphabetical
hl-assets hl-assets-uploader kustomization semver
firewalla-mcp firewalla-mcp kustomization semver
dns-control dns-control kustomization:/argocd/app-configs/dns-control-image semver
temporal-workers temporal-workers/terraform helmvalues:/argocd/app-configs/temporal-workers/values.yaml semver
agent-memory engram helmvalues:/argocd/app-configs/agent-memory/values.yaml semver

The app external-dns runs the image of dns-control as its webhook, with no ImageUpdater resource of its own. Decision 12 gives that image one pin for both apps.

The allowTags of each image matches its release tags only, and never latest or main-<sha>. A task lists the tags of each image with crane ls before it writes the expression. The date tags of life-agent-hermes have a fixed width, so an alphabetical sort orders them, and Image Updater reads no image metadata for them.

For a kustomize app, the manifests name the image without a tag, and the images: entry of kustomization.yaml holds newTag, plus digest for the two main images. Kustomize v5 renders an entry with both fields as <name>:main@sha256:<digest>, so these two render as today. A test with kubectl v1.33.9 and Kustomize v5.6.0 printed that form.

For a chart app, the content of valuesObject moves to a values file in git. The Application reads it through a ref: values source and $values/... in valueFiles, as external-dns did before decision 12. manifestTargets.helm names the repository key and the tag key of each image. The chart of the temporal workers builds each worker image as <global.image.registry>/<global.image.repository>/<worker>:<Chart.appVersion>. Up to chart 3.2.0, no value overrides that reference. The chart change in fzymgc-house/temporal-workers#197 adds workers.<worker>.image.repository and workers.<worker>.image.tag. The values file sets both, with the full image name, so Image Updater writes the long form that it reads.

The default target .argocd-source-<app>.yaml is rejected. It is a second file that overrides the manifest, and a reader of the manifest does not see it. ArgoCD also reads it only from a git directory, so it never applies to a chart from a registry.

6. Tag-only pins, guarded at the publisher

Section titled “6. Tag-only pins, guarded at the publisher”

On 2026-10-09 Sean accepted pins with a tag and no digest for first-party images. GHCR has no setting that makes a tag immutable. GitHub immutable releases lock release assets and git tags, not container tags. Each publisher therefore refuses to push a version tag that exists, as .github/workflows/build-hl-assets-uploader.yaml does. If docker buildx imagetools inspect <image>:<version> succeeds, the job fails. If the answer is “not found”, the job continues. Any other answer fails the job.

.github/workflows/release.yml runs the same step for dns-control before GoReleaser. The publishers in seanb4t/life-agent, seanb4t/engram, fzymgc-house/temporal-workers and fzymgc-house/firewalla-mcp get the same step. The tag main of the life agent moves by design, so those two pins keep their digest.

7. The changed files and lines decide the merge posture

Section titled “7. The changed files and lines decide the merge posture”

The workflow .github/workflows/image-updater-automerge.yml runs on pull_request (opened, reopened, synchronize). It acts only when the author is the bot user of the updater App and the head branch starts with image-updater-. If every changed file is a modified pin file of an app on the auto-merge list, and every changed line is a tag:, newTag: or digest: line, it turns on auto-merge with the squash method (gh pr merge --auto --squash). Otherwise it requests a review from seanb4t (gh pr edit --add-reviewer seanb4t). The workflow never merges a pull request itself. GitHub merges it after the required checks pass.

Image Updater v1.3.1 cannot label a pull request. Labels exist only on its unreleased main branch. The files and their diffs are facts of each pull request, so the workflow uses them.

A check of the files alone trusts the whole content of a pin file. A kustomization.yaml can add a resource, and a Helm values file can set any key of its chart. For the temporal workers, that file also holds the guarded-apply configuration and the Vault roles. The line check limits auto-merge to a version change. The updater App then can at most choose another tag of the same repository, and the tag guard of decision 6 keeps each tag fixed.

The default auto-merge list is life-agent, life-agent-gateway, hl-assets, firewalla-mcp and temporal-workers. The attended apps are dns-control and agent-memory, because dns-control is the only writer of pdns and an engram release can need a schema migration. The pin of dns-control also moves the webhook of external-dns, as decision 12 says.

The workflow uses GITHUB_TOKEN. A merge through that token starts no workflow on push. No such workflow reads argocd/app-configs/, and ArgoCD still gets the webhook of the push.

8. One Renovate rule hands each image over

Section titled “8. One Renovate rule hands each image over”

One rule in .github/renovate.json sets enabled: false for matchDatasources: ["docker"] and an exact matchPackageNames list of the images that Image Updater owns. Its description names Image Updater as the owner. It is the last package rule, so no later rule enables these images again. Each pull request that gives an image to Image Updater adds that image to the list. Pull requests 3 and 4 remove the two life agent rules.

To hold a bad version, a person adds it to ignoreTags of that image in its ImageUpdater resource, through a pull request. Image Updater then opens a pull request for the newest tag that is not held. GetNewestVersionFromTags picks the newest allowed tag and does not compare it with the running tag, so that pull request moves the pin back.

A revert of an updater pull request is not a hold. The live image is then older than the newest tag, and the next poll opens the same update again. For the two main images, a hold removes the image from its ImageUpdater resource, and a person sets the digest, as docs/operations/life-agent.md, section “Roll back an image”, describes.

10. engram and the temporal workers: the image moves alone

Section titled “10. engram and the temporal workers: the image moves alone”

Image Updater moves the image tag, and Renovate moves the chart. A first-party chart therefore accepts an image that was released after the chart. The worker chart accepts one only from the release of fzymgc-house/temporal-workers#197, so pull request 6 first moves the chart pin to that release.

The engram updater pull request is attended. If an engram release needs a newer chart, Sean adds the chart bump to the same pull request before the merge. The summarize CronJob moves into the engram chart, so one value drives every engram workload that ArgoCD runs. The three Jobs that a person runs by hand get no pin. Their runbook reads the running tag from the engram Deployment at run time. Today all engram pins move in one pull request, and this decision replaces that practice.

On 2026-10-11 Sean decided that Renovate merges an engram chart release with no release-age floor and no review, for every update type. A first-party release deploys at once, and Argo CD rolls a bad one back. A Qdrant downgrade stays a snapshot restore. The rule engram chart in .github/renovate.json lifts the major guard and the pre-1.0 guard for that one chart.

The summarize CronJob was already in the chart, behind memory.summarize.cronjob.enabled, but it ran /engram serve with the arguments of the sweep and failed at once. seanb4t/engram#808 (release 0.24.1) makes it run summarize-missing with the security context, limits and deadline of the hand-written CronJob.

Image Updater accepts the package webhook of GitHub. That needs a public route and a GitHub App that subscribes to package events. Two minutes of delay is enough, so this change uses polling only.

12. One pin for an image that two apps run

Section titled “12. One pin for an image that two apps run”

dns-control runs twice: as the Deployment of the app dns-control, and as the webhook in the pod of external-dns. Up to 0.22.0, each app had its own pin and its own ImageUpdater resource. Image Updater names the head branch of a pull request image-updater-<targetKey>-<sha256>. <targetKey> hashes the repository, the branch and the target path, and <sha256> hashes the update. Two target files gave two pull requests for one release. The releases 0.20.3, 0.21.0 and 0.22.0 each opened two, with one <sha256> and two values of <targetKey>. On 2026-10-11 Sean asked for one.

The pin moves to a kustomize Component, argocd/app-configs/dns-control-image/kustomization.yaml, which holds the one images: entry. The kustomization of dns-control includes it in components:. external-dns moves to the pattern of argocd/app-configs/clickstack-chart: one git source whose kustomization inflates the chart through helmCharts: and includes the same Component. Its values file sets the tag of the webhook image to set-by-kustomize, and the Component replaces that tag. Only the ImageUpdater resource of dns-control remains, and it writes the Component, so a release opens one pull request.

Image Updater edits only the images node of the target file, with kyaml, and keeps the rest of the file, so the file stays kind: Component. On 2026-10-11, kustomize build --enable-helm with Kustomize v5.8.3 rendered the new external-dns as the same nine resources as the Helm source plus the kustomization of today, with the webhook image from the Component.

The two pods now roll together, so dns-control no longer moves before its webhook. external-dns retries a failed call of the webhook at its next interval of one minute. A dns-control release therefore keeps the routes that the webhook of the previous version calls, and the minute between the two restarts costs at most one retry.

The chart version of external-dns moves from the Application to helmCharts:. The Renovate rule “Helm charts” matches the managers helm-values, helmv3 and argocd only, so a new rule after it gives the manager kustomize and the datasource helm the same settings. The ClickStack rule comes later and still overrides it for that chart.

Alternatives:

  • Two pull requests, one for each pin. Sean asked for one on 2026-10-11.
  • Fold the Deployment of external-dns into the app dns-control. This also gives one pin, but it moves the live resources of external-dns from one Argo CD app to another.
  • Auto-merge the pull request of external-dns after the one of dns-control. That is still two pull requests.
  • [A tag-only pin trusts the publisher. A re-pushed tag changes the content on the next pull, and each node caches its own copy under the same tag.] → Each publisher gets the guard of decision 6. Pull request 1 adds it here, and the cross-repository tasks add it in the other repositories.
  • [Image Updater v1.3 builds against the ArgoCD 3.5 API types, and the cluster runs ArgoCD v3.2.1.] → Pull request 3 uses the life agent stateless images first, which carry the least risk.
  • [Image Updater rewrites kustomization.yaml and can drop comments in it.] → The images: entries carry no comments. The first updater pull requests get a review against the requirement “An updater pull request changes only pin files”.
  • [Each pull request that changes a pin from <tag>@sha256:<digest> to <tag> changes the image string. The pods restart once with the same content.] → Pull request 5 restarts dns-control and the webhook of external-dns, so it merges in a quiet window.
  • [An image that no pod runs is not in status.summary.images, so Image Updater skips it.] → None of the images in scope scales to zero.
  • [dns-control and the webhook of external-dns restart together after decision 12.] → external-dns retries at its next interval, and a release keeps the routes of the previous webhook.
  • [Image Updater writes a file of kind Component for the first time.] → The first updater pull request after pull request 8 gets a review against the requirement “An updater pull request changes only pin files”.
  • [The updater App can write to the repository.] → It is installed on one repository, and branch protection still requires a pull request with pr-gates and Octopus Review.
  • [An engram image can need a newer engram chart.] → The engram pull request is attended, as decision 10 says.
  • [After a worker release, the old worker version runs until each workflow pinned to it ends, and then for one more hour (scaledownDelay: 1h). ArgoCD sorts the live images by name, and Image Updater takes the first match, which can be the old tag.] → Image Updater then writes the pin that is already there, and GitHub answers 422 each cycle until the old pods stop. No pull request comes from it. The runbook names this error as expected for up to five hours after a worker release, and a longer run as a workflow that does not end.

The pull requests merge in the order of proposal.md, section “Pull request split”. Each one has a rollback. An agent opens the revert pull request, and Sean merges it:

Terminal window
git switch -c revert/<name> origin/main
git revert -m 1 <merge-sha>
Step Rollback
1. The tag guard The revert pull request. Nothing in the cluster changes.
2. The updater The revert pull request. ArgoCD prunes the Application. The CRD stays if the chart ships it in crds/.
3 to 7. The handover of each app The revert pull request restores the old pin lines, the Renovate entries and the ImageUpdater resource. The pods restart once.
8. One pin for dns-control The revert pull request restores the two pins and the ImageUpdater resource of external-dns. Nothing restarts, because both renders stay the same.
Any merge in flight gh workflow disable image-updater-automerge.yml --repo fzymgc-house/selfhosted-cluster stops every new auto-merge at once.
A bad image The hold of decision 9.

After a rollback of pull request 5, this command waits until dns-control runs again:

Terminal window
kubectl --context fzymgc-house -n dns-control rollout status deployment/dns-control --timeout=5m

Each question has a default, and the tasks follow that default.

  1. engram: does decision 10 replace the practice that all engram pins move in one pull request? The default is yes, with the attended pull request and the chart bump that Sean adds when a release needs it.
  2. The auto-merge list. The default is the list in decision 7. A change moves an app between the list and the attended set, in the workflow and in the runbook only.
  3. GitHub immutable releases on fzymgc-house/selfhosted-cluster. They protect the dns-control/v<version> tags and the worker tarball. The release workflow makes a draft first and publishes it after a test, so the setting needs a test against that flow first. The default is to leave it off in this change.
  4. Does Renovate move the engram chart? In August 2026 the GHCR lookup of Renovate for the private engram packages failed with 403, and the dependency dashboard now lists the chart. Pull request 7 finds out. If Renovate cannot move it, Sean moves the chart by hand, as now. Answer: yes. The dependency dashboard detects the chart with no lookup warning. Renovate never opened a chart pull request, because Sean moved each release within an hour and the chart rules waited 3 or 7 days.
  5. An alert on the errors of Image Updater. The default is no alert in this change. The controller log reaches ClickStack like every pod log.
  6. octopus and headroom-otel join later, after octopus has a tag to follow and headroom-otel has one tag for each build.