Skip to content

Argo CD Image Updater

Argo CD Image Updater moves the pins of first-party images. It reads GHCR every two minutes. If an image that it owns has a new version or a new digest, it opens a pull request against main. That pull request moves the pin of the image. Renovate keeps every other dependency, and it keeps the charts of first-party apps too. kubectl commands on this page assume --context fzymgc-house.

Property Value
Namespace argocd
Workload deploy/argocd-image-updater-controller
Chart and pin argocd-image-updater from argo-helm, targetRevision in argocd/cluster-app/templates/argocd-image-updater.yaml. Renovate moves it
ImageUpdater resources argocd/app-configs/argocd-image-updater/<app>.yaml, one for each Application
Poll interval 2 minutes (--interval 2m)
Commit method api (--git-commit-method api). GitHub creates each commit through the App and marks it Verified
Git credential Secret argocd-image-updater-git, from Vault fzymgc-house/cluster/argocd-image-updater
GHCR credential Secret argocd-image-updater-ghcr, from Vault fzymgc-house/cluster/ghcr/pull-secret
GitHub webhook None. The poll is the only trigger
Image Application Pin Strategy Merge
ghcr.io/seanb4t/life-agent-sandbox life-agent argocd/app-configs/life-agent/kustomization.yaml digest of the tag main auto-merge
ghcr.io/seanb4t/life-agent-smokescreen life-agent argocd/app-configs/life-agent/kustomization.yaml digest of the tag main auto-merge
ghcr.io/seanb4t/life-agent-hermes life-agent-gateway argocd/app-configs/life-agent-gateway/kustomization.yaml alphabetical of the date tags YYYYMMDD.NNN auto-merge
ghcr.io/fzymgc-house/hl-assets-uploader hl-assets argocd/app-configs/hl-assets/kustomization.yaml semver auto-merge
ghcr.io/fzymgc-house/firewalla-mcp firewalla-mcp argocd/app-configs/firewalla-mcp/kustomization.yaml semver auto-merge
ghcr.io/fzymgc-house/dns-control dns-control argocd/app-configs/dns-control-image/kustomization.yaml, a Component that dns-control and external-dns include semver review
ghcr.io/fzymgc-house/temporal-workers/terraform temporal-workers workers.terraform.image.tag in argocd/app-configs/temporal-workers/values.yaml semver auto-merge
ghcr.io/seanb4t/engram agent-memory image.tag in argocd/app-configs/agent-memory/values.yaml semver review

The ImageUpdater resource of each Application is argocd/app-configs/argocd-image-updater/<app>.yaml. The pull request that hands an image to Image Updater adds a row here. The same pull request removes the image from Renovate, so each image has one owner at all times.

Image Updater opens one pull request for each target file. Its head branch is image-updater-<targetKey>-<sha256>, where <targetKey> hashes the repository, the branch and the target path. Two pin files for one image therefore give two pull requests for one release.

dns-control runs as the Deployment of dns-control and as the webhook of external-dns. Its one pin is the kustomize Component argocd/app-configs/dns-control-image/kustomization.yaml, which both kustomizations include in components:. external-dns inflates its chart through helmCharts:, so the Component can set the webhook image. Only the ImageUpdater resource of dns-control exists, and it writes the Component. Image Updater edits only the images: list of that file and keeps kind: Component.

Both pods restart in the same sync after the merge. external-dns retries the webhook at its next interval of one minute, so a dns-control release keeps the routes that the webhook of the previous version calls.

Only first-party images go to Image Updater. octopus, headroom-otel, k8s-utils, actions-runner and nas-otel stay out. octopus has commit tags only, and headroom-otel pushes the same version tag again for each build.

Each image that Image Updater owns has exactly one pin. The pin is the place where Image Updater writes:

  • For an app that kustomize builds, the pin is an entry in the images: list of its kustomization.yaml. The manifests name the image without a tag.
  • For an app that a chart builds, the pin is the image tag key in a values file in git. The Application reads that file through a ref: values source and $values/... in valueFiles.

A kustomize pin looks like this:

images:
- name: ghcr.io/fzymgc-house/hl-assets-uploader
newTag: 0.4.1

A pin holds a tag and no digest. Each first-party release workflow refuses to push a version tag that exists in GHCR, so a version tag never moves. The two life agent images that follow main are the exception. Their tag moves by design, so their pins keep newTag: main and a digest.

If YAML reads a tag as a number, put the tag in quotes. The date tags of the life agent gateway, such as "20261009.001", are an example. Kustomize does not render an unquoted one. Image Updater quotes such a tag when it writes the pin.

The repository holds no ArgoCD override file (.argocd-source*.yaml). To make sure of this, run this command on main. It must print nothing:

Terminal window
git ls-files ':(glob)**/.argocd-source*'

An updater pull request changes only pin files under argocd/app-configs/: a kustomization.yaml or a values.yaml. Its head branch starts with image-updater-. The bot user of the GitHub App of the updater is its author.

The workflow .github/workflows/image-updater-automerge.yml decides the merge posture from the files and the changed lines of the pull request. It acts only for the author fzymgc-image-updater[bot] and a head branch that starts with image-updater-. It turns on auto-merge with squash when both of these conditions are true:

  1. Every file is the kustomization.yaml or the values.yaml of an app on its auto-merge list, and GitHub reports the file as modified.
  2. Every changed line is a tag:, newTag: or digest: line with one value.

The workflow approves the pull request with GITHUB_TOKEN, because the ruleset of main needs one approving review, and GitHub then merges the pull request after pr-gates and Octopus Review pass. Otherwise, the workflow turns auto-merge off, dismisses its own approval, and requests a review from seanb4t. The step summary lists each change that is not a version change. The workflow never merges a pull request itself.

The line check exists because a pin file can set any value of its app. A kustomization.yaml can add a resource, and a Helm values file can set any key of the chart, such as the guarded-apply configuration of the temporal workers. Image Updater changes only the version line, so its pull requests pass. A change to a repository name, a configuration line or a resource waits for a review.

The auto-merge list holds these apps, by their directory under argocd/app-configs/: life-agent, life-agent-gateway, hl-assets, firewalla-mcp and temporal-workers. The attended apps are dns-control and agent-memory. dns-control is the only writer of pdns, and an engram release can need a schema migration. The pin of dns-control also moves the webhook of external-dns. To move an app between the two sets, change the list in the workflow and in this section, in one pull request.

To stop every new auto-merge at once, run:

Terminal window
gh workflow disable image-updater-automerge.yml --repo fzymgc-house/selfhosted-cluster

Image Updater moves image.tag in argocd/app-configs/agent-memory/values.yaml. The tag drives memory-mcp and the summarize CronJob memory-mcp-summarize, which both come from the engram chart. The pull request waits for a review. Before you merge it, read the release notes of engram:

  1. If the release needs a newer chart, push the new targetRevision of argocd/cluster-app/templates/agent-memory.yaml to the branch of the updater pull request. Image Updater finds its pin on the branch and pushes nothing, so your commit stays.
  2. If the release adds a payload migration, run the migrate Job after the merge. Use the command of Run a hand-run engram Job with engram-migrate-schema-v1-job.yaml.

Renovate moves the engram chart, ghcr.io/seanb4t/charts/engram, under the rule engram chart. Its pull request opens with no release-age wait and merges after its checks pass, for every update type. Its dependency dashboard detects the chart and reports no lookup warning for it. To roll a chart back, revert that pull request. A chart that moved the Qdrant image is the exception: a Qdrant downgrade is a snapshot restore (Memory backup and restore).

Three Jobs in argocd/app-configs/agent-memory/ run only by hand: engram-migrate-schema-v1-job.yaml, engram-summarize-missing-memory-v4.yaml and reindex-memory-v4-job.yaml. Argo CD does not apply them. Their image line has no tag, so kubectl create -f alone pulls latest. This command sets the image that memory-mcp runs and creates the Job. Replace <job file> with the file name:

Terminal window
IMAGE="$(kubectl --context fzymgc-house -n agent-memory get deployment memory-mcp \
-o jsonpath='{.spec.template.spec.containers[?(@.name=="memory-mcp")].image}')" \
yq '(.spec.template.spec.containers[] | select(.image == "ghcr.io/seanb4t/engram")).image = strenv(IMAGE)' \
argocd/app-configs/agent-memory/<job file> | kubectl --context fzymgc-house create -f -

A Job with a fixed name, such as engram-reindex-memory-v4, fails with AlreadyExists while an earlier run of it exists. Delete the earlier Job first.

To hold a bad version, add it to ignoreTags of that image in its ImageUpdater resource, through a pull request. The values of ignoreTags are glob patterns. For example, this holds dns-control 0.20.0:

spec:
applicationRefs:
- namePattern: dns-control
images:
- alias: dns-control
imageName: ghcr.io/fzymgc-house/dns-control
commonUpdateSettings:
updateStrategy: semver
ignoreTags:
- 0.20.0

The webhook of external-dns reads the same pin, so a hold in dns-control.yaml covers it too.

Image Updater then opens a pull request that sets the newest tag that is not held. If the cluster runs the held tag, that pull request moves the pin back.

Do not revert an updater pull request to hold a version. The next poll sees a live image that is older than the newest tag, and it opens the same update again.

The two life agent images that follow main have no version to hold. To hold one of them, remove the image from its ImageUpdater resource. Then set the digest by hand, as Roll back an image describes.

What failed Rollback
A bad image Hold the version, as Hold a version describes
Auto-merge of a bad pull request Disable the workflow with the command in What an updater pull request changes, then hold the version
The handover of an app Revert the pull request of the handover. The revert restores the old pin lines, the Renovate entries and the ImageUpdater resource. The pods of the app restart once
The updater itself Revert the pull request that added it. ArgoCD prunes the Application. The ImageUpdater CRD stays, because the chart marks it helm.sh/resource-policy: keep

An agent opens each revert pull request, and Sean merges it. The repository merges by squash, so each pull request is one commit on main. Revert that commit:

Terminal window
git switch -c revert/<name> origin/main
git revert <commit of the pull request on main>

Add an image only if its publisher refuses to push a version tag that exists. The step “Make sure that the version is new” in .github/workflows/release.yml is the model.

  1. Run crane ls <image>. Write an allowTags regular expression that matches the release tags only. It must not match latest, main, a commit tag or the tag of one architecture, such as 0.2.2-amd64. Make sure that crane ls <image> | rg -c '<expression>' prints the count of the release tags.
  2. Start the allowTags value with regexp:, and put the value in single quotes. Without the prefix, the controller matches no tag and never updates the image.
  3. Move the pin to the place that The pin of an image describes. Make sure that the render of the app before and after shows the same image, apart from the removed digest.
  4. Add argocd/app-configs/argocd-image-updater/<app>.yaml and list it in the kustomization.yaml of that directory. Give it the annotation argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true.
  5. Add the image to the handover rule of .github/renovate.json. Remove each other Renovate rule for that image.
  6. Add the image to the table in What Image Updater owns. If its app merges with no person, add the app to the auto-merge list of the workflow too.

To make sure that a new resource is valid, run a server dry run. It writes nothing:

Terminal window
kubectl apply --dry-run=server -f argocd/app-configs/argocd-image-updater/<app>.yaml

The updater writes to the repository through its own GitHub App. The App of ArgoCD (fzymgc-house-argocd-app) has no access to Pull requests, so it cannot open a pull request. The ArgoCD repo server never holds the credential of the updater.

The App has these permissions, and no webhook:

Permission Access
Contents Read and write
Pull requests Read and write
Metadata Read

It is installed on fzymgc-house/selfhosted-cluster only. Branch protection still requires a pull request with pr-gates and Octopus Review for each commit that the App makes.

To write the credential to Vault, run this on a host that has vault logged in. Read the installation ID from the URL of the installation page of the App (https://github.com/organizations/fzymgc-house/settings/installations/<installation id>):

Terminal window
vault kv put secret/fzymgc-house/cluster/argocd-image-updater \
github_app_id=<app id> \
github_app_installation_id=<installation id> \
github_app_private_key=@<path to the private key .pem file>

The ExternalSecret argocd-image-updater-git copies the three fields to the keys githubAppID, githubAppInstallationID and githubAppPrivateKey. The controller reads those key names. If they are missing, it commits with git instead of the API, and GitHub does not mark the commits Verified.

The controller reads the Secret at each write-back, so a change needs no restart. To rotate the private key, patch the field and make the ExternalSecret copy it at once:

Terminal window
vault kv patch secret/fzymgc-house/cluster/argocd-image-updater \
github_app_private_key=@<path to the new private key .pem file>
kubectl -n argocd annotate externalsecret argocd-image-updater-git force-sync="$(date +%s)" --overwrite
Terminal window
kubectl -n argocd logs deploy/argocd-image-updater-controller --since=1h

The controller logs JSON. For history beyond the pod, query default.otel_logs in ClickStack with ResourceAttributes['k8s.pod.name'] LIKE 'argocd-image-updater-controller-%'. The timestamp column is Timestamp.

The level of a line is in the JSON field level. ClickStack sets SeverityText to INFO for each line of the controller, also for an error. To find the errors, filter on JSONExtractString(Body, 'level') = 'error'.

Two error lines are expected. The first, couldn't find remote ref image-updater-..., comes at each new update. The controller looks for the branch of the pull request before it creates that branch.

The second, 422 Validation Failed with Field:head Code:invalid, comes once, in the cycle right after an updater pull request merges. In that cycle, a pod of the old version has not stopped yet. The controller sees the old version as live and builds the same update again. GitHub deleted the branch at the merge, so no pull request opens. The next cycle has no error.

After a release of a temporal worker, the 422 line repeats until the old worker version stops. That version runs until each workflow pinned to it ends, and then for one more hour (scaledownDelay). ArgoCD sorts the live images by name, and if the old tag sorts first, the controller reads it as the live version. No pull request comes from it. If the line repeats for more than five hours, a workflow on the old version does not end. See An old worker version does not stop.

In any other case, a 422 line in a later cycle is a fault.