Argo CD Image Updater
Argo CD Image Updater moves the pins of first-party images. It reads GHCR every two minutes. If an
image that it owns has a new version or a new digest, it opens a pull request against main. That
pull request moves the pin of the image. Renovate keeps every other dependency, and it keeps the
charts of first-party apps too. kubectl commands on this page assume --context fzymgc-house.
What exists
Section titled “What exists”| Property | Value |
|---|---|
| Namespace | argocd |
| Workload | deploy/argocd-image-updater-controller |
| Chart and pin | argocd-image-updater from argo-helm, targetRevision in argocd/cluster-app/templates/argocd-image-updater.yaml. Renovate moves it |
| ImageUpdater resources | argocd/app-configs/argocd-image-updater/<app>.yaml, one for each Application |
| Poll interval | 2 minutes (--interval 2m) |
| Commit method | api (--git-commit-method api). GitHub creates each commit through the App and marks it Verified |
| Git credential | Secret argocd-image-updater-git, from Vault fzymgc-house/cluster/argocd-image-updater |
| GHCR credential | Secret argocd-image-updater-ghcr, from Vault fzymgc-house/cluster/ghcr/pull-secret |
| GitHub webhook | None. The poll is the only trigger |
What Image Updater owns
Section titled “What Image Updater owns”| Image | Application | Pin | Strategy | Merge |
|---|---|---|---|---|
ghcr.io/seanb4t/life-agent-sandbox |
life-agent |
argocd/app-configs/life-agent/kustomization.yaml |
digest of the tag main |
auto-merge |
ghcr.io/seanb4t/life-agent-smokescreen |
life-agent |
argocd/app-configs/life-agent/kustomization.yaml |
digest of the tag main |
auto-merge |
ghcr.io/seanb4t/life-agent-hermes |
life-agent-gateway |
argocd/app-configs/life-agent-gateway/kustomization.yaml |
alphabetical of the date tags YYYYMMDD.NNN |
auto-merge |
ghcr.io/fzymgc-house/hl-assets-uploader |
hl-assets |
argocd/app-configs/hl-assets/kustomization.yaml |
semver |
auto-merge |
ghcr.io/fzymgc-house/firewalla-mcp |
firewalla-mcp |
argocd/app-configs/firewalla-mcp/kustomization.yaml |
semver |
auto-merge |
ghcr.io/fzymgc-house/dns-control |
dns-control |
argocd/app-configs/dns-control-image/kustomization.yaml, a Component that dns-control and external-dns include |
semver |
review |
ghcr.io/fzymgc-house/temporal-workers/terraform |
temporal-workers |
workers.terraform.image.tag in argocd/app-configs/temporal-workers/values.yaml |
semver |
auto-merge |
ghcr.io/seanb4t/engram |
agent-memory |
image.tag in argocd/app-configs/agent-memory/values.yaml |
semver |
review |
The ImageUpdater resource of each Application is argocd/app-configs/argocd-image-updater/<app>.yaml.
The pull request that hands an image to Image Updater adds a row here. The same pull request removes
the image from Renovate, so each image has one owner at all times.
An image that two apps run
Section titled “An image that two apps run”Image Updater opens one pull request for each target file. Its head branch is
image-updater-<targetKey>-<sha256>, where <targetKey> hashes the repository, the branch and the
target path. Two pin files for one image therefore give two pull requests for one release.
dns-control runs as the Deployment of dns-control and as the webhook of external-dns. Its one pin
is the kustomize Component argocd/app-configs/dns-control-image/kustomization.yaml, which both
kustomizations include in components:. external-dns inflates its chart through helmCharts:, so
the Component can set the webhook image. Only the ImageUpdater resource of dns-control exists, and
it writes the Component. Image Updater edits only the images: list of that file and keeps
kind: Component.
Both pods restart in the same sync after the merge. external-dns retries the webhook at its next interval of one minute, so a dns-control release keeps the routes that the webhook of the previous version calls.
Only first-party images go to Image Updater. octopus, headroom-otel, k8s-utils,
actions-runner and nas-otel stay out. octopus has commit tags only, and headroom-otel pushes
the same version tag again for each build.
The pin of an image
Section titled “The pin of an image”Each image that Image Updater owns has exactly one pin. The pin is the place where Image Updater writes:
- For an app that kustomize builds, the pin is an entry in the
images:list of itskustomization.yaml. The manifests name the image without a tag. - For an app that a chart builds, the pin is the image tag key in a values file in git. The
Application reads that file through a
ref: valuessource and$values/...invalueFiles.
A kustomize pin looks like this:
images: - name: ghcr.io/fzymgc-house/hl-assets-uploader newTag: 0.4.1A pin holds a tag and no digest. Each first-party release workflow refuses to push a version tag
that exists in GHCR, so a version tag never moves. The two life agent images that follow main are
the exception. Their tag moves by design, so their pins keep newTag: main and a digest.
If YAML reads a tag as a number, put the tag in quotes. The date tags of the life agent gateway,
such as "20261009.001", are an example. Kustomize does not render an unquoted one. Image Updater
quotes such a tag when it writes the pin.
The repository holds no ArgoCD override file (.argocd-source*.yaml). To make sure of this, run
this command on main. It must print nothing:
git ls-files ':(glob)**/.argocd-source*'What an updater pull request changes
Section titled “What an updater pull request changes”An updater pull request changes only pin files under argocd/app-configs/: a kustomization.yaml
or a values.yaml. Its head branch starts with image-updater-. The bot user of the GitHub App
of the updater is its author.
The workflow .github/workflows/image-updater-automerge.yml decides the merge posture from the
files and the changed lines of the pull request. It acts only for the author
fzymgc-image-updater[bot] and a head branch that starts with image-updater-. It turns on
auto-merge with squash when both of these conditions are true:
- Every file is the
kustomization.yamlor thevalues.yamlof an app on its auto-merge list, and GitHub reports the file as modified. - Every changed line is a
tag:,newTag:ordigest:line with one value.
The workflow approves the pull request with GITHUB_TOKEN, because the ruleset of main needs
one approving review, and GitHub then merges the pull request after pr-gates and Octopus Review
pass. Otherwise, the workflow turns auto-merge off, dismisses its own approval, and requests a review
from seanb4t. The step summary lists each
change that is not a version change. The workflow never merges a pull request itself.
The line check exists because a pin file can set any value of its app. A kustomization.yaml can
add a resource, and a Helm values file can set any key of the chart, such as the guarded-apply
configuration of the temporal workers. Image Updater changes only the version line, so its pull
requests pass. A change to a repository name, a configuration line or a resource waits for a
review.
The auto-merge list holds these apps, by their directory under argocd/app-configs/:
life-agent, life-agent-gateway, hl-assets, firewalla-mcp and temporal-workers. The
attended apps are dns-control and agent-memory. dns-control is the only writer of pdns, and an
engram release can need a schema migration. The pin of dns-control also moves the webhook of
external-dns. To move an app between the two sets,
change the list in the workflow and in this section, in one pull request.
To stop every new auto-merge at once, run:
gh workflow disable image-updater-automerge.yml --repo fzymgc-house/selfhosted-clusterengram
Section titled “engram”Image Updater moves image.tag in argocd/app-configs/agent-memory/values.yaml. The tag drives
memory-mcp and the summarize CronJob memory-mcp-summarize, which both come from the engram chart.
The pull request waits for a review. Before you merge it, read the release notes of engram:
- If the release needs a newer chart, push the new
targetRevisionofargocd/cluster-app/templates/agent-memory.yamlto the branch of the updater pull request. Image Updater finds its pin on the branch and pushes nothing, so your commit stays. - If the release adds a payload migration, run the migrate Job after the merge. Use the command of
Run a hand-run engram Job with
engram-migrate-schema-v1-job.yaml.
Renovate moves the engram chart, ghcr.io/seanb4t/charts/engram, under the rule engram chart. Its
pull request opens with no release-age wait and merges after its checks pass, for every update type.
Its dependency dashboard detects the chart and reports no lookup warning for it. To roll a chart back,
revert that pull request. A chart that moved the Qdrant image is the exception: a Qdrant downgrade is
a snapshot restore (Memory backup and restore).
Run a hand-run engram Job
Section titled “Run a hand-run engram Job”Three Jobs in argocd/app-configs/agent-memory/ run only by hand: engram-migrate-schema-v1-job.yaml,
engram-summarize-missing-memory-v4.yaml and reindex-memory-v4-job.yaml. Argo CD does not apply
them. Their image line has no tag, so kubectl create -f alone pulls latest. This command sets the
image that memory-mcp runs and creates the Job. Replace <job file> with the file name:
IMAGE="$(kubectl --context fzymgc-house -n agent-memory get deployment memory-mcp \ -o jsonpath='{.spec.template.spec.containers[?(@.name=="memory-mcp")].image}')" \ yq '(.spec.template.spec.containers[] | select(.image == "ghcr.io/seanb4t/engram")).image = strenv(IMAGE)' \ argocd/app-configs/agent-memory/<job file> | kubectl --context fzymgc-house create -f -A Job with a fixed name, such as engram-reindex-memory-v4, fails with AlreadyExists while an
earlier run of it exists. Delete the earlier Job first.
Hold a version
Section titled “Hold a version”To hold a bad version, add it to ignoreTags of that image in its ImageUpdater resource, through
a pull request. The values of ignoreTags are glob patterns. For example, this holds dns-control
0.20.0:
spec: applicationRefs: - namePattern: dns-control images: - alias: dns-control imageName: ghcr.io/fzymgc-house/dns-control commonUpdateSettings: updateStrategy: semver ignoreTags: - 0.20.0The webhook of external-dns reads the same pin, so a hold in dns-control.yaml covers it too.
Image Updater then opens a pull request that sets the newest tag that is not held. If the cluster runs the held tag, that pull request moves the pin back.
Do not revert an updater pull request to hold a version. The next poll sees a live image that is older than the newest tag, and it opens the same update again.
The two life agent images that follow main have no version to hold. To hold one of them, remove
the image from its ImageUpdater resource. Then set the digest by hand, as
Roll back an image describes.
Roll back
Section titled “Roll back”| What failed | Rollback |
|---|---|
| A bad image | Hold the version, as Hold a version describes |
| Auto-merge of a bad pull request | Disable the workflow with the command in What an updater pull request changes, then hold the version |
| The handover of an app | Revert the pull request of the handover. The revert restores the old pin lines, the Renovate entries and the ImageUpdater resource. The pods of the app restart once |
| The updater itself | Revert the pull request that added it. ArgoCD prunes the Application. The ImageUpdater CRD stays, because the chart marks it helm.sh/resource-policy: keep |
An agent opens each revert pull request, and Sean merges it. The repository merges by squash, so
each pull request is one commit on main. Revert that commit:
git switch -c revert/<name> origin/maingit revert <commit of the pull request on main>Add an image
Section titled “Add an image”Add an image only if its publisher refuses to push a version tag that exists. The step “Make sure
that the version is new” in .github/workflows/release.yml is the model.
- Run
crane ls <image>. Write anallowTagsregular expression that matches the release tags only. It must not matchlatest,main, a commit tag or the tag of one architecture, such as0.2.2-amd64. Make sure thatcrane ls <image> | rg -c '<expression>'prints the count of the release tags. - Start the
allowTagsvalue withregexp:, and put the value in single quotes. Without the prefix, the controller matches no tag and never updates the image. - Move the pin to the place that The pin of an image describes. Make sure that the render of the app before and after shows the same image, apart from the removed digest.
- Add
argocd/app-configs/argocd-image-updater/<app>.yamland list it in thekustomization.yamlof that directory. Give it the annotationargocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true. - Add the image to the handover rule of
.github/renovate.json. Remove each other Renovate rule for that image. - Add the image to the table in What Image Updater owns. If its app merges with no person, add the app to the auto-merge list of the workflow too.
To make sure that a new resource is valid, run a server dry run. It writes nothing:
kubectl apply --dry-run=server -f argocd/app-configs/argocd-image-updater/<app>.yamlThe GitHub App
Section titled “The GitHub App”The updater writes to the repository through its own GitHub App. The App of ArgoCD
(fzymgc-house-argocd-app) has no access to Pull requests, so it cannot open a pull request. The
ArgoCD repo server never holds the credential of the updater.
The App has these permissions, and no webhook:
| Permission | Access |
|---|---|
| Contents | Read and write |
| Pull requests | Read and write |
| Metadata | Read |
It is installed on fzymgc-house/selfhosted-cluster only. Branch protection still requires a pull
request with pr-gates and Octopus Review for each commit that the App makes.
To write the credential to Vault, run this on a host that has vault logged in. Read the
installation ID from the URL of the installation page of the App
(https://github.com/organizations/fzymgc-house/settings/installations/<installation id>):
vault kv put secret/fzymgc-house/cluster/argocd-image-updater \ github_app_id=<app id> \ github_app_installation_id=<installation id> \ github_app_private_key=@<path to the private key .pem file>The ExternalSecret argocd-image-updater-git copies the three fields to the keys githubAppID,
githubAppInstallationID and githubAppPrivateKey. The controller reads those key names. If they
are missing, it commits with git instead of the API, and GitHub does not mark the commits
Verified.
The controller reads the Secret at each write-back, so a change needs no restart. To rotate the private key, patch the field and make the ExternalSecret copy it at once:
vault kv patch secret/fzymgc-house/cluster/argocd-image-updater \ github_app_private_key=@<path to the new private key .pem file>kubectl -n argocd annotate externalsecret argocd-image-updater-git force-sync="$(date +%s)" --overwriteRead the controller log
Section titled “Read the controller log”kubectl -n argocd logs deploy/argocd-image-updater-controller --since=1hThe controller logs JSON. For history beyond the pod, query default.otel_logs in ClickStack with
ResourceAttributes['k8s.pod.name'] LIKE 'argocd-image-updater-controller-%'. The timestamp column
is Timestamp.
The level of a line is in the JSON field level. ClickStack sets SeverityText to INFO for each
line of the controller, also for an error. To find the errors, filter on
JSONExtractString(Body, 'level') = 'error'.
Two error lines are expected. The first, couldn't find remote ref image-updater-..., comes at each
new update. The controller looks for the branch of the pull request before it creates that branch.
The second, 422 Validation Failed with Field:head Code:invalid, comes once, in the cycle right
after an updater pull request merges. In that cycle, a pod of the old version has not stopped yet.
The controller sees the old version as live and builds the same update again. GitHub deleted the
branch at the merge, so no pull request opens. The next cycle has no error.
After a release of a temporal worker, the 422 line repeats until the old worker version stops.
That version runs until each workflow pinned to it ends, and then for one more hour
(scaledownDelay). ArgoCD sorts the live images by name, and if the old tag sorts first, the
controller reads it as the live version. No pull request comes from it. If the line repeats for more
than five hours, a workflow on the old version does not end. See
An old worker version does not stop.
In any other case, a 422 line in a later cycle is a fault.