Pin self-owned images by short-SHA, not mutable tags
Date: 2026-06-06 Status: Accepted Decision: hl-pvn Deciders: Sean
Context
Section titled “Context”In-cluster jobs that reference container images by mutable tags (:latest or floating semver) can silently receive different image content across node pulls, undermining reproducibility and GitOps auditability. The cluster runs mixed ARM (tpi-*) and amd64 nodes, so multi-arch manifest identity matters. The cf-ssh-bastion image already established short-SHA tagging via ${GITHUB_SHA::7}.
Decision
Section titled “Decision”All self-owned images published to GHCR are tagged with the 7-character short-SHA of the triggering commit, and Kubernetes manifests pin that tag (updated via a Git commit before ArgoCD syncs). :latest is also pushed for local convenience but is never referenced in manifests.
Note on immutability: an OCI registry tag is a mutable pointer — re-running a build for the same commit CAN overwrite an existing <short-sha> tag. The short-SHA scheme therefore gives immutability by policy, not by registry guarantee: do not rebuild-and-overwrite an existing short-SHA tag; if a rebuild is genuinely needed, publish under a new tag (e.g. <short-sha>-<run-id>). For workloads requiring a hard guarantee, pin by digest (@sha256:...) instead of by tag.
Rationale
Section titled “Rationale”Mutable tags allow the image content referenced by a manifest to change without a Git diff, defeating ArgoCD drift detection and audit. Pinning the short-SHA tag maps each manifest to a specific commit and (by the no-overwrite policy) a specific build. It is already the cf-ssh-bastion convention, so consistency reduces cognitive overhead. Tag-or-digest pinning makes rollback a single-line manifest revert.
Alternatives Considered
Section titled “Alternatives Considered”:latest or a floating semver tag — no manifest update needed after each build, but mutable: kubelet image-pull-policy can diverge across nodes, ArgoCD cannot detect drift, reproducibility breaks. Digest pinning (@sha256:...) — strongest guarantee (registry-enforced immutability) but the digest is only known post-build and is less human-readable; reserved for workloads that need a hard guarantee rather than the default.
Consequences
Section titled “Consequences”Positive: every running image version is traceable to a Git commit; ArgoCD drift detection works (a tag change in Git triggers a sync); rollback is a one-line revert. Negative: every image rebuild requires a manifest commit to update the tag, adding a post-build sequencing step; tag-level immutability depends on operational discipline (no overwrite) rather than a registry guarantee. Neutral: :latest is pushed but reserved for local testing only; digest pinning remains available for workloads that need registry-enforced immutability.