Design: `k8s-utils` container image

Bead: hl-ozs Date: 2026-06-05 Status: Draft (pending design-reviewer)

Problem

The agent-memory/qdrant-memory-snapshot CronJob fails on every run. It uses a bare alpine:3.21 image and installs tools at runtime with apk add --no-cache curl jq. The agent-memory pods have no internet egress to the Alpine package CDN (dl-cdn.alpinelinux.org), so the apk index fetch fails:

ERROR: unable to select packages:
  curl (no such package):
    required by: world[curl]
  jq (no such package):
    required by: world[jq]

(Empty world[] resolution = the package index could not be fetched at all.)

Image pulls succeed because they happen at the node/containerd layer; the pod’s egress to the public internet is what is blocked. Every run since the CronJob was created (~3 days) has failed, leaving ~10 Error pods (backoffLimit: 2 × failedJobsHistoryLimit: 3 × multiple daily runs).

Root cause: installing packages at container runtime couples a job that only needs to talk to an in-cluster service to external network availability.

Key grounding fact: the snapshot job only POSTs to the in-cluster Qdrant REST API (qdrant.agent-memory.svc.cluster.local:6333). The R2/Cloudflare snapshot credentials back the Qdrant pod’s env (native S3 snapshot upload), not the job. So the job needs zero internet egress once curl+jq are baked into the image.

Goals

• Eliminate runtime package installation from the snapshot CronJob.
• Provide a small, self-owned, multi-arch utility image for in-cluster REST/batch jobs, reusable beyond this one CronJob.
• Harden the snapshot job: non-root, read-only rootfs, all caps dropped.

Non-goals

• A general-purpose in-cluster debug/ops image (kubectl, tcpdump, ping). Those need a different security posture (API token + RBAC, CAP_NET_RAW) and would be a separate image if ever wanted.
• Changing the snapshot logic, schedule, retention (KEEP=14), or history limits.

Decisions

Decision	Choice	Rationale
Scope	Minimal job-utils	Purpose-built for in-cluster REST jobs; small attack surface; fast pulls on ARM tpi-* nodes
Base	alpine:3.21	Matches the image being replaced; tiny
Name	ghcr.io/fzymgc-house/k8s-utils	Follows repo GHCR-owner convention
User	USER 65532 (nonroot)	Enables runAsNonRoot + readOnlyRootFilesystem on consumers
Visibility	Public GHCR package	Stock Alpine tools, nothing sensitive; no imagePullSecrets needed
Tools	curl jq ca-certificates bind-tools netcat-openbsd ldns	HTTP + JSON + DNS (dig/drill) + TCP probe (nc)
Excluded	ping, kubectl	ping needs CAP_NET_RAW (broken under dropped caps); kubectl needs SA token/RBAC and pulls toward a broader image
Tag pinning	Short-SHA (${GITHUB_SHA::7})	Auditable, no :latest drift; immutable by policy (don’t overwrite a SHA tag) — see ADR hl-pvn for the digest-pinning caveat
Stale pods	Manual cleanup once	History config is not the bug; failedJobsHistoryLimit:3 unchanged

Architecture

Three coordinated pieces, following the existing images/ + .github/workflows/ convention (mirrors cf-ssh-bastion):

Component	Path
Dockerfile	images/k8s-utils/Dockerfile
Build workflow	.github/workflows/build-k8s-utils-image.yaml
CronJob repoint	argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml

Dockerfile

# SPDX-License-Identifier: MIT
# Minimal in-cluster job/debug utilities: HTTP + JSON + DNS + TCP connectivity.
# https://github.com/fzymgc-house/selfhosted-cluster
FROM alpine:3.21

RUN apk add --no-cache \
      curl \
      jq \
      ca-certificates \
      bind-tools \
      netcat-openbsd \
      ldns

# Internal CA trust (shared images/certs/), so curl against internal HTTPS works.
COPY certs/*.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

# No /etc/passwd entry for this UID; $HOME is undefined — fine for script
# workloads that don't resolve $HOME or invoke whoami.
USER 65532
# CMD (not ENTRYPOINT): a Kubernetes container `command:` overrides ENTRYPOINT
# entirely, so the snapshot CronJob's `command: ["/bin/sh","-c"]` already drives
# execution. This CMD is only the default for standalone `docker run`.
CMD ["/bin/sh"]

Notes:

• ca-certificates must be installed before update-ca-certificates runs.
• Build context is images/ (so the shared images/certs/ is COPY-able), matching build-bastion-image.yaml (context: images, file: images/<name>/Dockerfile).
• USER 65532 runs the process as a bare numeric UID with no /etc/passwd entry (distroless convention). Safe here because the script reads only; a future consumer relying on $HOME (e.g. curl --netrc) would need a passwd entry.
• readOnlyRootFilesystem: true holds: curl -fsS and jq -e only read; the script writes nothing (no /tmp, no $HOME).

Build workflow

Mirror .github/workflows/build-bastion-image.yaml:

• Triggers: push to main on images/k8s-utils/** and images/certs/**; workflow_dispatch.
• build-amd64 + build-arm64 jobs on runs-on self-hosted runners.
• create-manifest stitches a multi-arch manifest, pushes :<short-sha> and :latest.
• REGISTRY=ghcr.io, IMAGE_NAME=${{ github.repository_owner }}/k8s-utils.

Multi-arch is required: the cluster runs ARM tpi-* nodes; an amd64-only image would exec format error.

CronJob changes

argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml:

1. image: alpine:3.21 → image: ghcr.io/fzymgc-house/k8s-utils:<short-sha> (tag filled in after the first build publishes).
2. Remove the apk add --no-cache --quiet curl jq line from args.
3. Replace the container securityContext and delete the root/apk comment block:

securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

Pod-level seccompProfile: RuntimeDefault and automountServiceAccountToken: false unchanged. No imagePullSecrets (public package). backoffLimit, activeDeadlineSeconds, schedule, retention unchanged.

Data flow

GH push (images/k8s-utils/**) → build workflow → multi-arch image in GHCR
  → (manually flip package to public after first publish)
  → CronJob manifest pinned to :<short-sha>
  → ArgoCD syncs → kubelet pulls (no egress, no pull secret)
  → job runs: curl POST snapshot → Qdrant → Qdrant uploads to R2

Error handling

• Build: no || true masking; apk add / update-ca-certificates failures fail the workflow.
• Job: existing set -eu, curl -fsS, jq -e fail-fast on HTTP/JSON errors; backoffLimit: 2, activeDeadlineSeconds: 1800 unchanged.

Sequencing constraint

The image must exist in GHCR (and be public) before ArgoCD syncs the new CronJob, else ImagePullBackOff. Because the tag is a short-SHA, the manifest’s tag is updated after the first build publishes — either a follow-up commit in the same PR or a small second PR. The build workflow lands first.

Testing / verification

1. After merge, confirm the build workflow published ghcr.io/fzymgc-house/k8s-utils:<sha> (amd64+arm64 manifest).
2. Flip the GHCR package to public.
3. ArgoCD syncs the CronJob.
4. Manually trigger a run: kubectl create job -n agent-memory --from=cronjob/qdrant-memory-snapshot test-snap.
5. Confirm the job reaches Completed and a new snapshot lands in R2.
6. Clean up the ~10 stale Error pods/jobs once (manual, one-time).

If ArgoCD shows ImagePullBackOff after sync, the most likely cause is the GHCR package still being private — confirm/flip it at github.com/orgs/fzymgc-house/packages (the public-visibility step above). New GHCR packages default to private on first publish.

Rollback

Revert the CronJob commit (re-point to alpine:3.21 + restore apk add). The image and workflow are additive and harmless if unused.