Skip to content
fzymgc-house Cluster Docs

k8s-utils Image Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a self-owned, minimal, non-root, multi-arch ghcr.io/fzymgc-house/k8s-utils image (curl + jq + DNS/TCP tools) and repoint the failing agent-memory/qdrant-memory-snapshot CronJob at it, removing the broken runtime apk add.

Architecture: A new Dockerfile under images/k8s-utils/ (Alpine 3.21, tools baked in, USER 65532) is built multi-arch by a new GitHub Actions workflow mirroring build-bastion-image.yaml, publishing to GHCR. The CronJob manifest is then repointed at the new image by short-SHA, the apk add line removed, and its securityContext hardened to non-root + read-only rootfs. The image is made a public GHCR package so no pull secret is needed.

Tech Stack: Alpine Linux 3.21, Docker/BuildKit, GitHub Actions (runs-on self-hosted runners), GHCR, Kubernetes CronJob, ArgoCD (GitOps), jj VCS.

Design bead: hl-ozs Spec: docs/engineering/specs/2026-06-05-k8s-utils-image-design.md

File Structure

Section titled “File Structure”
FileStatusResponsibility
images/k8s-utils/DockerfileCreateDefines the minimal utility image (tools, CA trust, non-root user)
.github/workflows/build-k8s-utils-image.yamlCreateMulti-arch build + GHCR publish, mirrors build-bastion-image.yaml
argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yamlModifyRepoint image, drop apk add, harden securityContext

There are no automated unit tests for container images or Kubernetes manifests in this repo; verification is via local lint (yamllint, hadolint if available), kubectl --dry-run, the CI build itself, and a manual post-deploy job trigger. “Test” steps below are these verification commands.

Task 1: Create the k8s-utils Dockerfile

Section titled “Task 1: Create the k8s-utils Dockerfile”

Files:

  • Create: images/k8s-utils/Dockerfile
  • Test (verify): images/certs/ (shared CA certs COPY’d by the build)
  • Step 1: Write the Dockerfile

Create images/k8s-utils/Dockerfile with exactly this content:

# SPDX-License-Identifier: MIT
# Minimal in-cluster job/debug utilities: HTTP + JSON + DNS + TCP connectivity.
# Built to replace runtime `apk add` in batch jobs that have no internet egress.
# https://github.com/fzymgc-house/selfhosted-cluster
FROM alpine:3.21


# curl: HTTP(S) client | jq: JSON | ca-certificates: TLS trust store
# bind-tools: dig/nslookup/host | netcat-openbsd: nc (TCP probe) | ldns: drill
RUN apk add --no-cache \
      curl \
      jq \
      ca-certificates \
      bind-tools \
      netcat-openbsd \
      ldns


# Internal CA trust (shared images/certs/), so curl against internal HTTPS works.
# ca-certificates (installed above) provides update-ca-certificates.
COPY certs/*.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates


# Run as a bare numeric non-root UID (distroless "nonroot" convention). No
# /etc/passwd entry, so $HOME is undefined and `whoami` errors — fine for script
# workloads that don't resolve $HOME. Lets consumers set runAsNonRoot:true and
# readOnlyRootFilesystem:true.
USER 65532


# CMD (not ENTRYPOINT): a Kubernetes container `command:` overrides ENTRYPOINT
# entirely, so consuming jobs drive execution via their own command/args. This
# CMD is only the default for standalone `docker run`.
CMD ["/bin/sh"]
  • Step 2: Lint the Dockerfile (if hadolint available)

Run: command -v hadolint >/dev/null && hadolint images/k8s-utils/Dockerfile || echo "hadolint not installed; skipping" Expected: No errors, or the skip message. (DL3018 “pin apk versions” may warn — acceptable here, matching the unpinned style of images/cf-ssh-bastion/Dockerfile.)

  • Step 3: Verify the COPY source globs match real files

Run: ls images/certs/*.crt Expected: lists the 4 .crt files (root-ca, intermediate-ca1, vault-intermediate, intermediate-ca1-v1). Confirms the COPY certs/*.crt will not fail on an empty glob.

  • Step 4: Commit
Terminal window
jj describe -m "feat(k8s-utils): add minimal non-root curl+jq utility image (hl-ozs)


Alpine 3.21 with curl, jq, ca-certificates, bind-tools, netcat-openbsd, ldns.
Runs as UID 65532 with internal CA trust baked in. Replaces runtime 'apk add'
for in-cluster batch jobs with no internet egress.


Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"

(jj auto-tracks the new file; this is the working-copy commit @.)

Task 2: Create the multi-arch build workflow

Section titled “Task 2: Create the multi-arch build workflow”

Files:

  • Create: .github/workflows/build-k8s-utils-image.yaml
  • Reference (do not modify): .github/workflows/build-bastion-image.yaml
  • Step 1: Write the workflow

Create .github/workflows/build-k8s-utils-image.yaml with exactly this content (mirrors build-bastion-image.yaml; only name, IMAGE_NAME, and the paths/file references differ):

# SPDX-License-Identifier: MIT
# yaml-language-server: $schema=https://json.schemastore.org/github-workflow
---
name: Build k8s-utils Image


on:
  push:
    branches:
      - main
    paths:
      - 'images/k8s-utils/**'
      - 'images/certs/**'
  workflow_dispatch:


env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository_owner }}/k8s-utils


permissions:
  contents: read
  packages: write


jobs:
  build-amd64:
    runs-on:
      - runs-on=${{ github.run_id }}
      - runner=4cpu-linux-x64
      - image=ubuntu24-full-x64
      - spot=lowest-price
    outputs:
      image_tag: ${{ steps.vars.outputs.image_tag }}
    steps:
      - uses: runs-on/action@v2
      - name: Checkout repository
        uses: actions/checkout@v6


      - name: Set image tag
        id: vars
        run: echo "image_tag=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"


      - name: Log in to GHCR
        uses: docker/login-action@v4
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}


      - name: Build and push amd64 image
        uses: docker/build-push-action@v7
        with:
          context: images
          file: images/k8s-utils/Dockerfile
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.vars.outputs.image_tag }}-amd64


  build-arm64:
    runs-on:
      - runs-on=${{ github.run_id }}
      - runner=4cpu-linux-arm64
      - image=ubuntu24-full-arm64
      - spot=lowest-price
    steps:
      - uses: runs-on/action@v2
      - name: Checkout repository
        uses: actions/checkout@v6


      - name: Set image tag
        id: vars
        run: echo "image_tag=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"


      - name: Log in to GHCR
        uses: docker/login-action@v4
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}


      - name: Build and push arm64 image
        uses: docker/build-push-action@v7
        with:
          context: images
          file: images/k8s-utils/Dockerfile
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.vars.outputs.image_tag }}-arm64


  create-manifest:
    needs: [build-amd64, build-arm64]
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64
    env:
      IMAGE_TAG: ${{ needs.build-amd64.outputs.image_tag }}
    steps:
      - uses: runs-on/action@v2
      - name: Log in to GHCR
        uses: docker/login-action@v4
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}


      - name: Create and push multi-arch manifest
        run: |-
          set -euxo pipefail
          # Create versioned tag (short SHA)
          docker manifest create "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}" \
            "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}-amd64" \
            "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}-arm64"
          docker manifest push "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}"


          # Also tag as latest
          docker manifest create "${REGISTRY}/${IMAGE_NAME}:latest" \
            "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}-amd64" \
            "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}-arm64"
          docker manifest push "${REGISTRY}/${IMAGE_NAME}:latest"


          echo "Published image: ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}"
  • Step 2: Lint the workflow YAML

Run: yamllint .github/workflows/build-k8s-utils-image.yaml Expected: No errors. (If yamllint flags line length on the tags: lines, compare against build-bastion-image.yaml — the same lines pass there, so the config tolerates them.)

  • Step 3: Diff against the reference workflow to confirm only intended changes

Run: diff <(sed 's/cf-ssh-bastion/k8s-utils/g; s/SSH Bastion/k8s-utils/g' .github/workflows/build-bastion-image.yaml) .github/workflows/build-k8s-utils-image.yaml || true Expected: differences limited to the paths: trigger (images/k8s-utils/** vs images/cf-ssh-bastion/**), the file: path (images/k8s-utils/Dockerfile), and the leading --- document marker (the new workflow has one; the bastion sibling omits it — harmless, yamllint allows both). No structural divergence in jobs/steps. (This is a sanity diff, not an exact match.)

Note on the sed: normalize the reference identifiers onto the new workflow’s — cf-ssh-bastion → k8s-utils (IMAGE_NAME suffix) and SSH Bastion → k8s-utils (the name:). Do NOT use s/bastion/k8s-utils/g first: it would rewrite cf-ssh-bastion to cf-ssh-k8s-utils and mangle the comparison.

  • Step 4: Commit
Terminal window
jj commit -m "ci(k8s-utils): multi-arch build workflow for k8s-utils image (hl-ozs)


Mirrors build-bastion-image.yaml: amd64 + arm64 native builds on runs-on
runners, stitched into a multi-arch manifest pushed to ghcr.io as :<short-sha>
and :latest. Triggers on images/k8s-utils/** and images/certs/**.


Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"

Task 3: Repoint and harden the snapshot CronJob

Section titled “Task 3: Repoint and harden the snapshot CronJob”

Files:

  • Modify: argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml

Tag note: This task uses the placeholder tag REPLACE_WITH_SHA. The real short-SHA does not exist until the Task 2 workflow runs on main. See Task 4 for how the real tag is filled in. Do NOT invent a SHA here.

  • Step 1: Replace the container image and securityContext

In argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml, replace this block (the - name: snapshot container’s image, comment, and securityContext — currently lines ~28-36):

            - name: snapshot
              image: alpine:3.21
              # Runs as root (uid 0) because `apk add` needs write access to
              # install curl+jq; the job is short-lived with all caps dropped
              # and no privilege escalation. Root rootfs stays writable for apk.
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"]

with:

            - name: snapshot
              image: ghcr.io/fzymgc-house/k8s-utils:REPLACE_WITH_SHA
              # curl+jq are baked into the image (no runtime apk), so the job
              # needs no writable rootfs and no internet egress — it only POSTs
              # to the in-cluster Qdrant REST API.
              securityContext:
                runAsNonRoot: true
                runAsUser: 65532
                readOnlyRootFilesystem: true
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"]
  • Step 2: Remove the runtime apk add line from the script args

In the same file, in the args: heredoc block, delete this line (currently ~line 48):

                  apk add --no-cache --quiet curl jq

Leave the surrounding set -eu, QDRANT=..., COLL=..., KEEP=... lines and everything after intact. After the edit, the script begins:

              args:
                - |
                  set -eu
                  QDRANT="http://qdrant.agent-memory.svc.cluster.local:6333"
                  COLL="memory"
                  KEEP=14
                  echo "Creating snapshot of collection ${COLL} ..."
  • Step 3: Validate the manifest renders (kustomize build)

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/agent-memory | grep -A2 'name: snapshot' Expected: shows image: ghcr.io/fzymgc-house/k8s-utils:REPLACE_WITH_SHA and no apk add anywhere in the output: kubectl --context fzymgc-house kustomize argocd/app-configs/agent-memory | grep -c 'apk add' → 0

  • Step 4: Lint the YAML

Run: yamllint argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml Expected: No errors.

  • Step 5: Commit
Terminal window
jj commit -m "fix(agent-memory): repoint qdrant snapshot job at k8s-utils image (hl-ozs)


The CronJob failed every run doing 'apk add curl jq' at runtime — agent-memory
pods have no egress to the Alpine CDN. Use the baked-in ghcr.io/fzymgc-house/
k8s-utils image instead, drop the apk line, and harden the container to
non-root + readOnlyRootFilesystem (the apk root/writable-rootfs hack is gone).


Tag is a placeholder until the k8s-utils build publishes on main.


Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"

Task 4: Publish image, pin real tag, make package public

Section titled “Task 4: Publish image, pin real tag, make package public”

This task spans CI and a manual GHCR setting; its sub-steps are verification/operational, not code edits. It cannot complete until Tasks 1-2 have merged to main (or run via workflow_dispatch).

  • Step 1: Get the build workflow onto main

Open a PR with Tasks 1-3 (the Dockerfile, the workflow, and the CronJob with the REPLACE_WITH_SHA placeholder). The Dockerfile + workflow can merge first; the CronJob’s placeholder tag is corrected in Step 4 below. Prefer a single PR — the CronJob won’t sync to a real image until ArgoCD picks up the corrected tag anyway.

  • Step 2: Confirm the image published (multi-arch)

After the workflow runs on main, run: docker manifest inspect ghcr.io/fzymgc-house/k8s-utils:latest | jq -r '.manifests[].platform | "\(.os)/\(.architecture)"' Expected: lists both linux/amd64 and linux/arm64.

Capture the published short-SHA tag from the workflow’s create-manifest log line (Published image: ghcr.io/fzymgc-house/k8s-utils:<sha>).

  • Step 3: Make the GHCR package public

In github.com/orgs/fzymgc-house/packages, open the k8s-utils package → Package settings → Change visibility → Public. (New GHCR packages default to private; the CronJob has no imagePullSecrets, so it must be public to pull.)

Verify anonymously: docker pull ghcr.io/fzymgc-house/k8s-utils:<sha> from a logged-out context, or curl -fsSI https://ghcr.io/v2/fzymgc-house/k8s-utils/manifests/<sha> returns without auth challenge.

  • Step 4: Pin the real SHA in the CronJob manifest

Replace REPLACE_WITH_SHA in argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml with the real short-SHA from Step 2:

Run (substitute the real value): sed -i '' 's/k8s-utils:REPLACE_WITH_SHA/k8s-utils:<real-sha>/' argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml

Verify no placeholder remains: grep -c REPLACE_WITH_SHA argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml → 0

  • Step 5: Commit the tag pin
Terminal window
jj commit -m "fix(agent-memory): pin qdrant snapshot job to k8s-utils:<sha> (hl-ozs)


Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"

(If Tasks 1-3 and 4 are in the same PR, fold this into Task 3’s commit instead — replace the placeholder before opening the PR, once the image is published via workflow_dispatch on the feature branch. Either ordering is valid; the invariant is: real tag before ArgoCD syncs.)

Task 5: Verify in-cluster and clean up stale pods

Section titled “Task 5: Verify in-cluster and clean up stale pods”

Verification only — runs after ArgoCD syncs the merged CronJob. These commands are run by the user (the assistant has read-only Kubernetes access).

  • Step 1: Confirm ArgoCD synced the CronJob to the new image

Run: kubectl --context fzymgc-house get cronjob -n agent-memory qdrant-memory-snapshot -o jsonpath='{.spec.jobTemplate.spec.template.spec.containers[0].image}{"\n"}' Expected: ghcr.io/fzymgc-house/k8s-utils:<sha> (not alpine:3.21).

  • Step 2: Trigger a manual test run

Run: kubectl --context fzymgc-house create job -n agent-memory --from=cronjob/qdrant-memory-snapshot k8s-utils-smoke

  • Step 3: Confirm the job completes

Run: kubectl --context fzymgc-house wait -n agent-memory --for=condition=complete job/k8s-utils-smoke --timeout=120s Expected: job.batch/k8s-utils-smoke condition met.

If it fails: check kubectl --context fzymgc-house logs -n agent-memory job/k8s-utils-smoke.

  • ImagePullBackOff → the GHCR package is not public (revisit Task 4 Step 3).
  • HTTP errors from curl → Qdrant connectivity / collection-name issue, not the image.
  • Step 4: Confirm a snapshot was created

Run the snapshot list from inside the cluster (the job already does this), or check Qdrant: the job’s log prints .result.name of the new snapshot and a “Snapshot cycle complete.” line. Confirm the log shows both.

  • Step 5: Delete the smoke-test job and the stale Error pods (one-time)
Terminal window
# remove the smoke test
kubectl --context fzymgc-house delete job -n agent-memory k8s-utils-smoke


# remove the accumulated failed snapshot jobs (this also removes their Error
# pods). Match by NAME PREFIX, not by the generic batch.kubernetes.io/job-name
# label — that label is on every CronJob-spawned job, so a label+field-selector
# delete would also catch unrelated failed jobs if any existed in the namespace.
kubectl --context fzymgc-house get jobs -n agent-memory -o name \
  | grep '/qdrant-memory-snapshot-' \
  | xargs -r kubectl --context fzymgc-house delete -n agent-memory

Expected: kubectl --context fzymgc-house get pods -n agent-memory | grep -c Error → 0.

  • Step 6: Close the bead

Run: bd close hl-ozs --reason="k8s-utils image built + published public; snapshot CronJob repointed and verified Completed; stale Error pods cleaned"

Rollback

Section titled “Rollback”

If the new image misbehaves, revert the CronJob commit (re-point to alpine:3.21 and restore the apk add line). The image and workflow are additive — leaving them in place is harmless. Note that reverting to alpine:3.21 reintroduces the original egress bug, so rollback is only a stopgap.