Skip to content
fzymgc-house Cluster Docs

Headroom OTel-SDK derived image — implementation plan

Headroom OTel-SDK derived image Implementation Plan

Section titled “Headroom OTel-SDK derived image Implementation Plan”

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make Headroom’s native OTLP/HTTP metrics export to ClickStack actually work by building one thin derived image that adds the OpenTelemetry SDK the official images omit, and repointing both deployments at it.

Architecture: A single derived image ghcr.io/fzymgc-house/headroom-otel:0.27.0 is built FROM the official code-nonroot base (a superset that serves both the apps and agents instances) plus a pip install of opentelemetry-sdk + opentelemetry-exporter-otlp-proto-http, pinned to the OTel API version the base already bundles. The build reuses the repo’s existing multi-arch image lane. Because a self-built image’s digest only exists after its Dockerfile lands on main, the work is two-phase: publish the image first, then digest-pin the deployments.

Tech Stack: Docker (thin derived image), GitHub Actions (namespace.so runners, multi-arch manifest), ArgoCD/Kustomize manifests, Renovate, beads-rendered ADRs.

Design spec: docs/engineering/specs/2026-06-24-headroom-otel-derived-image-design.md Design bead: hl-v9xz.11 (epic hl-v9xz)

Files touched

Section titled “Files touched”
  • Create: images/headroom-otel/Dockerfile — the thin derived image.
  • Create: .github/workflows/build-headroom-otel.yml — multi-arch build+publish.
  • Modify: .github/renovate.json — explicit dockerfile manager + ghcr automerge for it.
  • Modify: docs/adr/hl-8ycr-use-official-ghcr-headroom-images-do-not-build-cluster-owned.md (re-rendered from bd) — scoped amendment.
  • Modify: docs/operations/headroom.md — derived-image upgrade runbook section.
  • Modify: argocd/app-configs/headroom-apps/deployment.yaml:29 — image ref.
  • Modify: argocd/app-configs/headroom-agents/deployment.yaml:29 — image ref.

Phasing: Tasks 1–5 form the first PR (image + workflow + config + docs/ADR). Task 6 is the merge/build checkpoint. Task 7 is the second PR (deployment repoint with the now-known digest).

Task 1: Create the derived Dockerfile

Section titled “Task 1: Create the derived Dockerfile”

Files:

  • Create: images/headroom-otel/Dockerfile

  • Step 1: Write the Dockerfile

Create images/headroom-otel/Dockerfile with exactly:

# SPDX-License-Identifier: MIT
# Thin derived Headroom image: adds the OpenTelemetry SDK + OTLP/HTTP exporter
# that the official ghcr.io/chopratejas/headroom images omit (built [proxy], not
# [proxy,otel]), so native HEADROOM_OTEL_METRICS_* export to ClickStack works.
# Base is code-nonroot (superset of nonroot) so one image serves both the apps
# and agents deployments. See docs/adr/hl-8ycr (amended) and bead hl-v9xz.11.
FROM ghcr.io/chopratejas/headroom:0.27.0-code-nonroot@sha256:a642ad983f35651272a5817f46b287132bad871cbbf0227bc65c8c825ab494e0


USER root
# Install ONLY what headroom-ai[otel] adds, pinned to the opentelemetry-api
# version already bundled by [proxy] -> api/sdk/exporter lockstep, no headroom
# reinstall, no version drift. headroom lives in system site-packages so these
# are importable by it.
RUN API_VER="$(python -c 'import importlib.metadata as m; print(m.version("opentelemetry-api"))')" && \
    pip install --no-cache-dir \
      "opentelemetry-sdk==${API_VER}" \
      "opentelemetry-exporter-otlp-proto-http==${API_VER}"
USER 65532
  • Step 2: Build the image locally to verify it builds

Run (Sean’s Mac is arm64, matching the cluster, so a native build works):

Terminal window
docker build -t headroom-otel:verify images/headroom-otel

Expected: build succeeds; the pip install resolves opentelemetry-sdk and opentelemetry-exporter-otlp-proto-http at the bundled API version.

  • Step 3: Run the in-image import check (the acceptance gate)
Terminal window
docker run --rm --entrypoint python headroom-otel:verify \
  -c "import opentelemetry.sdk, opentelemetry.exporter.otlp.proto.http.metric_exporter; print('otel-sdk-ok')"

Expected output: otel-sdk-ok (proves the SDK + OTLP HTTP exporter are importable in the same environment as Headroom).

  • Step 4: Commit
Terminal window
jj describe -m "feat(headroom): derived headroom-otel image with OTel SDK [hl-v9xz.11]"

(Working copy already on bookmark feat/headroom-otel-export; this folds the Dockerfile into the change. Subsequent tasks append commits with jj commit/jj new as appropriate.)

Task 2: Create the multi-arch build workflow

Section titled “Task 2: Create the multi-arch build workflow”

Files:

  • Create: .github/workflows/build-headroom-otel.yml

  • Step 1: Write the workflow

Create .github/workflows/build-headroom-otel.yml (a near-verbatim copy of build-openclaw-custom.yml, adapted: image name headroom-otel, context images/headroom-otel, and the published tag is the numeric base version 0.27.0, not the full base tag):

# SPDX-License-Identifier: MIT
# yaml-language-server: $schema=https://json.schemastore.org/github-workflow
---
name: Build Headroom OTel Image


on:
  push:
    branches:
      - main
    paths:
      - 'images/headroom-otel/**'
      - '.github/workflows/build-headroom-otel.yml'
    tags:
      - 'headroom-otel-*'
  workflow_dispatch:


env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository_owner }}/headroom-otel


permissions:
  contents: read
  packages: write


jobs:
  build-amd64:
    runs-on: namespace-profile-linux-amd64-2x4
    outputs:
      image_tag: ${{ steps.vars.outputs.image_tag }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3


      - name: Set image tag from base image version
        id: vars
        run: |
          # Base FROM is ...headroom:0.27.0-code-nonroot@sha256:...; publish the
          # numeric version only (one derived image serves both instances).
          IMAGE_TAG=$(grep '^FROM' images/headroom-otel/Dockerfile \
            | sed -E 's/.*headroom:([0-9]+\.[0-9]+\.[0-9]+).*/\1/')
          echo "image_tag=${IMAGE_TAG}" >> "$GITHUB_OUTPUT"
          echo "Derived image tag: ${IMAGE_TAG}"


      - name: Log in to GHCR
        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}


      - name: Build and push amd64 image
        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
        with:
          context: images/headroom-otel
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.vars.outputs.image_tag }}-amd64


  build-arm64:
    runs-on: namespace-profile-linux-arm64-2x4
    steps:
      - name: Checkout repository
        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3


      - name: Set image tag from base image version
        id: vars
        run: |
          IMAGE_TAG=$(grep '^FROM' images/headroom-otel/Dockerfile \
            | sed -E 's/.*headroom:([0-9]+\.[0-9]+\.[0-9]+).*/\1/')
          echo "image_tag=${IMAGE_TAG}" >> "$GITHUB_OUTPUT"
          echo "Derived image tag: ${IMAGE_TAG}"


      - name: Log in to GHCR
        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}


      - name: Build and push arm64 image
        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
        with:
          context: images/headroom-otel
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.vars.outputs.image_tag }}-arm64


  create-manifest:
    needs: [build-amd64, build-arm64]
    runs-on: namespace-profile-linux-amd64-2x4
    env:
      IMAGE_TAG: ${{ needs.build-amd64.outputs.image_tag }}
    steps:
      - name: Log in to GHCR
        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}


      - name: Create and push multi-arch manifest
        run: |
          set -euxo pipefail
          docker manifest create "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}" \
            "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}-amd64" \
            "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}-arm64"
          docker manifest push "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}"


          docker manifest create "${REGISTRY}/${IMAGE_NAME}:latest" \
            "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}-amd64" \
            "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}-arm64"
          docker manifest push "${REGISTRY}/${IMAGE_NAME}:latest"


          echo "Published image: ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}"
  • Step 2: Lint the workflow
Terminal window
yamllint .github/workflows/build-headroom-otel.yml

Expected: no errors (matches the .yamllint.yaml config the other workflows pass under).

  • Step 3: Diff against the precedent to confirm only the intended changes
Terminal window
diff <(sed -E 's/openclaw-custom|openclaw/HEADROOM/g' .github/workflows/build-openclaw-custom.yml) \
     .github/workflows/build-headroom-otel.yml || true

Expected: differences limited to the image name, context path, trigger paths, workflow name, and the numeric-version tag-extraction (vs openclaw’s full base-version extraction).

  • Step 4: Commit
Terminal window
jj commit -m "ci(headroom): build-headroom-otel multi-arch workflow [hl-v9xz.11]"

Task 3: Track the derived image’s base FROM in Renovate

Section titled “Task 3: Track the derived image’s base FROM in Renovate”

Files:

  • Modify: .github/renovate.json

Renovate’s dockerfile manager is already enabled by the config:recommended preset and discovers images/headroom-otel/Dockerfile by default — so no explicit manager block is needed (an explicit managerFilePatterns would replace the manager’s defaults and narrow discovery). The only change required is to make the base-image FROM bump automerge like the repo’s other ghcr images.

  • Step 1: Add dockerfile to the ghcr automerge rule

In the "description": "Container images from GitHub Container Registry" packageRule, add "dockerfile" to matchManagers so the FROM ghcr.io/chopratejas/headroom@sha256:... bump is grouped + automerged like the other ghcr images:

      "matchManagers": [
        "kubernetes",
        "argocd",
        "kustomize",
        "dockerfile"
      ],
  • Step 2: Validate JSON
Terminal window
python -c "import json; json.load(open('.github/renovate.json')); print('renovate.json valid')"

Expected: renovate.json valid.

  • Step 3: Commit
Terminal window
jj commit -m "chore(renovate): automerge images/**/Dockerfile FROM bumps [hl-v9xz.11]"

Task 4: Amend ADR hl-8ycr (scoped derived-image exception)

Section titled “Task 4: Amend ADR hl-8ycr (scoped derived-image exception)”

Files:

  • Modify: docs/adr/hl-8ycr-use-official-ghcr-headroom-images-do-not-build-cluster-owned.md (re-rendered from bd — do NOT hand-edit the rendered file)

  • Step 1: Amend the bd-rendered ADR

Use the dev-flow:evolve-adr skill on hl-8ycr. The amendment adds a scoped exception to the decision (do not rewrite history — append an amendment):

Amended 2026-06-24 (hl-v9xz.11): No published official image carries the OpenTelemetry SDK (every variant is built [proxy], not [proxy,otel]; confirmed via manifest HEAD). A thin derived image ghcr.io/fzymgc-house/headroom-otel is therefore built to add only the OTel SDK + OTLP/HTTP exporter on top of the official code-nonroot base. This is a scoped exception, not a reversal: build a derived image only when the official matrix lacks a required capability extra unsupplied by config, and keep it minimal (no Headroom fork) so the upstream supply-chain trust is preserved.

  • Step 2: Re-render the ADR markdown

The dev-flow:evolve-adr skill (Step 1) re-renders docs/adr/hl-8ycr-*.md from the bd decision as part of its own flow — there is no standalone render-adr binary. If the render must be triggered separately, invoke the /adr update hl-8ycr skill command (per the file’s adr-render header), run from the worktree root so docs/adr/ resolves correctly.

  • Step 3: Lint + verify frontmatter (Starlight + rumdl gates)
Terminal window
rumdl check docs/adr/hl-8ycr-use-official-ghcr-headroom-images-do-not-build-cluster-owned.md
rg -n '^title:' docs/adr/hl-8ycr-use-official-ghcr-headroom-images-do-not-build-cluster-owned.md

Expected: rumdl “No issues found”; a title: frontmatter line present (the ADR renderer emits no frontmatter — add ---\ntitle: "..."\n--- and demote the body H1 to H2 if missing, per the docs-build requirement).

  • Step 4: Commit
Terminal window
jj commit -m "docs(adr): amend hl-8ycr — scoped derived-image exception [hl-v9xz.11]"

Task 5: Add the derived-image upgrade runbook section

Section titled “Task 5: Add the derived-image upgrade runbook section”

Files:

  • Modify: docs/operations/headroom.md (add a section after ## Observability)

  • Step 1: Add the upgrade section

Insert a new section after ## Observability (before ## Rollback):

## Upgrading the headroom image (derived headroom-otel)


Both deployments run `ghcr.io/fzymgc-house/headroom-otel`, a thin image derived
from the official `code-nonroot` base that adds the OpenTelemetry SDK (the
official images omit it — see ADR hl-8ycr, amended). Upgrades are two-hop, both
Renovate-automerged:


1. Renovate's `dockerfile` manager bumps the `FROM ...@sha256:...` pin in
   `images/headroom-otel/Dockerfile` on a new upstream Headroom release.
2. Merging that PR triggers `.github/workflows/build-headroom-otel.yml`, which
   publishes a new multi-arch `ghcr.io/fzymgc-house/headroom-otel:<version>`.
3. Renovate's `argocd` manager then digest-bumps the `image:` refs in
   `argocd/app-configs/headroom-{apps,agents}/deployment.yaml`.


To rebuild manually (e.g. after editing the Dockerfile): merge to `main`, or run
`gh workflow run build-headroom-otel.yml`. After publish, resolve the new digest
with:


\`\`\`bash
crane digest ghcr.io/fzymgc-house/headroom-otel:0.27.0
# or, without crane:
curl -sI -H "Authorization: Bearer $(curl -s \
  'https://ghcr.io/token?scope=repository:fzymgc-house/headroom-otel:pull' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')" \
  -H 'Accept: application/vnd.oci.image.index.v1+json' \
  https://ghcr.io/v2/fzymgc-house/headroom-otel/manifests/0.27.0 \
  | tr -d '\r' | grep -i docker-content-digest
\`\`\`


Re-verify at each bump (standing ADR hl-8ycr risk): confirm Headroom still
installs to system site-packages and that the api/sdk pin still resolves.

(Replace the escaped fences with real triple-backticks when writing the file.)

  • Step 2: Lint
Terminal window
rumdl fmt docs/operations/headroom.md && rumdl check docs/operations/headroom.md

Expected: “No issues found”.

  • Step 3: Commit
Terminal window
jj commit -m "docs(headroom): derived-image upgrade runbook [hl-v9xz.11]"

Task 6: Merge phase 1, build, and capture the derived digest (checkpoint)

Section titled “Task 6: Merge phase 1, build, and capture the derived digest (checkpoint)”

This task is the build-then-pin boundary. The deployment digest in Task 7 cannot be known until the image is published from main.

  • Step 1: Push the branch and open the phase-1 PR

From the default workspace (non-default jj workspaces have no .git, so gh must run there):

Terminal window
jj git push -b feat/headroom-otel-export
cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster
gh pr create --head feat/headroom-otel-export --base main \
  --title "feat(headroom): derived headroom-otel image with OTel SDK [hl-v9xz.11]" \
  --body "Spec + plan + derived image + build workflow + ADR amendment + runbook. Deployment repoint follows in a second PR once the image is published. Closes path to hl-bdok."
  • Step 2: Watch the PR checks, then merge
Terminal window
gh pr checks feat/headroom-otel-export --repo fzymgc-house/selfhosted-cluster --watch

Expected: docs build, rumdl, yamllint, cocogitto all green. Merge when green.

  • Step 3: Confirm the build workflow ran on main and published the image
Terminal window
gh run list --workflow build-headroom-otel.yml --repo fzymgc-house/selfhosted-cluster --limit 3

Expected: a successful run after merge.

  • Step 4: Resolve the published multi-arch index digest
Terminal window
TOKEN=$(curl -s 'https://ghcr.io/token?scope=repository:fzymgc-house/headroom-otel:pull' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
curl -sI -H "Authorization: Bearer $TOKEN" \
  -H 'Accept: application/vnd.oci.image.index.v1+json' \
  https://ghcr.io/v2/fzymgc-house/headroom-otel/manifests/0.27.0 \
  | tr -d '\r' | grep -i docker-content-digest

Record the sha256:... value — it is the <DERIVED_DIGEST> used in Task 7.

Task 7: Repoint both deployments to the derived image

Section titled “Task 7: Repoint both deployments to the derived image”

Files:

  • Modify: argocd/app-configs/headroom-apps/deployment.yaml:29
  • Modify: argocd/app-configs/headroom-agents/deployment.yaml:29
  • Step 1: Update the apps image ref

In argocd/app-configs/headroom-apps/deployment.yaml, change line 29 from:

          image: ghcr.io/chopratejas/headroom:0.27.0-nonroot

to (substitute the <DERIVED_DIGEST> captured in Task 6):

          image: ghcr.io/fzymgc-house/headroom-otel:0.27.0@sha256:<DERIVED_DIGEST>
  • Step 2: Update the agents image ref

In argocd/app-configs/headroom-agents/deployment.yaml, change line 29 from:

          image: ghcr.io/chopratejas/headroom:0.27.0-code-nonroot

to the identical derived ref:

          image: ghcr.io/fzymgc-house/headroom-otel:0.27.0@sha256:<DERIVED_DIGEST>
  • Step 3: Validate both kustomizations render
Terminal window
kubectl kustomize argocd/app-configs/headroom-apps >/dev/null && echo apps-ok
kubectl kustomize argocd/app-configs/headroom-agents >/dev/null && echo agents-ok

Expected: apps-ok and agents-ok (no render errors; the new image ref parses).

  • Step 4: Commit, push, open the phase-2 PR, merge
Terminal window
jj commit -m "feat(headroom): repoint apps+agents to headroom-otel image [hl-v9xz.11]"
jj git push -b feat/headroom-otel-export
# from the default workspace:
gh pr create --head feat/headroom-otel-export --base main \
  --title "feat(headroom): repoint deployments to headroom-otel [hl-v9xz.11]" \
  --body "Repoints headroom-apps + headroom-agents to the published derived image. Enables OTLP metrics export to ClickStack."
  • Step 5: Post-deploy verification (after ArgoCD syncs)

In-pod, both instances (substitute pod names from kubectl get pods -n headroom-apps / -n headroom-agents):

Terminal window
kubectl exec -n headroom-apps deploy/headroom-apps -- \
  python -c "import opentelemetry.sdk; print('sdk-importable')"

Expected: sdk-importable. Then confirm Headroom reports configured export — query the proxy’s get_otel_metrics_status() (via /stats or the documented status path) and expect configured: True, enabled: True for both headroom-apps and headroom-agents.

  • Step 6: Confirm metrics land in ClickStack

Query ClickHouse (ClickStack MCP) after a few minutes of traffic:

SELECT ServiceName, count() AS n
FROM default.otel_metrics_sum
WHERE ServiceName IN ('headroom-apps', 'headroom-agents')
  AND TimeUnix > now() - INTERVAL 15 MINUTE
GROUP BY ServiceName;

Expected: both headroom-apps and headroom-agents present with n > 0. This is the bead’s acceptance criterion and unblocks hl-bdok.

Rollback

Section titled “Rollback”

Revert the two image: refs (Task 7) to the official 0.27.0-nonroot / 0.27.0-code-nonroot tags. Metrics export returns to its inert state; Headroom proxying is unaffected. The derived image and workflow can stay (unused).

Notes for the executor

Section titled “Notes for the executor”
  • Work continues on jj workspace headroom-otel-design / bookmark feat/headroom-otel-export. Run gh from the default workspace only.
  • Tasks 1–5 are independent edits; commit each separately. Task 6 is a hard checkpoint (image must publish before Task 7’s digest exists). Task 7 is a separate PR.
  • Do not skip the in-image import check (Task 1 Step 3) — it is the cheapest proof the whole change works before anything deploys.