Headroom OTel-SDK derived image — design

Bead: hl-v9xz.11 (epic hl-v9xz — Headroom central LLM compression) Date: 2026-06-24 Status: Design — pending design-review gate

Problem

Both Headroom deployments (headroom-apps, headroom-agents) are fully wired for native OTLP/HTTP metrics push to ClickStack — HEADROOM_OTEL_METRICS_ENABLED=1 plus the complete HEADROOM_OTEL_METRICS_* env (exporter otlp_http, endpoint http://cs-otel-collector.clickstack.svc.cluster.local:4318/v1/metrics, headers, service name). Yet no Headroom metrics reach ClickStack: the export is a silent no-op.

This blocks hl-bdok (central-compression validation needs Headroom metrics in ClickStack to measure real-traffic compression effectiveness).

Root cause (verified 2026-06-24)

The official ghcr.io/chopratejas/headroom images ship the OpenTelemetry API but not the SDK. They are built as headroom-ai[proxy], not headroom-ai[proxy,otel]. In-pod evidence:

• python3 -c 'import opentelemetry.sdk' → ModuleNotFoundError.
• get_otel_metrics_status() → {configured: False, enabled: False, exporter: None} despite the full env.

Headroom’s metrics setup finds no SDK and silently does nothing — no error logged. Two non-causes were ruled out and are red herrings:

• HEADROOM_TELEMETRY controls only the anonymous upstream usage beacon; it is independent of OTLP export. off is correct.
• The CiliumNetworkPolicy already permits egress to clickstack:4318.

No official image carries the OTel SDK

A read-only GHCR probe (manifest HEAD, the reliable check — tags/list paginates at 100 and reports false-missing, confirmed when 0.27.0-nonroot was absent from the listing yet returned HTTP 200) established that the published variant matrix is {code, slim, nonroot} only — there is no otel axis. Every candidate (otel, 0.27.0-otel, 0.27.0-nonroot-otel, 0.27.0-code-nonroot-otel, 0.6.7-otel, …) returned HTTP 404.

Therefore the only way to get the SDK into the running container is to build a derived image. That reverses ADR hl-8ycr (use official images; do not build cluster-owned), so this design includes a scoped amendment to that ADR.

Decision

Build one thin derived image — ghcr.io/fzymgc-house/headroom-otel — that adds the OpenTelemetry SDK and OTLP/HTTP exporter on top of the official code-nonroot base, and repoint both deployments at it.

Why code-nonroot for both instances

The two instances run different official bases today: headroom-apps on 0.27.0-nonroot, headroom-agents on 0.27.0-code-nonroot. Per ADR hl-8ycr’s own rationale, code-nonroot is a superset of nonroot — the same debian-slim base (so it keeps the shell headroom-apps uses for /stats tuning) plus tree-sitter. A single derived image from code-nonroot therefore serves both, collapsing what would otherwise be a per-base build matrix into one image.

Trade-off accepted: the apps pod carries tree-sitter it does not use (a few MB, no functional or runtime cost). This is the deliberate price of one image instead of two.

Approaches considered

• A — Thin derived image (chosen). FROM official + install the two OTel SDK packages. Matches the existing images/openclaw-custom and images/k8s-utils precedent; immutable digest; build runs on CI, not the cluster. Cost: reverses ADR hl-8ycr → scoped amendment.
• B — Runtime SDK injection via initContainer. Install the SDK into a shared PYTHONPATH volume at pod start. Avoids building an image (no ADR reversal) but requires PyPI egress at startup, a CiliumNetworkPolicy change (only clickstack:4318 egress is open today), and makes pod startup network-fragile. Rejected.
• C — Native Prometheus scrape. Headroom exposes a native /metrics endpoint; scrape it like the agentgateway metrics lane. Fully honors hl-8ycr with zero build, but abandons the user-directed OTLP/OTel path. Rejected.

Design

Component 1 — images/headroom-otel/Dockerfile

A single Dockerfile following the repo’s thin-derived-image convention:

# SPDX-License-Identifier: MIT
# Thin derived Headroom image: adds the OpenTelemetry SDK + OTLP/HTTP exporter
# that the official ghcr.io/chopratejas/headroom images omit (built [proxy], not
# [proxy,otel]), so native HEADROOM_OTEL_METRICS_* export to ClickStack works.
# Base is code-nonroot (superset of nonroot) so one image serves both the apps
# and agents deployments. See docs/adr/hl-8ycr (amended) and bead hl-v9xz.11.
FROM ghcr.io/chopratejas/headroom:0.27.0-code-nonroot@sha256:<pin-at-build>

USER root
# Install ONLY what headroom-ai[otel] adds, pinned to the opentelemetry-api
# version already bundled by [proxy] — guarantees api/sdk/exporter lockstep and
# avoids reinstalling headroom-ai itself (no version drift).
RUN API_VER="$(python -c 'import importlib.metadata as m; print(m.version("opentelemetry-api"))')" && \
    pip install --no-cache-dir \
      "opentelemetry-sdk==${API_VER}" \
      "opentelemetry-exporter-otlp-proto-http==${API_VER}"
USER 65532

Grounding facts (deepwiki + context7) that make this correct:

• Headroom is installed into system site-packages (/usr/local/lib/python3.11/site-packages, on PYTHONPATH); base is python:3.11-slim; pip is present. So packages installed here are importable by Headroom — no virtualenv mismatch.
• The [otel] extra is exactly opentelemetry-sdk>=1.24.0 + opentelemetry-exporter-otlp-proto-http>=1.24.0; opentelemetry-api>=1.24.0 is already a core dep. The OTLP HTTP metrics exporter (OTLPMetricExporter) lives in opentelemetry.exporter.otlp.proto.http.

Component 2 — .github/workflows/build-headroom-otel.yml

A near-verbatim copy of build-openclaw-custom.yml (the established multi-arch lane): a build-amd64 job and a build-arm64 job on the namespace.so namespace-profile-linux-{amd64,arm64}-2x4 runners, then a create-manifest job stitching a multi-arch manifest. Image name ghcr.io/fzymgc-house/headroom-otel; tag derived from the base version (0.27.0). Triggers on push to main touching images/headroom-otel/** or the workflow file, plus workflow_dispatch.

Multi-arch is retained (not arm64-only) to reuse the proven workflow pattern unchanged, even though the cluster is currently all-arm64.

Component 3 — deployment repoint (the only manifest change)

Both deployments already carry the complete OTLP env. The sole change is the image reference:

• argocd/app-configs/headroom-apps/deployment.yaml
• argocd/app-configs/headroom-agents/deployment.yaml

each set to:

image: ghcr.io/fzymgc-house/headroom-otel:0.27.0@sha256:<derived-digest>

--mode token (apps) and --mode cache (agents) and all other args/env stay exactly as-is.

Component 4 — image pinning and Renovate

• The deployment pins tag@sha256:<digest> to preserve the digest immutability ADR hl-8ycr values.
• The Dockerfile FROM is digest-pinned; Renovate’s dockerfile manager bumps it on upstream Headroom releases.
• Flow is two-hop, both automergeable: a dockerfile-manager PR bumps FROM → the build workflow publishes a new derived digest → an argocd-manager PR digest-bumps the two deployments. Documented in docs/operations/headroom.md.

ADR impact

ADR hl-8ycr (“use official images; do not build cluster-owned”) is reversed in a narrow, capability-driven way. Because hl-8ycr is bd-rendered (adr-render: source=bd:hl-8ycr), the amendment is captured via the beads ADR mechanism (/adr / evolve-adr) during the plan phase — not a hand-edit of the rendered file. The amendment scopes the rule to:

Prefer official images. Build a thin derived image only when the official image matrix lacks a required capability extra that cannot be supplied via configuration — here, the OpenTelemetry SDK, absent from every published [proxy] build.

The derived image stays minimal (no Headroom fork; only well-known OTel packages added) to preserve most of hl-8ycr’s supply-chain rationale. hl-8ycr gains a “partially amended by ⟨new ADR⟩” status.

Verification (acceptance criteria)

• Build / pre-publish: an in-image import check passes — python -c "import opentelemetry.sdk, opentelemetry.exporter.otlp.proto.http.metric_exporter" exits 0.
• Post-deploy (in-pod), both instances: get_otel_metrics_status() returns configured == True and enabled == True.
• Post-deploy (ClickStack): headroom-apps and headroom-agents appear in default.otel_metrics_sum (ClickHouse query).
• Unblocks: hl-bdok can then measure real-traffic compression in ClickStack.

Rollback

Revert the two deployment image: references to the official 0.27.0-{nonroot,code-nonroot} tags. Metrics export returns to its current inert state; Headroom proxying is otherwise unaffected.

Risks and consequences

• A build pipeline now exists for Headroom — the thing hl-8ycr avoided. Mitigated: it mirrors the existing openclaw-custom / k8s-utils lanes; the Dockerfile is ~6 lines; no Headroom source is forked.
• Two-hop Renovate flow adds one extra automerged PR per upstream bump. Documented in the ops runbook.
• Supply chain: the image now injects opentelemetry-sdk and opentelemetry-exporter-otlp-proto-http at build. Pinning to the bundled opentelemetry-api version avoids api/sdk skew; both are first-party OTel packages.
• Re-verify at each upstream bump (the standing hl-8ycr risk): confirm Headroom still installs to system site-packages and that the api/sdk pin still resolves.

Out of scope (YAGNI)

• arm64-only build (multi-arch retained to reuse the existing workflow).
• The [ml] / Kompress extra and any non-OTel Headroom extras.
• Changes to the OTLP env, headers secret, NetworkPolicy, or PVCs — all already correct.