Headroom compression service
Headroom is an HTTP proxy that compresses LLM request context (JSON tool outputs, message history)
before it forwards the request to the real upstream. One instance runs in the cluster,
headroom-agents. It serves Claude Code on the operator laptop, which opts in with
ANTHROPIC_BASE_URL. In-cluster applications call LiteLLM directly and do not use Headroom
(docs/adr/2026-09-23-headroom-apps-retired.md).
What exists
Section titled “What exists”| Instance | Namespace | Run mode | Surface | Upstream |
|---|---|---|---|---|
headroom-agents |
headroom-agents |
--mode cache |
Anthropic /v1/messages |
https://api.anthropic.com |
The manifests are in argocd/app-configs/headroom-agents/. Cache mode freezes earlier turns in
place, so Anthropic prefix-cache hits survive. A mode that compresses earlier turns again on each
request breaks the cache. The instance also runs --no-cache, which turns off the separate
semantic response cache that can return a stale completion for a near-duplicate prompt.
laptop Claude Code ──ANTHROPIC_BASE_URL──▶ headroom-agents ─────▶ api.anthropic.com (subscription OAuth) headroom.fzymgc.house cache mode (direct; bypasses LiteLLM) (LAN and Tailscale) headroom-otel image PVC · svc.name=headroom-agents
headroom-agents ──OTLP/HTTP :4318──▶ cs-otel-collector.clickstack (ClickStack)Claude Code sends its subscription OAuth bearer in the Authorization header. Headroom compresses
in cache mode, then forwards to api.anthropic.com with the bearer and the anthropic-* headers
unchanged. This path never touches LiteLLM or OpenRouter.
The deployment runs one headroom proxy process with replicas: 1. The --workers flag is not
set. Every worker shares one RWO PVC and writes the same proxy_savings.json and memory.db, so
a second worker causes diverging savings numbers or SQLITE_BUSY errors.
Use it from the laptop
Section titled “Use it from the laptop”export ANTHROPIC_BASE_URL=https://headroom.fzymgc.houseclaudeheadroom.fzymgc.house is reachable over the LAN and Tailscale. It is not on public DNS. To
bypass Headroom, unset the variable. Nothing changes on the Headroom side.
unset ANTHROPIC_BASE_URLObservability
Section titled “Observability”The instance pushes metrics through OTLP/HTTP to ClickStack:
| Setting | Value |
|---|---|
| Endpoint | http://cs-otel-collector.clickstack.svc.cluster.local:4318/v1/metrics |
| Auth header | HEADROOM_OTEL_METRICS_HEADERS from the headroom-otlp-headers Secret, as Authorization=<raw token> with no Bearer word |
| Service name | headroom-agents (HEADROOM_OTEL_SERVICE_NAME) |
| Resource attributes | service.namespace=headroom plus the pod and node names (HEADROOM_OTEL_RESOURCE_ATTRIBUTES) |
| Upstream telemetry | HEADROOM_TELEMETRY=off |
In HyperDX, filter by service.name = headroom-agents. The prefix-cache signal is
headroom_cache_write_ttl_tokens_total{provider=anthropic,ttl=<bucket>}, the tokens that landed
in each TTL bucket. The pod’s /stats endpoint carries prefix_cache.observed_ttl_buckets per
request.
An Uptime Kuma monitor reads /readyz on port 8787 (tf/uptime-kuma/monitors-services.tf).
Upgrade the image
Section titled “Upgrade the image”The deployment runs ghcr.io/fzymgc-house/headroom-otel, an image derived from the official
code-nonroot base that adds the OpenTelemetry SDK, which the official images omit
(docs/adr/hl-8ycr-use-official-ghcr-headroom-images-do-not-build-cluster-owned.md). An upgrade
takes two hops, and Renovate automerges both:
- Renovate’s
dockerfilemanager bumps theFROM ...@sha256:...pin inimages/headroom-otel/Dockerfileon a new upstream release. The base isghcr.io/headroomlabs-ai/headroom. If Renovate goes quiet for weeks, compare theFROMpath with the image location in the upstream release notes. - The merge runs
.github/workflows/build-headroom-otel.yml, which publishes a new multi-archghcr.io/fzymgc-house/headroom-otel:<version>. - Renovate’s
argocdmanager then bumps the digest of theimage:reference inargocd/app-configs/headroom-agents/deployment.yaml.
To rebuild by hand, for example after a Dockerfile edit, merge to main or run
gh workflow run build-headroom-otel.yml. After the build publishes, resolve the new digest:
crane digest ghcr.io/fzymgc-house/headroom-otel:<version>At each bump, make sure that Headroom still installs to system site-packages and that the OpenTelemetry api and sdk pin still resolves. That is the standing risk the ADR records.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Fix |
|---|---|---|
/readyz returns 503 |
The pod is still starting | Wait. The readiness probe starts after initialDelaySeconds: 10. |
| The OTLP push is rejected with 401 | An extra Bearer word in the header value |
HEADROOM_OTEL_METRICS_HEADERS must be Authorization=<raw token>. |
| SQLite errors or a diverging savings ledger | Two workers on one RWO PVC | Keep replicas: 1 and no --workers flag. |
| The Uptime Kuma monitor times out while the pod is Ready | The CiliumNetworkPolicy has no ingress rule for the uptime-kuma namespace on 8787 |
Make sure that argocd/app-configs/headroom-agents/networkpolicy.yaml allows uptime-kuma on 8787. |
Permission denied: '/.cache' from Hugging Face in the log |
The pod runs as UID 65532 with no home directory, so HOME is / |
Make sure that the deployment sets HF_HOME to a path on the PVC. |
headroom.fzymgc.house is unreachable from the laptop |
The laptop is not on the LAN or Tailscale | Connect to Tailscale or the LAN. The host is not on public DNS. |