Skip to content

Headroom compression service

Headroom is an HTTP proxy that compresses LLM request context (JSON tool outputs, message history) before it forwards the request to the real upstream. One instance runs in the cluster, headroom-agents. It serves Claude Code on the operator laptop, which opts in with ANTHROPIC_BASE_URL. In-cluster applications call LiteLLM directly and do not use Headroom (docs/adr/2026-09-23-headroom-apps-retired.md).

Instance Namespace Run mode Surface Upstream
headroom-agents headroom-agents --mode cache Anthropic /v1/messages https://api.anthropic.com

The manifests are in argocd/app-configs/headroom-agents/. Cache mode freezes earlier turns in place, so Anthropic prefix-cache hits survive. A mode that compresses earlier turns again on each request breaks the cache. The instance also runs --no-cache, which turns off the separate semantic response cache that can return a stale completion for a near-duplicate prompt.

laptop Claude Code ──ANTHROPIC_BASE_URL──▶ headroom-agents ─────▶ api.anthropic.com
(subscription OAuth) headroom.fzymgc.house cache mode (direct; bypasses LiteLLM)
(LAN and Tailscale) headroom-otel image
PVC · svc.name=headroom-agents
headroom-agents ──OTLP/HTTP :4318──▶ cs-otel-collector.clickstack (ClickStack)

Claude Code sends its subscription OAuth bearer in the Authorization header. Headroom compresses in cache mode, then forwards to api.anthropic.com with the bearer and the anthropic-* headers unchanged. This path never touches LiteLLM or OpenRouter.

The deployment runs one headroom proxy process with replicas: 1. The --workers flag is not set. Every worker shares one RWO PVC and writes the same proxy_savings.json and memory.db, so a second worker causes diverging savings numbers or SQLITE_BUSY errors.

Terminal window
export ANTHROPIC_BASE_URL=https://headroom.fzymgc.house
claude

headroom.fzymgc.house is reachable over the LAN and Tailscale. It is not on public DNS. To bypass Headroom, unset the variable. Nothing changes on the Headroom side.

Terminal window
unset ANTHROPIC_BASE_URL

The instance pushes metrics through OTLP/HTTP to ClickStack:

Setting Value
Endpoint http://cs-otel-collector.clickstack.svc.cluster.local:4318/v1/metrics
Auth header HEADROOM_OTEL_METRICS_HEADERS from the headroom-otlp-headers Secret, as Authorization=<raw token> with no Bearer word
Service name headroom-agents (HEADROOM_OTEL_SERVICE_NAME)
Resource attributes service.namespace=headroom plus the pod and node names (HEADROOM_OTEL_RESOURCE_ATTRIBUTES)
Upstream telemetry HEADROOM_TELEMETRY=off

In HyperDX, filter by service.name = headroom-agents. The prefix-cache signal is headroom_cache_write_ttl_tokens_total{provider=anthropic,ttl=<bucket>}, the tokens that landed in each TTL bucket. The pod’s /stats endpoint carries prefix_cache.observed_ttl_buckets per request.

An Uptime Kuma monitor reads /readyz on port 8787 (tf/uptime-kuma/monitors-services.tf).

The deployment runs ghcr.io/fzymgc-house/headroom-otel, an image derived from the official code-nonroot base that adds the OpenTelemetry SDK, which the official images omit (docs/adr/hl-8ycr-use-official-ghcr-headroom-images-do-not-build-cluster-owned.md). An upgrade takes two hops, and Renovate automerges both:

  1. Renovate’s dockerfile manager bumps the FROM ...@sha256:... pin in images/headroom-otel/Dockerfile on a new upstream release. The base is ghcr.io/headroomlabs-ai/headroom. If Renovate goes quiet for weeks, compare the FROM path with the image location in the upstream release notes.
  2. The merge runs .github/workflows/build-headroom-otel.yml, which publishes a new multi-arch ghcr.io/fzymgc-house/headroom-otel:<version>.
  3. Renovate’s argocd manager then bumps the digest of the image: reference in argocd/app-configs/headroom-agents/deployment.yaml.

To rebuild by hand, for example after a Dockerfile edit, merge to main or run gh workflow run build-headroom-otel.yml. After the build publishes, resolve the new digest:

Terminal window
crane digest ghcr.io/fzymgc-house/headroom-otel:<version>

At each bump, make sure that Headroom still installs to system site-packages and that the OpenTelemetry api and sdk pin still resolves. That is the standing risk the ADR records.

Symptom Cause Fix
/readyz returns 503 The pod is still starting Wait. The readiness probe starts after initialDelaySeconds: 10.
The OTLP push is rejected with 401 An extra Bearer word in the header value HEADROOM_OTEL_METRICS_HEADERS must be Authorization=<raw token>.
SQLite errors or a diverging savings ledger Two workers on one RWO PVC Keep replicas: 1 and no --workers flag.
The Uptime Kuma monitor times out while the pod is Ready The CiliumNetworkPolicy has no ingress rule for the uptime-kuma namespace on 8787 Make sure that argocd/app-configs/headroom-agents/networkpolicy.yaml allows uptime-kuma on 8787.
Permission denied: '/.cache' from Hugging Face in the log The pod runs as UID 65532 with no home directory, so HOME is / Make sure that the deployment sets HF_HOME to a path on the PVC.
headroom.fzymgc.house is unreachable from the laptop The laptop is not on the LAN or Tailscale Connect to Tailscale or the LAN. The host is not on public DNS.