Skip to content
fzymgc-house Cluster Docs

Headroom compression service

Headroom is an HTTP proxy that compresses LLM request context (JSON tool outputs, message history) before forwarding to the real upstream. Clients opt in by pointing OPENAI_BASE_URL or ANTHROPIC_BASE_URL at the relevant instance.

Overview

Section titled “Overview”

Two single-replica instances are deployed, one per namespace:

InstanceNamespaceRun modeSurfaceUpstream
headroom-appsheadroom-apps--mode tokenOpenAI /v1/chat/completionshttps://llm-gw.fzymgc.house/v1
headroom-agentsheadroom-agents--mode cacheAnthropic /v1/messageshttps://api.anthropic.com

Why two instances? Headroom’s run mode is a global per-process flag and the two client classes want opposite settings:

  • token mode maximises JSON compression — the right choice for in-cluster apps whose LLM calls carry large JSON tool outputs (mealie, karakeep, octopus). These apps route through llm-gw, which validates their virtual key and routes to OpenRouter/Ollama exactly as today.
  • cache mode freezes earlier turns in place to preserve Anthropic prefix-cache hits (cached input tokens are roughly 10x cheaper). Re-compressing earlier turns each request busts the cache and can cost more than compression saves — wrong for the laptop Claude Code subscription path.

Both instances run --no-cache to disable the separate semantic response cache, which risks returning a stale completion for a near-duplicate agentic prompt.

Topology

Section titled “Topology”
 laptop Claude Code ──ANTHROPIC_BASE_URL──▶ headroom-agents ─────▶ api.anthropic.com
 (subscription OAuth)   headroom.fzymgc.house    cache mode            (direct; bypasses llm-gw)
                         (Tailscale-internal)    code-nonroot image
                                                 PVC · svc.name=headroom-agents


 mealie / karakeep / ──OPENAI_BASE_URL────▶ headroom-apps ───────▶ llm-gw.fzymgc.house ──▶ OpenRouter / Ollama
 octopus (in-cluster)   ClusterIP only        token mode             (vkey validated here)
                                              nonroot image
                                              PVC · svc.name=headroom-apps


       both instances ──OTLP/HTTP :4318──▶ cs-otel-collector.clickstack (ClickStack)

Apps path (OpenAI surface). An app keeps its existing virtual key (vk_mealie, vk_karakeep, vk_octopus) as OPENAI_API_KEY and changes only OPENAI_BASE_URL. Headroom compresses the JSON-heavy message body in token mode, then forwards to llm-gw/v1 with the Authorization: Bearer vk_<app> header unchanged. llm-gw validates the key and routes to OpenRouter/Ollama as today.

Agents path (Anthropic surface). The laptop sets ANTHROPIC_BASE_URL=https://headroom.fzymgc.house and runs claude. Claude Code sends its subscription OAuth bearer in the Authorization header. Headroom compresses in cache mode (freezing prior turns), then forwards to api.anthropic.com with the bearer and anthropic-* headers unchanged. This path never touches llm-gw or OpenRouter.

Onboarding an app

Section titled “Onboarding an app”

For an in-cluster app that currently calls llm-gw.fzymgc.house/v1:

  1. Set OPENAI_BASE_URL=http://headroom-apps.headroom-apps.svc.cluster.local:8787/v1 in the app’s Deployment (via Vault/ExternalSecret or direct env).
  2. Keep the existing OPENAI_API_KEY / virtual key (vk_<app>) unchanged — Headroom forwards it to llm-gw which validates it as before.
  3. No model alias changes needed; Headroom preserves the model field.

Verify by checking the app’s LLM responses and then querying ClickStack for service.name=headroom-apps metrics.

Onboarding the laptop

Section titled “Onboarding the laptop”

For the operator laptop running Claude Code against the Anthropic subscription:

Terminal window
export ANTHROPIC_BASE_URL=https://headroom.fzymgc.house
claude

The subscription OAuth bearer in Authorization passes through unchanged. headroom.fzymgc.house is reachable over LAN and Tailscale; it is not on public DNS.

To stop routing through Headroom, unset the variable:

Terminal window
unset ANTHROPIC_BASE_URL

Worker model

Section titled “Worker model”

v1 runs the single-process headroom proxy form. Both deployments use the headroom proxy --host 0.0.0.0 --port 8787 ... CLI with replicas: 1.

The operator’s intended upgrade to 2 Gunicorn workers is DEFERRED because the 0.27.0 image’s process model could not be confirmed — Docker is unreachable in the build/drain sandbox, so it was not possible to run docker run --rm ghcr.io/chopratejas/headroom:0.27.0-nonroot --help and confirm whether a --workers flag (or a gunicorn binary on PATH) exists.

To resolve later, run this outside the sandbox:

Terminal window
docker run --rm ghcr.io/chopratejas/headroom:0.27.0-nonroot --help
# Look for a --workers flag.
docker run --rm --entrypoint sh ghcr.io/chopratejas/headroom:0.27.0-nonroot \
  -c "which gunicorn && gunicorn --version"

If a --workers flag exists, add --workers 2 to the args: list in the Deployment. If only gunicorn is present, wrap: gunicorn -w 2 -k uvicorn.workers.UvicornWorker headroom.proxy:app --bind 0.0.0.0:8787.

Caveat: 2 workers share one RWO PVC, so both processes write to the same proxy_savings.json and memory.db (SQLite). If ledger corruption appears (savings numbers diverge or SQLite SQLITE_BUSY errors appear in logs), drop back to 1 worker.

Observability

Section titled “Observability”

Both instances push metrics via OTLP/HTTP to ClickStack:

SettingValue
Endpointhttp://cs-otel-collector.clickstack.svc.cluster.local:4318/v1/metrics
Auth headerHEADROOM_OTEL_METRICS_HEADERS=Authorization=<RAW token> (no Bearer prefix — omit the word entirely; matches the deployed headroom-otlp-headers Secret)
Service nameheadroom-apps or headroom-agents (HEADROOM_OTEL_SERVICE_NAME)
Namespace tagservice.namespace=headroom (HEADROOM_OTEL_RESOURCE_ATTRIBUTES)

In HyperDX, filter by service.name = headroom-apps or service.name = headroom-agents to isolate each instance.

Key metrics:

  • headroom_cache_write_ttl_tokens_total{provider=anthropic,ttl=<bucket>} — prefix-cache signal on headroom-agents; shows how many tokens landed in each TTL bucket.
  • /stats.prefix_cache.observed_ttl_buckets — per-request prefix-cache observation (query the pod’s /stats endpoint for detail).

Upgrading the headroom image (derived headroom-otel)

Section titled “Upgrading the headroom image (derived headroom-otel)”

Both deployments run ghcr.io/fzymgc-house/headroom-otel, a thin image derived from the official code-nonroot base that adds the OpenTelemetry SDK (the official images omit it — see ADR hl-8ycr, amended). Upgrades are two-hop, both Renovate-automerged:

  1. Renovate’s dockerfile manager bumps the FROM ...@sha256:... pin in images/headroom-otel/Dockerfile on a new upstream Headroom release.
  2. Merging that PR triggers .github/workflows/build-headroom-otel.yml, which publishes a new multi-arch ghcr.io/fzymgc-house/headroom-otel:<version>.
  3. Renovate’s argocd manager then digest-bumps the image: refs in argocd/app-configs/headroom-{apps,agents}/deployment.yaml.

To rebuild manually (e.g. after editing the Dockerfile): merge to main, or run gh workflow run build-headroom-otel.yml. After publish, resolve the new digest with:

Terminal window
crane digest ghcr.io/fzymgc-house/headroom-otel:0.27.0
# or, without crane:
curl -sI -H "Authorization: Bearer $(curl -s \
  'https://ghcr.io/token?scope=repository:fzymgc-house/headroom-otel:pull' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')" \
  -H 'Accept: application/vnd.oci.image.index.v1+json' \
  https://ghcr.io/v2/fzymgc-house/headroom-otel/manifests/0.27.0 \
  | tr -d '\r' | grep -i docker-content-digest

Re-verify at each bump (standing ADR hl-8ycr risk): confirm Headroom still installs to system site-packages and that the api/sdk pin still resolves.

Rollback

Section titled “Rollback”

To bypass Headroom, revert the client’s *_BASE_URL to the direct upstream:

ClientRevert to
In-cluster appsOPENAI_BASE_URL=https://llm-gw.fzymgc.house/v1
Laptop Claude Codeunset ANTHROPIC_BASE_URL

No Headroom-side changes are needed; ArgoCD will keep the pods running.

Troubleshooting

Section titled “Troubleshooting”
SymptomLikely causeFix
/readyz returns 503Pod still initialisingWait — initial startup takes up to 10 s (initialDelaySeconds: 10)
OTLP push rejected with 401Extra Bearer word in header valueHEADROOM_OTEL_METRICS_HEADERS must be Authorization=<raw token> — not Authorization=Bearer <token>
SQLite errors / diverging savings ledgerTwo workers contending on same RWO PVCSet replicas: 1 (already the default) and confirm only one worker process is running
App gets completions but no compressionWrong run mode, or --no-cache misconfiguredCheck the pod args: list matches the intended mode for that instance
headroom.fzymgc.house unreachable from laptopNot on Tailscale or LANConnect to Tailscale or the office network; this host is not on public DNS