Headroom central LLM context-compression service

Headroom central LLM context-compression service

Status: Design (pre-plan) Bead: hl-v9xz Date: 2026-06-23 Deciders: Sean Brandt

Overview

Deploy the Headroom proxy (headroomlabs “Context Optimization Layer for LLM Applications”) as a central, in-cluster compression layer in front of the cluster’s existing LLM plane. Headroom is an HTTP proxy speaking both the Anthropic (/v1/messages) and OpenAI (/v1/chat/completions) wire formats; clients opt in by setting ANTHROPIC_BASE_URL / OPENAI_BASE_URL to it. It compresses request context (JSON tool outputs, message history) before forwarding to the real upstream, and forwards the caller’s credentials unchanged.

Headroom is complementary to agentgateway (llm-gw.fzymgc.house), which does auth, routing, model-aliasing, metering, and ZDR but no compression. Headroom chains in front of it for the OpenAI-surface in-cluster apps, and chains directly to Anthropic for the laptop Claude Code subscription path.

The design deploys two single-replica instances (not one) because Headroom’s single most important tuning knob — the run mode — is a global per-process flag, and the two client classes want opposite settings (see The driving constraint).

Goals

• Centralize LLM context compression for the in-cluster OpenAI-compatible apps (mealie, karakeep, octopus) routing through llm-gw.
• Front the operator’s laptop Claude Code (Anthropic subscription) so its context is compressed before reaching api.anthropic.com, stretching the subscription’s rate-limit / context budget.
• Centralize per-key budgets and rate-limits, and emit token-savings telemetry into ClickStack, reusing the agentgateway → ClickStack pattern landed in hl-rpm9.11.

Non-goals (YAGNI)

• No ML compression (headroom-ai[ml] / Kompress / ModernBERT). It needs model weights + RAM, runs a single global model behind a lock (sequential under concurrency), and only helps plain text (43–46%, latency-adding). The JSON SmartCrusher — which needs none of that — delivers the real wins.
• No semantic response cache. The LRU+TTL response cache risks returning a stale completion for a near-duplicate agentic prompt. Both instances run --no-cache.
• Engram is not onboarded. Its gateway traffic is embeddings — short query strings with no compressible JSON arrays or message history. Routing it through Headroom is a hop for ~zero benefit and would dilute the savings signal.
• No new edge authentication layer for Headroom itself (see Credentials & auth).

The driving constraint

Headroom’s run mode is a global per-process flag, and the two client classes want opposite modes:

Path	Wants	Why
Laptop Claude Code → Anthropic	--mode cache (“freeze prior turns”)	Preserve Anthropic prefix-cache hits (cached input tokens ~10× cheaper). Recompressing earlier turns each request busts the cache and can cost more than compression saves.
In-cluster apps → llm-gw	--mode token (maximize compression)	The apps’ value is compressing large JSON tool outputs (86–100%), where Headroom’s SmartCrusher actually fires.

One process cannot serve both optimally. That single fact splits the deployment into two instances. The split is not gold-plating — it falls directly out of an upstream constraint, and it resolves shared-state multi-tenancy (open-question #1) for free by giving each path its own workspace, budget, savings ledger, and telemetry service.name.

Grounded facts (Rule 7)

Grounding traces are recorded as bd note entries on hl-v9xz. Key findings that shape the design:

1. Code passes through by default. Headroom’s AST CodeCompressor is gated: protect_analysis_context=True protects all code in a conversation whenever the latest user message contains analyze/review/explain/fix/debug/optimize/error/bug; protect_recent_code=4 protects code in the last 4 messages; skip_user_messages=True. On coding-agent traffic, compression of the code under edit mostly does not fire — the win is JSON tool outputs and scoring-based dropping of stale old messages.
2. Two independent upstreams per instance. ANTHROPIC_TARGET_API_URL (default api.anthropic.com) and OPENAI_TARGET_API_URL (default api.openai.com) are separate; the docs show overriding the Anthropic target to an internal gateway. The Anthropic-surface → Anthropic and OpenAI-surface → llm-gw split is a first-class supported config.
3. Transparent credential forwarding. Headroom forwards the caller’s Authorization upstream and supplies none of its own (“api_key still needed for upstream”). The supported Claude Code pattern is literally ANTHROPIC_BASE_URL=<headroom> claude.
4. Per-process file/SQLite state under HEADROOM_WORKSPACE_DIR (default ~/.headroom): proxy_savings.json, toin.json, subscription_state.json, memory.db (SQLite), session_stats.jsonl, logs/. HEADROOM_CONFIG_DIR is read-mostly. Maps to one PVC per instance.
5. Official GHCR images ghcr.io/chopratejas/headroom, version-matched to the PyPI release. Tag matrix includes nonroot, slim/slim-nonroot (distroless), and code* variants (bundled tree-sitter), each per version (e.g. 0.27.0-code-nonroot). Existence confirmed by a manifest HEAD probe — the tags/list API paginates at 100/page and is unordered, so confirm a specific tag with HEAD /v2/chopratejas/headroom/manifests/<tag>, never a single tag-list page.
6. OTLP/HTTP metrics exporter with header auth: HEADROOM_OTEL_METRICS_HEADERS="key=value". Plus a Prometheus /metrics endpoint as a fallback. Provider prefix-cache usage is exposed via headroom_cache_write_ttl_tokens_total{provider,ttl} and /stats.prefix_cache.
7. Version. Current release is 0.27.0 (image and PyPI matched); the image tags 0.27.0, 0.27.0-nonroot, and 0.27.0-code-nonroot are confirmed present via manifest HEAD. The bead’s research saw ~0.5.21, so the project moves fast — a pre-1.0 single-vendor dependency. The grounded docs above track this 0.27.0 release, so they match the deployed image; still, re-verify the env-var / flag surface (HEADROOM_OTEL_METRICS_*, --mode, --no-cache) against the pinned image at implementation, and track bumps via Renovate.

Degraded grounding: deepwiki was used for Headroom-repo facts (image matrix, OTLP headers); agentgateway conventions are covered by repo-spine engram memories and repo prior art rather than a fresh deepwiki pass.

Architecture

 laptop Claude Code ──ANTHROPIC_BASE_URL──▶ headroom-agents ─────▶ api.anthropic.com
 (subscription OAuth)   internal DNS + TS    cache mode, --no-cache    (direct; bypasses gateway)
                                             code-nonroot image
                                             PVC · budget · svc.name=headroom-agents

 mealie / karakeep /  ──OPENAI_BASE_URL────▶ headroom-apps ───────▶ llm-gw.fzymgc.house ──▶ OpenRouter / Ollama
 octopus (in-cluster)   ClusterIP only       token mode, --no-cache   (vkey validated here, as today)
                                             nonroot image
                                             PVC · budget · svc.name=headroom-apps

       both instances ──OTLP/HTTP :4318, authorization=<raw token>──▶ cs-otel-collector (ClickStack)

Both instances live in their own namespaces (headroom-apps, headroom-agents) — separate namespaces give each a clean whole-namespace CiliumNetworkPolicy (endpointSelector: {}, the firewalla-mcp exemplar) and an independent Velero exclusion — are single-replica Deployments managed by ArgoCD, and are transparent: a client configured as if talking to its real upstream works unchanged except for the base URL swap.

Components

headroom-agents

Property	Value
Clients	Operator laptop Claude Code (Anthropic subscription)
Surface	Anthropic /v1/messages
Upstream	ANTHROPIC_TARGET_API_URL=https://api.anthropic.com (direct)
Run mode	--mode cache (freeze prior turns → preserve Anthropic prefix-cache)
Response cache	--no-cache
Context manager	Conservative initially — measure before enabling aggressive scoring-based drops (mid-conversation drops risk confusing the agent even with CCR retrieval)
Image	ghcr.io/chopratejas/headroom:0.27.0-code-nonroot (pin by tag now; resolve to @sha256: digest at implementation)
Port	Container headroom proxy --host 0.0.0.0 --port 8787; Service/probes target 8787 (8787 is Headroom’s documented default per docker-install)
Exposure	Hostname headroom.fzymgc.house → Traefik IngressRoute, reachable over LAN/Tailscale only (internal DNS, not public), real headroom.fzymgc.house cert-manager Certificate (mirrors llm-gw posture). Also a ClusterIP for future in-cluster Anthropic clients (fovea).
Workers	gunicorn --workers 2
Telemetry	service.name=headroom-agents, service.namespace=headroom

headroom-apps

Property	Value
Clients	mealie, karakeep, octopus (OpenAI-compatible, currently on llm-gw)
Surface	OpenAI /v1/chat/completions
Upstream	OPENAI_TARGET_API_URL=https://llm-gw.fzymgc.house/v1
Run mode	--mode token (maximize JSON compression)
Response cache	--no-cache (start off; revisit per-app if a clear win appears)
Context manager	IntelligentContextManager on (scoring-based message dropping)
Image	ghcr.io/chopratejas/headroom:0.27.0-nonroot (pin by tag now, resolve to @sha256: digest at implementation; debian-slim retains a shell for /stats curls during tuning)
Port	Container headroom proxy --host 0.0.0.0 --port 8787; Service/probes target 8787
Exposure	ClusterIP only (headroom-apps.headroom-apps.svc.cluster.local:8787)
Workers	gunicorn --workers 2
Telemetry	service.name=headroom-apps, service.namespace=headroom

Data flow

Apps path (OpenAI surface). An app keeps its existing virtual key (vk_mealie, vk_karakeep, vk_octopus) as OPENAI_API_KEY and changes only OPENAI_BASE_URL from https://llm-gw.fzymgc.house/v1 to http://headroom-apps.headroom-apps.svc.cluster.local:8787/v1. It sends POST /v1/chat/completions with its usual or-* model alias. Headroom compresses the JSON-heavy message body in token mode, then forwards to OPENAI_TARGET_API_URL=llm-gw/v1 with the Authorization: Bearer vk_<app> header unchanged. llm-gw validates the vkey, resolves the alias, and routes to OpenRouter/Ollama exactly as today. The model field is preserved (Headroom uses it for token counting). Multimodal/image content blocks pass through.

Agents path (Anthropic surface). The laptop sets ANTHROPIC_BASE_URL=https://headroom.fzymgc.house and runs claude. Claude Code sends its subscription OAuth bearer in the Authorization header (plus anthropic-* headers). Headroom compresses in cache mode (freezing prior turns to keep the prefix stable), then forwards to ANTHROPIC_TARGET_API_URL=https://api.anthropic.com with the bearer and Anthropic headers unchanged. Anthropic validates the subscription token; the request now egresses from the cluster’s IP rather than the laptop’s, which is benign. This path never touches llm-gw or OpenRouter.

Credentials & auth

Headroom is a transparent credential forwarder, which collapses open-question #2:

• Apps: no new pass-through secret. Each app keeps sending its own vkey, validated downstream at llm-gw. Per-app budgets and rate-limits work because Headroom keys on the incoming API key (the vkey).
• Agents: no cluster-held Anthropic credential. The laptop holds its own subscription token, which passes through.
• Edge auth: headroom-apps is ClusterIP-internal (no external surface). headroom-agents is reachable only over the Tailscale network boundary (an existing auth boundary — only the operator’s devices) and via the cluster-internal hostname, not public DNS — mirroring how llm-gw.fzymgc.house is internal-only. Upstream validation (Anthropic for agents, llm-gw vkey for apps) is the real gate. No Headroom-level authentication is added.
• Residual risk: a Tailscale peer could consume the headroom-agents budget, but cannot obtain completions without a valid Anthropic credential. Acceptable for a single-user homelab; a Cloudflare Access or agentgateway front-door can be added later if the threat model changes.

The single Vault secret either instance needs is the ClickStack OTLP ingest token (reused, not new): fzymgc-house/cluster/clickstack#otel_ingest_api_key.

Observability

Preferred (native OTLP push). Use the [otel] extra and push metrics directly:

HEADROOM_OTEL_METRICS_ENABLED=1
HEADROOM_OTEL_METRICS_EXPORTER=otlp_http
HEADROOM_OTEL_METRICS_ENDPOINT=http://cs-otel-collector.clickstack.svc.cluster.local:4318/v1/metrics
HEADROOM_OTEL_METRICS_HEADERS=authorization=<RAW ingest token, no "Bearer ">
HEADROOM_OTEL_SERVICE_NAME=headroom-agents   # or headroom-apps
HEADROOM_OTEL_RESOURCE_ATTRIBUTES=service.namespace=headroom

The raw-token-no-Bearer convention and the reuse of the clickstack ingest secret follow the agentgateway telemetry work (hl-rpm9.11). Note Headroom uses OTLP/HTTP :4318 (agentgateway used gRPC :4317).

Spec-time checks — both confirmed during design review:

1. The cs-otel-collector receiver and its MetalLB Service expose :4318 (argocd/app-configs/clickstack/otel-loadbalancer-service.yaml maps otlp-http 4318→4318). Residual: confirm the chart’s ClusterIP cs-otel-collector Service also lists :4318 (the in-cluster DNS target) — the LB sibling and the receiver both have it, so this is near-certain.
2. The clickstack-default CiliumNetworkPolicy already admits :4318 (argocd/app-configs/clickstack/networkpolicy.yaml, the world-ingress rule covers both :4317 and :4318). No CNP change needed.

Fallback (proven). If the OTLP/HTTP push is blocked, Headroom also serves Prometheus on /metrics; clone the agentgateway monitoring-otel-scraper static-job + ClusterIP-metrics-Service pattern.

Validation signals. headroom_tokens_saved_total / headroom_cost_usd_total per service.name quantify savings. For the apps path, cross-check against agentgateway’s gen_ai_client_token_usage at the other end of the compression. For the agents path, headroom_cache_write_ttl_tokens_total{provider="anthropic",ttl="5m|1h"} and /stats.prefix_cache.observed_ttl_buckets directly confirm cache mode is preserving Anthropic prefix-caching — the central design rationale becomes measurable.

State & concurrency

Each instance gets its own ReadWriteOnce PVC mounted at HEADROOM_WORKSPACE_DIR, holding proxy_savings.json, toin.json, SQLite memory.db, and session_stats.jsonl. Single replica per instance keeps RWO valid.

gunicorn --workers 2 runs two processes per pod sharing one workspace. The savings ledger and SQLite memory DB are written by both; the beacon lock is per-port. Residual risk: write contention / ledger corruption. Mitigation: start at 2 workers (operator decision), monitor for corruption, drop to --workers 1 if observed. Memory (memory.db) is disabled in proxy mode unless explicitly enabled, reducing the SQLite-contention surface.

Image & supply chain

Use the official ghcr.io/chopratejas/headroom images — do not build our own. Digest-pin 0.27.0 (0.27.0-nonroot for apps, 0.27.0-code-nonroot for agents), the current release — each tag confirmed to exist via a manifest HEAD probe. Confirm tag existence with HEAD /v2/chopratejas/headroom/manifests/<tag>, not the tags/list API — it paginates at 100/page and is unordered, so a tag’s absence from one page is not proof it is missing (a false “tag missing” reading occurred during design before the HEAD probe corrected it). Track upgrades via Renovate so version bumps (frequent, pre-1.0) arrive as reviewed PRs, and re-verify the env-var/flag surface against the pinned image at implementation. If the GHCR package is public, no pull secret is needed; if private, reuse the shared cluster/ghcr/pull-secret.

Change surfaces

Surface	Files
App-config A	argocd/app-configs/headroom-agents/ — Deployment, Service, PVC, ExternalSecret (OTLP token), env
App-config B	argocd/app-configs/headroom-apps/ — Deployment, Service, PVC, ExternalSecret (OTLP token), env
ArgoCD Applications	argocd/cluster-app/templates/headroom-apps.yaml + headroom-agents.yaml (one file per Application, repo convention), each with resources-finalizer.argocd.argoproj.io
Ingress	Traefik IngressRoute (Host('headroom.fzymgc.house') → headroom-agents:8787) + cert-manager Certificate (dnsNames: [headroom.fzymgc.house]) + the router-hosts.fzymgc.house/enabled: "true" DNS-sync annotation, following the llm-gw exposure pattern
Velero	Add headroom to the backup-schedule excludedNamespaces (argocd/app-configs/velero/backup-schedule.yaml) — the workspace PVCs hold only regenerable operational telemetry (savings ledger, session stats), not worth backup churn
Client repoints	OPENAI_BASE_URL for mealie / karakeep / octopus (their app-config); laptop ANTHROPIC_BASE_URL=https://headroom.fzymgc.house (operator-side, not in repo)
NetworkPolicy	Namespace baseline policy for headroom (egress to api.anthropic.com, llm-gw, and cs-otel-collector :4318). The clickstack-default CNP already admits :4318 — no change there.
Docs	docs/operations/headroom.md runbook; docs/reference/services.md + docs/reference/secrets.md entries

Rollout plan

1. Merge infra PR (both app-configs + Applications + ingress). ArgoCD syncs the two Deployments; verify /readyz (200) on each.
2. Wire telemetry, confirm headroom_* metrics land in ClickStack under both service.names.
3. Onboard apps incrementally — repoint one app’s OPENAI_BASE_URL (e.g. mealie first), verify a live request returns 200 through Headroom → llm-gw, and watch token-savings. Repeat per app.
4. Onboard the laptop — set ANTHROPIC_BASE_URL=https://headroom.fzymgc.house, run a Claude Code session, confirm responses and that prefix_cache buckets remain populated (cache mode preserving Anthropic caching). This step also validates the subscription-via-custom-base-URL assumption end-to-end before it is relied on.
5. Measure for a week, then tune (token vs context-manager aggressiveness, per-app --no-cache revisit).

Error handling & failure modes

• Compression failure is non-fatal. Headroom’s compressors “fail gracefully, return original content unchanged” — invalid JSON, AST parse failure, missing optional deps, or compression-makes-it-larger all pass through with a warning log.
• Headroom down ⇒ client down. Putting Headroom in the request path means an outage breaks the fronted client. Mitigation: liveness/readiness probes (/livez, /readyz — the latter 503s until ready), single-replica restart, and a documented one-line rollback (revert the client’s *_BASE_URL to the direct upstream). Apps stay one env var away from bypassing Headroom entirely.
• Budget exceeded returns a budget error and surfaces in /stats; budgets are per-key, so one app’s runaway cannot starve another.
• OTLP export failure must not affect request serving (metrics are best-effort); verify export errors are logged, not propagated.

Testing & validation

• Per-app smoke from inside each app pod using its own creds (the established pattern from the mealie/karakeep onboarding): a text completion (HTTP 200) and, for vision-capable apps, a base64 image request (HTTP 200).
• Agents smoke: a real Claude Code session through headroom-agents; confirm completions and prefix-cache TTL buckets.
• Savings assertion: compare headroom_tokens_saved_total and the agentgateway gen_ai_client_token_usage deltas over a representative window.
• Failure injection: send malformed JSON and oversized payloads; confirm graceful passthrough with a warning log, not a 5xx.

Risks & mitigations

Risk	Mitigation
Young, fast-moving pre-1.0 single-vendor dependency (rapid releases)	Digest-pin 0.27.0 (existence HEAD-verified); Renovate PRs for bumps; re-verify the env/flag surface against the pinned image at implementation
Claude Code subscription + custom ANTHROPIC_BASE_URL may be restricted by Anthropic ToS / client behavior	Verify the subscription path actually works end-to-end in rollout step 4 before relying on it; agents instance is independent of apps, so failure here doesn’t block app onboarding
cache mode insufficient to preserve prefix-cache; compression busts Anthropic caching net-negative	prefix_cache TTL counters make this directly observable; revert agents *_BASE_URL if net-negative
2-worker shared-state corruption	Monitor ledger/SQLite; drop to 1 worker
Headroom in-path becomes an availability SPOF for fronted clients	Probes + trivially reversible per-client *_BASE_URL rollback
Compression alters semantics for an app that parses raw model output strictly	Per-app onboarding + smoke test; --no-cache avoids cross-request staleness
ClickStack :4318 receiver/CNP not open	Spec-time check; Prometheus /metrics scrape fallback

Resolution of the bead’s open questions

1. Shared-state multi-tenancy → two instances, each with its own workspace PVC, budget, savings ledger, and service.name. No cross-tenant pooling.
2. Edge auth for Headroom → transparent pass-through; upstream validates; network boundary (ClusterIP / Tailscale) is the perimeter. No Headroom-level gate.
3. Semantic-cache correctness for agentic prompts → --no-cache on both; the cache run mode (different feature) is used only to preserve prefix-caching.
4. Value vs added hop/ops → accepted as a capability + centralization + observability investment; savings are measurable post-rollout via the telemetry, converting the open question into a data-backed review.
5. Relationship to in-flight agentgateway telemetry → reuses the hl-rpm9.11 ClickStack ingest pattern; complementary, not conflicting.

Future / phase-2

• Onboard fovea (Anthropic-direct PR-review engine) onto headroom-agents via the instance ClusterIP — same → api.anthropic.com target.
• Code compression tuning on the agents instance (the code-nonroot image already ships tree-sitter): selectively relax protect_analysis_context and measure quality.
• Cloudflare Access / agentgateway front-door for headroom-agents if the threat model expands beyond single-user Tailscale.