Skip to content
fzymgc-house Cluster Docs

Split Headroom into two instances by run mode

Date: 2026-06-23 Status: Accepted Decision: hl-g0fn Deciders: Sean Brandt

Context

Section titled “Context”

Headroom’s run mode (--mode cache vs --mode token) is a global per-process flag with no per-request override. The two client classes Headroom serves want opposite settings: the laptop Claude Code path (→ Anthropic) needs cache mode to preserve Anthropic prefix-cache economics; the in-cluster app path (mealie/karakeep/octopus → llm-gw) needs token mode to maximize JSON SmartCrusher compression on tool outputs. A single shared instance cannot serve both optimally.

Decision

Section titled “Decision”

Deploy two single-replica Headroom instances — headroom-agents (--mode cache, namespace headroom-agents) and headroom-apps (--mode token, namespace headroom-apps) — because the run mode cannot be scoped per request.

Rationale

Section titled “Rationale”
  • Headroom’s run mode is a global per-process flag with no per-request override; the two client classes require opposite values.
  • cache mode on the agents path preserves Anthropic prefix-cache (cached input tokens ~10× cheaper); token mode would bust it and cost more than compression saves.
  • token mode on the apps path fires the JSON SmartCrusher on tool outputs (86–100% JSON compression); cache mode would suppress those wins.
  • Two instances resolve cross-tenant state pooling (workspace PVC, budget, SQLite, telemetry service.name) without additional design work.

Alternatives Considered

Section titled “Alternatives Considered”
  • Two instances split by run mode (chosen): each instance optimally tuned; fully isolates workspace PVC, per-key budget, savings ledger, and telemetry service.name per client class; resolves shared-state multi-tenancy (open question #1) for free.
  • Single shared instance with one default run mode (rejected): simpler ops (one Deployment, PVC, ArgoCD Application) but either busts Anthropic prefix-cache (cache mode off) or foregoes JSON compression wins (token mode off), and pools cross-tenant state (savings ledger, budget, SQLite, telemetry) between structurally different client classes.

Consequences

Section titled “Consequences”
  • Positive: each instance is optimally configured for its client class; workspace, budget, savings ledger, and telemetry are isolated per client class; prefix-cache preservation is directly observable via headroom_cache_write_ttl_tokens_total.
  • Negative: double the manifest surface (two Deployments, PVCs, ExternalSecrets, ArgoCD Applications, namespaces); any Headroom-wide config change (e.g. an image bump) must be applied to both instances.
  • Neutral: single replica per instance is required by the RWO PVC constraint regardless of instance count; both instances share the same clickstack OTLP ingest secret but emit distinct service.name labels.