Skip to content
fzymgc-house Cluster Docs

Headroom compression service — implementation plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Deploy two single-replica Headroom proxy instances (headroom-apps, headroom-agents) as a GitOps-managed, in-cluster LLM context-compression layer — one tuned for in-cluster OpenAI apps (→ llm-gw), one for the operator’s laptop Claude Code subscription (→ api.anthropic.com).

Architecture: Two ArgoCD Applications, each a Kustomize app-config directory under argocd/app-configs/ with a Deployment + ClusterIP Service + RWO PVC + OTLP-headers ExternalSecret + CiliumNetworkPolicy. headroom-agents additionally gets a Traefik IngressRoute + cert-manager Certificate for headroom.fzymgc.house. Credentials pass through transparently (apps keep their vk_<app>; the laptop keeps its subscription token). Telemetry pushes OTLP/HTTP to the existing ClickStack collector. Design spec: docs/engineering/specs/2026-06-23-headroom-compression-service-design.md (bead hl-v9xz, design-review READY).

Tech Stack: Kubernetes, ArgoCD (GitOps), Kustomize, External Secrets Operator (Vault ClusterSecretStore), Cilium NetworkPolicy, Traefik IngressRoute, cert-manager, Velero. Image: ghcr.io/chopratejas/headroom:0.27.0 (public GHCR; -nonroot for apps, -code-nonroot for agents).

File Structure

Section titled “File Structure”

New directories / files (Create):

argocd/app-configs/headroom-apps/
├── namespace.yaml              # Namespace: headroom-apps
├── kustomization.yaml          # ties the below together, namespace: headroom-apps
├── deployment.yaml             # Deployment: token mode, OPENAI_TARGET=llm-gw, OTEL env, PVC mount
├── service.yaml                # ClusterIP :8787
├── pvc.yaml                    # RWO 1Gi longhorn workspace
├── otlp-headers-secret.yaml    # ExternalSecret -> HEADROOM_OTEL_METRICS_HEADERS
└── networkpolicy.yaml          # ingress from app namespaces; egress llm-gw + collector + DNS


argocd/app-configs/headroom-agents/
├── namespace.yaml              # Namespace: headroom-agents
├── kustomization.yaml
├── deployment.yaml             # cache mode, code-nonroot image, ANTHROPIC_TARGET=api.anthropic.com
├── service.yaml                # ClusterIP :8787
├── pvc.yaml                    # RWO 1Gi longhorn workspace
├── otlp-headers-secret.yaml    # ExternalSecret -> HEADROOM_OTEL_METRICS_HEADERS
├── certificate.yaml            # cloudflare-acme cert headroom.fzymgc.house (laptop-trusted)
├── ingress.yaml                # Traefik IngressRoute Host(headroom.fzymgc.house) -> :8787
└── networkpolicy.yaml          # ingress from traefik; egress api.anthropic.com + collector + DNS


argocd/cluster-app/templates/headroom-apps.yaml     # ArgoCD Application
argocd/cluster-app/templates/headroom-agents.yaml   # ArgoCD Application
docs/operations/headroom.md                          # runbook

Modify:

argocd/app-configs/velero/backup-schedule.yaml       # add both namespaces to excludedNamespaces
argocd/app-configs/mealie/deployment.yaml            # OPENAI_BASE_URL -> headroom-apps (Task 9)
argocd/app-configs/karakeep/deployment.yaml          # OPENAI_BASE_URL -> headroom-apps (Task 9)
docs/reference/services.md                            # add headroom entries
docs/reference/secrets.md                             # note reused clickstack OTLP token

Validation model (GitOps — no unit-test framework). Each manifest task’s “test” is, in order: yamllint -c .yamllint.yaml <files> → kubectl --context fzymgc-house kustomize argocd/app-configs/<dir> (builds clean) → kubectl --context fzymgc-house apply --dry-run=server -k argocd/app-configs/<dir> (server-side schema validation, read-only). Live behavior is verified post-merge via ArgoCD sync + curl smoke (Tasks 8–10). Never kubectl apply for real — ArgoCD owns deployment.

Task 1: headroom-apps app-config manifests

Section titled “Task 1: headroom-apps app-config manifests”

Files:

  • Create: argocd/app-configs/headroom-apps/namespace.yaml
  • Create: argocd/app-configs/headroom-apps/pvc.yaml
  • Create: argocd/app-configs/headroom-apps/otlp-headers-secret.yaml
  • Create: argocd/app-configs/headroom-apps/deployment.yaml
  • Create: argocd/app-configs/headroom-apps/service.yaml
  • Create: argocd/app-configs/headroom-apps/networkpolicy.yaml
  • Create: argocd/app-configs/headroom-apps/kustomization.yaml
  • Step 1: Create namespace.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: headroom-apps
  • Step 2: Create pvc.yaml (workspace state — savings ledger, TOIN, session stats)
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: headroom-apps-workspace
  namespace: headroom-apps
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 1Gi
  • Step 3: Create otlp-headers-secret.yaml (mirrors argocd/app-configs/fovea/otlp-headers-secret.yaml — RAW token, no Bearer)
---
# HyperDX ingest token for headroom-apps OTLP metrics export. cs-otel-collector
# rejects unauthenticated OTLP; HEADROOM_OTEL_METRICS_HEADERS takes this value
# VERBATIM. The collector expects the RAW token as the authorization value —
# NO "Bearer " scheme ("Bearer <token>" is rejected and echoes the token into
# sender logs). Reuses the same Vault key the otel-scraper/fovea senders use.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: headroom-otlp-headers
  namespace: headroom-apps
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: headroom-otlp-headers
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      data:
        headers: "authorization={{ .token }}"
  data:
    - secretKey: token
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: otel_ingest_api_key
  • Step 4: Create deployment.yaml

The official image’s default entrypoint is headroom proxy. Run it with explicit flags. HEADROOM_WORKSPACE_DIR is repointed to the PVC mount so all runtime state lands on the volume. --mode token maximizes JSON compression; --no-cache disables the semantic response cache (agentic-correctness). The worker count (operator chose 2) is applied in Step 9’s verification — the headroom proxy CLI process model must be confirmed against the 0.27.0 image first (see that step).

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: headroom-apps
  namespace: headroom-apps
  labels:
    app: headroom-apps
spec:
  replicas: 1            # single replica — RWO PVC is single-writer
  strategy:
    type: Recreate       # avoid two pods racing the same RWO volume on rollout
  selector:
    matchLabels:
      app: headroom-apps
  template:
    metadata:
      labels:
        app: headroom-apps
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        fsGroup: 65532    # make the PVC writable by the nonroot UID
      containers:
        - name: headroom
          image: ghcr.io/chopratejas/headroom:0.27.0-nonroot
          args:
            - "proxy"
            - "--host"
            - "0.0.0.0"
            - "--port"
            - "8787"
            - "--mode"
            - "token"
            - "--no-cache"
          ports:
            - containerPort: 8787
              name: http
          env:
            # Upstream: the in-cluster agentgateway LLM gateway. Headroom forwards
            # the caller's vkey (vk_mealie/vk_karakeep/...) unchanged to llm-gw,
            # which validates it exactly as today.
            - name: OPENAI_TARGET_API_URL
              value: "https://llm-gw.fzymgc.house/v1"
            # Relocate all runtime state onto the PVC.
            - name: HEADROOM_WORKSPACE_DIR
              value: "/workspace/.headroom"
            - name: HEADROOM_CONFIG_DIR
              value: "/workspace/.headroom/config"
            # OTLP/HTTP metrics push to ClickStack (reuses hl-rpm9.11 ingest token).
            - name: HEADROOM_OTEL_METRICS_ENABLED
              value: "1"
            - name: HEADROOM_OTEL_METRICS_EXPORTER
              value: "otlp_http"
            - name: HEADROOM_OTEL_METRICS_ENDPOINT
              value: "http://cs-otel-collector.clickstack.svc.cluster.local:4318/v1/metrics"
            - name: HEADROOM_OTEL_METRICS_HEADERS
              valueFrom:
                secretKeyRef:
                  name: headroom-otlp-headers
                  key: headers
            - name: HEADROOM_OTEL_SERVICE_NAME
              value: "headroom-apps"
            - name: HEADROOM_OTEL_RESOURCE_ATTRIBUTES
              value: "service.namespace=headroom"
            # Anonymous upstream beacon stays OFF (default), explicit for clarity.
            - name: HEADROOM_TELEMETRY
              value: "off"
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: false  # Python app writes temp files outside the PVC
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8787
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /livez
              port: 8787
            initialDelaySeconds: 20
            periodSeconds: 30
          volumeMounts:
            - name: workspace
              mountPath: /workspace
      volumes:
        - name: workspace
          persistentVolumeClaim:
            claimName: headroom-apps-workspace
  • Step 5: Create service.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: headroom-apps
  namespace: headroom-apps
  labels:
    app: headroom-apps
spec:
  type: ClusterIP
  selector:
    app: headroom-apps
  ports:
    - name: http
      port: 8787
      targetPort: 8787
      protocol: TCP
  • Step 6: Create networkpolicy.yaml

Ingress from the OpenAI-app namespaces; egress to DNS, the ClickStack collector (:4318), and the world on :443 (llm-gw is reachable via its MetalLB VIP; world covers it without hard-coding the VIP).

---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: headroom-apps-default
  namespace: headroom-apps
spec:
  endpointSelector: {}
  ingress:
    # OpenAI-compatible app clients reach the proxy on 8787.
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: mealie
        - matchLabels:
            io.kubernetes.pod.namespace: karakeep
        - matchLabels:
            io.kubernetes.pod.namespace: octopus
      toPorts:
        - ports:
            - port: "8787"
              protocol: TCP
  egress:
    # DNS resolution.
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
    # OTLP/HTTP metrics to the ClickStack collector.
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: clickstack
      toPorts:
        - ports:
            - port: "4318"
              protocol: TCP
    # llm-gw (MetalLB VIP) over HTTPS — reached as an in-cluster LoadBalancer.
    - toEntities:
        - world
        - cluster
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
  • Step 7: Create kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization


namespace: headroom-apps


resources:
  - namespace.yaml
  - pvc.yaml
  - otlp-headers-secret.yaml
  - deployment.yaml
  - service.yaml
  - networkpolicy.yaml
  • Step 8: Lint and build

Run: yamllint -c .yamllint.yaml argocd/app-configs/headroom-apps/ Expected: no errors (exit 0).

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/headroom-apps Expected: clean combined YAML of all 6 resources, no kustomize error.

  • Step 9: Confirm the image process model and finalize worker count

The operator chose 2 workers, but the headroom proxy CLI process model isn’t pinned from docs. Confirm against the actual 0.27.0 image before relying on multi-worker:

Run: docker run --rm ghcr.io/chopratejas/headroom:0.27.0-nonroot proxy --help Inspect for a --workers option, and check whether gunicorn is on PATH: Run: docker run --rm --entrypoint sh ghcr.io/chopratejas/headroom:0.27.0-nonroot -c "command -v gunicorn && headroom proxy --help | grep -i worker || true"

Then apply ONE of:

  • If headroom proxy --workers N exists: add "--workers" / "2" to the Deployment args.
  • Else if gunicorn is present: change the container to command: ["gunicorn"] with args: ["headroom.proxy.server:app","--workers","2","--worker-class","uvicorn.workers.UvicornWorker","--bind","0.0.0.0:8787"], and move --mode token / --no-cache to env (HEADROOM_MODE=token; disable the response cache via its env toggle — confirm the exact var from headroom proxy --help output, e.g. a HEADROOM_CACHE_* flag).
  • Else (single-process only): keep the Step-4 single-process headroom proxy form; record in the runbook that v1 is single-process and 2 workers is deferred pending upstream support.

Document which branch was taken in docs/operations/headroom.md (Task 7).

  • Step 10: Commit

Commit the headroom-apps/ directory using VCS-appropriate commands per references/vcs-preamble.md. Message: feat(headroom): add headroom-apps compression proxy app-config [hl-v9xz]

Task 2: headroom-apps ArgoCD Application

Section titled “Task 2: headroom-apps ArgoCD Application”

Files:

  • Create: argocd/cluster-app/templates/headroom-apps.yaml

  • Step 1: Create the Application (mirrors argocd/cluster-app/templates/firewalla-mcp.yaml)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: headroom-apps
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "6"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/fzymgc-house/selfhosted-cluster
    targetRevision: HEAD
    path: argocd/app-configs/headroom-apps
  destination:
    server: https://kubernetes.default.svc
    namespace: headroom-apps
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
  ignoreDifferences:
    - kind: ExternalSecret
      group: external-secrets.io
      jqPathExpressions:
        - .spec.data[].remoteRef.conversionStrategy
        - .spec.data[].remoteRef.decodingStrategy
        - .spec.data[].remoteRef.metadataPolicy
  • Step 2: Lint

Run: yamllint -c .yamllint.yaml argocd/cluster-app/templates/headroom-apps.yaml Expected: exit 0.

  • Step 3: Commit

Message: feat(headroom): register headroom-apps ArgoCD Application [hl-v9xz]

Task 3: headroom-agents app-config manifests

Section titled “Task 3: headroom-agents app-config manifests”

Files:

  • Create: argocd/app-configs/headroom-agents/namespace.yaml
  • Create: argocd/app-configs/headroom-agents/pvc.yaml
  • Create: argocd/app-configs/headroom-agents/otlp-headers-secret.yaml
  • Create: argocd/app-configs/headroom-agents/deployment.yaml
  • Create: argocd/app-configs/headroom-agents/service.yaml
  • Create: argocd/app-configs/headroom-agents/networkpolicy.yaml
  • Create: argocd/app-configs/headroom-agents/kustomization.yaml
  • Step 1: Create namespace.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: headroom-agents
  • Step 2: Create pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: headroom-agents-workspace
  namespace: headroom-agents
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 1Gi
  • Step 3: Create otlp-headers-secret.yaml (same as apps, namespace headroom-agents)
---
# HyperDX ingest token for headroom-agents OTLP metrics export. RAW token, no
# "Bearer " (collector rejects the Bearer scheme; see fovea precedent).
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: headroom-otlp-headers
  namespace: headroom-agents
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: headroom-otlp-headers
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      data:
        headers: "authorization={{ .token }}"
  data:
    - secretKey: token
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: otel_ingest_api_key
  • Step 4: Create deployment.yaml

Differences from apps: code-nonroot image (Claude Code is code-heavy), --mode cache (freeze prior turns → preserve Anthropic prefix-cache), ANTHROPIC_TARGET_API_URL=https://api.anthropic.com (explicit, though it is also the default), service.name=headroom-agents. No OPENAI_TARGET (this instance serves only the Anthropic surface). Apply the same Step-9 worker decision from Task 1.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: headroom-agents
  namespace: headroom-agents
  labels:
    app: headroom-agents
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: headroom-agents
  template:
    metadata:
      labels:
        app: headroom-agents
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        fsGroup: 65532
      containers:
        - name: headroom
          image: ghcr.io/chopratejas/headroom:0.27.0-code-nonroot
          args:
            - "proxy"
            - "--host"
            - "0.0.0.0"
            - "--port"
            - "8787"
            # Two ORTHOGONAL flags: `--mode cache` freezes prior turns to preserve
            # Anthropic prefix-cache hits; `--no-cache` disables the separate
            # semantic *response* cache (LRU/TTL). Both are intentional here.
            - "--mode"
            - "cache"
            - "--no-cache"
          ports:
            - containerPort: 8787
              name: http
          env:
            # Upstream: Anthropic directly (subscription path bypasses llm-gw/OpenRouter).
            # This is also Headroom's default ANTHROPIC target; set explicitly for clarity.
            - name: ANTHROPIC_TARGET_API_URL
              value: "https://api.anthropic.com"
            - name: HEADROOM_WORKSPACE_DIR
              value: "/workspace/.headroom"
            - name: HEADROOM_CONFIG_DIR
              value: "/workspace/.headroom/config"
            - name: HEADROOM_OTEL_METRICS_ENABLED
              value: "1"
            - name: HEADROOM_OTEL_METRICS_EXPORTER
              value: "otlp_http"
            - name: HEADROOM_OTEL_METRICS_ENDPOINT
              value: "http://cs-otel-collector.clickstack.svc.cluster.local:4318/v1/metrics"
            - name: HEADROOM_OTEL_METRICS_HEADERS
              valueFrom:
                secretKeyRef:
                  name: headroom-otlp-headers
                  key: headers
            - name: HEADROOM_OTEL_SERVICE_NAME
              value: "headroom-agents"
            - name: HEADROOM_OTEL_RESOURCE_ATTRIBUTES
              value: "service.namespace=headroom"
            - name: HEADROOM_TELEMETRY
              value: "off"
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: false
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8787
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /livez
              port: 8787
            initialDelaySeconds: 20
            periodSeconds: 30
          volumeMounts:
            - name: workspace
              mountPath: /workspace
      volumes:
        - name: workspace
          persistentVolumeClaim:
            claimName: headroom-agents-workspace
  • Step 5: Create service.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: headroom-agents
  namespace: headroom-agents
  labels:
    app: headroom-agents
spec:
  type: ClusterIP
  selector:
    app: headroom-agents
  ports:
    - name: http
      port: 8787
      targetPort: 8787
      protocol: TCP
  • Step 6: Create networkpolicy.yaml

Ingress from the traefik namespace (the IngressRoute proxies to the pod); egress to DNS, the collector, and the world on :443 (api.anthropic.com).

---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: headroom-agents-default
  namespace: headroom-agents
spec:
  endpointSelector: {}
  ingress:
    # Traefik terminates TLS for headroom.fzymgc.house and proxies to 8787.
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: traefik
      toPorts:
        - ports:
            - port: "8787"
              protocol: TCP
  egress:
    # DNS resolution.
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
    # OTLP/HTTP metrics to the ClickStack collector.
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: clickstack
      toPorts:
        - ports:
            - port: "4318"
              protocol: TCP
    # api.anthropic.com over HTTPS.
    - toEntities:
        - world
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
  • Step 7: Create kustomization.yaml (includes the ingress + certificate from Task 4)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization


namespace: headroom-agents


resources:
  - namespace.yaml
  - pvc.yaml
  - otlp-headers-secret.yaml
  - deployment.yaml
  - service.yaml
  - certificate.yaml
  - ingress.yaml
  - networkpolicy.yaml
  • Step 8: Lint (build deferred to Task 4 — kustomization references certificate.yaml/ingress.yaml created there)

Run: yamllint -c .yamllint.yaml argocd/app-configs/headroom-agents/ Expected: exit 0.

  • Step 9: Commit

Message: feat(headroom): add headroom-agents compression proxy app-config [hl-v9xz]

Task 4: headroom-agents TLS + Traefik ingress

Section titled “Task 4: headroom-agents TLS + Traefik ingress”

Files:

  • Create: argocd/app-configs/headroom-agents/certificate.yaml
  • Create: argocd/app-configs/headroom-agents/ingress.yaml

The laptop must trust the cert without installing the internal CA, so use the cloudflare-acme-issuer (publicly trusted, DNS-01 validated — same issuer karakeep’s public cert uses). Reachability stays internal via the router-hosts.fzymgc.house/enabled DNS-sync annotation (resolves to the internal Traefik endpoint, like other *.fzymgc.house services).

  • Step 1: Create certificate.yaml (mirrors argocd/app-configs/karakeep/certificate.yaml public cert)
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: headroom-agents-public-tls
  namespace: headroom-agents
spec:
  secretName: headroom-agents-public-tls
  issuerRef:
    name: cloudflare-acme-issuer
    kind: ClusterIssuer
  dnsNames:
    - headroom.fzymgc.house
  usages:
    - server auth
  • Step 2: Create ingress.yaml (mirrors argocd/app-configs/karakeep/ingress.yaml)
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: headroom-agents
  namespace: headroom-agents
  annotations:
    router-hosts.fzymgc.house/enabled: "true"
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`headroom.fzymgc.house`)
      kind: Rule
      services:
        - name: headroom-agents
          port: 8787
  tls:
    secretName: headroom-agents-public-tls
  • Step 3: Lint and build the full headroom-agents kustomization

Run: yamllint -c .yamllint.yaml argocd/app-configs/headroom-agents/ Expected: exit 0.

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/headroom-agents Expected: clean combined YAML of all 8 resources, no kustomize error.

  • Step 4: Commit

Message: feat(headroom): expose headroom-agents at headroom.fzymgc.house (Traefik + cert) [hl-v9xz]

Task 5: headroom-agents ArgoCD Application

Section titled “Task 5: headroom-agents ArgoCD Application”

Files:

  • Create: argocd/cluster-app/templates/headroom-agents.yaml

  • Step 1: Create the Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: headroom-agents
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "6"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/fzymgc-house/selfhosted-cluster
    targetRevision: HEAD
    path: argocd/app-configs/headroom-agents
  destination:
    server: https://kubernetes.default.svc
    namespace: headroom-agents
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
  ignoreDifferences:
    - kind: ExternalSecret
      group: external-secrets.io
      jqPathExpressions:
        - .spec.data[].remoteRef.conversionStrategy
        - .spec.data[].remoteRef.decodingStrategy
        - .spec.data[].remoteRef.metadataPolicy
  • Step 2: Lint

Run: yamllint -c .yamllint.yaml argocd/cluster-app/templates/headroom-agents.yaml Expected: exit 0.

  • Step 3: Commit

Message: feat(headroom): register headroom-agents ArgoCD Application [hl-v9xz]

Task 6: Velero — exclude both namespaces from backup

Section titled “Task 6: Velero — exclude both namespaces from backup”

Files:

  • Modify: argocd/app-configs/velero/backup-schedule.yaml

The workspace PVCs hold only regenerable operational telemetry (savings ledger, TOIN, session stats) — not user data — so per the repo’s exclude-only strategy they belong in excludedNamespaces.

  • Step 1: Add both namespaces under the Telemetry category — in BOTH schedules

argocd/app-configs/velero/backup-schedule.yaml defines two Schedule resources (daily-backup at the top, weekly-full-backup below) — each has its own # Telemetry - stateless collectors, data in external storage block (currently vector, grafana-mcp, clickhouse-mcp, firewalla-mcp, loki, prometheus, cloudflared). Add the two headroom namespaces to both blocks:

      # Telemetry - stateless collectors, data in external storage
      - vector
      - grafana-mcp
      - clickhouse-mcp
      - firewalla-mcp
      - loki
      - prometheus
      - cloudflared
      # Headroom compression proxies — PVCs hold only regenerable savings ledgers
      - headroom-apps
      - headroom-agents
  • Step 2: Lint

Run: yamllint -c .yamllint.yaml argocd/app-configs/velero/backup-schedule.yaml Expected: exit 0.

  • Step 3: Commit

Message: chore(velero): exclude headroom telemetry namespaces from backup [hl-v9xz]

Task 7: Operations runbook + reference docs

Section titled “Task 7: Operations runbook + reference docs”

Files:

  • Create: docs/operations/headroom.md
  • Modify: docs/reference/services.md
  • Modify: docs/reference/secrets.md
  • Step 1: Create docs/operations/headroom.md

Write a runbook with these sections (real content, not placeholders): Overview (two instances, what each fronts, the run-mode split); Topology (the two data-flow paths from the design spec); Onboarding an app (set OPENAI_BASE_URL=http://headroom-apps.headroom-apps.svc.cluster.local:8787/v1, keep the existing vk_<app>); Onboarding the laptop (ANTHROPIC_BASE_URL=https://headroom.fzymgc.house); Worker model (record which Task 1 Step 9 branch was taken); Observability (service.name in ClickStack; the headroom_cache_write_ttl_tokens_total / /stats.prefix_cache prefix-cache signal); Rollback (revert the client’s *_BASE_URL to the direct upstream — apps to https://llm-gw.fzymgc.house/v1, laptop unset); Troubleshooting (/readyz 503, OTLP 401 = Bearer-prefix mistake, PVC write-contention if 2 workers). Add the Starlight frontmatter (---\ntitle: "Headroom compression service"\n---) as the first lines.

  • Step 2: Add a headroom row/section to docs/reference/services.md

Document both hostnames/endpoints: headroom-apps (ClusterIP headroom-apps.headroom-apps.svc.cluster.local:8787, internal only) and headroom-agents (https://headroom.fzymgc.house, Tailscale-internal). Match the file’s existing table/section format (read the file first and mirror it).

  • Step 3: Note the reused OTLP secret in docs/reference/secrets.md

Add a note that both headroom namespaces consume the existing fzymgc-house/cluster/clickstack → otel_ingest_api_key (no new Vault path). Mirror the file’s existing format.

  • Step 4: Lint docs

Run: rumdl check --config docs/.rumdl.toml docs/operations/headroom.md docs/reference/services.md docs/reference/secrets.md Expected: no issues.

  • Step 5: Commit

Message: docs(headroom): add runbook + service/secret references [hl-v9xz]

Task 8: Open the infra PR, merge, verify both instances live

Section titled “Task 8: Open the infra PR, merge, verify both instances live”

Files: none (operational).

  • Step 1: Push the branch and open a PR

Open a PR with Tasks 1–7 (app-configs, Applications, velero, docs). Per the writing-plans docs-first guidance, this PR carries both the implementation and the spec/plan/ADR docs, so a single merge lands everything.

  • Step 2: After merge, watch ArgoCD sync

Run: kubectl --context fzymgc-house -n argocd get application headroom-apps headroom-agents -o wide Expected: both Synced / Healthy.

  • Step 3: Verify readiness of both pods

Run: kubectl --context fzymgc-house -n headroom-apps get pods and kubectl --context fzymgc-house -n headroom-agents get pods Expected: 1 Running pod each, READY 1/1.

Run (from inside each pod, apps shown): kubectl --context fzymgc-house -n headroom-apps exec deploy/headroom-apps -- sh -c "wget -qO- http://localhost:8787/readyz" Expected: JSON with "ready": true. (For the distroless-free nonroot image a shell + wget exist; if not, kubectl port-forward and curl from the workstation.)

  • Step 4: Confirm OTLP metrics land in ClickStack

Generate a test request (see Task 9 Step 2 for the apps smoke), then query ClickStack via the ClickHouse MCP / HyperDX for ServiceName IN ('headroom-apps','headroom-agents') in default.otel_metrics_*. Expected: rows for headroom_requests_total / headroom_tokens_saved_total. If absent, check the pod logs for an OTLP 401 (means the Bearer prefix slipped in — the value must be authorization=<raw token>).

  • Step 5: Confirm headroom-agents is reachable AND internal-only

From the laptop (on Tailscale): Run: curl -sS -o /dev/null -w "%{http_code}\n" https://headroom.fzymgc.house/livez Expected: 200 and the cert is trusted (no -k needed — cloudflare-acme is publicly trusted).

From outside the network (phone off-WiFi / public host): Run: curl -sS -m 5 -o /dev/null -w "%{http_code}\n" https://headroom.fzymgc.house/livez || echo "unreachable (expected)" Expected: timeout / unreachable. If it answers publicly, STOP and gate it (Cloudflare Access or restrict the Traefik route) before onboarding the subscription — a public unauthenticated proxy to api.anthropic.com is unacceptable.

Task 9: Incrementally onboard the in-cluster apps

Section titled “Task 9: Incrementally onboard the in-cluster apps”

Files:

  • Modify: argocd/app-configs/mealie/deployment.yaml
  • Modify: argocd/app-configs/karakeep/deployment.yaml
  • (octopus: same pattern if still live — confirm with kubectl -n octopus get deploy first)

Do one app at a time, each its own PR, verifying before the next.

  • Step 1: Repoint mealie

In argocd/app-configs/mealie/deployment.yaml, change the OPENAI_BASE_URL env value from https://llm-gw.fzymgc.house/v1 to http://headroom-apps.headroom-apps.svc.cluster.local:8787/v1. Leave OPENAI_API_KEY (the vk_mealie secret ref) and OPENAI_MODEL unchanged.

      - name: OPENAI_BASE_URL
        value: "http://headroom-apps.headroom-apps.svc.cluster.local:8787/v1"
  • Step 2: Lint, commit, PR, merge

Run: yamllint -c .yamllint.yaml argocd/app-configs/mealie/deployment.yaml Message: feat(mealie): route LLM traffic through headroom-apps compression [hl-v9xz]

  • Step 3: Smoke-test mealie through Headroom

After ArgoCD syncs and the mealie pod restarts (Reloader/rollout), exercise a real LLM call from inside the pod using its own env (the established mealie verification pattern): Run: kubectl --context fzymgc-house -n mealie exec deploy/mealie -- python3 -c "import os,urllib.request,json; req=urllib.request.Request(os.environ['OPENAI_BASE_URL'].rstrip('/')+'/chat/completions', data=json.dumps({'model':os.environ.get('OPENAI_MODEL','or-gemini-2-5-flash'),'messages':[{'role':'user','content':'ping'}],'max_tokens':5}).encode(), headers={'Authorization':'Bearer '+os.environ['OPENAI_API_KEY'],'Content-Type':'application/json'}); print(urllib.request.urlopen(req).status)" Expected: 200. Then confirm in ClickStack that headroom-apps recorded the request and that agentgateway still saw the downstream gen_ai_client_token_usage (proves the full chain).

  • Step 4: Repeat for karakeep, then octopus

Same edit + smoke per app. Each is an independent PR so any regression is isolated and revertible by flipping OPENAI_BASE_URL back.

Task 10: Onboard the laptop Claude Code subscription

Section titled “Task 10: Onboard the laptop Claude Code subscription”

Files: none (operator-side config).

  • Step 1: Point Claude Code at the agents instance

On the laptop: export ANTHROPIC_BASE_URL=https://headroom.fzymgc.house then run claude. (Set it in the shell profile only after the smoke passes.)

  • Step 2: Validate the subscription path end-to-end

Run a short Claude Code session (a couple of turns). Confirm responses come back normally — this validates the core unproven assumption that a subscription token tunnels through a custom ANTHROPIC_BASE_URL to api.anthropic.com. If Claude Code refuses or errors on auth, the subscription path is blocked: revert (unset ANTHROPIC_BASE_URL), record the finding in the runbook, and keep headroom-agents for any Anthropic-API-key clients (e.g. future fovea onboarding) instead.

  • Step 3: Confirm prefix-cache preservation (the cache-mode rationale)

Run: curl -sS https://headroom.fzymgc.house/stats | python3 -m json.tool | grep -A8 prefix_cache Expected: prefix_cache.by_provider.anthropic.observed_ttl_buckets populated (5m/1h buckets with tokens) across the session — proof that --mode cache is preserving Anthropic prefix-caching rather than busting it. Also confirm in ClickStack that headroom_cache_write_ttl_tokens_total{provider="anthropic"} is non-zero and headroom_tokens_saved_total{service.name=headroom-agents} is climbing.

  • Step 4: Close out

Record measured savings in the runbook after ~1 week. File follow-up beads for any tuning (context-manager aggressiveness, per-app --no-cache revisit, fovea phase-2 onboarding, resolving the image to an @sha256: digest).

Notes for the implementer

Section titled “Notes for the implementer”
  • GitOps discipline: never kubectl apply real changes — only --dry-run and read-only get/exec. ArgoCD owns sync after merge.
  • Branch protection: every task’s commit goes on a feature branch; open PRs, never push to main.
  • Digest pinning: the spec calls for resolving the 0.27.0-* tags to @sha256: digests for supply-chain integrity. Do this when creating the Deployments if convenient (crane digest ghcr.io/chopratejas/headroom:0.27.0-nonroot), or as the Task 10 Step 4 follow-up.
  • The worker-count branch (Task 1 Step 9) is the one item that genuinely depends on inspecting the pinned image. Resolve it once and apply the same form to both Deployments.