router-hosts 0.10.7 upgrade — implementation plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Upgrade both halves of router-hosts to 0.10.7 so the in-cluster operator can publish DNS changes again, resolving incident hl-37e4 (operator wedged since ~2026-06-17, openrouter-gw NXDOMAIN for ~8 days).

Architecture: router-hosts ships as two coordinated binaries from one release tag — a server (gRPC + dnsmasq emitter) that runs on the Firewalla router via Ansible, and an operator (gRPC client / Kubernetes controller) that runs in-cluster via ArgoCD. This repo pins both. The upgrade is staged and server-first: the server must land the commit-on-timeout rollback (0.10.6) and the compaction command (0.10.7) — and the bloated llm-gw aggregate must be compacted — before the operator is upgraded, otherwise the operator’s newly-correct writes reactivate the runaway.

Tech Stack: Ansible (router-hosts role, manual ansible-playbook against the Firewalla), ArgoCD multi-source Application (OCI Helm chart ghcr.io/fzymgc-house/charts/router-hosts-operator + in-repo values), the router-hosts CLI over mTLS gRPC, dig against the Firewalla resolver (192.168.20.1).

Tracking: bead hl-fq4n (this upgrade) · blocks hl-37e4 (incident) · related hl-ls5 (server-bump task, now folded in).

Verification model (read first — ops upgrade, not unit-tested code)

This plan touches a deployed binary on the router and a GitOps-managed operator, so the red-green TDD loop does not apply. The verification chain per phase is:

State capture before — record current versions, aggregate event counts, and DNS answers so “after” is comparable and rollback is possible.
Apply — Ansible run (server) or git merge (operator, via ArgoCD auto-sync).
Observe — container/pod up, router-hosts host list returns records, operator logs free of AlreadyExists / version conflict, HostMapping reaches phase: Synced.
End-to-end — a real host change publishes to DNS within one reconcile (dig @192.168.20.1).

GitOps coupling — the single most important operational fact in this plan: the operator Application (argocd/cluster-app/templates/router-hosts-operator.yaml) has syncPolicy.automated (prune: true, selfHeal: true). Merging the chart-version bump to main IS the operator rollout trigger. The server, by contrast, deploys only when someone runs ansible-playbook by hand. To honor server-first ordering with a single PR, the server Ansible run is executed from the unmerged PR branch, and the PR is merged last (Task 4). Do not merge before the server is upgraded, compacted, and verified.

Background: why this upgrade, why this order

Incident hl-37e4 root cause (operator 0.10.2, server 0.10.5):

Operator sent optimistic-concurrency expected_version=0 on every UpdateHost → “version conflict: expected 0, got N” on every pre-existing host (upstream #323).
Operator reconcileUpsert called create-only AddHost on existing hosts → AlreadyExists spam every reconcile (upstream #313).
Server committed events past the 30s gRPC deadline instead of rolling back → one aggregate (llm-gw / agentgateway-hosts) bloated to ~91k events; correct-version writes rehydrate all of them → exceed 30s → DeadlineExceeded-but-commit → runaway growth (upstream #330).
Server snapshots.trigger_type column missing → no snapshot/rollback safety net (upstream #331).

What the releases deliver (only #332 and later matter — server is already on 0.10.5):

Release	Change	Side	Relevance
0.10.3	docs only	—	no-op
0.10.4	`dnsmasq address=` output (#326)	server	already deployed
0.10.5	regenerate outputs on startup (#328)	server	already deployed
0.10.6	#332: operator idempotent reconcile via host-adoption (#313); server commit-on-timeout rollback; server `snapshots.trigger_type` migration + boot assertion (#331)	both	core fix
0.10.7	#336/#337: server aggregate compaction CLI (`compact`) + over-threshold guard; event-count gauges	server	shrinks the ~91k aggregate so writes finish < 30s

The reactivation hazard (why server-first is mandatory): the runaway is currently dormant by accident — the operator’s expected_version=0 writes are rejected (never committed), so the aggregate is stable at ~v91178. Upgrading the operator first removes that accidental safety valve: the operator’s host-adoption path learns the real version and sends a correct write to the 91k aggregate, which on a server still at 0.10.5 rehydrates >30s → commits-on-timeout → runaway resumes. Server-first (commit-on-timeout fixed + aggregate compacted) closes this hole before the operator is allowed to write successfully. Upstream #336 confirms the design: compaction folds the log to a single seed event at the preserved high-water version, so host update --version <V> (the operator’s adoption write) succeeds afterward.

Pre-grounded facts (verified 2026-06-26):

Chart 0.10.7 exists in ghcr.io/fzymgc-house/charts and its values schema (routerHosts.serverAddress / defaultIngressIP / tlsSecret.{name,mountPath}) is identical to our argocd/app-configs/router-hosts-operator/values.yaml → no operator config migration.
The 0.10.7 chart CRD has zero ipAddress references (uses spec.ip) → our argocd/app-configs/agentgateway/hostmapping.yaml (spec.ip) stays valid.
Operator code is unchanged between 0.10.6 and 0.10.7 (0.10.7 commits are all server/docs) → bumping the chart to 0.10.7 ships the same operator fix as 0.10.6, on the same tag as the server.

File structure

File	Change	Responsibility
`ansible/roles/router-hosts/defaults/main.yml`	Modify line 27: `router_hosts_version: "0.10.5"` → `"0.10.7"`	Server image tag deployed to the Firewalla
`argocd/cluster-app/templates/router-hosts-operator.yaml`	Modify line 31: `targetRevision: 0.10.2` → `0.10.7`; refresh the version comment (lines 27-30)	Operator Helm chart version (and its pinned operator image)
`docs/operations/router-hosts.md`	Append an “Aggregate compaction (0.10.7+)” runbook subsection	Operational reference for the new `compact` command

No Vault, ExternalSecret, or values.yaml changes (schema confirmed identical).

Task 1: Edit both version pins (single PR, not yet merged)

Files:

Modify: ansible/roles/router-hosts/defaults/main.yml:27
Modify: argocd/cluster-app/templates/router-hosts-operator.yaml:27-31

Both pins move to 0.10.7 in one commit; the PR is opened but not merged until Task 4.

Step 1: Bump the server version pin

In ansible/roles/router-hosts/defaults/main.yml, change line 27:

# before
router_hosts_version: "0.10.5"
# after
router_hosts_version: "0.10.7"

Update the adjacent comment (lines 21-26) to add a line noting 0.10.6 (commit-on-timeout rollback + snapshots.trigger_type repair, rh#332) and 0.10.7 (aggregate compaction CLI, rh#336/#337).

Step 2: Bump the operator chart pin

In argocd/cluster-app/templates/router-hosts-operator.yaml, change line 31:

# before
      targetRevision: 0.10.2
# after
      targetRevision: 0.10.7

Replace the comment block (lines 27-30) so it documents the wedge fix, e.g.:

      # 0.10.7 = Go operator. Resolves the 0.10.2 publish wedge (incident hl-37e4):
      # 0.10.6 #332 made reconcile idempotent on AlreadyExists via host adoption
      # (rh#313) so the operator learns the real OCC version instead of sending
      # expected_version=0 (rh#323). Server must be on >=0.10.6 AND the bloated
      # llm-gw aggregate compacted (server 0.10.7 `compact`) BEFORE this merges —
      # see docs/engineering/plans/2026-06-26-router-hosts-0.10.7-upgrade.md.
      targetRevision: 0.10.7

Step 3: Lint the changed files

Run: yamllint ansible/roles/router-hosts/defaults/main.yml argocd/cluster-app/templates/router-hosts-operator.yaml Expected: no errors (matches .yamllint.yaml).

Step 4: Render the operator Application to confirm the chart resolves at 0.10.7

Run: helm template rh oci://ghcr.io/fzymgc-house/charts/router-hosts-operator --version 0.10.7 -f argocd/app-configs/router-hosts-operator/values.yaml | rg -n "image:|router-hosts.fzymgc.house|kind: Deployment" | head Expected: a Deployment referencing ghcr.io/fzymgc-house/router-hosts at a 0.10.7-aligned tag/digest, no template errors.

Step 5: Append the compaction runbook to docs/operations/router-hosts.md

Add an “Aggregate compaction (0.10.7+)” subsection so the Task 3 procedure is captured for future operators (this lands in the same PR as the pins):

### Aggregate compaction (0.10.7+)

The event-sourced server can grow a single host's event log unboundedly when writes
repeatedly time out. 0.10.7 adds compaction, which folds a bloated log to one
`HostCompacted` seed event at the **preserved high-water version** (so
`host update --version <V>` still succeeds).

- Preview (non-mutating): `router-hosts compact --over 1000 --dry-run`
- Compact every aggregate over N events: `router-hosts compact --over 1000`
- Compact one aggregate: `router-hosts compact <aggregate-id>`

`--over <= 0` is rejected with `InvalidArgument`. Do **not** retry a write on
`DeadlineExceeded` — re-run compaction instead. Monitor the gauges
`router_hosts_aggregate_events_max` and `router_hosts_aggregates_over_threshold`.

Step 6: Commit on a feature branch (do NOT merge yet)

Commit with the commit-commands:commit skill (jj). Suggested subject: feat(router-hosts): upgrade server+operator 0.10.x → 0.10.7 (staged) [hl-fq4n] Push the branch and open a PR, but leave it open — merging triggers the operator rollout and must happen in Task 4.

Task 2: Upgrade the server from the PR branch (server-first)

Files: none (deploy action). Runs against the Firewalla; requires the router-hosts mTLS CLI certs and Ansible inventory access.

Step 1: Capture pre-upgrade state

ssh pi@192.168.20.1 "systemctl status docker-compose@router-hosts --no-pager | head -5"
router-hosts host list | tee /tmp/rh-hosts-before.txt
router-hosts host get llm-gw            # note the current version (high-water mark)
dig @192.168.20.1 llm-gw.fzymgc.house +short
dig @192.168.20.1 openrouter-gw.fzymgc.house +short   # expect 192.168.20.155 (manually remediated)

Expected: server active (running); host list returns the operator-managed records; llm-gw resolves to 192.168.20.155.

Step 2: Snapshot the server data dir (rollback safety)

The 0.10.6 v4 migration is additive/idempotent (snapshots.trigger_type) and 0.10.5→0.10.7 stays on the Go line (no Rust→Go DB migration like 0.10.0), but snapshot anyway:

ssh pi@192.168.20.1 "sudo tar czf /extdata/router-hosts-backup-$(date +%Y%m%d-%H%M%S).tgz -C /extdata router-hosts && ls -lh /extdata/router-hosts-backup-*.tgz"

Expected: a .tgz written (DB + config + certs). Record its path for rollback.

Step 3: Deploy the server from the PR branch checkout

From the ansible/ directory of the repo checked out at the PR branch (so router_hosts_version reads 0.10.7; the playbook and inventory/hosts.yml paths are relative to ansible/):

Run: cd ansible && ansible-playbook -i inventory/hosts.yml router-playbook.yml --tags router-hosts Expected: changed on the router-hosts container task; play ends failed=0.

Step 4: Verify the server is healthy on 0.10.7

ssh pi@192.168.20.1 "docker inspect --format '{{.Config.Image}}' \$(docker ps -qf name=router-hosts) "   # ...:0.10.7
ssh pi@192.168.20.1 "systemctl status docker-compose@router-hosts --no-pager | head -5"
router-hosts host list | diff -u /tmp/rh-hosts-before.txt -    # expect no host loss

Expected: image tag 0.10.7; service active (running); host set unchanged (0.10.5 startup-regeneration already keeps the dnsmasq conf current).

Task 3: Compact the bloated llm-gw aggregate (server 0.10.7)

Files: none (CLI action against the server). This is the step that actually clears the wedge — it shrinks the ~91k-event llm-gw aggregate so a correct-version write finishes inside the 30s deadline.

Step 1: Dry-run to preview what compaction would reclaim

Run: router-hosts compact --over 1000 --dry-run Expected (0.10.7 #337 makes dry-run informative): exactly one aggregate listed (the llm-gw host), projecting ~91178 → 1 (total events reclaimed: ~91177). No other aggregate exceeds 1000 events. If more than one appears, stop and investigate before the real run.

--over 1000 isolates the runaway: normally-reconciled hosts carry well under 1000 events. The --over <= 0 footgun is rejected with InvalidArgument (0.10.7 #337).

Step 2: Run the real compaction

Run: router-hosts compact --over 1000 Expected: success; the llm-gw aggregate folded to a single HostCompacted seed event at the preserved high-water version.

Step 3: Verify state preserved and aggregate shrunk

router-hosts host get llm-gw          # same IP (.155) + aliases (mcp-gw, openrouter-gw); version preserved
router-hosts compact --over 1000 --dry-run   # expect: no aggregate over threshold (0 candidates)
dig @192.168.20.1 openrouter-gw.fzymgc.house +short   # still 192.168.20.155

Expected: llm-gw host data byte-identical to before; no aggregate over threshold; DNS unchanged. Do not retry on any DeadlineExceeded — each retry appends an event; with commit-on-timeout fixed it now rolls back, but re-running compaction (not the write) is the correct response.

Task 4: Merge the PR — operator rolls out via ArgoCD

Files: none (this is the main merge of Task 1’s PR). With the server on 0.10.7 and the aggregate compacted, the operator can now adopt hosts and write correct versions safely.

Step 1: Merge the PR to main

Merge via the normal PR flow. The app-of-apps reconciles router-hosts-operator.yaml, the operator Application picks up chart 0.10.7, and (auto-sync) rolls the operator Deployment.

Step 2: Confirm ArgoCD synced the operator

kubectl --context fzymgc-house get application router-hosts-operator -n argocd \
  -o jsonpath='{.status.sync.status}{" "}{.status.health.status}{" rev="}{.status.sync.revision}{"\n"}'
kubectl --context fzymgc-house get deploy -n router-hosts-operator -o wide

Expected: Synced Healthy; the operator pod image resolves to the 0.10.7 chart’s pinned digest.

Step 3: Confirm the operator logs are clean (wedge cleared)

kubectl --context fzymgc-house logs -n router-hosts-operator deploy/router-hosts-operator --since=5m \
  | rg -i "AlreadyExists|version conflict|expected 0|DeadlineExceeded|UpdateFailed" || echo "clean: none of the wedge signatures present"

Expected: clean: ... — none of the 0.10.2 failure signatures recur.

Task 5: End-to-end verification — a change publishes within one reconcile

Files: none (observation; optional git-driven canary).

Step 1: Confirm the previously-stuck HostMapping reconciles

kubectl --context fzymgc-house get hostmapping agentgateway-hosts -n agentgateway \
  -o jsonpath='{.status.phase}{" gen="}{.metadata.generation}{" observed="}{.status.observedGeneration}{"\n"}'
kubectl --context fzymgc-house describe hostmapping agentgateway-hosts -n agentgateway | rg -i "phase|reason|message|synced|fail"

Expected: phase: Synced, observedGeneration == generation, no UpdateFailed reason. This alone proves the adoption + correct-version write succeeded against the compacted aggregate.

Step 2 (stronger, optional): git-driven canary alias

Per repo policy (no direct kubectl apply; ArgoCD owns manifests), prove a new publish end-to-end by adding a throwaway alias through git: add rh-canary.fzymgc.house to spec.aliases in argocd/app-configs/agentgateway/hostmapping.yaml, open + merge a tiny PR, then:

# within ~1 reconcile of sync
dig @192.168.20.1 rh-canary.fzymgc.house +short    # expect 192.168.20.155

Expected: resolves to .155 within one reconcile. Then revert with a second tiny PR (remove the alias) and confirm it stops resolving. Skip if Step 1 + a real pending change (none currently) is sufficient confidence.

Step 3: Close out tracking

bd close hl-fq4n --reason "Upgraded server+operator to 0.10.7, compacted llm-gw aggregate, verified clean publish"
bd close hl-37e4 --reason "Wedge cleared by 0.10.7 upgrade + llm-gw compaction"   # or add a verification note and let the owner close
bd note hl-ls5 "Server reached 0.10.7 via hl-fq4n"   # close hl-ls5 only once its grpcurl-reflection doc-fix sub-task is also done

Rollback

Phase	Symptom	Rollback
Task 2 (server)	server container won’t start / loses hosts	restore the Task 2 Step 2 snapshot: `ssh pi@192.168.20.1 "sudo systemctl stop docker-compose@router-hosts && sudo rm -rf /extdata/router-hosts && sudo tar xzf /extdata/router-hosts-backup-<ts>.tgz -C /extdata && sudo systemctl start docker-compose@router-hosts"`; revert `router_hosts_version` to `0.10.5` on the branch and re-run the playbook. (Branch not yet merged → `main` still says 0.10.5, no drift to undo.)
Task 3 (compaction)	`host get llm-gw` shows wrong IP/aliases after compaction	restore the snapshot as above (compaction is in-DB); re-evaluate before retrying.
Task 4 (operator)	operator unhealthy or re-wedges after merge	revert the merge PR (`targetRevision` back to `0.10.2`); ArgoCD auto-syncs the operator back. Server stays on 0.10.7 (forward-compatible: proto is stable across the 0.10.x line).

Out of scope

hl-620y (auth.fzymgc.house leaks AAAA) — a ctrld/Firewalla per-domain resolver override, explicitly not a router-hosts bug. Untouched here.
The remaining #330 work (snapshot-accelerated rehydration / time-travel) — upstream router-hosts-eda.2, a separate design pass; not required to clear this wedge.

Open verification notes

New 0.10.7 server gauges (router_hosts_aggregate_events_max, router_hosts_aggregates_over_threshold) are worth scraping/alerting so a future runaway is caught early — follow-up, not a blocker for this upgrade.