CoE: PowerDNS Cutover Walked Forward, Rolled Back, and Re-run — 2026-08-16

Field	Value
Date	2026-08-16
Severity	Low (deliberate window; one client-visible name outage while the VIPs sat on the converted node)
Duration	Total exposure about 492 s
Impact	With the floating addresses on portland, vault.fzymgc.house stopped resolving from a MainWireless client and vault status failed with Could not resolve host — PowerDNS did not yet carry the Kubernetes service estate
Resolution	Rolled back by the same keepalived mechanism in reverse, external-dns was pointed at PowerDNS, and the cutover was re-run and kept the same day
Standing procedure	The PowerDNS authoritative tier

This is the record of the FIRST window, kept because the numbers in it are dated readings that must not be trusted again. The operational residue — re-take the record-set delta by querying, and the ordering rules the conversion bought — lives in the runbook, not here.

What was done, and what it read

On 2026-08-16 the cutover was walked forward on the production pair, measured, and rolled back by the same mechanism — then run again the same day and kept. What follows is the FIRST window, recorded as it was performed rather than as it was designed. It is history, not the estate’s current state: the table at the top of this section is the current state, and both nodes have served PowerDNS since.

The move itself used the existing procedure unchanged — Promotion — moving the floating addresses deliberately, further down this page. Nothing new was tied to keepalived:

Step	Act	Reading
1	verify on the NON-holder (portland), where a mistake costs nothing	keepalived-check-house.sh rc=0, pdnsutil zone check fzymgc.house rc=0
2	sudo systemctl stop keepalived on sandiego, the holder	issued 1786884277.577, portland held all five by 1786884278.280
3	sudo systemctl start keepalived on sandiego	returned as BACKUP holding zero, under nopreempt

The transition is bounded above by 0.703 s — a bound rather than a measurement, because the sampler’s first surviving sample already read five and the move was never caught in flight. A deliberate stop releases the addresses immediately, which is a different mechanism from the tracker-fault failover this page measures at 9.472 s; the two numbers are not comparable.

Why it was rolled back: PowerDNS does not carry the Kubernetes service estate. With the VIPs on portland, vault.fzymgc.house stopped resolving from a client on MainWireless and vault status failed with Could not resolve host: vault.fzymgc.house. This was observed, not predicted.

blocky’s fallbackUpstream: false is what makes that final. It is the guard that keeps .house queries on the box, and it is working exactly as the gate above proves. That is precisely why an absent name is an OUTAGE on this tier rather than a slower lookup somewhere else. There is no “somewhere else”.

The delta, comparing unbound-control list_local_data against pdnsutil zone list fzymgc.house, A records only:

Reading, 2026-08-16	during the cutover window	later the same day	after external-dns, 13:54Z
A names unbound serves	64	87	87
A names pdns serves	39	39	70
in unbound, ABSENT from pdns	42	44	13

No reading here is a constant, which is the point of printing three. The first two were taken hours apart on the same day by comparing config dumps and they disagree; the third was taken after external-dns began publishing and supersedes both. Every one of them was already stale by the time it was written down, so re-take this delta before converting a node rather than trusting any number on this page.

Measure it by querying, not by diffing config. The 64-vs-87 disagreement is an artifact of parsing unbound-control list_local_data output; the honest instrument asks the serving listener what it actually answers, because that is what a client gets:

# on the pdns node, for each name unbound serves:
dig @127.0.0.1 -p 1053 <name> A +short   # empty output == the name is dark

As of the last reading, 11 of the 13 are stale — records the old operator created and never deleted, whose namespaces no longer exist (alloy-ingest, grafana, logs, mem-b, mem-c and their .k8s siblings, merlin, ntfy, office-printer). --policy=sync with a TXT registry is what stops external-dns accumulating the same debris. Verify staleness the same way — by asking the cluster, not by reading a list:

kubectl get ingressroute,ingressroutetcp,ingress,httproute -A -o json | grep -c '<name>.fzymgc.house'

Note the backticks in Traefik’s Host(...) match rules: a pattern anchored on double quotes matches Ingress objects but silently misses every IngressRoute, and reports a live name as stale.

Only two of the 13 had a live route, and both are now closed. argocd.fzymgc.house was closed by adding the ingress source to external-dns. rd-admin.fzymgc.house was closed by deleting the workload it pointed at rather than by publishing it: the in-cluster rustdesk namespace was a remnant of an earlier deployment attempt — production RustDesk Pro runs on the DigitalOcean edge node via Docker Compose (docs/operations/rustdesk-clients.md), not in k3s. The namespace was removed 2026-08-16 by operator decision; router-hosts-operator logged deleting host entry for deleted IngressRoute within seconds and dns-zone-sync carried the removal to the resolvers on its next tick.

As of that removal the delta is 11 absent of 65, and every one of the 11 is stale — verified by dumping every IngressRoute, IngressRouteTCP, Ingress, HTTPRoute and Service in the cluster and finding zero references to any of them. No live name is served by unbound and missing from pdns. That was the standing blocker on converting the second node; re-take the delta rather than trusting this paragraph, but the blocker as measured is gone.

One caution the removal exposed: docs/reference/services.md described that namespace as containing “no active workloads” while it held a pod with 96 days of uptime. A namespace being a remnant and being empty are different claims, and only the second one is safe to act on without looking.

The rollback used the same mechanism in reverse — sudo systemctl stop keepalived on portland, then start it again — and the estate was back on sandiego/unbound. Total exposure: about 492 s. Rollback was verified before it was trusted: sandiego’s keepalived-check-house.sh returned rc=0 and keepalived blocky unbound were all active before portland was told to release the addresses.

What this record is not

None of the counts above is a current reading. Every one was already stale when it was written down, which is the whole reason the runbook tells you to re-take the delta rather than quote a number from a page.