Make the Resolver Pair Actually Serve Guest and Telework

Date: 2026-08-31 Status: ready to execute Trigger: Telework DNS outage, 2026-08-31 ~09:15–11:0x local, ended by rolling Telework’s DHCPv4 Primary DNS Server back to 192.168.217.1.

Fix forward. Guest stays pointed at 192.168.79.5 and stays broken until this lands — that is a deliberate choice, and it buys a real canary (see Guest is the canary).

What is actually wrong

Phase 03 cut two production VLANs onto resolver addresses that exist on no machine in the estate. Two independent blockers, stacked. Fixing either alone reproduces the outage.

Blocker 1 — the VLAN legs were never created

Measured 2026-08-31 on both nodes:

eth0.3100    ABSENT      (not down — absent from /sys/class/net)
eth0.3200    ABSENT
eth0.3000    oper=up  rx_pkts=26038106     <- control, the legs that work

192.168.79.5/.6/.7 and 192.168.217.5/.6/.7 are configured nowhere. Phase 02 added the two dns_vlan_interfaces entries to host_vars/{portland,sandiego}.yml and committed them; the play was never run against the nodes afterwards.

roles/dns-common/tasks/network.yml iterates dns_vlan_interfaces with no gate — it splits untagged (id is none) from tagged and renders both. So the config is correct and complete. It has simply never been applied.

Corroborated from the router: ARP for all six addresses is INCOMPLETE on bond0.3100 and bond0.3200, while other hosts on those same segments (192.168.79.248, .75, 192.168.217.41) resolve normally. The segments are healthy; the pair is not on them.

Blocker 2 — nftables does not admit the new legs

The live ruleset on both nodes names exactly five legs:

iifname { "eth0", "eth0.1000", "eth0.3000", "eth0.3020", "eth0.3001" } ... dport 53 counter accept

eth0.3100 and eth0.3200: referenced 0 times. The accept lists derive from dns_vlan_interfaces too, so this ruleset also predates Phase 02. Chain policy is drop.

Both nodes agree exactly, which rules out a single-node fluke.

So creating the interfaces alone is not enough. With policy drop and the leg unnamed, DNS arriving on eth0.3200 is discarded, and the client sees precisely the same total failure.

Blocker 3 (latent) — the drop path is silent

log statements: 0

Neither node logs a dropped packet. A packet arriving on an unadmitted leg dies with no counter, no journal line, and nothing to find afterwards. This is why the outage produced no evidence on the nodes and had to be diagnosed entirely from the router and from off-box probes.

Why nothing caught it

Phase 03’s D-01 ruled all nine ROADMAP success criteria out, on the discriminator “ask what breaks if the action doesn’t take — if the answer is the status quo, no instrument earns its place.”

That discriminator is sound for the rollback direction and silent on the other one: what breaks if the action DOES take and the target cannot be reached? For a DNS cut that is a total outage on the segment, which is what happened.

Criterion 4 — a reading from the pair’s own untrusted vantage — was recorded as available and declined. One dig would have failed instantly.

The measurement trap that hid it afterwards

During triage, a probe from a trusted subnet to 192.168.217.5 returned NOERROR with real answers for example.com, and NODATA for *.fzymgc.house. That looked exactly like Blocky’s per-client view withholding the internal namespace — a coherent, wrong story.

Positive control that broke it:

192.168.217.5     status: NOERROR  ANSWER: 2
192.168.217.99    status: NOERROR  ANSWER: 2     <- address exists nowhere
192.168.217.222   status: NOERROR  ANSWER: 2     <- address exists nowhere

The Firewalla intercepts DNS to any address on a subnet it manages. From off-segment, every address on that subnet looks like a healthy DNS server. The probe never reached the pair.

Record this as a standing rule: a reachability probe for an on-segment service must be run from that segment. Off-segment it does not merely lose fidelity here — it fabricates a positive.

Guest is the canary

Guest is currently pointed at 192.168.79.5 and broken. Leaving it there is deliberate:

Once the pair genuinely serves VLAN 3100, Guest starts resolving with no appliance change at all. That gives an end-to-end, on-segment, real-client signal that the fix works — obtained before Telework is touched a second time, and without spending another Telework window to get it.

If Guest does not recover after both nodes converge, the fix is incomplete and Telework must not be re-cut.

The fix

ansible/dns-playbook.yml already encodes the constraint this work needs. Standing constraint C3 forbids changing both resolver nodes in the same window — the estate has exactly two and they are the whole DNS serving plane. The playbook’s per-role tagging exists to make C3 expressible on the command line:

scripts/dns-playbook.sh --limit <node> --tags dns-common

--limit reaches the node it names because inventory/hosts.yml gives each host its own Core address (.6/.7) as ansible_host, never the .5 VIP.

sandiego currently holds the VIPs (.5 present on eth0, eth0.3000, eth0.1000); portland is backup. Converge the backup first.

T1 — add drop logging before anything else

Blocker 3 is fixed first on purpose: it is the instrument every later step depends on. Without it, a failure in T2–T5 is as silent as the original outage.

Add a log prefix on the drop path in roles/dns-common/templates/nftables-dns.conf.j2, rate-limited so a hostile segment cannot flood the journal. Guest is a hostile segment by definition, so the limit is load-bearing, not decoration.

Gate: the rendered ruleset carries a log statement, and the render gate’s assertion count is unchanged otherwise.

T2 — converge portland (backup)

scripts/dns-playbook.sh --limit portland --tags dns-common --check --diff   # read the diff first
scripts/dns-playbook.sh --limit portland --tags dns-common

The --check --diff pass is not ceremony. This role converges more than the two legs — resolv.conf, the RA watch/deprecate units, multi-homing kernel settings, the nftables reload. Read what it intends to change on a live serving node before letting it.

Gates, all on portland:

ip -br addr show | grep -E '3100|3200' → both present, UP, correct addresses
sudo nft list ruleset | grep -c 'eth0.3100' → non-zero; same for eth0.3200
the four dport 53 accept rules name seven legs, not five
blocky and keepalived still active; the Main/Core/Lab legs still answer

T3 — verify from the segment, not from a trusted subnet

This is the step Phase 03 declined, and the one that would have prevented the outage. Run it from the router, which is on-link for both VLANs:

ssh router 'ping -c2 -W2 192.168.79.7 ; ip neigh show 192.168.79.7'      # portland's leg address
ssh router 'dig +time=3 @192.168.79.7 -b 192.168.79.1 example.com A'

Target portland’s node address (.7), not the .5 VIP — the VIP does not exist until keepalived brings it up, and at this point only one node is converged.

ip neigh must read REACHABLE, not INCOMPLETE. That single field is the whole finding of this plan; assert it explicitly rather than inferring it from a successful dig.

Positive control required. A dig against any address on that subnet will answer, because the Firewalla intercepts. So the control is: query an address that certainly does not exist (192.168.79.99) and confirm the two answers differ in a way only a real server explains — the node answers authoritatively for *.fzymgc.house, the interceptor returns NODATA. Without that control this step re-runs the exact trap described above and returns a false pass.

T4 — converge sandiego (VIP holder), then confirm the VIPs

Only after T3 passes.

scripts/dns-playbook.sh --limit sandiego --tags dns-common --check --diff
scripts/dns-playbook.sh --limit sandiego --tags dns-common

Then confirm 192.168.79.5 and 192.168.217.5 come up on whichever node holds them, and that a keepalived failover moves them. The VIP is what the appliance points at; a pair that serves only its node addresses is not finished.

T5 — Guest recovers on its own

No appliance change. Confirm a real Guest client resolves. If it does not, stop — Telework is not re-cut until it does.

T6 — re-cut Telework

Only after T5. Same one-field change as 03-02, same read-back-off-the-reopened-Edit-view discipline, which worked correctly and needs no revision.

Add the one gate Phase 03 lacked: immediately after Save, confirm the dport 53 accept rule counter for eth0.3200 is non-zero and rising. That distinguishes “the field changed” from “the segment can resolve” — the exact gap that made this outage possible, measured at the layer where it failed.

Record corrections owed

docs/operations/dns.md and docs/reference/network.md currently assert that both untrusted legs take their resolver from the pair. That is false for Guest (pointed at a dead address) and false for Telework (rolled back). Corrections follow this page’s existing convention — name the superseded text, date it, state why it is now false. Do not delete and do not silently reword.

The per-leg pre-cut records written by 03-01 and 03-02 stay exactly as they are. They are accurate, they are what made the rollback safe, and they are the part of Phase 03 that worked.

Not in scope

Re-litigating Phase 03’s verdict or its requirement completions.
Rolling Guest back.
The Secondary DNS Server ABSENT-vs-blank question (todos/pending/2026-08-31-re-read-iot-and-core-secondary-dns-field.md).