Skip to content
fzymgc-house Cluster Docs

CoE: WAN Resolver Failure Tore Down Internal DNS Redirect

Field Value
Date 2026-07-23
Severity Medium (all internal fzymgc.house names unresolvable for LAN and tailnet clients; external DNS degraded to ~50% timeouts)
Duration ≥4 h (onset at or before 2026-07-24 00:44 UTC — exact onset unrecoverable, see Latent Findings — resolved 2026-07-24 04:47 UTC)
Impact Internal names (vault, engram, id, grafana, ….fzymgc.house) returned public-zone NODATA or timed out; anything keyed on internal DNS (MCP gateway, Vault CLI, cluster ingress by name) failed
Resolution Replaced the dead statically configured eth1 WAN nameserver (1.252.0.14) with 9.9.9.9 / 1.1.1.1; FireMain re-added the :53 redirect rules automatically
Follow-ups #1717 (DNS answer probe), #1718 (router boot-ordering/telemetry hardening)

Correction 2026-07-30: one conclusion drawn from this incident was later found to be wrong. The client path does include a dnsmasq front tier — firerouter_dns, bound to each VLAN gateway address — and it was healthy throughout. What FireMain tore down was the :53 redirect in front of that tier, not dnsmasq itself. This changes nothing about the timeline, the root cause, or the remediation recorded below; it changes only the architecture description that was drawn from them. Corrected chain: Internal DNS.

Summary

Section titled “Summary”

The Firewalla health-gates the entire client DNS path — including the internal authoritative .house zone served by unbound — on plain-UDP reachability of the WAN-configured resolvers. When the statically configured primary WAN nameserver 1.252.0.14 (provenance unknown; not Control D, not Quad9 primary) stopped answering, FireMain’s upstream check (dig @1.252.0.14 firewalla.encipher.io) failed, and it removed the FW_PREROUTING_DNS_DEFAULT nat redirect rules that steer client :53 traffic into unbound.

With the redirect gone, Firewalla’s FW_PREROUTING_DNS_FALLBACK DNAT took over, round-robining client queries between the dead resolver (timeouts) and Quad9 (public answers). Internal-only names exist only in unbound’s generated zone, so public resolution returned NOERROR with zero answers — resolution “worked” while every internal name silently vanished. unbound itself was healthy and authoritative throughout; the actual resolution path (unbound → Control D over DoT) never uses the WAN resolvers that failed the health check.

The recurring-incident feel (“DNS is broken again”) is a smell worth naming: the 2026-07-16 and 2026-07-21 incidents were tailscale-layer failures with the same user-visible symptom (internal names dead). This one was router-native. All three share one property — no monitor asserted that internal names return internal answers, so each was user-detected.

Timeline (UTC)

Section titled “Timeline (UTC)”
Time Event
2026-07-07 16:27 Router reboot. firemasq (Firewalla’s DNS dnsmasq) never starts this boot cycle — the live client path is iptables :53 redirect → unbound. fireonboot.service times out (Firewalla’s own sync_time/clean_log; unrelated — Latent Finding 2). vector/otel-collector compose services are not restored → router telemetry dark in ClickStack (Latent Finding 3)
unknown eth1’s statically configured primary WAN resolver 1.252.0.14 stops answering. Static pair was 1.252.0.14 + 149.112.112.112 (Quad9 secondary); how 1.252.0.14 got there is unknown
≤2026-07-24 00:44 FireMain upstream check fails → DNS of network <X> is unreachable, remove DNS redirect rules loop begins (~every 6 s, all networks); FW_PREROUTING_DNS_DEFAULT emptied. Client :53 falls to fallback DNAT: nth-packet round-robin between 1.252.0.14 (dead → timeouts) and 149.112.112.112 (Quad9 → public-zone NODATA for .house)
2026-07-24 04:26 On-router docker stack (tailscale, router-hosts, vault-unseal) restarted during initial triage; no effect (correctly — those services were not the fault)
2026-07-24 04:36 Incident reported: engram.fzymgc.house / vault.fzymgc.house not resolving
2026-07-24 04:36–04:45 Hop-by-hop isolation: client NODATA carries the public Cloudflare SOA → not an unbound problem; unbound on 127.0.0.1:8953 answers internal names AND external names (Control D DoT healthy); FW_PREROUTING_DNS_DEFAULT found empty; dig @1.252.0.14 times out; eth1 static nameservers identified via firerouter API
2026-07-24 04:47 Operator replaced eth1 WAN DNS with 9.9.9.9 / 1.1.1.1 (Firewalla app). FireMain’s next check passed and it re-added the redirect rules automatically — no service restarts needed. Last error logged 04:48:03
2026-07-24 04:49 Client verification green: internal names → 192.168.20.x via the original failing path (tailnet MagicDNS split-DNS → 192.168.20.1 → unbound); TYPE65 NODATA guard intact; external resolution normal
2026-07-24 04:55 Collateral repair: vector + otel-collector compose services restored on the router (dark since the Jul 7 boot)

Root Cause

Section titled “Root Cause”

1. Trigger: a dead, statically configured WAN resolver

Section titled “1. Trigger: a dead, statically configured WAN resolver”

~/.router interface config for eth1 carried static nameservers: ["1.252.0.14", "149.112.112.112"]. The primary stopped answering UDP/53 entirely. The IP appears in no repo file, is not a Control D address (76.76.2.22), and is not Quad9 primary (9.9.9.9) — it may have been a typo’d or long-revoked entry. Nothing in the actual resolution path uses it.

2. Design: FireMain gates the whole client DNS path on WAN-resolver reachability

Section titled “2. Design: FireMain gates the whole client DNS path on WAN-resolver reachability”

FireMain’s Dnsmasq controller periodically resolves canary names against the WAN nameservers. On failure it declares each network’s DNS unreachable and removes the FW_PREROUTING_DNS_DEFAULT redirect rules — the only thing steering client :53 into unbound. The health check therefore tests a path clients never use (raw UDP to WAN resolvers) and, on failure, tears down the path clients do use (redirect → unbound → Control D over DoT, which was healthy the whole time).

3. The fallback converts a hard failure into wrong answers

Section titled “3. The fallback converts a hard failure into wrong answers”

FW_PREROUTING_DNS_FALLBACK DNATs client queries to the same WAN resolver set via nth-packet statistic rules. Result: ~half of all queries timed out (dead resolver), and the rest got authoritative-looking public NODATA for internal names. NOERROR/zero-answers looks “up” to anything that only checks that resolution succeeds — and looks like an unbound zone problem to a human, which is where triage time went first.

What Was Ruled Out

Section titled “What Was Ruled Out”
  • Tailscale split DNS / the #1714 subnet-route demotion — tailnet clients’ queries were reaching 192.168.20.1 fine; the split-DNS config (fzymgc.house → 192.168.20.1) is intact in tf/tailscale/dns.tf.
  • unbound / the router-hosts zone machinery — unbound answered every internal name and external names directly on 127.0.0.1:8953; the generated zone was current; the reload .path unit was active.
  • Control D DoT upstream — external resolution through unbound worked throughout (123 ms to github.com).
  • firemasq/dnsmasq config — firemasq has not run at all this boot cycle; it was not in the client path to break.

Latent Findings

Section titled “Latent Findings”
  1. Docs drift: docs/operations/dns.md and docs/reference/network.md described a dnsmasq interceptor tier in front of unbound. The live path this boot cycle is iptables redirect → unbound directly; firemasq.service is disabled and has never started since the Jul 7 boot. Corrected in this PR.

Correction 2026-07-30: this latent finding is where the error entered the docs. The “correction” it records replaced an accurate description with an inaccurate one: it read the legacy DNS unit’s inactive state as evidence that no dnsmasq tier existed. On a Firewalla Gold, platform/gold/GoldPlatform.js overrides the DNS and DHCP service names to firerouter_dns/firerouter_dhcp, so that legacy unit is inactive by design and its state is diagnostically worthless. The contradiction was already visible on the page being “corrected”: the same runbook told readers to run client verification from a real LAN client rather than from the box, because the box does not take the same dnsmasq → unbound path a client does — which only makes sense if a client-facing tier exists. Corrected chain: Internal DNS.

  1. fireonboot.service failed (timeout) at the Jul 7 boot — red herring for this incident. fireonboot.sh backgrounds Firewalla’s own clean_log.sh/sync_time.sh and waits (20 min cap); repo-managed boot scripts run from post_main.d via FireMain and were unaffected.
  2. Router telemetry dark since Jul 7: docker-compose@vector and docker-compose@otel-collector were dead with containers removed — the units are enabled but Requires=docker.service races Firewalla’s post_main.d/0050-start-docker.sh docker bring-up at boot, and the compose-up boot scripts evidently did not complete that boot. Router logs were absent from ClickStack during the incident, forcing on-box triage. Restored 2026-07-24; hardening tracked in #1718.
  3. FireMain log rotation under error loops is ~2 h (three ~1 MB files) — the teardown onset time was unrecoverable. If precise onset matters, the answer must come from ClickStack (which was dark — see 3).
  4. Detection gap: no monitor asserted that an internal name returns an internal answer. The failure mode returns NOERROR/NODATA, which passes any “does DNS resolve” check. Probe specified in #1717: a uptimekuma_monitor_dns check requiring a 192.168.20. answer for vault.fzymgc.house via 192.168.20.1.

Resolution and Durable Fix

Section titled “Resolution and Durable Fix”
  • Immediate: eth1 WAN DNS replaced with 9.9.9.9 / 1.1.1.1 (both live anycast). FireMain re-added the redirect rules on its next check cycle; client resolution verified end-to-end minutes later.
  • Why this is reasonably durable: the WAN resolvers are used only by Firewalla’s own health checks and the fallback DNAT — client resolution rides unbound → Control D over DoT. Two independent anycast providers make a simultaneous health-check failure (and thus another teardown) unlikely.
  • Detection: the DNS answer probe (#1717) will page cluster-edge when internal names stop returning VLAN-20 answers, whatever the cause — router-native (this incident) or tailscale-layer (the 2026-07-16/2026-07-21 class).

Lessons Learned

Section titled “Lessons Learned”
  1. Health checks must exercise the path clients use. FireMain’s check and the client path share zero components beyond the box itself; the check’s failure response destroyed a healthy path. When adding our own checks, probe through the front door (:53 on the VLAN gateway) — that is what the #1717 probe does.
  2. NOERROR/NODATA is the deceptive failure mode for split-horizon DNS. Internal names falling through to the public zone don’t error — they return empty success with the public SOA. The public-SOA-in-authority-section fingerprint immediately places the fault outside unbound; it’s the fastest triage discriminator for this architecture.
  3. Hand-entered infrastructure values need provenance. 1.252.0.14 sat as eth1’s primary DNS with no record of why. Anything typed into the Firewalla app is invisible to the repo; when it matters operationally (WAN DNS does), record it — this CoE now documents the intended pair (9.9.9.9/1.1.1.1).
  4. Telemetry gaps compound incidents. The router’s vector/otel pipeline had been dark for 16 days; nobody noticed because nothing asserted its liveness either. #1718 covers both boot-ordering hardening and a freshness check on router logs in ClickStack.