Authoritative Internal .house Resolver — Design

Design bead: hl-hqyv. Supersedes the mechanism of ADR hl-qra1 (per-name dnsmasq local=/address=), which itself refined ADR hl-2t4o. External dependency: a new router-hosts feature (see router-hosts feature contract).

Goal

Make the Firewalla’s native unbound the codified, self-maintaining authoritative resolver for fzymgc.house, so that:

1. Every internal name answers with its internal IP for A queries and NODATA for every other record type — closing the class of split-horizon RR-type leaks that has now recurred three times (AAAA: hl-zh3w/hl-qra1; HTTPS/type-65 ECH: hl-hqyv, which broke id.fzymgc.house in Chromium and blocked all mcp-gw OAuth).
2. The zone regenerates automatically from a single declarative inventory (the router-hosts store, managed by tf/router-hosts) — no hand-applied config, no dual-source merges.
3. Zone changes activate without human intervention (unbound reload on file change).

Everything currently live was hand-applied during the 2026-07-07 incident and exists only on the box. This design moves it into Git.

Current state (hand-applied, works, uncodified)

• Firewalla is in Unbound mode (native unbound v1.19.1, all devices): dnsmasq intercepts client DNS and forwards to unbound at 127.0.0.1:8953.
• /home/pi/.firewalla/config/unbound_local/ (auto-included by unbound.template.conf, the supported Firewalla seam) contains two hand-written files:
  • 10-fzymgc-authoritative.conf — ~60 per-name local-zone: "<name>." static + local-data: "<name>. 300 IN A <ip>" pairs, manually merged from the router-hosts hosts file plus Firewalla-app “Custom DNS” rules.
  • 20-controld-forward.conf — forward-zone "." to Control D over DoT (forward-tls-upstream yes; 76.76.2.22@853#mt3vg25yld.dns.controld.com + IPv6 equivalent; system CA bundle).
• The dnsmasq per-name overlay from hl-qra1 is still deployed (dnsmasq_conf_path → dnsmasq_local/router-hosts.conf, plus the hostsdir cutover) — now redundant duplicate authority in front of unbound.
• Known gaps found during the incident:
  • k8s-cluster.fzymgc.house (the break-glass admin kubeconfig endpoint) is NXDOMAIN. Mechanism: it exists in tf/router-hosts/hosts.tf only as a bare alias (k8s-cluster) on the k8s_api entry, and router-hosts name outputs emit stored names verbatim (no domain suffixing) — bare aliases never render as FQDN records. The Firewalla Custom-DNS layer never carried it either.
  • Host changes do not regenerate the unbound zone (router-hosts has no unbound output). (An earlier observation that the on_success hook script was “missing on the box” turned out to be an artifact of the 2026-07-07 extdata-mount outage — see hl-tl2k; the role deploys it correctly.)
  • No reload mechanism exists for either dnsmasq conf changes or unbound zone changes.

Decisions

#	Decision	Rationale
D1	router-hosts grows a native unbound_conf_path output (new feature in fzymgc-house/router-hosts), rather than Ansible/Jinja generation in this repo	The generator machinery already exists once, in Go, with tests: dnsmasqconf.go (110 lines) regenerates on startup + on every host change, and it took two point releases (0.10.4 added the output, 0.10.5 fixed regenerate-on-startup — rh#326–#328) to harden those semantics. A shell/Jinja re-implementation would re-earn those scars and still couldn’t regenerate between deploys.
D2	Activation decouples from generation: a host-side systemd .path unit watches unbound_local/ and runs unbound-control reload	The router-hosts [[hooks.on_success]] command runs inside the container — it can never systemctl and would need control certs + binary to speak to unbound. A host watcher reloads correctly no matter who writes the file (daemon at runtime, Ansible at deploy).
D3	The router-hosts store becomes the complete .house inventory: import Firewalla Custom-DNS .house rules and add k8s-cluster.fzymgc.house via tf/router-hosts; retire the Custom-DNS .house entries	A generated zone is only as complete as its inventory. Dual sources caused the k8s-cluster gap. Matches the established rule that router-hosts must own *.fzymgc.house (NAS precedent, hl-eyq7/hl-uegs).
D4	Retire the dnsmasq per-name overlay (dnsmasq_conf_path, dnsmasq_local/router-hosts.conf, hostsdir cutover): dnsmasq becomes pure interceptor→unbound	One authority. Dual answers drift, and if unbound were ever bypassed the dnsmasq layer would silently reintroduce the type-65 leak (pre-2.90 dnsmasq forwards unknown RR types regardless of local=).
D5	Per-name local-zone … static, not typetransparent, not zone-wide static	Live-validated during the incident: typetransparent passes missing types to recursion (leaks); zone-wide static NXDOMAINs unmanaged siblings (the hl-2t4o regression). Per-name static answers listed types and NODATAs the rest.
D6	fzymgc.house only — .net is external-only, permanently	User convention (2026-07-07): .net must never carry internal records or an internal view. The bead description’s ”+ .net internal view” is stale.
D7	Control D forward config stays Ansible-templated static config (20-controld-forward.conf)	It has no relationship to the host inventory; deploy-time templating is the right cadence.
D8	Accept endpoint-level attribution at Control D; per-client DNS visibility moves local	Per-client attribution upstream is structurally incompatible with a shared caching forwarder: unbound’s global forward-zone cannot select per-client Control-D client-ID hostnames, and cache hits never reach Control D at all (so even ctrld-based attribution only ever covered cache misses). Control D’s activity log now shows one client (“Home-Firewalla”); profile-level filtering/policy is unaffected. Per-client “who queried what” is answered locally — Firewalla’s built-in per-device DNS insights today; optionally unbound query logging (log-queries/dnstap → OTel → ClickStack) as a follow-up bead if queryable history is wanted. Rejected: reinstating ctrld as first hop (reverses the retire decision, rebuilds the multi-daemon chain behind this incident, and behind dnsmasq interception it may not see real client IPs anyway); EDNS Client Subnet from unbound’s subnetcache module (unverified Control D support, stratifies the cache per client, untested with DoT — at most a research spike).
D9	k8s-cluster.fzymgc.house answers 192.168.20.140 (kube-vip API VIP), not the group_vars k8s_cluster_endpoint_ip of 10.255.254.6	Empirically verified 2026-07-07: 10.255.254.6 is unreachable from LAN clients and from the router itself (the router routes it out the WAN interface — no internal route exists); the tailscale subnet router advertises only 192.168.20.0/22; the k8s-fzymgc tailscale ACL entry for .6 traces to the dead Tailscale operator (hl-o89e). 192.168.20.140 is reachable from LAN (and tailnet via the subnet route), is in the apiserver SANs, and is what the actual break-glass used during the incident. The k8s_cluster_endpoint_ip var remains untouched for its surviving role (k3s-agent join URL, intra-cluster). Reconciling/retiring the 10.255.254.x convention repo-wide is a follow-up bead, not this work.

Architecture

tf/router-hosts  (declarative, complete .house inventory)
      │  terraform apply → gRPC/mTLS
      ▼
router-hosts daemon (docker on Firewalla)
      │  atomic writes; regenerates on startup + every host change
      ├─ /hosts/main                                  (hosts file output, kept)
      └─ /unbound/10-fzymgc-authoritative.conf        (NEW: unbound_conf_path)
              per name:  local-zone: "<name>." static
                         local-data: "<name>. 300 IN A <ip>"
      ▼
host systemd path unit  (watches /home/pi/.firewalla/config/unbound_local/)
      └─ oneshot service: unbound-control reload   (LD_LIBRARY_PATH=<unbound run dir>)
      ▼
unbound (native, all devices)
      ├─ fzymgc.house: authoritative, all RR types (NODATA for unlisted → no ECH/AAAA leak)
      └─ forward-zone ".": Control D DoT (20-controld-forward.conf, Ansible-templated)

dnsmasq remains the client-facing interceptor (Firewalla-managed) but carries no .house records of its own.

router-hosts feature contract

Filed as router-hosts#349; this repo’s changes gate on the released version.

• New server config option unbound_conf_path (string, optional). Validation: at least one of hosts_file_path, dnsmasq_conf_path, unbound_conf_path required.

• Output format, one entry per host record, exactly the live-validated shape:

  local-zone: "<fqdn>." static
  local-data: "<fqdn>. 300 IN A <ipv4>"

  IPv6 addresses emit IN AAAA. Multiple records for one name emit one local-zone line and N local-data lines.

• Same write semantics as dnsmasq_conf_path: atomic write (temp + rename), regenerate on startup (0.10.5 semantics, rh#327) and on every successful host mutation, participate in the existing [[hooks.on_success]]/on_failure flow.

• Name handling matches the dnsmasq generator: each entry’s hostname and each alias emit their own directive pair, verbatim as stored — no domain suffixing. Bare (non-FQDN) aliases therefore produce no useful zone record; the inventory must carry FQDNs for any name the zone must answer (this is exactly how the k8s-cluster gap happened, and the contract states it explicitly so it can’t happen silently again).

• Mirror of internal/server/dnsmasqconf.go with equivalent unit tests.

Changes in this repo

ansible/roles/router-hosts

• server.toml.j2: add unbound_conf_path = "/unbound/10-fzymgc-authoritative.conf"; remove dnsmasq_conf_path.
• docker-compose.yml.j2: mount {{ router_hosts_firewalla_config_dir }}/unbound_local → /unbound:rw.
• New template unbound-controld-forward.conf.j2 → unbound_local/20-controld-forward.conf (Control D endpoints/resolver-id as role vars; secrets not required — the resolver id is not sensitive).
• New host-side units (installed via role, with Firewalla boot persistence — same post_main.d re-assert pattern as the existing z0100-start-router-hosts.sh; exact persistence mechanics verified at plan time):
  • router-hosts-unbound-reload.path — PathChanged= on unbound_local/.
  • router-hosts-unbound-reload.service — oneshot unbound-control -c /home/pi/.firewalla/run/unbound/unbound.conf reload with Environment=LD_LIBRARY_PATH=/home/pi/.firewalla/run/unbound.
• The on-hosts-update.sh hook stays a logger (activation is the path unit’s job; the once-suspected “scripts drift” was the unmounted-extdata artifact, hl-tl2k).
• Remove the dnsmasq cutover: delete the router_hosts_enable_dnsmasq_cutover var, its tasks, the dnsmasq_local/local-hosts.conf template, and the reload dnsmasq handler usage that serviced them.
• Add a one-time cleanup task that removes the leftover dnsmasq_local/router-hosts.conf and dnsmasq_local/local-hosts.conf from the box (the retired overlay’s on-disk remnants), notifying a dnsmasq reload.
• verify.yml additions: zone file exists and non-empty; unbound-control status healthy; on-router dig @127.0.0.1 -p 8953 checks (a managed name → internal A; same name TYPE65 → NODATA; an external name resolves).
• Version gate: router_hosts_version bumped to the release carrying unbound_conf_path; preflight asserts the running image satisfies it.

tf/router-hosts

• Promote k8s-cluster from bare alias to FQDN on the existing k8s_api entry: aliases = ["k8s-api", "k8s-cluster.fzymgc.house"] (the module convention already documents FQDN aliases; the entry’s IP stays 192.168.20.140 per D9 — no new map key, no collision).
• Import the Firewalla-app Custom-DNS .house entries as declarative host records (enumerated at plan time from the Firewalla app alongside the hand-applied zone file). Watch for the AlreadyExists trap from hl-uegs.1: entries manually present in the store need router-hosts host delete <ulid> before the TF apply, or TF import.

Manual runbook (not automatable)

After the TF import is applied and verified: delete the Custom-DNS .house rules in the Firewalla app (router-hosts/CLI cannot touch Firewalla-native entries — hl-uegs precedent), then re-verify resolution for each migrated name.

Docs

• Update the DNS/resolver description in docs/reference/network.md (dnsmasq-overlay → unbound-authoritative), including the one-line correction that MetalLB lives in namespace metallb (not metallb-system).
• Operations notes: unbound reload/validation commands (unbound-control env quirk, throwaway-instance validation on port 18953 — no unbound-checkconf on the box).
• ADR: supersede hl-qra1 via capture-adrs after this spec is READY.

Failure modes

Failure	Behavior
router-hosts write fails	on_failure hook logs; previous zone file intact (atomic writes) — unbound keeps serving last-good data (fail-static).
unbound-control reload fails	Oneshot service fails visibly in journald; unbound continues serving the previously loaded zone. Stale-but-valid beats broken.
unbound down / mode toggled off	dnsmasq forwards .house upstream → public answers return (ECH exposure for proxied names, NXDOMAIN for private-only names). Unchanged platform risk, now visible rather than silently masked by the retired dnsmasq overlay. Monitoring for it is out of scope (candidate follow-up bead).
Name missing from inventory	NXDOMAIN, as today — mitigated by D3 (single reviewed inventory) and the PR-reviewed TF workflow.
Firewalla firmware update resets units	post_main.d re-assert restores the path/service units at boot (same persistence pattern the role already uses for the container).

Rollout & rollback

1. rh feature ships first (separate repo/release; this repo’s role bumps router_hosts_version only after release).
2. TF inventory import (k8s-cluster + Custom-DNS names) — additive, safe before the role change.
3. Ansible role apply — rh takes over writing the same file path the hand-applied config lives at (10-fzymgc-authoritative.conf), so first regeneration simply overwrites it: idempotent takeover, no cutover window. The Control D forward file is overwritten byte-compatibly by the template.
4. Retirement: dnsmasq overlay removal (same role apply) + manual Custom-DNS cleanup.
5. Rollback: disable the path unit and redeploy the previous role version — the last-written zone file keeps serving; the hand-applied file is preserved in the bead trail if full manual restore is ever needed.

Verification

• rh repo: unit tests for the generator (format, IPv6, multi-record, empty-store, atomic write) mirroring the dnsmasq generator’s suite.
• Role verify.yml: as above (file, unbound-control status, on-router digs).
• Client-side runbook (from a real client, not the box — they take different DNS paths):
  • dig id.fzymgc.house A +short → 192.168.20.145; same for mcp-gw → .155, k8s-cluster → .140.
  • dig id.fzymgc.house TYPE65 +short → empty, status NOERROR.
  • An external name resolves (Control D DoT path); curl https://mcp-gw.fzymgc.house/mcp/engram → HTTP 401 (reachable).
  • Break-glass kubeconfig works without --server overrides.

Out of scope

• .net anything (external-only domain, permanently — D6).
• Unbound availability monitoring/alerting (candidate follow-up bead).
• Unbound per-client query logging into ClickStack (candidate follow-up bead — see D8; only if Firewalla’s built-in per-device DNS insights prove insufficient).
• Reconciling/retiring the 10.255.254.x endpoint convention (k8s_cluster_endpoint_ip in group_vars, the stale k8s-fzymgc tailscale ACL entry) — candidate follow-up bead; see D9.
• The Keycloak/OIDC and Little Snitch incident remediations (complete; documented on hl-hqyv).
• Tailscale MagicDNS path: unchanged by this design (unbound sits behind dnsmasq exactly where ctrld/Control D did; the hl-2t4o failure mode — zone-wide NXDOMAIN breaking MagicDNS names — is specifically avoided by D5’s per-name scoping). Called out here so reviewers check the reasoning, not because work is planned.