Skip to content
fzymgc-house Cluster Docs

Authoritative Internal fzymgc.house Zone — Implementation Plan

⚠️ Superseded (2026-06-19). This plan implemented zone-wide local=/fzymgc.house/, which was abandoned (hl-zh3w.2) and replaced by per-name local=+address= emitted by router-hosts >=0.10.4 (dnsmasq_conf_path, PR #1338). Current decision: ADR hl-qra1 (supersedes hl-2t4o). The Task 1 pre-flight audit and dig-based verification remain useful; the local=/<domain>/ mechanism in the tasks below is superseded.

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make the Firewalla’s dnsmasq authoritative for fzymgc.house so internal AAAA queries return NODATA instead of leaking public Cloudflare IPv6, fixing the dual-stack OIDC stall (kubectl / k8s-MCP).

Architecture: Add local=/{{ router_hosts_domain }}/ to the role-owned dnsmasq overlay template, and make the latent router-hosts-dnsmasq Ansible tag functional so apply/rollback are surgical. A pre-flight audit confirms no internal name NXDOMAINs under authoritative mode. Verification is dig-based (A unchanged, AAAA NODATA, unlisted NXDOMAIN) plus the real-world kubectl recovery.

Tech Stack: Ansible (router-hosts role), dnsmasq on Firewalla, jj VCS, rumdl markdown lint.

Spec: docs/engineering/specs/2026-06-18-authoritative-internal-fzymgc-house-zone-design.md · Bead: hl-zh3w

Task 1: Pre-flight audit + baseline capture (safety gate)

Section titled “Task 1: Pre-flight audit + baseline capture (safety gate)”

Confirm every public fzymgc.house name already has an internal record (no NXDOMAIN after the flip), and record the current leaking behavior as the baseline. No file changes — this is a gate. If a gap is found, STOP and add the missing internal record (operator annotation or tf/router-hosts/hosts.tf) before proceeding.

Files: none (read-only — live Cloudflare zone and ssh router:/extdata/router-hosts/hosts/main).

  • Step 1: Enumerate public fzymgc.house names

The documented public set (docs/reference/network.md → External DNS): vault auth grafana argocd mealie longhorn status. Confirm completeness against the live Cloudflare zone (catches any name added out-of-band):

Run:

Terminal window
cd tf/cloudflare && terraform state list 2>/dev/null | rg 'cloudflare_dns_record' \
  | while read -r r; do terraform state show "$r" 2>/dev/null \
      | rg -o 'name *= *"[^"]*"|hostname *= *"[^"]*fzymgc\.house"'; done | sort -u

Expected: the seven names above (HCP auth required; if unavailable, fall back to the documented set and note the gap inline).

  • Step 2: Diff public names against the live hostsdir

Run:

Terminal window
for h in vault auth grafana argocd mealie longhorn status; do
  printf '%-12s ' "$h"
  ssh router "grep -wq '${h}.fzymgc.house' /extdata/router-hosts/hosts/main \
    /home/pi/.firewalla/config/dnsmasq_local/00-bootstrap-vault.conf 2>/dev/null \
    && echo COVERED || echo '!!! GAP — add internal record before flip'"
done

Expected: every name COVERED. Any GAP → add the internal record first (see spec “Blast radius”), then re-run.

  • Step 3: Capture baseline (the bug) — A internal, AAAA leaking

Run (from an internal client, NOT via the /etc/hosts workaround — grep -v guards against a stale pin):

Terminal window
grep -q 'auth.fzymgc.house' /etc/hosts && echo "WARNING: /etc/hosts pin present — comment it out for an accurate baseline"
echo "A:";    dig +short A    auth.fzymgc.house @192.168.20.1
echo "AAAA:"; dig +short AAAA auth.fzymgc.house @192.168.20.1

Expected baseline: A = 192.168.20.145; AAAA = a 2606:4700:: Cloudflare address (the leak we will eliminate).

  • Step 4: No commit — audit only. Record the result in the bead.

Run: bd note hl-zh3w "Task1 audit: all 7 public names COVERED in hostsdir; baseline AAAA leak confirmed (2606:4700::)."

Task 2: Implement the two edits + dry-run

Section titled “Task 2: Implement the two edits + dry-run”

Files: Modify ansible/roles/router-hosts/templates/local-hosts.conf.j2 and ansible/roles/router-hosts/tasks/main.yml.

  • Step 1: Add the authority directive to the template

Edit ansible/roles/router-hosts/templates/local-hosts.conf.j2 to:

# {{ ansible_managed }}
# Point dnsmasq to router-hosts managed hosts directory
hostsdir={{ router_hosts_hosts_dir }}
expand-hosts
domain={{ router_hosts_domain }}
# Authoritative internal zone: answer fzymgc.house only from local data, never
# forward upstream. Makes AAAA for A-only names return NODATA instead of leaking
# the public Cloudflare AAAA. Unlisted names return NXDOMAIN (see design spec).
local=/{{ router_hosts_domain }}/
  • Step 2: Make the router-hosts-dnsmasq tag functional

In ansible/roles/router-hosts/tasks/main.yml, add router-hosts-dnsmasq to the Include configuration deployment block so the dynamic include is processed under that tag (today it carries only router-hosts/router-hosts-configure, so --tags router-hosts-dnsmasq is a no-op):

- name: Include configuration deployment
  ansible.builtin.include_tasks: configure.yml
  tags:
    - router-hosts
    - router-hosts-configure
    - router-hosts-dnsmasq

(Inside configure.yml, only the two dnsmasq tasks carry router-hosts-dnsmasq, so this tag now selects exactly those two for apply/rollback.)

  • Step 3: Syntax check

Run:

Terminal window
cd ansible && ansible-playbook -i inventory/hosts.yml router-playbook.yml --syntax-check

Expected: playbook: router-playbook.yml with no errors.

  • Step 4: Dry-run the dnsmasq-only converge (no changes applied)

Run:

Terminal window
cd ansible && ansible-playbook -i inventory/hosts.yml router-playbook.yml \
  --tags router-hosts-dnsmasq --check --diff --limit router

Expected: the diff shows local=/fzymgc.house/ being added to /home/pi/.firewalla/config/dnsmasq_local/local-hosts.conf, the two dnsmasq tasks reported changed, and no other tasks run (confirms the tag is now selective). If unrelated tasks run, the tag wiring is wrong — fix before apply.

  • Step 5: Commit

Run:

Terminal window
jj --no-pager commit -m "fix(dns): make Firewalla authoritative for fzymgc.house (local=) to stop AAAA leak [hl-zh3w]"

Task 3: Apply to the router + verify

Section titled “Task 3: Apply to the router + verify”

Production change to the live Firewalla. Apply deliberately.

Files: deploys /home/pi/.firewalla/config/dnsmasq_local/local-hosts.conf on the router (via the role; no repo files change).

  • Step 1: Apply the dnsmasq-only converge

Run:

Terminal window
cd ansible && ansible-playbook -i inventory/hosts.yml router-playbook.yml \
  --tags router-hosts-dnsmasq --limit router

Expected: PLAY RECAP failed=0; the reload dnsmasq handler fires.

  • Step 2: Confirm the deployed config

Run:

Terminal window
ssh router "grep -n 'local=' /home/pi/.firewalla/config/dnsmasq_local/local-hosts.conf"

Expected: local=/fzymgc.house/ present.

  • Step 3: Verify the fix — A unchanged, AAAA NODATA, unlisted NXDOMAIN

Run:

Terminal window
echo "A (want 192.168.20.145):"; dig +short A    auth.fzymgc.house @192.168.20.1
echo "AAAA (want EMPTY):";       dig +short AAAA auth.fzymgc.house @192.168.20.1
echo "AAAA status (want NOERROR/NODATA, NOT a 2606: answer):"
dig AAAA auth.fzymgc.house @192.168.20.1 | rg -i 'status:|2606:'
echo "Unlisted (want NXDOMAIN):"
dig A nonexistent.fzymgc.house @192.168.20.1 | rg -i 'status:'
echo "Spot-check still-internal A:"
for h in grafana.fzymgc.house vault.fzymgc.house whoami.k8s.fzymgc.house loki.fzymgc.house; do
  printf '%-28s %s\n' "$h" "$(dig +short A "$h" @192.168.20.1)"
done

Expected: A = 192.168.20.145; AAAA empty with status: NOERROR (NODATA) and no 2606: line; nonexistent → status: NXDOMAIN; all spot-check names return their internal A.

  • Step 4: Verify real-world recovery

Run (remove the temporary /etc/hosts pin first if present):

Terminal window
sudo sed -i '' '/auth.fzymgc.house/d' /etc/hosts 2>/dev/null || true
rm -f ~/.kube/cache/oidc-login/*
kubectl get nodes

Expected: oidc-login completes (no stall), node list returns. Then reconnect kubernetes-mcp-server and the gateway MCPs via /mcp.

  • Step 5: Record outcome (no commit — deploy verification)

Run: bd note hl-zh3w "Task3 applied + verified: AAAA NODATA, NXDOMAIN for unlisted, kubectl recovered."

Task 4: Update documentation

Section titled “Task 4: Update documentation”

Files: Modify docs/reference/network.md and docs/operations/router-hosts.md.

  • Step 1: Document the authoritative zone in the network reference

In docs/reference/network.md, in the DNS Configuration section, add after the Internal DNS Subdomain block:

### Internal Zone Authority


The Firewalla's dnsmasq is **authoritative** for `fzymgc.house`
(`local=/fzymgc.house/` in the `router-hosts` overlay). Internal queries are
answered only from local data (hostsdir + `address=` overlays) and never
forwarded upstream. Consequences:


- `A` → internal IP (e.g. Traefik VIP `192.168.20.145`).
- `AAAA` for an A-only name → **NODATA** (prevents leaking the public Cloudflare
  IPv6 of proxied names such as `auth.fzymgc.house`).
- A `fzymgc.house` name absent from the hostsdir → **NXDOMAIN**. Every internal
  name MUST have a local record (operator annotation or `tf/router-hosts/hosts.tf`).
  • Step 2: Add the operational note

In docs/operations/router-hosts.md, add a note that the overlay sets local=/fzymgc.house/, and that adding a new public-but-internal name requires a hostsdir entry or it will NXDOMAIN internally.

> **Authoritative zone:** `local-hosts.conf` sets `local=/fzymgc.house/`, so the
> router answers the whole zone locally and never forwards. Any new
> `*.fzymgc.house` name an internal client must reach needs a hostsdir entry
> (IngressRoute `router-hosts.fzymgc.house/enabled: "true"` annotation, or a
> `tf/router-hosts/hosts.tf` entry) — otherwise it returns NXDOMAIN internally.
  • Step 3: Lint

Run:

Terminal window
rumdl check docs/reference/network.md docs/operations/router-hosts.md

Expected: Success: No issues found.

  • Step 4: Commit

Run:

Terminal window
jj --no-pager commit -m "docs(dns): document authoritative internal fzymgc.house zone [hl-zh3w]"

Follow-up (separate bead, not a task here)

Section titled “Follow-up (separate bead, not a task here)”

File a bead for fzymgc.net: tf/cloudflare/tunnel.tf internal_services/ssh_services publish ${svc}.fzymgc.net. Confirm whether the same internal AAAA leak exists there and, if so, apply the equivalent authority directive. Tracked separately to keep this change focused.