Internal DNS (the resolver pair, and the appliance chain behind it)

The resolver pair sandiego/portland answers client DNS for every VLAN over IPv4 and for Core, Main, MainWireless and IoT over IPv6; Lab, Guest and Telework expose no DHCPv6 DNS field, so their v6 clients reach the appliance (Lab) or learn the pair’s VIP from the pair’s own RA (Guest, Telework). The appliance keeps DHCPv6 on every leg and DHCPv4 on Guest and Telework, answers any client that has not renewed its lease, and forwards everything it does not hold itself to the pair; it produces no curated zone and runs no resolver of its own. In an incident, read Which tier answers each VLAN, and how to prove it first, then Things that will bite you.

Which tier answers each VLAN, and how to prove it

Client	Asks	Path	Answered by
Trusted VLAN (Core, Lab, Main, MainWireless, IoT), IPv4	192.168.<vlan>.5, the leg’s floating address	on-link, switched at L2 — never routed, so the appliance’s DNAT never sees it — to whichever node holds the VRRP sync group	blocky on that node; fzymgc.house to pdns at 127.0.0.1:1053, everything else to Control D over DoT
Guest or Telework, IPv4	192.168.79.5 / 192.168.217.5	the same L2 path to the same node	the same blocky, with trusted: false, so fzymgc.house is NXDOMAIN by design
Remote client over the tailnet, IPv4	100.102.168.37 (sandiego) or 100.124.42.55 (portland) — each node’s own tailnet address	no VIP and no subnet route in the path	the same blocky on that node
IoT, Core, Main or MainWireless, IPv6	the leg’s v6 VIP on the resolver pair	on-link	the same blocky on that node
Lab, IPv6	the appliance’s own per-VLAN v6 address — Lab has DHCPv6 off and no DNS field to set	the appliance’s dnsmasq, which answers the per-device names it holds itself and forwards everything else to the pair at 192.168.20.5 through the WAN DNS field	writers 2 and 3 on the appliance for the names they hold; blocky on the pair for everything else
Guest or Telework, IPv6	the leg’s v6 VIP, learned as RDNSS from the pair’s own RA — neither leg exposes a DHCPv6 DNS field	on-link	the same blocky on that node

This table says nothing about DHCP: the appliance keeps DHCPv6 on every leg and DHCPv4 on Guest and Telework.

The seven VLAN legs and their floating addresses

VLAN	802.1Q	Network	Floating address
Core	untagged native	192.168.40.0/22	192.168.40.5
Lab	1000	192.168.200.0/22	192.168.200.5
Main	3000	192.168.20.0/22	192.168.20.5
IoT	3020	192.168.218.0/24	192.168.218.5
MainWireless	3001	192.168.24.0/22	192.168.24.5
Guest	3100	192.168.79.0/24	192.168.79.5
Telework	3200	192.168.217.0/24	192.168.217.5

The pair holds a floating address on seven legs, and all seven are cut over for client DNS. An operator reading a keepalived state dump, or ip -br addr on either node, sees seven VIPs and this table accounts for all seven. What Guest and Telework do NOT have is a Kea subnet or a DHCPv4 pool: each is a dns_vlan_interfaces entry declaring trusted: false, with no kea_subnet_id and no kea_pool, and the appliance serves DHCPv4 on both segments. See The one value an operator edits to change trust below.

Bringing a new leg into service, and how to tell whether it actually is

Declaring a leg in dns_vlan_interfaces does not serve it. Five consumers read that structure and each needs its own converge and its own adoption step. Check them individually — a leg that is short any one of these presents to a client as total DNS failure, identically in every case:

consumer	check on the node	adoption step
networkd legs	ip -br addr show | grep <vlan-id>	networkctl reload — a converge writes the files only
nftables accepts	sudo nft list ruleset | grep -c '<iface>'	reload handler; verify the dport 53 sets name every leg
Blocky listeners	sudo ss -tlnup | grep '<leg-addr>:53'	restart, via --tags blocky
keepalived VIPs	sudo grep -c '^vrrp_instance' /etc/keepalived/keepalived.conf	never a converge alone — see Arming and disarming below
RA announcer	ActiveEnterTimestamp vs config mtime	systemctl restart dns-common-ra-announce (no reload path)

The appliance points at the VIP, not the node address, so keepalived is the consumer that decides whether a client sees anything at all.

systemctl is-active cannot tell you a daemon matches its config. It reports that a process exists. Compare its ActiveEnterTimestamp against the config file’s mtime — a unit can be active and healthy while executing a configuration weeks older than the file on disk.

Never probe an on-segment resolver from off-segment. The Firewalla intercepts DNS to any address on a subnet it manages, so a query from a trusted subnet to a leg address returns NOERROR with real answers whether or not anything is listening — including to addresses that exist nowhere. It does not intercept locally-originated traffic, so the same query from the router behaves honestly. Probe from the segment, keep a control query to a deliberately non-existent address on that subnet, and treat ip neigh show <addr> reading INCOMPLETE as the reliable negative.

What the cutover actually was — one field per network, and nothing was removed

No rule was deleted from the appliance and nothing on it was hand-edited. This is the single most important mechanical fact on this page.

FW_PREROUTING_DNS_DEFAULT is nine independent tcp/udp rule pairs, each matching a per-network ipset as SOURCE and DNATing to that VLAN’s own gateway :53. The in-scope source sets are plain subnet CIDRs. So the redirect is source-matched, and it only ever sees routed traffic. 192.168.x.5 is on-link for a client on that VLAN, so once option 6 advertises the floating address those queries are switched at layer 2 and never reach the appliance at all.

The cutover is therefore one field per network — Primary DNS Server, 192.168.x.1 → 192.168.x.5: in the Firewalla app for the two legs whose DHCPv4 the appliance keeps, and in Kea’s option 6 ({{ leg.vip }}, always-send) for the five legs Kea serves. Nothing needs removing or disabling, which is just as well: the rules are appliance-managed by FireMain’s own loop and cannot be hand-edited. The back-out is the same field, set back.

Consequence for triage: the chain is full, and that is CORRECT. Read Failure mode: internal names return public NODATA before reading an empty chain as a success signal — an empty FW_PREROUTING_DNS_DEFAULT is a FireMain teardown, not evidence the pair is serving.

The DNS fields each untrusted leg carried before its cut, and what a rollback restores

This is the DNS half of the rollback record, and it exists because these values exist nowhere else. None of them is in Git, in inventory, or in any export — there is no API, no config file and no Ansible path to any of them. They are the rollback’s only input. The DHCPv4-server half of the record — the toggle, the range and the lease time each trusted leg carried when its DHCP service moved to Kea — is at The appliance values each rollback restores below.

Fields are recorded BY NAME and restored BY NAME, never by position. An ABSENT field and a BLANK field are different findings: some networks on this appliance offer four DHCPv4 fields and some offer five, and a restorer hunting for a fifth value on a four-field network will either invent one or report the restore incomplete. Neither is right.

READ THE EDIT VIEW, NOT THE READ-ONLY DISPLAY VIEW — the two surfaces DISAGREE. The display view shows no Secondary DNS Server row at all where the Edit view shows that row rendered with the greyed placeholder Optional, which is PRESENT-BUT-EMPTY. The Edit view is the only authoritative surface and is identifiable on sight: a Cancel / Save header, toggles rendered as switches, and the Surprise Me, Add DHCP Option and Delete This Network actions present.

Guest (VLAN 3100, 192.168.79.0/24, gateway 192.168.79.1, mask 255.255.255.0) and Telework (VLAN 3200, 192.168.217.0/24, gateway 192.168.217.1, mask 255.255.255.0) carry the same field set, field for field:

Field	Guest	Telework
DHCPv4 Server	On	On
Start IP Address / End IP Address	192.168.79.20 / 192.168.79.249	192.168.217.20 / 192.168.217.249
Primary DNS Server (the field the cut moved)	192.168.79.5; rollback restores 192.168.79.1	192.168.217.5; rollback restores 192.168.217.1
Secondary DNS Server	present, empty (Optional)	present, empty (Optional)
Lease Time	600	600
Search Domain	empty — a > disclosure row whose sub-screen was not opened	empty — same caveat
IPv6 / IPv6 Interface Type / IPv6 Delegation	On / Prefix Delegation / Automatic (VZFios)	same
DHCPv6	Off, alone on its card — no DHCPv6 Primary DNS Server field exists	same
mDNS Relay / SSDP Relay / Block ICMP (Ping)	Off / Off / Off	same
DHCP options under Add DHCP Option	none	none

The v6 half of both cuts is NOT ATTEMPTED, and that is a clean outcome rather than a shortfall. DHCPv6 is Off on both, there is no field to set, and DHCPv6 was not switched on to create one — enabling it would change an untrusted segment’s addressing posture as a side effect of a DNS change. Lab is the same shape. The resolver pair announces RA with RDNSS on both legs, so a segment with no appliance DHCPv6 DNS field has the pair as its only v6 DNS announcement: Guest’s v6 VIP is fddb:f665:73f7:2::5 and Telework’s is fddb:f665:73f7:5::5, neither of which is typed into any appliance field. Whether toggling DHCPv6 on would reveal a DNS row is UNKNOWN and was deliberately not probed — a different statement from “no such field can exist on this network”.

A cut moves what the network ADVERTISES in DHCP option 6, not any client that has not renewed. At a 600 s lease a client renews at about half of it, so the advertisement residue is bounded by ten minutes from the Save. Statically configured clients never renew at all and are not covered by that bound. A displaced RDNSS announcement carries an 86400 s lifetime and is never withdrawn, so a v6 cut leaves a client holding two resolvers for up to 24 h — see The 24 h RDNSS residue below; no DHCPv6 field moved on either untrusted leg, so neither cut displaced one.

The appliance is out of the query path for IPv4 only — how to prove it in one query

whoami.dns.invalid is the instrument. Blocky answers this RFC 6761 special-use name with the answering node’s own address; the appliance returns nothing. One query separates my resolver is serving from the appliance is. A .house name cannot do this job — the appliance forwards .house to the pair, so a right-looking answer tells you nothing about which tier the client asked.

IT MUST BE TAKEN FROM A HOST ON THAT VLAN. A query from anywhere else is routed, meets the source-matched DNAT, and is answered by the appliance while looking exactly like a resolver reading: a query to @192.168.218.5 taken from a Main host reads empty and records IoT as not serving; re-taken from an IoT-attached vantage it answers correctly.

# From a host holding an address on that VLAN. On a resolver node, -b selects the leg.
dig -b 192.168.24.7 @192.168.24.5 whoami.dns.invalid +short   # want: the VIP holder's Core address
dig -b 192.168.24.7 @192.168.24.1 whoami.dns.invalid +short   # CONTROL, want: empty (the appliance)

Both readings are required. The empty control is what makes the non-empty answer attributable; a lone positive reading proves only that something answered.

The discriminator does not verify a cut. It verifies that Blocky answers a query from that source. What a cut changes is what the network advertises, and no query addressed to the VIP can observe that; the DHCP OFFER proves the advertisement moved and the discriminator proves the floating address is answered by Blocky. Both are needed, and they are different claims.

Un-renewed clients route to the appliance until they renew

A cutover moves what each network ADVERTISES. It does not move any client that has not renewed its lease, and a client with a hand-configured 192.168.x.1 nameserver never renews at all. Such a client is answered by the appliance’s dnsmasq, which forwards to the pair — it gets a correct answer with no per-client view, no Blocky filtering and no query logging.

The instrument is a delta over the redirect chain’s own packet counters, per network, on the appliance — see Reading the redirect chain below. These counters are cumulative since FireMain last installed the rules, so an absolute reading says nothing; two samples from one boot, an hour or more apart, do. A counter is per rule, not per source, so it cannot attribute a packet to a client: un-renewed DHCP clients and hosts with a hand-configured nameserver are indistinguishable here.

The residue is ACCEPTED, not closed. The remedy for a network that keeps counting once every client has had a lease interval to renew is unscheduled hand work: each statically configured client has to be found and repointed by hand — on devices with no automatable shell on IoT and MainWireless. That is not repo work.

The tailnet path — a firewall rule and a listener that must not be removed as unexplained

Remote clients resolve .house at each node’s own tailnet address, not at a floating address: sandiego 100.102.168.37, portland 100.124.42.55. The coordination server assigns them and nothing in this repository guarantees them. They are not adjacent and must not be “tidied” into a pattern; unlike every other per-node value in this pair, they share no relationship at all.

Three artifacts carry this path, and removing any one of them takes remote .house resolution with it — including the recovery path the back-out below leans on:

Artifact	Where	Scope
the nftables accept	roles/dns-common/templates/nftables-dns.conf.j2, guarded by dns_common_tailnet_dns_enabled	iifname the tailnet device and ip saddr 100.64.0.0/10, udp and tcp, each carrying a counter
the client-tier listener	roles/blocky listener derivation, from that host’s own dns_tailnet_address	one address per node, its OWN — never the peer’s, never a wildcard
the tailnet policy	tailscale/policy.hujson, applied by a separate GitHub Action	grants port 53 on tag:ns and nothing wider

The tailnet’s own access-control policy is the authorization tier; the node ruleset is a narrowing. 100.64.0.0/10 is every tailnet address there is, so the source scope excludes a spoofed non-tailnet source on the tunnel and nothing more. Anything policy.hujson admits under tag:ns reaches this listener. Narrowing which peers may reach port 53 is an access-control change owned outside this workstream.

There is no subnet route in this path and no floating address: the destination is a 100.x address, which a subnet route does not carry. A packet arriving on a VLAN leg cannot match iifname "tailscale0", and the accepts’ counters advance on each remote query.

The tailnet device is deliberately not in dns_vlan_interfaces: that structure has five consumers, and a sixth pseudo-leg would produce a VLAN device, a floating address and a VRRP instance electing over a tunnel.

The client view — which clients see fzymgc.house, and which get NXDOMAIN

Blocky hands the internal namespace to an enumerated set of trusted clients and answers NXDOMAIN for the whole of fzymgc.house and both reverse zones to everybody else.

The decision behind this, including what it deliberately costs, is Untrusted clients lose the whole fzymgc.house apex.

Who is exempt, and why each member is a member

Exemption is by client source address. A client whose source falls in one of these ranges is handed a named, source-less client group that matches nothing, so the denylist never applies to it.

Member	Rendered exempt key(s)	Why it is a member	Where the value comes from
Core leg	192.168.40.0/22, fddb:f665:73f7:6::/64	the core entry declares trusted: true	the trusted attribute on that entry in ansible/inventory/host_vars/{sandiego,portland}.yml
Lab leg	192.168.200.0/22, fddb:f665:73f7:4::/64	the lab entry declares trusted: true	same
Main leg	192.168.20.0/22, fddb:f665:73f7:1::/64	the main entry declares trusted: true	same
IoT leg	192.168.218.0/24, fddb:f665:73f7:3::/64	the iot entry declares trusted: true	same
MainWireless leg	192.168.24.0/22, fddb:f665:73f7:7::/64	the mainwireless entry declares trusted: true	same
keepalived health client	127.0.0.1	dns-health.fzymgc.house sits under the withheld apex and is dug from loopback on each node. A group catching it blackholes the health check on both nodes at once — total estate outage, not leak. It is also allowlisted out of rateLimit for the same reason.	blocky_health_client_address in ansible/roles/blocky/defaults/main.yml
the appliance’s forwarding source	192.168.20.1	The Firewalla’s dnsmasq forwards from its own address, so its entire population — un-renewed leases, wg0, wg_ap, and the appliance’s own resolution — arrives as ONE client. Keyed separately even though the Main /22 already covers it, so that trusting the appliance stays independently revocable. It is also allowlisted out of rateLimit for that same aggregation reason: a per-client cap on an aggregate is an outage ceiling for everyone behind it rather than an abuse cap on a host.	a named literal in ansible/inventory/host_vars/sandiego.yml
the tailnet	100.64.0.0/10	A tailnet client’s 100.x source is in no leg CIDR, is not the health client and is not the appliance, so without this key the per-node tailnet :53 listener would answer, and would answer NXDOMAIN. 100.64.0.0/10 is every tailnet address there is, so this trusts every tailnet peer — the same posture the node ruleset already takes, since nftables cannot distinguish peers either. Narrowing that is a tailscale/policy.hujson change, owned elsewhere.	a declared value alongside dns_tailnet_address

Who is NOT exempt, and exactly what they see

Everything else. A client the configuration does not name is untrusted, including any leg in dns_vlan_interfaces whose entry does not declare trusted: true — that is deny by non-declaration, and it is the whole mechanism rather than a side effect. Guest (3100) and Telework (3200) therefore arrive untrusted with no edit to the Blocky role at all.

An untrusted client receives NXDOMAIN, not NOERROR with an empty answer, for:

• fzymgc.house and every name beneath it, for every query type — A, AAAA, PTR, SOA, TXT, MX, SRV and HTTPS alike. blockType: nxDomain does not branch on question type.
• every PTR under 168.192.in-addr.arpa and under 7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa, so internal hostnames cannot be walked in reverse.

THE ACCEPTED COST, and it is the surprising one: this includes the zone’s REAL PUBLIC NAMES. fzymgc.house is a genuine public zone whose source of truth is tf/cloudflare/. An untrusted client asking for id.fzymgc.house or cluster-docs.docs.fzymgc.house gets NXDOMAIN even though both resolve from the public internet. That was ruled deliberately and is recorded in the ADR linked above. If you are triaging “a guest device cannot reach the docs site”, this is the answer and it is working as designed.

The one value an operator edits to change trust

The trusted boolean on the leg’s own dns_vlan_interfaces entry, in both ansible/inventory/host_vars/sandiego.yml and ansible/inventory/host_vars/portland.yml. That is the whole interface, and it is the estate’s single declaration of leg trust. It is deliberately not kea_served_legs — serving DHCP on a leg and handing that leg the internal namespace are different claims that are expected to diverge.

A real YAML boolean, not a quoted string. trusted: "true" is a non-empty string and therefore truthy, so it would grant trust while reading like a value under review. Every entry must carry the key; an entry that omits it is a leg with no trust declaration, which the render must not silently filter out.

The conflation the single axis accepts, and its falsifying condition. A udp/67 exclusion really means “we do not serve DHCP here” and a TSIG exclusion really means “this leg has no DDNS”, and neither is literally trust — they coincide on Guest and Telework only because the appliance keeps DHCP on both segments. If a future leg is ever untrusted AND DHCP-served by this pair, the attribute must be split into two axes. The full statement is in the trusted header block in host_vars/sandiego.yml.

The two hazards a reader must not rediscover

Both of these read as “exempt” and behave as the opposite. They are recorded so nobody arrives at them a second time.

An exempt group written as an empty clientGroupsBlock value — 192.168.20.0/22: [] — falls through to default and carries the FULL denylist. Two independent code paths produce that, either one sufficient: the client-group index is built by iterating each identifier’s groups, so an empty list never inserts the identifier at all; and the default is selected on the collected group count, not on whether the identifier matched. The form that works is a named group with an empty source list, which is why the exemption is spelled the way it is.

An exempt group written under allowlists: instead of denylists: blocks everything else for every client carrying it — and is harmless until it isn’t. A group present in the allowlists with no entries is inert. The moment somebody adds one entry, every client in that group has every non-allowlisted domain blocked. A failure that arms itself later is strictly worse than one that breaks now.

What the pair emits onto an untrusted segment

:53 over both address families and both transports, ICMP echo, VRRP, and RA carrying RDNSS. NOT udp/67, NOT a DHCPv4 subnet, NOT a per-leg TSIG key, and NOT an exempt key in Blocky’s client-group map. The two rulings that decide the surprising halves of that are in the tree, with their reasoning: ansible/roles/dns-common/templates/nftables-dns.conf.j2 for the DHCP-port split, and ansible/roles/dns-common/tasks/ra-announce.yml for the announcement. The keepalived sync group is the trusted instances only — dns_vlan_interfaces | selectattr('trusted') | list — so VI_guest and VI_telework are not in the estate-wide group.

The IPv6 half — what the pair answers, what the appliance keeps, and the hazards

Everything under this heading is v6-only. The family-neutral per-leg records are under The DNS fields each untrusted leg carried before its cut, and what a rollback restores above.

IPv6 on the pair — four VLANs cut, three with no field to set

Four of the seven legs — IoT, Core, Main and MainWireless — have their IPv6 clients pointed at the resolver pair through the appliance’s DHCPv6 Primary DNS Server field. The other three — Lab, Guest and Telework — have DHCPv6 Off and expose no Primary DNS Server field, so their cut is NOT ATTEMPTED rather than failed: there was no field to set, and the wire confirms Lab advertises no RDNSS from the appliance. On Guest and Telework the pair’s own RA carries RDNSS, so the pair is the only v6 DNS announcement there; on Lab a v6 client is answered by the appliance’s dnsmasq, which forwards to the pair.

What makes the four cuts possible:

• Blocky has IPv6 listeners on every declared leg.
• Every declared leg has a floating ULA VIP at ::5, which is why a DHCPv6 field can name a stable address at all. See IPv6 ULA Addressing.

The sections below are the operator-facing consequences.

Pointing a VLAN’s IPv6 clients at the resolver pair

The mechanism is the Firewalla app’s per-network Primary DNS Server field, under DHCPv6, in the network’s Edit view. There is no other path.

The field is absent from the read-only display view and only appears under Edit. If you are looking at a network’s settings and there is no DHCPv6 DNS field, you are in the display view. And the display view is not a lower-fidelity substitute for a READ-BACK either — it omits the Secondary DNS Server row entirely and so cannot answer did the untouched fields stay untouched? at all. The Edit view’s on-sight identifiers are under The DNS fields each untrusted leg carried before its cut, and what a rollback restores above.

This repository cannot automate it. There is no API, no config file, and no Ansible path to that field. It is a human editing a value in a mobile app.

VLAN	802.1Q	value for Primary DNS Server
Main	3000	fddb:f665:73f7:1::5
IoT	3020	fddb:f665:73f7:3::5
Lab	1000	(no field — DHCPv6 is Off on this network)
Core	untagged	fddb:f665:73f7:6::5
MainWireless	3001	fddb:f665:73f7:7::5
Guest	3100	(no field — DHCPv6 is Off on this network; fddb:f665:73f7:2::5 is the leg’s UNUSED v6 VIP, typed nowhere)
Telework	3200	(no field — DHCPv6 is Off on this network; fddb:f665:73f7:5::5 is the leg’s UNUSED v6 VIP, typed nowhere)

The ULA index and the node suffix are small integers in different hextets — a digit collision this estate has been bitten by — so read a value from its own row, never from a neighbouring one. Whether toggling DHCPv6 on would reveal a field on Lab, Guest or Telework is UNKNOWN and was deliberately not probed.

Secondary DNS Server stays empty, deliberately. Both plausible values are wrong:

• A resolver node’s own address (::6/::7) bypasses the VIP, so the client keeps using a specific node after that node stops being the one holding the address — and it sits behind a health check that does not test IPv6.
• A public resolver sends internal queries out of the house, unfiltered and unlogged, and returns public NODATA for every .house name.

A value in that field surfaces on the wire as a second RDNSS option, so the emptiness is checkable rather than merely intended.

Two IPv6-specific hazards on the floating addresses

Both are v6-only. Neither has an IPv4 analogue, and the second is the one an operator meets first.

1. Every floating v6 address must carry -nodad. Without it, Duplicate Address Detection can leave the VIP in the dadfailed state — unusable, with no service-level error anywhere. keepalived reports the address as held, the interface lists it, and nothing answers on it.

   ip -6 addr show | grep -i dadfailed   # grep, not rg — the resolver nodes have no ripgrep

2. keep_addr_on_down is 0 on every leg, so an admin ip link set <leg> down DELETES the v6 VIP and not the v4 one. The two families behave differently on the same command: the IPv4 VIP comes back with the link, the IPv6 VIP is gone until keepalived reasserts it. An operator who bounces a leg to “reset” something has silently removed IPv6 service for that VLAN and left IPv4 working — which is exactly the shape that gets diagnosed as a DNS problem.

The 24 h RDNSS residue — a client holding two resolvers for a day is working, not broken

A new RDNSS appears within about 200 s. The displaced one carries an 86400 s lifetime and is never withdrawn. The appliance does not retract the old announcement; it simply stops renewing it, and the client ages it out on its own schedule. So for up to 24 hours after a cutover a client can hold both the pair’s ULA VIP and the appliance’s address, and that is the designed behaviour.

Client RDNSS consumers differ, and the difference is not a protocol guarantee. macOS replaces the per-interface resolver list from the newest advertisement; NetworkManager accumulates. So “both servers present” is a NetworkManager behaviour. On a macOS client the displacement is complete and immediate. Do not treat either shape as the correct one — read the client you actually have.

If the appliance’s address is present past 24 h, that is a distinct finding and not a lifetime counting down. It means the appliance is re-advertising it, which is a different fault with a different fix: check that the network’s Primary DNS Server field holds the ULA VIP and that nothing rewrote the RA configuration.

What the appliance answers for internal names over v6

The appliance’s writer 2 (LocalDomainSensor) publishes every device it sees under wireless.lan.fzymgc.house — /home/pi/.firewalla/run/hosts/, one file per device MAC — at every address the appliance has seen the device use: a GUA, a ULA and a v4 address, under both the short device name and the suffixed one. Three things follow:

• Writer 2 publishes AAAA records, not just A records.
• Real clients use the ULA prefix. The ULA in such a record is a client address from the per-leg fddb:f665:73f7:N::/64.
• These names are NOT in the zone the pair serves, and they are not the same names as the pair’s own dynamic ones. The pair populates <host>.<leg>.lan.fzymgc.house from Kea leases via D2 (The DDNS tier at the end of this page); the appliance publishes a flatter *.wireless.lan.fzymgc.house shape from its own writer 2, and nothing transports one to the other. So an appliance-published name will not answer against a floating address, in either family.

Consequence for a v6-preferring client on a cut VLAN. It asks the pair, and the pair has no *.wireless.lan.fzymgc.house data. Any workflow that depended on a per-device name resolving is on the appliance’s answer, and a cut VLAN does not receive it. This is a standing gap rather than an observed breakage — but it is the gap to suspect first if a per-device name stops working.

Working on the appliance — its directories and its adoption test

This repository owns nothing on the appliance and shares all of it. These are the read-only and operator-run procedures for working on that box: how to make it adopt a change, whose directory is whose, and how to read its ULA artifacts and its redirect chain.

Applying an appliance configuration change — stop, mutate, start, prove

SIGHUP does not make dnsmasq re-read its configuration, and systemctl reload firerouter_dhcp is not a gentler option: its ExecReload is literally pkill -HUP dnsmasq — the same signal, the same non-effect.

The procedure is stop, mutate, start, prove, and it is an operator procedure — never a role handler. Read the wire either side of it; where the old and new configurations advertise the same prefix, the advertised lifetime is the discriminator.

The adoption test is ExecMainStartTimestamp in epoch seconds, strictly later than the config file’s mtime in epoch seconds.

# On the appliance (pi@192.168.40.1). grep, not rg — this box has no ripgrep.
CONF=/home/pi/.router/config/dhcp/conf/zzzz-ula-dns.conf
stat -c %Y "$CONF"
systemctl show -p ExecMainStartTimestamp --value firerouter_dhcp
# Convert that timestamp to epoch seconds and require it to be STRICTLY GREATER than the mtime.

Three ways this test is got wrong:

• systemctl start returning 0 is not the proof. It reports that systemd accepted the request, not that the daemon read the file. A start that exits 0 against an unchanged running process is indistinguishable from a real adoption on exit status alone.
• ExecMainStartTimestampMonotonic is microseconds-since-boot and is not commensurable with an mtime. Used raw it is an enormous number and the comparison passes unconditionally. Divided by 1e6 it is a small number and the comparison fails spuriously. Neither direction is a measurement. Use the wall-clock ExecMainStartTimestamp.
• NRestarts counts restart-on-failure events, not a supervised stop/start. A clean operator restart leaves it unchanged. Use MainPID and ExecMainStartTimestamp.

Blast radius, stated because it is easy to underestimate: no DHCP and no RA anywhere in the house for the duration of the window. Every VLAN, both families, wired and wireless.

Which appliance directory belongs to which subsystem

This repository owns none of these and shares all of them. Writing to the wrong one is silently undone by whichever subsystem does own it.

Directory	Owner	What belongs there
/home/pi/.router/config/dhcp/conf	FireRouter	DHCP and RA configuration — a dhcp-range drop-in belongs here
/home/pi/.firewalla/config/dnsmasq_local	FireMain	DNS configuration for the front dnsmasq tier
/home/pi/.firewalla/config/post_main.d	FireMain	Start scripts — this is how anything survives a boot

The third row is the one that matters after a restart: a file written into either config directory without a corresponding post_main.d entry does not survive a reboot.

V6-05 — reading the appliance-side ULA artifacts absent

Host: the Firewalla appliance, as inventory host router (pi@192.168.40.1, the Core leg). The remote program uses mount, ls, sed, grep and test and nothing else — the tools that box has.

Single-quote the remote program. This is not a style preference. A double-quoted -a argument is expanded by the control shell before Ansible dispatches it: an unset loop variable becomes the empty string, the appliance evaluates [ -e "" ], and the loop prints an absence verdict for every iteration whatever is on disk — so an absence check passes whether or not anything was ever removed. Single quotes send every variable to the appliance unexpanded. Every quote inside the program is a double quote, so nothing needs escaping.

A dispatch’s exit status is its last command’s. The program therefore ends in exit $rc and not in the echo, so a control that did not fire fails the read rather than leaving a human to notice a missing line. The | tee is a capture, not a gate — a pipeline exits with tee’s status, and set -o pipefail is shell-dependent. The gate is the reading below it.

# On the appliance, via the control node. grep, not rg — this box has no ripgrep.
cd ansible && ansible router -i inventory/hosts.yml -b -m ansible.builtin.shell -a 'rc=0; m=$(mount | grep -c extdata); echo "CONTROL-MOUNT extdata=$m"; [ "$m" -ge 1 ] || rc=1; for d in /home/pi/.router/config/dhcp/conf /extdata/scripts; do if [ -d "$d" ]; then echo "CONTROL-PRESENT $d"; else echo "CONTROL-MISSING $d"; rc=1; fi; done; if [ -e /extdata/scripts/v6-05-control-never-exists ]; then echo "CONTROL-UNEXPECTED /extdata/scripts/v6-05-control-never-exists"; rc=1; else echo "CONTROL-ABSENT /extdata/scripts/v6-05-control-never-exists"; fi; for f in /home/pi/.router/config/dhcp/conf/zzzz-ula-dns.conf /home/pi/.firewalla/config/post_main.d/0001-ipv6-ula.sh /extdata/scripts/configure-ipv6-ula.py /var/log/ipv6-ula-failure.log /extdata/ipv6-ula-boot.log; do if [ -e "$f" ]; then echo "PRESENT $f"; ls -ld "$f" | sed "s/^/OWNER /"; else echo "ABSENT $f"; fi; done; ls -ld /home/pi/.router/config/dhcp/conf /extdata/scripts | sed "s/^/OWNER /"; echo "V6-05-VERDICT controls_rc=$rc"; exit $rc' 2>&1 | tee /tmp/v6-05-pre.txt

Read the output in this order. Each item names what its absence would mean:

1. CONTROL-MOUNT extdata= must read at least 1. Two of the five targets live on /extdata. If /extdata is not mounted they read absent for a reason that is not removal, and the whole reading is void.
2. Two CONTROL-PRESENT lines — /home/pi/.router/config/dhcp/conf and /extdata/scripts, the directories that survive the removal. These are the positive controls: they prove, in the run that reports the targets, that this program can print a present verdict. A reading with no CONTROL-PRESENT line is uninterpretable, not negative — it must be re-taken, never read as evidence of absence.
3. One CONTROL-ABSENT line for a path under /extdata/scripts that has never existed. It proves the absent branch is reachable and the test is not inverted.
4. Five target lines, each naming its own path exactly once. Compare them as a set against the five literals in the program, not as a count: a loop that read one path five times satisfies a bare count. Absence of a path from that set means the loop, not the appliance, is what changed.
5. OWNER lines — ls -ld on each surviving target and both parent directories. This is where you learn whether become is required before a removal, rather than mid-removal.
6. V6-05-VERDICT controls_rc=0. Any other value means a control did not fire, and nothing above it may be read as a measurement.

This is a one-shot precondition read, never a standing check. Appliance filesystem state must not acquire a standing or scheduled green bar in this repository: FireRouter and FireMain rebuild that filesystem on events this repository neither controls nor observes, so a recurring check over it would go red for reasons nobody here can fix. Run this block when the precondition is being established or re-established — never on a timer.

Reading the redirect chain

Existence and counters need different commands, and reaching for the wrong one is a mistake that looks like diligence.

• sudo ip6tables -t nat -S prints rule specifications and no packet or byte counter at all. It can prove the rules exist. It can say nothing whatever about whether they carry traffic.
• sudo ip6tables-save -c -t nat prefixes each rule with [packets:bytes]. Counters come from here.

sudo is required on both. Without it each returns an empty ruleset, which presents as a clean zero — the most dangerous possible failure mode for this particular reading.

Existence is two numbers, never one.

# On the appliance. grep, not rg — this box has no ripgrep.
sudo ip6tables -t nat -S > /tmp/nat.txt
grep -c 'j DNAT --to-destination \[fe80::226d:31ff:fe51:61fb\]' /tmp/nat.txt   # expect 14
grep -c '^-A FW_PREROUTING_DNS_FALLBACK ' /tmp/nat.txt                         # expect 5

The FALLBACK count must be anchored to rule position. A bare grep -c 'FALLBACK' reads 7 against a perfectly correct estate, because the -N FW_PREROUTING_DNS_FALLBACK chain declaration and the -A FW_PREROUTING -j FW_PREROUTING_DNS_FALLBACK jump match the same token as the five rules inside the chain. Anchoring on ^-A FW_PREROUTING_DNS_FALLBACK followed by a space counts rules; the bare token counts lines that mention a string.

An empty chain is an incident signal, not a pass. This is the state a successful cutover most resembles to a careless reading, and it is the state the estate enters when FireMain health-gates the redirect away. Zero is never the answer you are looking for here.

Read counters as two samples at least an hour apart on the same boot, and compare growth. A single absolute number is not a rate. A reboot resets every counter, so a delta spanning one is meaningless in both directions — it can read near-zero or negative whether or not the redirect is carrying traffic. Two samples from one boot are immune to that by construction:

# Read the boot id in the SAME ssh transaction as the counters, at both ends.
# Two separate ssh calls are NOT one session: a reboot landing between them pairs
# counters from one boot with an id from another, and no later check can detect it.
cat /proc/sys/kernel/random/boot_id
sudo ip6tables-save -c -t nat
cat /proc/sys/kernel/random/boot_id

If the two ids differ, the samples are void: discard both and restart the interval. Do not reason about which way the reset pushed the numbers.

There is no uncut control leg on this estate. Every leg is cut, and the resolver pair announces RA carrying RDNSS on Guest and Telework, so no leg’s counters are structurally unable to grow. What discriminates is the contrast between cut legs, which holds the house’s business constant: cut legs whose counters stay flat beside cut legs whose counters grow. Growth on a cut leg is un-renewed clients routing to the appliance — expected in kind (see Un-renewed clients route to the appliance until they renew above) — but at T+24 h it is a finding to act on, not a pass.

What an operator triaging IPv6 on Guest or Telework needs from this, stated plainly. A dual-stack client on either leg may learn the pair’s VIP there — fddb:f665:73f7:2::5 on Guest, fddb:f665:73f7:5::5 on Telework — as a recursive resolver over IPv6, from the pair’s own RA. What such a client gets is a working resolver that answers nothing under fzymgc.house: both legs are trusted: false, so every name in the internal namespace and every internal PTR comes back NXDOMAIN by design — see Who is NOT exempt, and exactly what they see above. Public names resolve normally. The symptom to expect is therefore “internal names fail, the internet works, and the resolver is up”, and the pair IS in the path on those legs — over IPv6 by RA and over IPv4 by the leg’s Primary DNS Server field — even though the appliance keeps DHCPv4 on both.

Writing commands for the appliance and the resolver nodes

The rule is scoped by WHERE a command runs, not by which file it is written in.

The appliance and the resolver nodes have no ripgrep

The grep invocations in the appliance-side and node-side command blocks on this page are deliberate. Do not “fix” them to rg.

The Firewalla has /usr/bin/grep, awk and sed and no ripgrep; neither resolver node has it either. An unscoped reading of this repository’s “always use rg” rule once rewrote a working on-box grep on this page into an rg that cannot execute there, silently breaking a documented runbook command — which is why CLAUDE.md’s Search rule exempts commands written for a third-party appliance.

• Commands you run against this repository, on your own workstation — use rg.
• Commands written to run on the appliance or on a resolver node — use grep.

Validating a resolver role off the estate — one command per role

Each role below carries a molecule scenario whose converge runs that role’s own tasks/verify.yml against containers built from the trixie image the resolvers run. The scenarios link ansible/inventory/ in, so a converge exercises the real group_vars and host_vars rather than a fixture.

Four roles have a scenario: dns-common, blocky, pdns and keepalived. kea, lightningstream, dns-otel-collector and dns-tailscale have none, by decision: a role whose verify.yml asserts nothing a converge can falsify gets no scenario. A missing entry below is that decision, not an omission.

MUST run these from a clone of this repository on your workstation. They are not node-side commands: each builds local containers and reads the inventory out of the checkout.

MUST have Docker running on the workstation. The scenarios use molecule’s docker driver; with the daemon down the first sign is a docker.errors.DockerException traceback, not a molecule message.

MUST run one scenario at a time. Platforms are named after the real resolvers — sandiego, plus portland where the role dereferences a peer through hostvars — so two scenarios up at once collide on the container names.

dns-common and pdns MUST have VAULT_TOKEN exported. Both read from Vault on the control node, never from a container, so no container needs Vault reachability. vault login -method=oidc writes ~/.vault-token but does not export it.

dns-common

export VAULT_TOKEN="$(cat ~/.vault-token)"
task ansible:molecule-dns-common
# want: PLAY RECAP with failed=0 and unreachable=0 for BOTH sandiego and portland.
#   Two platforms because tasks/firewall.yml asserts the VRRP peer declares exactly the
#   legs this node does, which dereferences hostvars[keepalived_peer_host].
#   A 403 on the ICA1 chain read is an absent or expired token, not a broken role:
#   tasks/ca-trust.yml reads that chain unconditionally, on localhost.

blocky

task ansible:molecule-blocky
# want: PLAY RECAP with failed=0 and unreachable=0 for sandiego, and the roll-up
#   "criteria passed: 4 of 4". One platform: the role dereferences no hostvars.
#   The scenario runs the daemon at its real inventory value, so blocky comes up and
#   binds every per-VLAN listener on a container carrying none of those addresses —
#   freeBind in blocky-config.yml.j2. An `active (running)` unit here is expected.
#   The three `WARN upstream: initial resolver test failed` lines for 127.0.0.1:1053
#   are the pdns loopback socket a blocky-only container has no reason to carry.

pdns

export VAULT_TOKEN="$(cat ~/.vault-token)"
task ansible:molecule-pdns
# want: exit status 201, go-task's code for a failed command, and its last line
#   `task: Failed to run task "ansible:molecule-pdns": exit status 2` — molecule's own.
#   PLAY RECAP with failed=1 and unreachable=0 for sandiego, and the roll-up
#   "criteria passed: 2 of 14", stopped at `zone-set` with observed (pdnsutil): (none).
#   That red is the scenario's boundary, NOT a defect: the role creates no zone —
#   `tf/dns-records` applies every zone through the HTTP API — and the block stops at
#   its first red. Do not seed a zone and do not add a `when:`; the boundary is
#   recorded in the scenario's converge.yml. One platform: the role dereferences no
#   hostvars. The scenario connects as `fzymgc`, created by its prepare.yml, because
#   `api-key-drop-in-unreadable` is a read that must fail and cannot fail as root.
#   A 403 on the API-key read is an absent or expired token, not a broken role.

keepalived

task ansible:molecule-keepalived
# want: PLAY RECAP with failed=0 and unreachable=0 for BOTH sandiego and portland, and
#   the roll-up "criteria passed: 3 of 3" on each. The three are SC2-address-read,
#   SC2-service and DNS-07-script-account. The peer-aware external tracker's three
#   criteria are retired, so this scenario no longer stops part way through the block.
#   Two platforms because keepalived.conf.j2 dereferences hostvars[keepalived_peer_host].
#   The daemon's own `--config-test` validates the render against six dummy links the
#   scenario's prepare.yml creates (down, unaddressed) for the tagged legs the container
#   lacks. `keepalived_enabled: false` in converge.yml is deliberate: at the real value
#   the daemon starts, every instance enters FAULT, and the roll-up is the same.

The PowerDNS authoritative tier

pdns holds 127.0.0.1:1053 on sandiego and on portland, and lightningstream runs on both. Every .house answer a client gets comes from PowerDNS, whichever node happens to be holding the floating addresses. There is no unbound on either node.

A read that tells you which state a node is in, without changing anything:

# On a resolver node. Reach it with:
#   scripts/dns-adhoc.sh <node> -m ansible.builtin.shell -a '<the line below>'
# The resolver SSH key lives in Vault and the SSH agent is deliberately not used; see the
# header of scripts/dns-adhoc.sh for why.
echo "$(hostname -s) vip=$(ip -br addr show | grep -c 192.168.20.5) pdns=$(systemctl is-active pdns) ls=$(systemctl is-active lightningstream)"
# want: <node> vip=<0 or 1> pdns=active ls=active
# `vip=` is the only field expected to differ between the nodes — either may hold the addresses.
# Any other difference is either a converge in progress or a finding. grep, not rg — the resolver
# nodes have /usr/bin/grep and no ripgrep.

The apex, and what it answers

None of these is a defect. All of them look like breakage to somebody meeting them for the first time in the middle of an incident. That is the entire reason they are written here.

An unlisted RR type on a declared name returns NOERROR with no answer. This is the property that closes the type-65/HTTPS ECH leak.

dig @127.0.0.1 -p 1053 dns.main.fzymgc.house TYPE65
dig @127.0.0.1 -p 1053 dns.main.fzymgc.house HTTPS

Both answers carry NOERROR with an empty answer section and the apex SOA in AUTHORITY — NODATA, not NXDOMAIN.

An undeclared name returns NXDOMAIN with the apex SOA.

dig @127.0.0.1 -p 1053 nonexistent-name-for-the-gate.fzymgc.house A

The status is NXDOMAIN, with the apex SOA in the AUTHORITY section and no answer.

The AUTHORITY SOA’s MINIMUM field reads 60. That is how long a client keeps believing this NXDOMAIN, and it is the dominant term in the convergence window computed in the next section. Note that this is a different answer from the NODATA above, and an operator at 2am needs both.

No .house query leaves the box, and it holds on two independent grounds. blocky will not fall back, and PowerDNS as configured cannot recurse or forward at all. blocky’s guards are fallbackUpstream: false and the conditional mapping fzymgc.house: "127.0.0.1:1053". Read them like this, and read the exit status rather than a || echo branch:

# /etc/blocky/config.yml is ROOT-OWNED, so this needs -b:
#   scripts/dns-adhoc.sh <node> -b -m ansible.builtin.shell -a '<the line below>'
grep -n 'fallbackUpstream\|127.0.0.1:1053' /etc/blocky/config.yml; echo "RC=$?"
# want: RC=0 and two lines — the fallback guard and the conditional mapping.
#   RC=1 is a genuine absence and a finding.
#   RC=2 means you read it WITHOUT privilege, and the guards may be present and unreadable.
# `grep PATTERN FILE || echo "NOT FOUND"` CANNOT tell those two apart and reports a present
# security guard as missing. Capture the rc.
# grep, not rg — the resolver nodes have no ripgrep.

The apex itself answers.

dig @127.0.0.1 -p 1053 fzymgc.house SOA

The status is NOERROR and the SOA is in the answer section.

dig @127.0.0.1 -p 1053 fzymgc.house A

NOERROR/NODATA. This is also intended. No A record is declared at the apex — what the bare apex name should address to is an open decision — so the correct answer to “what is the address of fzymgc.house” is “the name exists and has no address of that type”.

dig @127.0.0.1 -p 1053 fzymgc.house NS

The answer section carries both nodes as NS, with their addresses in ADDITIONAL.

Convergence lag — the arithmetic, not a reassurance

PowerDNS runs multi-primary with no zone transfer: each node owns a local LMDB and LightningStream reconciles the two through S3. There is no primary to lose and therefore no primary-outage cost — what replaces it is a window during which the two nodes disagree.

The window is four terms plus a cache:

Term	Value	What it is
lmdb_poll_interval	1s	how long the WRITING node takes to notice its own change
the snapshot upload	sub-second at this zone size	pushing the snapshot to S3
storage_poll_interval	1s	how long the PEER takes to list the storage and see it
the download and merge	sub-second at this zone size	the peer pulling and merging it

Roughly two to five seconds at this zone size for a record written on one node to be answerable on the other. Both intervals are rendered from ansible/roles/lightningstream/defaults/main.yml. Grant metadata written by Terraform through one node’s API takes longer to land on the peer — on the order of 45 s end to end — see The two latencies under The DDNS tier.

That is not the number an operator experiences. A client that asked the other node inside that window got NXDOMAIN, and it keeps believing that answer for the SOA minimum of 60 seconds — the RFC 2308 negative-cache TTL, declared on every zone in tf/dns-records/. So:

The worst case an operator can observe is about 65 seconds.

That number was chosen, not tuned down to, and it is not going lower. Both polls are already at their floor of one second, and the dominant term is the negative cache. Shaving a second off a poll buys nothing against a 60-second window; lowering the negative-cache TTL is a zone-wide decision with its own costs and is not an operational knob.

One boundary case, and it is a bring-up property rather than an operational one. A record added to an existing zone is not delayed by the zone cache at all. A brand-new zone takes up to zone-cache-refresh-interval (60s, ansible/roles/pdns/defaults/main.yml) longer to become visible on the peer, because that is when the peer next re-reads the zone list. zone-cache-refresh-interval caches the zone list, not records. A reader who takes it for a record-replication delay will draw the wrong conclusion from every convergence question they ever ask this page.

Reverse resolution — the covering zones, the carves, and the one deliberately not attempted

Every zone is Native for the same reason the forward zones are — there is no transfer and no primary:

Zone	Covers
168.192.in-addr.arpa.	the whole of 192.168.0.0/16 outside the carved cuts — the curated PTRs
the per-leg /24 carves (200.–203., 22.–23., 41.–43., 218., and mainwireless’s)	the DDNS-writable reverse space, one real /24 zone each — see The DDNS tier
7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa.	the estate’s ULA fddb:f665:73f7::/48

A consequence worth reading rather than discovering: the IPv4 zone makes us authoritative for address ranges the pair does not serve DHCP for — Guest 192.168.79.0/24, Telework 192.168.217.0/24 and WireGuard 192.168.80.0/24. Reverse queries for those addresses return NXDOMAIN from our zone.

The 2600: GUA reverse is explicitly not attempted. The estate’s global IPv6 prefix is delegated by Verizon, and so is its ip6.arpa reverse. We cannot be authoritative for a reverse zone under somebody else’s delegation, and nothing in tf/dns-records/reverse.tf tries. This is an exclusion, not a gap. “IPv6 PTRs don’t work for our global addresses” must not be filed as a defect against this tier. ULA reverse works; GUA reverse is Verizon’s.

PTRs are not a side effect of the forward record. powerdns_record’s set_ptr has been deprecated since PowerDNS 4.3 and — the part that matters — does not remove the PTR when the A record is deleted. tf/dns-records/hosts.tf carries an explicit ptr boolean per host so that the canonical name for an address is a decision with a resource in state, rather than a side effect nothing notices going stale.

Record write paths, and the update boundary

Three writers, and they do not overlap:

Writer	What it writes	How
Terraform, tf/dns-records/	the curated static records, the zones, the SOAs, the PTRs, the TSIG grants	the mmianl/powerdns provider against the pdns API on :8081
external-dns	the Kubernetes service estate	the same API, --provider=pdns, TXT-registry ownership
Kea D2 and dns-aaaa-publish	DHCP-derived names in <leg>.lan.fzymgc.house and the carved reverse zones	DNS UPDATE with TSIG

The update boundary is declared, not assumed. Each leg’s child zone <leg>.lan.fzymgc.house and each carved reverse /24 carries a TSIG-ALLOW-DNSUPDATE grant in tf/dns-records/metadata.tf; the parent lan.fzymgc.house, the covering /16 and the apex carry none, and dnsupdate-require-tsig=yes makes that absence a REFUSED. How the grants and the global switches interlock is Where the boundary is, and what actually enforces it under The DDNS tier.

dig TXT <managed-name> returns an ownership record that looks like junk. It is not junk. external-dns’s pdns provider has no --txt-prefix support, so the TXT registry writes its ownership record at the same name as the record it owns rather than at a prefixed sibling. A TXT beside every external-dns-managed A record is expected rather than junk, and deleting one by hand makes external-dns believe it no longer owns the record beside it.

Zone syntax is checked with pdnsutil zone check <zone> — the PowerDNS 5.x object-verb spelling. pdnsutil check-zone is the deprecated alias.

Adding a name — one entry per writer, and the rule that keeps the writers apart

A name enters the zone the pair serves through exactly one of three writers, and which writer is decided by what the name IS. A device or host is a curated entry; a Kubernetes service is an annotation; a DHCP client is a lease. There is no fourth path and no reviewed line in any gate: the declaration files are the inventory, and the apply that publishes them is the drift control.

A device or host — a curated static record or a DHCP reservation — is one entry in tf/dns-records/hosts.yml. Seven fields, six of them required: hostname (a lowercase FQDN with a trailing dot, which is what “the same name” means everywhere on this page), address, address6 (or null), ptr (an explicit boolean with no safe default; write false unless the address is outside every zone listed under dns_ddns_reverse_zones in ansible/inventory/group_vars/dns_hosts.yml, because a curated PTR inside DDNS-writable space is parent-held under the carve and answers nothing), aliases, comment, and the optional mac. Append the entry at the end of the file; never re-sort the file for readability, and never rename a key — the key is a live Terraform resource address, and renaming it is a destroy-and-create of a live DNS name. When the PR merges, main-cluster-dns-records on HCP Terraform applies the A and AAAA (and the PTR when ptr: true): hosts.tf writes every entry into powerdns_zone.apex. An entry carrying a mac is also a Kea reservation, rendered on the next --tags kea converge from the same literal. Nothing pins this file’s entry count or its key set, so adding an entry costs no reviewed line anywhere else. The infrastructure name families that are not host entries — dns.<leg>, the per-leg node names, the node apexes — live in records.tf.

A Kubernetes service is a Host() matcher plus the external-dns.alpha.kubernetes.io/target annotation, and nothing under tf/. external-dns publishes it through the same pdns API with TXT-registry ownership, and it will delete what it believes it no longer owns. The two ways that goes wrong without a log line are the next section — read The two external-dns traps before adding one.

A DHCP client needs nothing done: Kea D2 writes <host>.<leg>.lan.fzymgc.house on the lease for core, lab, main, iot and mainwireless — the five zones under dns_ddns_forward_zones in ansible/inventory/group_vars/dns_hosts.yml. Which daemon publishes which half, and how each half fails, is The DDNS tier below.

The rule that keeps the writers apart: the two apex writers are disjoint, and neither declares under lan.fzymgc.house. external-dns drops a lan. name silently by --exclude-domains=lan.fzymgc.house (argocd/app-configs/external-dns/values.yaml). A curated entry whose hostname sits under a delegated child (lan.fzymgc.house or a leg zone beneath it) is written into the apex by hosts.tf anyway, is parent-held under the delegation, and answers NXDOMAIN with the child’s authority — published, invisible, and undetectable by a dig of the parent. The TXT registry keeps external-dns from deleting a curated record; nothing keeps a curated record out of a child zone except this rule, and no gate holds it — it is written here and nowhere else. A name under no writer is dark by design: it answers NXDOMAIN and is not forwarded — fallbackUpstream: false — see What the namespace closure covers for the scope of the closure and its revert.

The two external-dns traps, both of which fail silently

Neither raises an error. Neither logs a failure. Both end in records disappearing, because external-dns runs with --policy=sync, which deletes what it believes it no longer owns.

Trap 1 — the annotation prefix is coupled to the pinned tag.

Ref	DefaultAnnotationPrefix
tag v0.21.0 (pinned, running)	external-dns.alpha.kubernetes.io/
master, after that tag	external-dns.kubernetes.io/

Every converted manifest is annotated with the alpha spelling. A minor version bump renames every one of them at once: external-dns simply stops seeing the annotations, generates no endpoints from them, and under --policy=sync deletes the records they used to produce. Nothing in its log says so.

• Symptom to search for: names that worked yesterday returning NXDOMAIN, with external-dns healthy and its log carrying no error at all.
• The string that confirms it: the constant lives at source/annotations/annotations.go in kubernetes-sigs/external-dns. Compare its value at the tag you have deployed against the prefix written in argocd/app-configs/.
• The escape hatch when the bump comes: the --annotation-prefix flag pins the old spelling, so the image and the manifests can move independently.

Trap 2 — the traefik-proxy source builds targets from the target annotation and from nowhere else.

Traefik CRDs do not expose a load balancer IP or hostname in their status. Upstream states the consequence plainly: “Without it, no endpoint is generated and --default-targets cannot apply.” So every Traefik IngressRoute needs its own external-dns.alpha.kubernetes.io/target: "192.168.20.145".

• Symptom to search for: a newly added IngressRoute whose host never appears in DNS, while every older one resolves. external-dns logs nothing about it — the resource simply produces no endpoint.
• The check: the resource carries a Host(...) match and no target annotation.

The S3 endpoint and its certificate — a convergence dependency, not a serving one

LightningStream reconciles the two LMDBs through https://s3-direct.nas.fzymgc.house, which is the NAS’s own direct S3 address (192.168.20.203) presenting an estate-issued certificate. The credentials come from Vault at secret/fzymgc-house/infrastructure/nas/rustfs-s3; nothing in this path is written into a file on a resolver node in plaintext.

This path is not in the serving path. PowerDNS answers every query from its own local LMDB whether or not S3 is reachable and whether or not its certificate validates. An S3 outage, a certificate expiry, or a NAS reboot degrades convergence and nothing else: the two nodes stop reconciling and each keeps answering correctly from what it already has. It is a ticket, not a page.

There is no dedicated availability detector for this endpoint, and that is deliberate. Adding one would create a second arbiter over the resolver pair — an instrument that can take a healthy, correctly-answering node out of VRRP eligibility because a storage endpoint is down. The whole point of the paragraph above is that those two facts are unrelated. The endpoint’s health is read when somebody is already looking at replication, not continuously by something that can act on it.

The certificate is a scheduled date rather than a risk: it is renewed through Traefik’s DNS-01 resolver, which needs the WAN to renew and nothing to serve. A WAN outage lasting past a renewal window would eventually stop convergence; it would never stop an answer.

The endpoint name is pinned in two files, and a reboot is the reason for the second. The name has no DNS record by design, so roles/lightningstream pins it in /etc/hosts. The nodes’ cloud-init user-data sets manage_etc_hosts: true, which re-renders /etc/hosts from /etc/cloud/templates/hosts.debian.tmpl at every boot and discards that pin. The role therefore pins the name in the template as well, and --tags lightningstream-verify asserts both.

If replication has stopped and the journal shows lookup s3-direct.nas.fzymgc.house on 127.0.0.1:53: no such host, the pin is gone. Compare the mtime of /etc/hosts with uptime -s. Repair one node at a time:

scripts/dns-playbook.sh --limit portland --tags lightningstream-config
scripts/dns-playbook.sh --limit sandiego --tags lightningstream-config

Both nodes log Loaded remote update and startup phase completed successfully within seconds. The symptom a client sees is a name that answers on one node and returns NXDOMAIN on the other, which reads as flaky resolution. Nothing alerts on this today. The gap is tracked in issue #2219.

Handing the authoritative socket to another daemon — the rules a further conversion inherits

Both nodes run PowerDNS; nothing here is pending. What survives from the conversion is the set of rules any future handover of the authoritative tier inherits. The walked conversion, its rollback and the readings it produced are PowerDNS Cutover Walked Forward, Rolled Back, and Re-run.

Measure the name delta by querying, not by diffing config. The question is how many names the tier being replaced serves that the successor does not, and it moves every time external-dns publishes:

# on the pdns node, for each name the tier being replaced serves:
dig @127.0.0.1 -p 1053 <name> A +short   # empty output == the name is dark

A name that is dark is only acceptable if it is stale, and staleness is verified by asking the cluster rather than by reading a list:

kubectl get ingressroute,ingressroutetcp,ingress,httproute -A -o json | grep -c '<name>.fzymgc.house'

Note the backticks in Traefik’s Host(...) match rules: a pattern anchored on double quotes matches Ingress objects but silently misses every IngressRoute, and reports a live name as stale. A namespace being a remnant and being empty are different claims, and only the second one is safe to act on without looking.

Why a dark name is an outage rather than a slow lookup. blocky ships fallbackUpstream: false, which is the guard that keeps .house queries on the box. There is no “somewhere else” for an absent name to be answered from, so an absent name is an OUTAGE on this tier.

Prove the rollback target before releasing the addresses, never after. The rollback is sudo systemctl stop keepalived on the holder, and the node you are rolling back to must be read healthy first: keepalived-check-house.sh rc=0 and its client and authoritative tiers both active, before the holder is told to let go. Start the drained node again afterwards and it returns as BACKUP under nopreempt, holding nothing — moving the addresses back is a separate deliberate act, described at Promotion — moving the floating addresses deliberately.

Two daemons cannot both hold 127.0.0.1:1053. The play that STOPS the incumbent must run before the play that starts the successor, and ansible/dns-playbook.yml’s play order is what sequences that by construction — so a single tagged run is correct and two runs are not. The reverse direction needs the reverse order. Getting it backwards fails CLOSED — there is no dual bind, since SO_REUSEPORT needs every binder to set it and matching effective UIDs — but it leaves the loser enabled and restart-looping, so a reboot resumes the fight unattended. A failed host runs no further plays, so the play that would have freed the socket is never reached. Run a handover on the node holding no floating address, and budget the interval in which neither daemon holds the socket: vault.fzymgc.house keeps answering through it only if the successor already holds it, and keepalived-check-house.sh keeps the node VRRP-eligible while it serves a stale set.

LightningStream must push from a populated store before a peer pulls. A node whose pdns starts on an LMDB pulled from an empty store answers nothing for .house and takes chk_house_answer down — removing it from VRRP eligibility and leaving the estate single-homed. And lightningstream_group: pdns names a group the pdns role creates: converging LightningStream first fails with chgrp failed: failed to look up group pdns, which reads as a permissions problem and is an ordering one. The single-run play order sequences both.

Where the resolver pair is cabled — one switch, accepted

Nothing in this repository re-checks where the nodes are cabled. Take the reading again before relying on it: one LLDP frame per node, on eth0 (the trunk), gives the chassis id, the port id and the system name.

Both resolvers hang off switch-main (chassis f4:e2:c6:ad:b2:48): sandiego on Port 12, portland on Port 13. So both share one switch as a single failure domain — accepted, because a failure of that switch takes the whole estate down and leaves a surviving resolver nothing to serve, and because the switch does not need DNS to boot, so recovery carries no circular dependency on the resolvers.

The different-chassis requirement does not apply to this topology, and the backlogged move has a precondition

Stated here with its reasoning because it reads as a violation to anyone who finds it without one. An earlier gate required the two resolver nodes to sit on different PoE switches before the VIP cutover. That rule is retired for this topology by operator decision, not deleted: both nodes hang off the main core switch and neither is solely PoE-powered, so a PoE failure does not take them; if that switch is down the whole estate is down and a surviving resolver has nothing left to serve; and the switch does not need DNS to boot, so recovery carries no circular dependency.

Re-asserting that the chassis must differ would be worse than silence — it reddens on the topology the operator deliberately chose, and a gate that fails on a correct configuration trains its reader to ignore it.

Moving portland to a second switch is backlogged, not abandoned, and its precondition travels with it: the new port needs the same native-VLAN-to-Core treatment sandiego’s port got, or that node’s Core leg drops. That is half the estate’s DNS. Do not do the move without the port work.

The surviving half of that requirement is live and is what the promotion procedure below implements: never change both resolver nodes in the same window.

Promotion — moving the floating addresses deliberately

Both instances start in the backup state with nopreempt. A node that is up and returns from a tracker FAULT does not take the addresses back, so promotion is an explicit act. That is deliberate: it stops a node that has just come back from yanking five live floating addresses.

nopreempt DOES NOT SURVIVE A REBOOT OF THE HIGHER-PRIORITY NODE. nopreempt suppresses the BACKUP→MASTER transition only for an instance that hears an existing master. On a reboot of sandiego (priority 150) its master-down timer expires before its VLAN sub-interfaces pass VRRP, so it reads the group as unowned rather than owned by someone lower, and takes it: the sync group goes FAULT (the .house tracker fails while blocky is not up), then BACKUP, then MASTER within seconds of keepalived starting, with no advert ever received. By the time portland’s adverts arrive, sandiego is MASTER at priority 150 and outranks them; portland steps down.

Operationally: after rebooting the priority-150 node, expect the group back on it. Read ownership, do not assume the peer kept it. The failback is automatic on this path and deliberate on the tracker-fault path — the difference is whether keepalived itself restarted.

Verify on the node that is NOT holding the addresses, promote deliberately, then work on the other one. Doing it in that order is what makes one-node-per-window expressible: a mistake on the non-holder costs nothing.

# STEP 1 — verify on the node that is NOT holding the addresses, where a mistake costs nothing
ssh <non-holder> 'systemctl is-active keepalived blocky pdns'
ssh <non-holder> 'sudo /usr/local/sbin/keepalived-check-house.sh; echo $?'   # expect 0
ssh <non-holder> 'dig +short @127.0.0.1 dns-health.fzymgc.house A'          # expect 192.168.x.x

# STEP 2 — move the group deliberately
ssh <holder> 'sudo systemctl stop keepalived'
# confirm from BOTH nodes that the peer now holds all five
ssh 192.168.40.6 'ip -br addr'; ssh 192.168.40.7 'ip -br addr'

# STEP 3 — work on the drained node, then return it as BACKUP
ssh <old-holder> 'sudo systemctl start keepalived'
# nopreempt: it returns as BACKUP and does NOT take the group back

systemctl stop keepalived is the role’s own stated position: roles/keepalived/tasks/config.yml carries no stop branch, precisely so that dropping five live floating addresses is an operator act typed by a human who meant it.

Which node holds the addresses is a reading, not a convention. Read it from both nodes before acting:

ssh 192.168.40.6 'ip -br addr | grep -c "192\.168\..*\.5/"'   # grep, not rg — the nodes have no ripgrep
ssh 192.168.40.7 'ip -br addr | grep -c "192\.168\..*\.5/"'
# want: one node reports 5 and the other 0. Any other pair is a finding, and 5/5 is a split brain.

What a failover looks like

A tracker-induced failover moves all five addresses together inside roughly ten seconds: detection is check_interval 2 × fall 5, plus up to 3 × advert_int 1, plus the priority skew (256−150)/256 ≈ 0.41 s. The group may be briefly unowned in transit, which is legal. The protocol specifies no ordering among a synchronisation group’s members.

What a DOUBLE failure looks like — know this before you see one. With both nodes faulted the floating addresses go UNOWNED, not duplicated: neither node holds any of them, keepalived and blocky both report active on both nodes, and both journals carry Entering FAULT STATE. So the shape to recognise is nothing holds the addresses and every unit looks healthy — the estate has no resolver on those addresses and no process is dead.

Address resolution: what makes a client’s mapping follow the address

keepalived sends a gratuitous ARP on entering the master state, and that is what causes a client’s cached IPv4-to-MAC mapping for a floating address to follow it to the new node. That is the daemon’s documented behaviour; no client class has been tested against it.

If a class of client is observed holding a stale mapping after a move — a failover that “worked” while one device class keeps talking to the old node — work in this order:

1. Confirm the new master sent the gratuitous ARP: journalctl -u keepalived shows the Entering MASTER STATE transition that triggers it.
2. Check whether that client’s segment has a switch or AP dropping unsolicited ARP.
3. For unmanaged devices with long cache lifetimes — which this estate has — accept a bounded window, and consider re-issuing GARPs.

Breaking one node’s external DNS path on purpose

scripts/dns-break-external.sh is the committed instrument for inducing real breakage on one node. It drops tcp dport 853 on one named resolver’s output hook, in a table of its own on the inet family, so Blocky’s ControlD DoT egress stalls while .house, VRRP, SSH and tailscale are untouched. Four verbs, one node at a time:

BEFORE=$(scripts/dns-break-external.sh sandiego digest)   # normalised whole-ruleset digest
scripts/dns-break-external.sh sandiego apply
scripts/dns-break-external.sh sandiego status             # both counters, unfiltered
scripts/dns-break-external.sh sandiego revert
scripts/dns-break-external.sh sandiego digest "$BEFORE"   # exits non-zero if it did not come back

A cached answer is served without touching the network stack this break sits on. A name with TTL remaining in Blocky’s cache answers NOERROR at Query time: 0 msec with no egress at all, so the break does not make every lookup fail. No additional port in that ruleset reaches a cached answer, which is why the scope is not widened.

Nothing consumes the result automatically. The peer-aware external tracker that once formed a failover verdict from this breakage is retired. The script induces the fault and reports it, and an operator reads the outcome.

Back-out

Two different halves for two different changes. Take the one that matches what you changed.

Backing out a VLAN cutover

Set the field back. In the Firewalla app, that network’s Primary DNS Server, 192.168.x.5 to 192.168.x.1. That is the whole of it: nothing was disabled and nothing was removed, so nothing needs restoring. The appliance’s redirect rules were never touched and remain in place, and FireMain owns them.

Existing clients keep whatever nameserver they hold until they renew — up to 24 h on the estate’s 86400 lease — so a back-out is not instant for already-leased clients any more than the cutover was. This has not been rehearsed on any VLAN; it is a one-field change, but it is recorded as untested rather than presented as a proven procedure.

Backing out a node — apply first, converge second

For a node stranded by a network change or a fired auto-revert, the hand-run apply script comes first and the converge second. That order is load-bearing: no Ansible verb on the converge path lifts a systemd mask, so a converge alone dies at include position 3. The full procedure is The timed auto-revert.

Things that will bite you — read these before touching the pair

NEVER systemctl reload blocky. Restart it

Blocky does not handle SIGHUP, so the signal takes its default disposition and terminates the process: systemd logs Reloading…, Reloaded…, Deactivated successfully, reports Result=success, and leaves the client tier down. Restart=on-failure does not catch it, because the exit is clean.

ExecReload= is absent from blocky.service for exactly this reason, so systemd refuses the verb outright (Job type reload is not applicable, rc 3). That command is a house-wide DNS outage with every surface reporting success. Use systemctl restart blocky.

networkctl reload STRIPS FOREIGN ADDRESSES — never run it on the node holding the VIPs

keepalived’s floating addresses are foreign to systemd-networkd. networkctl reload removes foreign addresses from a link whose .network changed: add a scratch address to eth0, run networkctl reload, and the address is gone. Running it on the VIP holder pulls the live floating addresses while keepalived goes on reporting MASTER — a silent outage with the daemon insisting everything is fine.

To add or change a leg, the ordering is a requirement rather than a preference:

1. Converge the node that holds no floating address.
2. networkctl reload that node.
3. Move the group (systemctl stop keepalived on the holder — the documented operator act).
4. Repeat on the newly freed node.

A converge reports success and the machine does not change — three ways, on three tiers

All three bite whoever next edits dns_vlan_interfaces.

What	The reading	Consequence
keepalived has no path from a changed config to the running daemon	after --tags keepalived reports changed=1: the new instances are on disk, the process uptime is unchanged, and the addresses held are unchanged	changed=1 on the VRRP tier means the opposite of what it reads as. The rendered config is adopted silently at the next restart or reboot — a VRRP topology change at an unplanned moment
Nothing reloads systemd-networkd, so a NEW leg never comes up	after --tags dns-common reports changed=N: both files rendered, the new eth0.<tag> ABSENT, and the nftables accepts naming it have reached the kernel	the node ends a successful converge admitting DNS on an interface it does not have
Tag filtering silences notified handlers, so a deletion lands with no replacement	a converge reporting failed=0 with no RUNNING HANDLER section at all, having deleted a file and written nothing in its place	the recap is entirely green and the machine is left in a state neither the old nor the new configuration describes

After any change to dns_vlan_interfaces, read the machine, not the recap: ip -br addr for the leg, systemctl show -p ExecMainStartTimestamp keepalived against the config’s mtime, and the addresses actually held.

The rule all three share: failed=0 is not evidence that a mutation ran

Every one of these reports success. What none of them reports is that the thing you wanted to happen happened. The recap counts tasks that executed, not state that changed on the machine.

For the handler case specifically:

• --skip-tags always drops notified handlers. Handlers inherit the tags of the play, so a tag filter that excludes them means a notify: fires into nothing. The converge nonetheless reports changed=N, because the task changed something — the handler that would have made the change effective simply never ran.
• --force-handlers does not rescue it. That flag runs handlers that were notified before a failure aborted the play. It does not run handlers the tag filter excluded, because those were never notified in the first place.
• --list-tasks cannot see it. Handlers do not appear in its output, so the gap is invisible to the one command an operator would reach for to check.
• The correct form skips the two roles by name rather than filtering on always.

The check that actually discriminates: require the RUNNING HANDLER section to be PRESENT in the output. Its absence is the signal. A converge that was supposed to restart or reload something and shows no RUNNING HANDLER section did not do it, whatever the recap says.

Anything you arm with systemd-run must pin AccuracySec

systemd-run --on-active defaults to AccuracySec=1min, so a --on-active=5 timer can fire roughly sixty seconds late. For a safety net that matters twice: a rehearsal checked at 12 s reads as the restore never ran on a restore that is merely pending, and a 900 s restore that fires at 960 is an extra minute of a bad state. Pin --timer-property=AccuracySec=1s on every one.

Verifying a DHCP advertisement: select the packet by its own transaction id

The DHCP OFFER is what proves what a network advertises. Do not read one with a bare grep Domain-Name-Server over a shared capture — two independent things go wrong at once:

1. A shared capture carries other clients, so a grep attributes whatever it finds to your probe.
2. Domain-Name-Server appears in a client’s Parameter-Request LIST, not only in an OFFER, so the pattern matches a request for the option and -A1 then prints a wholly different line class.

So:

• Select the packet by the probe’s own transaction id (or its synthetic 02:00:00:* MAC) before reading any option.
• Assert the message type is Offer, not Request.
• Assert the lease alongside option 6 — a free second opinion on whether the packet came from the network under test.
• Never 2>/dev/null the probe injection. A probe that never went out must be a red, not a quiet empty capture.

Trigger the OFFER with a crafted DISCOVER rather than waiting: passive capture waits on the renewal timer, and the resolver nodes carry no DHCP client tooling.

Tear a capture down BY PID. pkill -f "tcpdump -i ..." matches the invoking ssh shell’s own command line and kills the session you are working through.

tcpdump is declared for these nodes through dns_common_diagnostic_packages, flipped on at the inventory layer for the resolver pair; the role default is [], so no other consumer of dns-common gains it.

The two deliberate exceptions to standing constraints

Both read as violations to anyone who finds them without their reasons, which is how a correct decision gets reverted by a well-meaning reader.

1. A third daemon on a node the standing constraints declare single-purpose

dnscrypt-proxy runs on both resolvers as the node’s own encrypted fallback resolver, bound to 127.0.0.2:53. The standing constraint declares these nodes single-purpose, and this is a second daemon class on them. The reasoning is recorded at the head of roles/dns-common/templates/dnscrypt-proxy.toml.j2:

THE ALTERNATIVE THAT WAS DECLINED, NAMED. The natural home for encrypted transport on a Debian node is systemd-resolved with a DNSOverTLS=yes drop-in. It adds no process — resolved is already installed — and it is exactly what this tier would have been.

WHY IT WAS UNAVAILABLE. templates/dns-stub-listener.conf.j2 records the decision that the resolved unit is NEITHER STARTED NOR MASKED, with four reasons. A DNSOverTLS drop-in that is never read by a unit that never starts is dead configuration; and starting the unit to make it live is precisely the verb that decision forbids.

WHAT WAS BOUGHT FOR THE PRICE. Without this file a resolver node whose client tier is down cannot resolve anything at all — including the tailnet name of the host an operator would use to fix it.

The bound on what it achieves, so the arrangement is not trusted further than it goes. The glibc stub resolver advances to the second nameserver on a transport-level failure only — a refusal, a timeout, or a server error. It does not advance past a successful response code. So the self-first /etc/resolv.conf recovers a node from a dead client tier; it does not recover it from a misbehaving one. A Blocky that is up and answering NXDOMAIN for everything is, to the stub resolver, a working nameserver. That case is out of reach of any two-nameserver resolver file.

2. The retired different-chassis siting requirement

Stated in full, with its reasoning and with the backlogged move’s native-VLAN precondition, at The different-chassis requirement was RETIRED above.

Known risks nobody owns

These are not the deliberate exceptions above. An exception is a decision somebody took with a rationale; what follows is exposure with no detector and no owner. Both read the same way to a reader who finds them cold, which is exactly why they are kept apart — filing one of these under “deliberate” would tell the next person it was handled.

The appliance can put a second DHCP server back on a live leg, and nothing would notice

Every leg’s DHCPv4 Server toggle is unmanaged GUI state. It is not in Git, not in inventory, not in any export, and this repository has no API, no config file and no Ansible path to it. All five Kea legs are served by the pair because a human set five toggles to Off in a mobile app.

A vendor firmware update, a factory restore, or an app-side revert would set one back to On, and that puts a second DHCP server on a live broadcast domain. Two DHCPv4 servers on one segment fight and the client takes the first OFFER, so the leg does not fail — it becomes non-deterministic, handing some clients the appliance’s addresses, options and DNS and others Kea’s.

On this estate, nobody owns detection. No gate in this repository reads that toggle, no timer probes for a second server, and no alert fires on a foreign OFFER. Every instrument this estate has would stay green through it. The subject is vendor drift, not DNS or DHCP; do not file this as a defect against the DHCP tier.

It is sharpest on the three coincident-pool legs, because on those legs a leaked OFFER has no wrong-subnet tell. On lab and iot the two pools are disjoint and adjacent, so a client holding an address from the wrong range is visible to anybody who looks. On mainwireless, main and core Kea’s pool is IDENTICAL to the appliance’s, so a leaked OFFER carries an address that is exactly as plausible as a correct one.

On those three legs the discriminator is lease ORIGIN, never the address. Read the lifetime and the lease table, both of which the appliance cannot forge into agreement:

• Kea’s own lease4-get-all lists it on both nodes — an appliance lease never appears there. The command is Reading Kea lease state from a workstation (DHCP-12) below.
• The valid-lft is Kea’s 43200 seconds against the appliance’s 600. Read the lifetime, not the octets.

The range is unmanaged too, not only the on/off toggle

A leg’s Start IP and End IP are as unmanaged as its toggle, and one leg’s disjointness depends on a range value that exists in no git-tracked file. iot’s End IP is .100 so that Kea takes the space above the appliance’s range; that single value is the whole of iot’s disjointness, and restoring the appliance’s original .249 would re-create the overlap. Nothing verifies it. The recorded values for every leg are in The appliance values each rollback restores below; they are the rollback’s only input and they are also the only baseline a drift check would have.

main rollback needs a DHCP Guard change first

UniFi DHCP Guard vetoes the appliance on Main. The trusted DHCP server list for that network is 192.168.20.6, .7 and .5 — the pair and its VIP — and 192.168.20.1 is deliberately not on it. The appliance cannot serve DHCP on that leg even with its toggle back on; its offers would be dropped.

Any rollback of main must put 192.168.20.1 on the DHCP Guard trusted list FIRST, before the toggle is restored. The list is declared in tf/unifi/networks.tf (unifi_network.main) and holds at most three addresses, so .20.1 replaces an entry: drop the VIP .20.5, since the pair’s offers are sourced from the node addresses .6/.7. Land it as a PR and let main-cluster-unifi apply it, then restore the toggle.

What the namespace closure covers, and what is deferred by decision

The apex is closed, and the scope of the closure is stated narrowly on purpose: every name the pair serves has a declared source. That is narrower than “every name that resolves”, because two classes of name cannot be kept green and no gate pretends to: search-suffix shadows — <public-name>.fzymgc.house, <svc>.svc.cluster.local.fzymgc.house, denver.lan.fzymgc.house — and the appliance’s own writers (see Why does this name resolve when there is no Terraform entry for it?). The DDNS children are outside the closure scope by construction: external-dns’s --exclude-domains=lan.fzymgc.house (argocd/app-configs/external-dns/values.yaml) and D2’s exclusive per-leg grants put them there. So the scope applies to the curated apex; the writers, and how each is fed, are at Adding a name.

How the closure is enforced in the Blocky+PowerDNS chain. PowerDNS is authoritative for fzymgc.house, so an unlisted name is NXDOMAIN and an unlisted type is NOERROR/NODATA from the zone itself; Blocky’s conditional.mapping sends every .house query to pdns, and conditional.fallbackUpstream: false (ansible/roles/blocky/templates/blocky-config.yml.j2, with its rationale beside it; held at converge by roles/blocky/tasks/verify.yml) means neither answer is retried upstream. That IS the closure, and there is no switch left to flip.

Revert. Set fallbackUpstream: true in blocky-config.yml.j2 and flip the hold that pins the value in verify.yml in the same edit, then converge --tags blocky one node at a time (scripts/dns-playbook.sh --limit <node> --tags blocky). There is no inventory variable for it; none exists. This reopens the type-65/HTTPS leak the template’s own comment names, which is why it is a revert and not a knob.

The peer-external verdict path is retired. It published each node’s external-reachability verdict and let the peer read it as a second VRRP tracker. Its writer used unbound-control, and unbound is not on these nodes, so nothing ever published a record. retire and rehome were both put to the operator, and retire was chosen. The VIP is now tracked by chk_house_answer alone, so a node that resolves .house locally keeps its addresses even when it cannot reach upstream.

What the appliance owns

Everything from here down describes the Firewalla. It is where DHCPv6 lives, where DHCPv4 for Guest and Telework lives, and where an un-renewed or statically configured client is answered.

• DHCPv6 for every network, and DHCPv4 for Guest and Telework only. DHCPv4 on the five trusted legs is Kea’s, on the resolver pair. What the appliance sets for a trusted leg is that leg’s DHCPv6 Primary DNS Server field, and there is no agent path to it — it is set in the app. On three of the seven legs — Lab, Guest and Telework — there is no DHCPv6 Primary DNS Server field to set at all: DHCPv6 is Off and none exposes the field. On the two untrusted legs the pair announces RA carrying RDNSS, so the pair — not the appliance — is the v6 DNS announcement there. See IPv6 on the pair.
• IPv4 DNS — none. All seven legs advertise the pair over IPv4 and the appliance owns IPv4 DNS for none of them. What it answers over IPv4 is a client that has not renewed its lease or carries a hand-configured nameserver — a residue, not an ownership. See The DNS fields each untrusted leg carried before its cut, and what a rollback restores above.
• No curated .house zone. The appliance produces no curated zone and serves no .house name from its own store. Terraform writes the curated records into PowerDNS through its API, and lightningstream reconciles the two nodes. The appliance’s own resolution, and every query it forwards, goes to the resolver pair through the WAN DNS field (192.168.20.5 primary, 192.168.20.6 secondary, on both WANs); Unbound is off.
• The per-device names under wireless.lan.fzymgc.house (writer 2 below). These are NOT in the zone the pair serves, and the appliance is the only place they exist. A per-device name that answers through the appliance and is queried against a floating address will not answer. This is a standing gap, not an observed breakage: the pair’s dynamic namespace is populated by Kea’s D2 under <leg>.lan.fzymgc.house, a different name shape.
• No Custom DNS Rules. The app’s Custom DNS Rules screen reads “No Rules”; writer 3 has no entries. There is no router-hosts path and no CLI path to a Custom DNS Rule, so the app UI is the only inventory a rule would ever have.

Architecture — the appliance’s own DNS chain

This section describes what the appliance does when asked. It is the live client path for IPv6 on Lab and for any client, on any leg, that has not renewed its lease or carries a hand-configured nameserver; every cut VLAN’s client asks the pair directly.

Client DNS on the appliance follows a fixed chain, not a flat forward:

1. FW_PREROUTING_DNS_DEFAULT — a Firewalla nat REDIRECT, health-gated by FireMain.
2. firerouter_dns (dnsmasq) on <vlan-gateway>:53 — the front tier. It answers from addn-hosts=/home/pi/.firewalla/run/hosts (per-MAC device names, LocalDomainSensor) and address=/name/ip (Firewalla Custom DNS Rules), and hands everything else to the WAN resolver file /home/pi/.router/run/eth1.resolv.conf, which FireRouter writes from the WAN Primary/Secondary DNS field.
3. The resolver pair — 192.168.20.5 primary, 192.168.20.6 secondary — for everything dnsmasq does not hold itself, .house included. Blocky answers there with the appliance’s forward address in its trusted set, so .house resolves through this path.

• Unbound is off on the appliance. It was the authoritative tier for .house on the box; with it off the appliance resolves .house for nobody, itself included, and forwards instead.
• Client queries land on firerouter_dns (Firewalla’s dnsmasq), bound to the VLAN gateway address the client’s traffic arrives on. dnsmasq is the front tier: it answers from addn-hosts and address= first, and forwards the rest.
  • firemasq.service reporting inactive is expected on a Firewalla Gold and is diagnostically worthless. platform/Platform.js defaults the DNS/DHCP service names to firemasq/firedhcp, but platform/gold/GoldPlatform.js overrides them to firerouter_dns/firerouter_dhcp. So firemasq is simply the wrong unit name on this hardware — its state says nothing whatsoever about whether a :53 listener exists. Check the listeners instead — see Verify the appliance chain is what is running.
  • The client-side verification runbook further down this page tests from a real LAN client because the box does not take the same dnsmasq path a client does.
• FireMain health-gates the redirect on WAN-resolver reachability — and the WAN resolvers are the pair, so a whole-pair outage also takes the redirect down. See the failure-mode section below.
• Eight of the nine WAN-pointed legs share one resolver file, so the WAN DNS field is the one lever that moves the appliance’s fallback upstream; the per-network Primary DNS Server is a DHCPv4 option whose only write surface is the DHCPv4 form, which is not rendered on a leg whose DHCPv4 server is off. Set the WAN DNS field on both WANs, or a failover reverts it silently.

Design and rationale: docs/engineering/specs/2026-07-07-authoritative-house-resolver-design.md (the original chain) and docs/adr/2026-09-09-the-firewalla-leaves-the-dns-path.md (Unbound off, the WAN DNS lever). The reference material for the pair starts at The resolver pair, and what it serves is described at Which tier answers each VLAN.

Verify the appliance chain is what is running

This section verifies the APPLIANCE’s tiers, which is not the same question as “is trusted-VLAN DNS working”. It is the right section when triaging IPv6 resolution on Lab, an un-renewed client on any leg, or the curated zone’s producer. A DNS complaint on Guest or Telework is not an appliance question — both legs are served by the pair. To ask whether the resolver pair is serving, use the one-query discriminator at The appliance is out of the query path for IPv4 only instead.

MUST run these on the Firewalla (ssh pi@192.168.20.1); they are read-only. This is the exact inverse of the client-side verification runbook further down, which MUST run from a real LAN client. The two are not interchangeable: these commands inspect the tiers from the inside — which units are up, what is bound where — whereas the client-side runbook exercises the redirect and the tiers end to end along the path a client actually takes, a path the box itself does not take. Neither one substitutes for the other, and passing one while the other fails is diagnostic in itself.

systemctl is-active firerouter_dns firerouter_dhcp firemasq
# want: active  active  inactive
#   `firemasq` inactive is EXPECTED on a Gold: platform/gold/GoldPlatform.js overrides the
#   DNS/DHCP service names to firerouter_dns/firerouter_dhcp. Its state says NOTHING about
#   whether a :53 listener exists. Never infer the presence or absence of a tier from it.

sudo ss -tlnup | grep :53
# want: dnsmasq bound on each of the NINE VLAN gateway addresses — 192.168.20.1,
#       192.168.24.1, 192.168.40.1, 192.168.79.1, 192.168.80.1, 192.168.200.1,
#       192.168.217.1, 192.168.218.1, 10.230.236.1. No unbound listener on 127.0.0.1:8953:
#       Unbound is off on this box, and a listener there is a re-enabled setting.
#   SEVEN IPv6 listeners are present too — [2600:4040:27d5:5d00::1] through [...:5d06::1],
#   all :53. They are the reason a dual-stack client on Lab resolves through this box; a
#   v4-only reading of this command cannot see the tier that is actually answering such a client.
#   dnsmasq NEVER binds 0.0.0.0: every per-VLAN config sets an explicit `interface=` plus
#   `bind-interfaces`, so a 0.0.0.0:53 listener here is something other than the front tier.

grep -rn 'addn-hosts' /home/pi/.firewalla/config/dnsmasq/
# want: addn-hosts=/home/pi/.firewalla/run/hosts   (writer 2's feed into the front tier)

ls /home/pi/.firewalla/run/hosts/ | wc -l
# want: nonzero — one file per device MAC. The writer adds a file when it sees a device and
#   deletes entries for devices idle beyond a few days, so any exact number is a point in time.

redis-cli get local:domain:suffix
# want: wireless.lan.fzymgc.house — a three-label subdomain, not the apex. The apex value is
#   one `redis-cli set` away and would move every published name back into the apex.

If ss shows no dnsmasq :53 listener, the front tier really is down — and that is a different incident from the WAN-resolver redirect teardown, which was the nat redirect being torn down while dnsmasq itself stayed healthy. Read the failure-mode section below before assuming either.

Why does this name resolve when there is no Terraform entry for it?

Because fzymgc.house has three writers, and only the first is declared in git. This is the single most common source of confusion during triage: a name answers, rg across tf/dns-records/ finds nothing, and the reasonable-but-wrong conclusion is that DNS is lying.

Writers 2 and 3 answer only a client that is answered by the appliance — an un-renewed lease, or Lab over IPv6. Every cut-over leg asks the pair, whose writers and entry paths are at Adding a name; the three-writer framing is kept because the un-renewed client is the residue a triage lands on.

#	Writer	Mechanism	Declared in git?
1	Terraform, tf/dns-records/hosts.yml and records.tf → PowerDNS	the mmianl/powerdns provider writes every entry through the pdns API into powerdns_zone.apex, applied by main-cluster-dns-records on merge	Yes
2	Firewalla LocalDomainSensor (feature flag local_domain)	Writes one file per device MAC under /home/pi/.firewalla/run/hosts/, consumed by dnsmasq via addn-hosts=/home/pi/.firewalla/run/hosts; entries for devices idle more than 3 days are deleted	No — exists only on the box
3	Firewalla Custom DNS Rules	Renders address=/name/ip lines under ~/.firewalla/config/dnsmasq*, read by the same dnsmasq front tier	No — the app UI is the only inventory

Writers 2 and 3 both land in the dnsmasq front tier, ahead of the appliance’s unbound. That is why a name with no Terraform entry can answer, and why unbound looking healthy proves nothing about where an answer came from. Writer 2 publishes far more names than the appliance holds leases for — a file count and a name count are different populations, and neither is bounded by the lease count.

local:domain:suffix auto-publishes every device the Firewalla sees

The redis key local:domain:suffix is set to wireless.lan.fzymgc.house. Every device the Firewalla sees gets auto-published with no declaration anywhere, by writer 2 — under that three-label subdomain, and none directly in the fzymgc.house apex. The apex is therefore not a superset of the declared inventory — but it is one redis-cli set from being one again: with the key at fzymgc.house, writer 2 publishes every device it sees into the infrastructure domain, and the set of names in the apex becomes a superset of the declared inventory by construction, changing as devices come and go.

Worked example — and a deliberate demonstration of the naming hazard below. heimdall.wireless.lan.fzymgc.house resolves to two IPv4 addresses, one per interface (192.168.40.20 on Core, and a second on the IoT leg). Both come from writer 2’s per-MAC files under /home/pi/.firewalla/run/hosts/ — the Pi has two interfaces on two different subnets, and writer 2 publishes each interface it sees with no notion that they are the same host; it also emits a global v6 address and a ULA for the same name. Neither address appears in any git-tracked file in this repo. Searching tf/ for either will always come up empty; that is expected, not a bug.

Four distinct identities answer to the name “heimdall”, and this is precisely the section where a reader is most likely to conflate them. Never use bare heimdall in a networking context:

Identity	Address	Source of truth
The Raspberry Pi, over the tailnet	heimdall.hound-skate.ts.net → 100.68.116.105	tailnet MagicDNS
The same Pi, auto-published on the LAN	heimdall.wireless.lan.fzymgc.house → 192.168.40.20 and its IoT-leg lease	writer 2, on the Firewalla only
A tailscale ACL alias — not DNS at all	192.168.40.20	tailscale/policy.hujson
An unrelated public machine	heimdall.fzymgc.net → 159.203.126.35	tf/cloudflare/dns-fzymgc_net.tf

Note that row 2 returns two addresses for one name — eth0 on the Core VLAN and wlan0 on a separate subnet — and a client gets whichever the resolver hands back. That is the cross-writer collision class this section exists to explain: an undeclared writer, invisible to git, shaping answers that no declared source can reconcile or even enumerate.

Rows 2 and 3 agree because heimdall’s eth0 lease is pinned to 192.168.40.20 by a Firewalla DHCP reservation — the value tailscale/policy.hujson claims. Read that agreement as a coincidence of maintenance, not a mechanism. Nothing keeps the two in step: the ACL alias is hand-maintained and will drift the moment the lease changes without someone editing it.

Landmines

1. Never set redis local:domain:no_forward. Setting it makes Firewalla write a server-uhigh=/fzymgc.house/127.0.0.1#33333 directive (into local_domain_block.conf), pointing the entire suffix at a dead port. That blackholes every .house name in front of unbound — the query never reaches the authoritative tier at all. The failure is nasty precisely because unbound stays perfectly healthy while all internal resolution is gone.
   • Symptom from a client: every .house name fails (SERVFAIL or an empty answer), while external names keep resolving normally.
   • Confirm it: on the Firewalla, redis-cli get local:domain:no_forward — want empty. Then grep -rn 'server-uhigh' ~/.firewalla/config/dnsmasq* — any hit for fzymgc.house is this landmine. dig @127.0.0.1 -p 8953 vault.fzymgc.house A answering correctly while a client gets nothing confirms the break is in front of unbound, not in it.
2. Never conclude “there is no dnsmasq tier” from a service unit’s state. On a Firewalla Gold, platform/gold/GoldPlatform.js overrides the DNS/DHCP service names, so firemasq reporting inactive is expected and carries no diagnostic information whatsoever. Check the listeners instead — sudo ss -tlnup | grep :53 — never the legacy unit name.

Client-side verification runbook

MUST run this from a real client on the LAN, not from the Firewalla box itself — the box does not take the same path a client does.

for n in id llm grafana vault k8s-cluster; do echo -n "$n: "; dig $n.fzymgc.house A +short; done
dig id.fzymgc.house TYPE65 +short          # want: empty (NODATA)
curl -sS -o/dev/null -w"%{http_code} %{remote_ip}\n" https://llm.fzymgc.house/v1/models  # want: 401 via 192.168.20.145

This block alone does not tell you WHICH tier answered. Every name above answers correctly from the appliance and from the pair, so a clean run here is consistent with the cutover having silently reverted. Add the discriminator, and add the address-family check that the block above cannot make:

dig whoami.dns.invalid +short   # want: 192.168.40.6 or .7 — a resolver node.
                                # EMPTY means the appliance answered you: either this VLAN is not
                                # cut over, or this client has not renewed its lease, or you are
                                # being answered over IPv6.

# WHICH server is this client actually using? On macOS:
scutil --dns | grep nameserver | sort -u
# A 2600:… or fe80:… entry ahead of 192.168.x.5 means IPv6. Whether that is the appliance depends
# on the VLAN: Lab is appliance-served over v6; IoT, Core, Main and MainWireless answer from the
# pair's v6 VIP; Guest and Telework have no DHCPv6 DNS field at all, so a v6 resolver they learn
# comes from the PAIR's own RA (RDNSS) and is a fddb:… VIP. Check the VLAN before concluding the
# appliance is in the path.
# On Linux: `resolvectl status` or `cat /etc/resolv.conf`.

Two probes that manufacture a false incident

An NXDOMAIN for a name the zone does not curate is CORRECT. fzymgc.house is a set of declared names, and an undeclared one answers NXDOMAIN with the apex SOA in AUTHORITY — that is the authoritative tier working as designed, not a failed zone publish or a broken delegation. Verify with a known curated name:

dig +short nas.fzymgc.house   # 192.168.20.200 — THIS is the liveness probe.

A workstation with tailnet-claimed routes over estate subnets is not a valid vantage point. If the machine you are testing from runs Tailscale advertising or accepting routes that overlap the estate — 192.168.20.0/22 and 192.168.40.0/22 are both claimed by utun4 on at least one workstation here — then queries aimed at Core addresses transit the tunnel instead of the LAN. What comes back is a non-authoritative answer no estate server actually sent: NODATA for names that resolve perfectly from a LAN client, which reads exactly like a resolver serving an empty zone.

# Before trusting ANY negative result, check whether the query is even leaving on the LAN:
netstat -rn -f inet | grep -E '192\.168\.(20|40)'   # macOS; want the LAN interface, not utun*

If a tunnel interface owns the route, move to a client that is genuinely on the VLAN before concluding anything. On the known dual-homed workstation the corrective routes are added by hand and auto-clear on reboot, so a machine that was a valid vantage point last week may not be one now.

Failure mode: internal names return public NODATA

FireMain health-gates the :53 redirect on plain-UDP reachability of the WAN nameservers (firerouter eth1 nameservers) — a path client resolution never uses. If those checks fail, FireMain removes the redirect rules and clients on the appliance path fall to the FW_PREROUTING_DNS_FALLBACK DNAT toward the same WAN resolvers. With a public resolver in that field every .house name then returns NOERROR with zero answers and the public Cloudflare SOA in the authority section, while external resolution keeps (partially) working — the shape of the WAN-resolver redirect teardown. With the WAN DNS field naming the resolver pair, a public secondary is deliberately not configured: fzymgc.house is a real Cloudflare-hosted public zone that answers those names NODATA, and dnsmasq would cache whichever answer it got.

This is an appliance failure mode: it reaches a client only when that client is answered by the appliance — an un-renewed client, or Lab over IPv6. A whole-pair outage also fails these checks, and the estate has no DNS in that case regardless of what the appliance concludes about its WANs.

Triage, in order:

1. dig vault.fzymgc.house from a client — a public Cloudflare SOA in the AUTHORITY section means queries are falling through to the public zone.
2. sudo iptables -t nat -S FW_PREROUTING_DNS_DEFAULT — an empty chain means FireMain tore the redirect down. The VLAN cutover to the pair did not empty this chain and never will: it changed one DHCP field per network and removed nothing, so nine tcp/udp rule pairs is the healthy state on a cut-over estate too, and an empty chain here is this incident.
3. grep Dnsmasq $(ls -t /home/pi/logs/FireMain*.log | head -1) | tail — look for DNS upstream check, failed to resolve ... on <ip> and remove DNS redirect rules. Under this error loop the FireMain logs rotate in ~2 h, so capture them early if onset time matters.
4. Test each configured WAN nameserver directly: curl -s localhost:8837/v1/config/active | jq '.interface.phy.eth1.nameservers' then dig @<ip> example.com for each.

Fix: restore the WAN nameserver (Firewalla app → Network → WAN → DNS, on both WANs) or restore the pair it names. FireMain re-adds the redirect rules automatically within a check cycle or two — no restarts needed. Then run the client-side verification runbook above.

The resolver pair

The pair is converged on both nodes, elected under keepalived, holding the floating addresses, and serving every declared leg.

The two nodes and their addresses

Held in ansible/inventory/host_vars/{sandiego,portland}.yml. Seven legs per node, one address per leg. The .5 column is a floating address held by whichever node won the election.

Leg	VLAN id	Network	Gateway	VIP	sandiego	portland
core	untagged native	192.168.40.0/22	.1	.5	.6	.7
lab	1000	192.168.200.0/22	.1	.5	.6	.7
main	3000	192.168.20.0/22	.1	.5	.6	.7
iot	3020	192.168.218.0/24	.1	.5	.6	.7
mainwireless	3001	192.168.24.0/22	.1	.5	.6	.7
guest	3100	192.168.79.0/24	.1	.5	.6	.7
telework	3200	192.168.217.0/24	.1	.5	.6	.7

Each leg also carries a ULA /64 (fddb:f665:73f7:N::/64) with the same ::5 / ::6 / ::7 convention. mainwireless carries trusted wireless clients under the same trust policy as Main. Guest and Telework are separate trust tiers: both entries declare trusted: false with a recorded reason, and the appliance keeps DHCPv4 on both segments.

The authoritative loopback listener on the resolver pair is 127.0.0.1:1053, held by pdns. Loopback :53 is not available on these nodes: Blocky binds 127.0.0.1:53 for the health client that performs the resolved-answer failover check, which is also why the systemd-resolved stub listener is taken down. 1053 is unprivileged, so the listener needs no CAP_NET_BIND_SERVICE.

The nine roles and what each one owns

Role	Owns
dns-common	The node baseline every tier stands on: the tagged VLAN legs and their systemd-networkd units, the ip_forward=0 drop-in and the read-only assert over it, the systemd-resolved stub-listener drop-in that frees :53, the nftables ruleset, the RA announcer and the RA-watch detector, the timed auto-revert, and the shared uv-built Python interpreter at /opt/dns-resolver/venv
pdns	The authoritative tier for fzymgc.house and its children: pdns-auth on 127.0.0.1:1053, its LMDB backend, the DNS UPDATE boundary and the per-leg TSIG grants
lightningstream	Multi-primary replication of that LMDB between the two nodes through S3 — no zone transfer and no primary
blocky	The client-facing tier: the pinned blocky binary, explicit per-VLAN listeners, the conditional mapping of fzymgc.house to the authoritative loopback socket, and the Control D DoT upstream
dns-otel-collector	Query-log and journal transport into ClickStack, including the unit list the RA-watch alert depends on
kea	The DHCPv4 serving tier: kea-dhcp4 and its HA pair, the per-leg pools and option surface, and kea-dhcp-ddns (D2)
keepalived	Per-VLAN floating-address failover: the VRRP sync group and the .house tracker
dns-tailscale	The tailnet client on each resolver node, which is what makes remote .house resolution reach each node’s own 100.x address
dns-aaaa-publish	The observed-AAAA publisher for the dynamic zone: the neighbour-table sweep, its TSIG key and its timer

The table follows ansible/dns-playbook.yml: add or remove the play and the row follows.

The dynamic namespace is lan.fzymgc.house, populated by Kea’s D2 — see The DDNS tier — how a DHCP lease becomes a name, and how each half fails at the end of this page.

The interpreter dns-aaaa-publish runs under

/opt/dns-resolver/venv belongs to dns-aaaa-publish alone: it is the only distribution installed there. The converge asserts that the package imports under the interpreter the unit starts, because the alternative failure is invisible — an install can land the code somewhere the unit never starts, every structural check stays green, and the service fails at boot on an ImportError. Remove a package from that venv with uv pip uninstall, never uv pip sync, which equals the lockfile and would take the survivor with it.

The ruleset, and where SSH is accepted

The nftables ruleset is policy drop permitting udp/tcp destination port 53 on every declared leg, udp destination port 67 (the DHCPv4 accept, below), established/related, iif lo, VRRP and the tailnet accepts, and tcp destination port 22 accepted on every leg the inventory marks trusted: true, in both address families. It is a source-restricted SSH accept: not console-only access, and not an empty jump chain. The rule is part of the ruleset with no second flag of its own, so dns_common_nftables_enabled is the single switch and flipping it does not remove inbound SSH. The role installs the include into /etc/nftables.conf that makes nftables.service read the rendered ruleset, appended below Debian’s flush ruleset because an include above it is read and then silently wiped. Console-only was rejected because it would turn every scripts/dns-playbook.sh converge into console work. Guest and Telework get no SSH path at all. They are the two legs the inventory marks trusted: false. Source-address matching is spoofable and is not an authentication control. It reduces exposure, and SSH’s own authentication is what protects the host.

The two source sets are built differently. IPv4 reads the role variable dns_common_ssh_source_set_v4, because 192.168.16.0/20 covers both the Main and the MainWireless leg and nft refuses overlapping intervals inside one anonymous set. IPv6 derives each trusted leg’s own ULA /64, and those cannot overlap.

Leg	v4 source entry	v6 source entry
core	192.168.40.0/22	fddb:f665:73f7:6::/64
main, mainwireless	192.168.16.0/20	fddb:f665:73f7:1::/64, fddb:f665:73f7:7::/64
lab	192.168.200.0/22	fddb:f665:73f7:4::/64
iot	192.168.218.0/24	fddb:f665:73f7:3::/64

Reach a node at any trusted leg address. Until this widening the node answered on its Core address alone, and a bare ssh sandiego hung: the name publishes an address on every leg, the client walks that list, and a dropped address costs a full TCP timeout before the next one is tried. sandiego.core.fzymgc.house resolves to the Core address alone and is the form to use when one address is wanted.

ansible/inventory/hosts.yml still gives each host its own Core address as ansible_host and never the .5 VIP, so dns-playbook.sh takes one of the permitted paths. A session opened from Main to a node’s Core address is asymmetrically routed, reasoned from net.ipv4.conf.all.rp_filter = 2 and not observed: the reply egresses the directly-connected Main leg instead of retracing its Core ingress, so the Firewalla never sees the return half and its connection state for that session will be empty.

Ping, neighbour discovery, and what ICMP the resolver answers

echo-request is accepted on every declared leg in dns_vlan_interfaces — all seven — in both address families, and both rules are source-unscoped. The invariant is ping surface == DNS surface, held by the two echo rules sharing the port-53 accepts’ loop expression in ansible/roles/dns-common/templates/nftables-dns.conf.j2 and by nothing else — it is true by construction and ungated. Echo is deliberately NOT accepted on the tailnet leg.

The SSH scope does not move with it, and must not be “restored to symmetry”. Echo is accepted on all seven legs and SSH on the five trusted ones. Ping and SSH do not share a scope, and the template says so beside the rules.

Neither echo rule is source-scoped, and the v6 rule cannot be. docs/reference/network.md records a ULA /64 for Main, Guest, IoT, Lab and Telework and none for Core, so there is no Core prefix to name; and narrowing either rule to a source set would reintroduce exactly the drift the ping-equals-DNS invariant removes. The ruling is written beside the two rules in the template.

IPv6 Neighbour Discovery, MLD and the ICMP error types are accepted unconditionally, above the invalid-state drop, and that placement is load-bearing. NDP is not valid-new for nf_conntrack_icmpv6, so types 133–137 arrive marked INVALID; an accept placed below ct state invalid drop is dead code that parses, loads, and reads as though it works. Without these, the node becomes IPv6-unreachable from any neighbour that must resolve it — one RA lifetime after the flag is flipped, on every leg at once, with a green converge behind it. Both MLD versions are named: the v1 types (130/131/132) and the MLDv2 report (143) are different types, and modern Linux speaks v2; carrying only v1 lets an MLD-snooping switch age this node’s multicast listener state out. packet-too-big carries IPv6 PMTU discovery, without which large DNS answers blackhole in a way that reads as an upstream fault. The DHCPv6 client port (udp 546) is accepted from link-local sources only, so an RA that sets the managed or other-config flag does not leave the node unable to hear the reply.

None of the ICMP behaviour above has been exercised deliberately. The rules are loaded in both kernels; loaded is a different claim from tested.

What the dynamic namespace publishes — A from leases, AAAA from observation

The dynamic namespace beneath lan.fzymgc.house carries A and PTR records written by Kea’s D2 from leases, and AAAA records written by dns-aaaa-publish from the resolvers’ own IPv6 neighbour tables. A resolver learns a client’s real address the moment that client speaks IPv6 to it, and ip -6 neigh reports it — no probing, no derivation. Joining that table to the Kea lease table on MAC address yields the address the device is actually using, which is what a published AAAA has to be; deriving one instead would be arithmetic on a false premise, since Windows, Android, macOS/iOS and Linux under NetworkManager all use RFC 7217 stable-privacy addresses that are not derived from the MAC. See The DDNS tier at the end of this page for how it publishes and how it degrades.

Global (2600:) addresses are deliberately out of scope. The ISP-delegated prefix is re-delegated periodically on this estate, so a published GUA becomes a lie at the next re-delegation. Publication is scoped to the site ULA fddb:f665:73f7::/48, a /64 per VLAN.

Blocky’s upstream is dual-family on purpose, and the IPv4 entry is an anycast literal

upstreams.groups.default carries two entries that reach ONE endpoint in two address families, and the second one is a deliberate correction rather than redundancy for its own sake:

default:
  - "tcp-tls:mt3vg25yld.dns.controld.com:853"
  - "tcp-tls:76.76.2.22:853#mt3vg25yld.dns.controld.com"

Why the second entry exists. Blocky resolves a named upstream and pins the address it gets. With the name as the sole entry it pins the AAAA, [2606:1a40::22]:853, and has no IPv4 fallback — every attempt of every query goes to that one address. The estate’s IPv6 LAN→internet forwarding is a black hole (a router-owned fault), so with a single named entry every non-.house query SERVFAILs while .house keeps answering from the authoritative tier and every structural gate stays green. A resolver whose only upstream is reachable in one address family is a single point of failure whichever family that is, and it stays one after v6 forwarding is fixed — which is why this entry is not removed when v6 forwarding returns.

Expect one WARN at every Blocky start while v6 forwarding is broken, and do not read it as a failure: the line reads WARN upstream: initial resolver test failed, names the tcp-tls:mt3vg25yld.dns.controld.com entry with its IPv6 literal, and ends connect: network is unreachable. That is the named entry’s start-up probe failing over dead IPv6. The daemon starts, and the query log attributes answers to the entry that actually served them — response_reason=RESOLVED (tcp-tls:76.76.2.22) is the healthy steady state.

76.76.2.22 is an ANYCAST literal and Control D may re-point it without notice. That cost was accepted deliberately over connectIPVersion: v4. Read exactly what a re-point does and does not break:

It cannot leak	#mt3vg25yld.dns.controld.com pins TLS verification to the Control D hostname, so an address that stops being Control D fails the handshake. The entry errors; it never hands internal query names to whoever holds the address next.
It fails silently	The named entry goes on answering, so the resolver goes on working while the estate quietly returns to the single-family exposure this entry exists to remove. Nothing goes red — not even the runtime external-resolution gate, which asks whether the node resolves and gets a yes from the other entry.
The symptom	Blocky upstream-error lines in the journal naming 76.76.2.22, and nothing else.
The drift check	dig +short mt3vg25yld.dns.controld.com A. If it no longer prints this value, update the literal.
The repair is not deletion	Deleting the entry to make a mismatch go away restores the single-family exposure exactly, and the next thing that notices is a client.

The value is set once, at ansible/inventory/group_vars/dns_hosts.yml (dns_upstream_v4_address), and the template derives the common name from dns_upstream_host rather than restating it, so the two entries cannot drift into naming different endpoints.

Ask the node itself, never the control node — the Firewalla DNATs LAN port-53 to its own dnsmasq, so a probe from the control node measures the router:

uv run scripts/dns-external-resolution.py \
  --ssh fzymgc@192.168.40.6 --resolver 127.0.0.1 --decoy 192.168.40.99

The --decoy probe is mandatory and aborts the run as GATE INVALID (exit 4) if anything answers for an address where nothing exists. A green with a failed or absent decoy is not a pass — it means nothing was measured.

Running a play, and how one-node-per-window is expressed on the command line

Use the wrapper; it materializes credentials and pins the inventory:

scripts/dns-playbook.sh --limit sandiego --tags pdns-verify --check --diff  # dry-run, one node
scripts/dns-playbook.sh --limit sandiego --tags blocky                      # one node, one tier
scripts/dns-playbook.sh --limit portland --tags kea -vv                     # verbose

Ansible does not fail on an unmatched tag — it selects zero tasks, runs nothing, and prints a green PLAY RECAP, so a mistyped --tags value is a success report for a converge that did nothing at all. Check the tag against ansible/dns-playbook.yml before trusting a green recap.

ansible/dns-playbook.yml is nine separate plays, one role each, every role individually tagged — dns-common, pdns, lightningstream, blocky, dns-otel-collector, kea, keepalived, dns-tailscale, dns-aaaa-publish, in that dependency order — and most roles additionally expose a -verify tag selecting only their read-only assertions.

The tag structure is a safety mechanism, not a convenience. Never change both resolver nodes in the same window: the estate has exactly two, they are the whole serving plane, and a change applied to both at once leaves no surviving node to fall back to. --limit sandiego --tags blocky is the expressible form of that rule. Collapse the nine plays into one, or drop the per-role tags, and the rule stops being enforceable from the command line and becomes a note in a document.

--limit reaches the node it names because ansible/inventory/hosts.yml gives each host its own core address (.6/.7) as ansible_host, never the .5 VIP. A VIP target would route to whichever node holds it — which is the other node exactly when the first one is being worked on, the failure mode the rule exists to prevent, reintroduced by the transport.

The timed auto-revert — arm, disarm, fire, and how to get a node back

dns-common ships a dead-man switch. dns-common-revert.timer starts dns-common-revert.service a fixed interval after it is armed, and that service unmasks and starts NetworkManager, masks and stops systemd-networkd, and stops nftables — putting the node back on the network manager it was reachable through before the converge touched it. Arm it before a change that could strand you; disarm it once you have confirmed you can reach the node.

MUST be run against one node at a time, and these commands are NOT read-only — the exact inverse of Verify the appliance chain is what is running above. The arm schedules a network teardown, and the session that issues the hand-run apply in the recovery below does not survive it; the script says so in its own header. Reconnect on the node’s Core address. Every trusted leg now accepts SSH, but Core is the untagged parent leg and is the address the rest of this runbook uses.

Set these two first. Every command in this section reads them, including the recovery.

NODE=sandiego              # the ONE node being worked on — sandiego or portland, never both
NODE_IP=192.168.40.6       # that node's CORE address: sandiego .6, portland .7
# NEVER the .5 VIP, which routes to whichever node holds it — the other node exactly when this
# one is being worked on (see `--limit` above).
# The Main address 192.168.20.x answers too. Use Core here, as the rest of this runbook does.

Arm it.

scripts/dns-playbook.sh --limit "$NODE" --tags dns-common \
  --skip-tags dns-common-venv,dns-common-daemons \
  -e dns_common_revert_enabled=true -e dns_common_revert_delay=600s
# want: PLAY RECAP ... changed=2  unreachable=0  failed=0
#   The two changed tasks are `Render the revert timer unit` and `Set the revert timer's
#   state from its guard`. Nothing else moves.
#   `revert-arm.yml` is the LAST include, after verify.yml — so a converge that fails earlier
#   never arms anything.
#   600s is the PRODUCTION delay. A shortened value is a DEVIATION from the production shape —
#   it stays rendered on disk until a converge WITHOUT that `-e` puts 600s back.
#   Pass both values as `-e`, never by editing dns_hosts.yml: an armed flag left `true` in a
#   commit is a self-inflicted outage on somebody else's next converge.

sudo systemctl start dns-common-revert.timer
# want: rc=0, nothing printed.
#   The hand re-arm — a second window against the unit already on disk, with no second
#   converge. OnActiveSec counts from THIS start, not from the converge that rendered it.

Disarm it.

sudo systemctl stop dns-common-revert.timer
# want: rc=0, nothing printed. Afterwards: ActiveState=inactive, SubState=dead.
#   A `stop` clears the RUNTIME half only. The unit-file half stays `enabled` with whatever
#   delay is rendered, so a REBOOT re-arms it. Only a converge with `dns_common_revert_enabled`
#   back at its `false` default clears that, and it is what makes the disarm survive a boot.

Type the disarm into an ALREADY-OPEN session, and put the reading and the stop in one invocation rather than in two round trips: a sequence that round-trips between steps can lose the whole window.

Read it — and read SubState, never is-active.

sudo systemctl show dns-common-revert.timer \
  -p ActiveState -p SubState -p LastTriggerUSec -p NextElapseUSecMonotonic
# want (armed): ActiveState=active  SubState=waiting  LastTriggerUSec=            (empty)
# want (fired): ActiveState=active  SubState=elapsed  LastTriggerUSec=<a stamp>
#   KEEP THE KEYS. `--value` strips them, and systemd emits properties in ITS OWN internal
#   order rather than in `-p` flag order — so values land against the wrong labels, silently,
#   with correct data and no error at all.
#   `NextElapseUSecRealtime` is EMPTY on this timer and that is correct rather than a sign it
#   is unarmed: `OnActiveSec=` is a monotonic directive, so only the monotonic field is filled.

Timer state	SubState	LastTriggerUSec	ActiveState
Armed, has not fired	waiting	empty	active
Fired, not stopped	elapsed	set	active
Fired then stopped	dead	empty	inactive
Disarmed, then stopped	dead	empty	inactive

The last two rows are byte-identical, which reads like a bug and is correct. systemctl stop destroys SubState, LastTriggerUSec and both NextElapse fields, so after any stop a timer that fired and a timer that was disarmed in time are indistinguishable from the unit object. journalctl -t dns-common-revert is the only instrument that discriminates after the fact, so capture it BEFORE you disarm. is-active and is-enabled read active/enabled in both the armed and the fired-but-not-stopped states and are evidence of neither.

Expect the fire LATE rather than on time. AccuracyUSec=1min is systemd’s default and this unit sets no AccuracySec=, so the real window is OnActiveSec through OnActiveSec + 60s — 600–660 s in production. list-timers NEXT is a lower bound, so a reading taken at NEXT + 15 s can catch a timer that has not fired and misreport it as “did not fire”.

What a complete fire writes.

journalctl -t dns-common-revert -o short-iso --no-pager | tail -7
# want: SEVEN lines — one opening line, five ACTION-OK lines in manifest order, one decision:
#   ... restoring the reachable pre-transaction state (no-act mode: 0)
#   ... ACTION-OK unmask NetworkManager.service exited 0
#   ... ACTION-OK start NetworkManager.service exited 0
#   ... ACTION-OK mask --now systemd-networkd.socket exited 0
#   ... ACTION-OK mask --now systemd-networkd.service exited 0
#   ... ACTION-OK stop nftables.service exited 0
#   ... DECISION=REVERT-COMPLETE every action in the manifest exited 0
#   Assert this as a DELTA of +7 from a boundary taken BEFORE arming, never as an absolute:
#   the journal is not rotated between runs.
#   The socket is masked at index 2, AHEAD of the service at index 3. Read this BY INDEX — a
#   set comparison over the five actions is green in either order, and the order is the point.

The service’s exit status is a contract: 0 when every action was performed and each returned 0, 64 for an unrecognised argument (refused before anything is evaluated, so a mistyped no-act flag cannot select acting mode), 78 for an empty systemctl command refused inside the runner, and otherwise the number of actions that did not succeed, each named in the journal under DECISION=REVERT-INCOMPLETE.

Stopping systemd-networkd does NOT release its addresses or destroy its VLAN netdevs, and that is the most counter-intuitive fact on this page. After a fire the designed addresses and the VLAN netdevs remain present and UP, and NetworkManager adds one DHCP address of its own on top. So a reverted node stays reachable at its inventory address, and the DHCP address is additional rather than a replacement. The session you recover through is riding networkd’s surviving static address, so anything that deletes addresses on eth0 can cut the connection you are working through — which is why the recovery below touches no addresses at all and the residual lease is left to expire.

A fired revert leaves the node with NO packet filter. nftables.service is stopped and the table count drops to zero, so the node is default-allow on every VLAN it holds a leg on until a converge restores the ruleset. Close it deliberately; do not leave it open while triaging something else.

The converge will REFUSE on a reverted node, and the refusal is deliberate.

scripts/dns-playbook.sh --limit "$NODE" --tags dns-common \
  --skip-tags dns-common-venv,dns-common-daemons
# want, against a REVERTED node — it stops at include position 3:
#   TASK [dns-common : Set systemd-networkd's enabled state from its guard]
#   Unable to enable service systemd-networkd.service: Failed to enable unit:
#     Unit /etc/systemd/system/systemd-networkd.service is masked
#   PLAY RECAP ... unreachable=0  failed=1
#   This is the ONLY detection surface a fired revert has. It MUST NOT be "fixed" by adding
#   `masked: false` to that task: unmasking from there lifts the revert's protection silently,
#   on a node whose network is by definition already wrong.

Get the node back — apply first, converge second.

# 1. Lift the masks by hand. Detached, because the session will not survive it.
ssh -o ControlPath=none fzymgc@"$NODE_IP" \
  'sudo -n systemd-run --collect --unit=dns-common-network-apply-recover \
     /usr/local/sbin/dns-common-network-apply.sh'
# want: rc=0 and `Running as unit: dns-common-network-apply-recover.service`

# 2. Reconnect on CORE, then confirm both masks are gone. Count; never a bare `ls`.
ssh -o ControlPath=none fzymgc@"$NODE_IP" \
  'ls -1 /etc/systemd/system/systemd-networkd.service /etc/systemd/system/systemd-networkd.socket \
     2>/dev/null | wc -l'
# want: the literal 0. A bare `ls` on an absent path exits 2 with a message on stderr, which is
#       byte-identical to what a typo or a wrong directory produces — count instead.

# 3. Now converge. This re-enables networkd, restarts nftables, restores OnActiveSec=600s and
#    disarms the timer on both axes.
scripts/dns-playbook.sh --limit "$NODE" --tags dns-common \
  --skip-tags dns-common-venv,dns-common-daemons
# want: PLAY RECAP ... unreachable=0  failed=0, and the task that reported `fatal` above now
#       reports `ok: [$NODE]`.

# 4. Confirm the production delay is back.
ssh -o ControlPath=none fzymgc@"$NODE_IP" \
  "systemctl cat dns-common-revert.timer | grep -E '^OnActiveSec='"
# want: OnActiveSec=600s   (grep, not rg — the node has no ripgrep)

Check journalctl -t dns-common-network-apply first — the apply script logs every action with a timestamp, in manifest order, and a complete run ends DECISION=HANDOVER-COMPLETE. It is the only place the two unmask actions are visible as having happened.

The order is apply first, converge second, and it is load-bearing rather than stylistic. A converge alone cannot recover a reverted node, because no Ansible verb on that path lifts a systemd mask — which is why step 3 dies at include position 3 when step 1 has not run. The apply script’s two unmask actions sit at manifest indices 0 and 1, ahead of the sever at index 3, and that placement is what makes the script re-runnable on a node that has already reverted.

A residual NetworkManager lease needs no step. NetworkManager’s DHCP lease release is not uniform — a clean systemctl stop inside the apply may release it, or the lease may persist with a countdown the kernel honours — but neither outcome asks anything of the operator: nothing is left behind after roughly a day, and the converge in step 3 does not depend on it. Where it matters: the lease lands in Core (192.168.40.0/22) on the same eth0 as the static .6, so while held it un-pins source-address selection for Core destinations and is not matched by the address-specific ruleset the converge restores. If you are chasing odd source-address or firewall behaviour on a freshly recovered node, look for it as a count, never as an error:

ssh -o ControlPath=none fzymgc@"$NODE_IP" 'ip -o -4 addr show dev eth0 | wc -l'
# want: the literal 1 — the static Core address alone.
#   2 means the lease is held. It will expire on its own; delete it only if you have a
#   reason not to wait, and delete the address the read above actually shows.

Three /dev/null mask symlinks in /etc/systemd/system/ is the CORRECT steady state, not residue. NetworkManager.service, unbound-resolvconf.service and userconfig.service are all masked on a healthy resolver, and neither networkd unit is among them. To assert the directory is unchanged, compare a name-only digest — ls -1 /etc/systemd/system/ | LC_ALL=C sort | sha256sum. An ls -la | sha256sum embeds mtimes and reports a false mismatch after any re-render, which reads exactly like a real regression.

ULA prefix announcement (the resolver pair)

Both resolvers advertise the ULA prefix on every in-scope leg, permanently, at a finite lifetime. Committed as dns_common_ra_announce_enabled: true in ansible/inventory/group_vars/dns_hosts.yml.

What it is

dns-common-ra-announce.service runs radvd as a standing daemon on both nodes, reading a role-owned config at /etc/dns-common/radvd-announce.conf rendered from dns_vlan_interfaces — one interface stanza per in-scope leg, five legs, Lab included. Each leg advertises its own fddb:f665:73f7:N::/64:

Directive	Value	What it buys
AdvOnLink	on	the client forms an on-link route to that leg’s ULA VIP
AdvAutonomous	on	the client may configure an address from the prefix. Load-bearing: with it clear a client silently ignores the whole prefix-info option (RFC 4862 §5.5.3(a)) and never reaches the two-hour clamp
AdvPreferredLifetime	3600	finite — an operator ruling, not radvd’s default
AdvValidLifetime	7200	finite — the ceiling on how long a withdrawal takes
AdvDefaultLifetime	0	neither resolver becomes a default router. Observed on the wire as router_lifetime=0
MinRtrAdvInterval / MaxRtrAdvInterval	200 / 600	radvd’s own defaults; the unsolicited cadence is a backstop, not the join path
RDNSS	the leg’s vip6	gated by dns_common_ra_announce_rdnss_enabled alone, never the prefix. The appliance’s per-network DHCPv6 Primary DNS Server field advertises the same vip6 on the four non-Lab legs, so the two sources agree (RFC 8106 §5.3.1); on Lab the pair is the only source

The pidfile is /run/dns-common-ra-announce.pid — see below, it is not a detail.

Without this announcement nothing on the trunk advertises the prefix the floating addresses live in, and a client handed a ULA VIP but never told the prefix is on-link has no route to it.

Why a standing daemon rather than bursts

Only a listening daemon answers a Router Solicitation inside RFC 4861’s MAX_RA_DELAY_TIME. That is both how a fresh client joins in under a second and the only reason scripts/solicit-ra.py --router is a valid instrument at all — it solicits. A bounded-burst design is not listening between windows, so a client that joins between bursts waits, and every verification below would be measuring the burst schedule instead of the estate.

Why both nodes

• Prefix information is idempotent — two sources advertising one prefix is harmless per RFC 4861.
• Each node hears the other, so the “a node does not receive its own multicast” asymmetry has nowhere to live in the announce path.
• The estate survives one node down with zero failover logic and no coupling to the VRRP path.

How to read whether it adopted

A zero exit from systemctl start is not the proof — same rule, same reason, as the appliance adoption test further up this page. The proof is ExecMainStartTimestamp in wall-clock epoch seconds, strictly later than the config file’s mtime in epoch seconds.

# On a resolver node. grep, not rg — these boxes have no ripgrep.
SVC=dns-common-ra-announce.service
CONF=/etc/dns-common/radvd-announce.conf
TS=$(systemctl show "$SVC" -p ExecMainStartTimestamp --value)
echo "START_EPOCH=$(date -d "$TS" +%s)"
echo "CONF_MTIME_EPOCH=$(stat -c %Y "$CONF")"
systemctl show "$SVC" -p ActiveState -p SubState -p MainPID
radvd -c -C "$CONF" -p /run/radvd-configtest.$$.pid ; echo "configtest=$?"

Want: START_EPOCH strictly greater than CONF_MTIME_EPOCH, active / running, configtest 0.

Never compare ExecMainStartTimestampMonotonic against an mtime. It is microseconds-since-boot. Raw it is ~338× larger than an epoch mtime and the comparison passes unconditionally, for every config, forever; divided by 1e6 it is ~1.7 × 10⁹ smaller and fails spuriously, for every config, forever. Two opposite wrong answers from one field, neither depending on whether adoption happened. Both sides of the comparison must be wall-clock epoch seconds.

How to read it on the wire

From the repo root, one dispatch per node, each attributed to the peer — a node does not receive its own multicast, so a node asked to attribute to itself reads UNINTERPRETABLE on all five legs:

scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.script -a "../scripts/solicit-ra.py --router fe80::8aa2:9eff:fede:ae7c"   # attributes to portland
scripts/dns-adhoc.sh portland -b -m ansible.builtin.script -a "../scripts/solicit-ra.py --router fe80::8aa2:9eff:fed4:6167"   # attributes to sandiego

Those two link-locals are each node’s, identical across all five legs, recorded beside dns_common_ra_watch_expected_lladdrs in ansible/inventory/group_vars/dns_hosts.yml. That file is their home; do not re-derive them.

Three dispatch traps:

• The ../ prefix is load-bearing. ansible.builtin.script resolves a relative path against the playbook directory (ansible/), not the repo root. Without it the module cannot find the script.
• -b is required — the script opens a raw ICMPv6 socket.
• A non-zero exit prints the host FAILED in red, and that is the gate working, not the dispatch breaking. Read rc and the SUMMARY: line. Exit 1 is measured and absent; exit 2 is uninterpretable and no verdict is claimed for the set. Exit 2 dominates exit 1 deliberately: an incomplete measurement outranks any conclusion drawn from the part that did report.

Want, per node: rc 0 and SUMMARY: 5 advertising, 0 not advertising, 0 uninterpretable, of 5 legs.

A leg with zero frames from the named router reads UNINTERPRETABLE, never “no ULA”. The script prints the source of every frame it collects, so a leg answered only by something else names that something else beside the verdict.

The rule that matters most: a per-node count must be scoped to that node’s RA source

The exit status is a prefix-PRESENCE verdict only. A claim about the lifetimes must be read from the printed vlt= / plt= fields with an anchored count — never inferred from rc.

Because BOTH nodes announce, an unscoped count measures THE LINK, not THE NODE. A Router Solicitation is answered by every router on the link, so the peer’s five advertisements are in the captured output either way:

# WRONG for a per-node claim — source-blind: it counts the peer's frames too.
grep -o 'vlt=7200 plt=3600' sandiego.out | wc -l

# RIGHT — count only inside blocks whose PRINTED SOURCE is the node under test.
# grep and awk, not rg: this form is meant to run anywhere, including on a node.
awk -v tgt=fe80::8aa2:9eff:fed4:6167 '
  /^ *from /  { intgt = ($2 == tgt); next }
  /^=== /     { intgt = 0; next }
  intgt && /vlt=7200 plt=3600/ { n++ }
  END { print n+0 }
' sandiego.out

Want: exactly 5 — one per leg. Keep a sanity count (grep -c 'vlt=') beside it: without it, a zero cannot be told apart from a field that never appears. Anyone re-verifying a single resolver must scope the count to that node’s printed RA source, or the check is green regardless of whether that node advertises anything.

Also want, inside the target’s blocks: L=1 A=1 on every prefix-info line, router_lifetime=0 on every block, and an rdnss line carrying the leg’s vip6.

How to check containment, and the count that is wrong

Both nodes in one invocation:

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell -a "systemctl is-enabled radvd.service ; ps -o pid,unit,cmd -C radvd --no-headers || echo 'no radvd process'"

Want: systemctl is-enabled radvd.service prints masked on both nodes — Debian’s own unit, masked whenever the package is present, keyed on the package and never on the arm flag. And every row of ps shows a role unit in the UNIT column (dns-common-ra-announce.service or dns-common-ra-deprecate.service).

Two concurrent radvd is COMPLIANT — the standing announcer plus a bounded deprecation window. A check written as a process count equal to 1 is red on a correct estate, and a count equal to 2 is green on a node running one announcer that forked. Count by unit, never by process name. radvd forks, so even a single healthy announcer shows two rows sharing one unit.

The pidfile, and why it is not a detail

The announcer runs with -p /run/dns-common-ra-announce.pid. radvd 2.20 takes an exclusive non-blocking lock on its pidfile above the daemonize branch, and the bounded dead-prefix emitter (next section) passes no -p, so it takes the compiled-in default. If the announcer ever used that default it would hold the lock from boot, the emitter’s OnFailure=-triggered start would exit 255, dns-common-ra-deprecate.service would land failed, and the dead-prefix withdrawal would never be sent — silently, with every wire reading green.

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell -a "ls -l /run/dns-common-ra-announce.pid /run/radvd.pid /run/radvd/radvd.pid 2>&1 ; systemctl is-failed dns-common-ra-deprecate.service ; systemctl show dns-common-ra-deprecate.service -p Result -p ExecMainStatus"

Want: the announcer’s own pidfile present; the two compiled-in defaults absent; the emitter not failed, with Result=success and ExecMainStatus=0. inactive (dead) is the correct resting state for a bounded one-shot with no pending request. The journal signature of the collision is a message about being unable to lock the pid file with a resource-temporarily-unavailable errno:

# On a resolver node. grep, not rg — these boxes have no ripgrep.
journalctl -u dns-common-ra-deprecate.service --no-pager | grep -i 'lock pid file'

The same lock is why radvd -c needs its own pidfile: a bare config test on a node with a running announcer fails for a reason that has nothing to do with the config. Pass a scratch path, as the adoption read above does; the converge-time validate: uses a fixed scratch path for the same reason.

How to disarm, and what withdrawal means

Set dns_common_ra_announce_enabled: false and converge, one node per window:

scripts/dns-playbook.sh --limit portland --tags dns-common-ra-announce
# verify, then the same command with --limit sandiego

The arm is two-sided — ra-announce-arm.yml derives both enabled and state from that one flag — so the converge disables and stops the unit rather than merely declining to start it. The same asymmetry is why the flag must stay committed true: with it false, any routine --tags dns-common converge stops both announcers, and nothing errors.

There is no withdrawal code and none is wanted. The prefix ages out within the advertised valid lifetime — at most two hours — with no residue. That finite lifetime is the withdrawal mechanism, and it is why the lifetimes are finite in the first place.

radvd’s shutdown-deprecation prefix option is OFF and must stay off. Upstream states it is only safe when a single router announces the prefix onto the link; here two do, so enabling it would make restarting one announcer deprecate the prefix estate-wide. With it off, a restart is a sub-second RA gap and nothing more.

How to withdraw the RDNSS advertisement (and leave the prefix alone)

This withdraws the recursive-DNS advertisement only. Use it when the pair is advertising DNS over IPv6 and you need that to stop; use “How to disarm” above when you need the whole announcer to stop.

All three steps are required. A converge alone changes nothing on the wire.

# 1. Override the arm flag OFF, one node per window.
#    Note the JSON form: see the warning below before using -e key=value.
scripts/dns-playbook.sh --limit portland --tags dns-common-ra-announce \
  -e '{"dns_common_ra_announce_rdnss_enabled": false}'

# 2. Confirm the rendered config actually lost the blocks (expect 0; it is 5 when armed).
scripts/dns-adhoc.sh portland -b -m ansible.builtin.shell \
  -a 'c=$(grep -c "^ *RDNSS " /etc/dns-common/radvd-announce.conf || true); echo "rdnss_block_count=$c"; test "$c" -eq 0'

# 3. RESTART THE ANNOUNCER. Without this the wire does not change.
scripts/dns-adhoc.sh portland -b -m ansible.builtin.shell \
  -a 'systemctl restart dns-common-ra-announce.service'

To restore, run the same three steps without the -e override — the committed inventory value supplies the armed truth — and restart again. The restart is required in both directions.

Why step 3 is not optional. The conf-render task carries no notify:, the announcer unit declares no ExecReload=, and no role handler names the announcer — all three by design. So after step 1 the file on disk is correct and the running radvd is serving the previous config out of memory. A reader who skips step 3 will read an unchanged wire and conclude the lever is broken.

Use the JSON -e form, not -e key=value. ansible-playbook -e key=value passes the value as a string, and a non-empty string is truthy in Jinja. The guard applies | bool, so both spellings work and the JSON form is belt-and-braces.

The prefix survives the withdrawal, and that is the point. Only the RDNSS block is gated; the prefix block is unconditional. After the withdraw restart every leg reads rdnss (0): NONE with prefix-info (1): <prefix>/64 L=1 A=1 vlt=7200 plt=3600 unchanged. This matters because clients need that prefix to form an on-link route to the VIP at all — a withdrawal that took the prefix with it would remove the route to the very address it stopped advertising.

This lever is coupled to nothing in DHCPv4 and rolls back independently of any v4 change.

scripts/solicit-ra.py’s exit code covers the PREFIX, not the RDNSS. Its verdict is computed from prefix presence only; the rdnss (n): <addr> line is printed but never gated. Read the printed option, not rc. Exit 2 dominates and means a leg was uninterpretable — silence from the target is never absence.

How to restart

Plan-sequenced, never a role handler — the unit deliberately carries no Restart= policy, and the role installs no handler, following the same ruling as keepalived. A restart really is only a sub-second RA gap, but an automatic restart-on-change would move VIPs on any VRRP-tier converge estate-wide, and a second exception would erode the rule that a converge must never be the thing that moves the serving plane. A radvd that has exited is a failed unit an operator reads, which is the intended report.

These commands run where they are pointed

Every host-side command in this section uses systemctl, ps, ip, stat, date, awk, grep or journalctl. None uses ripgrep — the resolver nodes do not have it, exactly as the appliance does not. See “The appliance and the resolver nodes have no ripgrep” above.

Dead-prefix deprecation — when the ISP re-delegates

A re-delegation leaves the old prefix live on every client, and the appliance will not withdraw it. When the ISP re-delegates, FireRouter rewrites every bond0.*_v6.conf and restarts the DHCP dnsmasq. dnsmasq’s constructor:<iface> mode handles address deprecation and deletion — but a restarted dnsmasq has no memory of the prefix it previously advertised, so it cannot deprecate what it never knew. It stops mentioning the old prefix instead of withdrawing it with preferred lifetime 0 (RFC 4862 §5.5.3).

So the failure to expect is a client holding a dead prefix for its remaining valid lifetime — up to a day — while every unit reports healthy and nothing logs an error. RFC 6724 default source selection keeps choosing the dead prefix, so the client’s IPv6 fails while a probe forced onto the new prefix succeeds. That contrast is the diagnostic: compare the default source address against one pinned to the current prefix before concluding anything about the router. The incident record is ISP Prefix Re-delegation Left a Dead IPv6 Prefix on Every Client.

What detects, and how long it takes to notice

The standing detector (dns-common-ra-watch.service, one 900 s capture per run, one run every 6 h) harvests prefix-information options from appliance frames only, per in-scope VLAN, and classifies each observed prefix by family — global under 2000::/3, unique-local under fc00::/7, RFC 3849 2001:db8::/32 as documentation, anything else other. Three journal literals carry it:

Literal	Emitted when
RA-WATCH-PREFIX	one per distinct observed prefix — vlan, prefix, class
RA-WATCH-PREFIX-BASELINE	first observation for a VLAN, no recorded state. Never an action — a first run must not deprecate a prefix that was never dead
RA-WATCH-PREFIX-CHANGE	an in-scope global prefix changed. This is the ClickStack alert’s subject

Detection latency is up to 6 h 15 m — one full cadence plus one capture window, worst case. Shortening it is one role default away (dns_common_ra_watch_interval) and is deliberately not done: a re-delegation is a rare event whose damage window is ~22 hours, so buying an hour of latency at four times the tcpdump duty cycle on the two nodes that are the entire DNS serving plane is a bad trade. Change it if that reasoning stops holding, not because the number looks large.

A ULA change is reported and never acted on. Only the global class participates in change detection. The ULA announcement is the section immediately above.

The journal only reaches ClickStack because the collector is told to ship it

dns_otel_journal_units in ansible/roles/dns-otel-collector/defaults/main.yml names dns-common-ra-watch.service alongside blocky.service. Removing it silently un-arms the alert — the detector’s output reaches default.otel_logs nowhere and an alert over it is green forever, a content alert with a dead subject.

The collector does not re-read its config on its own. state: started is idempotent, so a converge that renders a changed config.yaml and reports changed does not restart the daemon. The role carries a restart handler guarded on dns_otel_service_enabled. If you ever edit that config outside Ansible, restart the unit yourself.

What announces

dns-common-ra-deprecate.service runs dns-common-ra-deprecate.sh, which emits a bounded deprecating RA with radvd:

• Preferred lifetime 0 is the whole fix. RFC 4862 §5.5.3(e) makes a host ignore a reduction of the valid lifetime below two hours unless the RA is authenticated, so AdvValidLifetime 0 may be silently discarded and must never be what this depends on. The valid lifetime is set above the clamp and stays out of the argument; the preferred lifetime is the field that deprecates.
• Router lifetime 0 on every stanza — it deprecates without ever claiming to be a default gateway. The Thread border router on this trunk already demonstrates that shape.
• Bounded by design. --announce runs under timeout for dns_common_ra_deprecate_duration_seconds (600 s) and stops. timeout’s exit 124 is the expected end and the unit declares it a success.
• Never Debian’s own radvd.service, and never /etc/radvd.conf. apt-get install radvd creates multi-user.target.wants/radvd.service in the same breath, held off from starting only by ConditionPathExists=/etc/radvd.conf being unsatisfied. Writing that path would arm a second, unbounded, boot-persistent radvd nothing in this role controls. The converge masks the Debian unit whenever the package is present, keyed on the package rather than on the arm flag.
• No recursive-DNS-server option, ever. The emitter is a bounded ten-minute burst whose single job is to deprecate a dead prefix, and attaching a resolver-list change to a transient withdrawal burst would make a client’s DNS depend on whether it happened to be listening during those ten minutes. The standing announcer is where RDNSS belongs, and it carries it.

Expect our own frames in the peer’s detector output. While an announcement is running, the other resolver sees a non-appliance RA source on an in-scope VLAN with router lifetime 0s and no RDNSS. That lands in the RA-WATCH-OBSERVED bucket by membership of the expected-source set, which is correct and is not a finding.

The two flags, and what each one is worth

They are separate on purpose.

Flag	False	True
dns_common_ra_prefix_watch_enabled	The detector prints RA-WATCH-PREFIX* and records prefix state — pure observability	The exit-3 path and the deprecation request file are live
dns_common_ra_deprecate_enabled	No radvd installed; /etc/dns-common/ra-deprecate.armed absent, so the emitter unit is skipped	radvd installed, arm file present, the emitter can announce

The second is a file, not a unit state: dns-common-ra-watch.service carries OnFailure=dns-common-ra-deprecate.service, and that fires on every non-zero detector exit — instrument failure and violations included. An unsatisfied ConditionPathExists= is a skip rather than a failure, which is exactly what makes that trigger safe.

Bring-up order

One node per window, portland (the non-holder) first, then sandiego (the ruling node) — the same order group_vars/dns_hosts.yml records for the detector’s own arm. Verify the first node is healthy before touching the second.

scripts/dns-playbook.sh --limit portland --tags dns-common-ra-watch --check --diff
scripts/dns-playbook.sh --limit portland --tags dns-common-ra-watch
# verify, then the same two commands with --limit sandiego

Arming is -e dns_common_ra_prefix_watch_enabled=true (and -e dns_common_ra_deprecate_enabled=true) on that node’s converge; making it durable is the committed inventory value. Both arms are two-sided — a converge with a flag false actively disarms rather than declining to arm.

systemctl start on a detector that is already running is a silent no-op, and a window that is already running keeps the OLD script: Ansible’s template render writes a temp file and renames it, so a running detector holds the previous inode and its output describes the previous script. A window read as evidence for a script change must have started after the converge finished. Before reading a triggered run as evidence, compare systemctl show dns-common-ra-watch.service -p ExecMainStartTimestamp --value against the script’s own mtime.

Which tag deploys what, and the one that does not

--tags dns-common-ra-watch renders the whole detector tier: the detector script, both detector units, the emitter script and the emitter unit. One tag, one tier.

--tags dns-common-ra-announce opens the same include but runs only the three tasks that carry that tag themselves — the radvd install, the package stat and the mask of Debian’s unit. The announce tag does not deploy the detector or the emitter.

That asymmetry is deliberate and is documented where it lives in ansible/roles/dns-common/tasks/main.yml: the task-level tag list is what makes the include selectable, and apply.tags is what every included task additionally inherits. Putting the announce tag in apply.tags would make an announcer converge sweep a tier the operator did not ask for. The cost: a change to the detector or the emitter is not deployed by an announcer converge and needs its own. If you changed a detector or emitter template, converge the detector tag or nothing you changed is on the box.

The check that settles which tasks a tag reaches, without touching a node — this one runs on the control node, not on a resolver:

cd ansible && ansible-playbook -i inventory/hosts.yml dns-playbook.yml \
  --limit portland --tags dns-common-ra-watch --check --diff

--list-tasks is not the instrument for this question: these are include_tasks, which are dynamic, so --list-tasks prints the include and cannot expand what is inside it. --check --diff runs the include and names each rendered file.

The expected-source set, and how to read it

Every RA-WATCH-OBSERVED line carries expected=yes or expected=no, set by membership of dns_common_ra_watch_expected_lladdrs (group_vars/dns_hosts.yml), which carries both resolvers’ link-local addresses: fe80::8aa2:9eff:fed4:6167 (sandiego) and fe80::8aa2:9eff:fede:ae7c (portland).

Membership decides which BUCKET the frame lands in. The membership test runs above the RDNSS test, because the resolver pair advertises its own RDNSS on every in-scope leg; under the opposite order that is ten permanent violations per window (five legs × two nodes) that nobody can fix, and a detector that reports a finding nobody can act on stops being read.

What that means when reading a window:

• An expected=yes line carrying an address in its rdnss= field is the pair announcing. It is not a finding; for the pair’s own frames the field reports the leg’s vip6.
• An expected=no source carrying an RDNSS on a TRUSTED leg is a violation, exits 2, and lands dns-common-ra-watch.service in failed, reported on a RA-WATCH-VIOLATION line. The SOURCE exemption is bounded by the two addresses in that list and by nothing else.
• On an UNTRUSTED leg (Guest, Telework) the same frame is reported on a RA-WATCH-UNTRUSTED-RDNSS line and does NOT exit 2. Those are segments whose definition is that we do not police them — a visitor advertising RDNSS there is the expected weather, and holding the detector failed on it would make a genuine rogue on Main indistinguishable from it. The frame is reported in full; only the exit code differs.
• The observed bucket’s count is reported as non_appliance_non_violating= on the RA-WATCH-SUMMARY line. A node printing non_appliance_no_rdnss= has not converged the detector tag.

The blind spot this creates, stated rather than left to be discovered. The exemption keys on the source link-local alone, and a link-local is spoofable on-link. An attacker who spoofs one of the two pair addresses and advertises a hostile recursive-DNS server is invisible to this detector. This is accepted rather than overlooked: the detector reports and does not prevent, so it was never the control that stopped that attack — nothing on this link is, since RA is unauthenticated by design and SEND is not deployed. Note the asymmetry deliberately: the appliance branch above the exemption requires both the MAC and the link-local to agree, while the exemption requires one identifier. Do not widen the exemption to compensate; widening it is how a detector becomes an allowlist.

The read is a cross-tab, not a grep:

journalctl -u dns-common-ra-watch.service --since '<window start>' --until '<now>' --no-pager > /tmp/ra.txt
grep -o 'RA-WATCH-OBSERVED' /tmp/ra.txt | wc -l
sed -n 's/.*src=\([^ ]*\).*expected=\([^ ]*\).*/\1 \2/p' /tmp/ra.txt | sort | uniq -c

Three rules for reading it:

1. The cross-tab counts must SUM to the RA-WATCH-OBSERVED count. If they do not, some lines are unannotated and the detector on that node predates the field. The fix is a --tags dns-common-ra-watch converge, not an edit.
2. No source may appear on both sides. A source that reads expected=yes in one line and expected=no in another is a broken set, not a finding about the wire.
3. An expected=no source that is not one of the two long-standing non-resolver sources is the change this annotation exists to surface. Name it by source and by VLAN before doing anything else.

The detector sees its own node’s advertisements

This is a property, not an anomaly. The detector captures with tcpdump -i <iface> and passes no direction restriction. tcpdump’s default direction is inout and it taps at the device layer, so it sees frames the node transmits as well as frames it receives. “A node does not receive its own multicast” is a property of IP-layer multicast reception, and a packet sniffer sits below that layer entirely.

The operational consequence: dns_common_ra_watch_expected_lladdrs must carry both nodes’ link-locals. Scoped per node to the peer’s address only, every node would classify its own advertisements expected=no, putting ten permanent unexpected-source lines across the pair into the report and destroying exactly the property the annotation buys. Do not narrow that list to a peer-only list.

Proving the announcement on demand

This path fires only during a real re-delegation, which cannot be summoned. Do not wait for one. The deprecating RA is observable on the wire on demand, against an RFC 3849 documentation prefix no client on this estate can hold. The detector classes 2001:db8::/32 as documentation and its change loop filters on class == "global" by equality, so the rehearsal frame cannot reach the deprecation path or name the live prefix as old_prefix.

Cheap confirmation before you start, on both nodes:

• systemctl is-active dns-common-ra-watch.service — whether a capture is running.
• systemctl list-timers dns-common-ra-watch.timer --all --no-pager — when the next one opens.
• cat /var/lib/dns-common/ra-prefix.state (dns_common_ra_prefix_state_path) — whether a global prefix is recorded for the leg you are about to advertise on. A VLAN with no recorded prefix reaches RA-WATCH-PREFIX-BASELINE, which is never an action.

If you only need to prove that the self-deprecation applies preferred_lft 0, use the self-deprecation procedure below instead. It exercises the whole mutation path and transmits nothing.

# 1. A scratch config for ONE leg. No recursive-DNS-server directive, and no comment naming one.
scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.shell -a 'cat > /tmp/probe.conf <<EOF
interface eth0
{
    AdvSendAdvert on;
    AdvDefaultLifetime 0;
    MinRtrAdvInterval 3;
    MaxRtrAdvInterval 4;
    prefix 2001:db8:dead:5f00::/64
    {
        AdvOnLink on;
        AdvAutonomous on;
        AdvPreferredLifetime 0;
        AdvValidLifetime 7200;
    };
};
EOF'

# 2. Run radvd in the foreground for ~60s and capture on the NAMED interface.
#    NEVER -i any: cooked-mode offsets differ, so ip6[40] silently matches nothing
#    and a zero is uninterpretable rather than negative.
scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.shell -a \
  '(timeout 60 radvd --logmethod stderr -n -d 5 -C /tmp/probe.conf > /tmp/radvd.out 2>&1 &) ; sleep 3;
   timeout 30 tcpdump -i eth0 -e -nn -v -s 0 -w /tmp/ra.pcap "icmp6 and ip6[40] == 134"'

# 3. Read it back with -v — options only print under -v.
scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.shell -a 'tcpdump -e -nn -v -r /tmp/ra.pcap'

Assert all three, on the frame sourced from this node’s own link-local:

• router lifetime 0s;
• a prefix info option for the dead /64 with Flags [onlink, auto] and pref. time 0s;
• no RDNSS option on our frames — the appliance’s frames in the same capture do carry one, which is the positive control proving the pattern can match.

The 30 s window is sufficient here and only here, because the 3/4 s interval is ours. Any capture of appliance frames must run longer than 200 s — ra-param=<if>,200,3600 sets its maximum interval — and must be bounded by timeout, never by -c N.

Clean up: kill radvd, remove the scratch config, apt-get purge -y radvd, and re-read the forwarding sysctl to confirm it is unchanged.

Proving the self-deprecation on demand

The procedure above proves the announcement. This proves the other half — that --deprecate-local actually fires and stamps preferred_lft 0 on this node’s own address for a named prefix. It puts nothing on the wire.

Why the condition has to be manufactured. The self-deprecation only has work to do when an address is held under a prefix the ISP has already retired — and a retired prefix ages out on its own within its remaining valid lifetime. Read at rest, the estate shows zero such addresses, because the same clock that made the prefix dead also removed the evidence. “The deprecation fired” and “the prefix timed out” are indistinguishable from that absence. A reading taken with no address under the named prefix has a false antecedent and proves nothing, however green it looks.

The two things that make the reading mean anything. Neither is optional:

1. The antecedent. An address must exist under the named prefix, on the target leg, with a non-zero preferred lifetime, read and written down before the invocation. If that read does not come back non-zero, stop — do not invoke, and do not report a result.
2. The discriminator. Plant it with a preferred lifetime of an hour, invoke, and take the post-fire read within a couple of minutes — then record the elapsed seconds. Expiry cannot explain a preferred_lft 0sec observed 42 seconds into a 3600-second preferred lifetime. Every second you spend before the read weakens the discriminator, so this is a figure to minimise, never a window to wait out.

Do not use the unit for this. systemctl start dns-common-ra-deprecate.service also runs --announce, which puts the named prefix on the wire. Invoke the script directly. Direct invocation exercises the entire mutation path (the staleness fence, the leg-set check, the unique-local fence, the hextet matcher, the settable-flag filter and ip -6 addr change) and transmits nothing.

Run it on the non-holder — the node holding no floating address, per ip -br addr show. <NODE> is that node and <LEG> is eth0.1000 (Lab): it is in the rendered leg set, it is the lowest-traffic in-scope leg, and its only RA source is the pair.

# 1. THE ANTECEDENT, part one: pre-flight. Both counts must read 0, and this
#    output is also the recovery record — every lifetime here is what a repair restores.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; ip -6 addr show scope global ; \
  ip -6 addr show scope global | grep -o 'preferred_lft 0sec' | wc -l ; \
  ip -6 addr show scope global | grep -o '2001:db8:' | wc -l ; \
  ls -l /var/lib/dns-common/ra-deprecate.request 2>&1 ; \
  sha256sum /var/lib/dns-common/ra-deprecate.request 2>&1"

# 2. THE ANTECEDENT, part two: plant the synthetic address with a KNOWN finite lifetime.
#    Read it back and confirm a NON-ZERO preferred_lft before going any further.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; \
  ip -6 addr add 2001:db8:04f3::1/64 dev <LEG> noprefixroute valid_lft 7500 preferred_lft 3600 ; \
  ip -6 addr show dev <LEG> scope global"

# 3. The request, written through the emitter's OWN file interface, with a fresh stamp.
#    Then the SAFETY PRE-FLIGHT: the doc-prefix count MUST read exactly 1.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; \
  printf 'requested_at=%s\nvlan=lab leg=<LEG> old_prefix=2001:db8:04f3::/64\n' \$(date +%s) \
    > /var/lib/dns-common/ra-deprecate.request ; \
  cat /var/lib/dns-common/ra-deprecate.request ; \
  ip -6 addr show dev <LEG> scope global | grep -o 'inet6 2001:db8:4f3:' | wc -l ; \
  ip -6 addr show dev <LEG> scope global | grep -o 'inet6 2600:' | wc -l ; \
  ip -6 addr show dev <LEG> scope global | grep -o 'inet6 fddb:' | wc -l"

# 4. Fire. DIRECTLY — never via the unit. Read the printed rc, not ansible's colour.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; \
  /usr/local/sbin/dns-common-ra-deprecate.sh --deprecate-local 2>&1 ; echo rc=\$?"

# 5. The reading, taken IMMEDIATELY. Record the elapsed seconds against step 2.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; ip -6 addr show scope global ; \
  ip -6 addr show scope global | grep -o 'preferred_lft 0sec' | wc -l"

The count in step 3 is anchored on 4f3, not 04f3, and that is load-bearing. The kernel prints addresses in RFC 5952 canonical form, which strips leading zeros from every hextet: you write 2001:db8:04f3::1 and ip reports 2001:db8:4f3::1. A count anchored on the string you typed reads 0 on a leg that genuinely carries the address — a green-looking pre-flight that has checked nothing. The old_prefix= in the request file is unaffected either way, because the matcher compares hextet by hextet with each hextet zero-padded rather than as a string.

The request-file interface is the way in, and forging detector state is not. A first observation for a VLAN always emits RA-WATCH-PREFIX-BASELINE and never deprecates, so a hand-edited prefix-state file cannot drive this path. The file carries a staleness fence read from its own requested_at stamp rather than from its mtime; the ceiling is dns_common_ra_deprecate_request_max_age_seconds, 3600 s. Give old_prefix an explicit /64 — a prefix with no length yields zero groups in the matcher, which prints nothing and reaches the harmless no-address branch rather than matching everything.

How to read it — both sides, and the second side is the one people skip.

• Positive. The planted address reads preferred_lft 0sec with a finite, non-zero valid_lft at or below the 7500 you planted, and the kernel adds its own deprecated token.
• Collateral. The count of addresses reading preferred_lft 0sec on that node goes from 0 to exactly 1, and every fddb: address and every live 2600: address reads a non-zero preferred lifetime, counting down normally against the pre-flight output. Name the planted address as the only row whose preferred lifetime went to zero — a bare “nothing else broke” is not a control.

preferred_lft 0 is never clamped; valid_lft is. RFC 4862 §5.5.3(e) makes a host ignore a reduction of the valid lifetime below two hours unless the RA is authenticated, so a valid lifetime is planted at 7500 rather than at 7200 or 0. The preferred lifetime is the field that deprecates, and it is honoured immediately.

The flags are not round-tripped, and a read-then-reissue that copies the observed string verbatim looks correct and is wrong. ip -6 addr change is add carrying the netlink replace flag: it replaces the whole attribute set, so a flag not named is a flag cleared. mngtmpaddr and noprefixroute are settable and are re-stated; dynamic and scope global are derived — dynamic follows from supplying a finite valid lifetime and scope global is the default for a global unicast address — and re-stating either is an iproute2 usage error, not a preservation. The emitter filters the observed flags to the settable subset for exactly this reason.

The unique-local fence, and it is worth probing. A request naming an fddb: prefix is refused before any address is read and before anything is mutated. Probe it against a prefix the node genuinely holds an address under, so the probe is not vacuous:

scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "\
  printf 'requested_at=%s\nvlan=lab leg=<LEG> old_prefix=fddb:f665:73f7:4::/64\n' \$(date +%s) \
    > /var/lib/dns-common/ra-deprecate.request ; \
  /usr/local/sbin/dns-common-ra-deprecate.sh --deprecate-local 2>&1 ; echo rc=\$? ; \
  ip -6 addr show dev <LEG> scope global"

Expect rc=1 with RA-DEPRECATE SKIP: the deprecation request names the unique-local prefix …, and every fddb: row unchanged. Exit 1 is the refusal; exit 2 belongs to the strict-argv refusals only (no mode, wrong arity, unknown argument). A caller that reads exit 1 as a failure will misdiagnose a request the script correctly declined — refuse exits 1 precisely so that an ExecCondition= treats “nothing to do here” as a skip rather than landing a unit in failed.

Clean up, and verify the cleanup rather than intending it.

scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "\
  ip -6 addr del 2001:db8:04f3::1/64 dev <LEG> ; \
  rm -f /var/lib/dns-common/ra-deprecate.request ; \
  ls -l /var/lib/dns-common/ra-deprecate.request 2>&1 ; \
  ip -6 addr show scope global | grep -o '2001:db8:' | wc -l ; \
  ip -6 addr show scope global | grep -o 'preferred_lft 0sec' | wc -l ; \
  systemctl list-units --failed --no-pager"

Both counts must read 0 and the request file must be absent — or, if the node already had a real one, restored from the copy you took in step 1 and proved byte-identical with sha256sum. Leaving a hand-written request behind arms the emitter against the next detector failure, because dns-common-ra-watch.service carries OnFailure=dns-common-ra-deprecate.service and that fires on every non-zero detector exit. The 3600-second staleness fence is the second lock on that door. Removal, verified by absence, is the first.

Finally, re-read the two-node invariants — the VIP line counts and both keepalived ExecMainStartTimestamp values — and confirm they are unchanged. A VRRP transition is a two-node event, so that read is only meaningful across both nodes in one invocation.

IPv6 forwarding is not touched

radvd’s check_ip6_forwarding() reads /proc/sys/net/ipv6/conf/all/forwarding, which is 0 on these deliberately hardened multi-homed resolvers. radvd logs IPv6 forwarding seems to be disabled, but continuing anyway and keeps advertising. No sysctl is flipped anywhere in this design.

Reading Kea lease state from a workstation (DHCP-12)

This is the supported way to read DHCP lease state off the resolver pair. It is dispatched from a workstation and it needs no interactive shell on either node.

The command

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
  -a 'kea-shell --host 127.0.0.1 --port 8000 --auth-user kea-ctrl --auth-password-file /etc/kea/creds/ctrl-basic-auth.pw lease4-get-all </dev/null'

Three details are load-bearing:

• The outer quotes on -a are SINGLE. A double-quoted dispatch argument is expanded by the control shell, which produces a command that never runs on the machine it describes and reports the same answer regardless of the target’s state.
• -b is required. The credential file is 0640 root:_kea and the login account is neither. Without it you get a permission error that reads like a broken listener.
• </dev/null is required. kea-shell reads the command’s JSON arguments from standard input and waits for EOF. Without it the dispatch hangs until Ansible’s timeout.

Replace dns_hosts with sandiego or portland to ask one node. The listener is bound to 127.0.0.1, so this command only works on the node — which is the point: the control channel has no read-only mode, and lease4-get-all arrives on the same socket as config-set and shutdown.

What it returns, and how to tell “no leases” from “command failed”

The response is a JSON array with one object per answering service, emitted as one line, and three fields matter:

Field	Carries	Read it as
result	Kea’s control result code	0 success · 1 error · 2 command unsupported · 3 success, EMPTY
arguments.leases	the lease payload	the actual answer; [ ] means no leases
text	a human sentence	a summary, never the datum

"result": 3 IS NOT AN ERROR. It is Kea’s CONTROL_RESULT_EMPTY: the command ran, the lease table was read, and it held nothing — the correct reading for any leg this pair does not serve. An operator who reads any non-zero result as failure will report a broken read surface on a healthy pair. Conversely, "result": 1 with an empty leases array is a failed command, and text is where it says why.

"result": 0 is the populated success code. Each lease under arguments.leases carries client-id, cltt, hostname, hw-address, ip-address, subnet-id, valid-lft and state. Read the count out of arguments.leases, never out of text — and read state before concluding a client is present: 0 is assigned, 2 is expired-reclaimed. An entry with state: 2 is a lease the server is remembering, not a client holding an address, and a leg whose entries are all 2 has a population that has not renewed since its lease ran out.

Both nodes must return the identical payload. They are HA partners and lease updates replicate, so a difference between sandiego and portland is a replication finding, not a reading artifact — which is why the documented dispatch targets dns_hosts rather than one node.

The lease CSV is NOT the read surface

/var/lib/kea/kea-leases4.csv exists, is greppable, and is the wrong thing to read. Kea’s lease file cleanup rewrites it on its own lfc-interval — it copies the file aside, compacts it and moves it back — so a reader of the file races that rewrite and can observe a truncated, doubled or absent file with no error of any kind. The control command does not race it, which is the entire reason this read is an API rather than a file. Do not “simplify” this procedure into a cat of the CSV.

Confirming the pair is peered

Same listener, different command:

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
  -a 'kea-shell --host 127.0.0.1 --port 8000 --auth-user kea-ctrl --auth-password-file /etc/kea/creds/ctrl-basic-auth.pw status-get </dev/null'

The high-availability state lives at arguments.high-availability[0].ha-servers.local.state, and its partner’s last known state at …ha-servers.remote.last-state. A healthy pair reports hot-standby on both nodes, one as primary and the other as standby.

Read the state field BEFORE the lease count, always. An unparseable response, a response whose high-availability object never appeared, and a genuinely empty lease table all look like “zero leases” from the outside, and the three are not the same finding. A zero reported without the state field beside it is not an observation.

If it does not work

Symptom	Almost always
kea-shell: command not found	isc-kea-admin is not installed — re-run scripts/dns-playbook.sh --limit <node> --tags kea-install
Connection refused on 127.0.0.1:8000	the daemon is stopped, or kea_ctrl_http_enabled was overridden false
401 from the listener	the credential file and the Vault entry disagree — re-run --tags kea-certs
the dispatch hangs	</dev/null was dropped; kea-shell is waiting on standard input
permission denied opening the credential	-b was dropped

The DHCPv4 accept and the raw-socket asymmetry — acquisition works without it, renewal does not (DHCP-09)

Kea will hand out addresses on a leg whose firewall never admitted a single DHCP packet, and it will stop renewing them hours later. That asymmetry is the whole of DHCP-09, and it is why table inet dns_resolver chain input carries a udp dport 67 accept on both resolver nodes.

Why acquisition works with no accept at all

kea-dhcp4 opens AF_PACKET raw sockets by default, and it has to: a client in INIT holds no address, so its DISCOVER is a broadcast from an unspecified source that no interface-bound UDP socket can receive. ISC states the consequence plainly — raw sockets bypass system firewalls like iptables — so DISCOVER and OFFER reach the daemon whether or not any accept exists. A chain missing this rule looks entirely healthy, and every first acquisition on the leg succeeds.

Why renewal does not

The RENEWING path does not bypass the chain, and that is the whole point. A client at T1 unicasts its DHCPREQUEST to the server identifier — this node’s own leg address — and that is an ordinary routed UDP datagram to port 67. It traverses the normal input path and meets policy drop.

So the failure mode is not a failure to acquire. It is working acquisition followed by silent renewal failure hours after a converge that looked clean, on a leg nobody is watching any more. The client sits unanswered from T1 until T2, then falls through to a broadcast REBIND — which the raw socket does answer, so service limps back on the slow path. At the committed 12 h lease that first shows up roughly six hours after the change that caused it.

Why it is scoped to every leg, not to the served ones

The accept must land no later than the change that starts Kea on a leg. Added early the rule is inert — it admits packets to a port nothing is listening on. Added late it is an outage that began hours before anyone could see it. Inert is the cheap error, so the rule is converged before a window rather than raced inside it.

That is only available because of how the rule is scoped. It is derived from dns_vlan_interfaces — the full VLAN leg set — exactly like the four dport 53 accepts beside it, so it renders byte-identically before and after any leg is armed. Nothing about it changes at cutover.

It is deliberately NOT scoped to kea_served_legs, and that refusal is load-bearing. The served-legs list is a per-converge -e value. Scoping the firewall to it would make the live ruleset depend on an extra-var somebody remembered to pass, so a routine converge that omitted it would silently withdraw the accept from a leg Kea is actively serving — reintroducing exactly the delayed failure above, on a leg already in production. It would also render iifname { } under an empty served-legs list, which nft -f rejects outright, taking the entire ruleset with it.

Reading the accept back off a live kernel

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
  -a 'nft list ruleset | grep "udp dport 67" | grep -c accept'
# want: 1 on each node. grep, never ripgrep — the resolver nodes do not have it.

Two details are load-bearing:

• The port match and the verdict are matched SEPARATELY, and that is not stylistic. The kernel prints a counter clause between them: ... udp dport 67 counter packets 0 bytes 0 accept. A pattern anchoring on the contiguous string dport 67 accept therefore matches zero and reports “the rules never reached the kernel” about a kernel that is holding them. Do not “simplify” this read into one pattern.
• The outer quotes on -a are SINGLE. A double-quoted dispatch argument is expanded by the control shell, which produces a command that never runs on the machine it describes and reports the same answer regardless of the target’s state.

The rule is IPv4-only by design — meta nfproto ipv4 is explicit, there is no IPv6 sibling, and DHCPv6 is not served by this pair at all. The negative control is the same shape:

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
  -a 'nft list ruleset | grep -c "nfproto ipv6 udp dport 67"'
# want: 0 on each node. A non-zero reading is a finding, not a hardening opportunity.

Cutting a leg over to Kea, and rolling it back — one procedure, invoked two ways

This is written for ANY leg. Substitute the leg name, its pool boundary and its host_vars entry — nothing else changes. Every leg is cut; the procedure is a rollback and rehearsal instrument.

There is exactly ONE rollback here and it is reached two ways: by an operator who decided to roll it back, and by an operator who hit one of the abort triggers below. Both run the same numbered steps. There is deliberately no separate fast-abort path, because a separate path makes the emergency path the one path that has never been run.

## Back-out is a different change — it backs out the DNS cutover, one Primary DNS Server field per network. It is not this procedure and neither supersedes the other.

The ordering rule, stated in both directions, and it is load-bearing

To cut, the appliance’s per-network DHCPv4 Server toggle goes OFF first, then Kea is armed. To roll back, Kea is disarmed on BOTH nodes first, then the appliance toggle goes back ON.

Direction	Step 1	Step 2
Cut	appliance DHCPv4 Server for that network → Off	arm Kea on the leg, both nodes
Roll back	disarm Kea on both nodes	appliance DHCPv4 Server for that network → On
Re-cut	appliance → Off again	arm Kea again

The reason is one sentence: two DHCPv4 servers on one broadcast domain fight and the client takes the first OFFER, so there is no safe overlap window. An operator who flips the appliance back on before disarming Kea has recreated the forbidden state — and has done it during a rollback, when attention is already spent. An unserved leg for minutes is accepted; nothing is being restored to during it.

On the coincident-pool legs this ordering is the ONLY control. On lab and iot Kea’s pool sits disjoint from and adjacent to the appliance’s, so a leaked OFFER arrives as an address from the wrong range and an operator can see it. On mainwireless, main and core the two pools COINCIDE EXACTLY, so a fight there is silent. Disarm Kea on both nodes and confirm it before the toggle goes back on.

The appliance half of a rollback is GUI state with no CLI and no API, so an automated step may disarm Kea and converge and nothing else, then halt and hand over. A rollback that stops between those two halves leaves the leg with no DHCP server on it at all, which is why the halt is an escalation rather than a pause.

Three machines, three roles — and the instrument must outlive its subject

Name all three before the window opens, so nobody improves the setup by opening an SSH path from the leg being cut.

Machine	Reached how	Does what
The instrumented client	over a different leg from the one being cut, through its own sanctioned wrapper (scripts/nas-adhoc.sh for the NAS)	every client-side read: address, lease, forced renew, v6 address and resolvers
The resolver pair	over the untagged Core leg, scripts/dns-adhoc.sh / scripts/dns-playbook.sh	every converge, and the server-side lease read
The control-node workstation	—	runs both wrappers; it is on neither the cut leg nor the pair’s DHCP path

The instrument must outlive its subject. The client is read over a leg its own DHCP state cannot break, and the pair is reached over Core, which has no DHCP dependency in either direction. SSH is accepted on the five trusted legs. Guest and Telework have no SSH path by design. Do not propose one “just for the window”.

Abort triggers, named up front. Any one of them runs the rollback below and closes the window

• (a) No client on the leg holds an address from Kea’s pool within N minutes of arming. For a leg instrumented at a ten-minute lease, N = 15 minutes: the client’s own T2 at that lease (0.875 of the lease, ≈525 s) plus one acquisition round trip.
• (b) A client observed to have JUST ACQUIRED an address holds an address outside Kea’s pool. This is the leaked-OFFER signal: the appliance answered after its scope was withdrawn. The “just acquired” scoping is not optional. Every client on the leg that has not renewed holds an appliance address after the toggle, so an unscoped reading of this trigger fires on the entire leg the instant the window opens. Only a client force-renewed or rebooted after the toggle can be read against it.
• (c) The static management path to either resolver node is lost. The nodes’ leg addresses are statically configured and have no DHCP dependency, so losing the path means something other than DHCP has moved, and the window can no longer be driven or rolled back safely.

The pool boundary is what makes trigger (b) readable

On a disjoint leg the appliance’s scope and Kea’s pool are disjoint and adjacent, so the address a client holds tells you which server answered. On Lab:

Server	Range
Kea	192.168.200.160 – 192.168.201.19
Appliance	192.168.201.20 – 192.168.203.249

On Lab, .200.140-.200.159 below the pool is static (the k3s node Lab legs, the MetalLB lab pool and the address reserved for kube-vip), so an address there in a lease is a misconfiguration, not an appliance answer.

They meet at .201.19 / .201.20 with no overlap and no gap. Read every other leg’s boundary from that leg’s host_vars disjointness table — do not carry Lab’s numbers across. iot is the one leg whose Kea pool sits ABOVE the appliance’s (.101-.223 against .20-.100); its upper bound is .223 because 192.168.218.224 is heimdall’s second interface, which carries no reservation. Do NOT widen that pool into .224.

Which client-side fields discriminate, and which only assert correctness

Field	Changes at the cut?	Read it as
The address range	yes, on a disjoint leg	primary evidence — which pool it came from
The DHCP server identifier	yes	primary evidence — which server answered, unambiguously
The gateway	no	correctness only; the appliance stays the default router
The advertised resolver	no	correctness only; the appliance already advertises the leg VIP that Kea also advertises
The domain-search list	yes	Kea sends fzymgc.house, lan.fzymgc.house; the appliance sends none or one legacy entry

A change in either “correctness only” field is a finding and both checks stay. But a reading that concludes “DNS is correct, therefore Kea answered” has concluded nothing.

On a coincident-pool leg the address proves nothing. Confirm by lease ORIGIN: the lease appears in Kea’s own lease4-get-all on both nodes (the appliance’s leases never do), and its lifetime is Kea’s 43200, not the appliance’s 600. DHCP4_PACKET_SEND ... DHCPACK (type 5) from <node address>:67 in the isc-kea-dhcp4-server journal names the sending address outright.

Why the lease is shortened before every window

A rollback is a cold acquisition, not a renewal. At T1 the client unicasts its request to a server identifier that is no longer answering and stays put; it does not look for another server until T2 (the rebind fraction, 0.875 of the valid lifetime), when it broadcasts. So the unattended cost of a rollback is bounded below by T2: about 8 min 45 s at a 600 s window lease, and about 10 h 26 min at the committed 43200 s steady state. A leg rolled back at the steady-state lease is a leg whose clients come back over the following ten and a half hours, and a leg full of devices with no shell cannot be forced. Shorten the lease before every leg’s window.

A forced renew on one instrumented client measures how fast the restored server answers once asked, not how long an unattended device takes to ask. It is a floor, never a forecast.

Standing cautions for any window on this estate

• Do not run the RA rehearsal in a cut window. It has a path into deprecating the live GUA prefix. Use the self-deprecation route, which transmits nothing.
• The appliance toggle’s blast radius is estate-wide and brief. Flipping a per-network DHCP setting regenerates the appliance’s DHCP configuration and restarts its DHCP daemon: no DHCP and no RA anywhere in the house for the duration — every VLAN, both families, wired and wireless. Schedule for it. It is unavoidable and it is not a reason to change the design.
• The appliance REGENERATES a network’s scope when its toggle comes back on; it does not restore it. Read the range back off the app after every flip rather than trusting it, and take the disjointness reading from the app’s own screen before every withdrawal.
• The per-network toggle is unmanaged GUI state. It is not in Git, and a vendor firmware update or an app-side revert could restore it, putting a second DHCP server on a live leg with nothing going red.
• The IPv6 half is the appliance’s and is never flipped by this procedure. “The appliance’s DHCP server is off for this leg” is true only of v4; DHCPv6 stays On on every leg.

Preparation, before the window opens

Each of these cites a command already documented on this page. Do not re-derive them.

1. The appliance ULA artifact absence read, with its own positive controls — see V6-05 — reading the appliance-side ULA artifacts absent above. Its CONTROL-PRESENT / CONTROL-ABSENT lines are what stop the absence claim being vacuous.

2. The reservation inventory. Read directly by Terraform and by the role, with no generated artifact between them to go stale — so this is a read, not a regeneration.

3. The appliance-side lease shortening, and its lead time. The condition to satisfy is every DHCPv4 client on the leg holds the shortened lease — measured client-side, on the leg, not inferred. One full PREVIOUS lease-time after the shortening is the calendar proxy for that condition, because a lease-time change takes effect only as clients renew. Where the population is enumerable and measured short, the condition is met and the date adds nothing.

4. The DHCP Guard check. UniFi DHCP Guard drops every DHCP server-side frame that is not from a trusted address: Kea can be correct — HA synced, sockets open, offers transmitted — while clients DISCOVER endlessly with zero DHCPREQUESTs. tf/unifi/networks.tf declares each network’s dhcp_guarding (enabled flag and at most three trusted servers), and a clean plan of main-cluster-unifi means the controller matches it. On the controller the same state is dhcpguard_enabled plus dhcpd_ip_1..3 in /proxy/network/api/s/default/rest/networkconf (see UniFi); the join key is the VLAN tag, not UniFi’s ip_subnet. A cut that brings a new DHCP server onto a guarded VLAN must add its address to that list first. Where the guard is on, prove it behaviourally with a throwaway macvlan client and a known-good VLAN as the control, BEFORE touching the appliance:

   ip link add dhcptest0 link eth0.<vlan> address 02:00:00:de:ad:01 type macvlan mode bridge
   ip link set dhcptest0 up
   busybox udhcpc -i dhcptest0 -s /bin/true -n -q -R -t 4 -T 5 -f
   ip link del dhcptest0

   A lease on the control VLAN and none on the target is the guard, not the daemon.

The converge commands, exact

The ruleset converge runs BEFORE the window and is firewall-scoped only:

scripts/dns-playbook.sh --limit <node> --tags dns-common-firewall
# One node at a time, non-VIP-holder first. Read the holder before trusting the name.

The in-window converges are the Kea role tag only:

scripts/dns-playbook.sh --limit <node> --tags kea \
  -e kea_service_enabled=true \
  -e 'kea_served_legs=["lab"]' \
  -e kea_valid_lifetime=600 \
  -e kea_renew_timer=300

Never the bare dns-common role tag in the window. It sweeps the VLAN legs and the dns-common-network-apply.sh transaction and re-arms the RA announcer — the v6 half this window must leave untouched.

The served-legs value must be passed in JSON list form. -e 'kea_served_legs=["lab"]' passes a list; -e kea_served_legs=lab passes a string, and the template’s leg.name in kea_served_legs membership test silently becomes a substring test that matches any leg whose name is a substring of it.

Disarming: pass the served-legs list explicitly. Do not simply drop the -e.

scripts/dns-playbook.sh --limit <node> --tags kea \
  -e kea_service_enabled=true \
  -e 'kea_served_legs=["<every leg that stays served>"]'

# Rolling back the ONLY served leg:
scripts/dns-playbook.sh --limit <node> --tags kea \
  -e kea_service_enabled=false \
  -e 'kea_served_legs=[]'

Dropping the -e falls back to the committed default, and every leg is committed to group_vars — so a bare converge re-arms. A rollback written as “drop the extra-vars” re-cuts the leg.

The three-part read that says a leg is served

A leg is served when three things are true at once. Substitute the leg name, its interface, its subnet id and its pool boundary from that leg’s host_vars entry; nothing else changes.

1. Kea’s journal shows sockets open on the new leg.
2. one lease allocated from that leg’s own pool, agreed by the pair’s own lease table.
3. one dynamic name resolving in that leg’s carved zone, forward and reverse.

All three, or the leg is not proved served. Any one of them alone has a healthy-looking failure mode: a socket with no lease is a leg nobody has asked, a lease with no name is a DDNS failure whose only symptom is in D2’s journal, and a name with no lease can be a curated record that was always there.

1 — sockets open on the new leg

The leg’s interface has to be in the set the daemon was actually handed, and the daemon must not have reported an empty socket set in the window:

# ON A RESOLVER NODE. These boxes have no ripgrep — use grep.
# (a) the leg's interface is in the served set the running daemon was given
grep -c '"<leg-interface>"' /etc/kea/kea-dhcp4.conf
# want: 1 on each node.

# (b) and the daemon did not report an empty socket set after the arm converge
SINCE='<the arm time, as YYYY-MM-DD HH:MM:SS UTC>'   # TZ=UTC and --utc below are what make that true
sudo TZ=UTC journalctl --utc -u isc-kea-dhcp4-server --since "$SINCE" --no-pager -o cat -q \
  > /tmp/kea4.log
echo "total=$(wc -l < /tmp/kea4.log)"                                # the denominator, always first
echo "nosockets=$(grep -c DHCPSRV_NO_SOCKETS_OPEN /tmp/kea4.log)"    # the numerator
# want: total greater than 0, AND nosockets 0.

A zero with no denominator beside it is not observation 1. journalctl exits 0 and prints nothing whether the daemon is clean or the query was wrong — a unit name that is really the process name, or a --since parsed in the node’s own America/New_York clock, both produce a false zero. total=0 means you measured your own query.

Never count a journal’s errors with a bare grep -i error: Kea’s LFC_READ_STATS and LFC_WRITE_STATS INFO lines carry errors: 0, so the word matches on a healthy daemon. Match on the severity Kea prints, or exclude the stat lines by name:

# ON A RESOLVER NODE. These boxes have no ripgrep — use grep.
grep -c ' ERROR ' /tmp/kea4.log
# want: 0. Kea prints its own severity, so an INFO stat line cannot match this at all.

grep -iE 'error' /tmp/kea4.log | grep -vc 'errors: 0'
# want: 0. The same read stated as an exclusion, for a log whose severity field you cannot trust.

2 — one lease allocated from that leg’s own pool

Read the leases back through the documented control-socket command — see Reading Kea lease state from a workstation (DHCP-12) above, whose three load-bearing details (single-quoted -a, -b, and the </dev/null) all apply here unchanged:

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
  -a 'kea-shell --host 127.0.0.1 --port 8000 --auth-user kea-ctrl --auth-password-file /etc/kea/creds/ctrl-basic-auth.pw lease4-get-all </dev/null'

What makes this observation 2 rather than “a lease exists” is that the lease’s subnet-id is this leg’s id from host_vars and its ip-address falls inside this leg’s kea_pool. A lease on another leg’s subnet id proves the pair is serving something else.

"result": 3 is EMPTY. On a leg armed minutes ago that reading is abort trigger (a)’s clock, not a pass. No client has asked. Wait it out or force a renew on the instrumented client; do not record an empty lease table as observation 2.

3 — one dynamic name resolving in that leg’s carved zone

Take the hostname and address from the lease in observation 2, and ask both nodes directly:

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
  -a 'dig +short <hostname>.<leg>.lan.fzymgc.house A @127.0.0.1 -p 1053'
# want: the leased address, on BOTH nodes.

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
  -a 'dig +short -x <the leased address> @127.0.0.1 -p 1053'
# want: the same name, on BOTH nodes.

Ask the authoritative tier on @127.0.0.1 -p 1053, never on :53. On these nodes blocky holds :53 and answers NOTIMP to anything it does not serve, which reads like a broken authoritative server and is not one.

The reverse half is the one that fails independently, and it fails in a way the forward half hides: D2 sends forward and reverse as one transaction but reports them separately, so a leg whose reverse /24s were not carved before its first lease answers the forward query perfectly and the reverse query not at all. That is the whole reason the carve lands before the window opens. On a leg whose pool spans three /24s (core: .41–.43), carve the middle one too — an endpoint-only containment check cannot see it, and every lease in it takes a D2 reverse rejection. Reservations in a deliberately uncarved /24 (main’s .20, core’s .40) take a DHCP_DDNS_NO_REV_MATCH_ERROR on their reverse half and keep answering in reverse from the covering /16; that is the carve working, not a failed cut.

The procedure

Numbered so the abort triggers above and the deliberate drill point at the same steps.

Cut

1. Take the pre-cut client reads: v4 address, lease (address, lease time, server identifier, option surface), v6 address and resolvers. Verbatim — these are one end of every later comparison.
2. Read the appliance’s DHCPv4 values for the network off the app and record them (see The appliance values each rollback restores below). With the toggle off the whole DHCPv4 block collapses to a single Off row — the values are gone from the page, not greyed out. Verify → RECORD → flip.
3. Appliance app → that network → DHCPv4 Server → Off.
4. Prove the static management path with DHCP off on that leg: dispatch to both resolver nodes and confirm both answer. Its failure is abort trigger (c) with a defined response rather than an improvised one.
5. Arm the non-VIP-holder, then the VIP holder, with the in-window command above.
6. Confirm each daemon opened a socket for the leg’s interface and its journal carries no “no sockets open” line. A failed bind is fatal by design and puts the window straight into abort territory with the appliance scope already off — which is the case this rollback exists to make survivable.

Verify

1. Force a renew on the instrumented client and re-read all four discriminating fields.
2. Read the leases back from the pair through the documented control-socket command. Single-quoted -a, -b, and the stdin redirect are all required, and "result": 3 is Kea’s EMPTY code and not an error.
3. Agreement between the client’s own view and the server’s lease is the evidence. Either alone leaves a gap: a client can hold a stale address the server never issued, and a server lease says nothing about what the client actually configured.
4. Re-read the v6 address and resolvers. They must be unchanged.

Roll back — the same steps whether this is the drill or an abort

0. RECOVER THE APPLIANCE’S VALUES FOR THIS LEG FIRST, before anything else. They are the rollback’s INPUT, they exist in no git-tracked file, and they are recorded at The appliance values each rollback restores below. Restore them BY FIELD NAME, never by position — iot and core each carry an ABSENT Secondary DNS field beside a BLANK Search Domain field, which are different findings, and a positional restore writes a search domain into a secondary-DNS slot. The appliance regenerates the network’s scope, so read the range back off the app afterwards rather than trusting the flip.
1. Disarm Kea on both nodes, using the explicit served-legs form above, and confirm both daemons are down before the toggle moves. On a coincident-pool leg this confirmation is the only control there is.
2. main ONLY, and it is a precondition rather than a step: put 192.168.20.1 on UniFi DHCP Guard’s trusted list for that network BEFORE the toggle, through unifi_network.main in tf/unifi/networks.tf (it replaces the VIP .20.5; the list holds at most three). The trusted list for Main holds the resolver pair and its VIP — 192.168.20.6, .7, .5 — and not the appliance, so with its toggle back on the appliance’s offers are dropped exactly as Kea’s would be, and the leg stays unserved.
3. Appliance app → that network → DHCPv4 Server → On, with the values from step 0.
4. Force a renew on the instrumented client.
5. Confirm by lease ORIGIN, not by address range. On lab and iot the pools are disjoint, so an appliance-range address is sufficient. On the three coincident-pool legs the address proves nothing: read the valid-lft instead — the appliance’s 600 against Kea’s 43200 — and confirm the lease has stopped appearing in Kea’s own lease4-get-all on both nodes. A client carrying a Kea lifetime means the rollback did not complete, whatever address it holds.
6. Re-read the v6 address and resolvers. This is the leg that matters — a coupling that only bites on the way back is invisible from the config and shows up only here. The v6 half is never flipped, so anything that moved here is a finding rather than a restoration.

Re-cut

1. Appliance → Off again.
2. Arm both nodes without the lease-instrument extra-vars, so the leg lands on the committed steady-state lease. The instrument’s lifetime is exactly the window’s, and the last action of the window is what ends it.
3. Verify again, including that the on-node rendered lease time is the steady-state value and the instrument value appears nowhere.
4. Commit the leg’s arm values to group_vars the same day, then run one bare converge with no extra-vars at all and confirm Kea is serving. That is the only reading which proves the steady state is self-sustaining rather than held up by a flag somebody remembered to pass.

The appliance values each rollback restores

Each leg’s appliance-side DHCPv4 settings are recorded HERE because they exist nowhere else. The per-network toggle and its fields are unmanaged GUI state: not in Git, not in inventory, not in any export. They are the rollback’s INPUT. A value nobody wrote down is a value nobody can restore, and the moment it is needed is the moment nobody has time to reconstruct it.

This is the DHCPv4-SERVER half only. The DNS fields the appliance held for Guest and Telework are recorded at The DNS fields each untrusted leg carried before its cut, and what a rollback restores above.

Leg	Network	Gateway / mask	Start	End	Lease	Primary DNS	Search Domain	Secondary DNS
mainwireless	VLAN 3001, 192.168.24.0/22	192.168.24.1 / 255.255.252.0	192.168.24.200	192.168.27.254	600	192.168.24.5	field absent (was wireless.lan.fzymgc.house, retired)	field absent
main	VLAN 3000, 192.168.20.0/22	192.168.20.1 / 255.255.252.0	192.168.22.20	192.168.23.249	600	192.168.20.5	fzymgc.house	field absent
iot	VLAN 3020, 192.168.218.0/24	192.168.218.1 / 255.255.255.0	192.168.218.20	192.168.218.100	600	192.168.218.5	present, BLANK	field absent
core	untagged, 192.168.40.0/22	192.168.40.1 / 255.255.252.0	192.168.41.10	192.168.43.249	600	192.168.40.5	present, BLANK	field absent

Lab’s appliance range is the one in the pool-boundary table above (192.168.201.20 – 192.168.203.249`); its remaining fields were not recorded before its cut.

“Field absent” is a complete record, not a missing value. mainwireless, main, iot and core render no Secondary DNS row at all; a restorer hunting for one will either invent it or report the restore incomplete, and neither is right. A BLANK Search Domain is a different finding from an absent one: restore it empty rather than inventing a value.

iot’s End IP .100 is the value to restore. Restoring .249 re-creates an overlap with Kea’s pool, which sits above the appliance’s range on this leg.

mainwireless’s search domain left with the appliance deliberately. Kea sends fzymgc.house, lan.fzymgc.house on every leg; wireless.lan.fzymgc.house is not a zone.

The pools coincide exactly on mainwireless, main and core — Kea’s pool is the range recorded above, so no client renumbers through a cut or a rollback and the address is not evidence of which server answered. The carves are per /24: main carves 22. and 23., core carves 41., 42. and 43.; 20. and 40. stay with the covering /16, which is what keeps unifi, heimdall and pdu answering in reverse.

The DDNS tier — how a DHCP lease becomes a name, and how each half fails

Read this section before triaging a missing name. The two halves below publish different record types through different daemons with different failure surfaces, and the single most expensive mistake available here is collapsing them into one question.

What publishes what

The forward chain — A and PTR — is Kea’s, end to end. But the A half and the PTR half are two separate updates, to two different zones, accepted by two different grants, and collapsing them is the mistake this subsection exists to prevent:

1. kea-dhcp4 grants or renews a lease and builds an FQDN from the client’s hostname plus the leg’s own ddns-qualifying-suffix (lab.lan.fzymgc.house. for Lab; one per leg, derived from the leg name so a client cannot mint a name outside its own VLAN).
2. It emits an NCR — a Name Change Request — over the dhcp-ddns channel to 127.0.0.1:53001, this node’s own kea-dhcp-ddns (D2). Loopback is forced; the NCR channel never crosses the wire.
3. D2 matches the FQDN against its configured forward domain, turns the request into RFC 2136 DNS UPDATE signed with that leg’s own kea-d2-<leg> TSIG key, and sends it to PowerDNS on 127.0.0.1:1053.
4. The forward update targets <host>.<leg>.lan.fzymgc.house in the leg’s own child zone <leg>.lan.fzymgc.house, and PowerDNS accepts it because that child zone grants DNS UPDATE to that leg’s key (and to the AAAA publisher’s) and the global dnsupdate switch is on. The parent lan.fzymgc.house grants nobody.
5. The reverse update is a second update to a different zone, signed with a different key. D2 builds the plain reverse FQDN from the lease address, longest-suffix-matches it against its configured ddns-domains, and signs it with the zone-wide kea-d2-lan key — and the zone it lands in is the leg’s own carved /24. For Lab (192.168.200.0/22) those are 200.168.192.in-addr.arpa through 203.168.192.in-addr.arpa, each NS-delegated from the covering /16 and each granting DNS UPDATE to kea-d2-lan and to nothing else.

168.192.in-addr.arpa is writable by NOBODY, and that is the design rather than an oversight. The covering /16 is authoritative for everything in 192.168.0.0/16 outside the carved cuts, and it carries the curated PTRs whose sole source is ansible/inventory/group_vars/hosts.yml. A DHCP client must have no path to those records, so the covering zone carries no TSIG-ALLOW-DNSUPDATE and no ALLOW-DNSUPDATE-FROM at all. A reverse update that lands there is REFUSED, and that refusal is correct. The carve boundary is drawn at the leg, not at the kea_pool, so widening a pool needs no zone change — but a leg whose pool spans a /24 that is not carved (main’s .20, core’s .40, deliberately) takes a DHCP_DDNS_NO_REV_MATCH_ERROR on every lease there, and those addresses keep answering in reverse from the covering /16.

If dig -x on a lease address returns nothing, first check whether the address is inside a carved zone at all. From the resolver’s side “the update was refused” and “there is no zone willing to accept it” look identical, and they have different fixes. Every command below runs ON a resolver node, which has /usr/bin/grep and no ripgrep:

pdnsutil zone list-all | grep in-addr.arpa
pdnsutil metadata get 200.168.192.in-addr.arpa TSIG-ALLOW-DNSUPDATE
# want: TSIG-ALLOW-DNSUPDATE = the zone-wide kea-d2-lan key at the granted generation
pdnsutil metadata get 168.192.in-addr.arpa TSIG-ALLOW-DNSUPDATE
# want: EMPTY — the covering /16 grants nobody, and the read above is this read's control
journalctl -u isc-kea-dhcp-ddns-server | grep -E 'RCODE|Transaction outcome'

pdnsutil prints multi-valued metadata comma-joined onto one line, so a zone granting two keys reads as a single string. Split on the comma before concluding a grant names one writer.

Reverse Change: yes on a DHCP_DDNS_REMOVE_SUCCEEDED line means the reverse half did not ERROR. It does NOT mean a PTR was withdrawn. Deleting an absent RRset is a success in RFC 2136, and D2 reports it as one. The instrument that settles it is the reverse zone’s SOA serial, because PowerDNS advances the serial only on an UPDATE that actually changes records. Reading Reverse Change: yes as proof a PTR was withdrawn sends you looking in the wrong place — at the zone’s grants, or at D2’s domain list, for a failure that never happened. Confirm with the zone or with the serial, never with the journal line alone:

pdnsutil zone list 200.168.192.in-addr.arpa | grep -E '[[:space:]]SOA[[:space:]]'
# want: the serial field to move ONLY across an update that changed records

Why real /24 zones and not RFC 2317. Anyone who reads “per-/24 zones for a /22” will reach for classless delegation, and it cannot work here. RFC 2136 resolves the target zone by walking the ancestor labels of the name being updated, and an update never follows CNAME indirection; ISC’s D2 builds the plain reverse FQDN from the lease address, longest-suffix-matches it against ddns-domains, and rejects when nothing matches. It has no RFC 2317 awareness at all. A real /24 is an ancestor of every name inside it, which is why the carve is real zones. And TSIG-ALLOW-DNSUPDATE / ALLOW-DNSUPDATE-FROM are per-zone only — the zone is the only grant boundary PowerDNS offers, so a narrower one has to be a narrower zone.

The AAAA half is a different program on a different schedule. dns-aaaa-publish runs from a systemd timer every 300 s on the VIP holder only, reads that node’s IPv6 neighbour table, joins it to the Kea lease table on MAC address, and publishes the address the device is actually using — signed with its own key, dns-aaaa-publish, not Kea’s, into the child zone lab.lan.fzymgc.house (its dynamic_zone is scoped to Lab alone).

So each leg’s zone has two writers with two keys, and they fail independently. A records can be publishing perfectly while AAAA records are stale, and vice versa. Ask the daemon rather than trusting this sentence:

# each leg's child: that leg's key and the publisher's
pdnsutil metadata get lab.lan.fzymgc.house TSIG-ALLOW-DNSUPDATE
# want: TSIG-ALLOW-DNSUPDATE = kea-d2-lab<gen>, dns-aaaa-publish<gen> — the granted generation's spelling

# the parent: nobody
pdnsutil metadata get lan.fzymgc.house TSIG-ALLOW-DNSUPDATE
# want: an empty value

An empty parent reading is the CORRECT state and a two-key reading is an unapplied module — the opposite of how every other grant read on this page is interpreted, because this one was removed rather than added. tf/dns-records/metadata.tf records the ruling and why withdrawing it costs no publishing path: the grant governs RFC 2136 dynamic updates only, and Terraform and external-dns write over the HTTP API.

Where the boundary is, and what actually enforces it

The curated apex fzymgc.house accepts no DNS UPDATE — and the reason is NOT that it carries no metadata.

pdnsutil metadata get fzymgc.house TSIG-ALLOW-DNSUPDATE
# want: an empty value

It is false that a zone carrying neither TSIG-ALLOW-DNSUPDATE nor ALLOW-DNSUPDATE-FROM is closed by that absence alone. PowerDNS’s authorisation logic falls back to the global allow-dnsupdate-from range when a zone names no per-zone source, and an update arriving from inside that range against a zone with no TSIG grant would be accepted on that basis. What closes the apex is configuration:

grep -E '^(dnsupdate|dnsupdate-require-tsig|allow-dnsupdate-from)' /etc/powerdns/pdns.conf
# want:
#   dnsupdate=yes
#   dnsupdate-require-tsig=yes
#   allow-dnsupdate-from=127.0.0.0/8,::1

dnsupdate-require-tsig=yes makes the absence of a TSIG grant a REFUSED, regardless of source address. The global range is loopback-only as a second, independent layer — an update from another host is refused before TSIG is even considered. Both lines are required and neither is redundant: drop the first and loopback processes could write the apex; drop the second and the only thing standing between the wider network and a signed update is key custody. The same reasoning is written into tf/dns-records/metadata.tf beside the resource it governs.

TSIG key rotation

Rotation here is by key NAME, with a server-side overlap — never by replacing the secret behind a name the writers are already using. Every key the estate signs DNS UPDATE with carries a generation: the zone-wide kea-d2-lan, the five per-leg kea-d2-<leg> keys and the publisher’s dns-aaaa-publish — seven names at each generation. Generation 1 is the UNSUFFIXED spelling (kea-d2-lab, Vault field secret_lab); a generation n > 1 renders -<n> on the name (kea-d2-lab-2) and _<n> on the Vault field (secret_lab_2). Two knobs choose the generation, and they are the only place one is chosen:

• dns_tsig_key_generation in ansible/inventory/group_vars/dns_hosts.yml — the generation every WRITER (Kea D2, the AAAA publisher) signs with, and the one every current-generation name and Vault field renders from.
• dns_tsig_key_generations_granted in the same file — every generation the PowerDNS STORE holds and every TSIG-ALLOW-DNSUPDATE grant names; and var.tsig_key_generations_granted in tf/dns-records/variables.tf, the SAME list on the Terraform side. The two lists must be equal as sets — a grant on a key the store does not hold is a half-rotation.

The Vault side is FIELDS of the two entries that already exist: secret_<n> and secret_<leg>_<n> on fzymgc-house/infrastructure/dns/kea-d2-tsig, secret_<n> on fzymgc-house/infrastructure/dns/dns-aaaa-publish-tsig. No new path, no new policy grant, no Terraform read of either: a KV v2 read returns the whole entry, so a new field of an existing entry is visible to both roles with nothing else changed.

This procedure gets its own window. Never in the same window as a leg cut or the appliance repoint, because a rotation gone wrong and a cut gone wrong have the identical symptom — names stop appearing — and no way to tell them apart from outside. The same-name delete-and-reimport at the end of this section is break-glass, and it is destructive; the generation path exists so that it is never the first move.

The two latencies, recorded separately

zone-metadata-cache-ttl=60 — the documented upper bound, and on this path NOT the latency. PowerDNS caches domain metadata for 60 s for consumers that read it through the cache. The RFC 2136 grant check does not: isUpdateAllowed reads domain metadata through the backend directly, which the LMDB backend answers from a direct read transaction, and pdnsutil metadata get reads the same uncached store from its own process. Quote the 60 s as the upper bound a metadata consumer that caches would see; do not size anything on this path by it.

LightningStream replication — the latency that governs. Terraform writes a grant through ONE node’s API; the other node receives it by LightningStream (lmdb_poll_interval: 1s, storage_poll_interval: 1s), which takes on the order of 45 s end to end. A grant is therefore present on one node and absent on the other for most of a minute after every merge — a writer that signs against the VIP holder during that minute is REFUSED on one node and accepted on the other, and which one depends on where the VIP sits.

The role does not sleep for either. It reads until the value is there. --tags pdns carries two grant reads (the forward carve and the reverse carve, roles/pdns/tasks/verify.yml), and each is an until over pdnsutil metadata get on the node being converged, retries: 18, delay: 10 — up to ~3 min per host — until the parsed set equals the expected one for the granted generations. A converge that starts before replication finishes waits; a converge that reports GREEN on a host has read the grant on THAT host.

The order, and the three converges

(0) The Vault fields, from the control node, in fish. One value per field, minted independently, never shared across legs; the zone-wide D2 field is spelled secret_<n> (the base field has no leg component). vault kv patch, never vault kv put — a put REPLACES the entry and drops the fields the current generation is signing with:

for leg in core lab main iot mainwireless
    vault kv patch -mount=secret fzymgc-house/infrastructure/dns/kea-d2-tsig secret_{$leg}_<n>=(openssl rand -base64 32)
end
vault kv patch -mount=secret fzymgc-house/infrastructure/dns/kea-d2-tsig secret_<n>=(openssl rand -base64 32)
vault kv patch -mount=secret fzymgc-house/infrastructure/dns/dns-aaaa-publish-tsig secret_<n>=(openssl rand -base64 32)
vault kv get -mount=secret -format=json fzymgc-house/infrastructure/dns/kea-d2-tsig | jq -r '.data.data | keys[]'
vault kv get -mount=secret -format=json fzymgc-house/infrastructure/dns/dns-aaaa-publish-tsig | jq -r '.data.data | keys[]'

The two jq lines read field NAMES only, never values. Want the current generation’s fields beside the new ones on both entries.

(1) PR A — widen. dns_tsig_key_generation: <n> and dns_tsig_key_generations_granted: [<n-1>, <n>] in dns_hosts.yml; default = [<n-1>, <n>] on var.tsig_key_generations_granted. Read the main-cluster-dns-records speculative plan on the PR before merging: the powerdns_zone_metadata.*_tsig_allow_dnsupdate updates in place, each metadata set gaining the -<n> names, nothing added and nothing destroyed. On merge the workspace applies them. Then converge the authoritative tier, both nodes in ONE run:

scripts/dns-playbook.sh --tags pdns

One run and not two --limit runs, deliberately: the store read and the import are run_once on the first host — one writer into the replicated LMDB, one absence guard — and the grant reads are per host, so the until on the second host is what proves the peer received both the grant and the imported keys. Two --limit runs would give the second host its own absence guard against a store that may not have replicated. A writer must not converge until --tags pdns has passed on BOTH resolvers. The writer signs against whichever node holds the VIP, and a grant that has landed on one node is a REFUSED update on the other; the until reading GREEN on each host is the only evidence that both hold it. Do not read the first host’s GREEN as the pair’s.

(2) The writers — straight to <n>, no writer-side overlap.

scripts/dns-playbook.sh --tags kea,dns-aaaa-publish

D2 restarts onto the new names in one converge (tsig-keys[] and every ddns-domains key-name move together, there is no D2-side overlap by design) and its per-key counters start from zero under the new key objects; the publisher’s key name and key file switch, and the running process picks them up at its next tick. Take the reads in the next subsection before going on.

(3) PR B — narrow. dns_tsig_key_generations_granted: [<n>] and default = [<n>], on both sides again; on merge the grants narrow to the -<n> names alone. Then --tags pdns on both nodes once more: the run_once pdnsutil tsigkey delete removes the previous generation’s seven keys from the store (guarded on presence, for an honest changed), and d2-tsig-key-present proves the store equals the declared set — in both directions — on both nodes. The previous generation’s .tsig files stay on the Kea nodes, 0640 root:_kea, material for a key the server no longer holds, until the next --tags kea converge removes them; nothing loads them in the meantime, and nothing restarts for their removal.

Never the narrowing before the writers have converged. That is withdraw-then-converge, the destructive order named below.

Reading the keys and the grants on-box

Every command here runs ON a resolver node, which has /usr/bin/grep, cut and awk and no ripgrep. pdnsutil tsigkey list prints the SECRET as its third field — never paste its raw output anywhere. The name-only list is pdnsutil tsigkey list | cut -d " " -f1 | sort; want fourteen names during an overlap (seven at each generation) and seven after the narrowing. The grant on a carved forward zone reads comma-joined on one line, in the order stored; the reverse twin carries the zone-wide key alone in each spelling. And the reading that NAMES the key — the only one that does — is D2’s own per-key statistics: statistic-get-all on the rendered control socket /run/kea/kea-ddns-ctrl-socket (0750 _kea:_kea, hence sudo) reports key[<name>].update-success per configured key. Neither PowerDNS’s success line (UPDATE (<id>) from <ip> for <zone>: Update completed, N changed records committed.) nor D2’s Transaction outcome line names the key, which is why the statistic is the reading of record. A positive update-success under a new-generation key is post-converge by construction: the counters live on the key object and reset when D2 recreates it, so nothing from before the converge can be counted under the new name.

pdnsutil tsigkey list | cut -d " " -f1 | sort
# want (overlap): fourteen names, seven of them ending -<n>; want (narrowed): the seven -<n> names
pdnsutil metadata get lab.lan.fzymgc.house TSIG-ALLOW-DNSUPDATE
# want (overlap): TSIG-ALLOW-DNSUPDATE = kea-d2-lab<n-1>, dns-aaaa-publish<n-1>, kea-d2-lab-<n>, dns-aaaa-publish-<n>
# want (narrowed): TSIG-ALLOW-DNSUPDATE = kea-d2-lab-<n>, dns-aaaa-publish-<n>
pdnsutil metadata get 200.168.192.in-addr.arpa TSIG-ALLOW-DNSUPDATE
# want (overlap): TSIG-ALLOW-DNSUPDATE = kea-d2-lan<n-1>, kea-d2-lan-<n>
# want (narrowed): TSIG-ALLOW-DNSUPDATE = kea-d2-lan-<n>
sudo /opt/dns-resolver/venv/bin/python -c 'import json, socket; s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM); s.connect("/run/kea/kea-ddns-ctrl-socket"); s.sendall(json.dumps({"command": "statistic-get-all"}).encode()); s.shutdown(socket.SHUT_WR); data = b"".join(iter(lambda: s.recv(65536), b"")); stats = json.loads(data)["arguments"]; print({k: v[0][0] for k, v in stats.items() if k.startswith("key[") and k.endswith("update-success")})'
# want (after step 2, once a lease event has happened): a positive count under a key[...-<n>...] name

Split the grant’s right-hand side on the comma before concluding anything about how many writers it names; the whole line is one string.

The destructive orders, named

Withdraw-then-converge is the order this procedure exists to avoid. Narrowing the grant (PR B) while a writer signs with the previous generation REFUSES every one of its updates from the moment the merge applies until the writers converge — and the refusal reads like a zone problem on the resolver, hours later, with the writer’s own service reporting success the whole time. The sequence above puts the writers’ converge (2) strictly between the widen (1) and the narrow (3), and the reads in between, so that no grant is ever withdrawn from a key that is in use.

The break-glass, same-name path — destructive, kept for the case the generation path cannot cover. When the secret behind a name must change WITHOUT a generation bump — the material leaked, say, and there is no time for two PRs:

pdnsutil tsigkey delete <name>

on ONE node only (the store is LightningStream-replicated and the import is run_once; two nodes deleting is two writers racing one database), then wait ~45 s for replication, confirm absence on BOTH nodes with the name-only list above, then converge --tags pdns — the absence guard fires and re-imports the name from whatever Vault holds. Nothing under that name verifies between the delete and the converge: every UPDATE signed with it is REFUSED for the whole interval, on both nodes, and nothing announces it. That is what “destructive” means here, and why the generation path above is the default and this one is not.

“Refused” means DNS-refused, and NEVER lease-refused

This is the sentence most likely to be misread during an incident, so it is stated on its own.

A client that loses a DNS name conflict — DHCID says the name belongs to a different client — keeps its address, its gateway, its subnet mask and its resolvers, in full. It simply has no forward record. DDNS sits strictly downstream of leasing: a DNS failure of any kind, at any layer, degrades nothing about DHCP.

So when someone reports “Kea is refusing clients” and points at a REFUSED in a DNS log, those are two unrelated statements. Confirm DHCP health from the lease table (DHCP-12 above), not from anything in this section. A leg whose entire DDNS tier is dead serves addresses normally.

A client that supplies a MULTI-LABEL hostname gets no name, silently

Kea appends the leg’s ddns-qualifying-suffix only to a partial — single-label — hostname. A client that sends a hostname already containing dots is treated as fully qualified and the suffix is not appended. The FQDN then falls outside lan.fzymgc.house, D2 matches no configured forward domain, and discards the request without ever contacting PowerDNS.

A client sending nas.fzymgc.house produces a DHCP_DDNS_NO_MATCH No DNS servers match FQDN nas.fzymgc.house. warning followed by DHCP_DDNS_NO_FWD_MATCH_ERROR … the configured list of forward DDNS domains does not contain a match.

Why it is hard to see. Everything else looks healthy:

What you will look at	What it says	What is actually true
The lease (lease4-get-all)	"fqdn-fwd": true	Kea asked for a forward update. It says nothing about whether one happened.
journalctl -u isc-kea-dhcp4-server	clean	The discard happens in D2, a different daemon in a different unit.
--tags pdns-verify / kea-verify	all green	They assert configuration. Neither observes a published name.
The zone	no record	Correct — and the only error anywhere is in D2’s journal.

fqdn-fwd: true IS NOT EVIDENCE THAT A NAME WAS PUBLISHED. Read the zone, or read D2’s journal.

This is a property of the naming policy, not a bug in it: the same mechanism is what stops a Lab client asserting a name in the curated apex. Changing it means revisiting the naming requirement, which is an operator decision rather than a triage step.

A name D2 missed comes back on the client’s next renewal — and a deleted record comes back with it

ddns-update-on-renew is true on both nodes. Automatic recovery is a property of this estate, and it is not free — the two costs it was taken with are stated here rather than left to be discovered.

• A released or expired lease’s name is removed by D2 through the normal CHG_REMOVE path. The ordinary case needs nothing from you.
• A name D2 MISSED while it was down DOES come back on the client’s next renewal. A renewal emits a Name Change Request even when neither the FQDN nor the DHCID has changed — Kea completes the DHCPREQUEST → lease allocation → DHCPACK and then hands D2 an update for the lease it has just renewed. Renewal therefore republishes a name D2 missed, unattended, within one renewal cadence: kea_renew_timer, 21600 seconds (six hours) in ansible/roles/kea/defaults/main.yml. So the recovery path is to WAIT ONE RENEWAL, and a client RELEASE and RE-ACQUIRE is the way to go faster — the manual step is an accelerator rather than the only way back.
• A renewal queues TWO Name Change Requests, not one: a CHG_REMOVE for the old lease, then a CHG_ADD for the new one. Every renewing client’s name therefore has a brief ABSENCE WINDOW once per renew interval. The window is milliseconds — the two requests queue back to back — but a resolver querying inside it gets NXDOMAIN and caches that negative answer for up to the zone’s 60 s SOA minimum. The exposure is a small probability of a 60-second negative cache, per client, per six hours. This cost was ruled ACCEPTABLE and is recorded here rather than mitigated: shortening the zone’s negative TTL would shorten every negative answer this estate gives and not only this window.
• Record deletion is not durable, and that is the second accepted cost. Deleting a name from the zone by hand removes it only until that client next renews — a deleted record returns on the client’s next renewal, within the cadence above, and there is no way to hold a name down while its lease is alive. A hold-down mechanism was offered and declined. If you need a name gone and to stay gone, take the LEASE away, not the record.
• A claim refused by a seeded reserved label fails TWICE per renewal, per offending client. The CHG_REMOVE fails and the CHG_ADD then fails at the replace step, so a device claiming a reserved label produces two failed D2 transactions every renew interval rather than one at acquisition. That is recurring noise in D2’s journal by design, not a new fault.
• There is no A-record reconciler and none is planned. The only reconciler in the tree belongs to the AAAA publisher, and it skips A-only names by design — its own docstring says “a name holding only an A record is never a candidate, so D2’s forward records are untouched”. That scoping is correct and is not a gap: a reconciler scoped to “everything” would read a failed or empty lease read as an empty desired state, reap the entire authoritative zone on its first cycle, and report a clean run.

Reading the journal when a claim is refused. Two identifiers, and both are LOG_ERROR, so both appear at this estate’s default kea_log_severity:

# ON A RESOLVER NODE. These boxes have no ripgrep — use grep.
journalctl -u isc-kea-dhcp-ddns-server --since '-1h' \
  | grep -E 'DHCP_DDNS_REMOVE_FAILED|DHCP_DDNS_FORWARD_REPLACE_REJECTED'

DHCP_DDNS_FORWARD_REPLACE_REJECTED carries the request id, the server, the FQDN and the RCODE — that is the line proving a seeded label refused a claim. DHCP_DDNS_REMOVE_FAILED is its partner half of the same renewal.

An EMPTY journal is a third answer and not an absence of one. If nothing reached D2 at all, Kea skipped the request before it was ever queued: it returns early when the lease’s hostname is empty or both FQDN flags are false, and it says so as DHCPSRV_QUEUE_NCR_SKIP, which is LOG_DEBUG and therefore INVISIBLE at the default severity. Raise kea_log_severity on the node before concluding that nothing happened:

# ON A RESOLVER NODE.
journalctl -u isc-kea-dhcp4-server --since '-1h' | grep -F 'DHCPSRV_QUEUE_NCR_SKIP'

A ZERO IS ONLY A RESULT IF YOU READ ITS DENOMINATOR IN THE SAME COMMAND. Both reads above answer 0 just as readily when they were never pointed at anything, and neither announces itself: journalctl exits 0 and prints nothing whether the service is clean or the query was wrong.

• The unit is NOT the process name — and the log lines carry the process name. The unit is isc-kea-dhcp-ddns-server.service; every line it emits is tagged kea-dhcp-ddns[<pid>], which is the string an operator reaches for. journalctl -u kea-dhcp-ddns matches no unit, prints -- No entries --, and exits 0 — an empty result that reads as a clean service.
• --since parses the NODE’s local time, and these nodes run America/New_York. A UTC timestamp pasted straight into --since therefore slides the window four hours forward, and for a recent event that window is empty. Pin both ends with TZ=UTC and --utc.

So read the total and the match in the same command, and treat a zero total as “the query was wrong”, never as “the service was quiet”:

# ON A RESOLVER NODE. These boxes have no ripgrep — use grep.
# -o cat -q is what makes an empty result count 0 rather than the 1 line of `-- No entries --`.
SINCE='<the window start, as YYYY-MM-DD HH:MM:SS UTC>'   # TZ=UTC and --utc below are what make that true
sudo TZ=UTC journalctl --utc -u isc-kea-dhcp-ddns-server --since "$SINCE" --no-pager -o cat -q \
  > /tmp/d2.log
echo "total=$(wc -l < /tmp/d2.log)"                                 # THE DENOMINATOR, always first
echo "errors=$(grep -c DHCP_DDNS_NO_FWD_MATCH_ERROR /tmp/d2.log)"   # the numerator

total=0 means you have measured your own query. A non-zero total beside errors=0 means the service processed work and none of it failed. Only the second shape is evidence.

So when a name is missing after a D2 outage: first confirm it is genuinely absent — read the zone (pdnsutil zone list <leg>.lan.fzymgc.house on the node, and resolve the name at both daemons) and read D2’s journal for the transaction, because fqdn-fwd: true on the lease says only that Kea asked. Then decide whether you need it back sooner than the cadence above: if you do, have the client release and re-acquire its lease; if you do not, the client’s next renewal republishes it without you. Do not go looking for a reconciler to run — there isn’t one, and it is renewal, not a reconciler, that brings the name back.

A lease that never renews never heals. A client holding a name whose parent-held copy predates the carve — occluded by the child zone’s delegation, so it answers NXDOMAIN — gets its child-zone name on its next renewal and not before. A device that is not renewing is a device or IPAM question, not a carve defect. Recovering such a name is a one-off release and re-acquire on that host, an operator choice: it is not scheduled work.

How to read each tier

Every command below runs ON a resolver node. The resolver nodes have no ripgrep — use grep.

The three verify tags, each answering a different question:

scripts/dns-playbook.sh --limit <node> --tags pdns-verify
scripts/dns-playbook.sh --limit <node> --tags kea-verify
scripts/dns-playbook.sh --limit <node> --tags dns-aaaa-publish-verify

Tag	Answers	Does NOT answer
pdns-verify	Are the granted generations’ keys in this node’s key store? Do the carved zones grant exactly their writers, and do the parent dynamic zone, the covering /16 and the apex grant nobody? Is the running daemon armed for DNS UPDATE?	Whether any update was ever accepted
kea-verify	Is the D2 config 0640 root:_kea, is its TSIG secret readable as the daemon, does the config parse?	Whether D2 is running, and whether a signed update is accepted — it says so in its own roll-up
dns-aaaa-publish-verify	Is the key 0600, does the config name a present key file, does the timer’s state match its guard, is the last-success stamp fresh?	Whether any AAAA was published

None of the three observes a published name. That is deliberate and is stated in the kea-verify roll-up’s own text. To know whether names are being published, read the zone.

A CONFIG FILE IS NOT A RUNNING PROCESS. PowerDNS reads its configuration only at start. A converge that writes pdns.conf and then fails before its handlers flush leaves the file correct and the process unchanged; every subsequent converge finds the file already correct, reports ok rather than changed, and never re-notifies the restart. A node in that state carries dnsupdate=no in the running daemon while its config file says yes, and every instrument that reads the file, the render or the replicated LMDB stays green over it.

The reading that tells a file from a process:

pdns_control current-config | grep dnsupdate
systemctl show pdns -p ExecMainStartTimestamp
stat -c '%y %n' /etc/powerdns/pdns.conf

Two things to know before you read that output:

• A COMMENTED # dnsupdate=no is how pdns reports a setting left at its DEFAULT. It means NOT ARMED — it does not mean “not mentioned”, and it is not an absence you can shrug at. The armed reading is an uncommented dnsupdate=yes with an uncommented dnsupdate-require-tsig=yes.
• A process start timestamp OLDER than the config file’s mtime is the signature. If ExecMainStartTimestamp predates the stat mtime, the running daemon is not running the file you are reading. The fix is systemctl restart pdns on that node.

The running-dnsupdate-armed criterion in ansible/roles/pdns/tasks/verify.yml is the one criterion in the tier that reads the running daemon — the rest read the LightningStream-replicated LMDB. When a gate is GREEN and the live estate is RED, the first hypothesis is the instrument, not the estate — specifically, that the instrument is reading a different environment from the one production reads.

The zone itself:

pdnsutil zone list lab.lan.fzymgc.house
pdnsutil zone list lab.lan.fzymgc.house | grep -c -E "[[:space:]]IN[[:space:]]+A[[:space:]]"

pdnsutil zone list emits TAB-separated records. A pattern written with a literal space — " IN A " — matches nothing on a perfectly healthy zone. Match whitespace as a class, as above. And do not wrap a counting command in || true: grep -c exits 1 on zero matches, and that non-zero exit is the useful signal. Suppressing it turns “I found nothing” into a pass.

The key store — and it PRINTS SECRETS:

pdnsutil tsigkey list | awk '{print $1}'
# want: the seven names of the granted generation — kea-d2-lan, the five kea-d2-<leg> and dns-aaaa-publish, each with its generation suffix

Never run pdnsutil tsigkey list bare into a terminal you are pasting from, a ticket, or a log. Its third field is the base64 HMAC secret. Reduce to names on the box, as above.

Is D2 alive and listening:

systemctl is-active isc-kea-dhcp-ddns-server
ss -lnup | grep 53001
journalctl -u isc-kea-dhcp-ddns-server --since '-1 hour' --no-pager

Note the asymmetry that will waste your time once: the unit is isc-kea-dhcp-ddns-server.service, the package is isc-kea-dhcp-ddns. dpkg -l isc-kea-dhcp-ddns-server reports Installed: (none) on a node where D2 is running perfectly.

The publisher’s stamp:

systemctl is-active dns-aaaa-publish.timer
systemctl list-timers 'dns-aaaa-publish*' --all
cat /run/dns-aaaa-publish/last-success
journalctl -u dns-aaaa-publish.service --since '-1 hour' --no-pager

What a stale or absent stamp does and does NOT mean. The stamp is written only on a cycle that completed its work, and only on the VIP holder.

• On the non-holder, an absent stamp is CORRECT. The publisher is a singleton; the non-holder logs sweep not-holder and exits without stamping. /run/dns-aaaa-publish/ may not even exist there. This is the expected steady state on one of the two nodes at all times.
• On the holder, a stale stamp means the last cycle did not complete — and it deliberately does not tell you the zone is wrong. Records already published keep their values; nothing is withdrawn by a publisher that is not running.
• A stamp is not a pulse. It records the last success, not the last attempt. Read the journal for attempts.

Because the VIP moves, so does the stamp. After a failover, the new holder starts stamping and the old one stops; that is the mechanism working, not a fault.

The two silent degradations, named

Neither of these produces an error anywhere except the place named.

1. D2’s queue limit. kea-dhcp-ddns accepts NCRs into a bounded queue — max-queue-size, 1024 (kea_ddns_max_queue_size). When the queue is full, kea-dhcp4 suspends sending DDNS updates until the backlog clears. Leasing is unaffected and no DHCP client notices. The only places this is visible are D2’s journal and the growing gap between leases and records. If names are appearing late or in bursts under load, read D2’s journal before suspecting the zone.

2. The publisher’s failure modes, which are asymmetric by design.

What failed	What the publisher does	Stamp
Lease read (Kea control socket)	publishes nothing, withdraws nothing, logs the reason	not written
Zone read (pdnsutil)	publishes nothing, withdraws nothing, logs the reason	not written
Holder check	assumes it is not the holder and does nothing	not written
Peer neighbour read	publishes the holder’s own observations only, logs degraded and names the peer	written

A failed read publishes NOTHING rather than an empty set, and that is the single most important property in the tool. An empty desired state would mean “withdraw everything”, so a reconciler that reads a failed read as an empty table reaps the zone and reports a clean run. The tool refuses to express a failed read as an empty answer, and the absent stamp is how you find out.

The peer row is dormant. The ruled configuration is a single-node read on the VIP holder with no cross-node transport — peer_host is empty in the deployed config — so a peer read cannot fail because none is attempted. The row is documented because the code path exists and becomes live if a peer is ever configured.

What observation-based AAAA does not cover

A device holding a lease that has sent no IPv6 to either resolver appears in neither neighbour table and gets no AAAA record. This is a property of the mechanism rather than a defect in it: the publisher publishes addresses it has observed, which is precisely what stops it emitting AAAA records pointing at addresses nothing holds. Deriving addresses instead would be arithmetic on a false premise — Windows, Android, macOS/iOS and Linux under NetworkManager all use RFC 7217 stable-privacy addresses that are not derived from the MAC — and RFC 6724 makes resolvers try AAAA first, so every such lookup would hang until the v6 timeout before falling back.

A Lab client can hold an in-scope ULA address configured by SLAAC from the pair’s own RA while neither resolver’s neighbour table carries it, because it has never sent IPv6 to a resolver from that address. The publisher’s input is then the empty set, and a cycle reporting sweep ok: 0 name(s) published is the correct, healthy reading — the chain is connected and has nothing to carry. Do not triage it as a fault. The reading that would change it is a non-zero in-scope count:

ip -6 neigh show dev eth0.1000 | grep "fddb:f665:73f7:4:"

The empty set is an INPUT STATE, not a defect. No work is planned to manufacture IPv6 traffic so that the count stops being zero.

One residual risk, accepted rather than defended

A release carrying another client’s identifier can delete that client’s record. The published DDSpoof technique abuses the fact that a DHCP Release is unauthenticated: a host on the leg can send a Release carrying a victim’s client identifier and cause the victim’s name to be withdrawn. DHCID does not stop this and no configuration available here does.

It is recorded as an accepted, homelab-sized residual rather than claimed as mitigated. The blast radius is a missing name for one client until its next renewal, and the attacker must already be on the leg. Know that it exists so that a name disappearing for no apparent reason has a candidate explanation that is not a broken pipeline.