Skip to content

Internal DNS (the resolver pair, and the gateway beside it)

The resolver pair sandiego and portland answers client DNS for every VLAN over IPv4 and over IPv6. The UCG-Fiber gateway serves no DHCP. Its dnsmasq answers a client on a trusted leg that carries a hand-set .1 nameserver, and forwards every query to the pair. It produces no curated zone and redirects no DNS query. In an incident, read Which tier answers each VLAN, and how to prove it first, then Things that will bite you.

A Firewalla appliance was the gateway until the cutover of 2026-10-01.

The pair has no ripgrep. Every command written to run on a resolver node uses grep. Commands that run against this repository on a workstation use rg.

Which tier answers each VLAN, and how to prove it

Section titled “Which tier answers each VLAN, and how to prove it”
Client Asks Path Answered by
Trusted VLAN (Core, Lab, Main, MainWireless, IoT), IPv4 192.168.<vlan>.5, the leg’s floating address on-link at layer 2 to the node that holds the VRRP sync group. The gateway is not in the path the main recursor process pdns-recursor@main on that node. fzymgc.house goes to pdns at 127.0.0.1:1053, everything else to dnscrypt-proxy on 127.0.0.1:5300, which reaches a public resolver through a relay
Guest or Telework, IPv4 192.168.79.5 or 192.168.217.5 the same layer 2 path to the same node the public recursor process pdns-recursor@public on that node, which answers every name from public DNS only. fzymgc.house and both reverse zones get NXDOMAIN from the withheld zone of the process, and no forwarder and no relay sees the name
Remote client over the tailnet, IPv4 100.102.168.37 (sandiego) or 100.124.42.55 (portland), each node’s own tailnet address no VIP and no subnet route in the path the same main process on that node
Trusted VLAN (Core, Lab, Main, MainWireless, IoT), IPv6 the leg’s v6 VIP on the resolver pair, learned as RDNSS from the RA of the gateway and from the RA of the pair on-link the same main process on that node
Guest or Telework, IPv6 the leg’s v6 VIP, learned as RDNSS from the same two RAs on-link the public process on that node
Trusted VLAN, a hand-set 192.168.<vlan>.1 the gateway the gateway’s dnsmasq forwards the query to the Core VIP 192.168.40.5 the main process on the node that holds the Core VIP

This table says nothing about DHCP. No device serves DHCPv6. Kea on the pair serves DHCPv4 on all seven legs.

The seven VLAN legs and their floating addresses

Section titled “The seven VLAN legs and their floating addresses”
VLAN 802.1Q Network Floating address
Core untagged native 192.168.40.0/22 192.168.40.5
Lab 1000 192.168.200.0/22 192.168.200.5
Main 3000 192.168.20.0/22 192.168.20.5
IoT 3020 192.168.218.0/24 192.168.218.5
MainWireless 3001 192.168.24.0/22 192.168.24.5
Guest 3100 192.168.79.0/24 192.168.79.5
Telework 3200 192.168.217.0/24 192.168.217.5

The pair holds a floating address on all seven legs. ip -br addr on either node lists seven VIPs, and this table accounts for all seven. Guest and Telework are dns_vlan_interfaces entries that declare trusted: false, a kea_subnet_id and a kea_pool. Kea serves DHCPv4 on both. See The one value an operator edits to change trust below.

Bringing a leg into service, and how to tell whether it is

Section titled “Bringing a leg into service, and how to tell whether it is”

Declaring a leg in dns_vlan_interfaces does not serve it. Five consumers read that structure, and each needs its own converge and its own adoption step. Check them one by one. A leg that is short any one of them presents to a client as total DNS failure:

Consumer Check on the node Adoption step
networkd legs ip -br addr show | grep <vlan-id> networkctl reload. A converge writes the files only
nftables accepts sudo nft list ruleset | grep -c '<iface>' the reload handler. Make sure that the dport 53 sets name every leg
recursor listeners sudo ss -tlnup | grep '<leg-addr>:53'. On Guest or Telework the PID is systemctl show -p MainPID --value pdns-recursor@public a restart, through --tags pdns-recursor
keepalived VIPs sudo grep -c '^vrrp_instance' /etc/keepalived/keepalived.conf never a converge alone. See Promotion
RA announcer ActiveEnterTimestamp against the config mtime systemctl restart dns-common-ra-announce. There is no reload path

Kea, both router advertisements and the gateway point at the VIP, not at a node address, so keepalived is the consumer that decides whether a client sees anything at all.

systemctl is-active cannot tell you that a daemon matches its configuration. It reports that a process exists. Compare ActiveEnterTimestamp against the config file’s mtime. A unit can be active while it runs a configuration older than the file on disk.

Probe an on-segment resolver from the segment. The UCG-Fiber redirects no DNS query, so a query from another subnet is routed to the address it names. It then measures the routed path, the zone firewall and the node ruleset, and not what a client on the segment sees. Keep a control query to an address on that subnet that does not exist, and treat ip neigh show <addr> reading INCOMPLETE as the negative.

192.168.x.5 is on-link for a client on that VLAN, so a query to the floating address is switched at layer 2 and never reaches the gateway. The pair is in the path because every leg advertises the floating address as its DNS server: Kea’s option 6 over IPv4, and RDNSS over IPv6. The gateway holds no DNS redirect, no ad blocking and no content filter.

whoami.dns.invalid is the instrument. Each recursor process serves the zone dns.invalid. from a file, and whoami in it holds the Core address of the node. The main process answers that address, which shows which node answered. The public process answers NODATA, because its rebinding zone drops the private address, so take this reading on a trusted leg. It does not show which address the client asked, because the gateway forwards every name to the pair: a query to 192.168.x.1 returns the same node address as a query to the VIP.

Take the query from a host on that VLAN. A query from anywhere else is routed.

Terminal window
# From a host holding an address on that VLAN. On a resolver node, -b selects the leg.
dig -b 192.168.24.7 @192.168.24.5 whoami.dns.invalid +short # want: the VIP holder's Core address
dig -b 192.168.24.7 @<unused address on the leg> whoami.dns.invalid +short +time=2 +tries=1 # CONTROL, want: a timeout

Both readings are required. The control that times out is what makes the answer attributable.

The discriminator makes sure that the recursor answers a query from that source. It does not show what the network advertises. The DHCP OFFER shows the advertisement, and the discriminator shows that the recursor answers the floating address. Both are needed, and they are different claims.

A client that carries a hand-configured 192.168.x.1 nameserver asks the gateway. The gateway’s dnsmasq forwards every query to the Core VIP 192.168.40.5. A client on a trusted leg gets a correct answer, with the internal view and the public lists of the gateway’s source address and not of its own leg. The query row names that address as the client, not the device. The source address is not measured. 192.168.40.1 is the likely one, because the VIP is on-link on Core. A Guest or Telework client gets no reply from the gateway, because a zone policy blocks DNS from each of the two legs to it.

The remedy is hand work: find each statically configured client and repoint it to the VIP of its leg. That is not repository work.

Remote clients resolve .house at each node’s own tailnet address, not at a floating address: sandiego 100.102.168.37, portland 100.124.42.55. The coordination server assigns them and nothing in this repository guarantees them. They share no pattern. Do not tidy them into one.

Three artifacts carry this path. Removing any one of them takes remote .house resolution with it, including the recovery path the auto-revert leans on:

Artifact Where Scope
the nftables accept roles/dns-common/templates/nftables-dns.conf.j2, guarded by dns_common_tailnet_dns_enabled iifname the tailnet device and ip saddr 100.64.0.0/10, udp and tcp, each with a counter
the client-tier listener roles/pdns-recursor, derived from that host’s own dns_tailnet_address into the listen list of the main process one address per node, its own. Never the peer’s and never a wildcard
the tailnet policy tailscale/policy.hujson, applied by its own GitHub Action grants port 53 on tag:ns and nothing wider

The tailnet’s own access-control policy is the authorization tier. The node ruleset is a narrowing. 100.64.0.0/10 is every tailnet address there is, so the source scope excludes a spoofed non-tailnet source on the tunnel and nothing more. Narrowing which peers reach port 53 is a tailscale/policy.hujson change.

There is no subnet route in this path and no floating address. The tailnet device is not in dns_vlan_interfaces, because that structure has five consumers and a sixth pseudo-leg would produce a VLAN device, a floating address and a VRRP instance electing over a tunnel.

The client view: which clients see the internal fzymgc.house, and what the others see

Section titled “The client view: which clients see the internal fzymgc.house, and what the others see”

Each resolver node runs two PowerDNS Recursor processes from one unit template, pdns-recursor@.service. Until 2026-10-14 the tier was Blocky, in two processes with the same split.

  • pdns-recursor@main.service listens on the trusted legs, the tailnet and 127.0.0.1. It forwards fzymgc.house and both reverse zones to pdns-auth on 127.0.0.1:1053, with no fallback, and every other name to dnscrypt-proxy on 127.0.0.1:5300. It answers NXDOMAIN for the whole of fzymgc.house and both reverse zones to a client on no declared leg. It blocks the public lists that each trusted leg declares. See The public blocklists. Its configuration is /etc/powerdns/recursor-main.yml and the include directory /etc/powerdns/recursor-main.d/.
  • pdns-recursor@public.service listens on the untrusted legs (Guest and Telework) and 127.0.0.3. It has no forward zone to pdns-auth and loads no zone from it. Its withheld zone answers NXDOMAIN for fzymgc.house and both reverse zones before any forwarder sees the name, the public names of the zone included. Its rebinding zone drops any answer that carries a private address. Every other name goes to dnscrypt-proxy. It blocks the public lists that Guest and Telework declare. Its configuration is /etc/powerdns/recursor-public.yml.

Every block rule and allow rule of both processes is a response policy zone (RPZ). An RPZ is a DNS zone whose records are rules: the owner name is the trigger, and the data is the action. A block is CNAME . and answers NXDOMAIN. An allow is CNAME rpz-passthru. and lets the name resolve. A process consults its zones in a fixed order, and the first zone that matches a name decides the answer. Which zones a client gets, and in which order below lists them.

The trusted attribute on each dns_vlan_interfaces entry selects the process. The decisions are Guest and Telework resolve public DNS only and, for the main process, Untrusted clients lose the whole fzymgc.house apex.

To talk to a process, name its configuration and its socket directory:

Terminal window
# On a resolver node, as root or as the user dns-worker. <p> is main or public.
rec_control --config-dir=/etc/powerdns --config-name=<p> --socket-dir=/run/pdns-recursor-<p> rpzstatistics

Each process also serves an HTTP API on loopback: 127.0.0.1:8082 for main and 127.0.0.1:8083 for public. The key is in /etc/powerdns/recursor-api.key (root, mode 0600), with a copy for the node worker at /etc/dns-node-worker/recursor-api.key. /metrics on each API port serves the Prometheus metrics of the process with no key.

Who is exempt, and why each member is a member

Section titled “Who is exempt, and why each member is a member”

The exemption is by client source address. The Lua script /etc/powerdns/policy.lua reads the leg map of its process, /etc/powerdns/legs-main.map or legs-public.map. The map holds one line for each leg network, with the list groups of the leg. A client whose source address is in a line of the map of the main process is on a declared leg, so the withheld zone does not apply to it. A client on no line gets the withheld zone and the groups of dns_default_list_groups.

Member Lines in legs-main.map Why it is a member Where the value comes from
Core leg 192.168.40.0/22, fddb:f665:73f7:6::/64 the core entry declares trusted: true the trusted attribute on that entry in ansible/inventory/host_vars/{sandiego,portland}.yml
Lab leg 192.168.200.0/22, fddb:f665:73f7:4::/64 the lab entry declares trusted: true same
Main leg 192.168.20.0/22, fddb:f665:73f7:1::/64 the main entry declares trusted: true same
IoT leg 192.168.218.0/24, fddb:f665:73f7:3::/64 the iot entry declares trusted: true same
MainWireless leg 192.168.24.0/22, fddb:f665:73f7:7::/64 the mainwireless entry declares trusted: true same
keepalived health client 127.0.0.1/32 dns-health.fzymgc.house sits under the withheld apex and is dug from loopback on each node. A withheld zone that catches it blackholes the health check on both nodes at once. The nftables rate meter also leaves loopback out, for the same reason pdns_recursor_health_client_address in ansible/roles/pdns-recursor/defaults/main.yml
the tailnet 100.64.0.0/10 A tailnet client’s 100.x source is in no leg CIDR, so without this line the per-node tailnet :53 listener answers NXDOMAIN. This trusts every tailnet peer, the same posture the node ruleset takes. Narrowing it is a tailscale/policy.hujson change pdns_recursor_tailnet_cidr in ansible/roles/pdns-recursor/defaults/main.yml

The loopback line and the tailnet line carry the groups of dns_default_list_groups.

Which zones a client gets, and in which order

Section titled “Which zones a client gets, and in which order”

The zone list of a process is the precedence. The Lua script discards, for each query, every owned zone that is not in the zone set of the client, and the recursor applies the rest in this order:

Order Zone Source Holds Who gets it
1 withheld.rpz.internal. a file that the role renders fzymgc.house and both reverse zones, as NXDOMAIN every client of the public process, and a client of the main process on no declared leg
2 rebinding.rpz.internal. a file of the role every private, ULA, loopback and link-local range, as NODATA every client of the public process. The main process has no such zone
3 allow.rpz.internal. a file that the role renders the owned zones of dns_list_allowlist and the names of dns_list_excepted_names, as rpz-passthru. every client of both processes
4 <scope>.scope.rpz.internal. pdns-auth on 127.0.0.1:1053, by zone transfer the entries of one scope of dns-control, in the order of the scopes the members of the scope, on the main process
5 operator.rpz.internal. pdns-auth on 127.0.0.1:1053, by zone transfer the list entries of dns-control with no scope every client of the main process
6 <group>.feed.rpz.internal. /var/lib/pdns-recursor-feeds/<group>.feed.rpz one zone for each public list group, in the order hagezi, tif, crypto, domains the clients of a leg that declares the group in dns_list_groups

The order sets four rules. A client of the public process, and a client of the main process on no declared leg, never gets an internal name, and no such query leaves the process. An owned name or an excepted name always resolves for every other client. A scoped entry beats an entry with no scope. An operator entry beats every feed.

The main process spreads this list over three files, and the file order is the zone order:

File Holds Written by
/etc/powerdns/recursor-main.yml the withheld zone and the allow zone the role
/etc/powerdns/recursor-main.d/50-scopes.yml each scope zone the node worker
/etc/powerdns/recursor-main.d/90-tail.yml the operator zone and the feed zones the role

The scope map /var/lib/dns-worker/scopes.map names the members of each scope. The node worker writes it: the line # version N first, then one <scope> <cidr> line for each member. The public process has no scope map. The main process keeps a copy of each zone from pdns-auth in /var/lib/pdns-recursor-main/, and starts from the copy when pdns-auth is down.

Guest (3100) and Telework (3200) declare trusted: false, so the public process serves them. A client on those legs gets what public DNS returns, with the internal zones withheld:

Query Answer
fzymgc.house and every name under it, id.fzymgc.house included NXDOMAIN, from the withheld zone of the public process. The query never reaches dnscrypt-proxy
a PTR under 168.192.in-addr.arpa or under 7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa NXDOMAIN, from the same zone
whoami.dns.invalid NODATA. The identity zone holds the Core address of the node, and the rebinding zone drops the private answer
health.dns.invalid NXDOMAIN, from the identity zone dns.invalid. of the process. The keepalived check asks for it
a public name whose answer is a private address NODATA, from the rebinding zone
any other name the public answer, through dnscrypt-proxy and a relay

No internal address appears in an answer. If a guest device cannot reach an internal service by name, that is working as designed. Until 2026-10-14 a Guest client got the public Cloudflare addresses for id.fzymgc.house and REFUSED for an internal name, because Blocky forwarded the zone to dnscrypt-proxy, which refused it.

The UCG-Fiber redirects no DNS query. A Guest or Telework query to the gateway gets no reply, because a zone policy blocks DNS from each of the two legs to the gateway. A query to a public resolver reaches that resolver and gets the public answer.

Take each reading from a host on that leg. A reading from any other host is routed, and the zone firewall of the gateway decides it. Compare with DNS over HTTPS (DoH), which no device on the path can rewrite.

No workstation needs to be on the leg. The untrusted-leg-dns-check skill takes these readings from a throwaway container that the nas-throwaway-lxc skill puts on the leg from the NAS. Both live under .claude/skills/, and the scripts need a Vault session. The script of the skill also takes --server <address>, which runs the answer readings against one server and skips the on-link readings. The commands below are the same readings, for a host that is already on the leg.

Terminal window
# From a host on Guest. For Telework, replace 192.168.79 with 192.168.217 throughout.
# Repeat the first and third lines against the v6 VIP, fddb:f665:73f7:2::5 (Telework ::5:5).
dig @192.168.79.5 cloudflare.com A +short | sort
curl -s -H 'accept: application/dns-json' \
'https://cloudflare-dns.com/dns-query?name=cloudflare.com&type=A' # want: the same addresses
dig @192.168.79.5 id.fzymgc.house A +noall +comments | grep -c 'status: NXDOMAIN' # want: 1
dig @192.168.79.5 dns-health.fzymgc.house A +short # want: no output, never a 192.168. address
dig @192.168.79.5 whoami.dns.invalid A +noall +comments +answer # want: NOERROR and no answer record
dig @<unused address on the leg> cloudflare.com +time=2 +tries=1 # decoy: want a timeout
dig @192.168.79.1 dns-health.fzymgc.house A +time=3 +tries=1 # hand-set gateway: want a timeout or no address
dig @1.1.1.1 dns-health.fzymgc.house A +short # hand-set public resolver: want no output

If the decoy answers, something other than the pair answered the other readings. The reading for whoami.dns.invalid tells the two processes apart. The main process answers the Core address of the node, and the public process answers NODATA, because its rebinding zone drops that address. An address in that answer means that the main process answered the leg, which is a finding.

The last two readings are for a client that sets its resolver by hand. The UCG-Fiber blocks DNS from Guest and Telework to itself, so the query to .1 gets no reply, and the check script counts that as a PASS. The query to 1.1.1.1 reaches the public resolver, which holds no address for the name.

This section covers the public recursor process, pdns-recursor@public. The alert message points at this heading, so a rename of the heading also changes the message in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml.

The ClickStack alert DNS public resolver failing (Guest/Telework) fires when failed resolutions reach 0.05 of the answers of the public process over 15 minutes. A failed resolution reaches the client as SERVFAIL. The collector on each node scrapes pdns_recursor_servfail_answers and pdns_recursor_questions from the API port of the public process, 127.0.0.1:8083, and copies them to dns_resolver_errors_total and dns_resolver_responses_total under the ServiceName dns-resolver-public. The alert reads the copies with the filter ServiceName = 'dns-resolver-public', and the ratio sums both nodes.

The ratio grows with the duration of a failure, so a short upstream blip does not page. The keepalived check on each node asks the public process for health.dns.invalid every 2 seconds, and each answer counts in the denominator. So a stray error on a quiet leg does not move the ratio. The keepalived check itself does not see an upstream failure, because the process answers that name from its identity zone dns.invalid. with no upstream query. This alert is the signal for an upstream failure on these legs.

To read the ratio, open the tile DNS public resolver failed resolutions per answer on the Cluster — metric alerts dashboard in HyperDX.

To triage:

  1. Find the node that holds the untrusted VIPs: ip -br addr | grep -E '192\.168\.(79|217)\.5/'.
  2. On that node, read the SERVFAIL count of the public process: rec_control --config-dir=/etc/powerdns --config-name=public --socket-dir=/run/pdns-recursor-public get servfail-answers. Read it twice, one minute apart. A count that moves is a live failure.
  3. Compare with the main process. Run the same command with main in place of public. If the main process fails too, the fault is the WAN, dnscrypt-proxy or its relays. Read journalctl -u dnscrypt-proxy on that node and the alerts of The anonymizing upstream.
  4. If only the public process on one node fails, restart it: sudo systemctl restart pdns-recursor@public. keepalived moves the Guest and Telework VIPs only after 5 failed checks, 10 seconds, so a restart does not move them.

Each recursor process sends one protobuf message for each answer to DNS-collector 3.1.0 on the node, the unit dns-resolver-reader.service. The reader listens on 127.0.0.1:6000 for the main process and 127.0.0.1:6001 for the public process. Its binary is /usr/local/bin/dnscollector and its configuration is /etc/dns-resolver-reader/dnscollector.yml. It writes one JSON row for each answer to its journal, and the collector otelcol-dns parses the row.

In default.otel_logs, a query row carries the ServiceName of the process that answered: dns-resolver for the main process and dns-resolver-public for the public process. The name comes from the field dnstap.identity of the row, which is the server_id of the process. The daemon lines of pdns-recursor@main.service and pdns-recursor@public.service carry the same two names, from the map dns_otel_recursor_log_service_names in ansible/roles/dns-otel-collector/defaults/main.yml. The other lines of the reader carry dns-resolver-reader. Every other unit keeps otelcol-dns. The node worker sends its own log over OTLP with the ServiceName dns-node-worker, so the collector reads no journal of dns-node-worker.service. Until 2026-10-14 the query rows carried the names blocky and blocky-public.

The collector gives each query row the engine-neutral names that dns-control reads: client.address, dns.question.name, dns.question.type, dns.response.code and dns.answer. A row that a policy zone decided also gets dns.policy.zone, dns.policy.trigger, dns.policy.hit and dns.policy.action. The zone is the policyName of the RPZ, such as hagezi.feed.rpz.internal.. The trigger is the rule that matched, and the hit is the name that matched it. The action is nxdomain for a block, nodata for the rebinding zone and passthru for an allow. A row that no rule decided carries no policy attribute. The table in ClickStack lists the names. The raw names of the reader, such as dns.qname and powerdns.applied-policy, stay on the row beside them.

No row of dns-health.fzymgc.house or health.dns.invalid reaches ClickStack. The Lua script of each process keeps the two names out of the protobuf, and the processor filter/health of the collector drops a row that arrives anyway.

A row filter in argocd/app-configs/clickstack-chart/values.yaml hides the rows of these names from mcp_readonly and mcp_readwrite. HyperDX and every other ClickHouse user read the rows. If you change a name in the map, change the filter to the same name. Nothing checks that the two are equal.

The collector scrapes three loopback endpoints every 60 seconds: the public process on 127.0.0.1:8083, the main process on 127.0.0.1:8082 and lightningstream on 127.0.0.1:8500. The job name is the metric ServiceName, so the two recursor jobs are dns-resolver-public and dns-resolver. Each job passes only its own named series. The two recursor series are pdns_recursor_questions, every query with the RD bit set, and pdns_recursor_servfail_answers, every SERVFAIL the process gave. The collector copies them to dns_resolver_responses_total and dns_resolver_errors_total under the same ServiceName. dns-control and the ClickStack tiles read the copies.

The collector also takes OTLP over HTTP on 127.0.0.1:4318, for the node worker. Nothing off the node can reach the port. The worker sends its traces, its logs and its metrics there, each with the ServiceName dns-node-worker and the host.name of the node. Its three gauges are dns_worker_client_cert_not_after_seconds, dns_worker_drained and dns_worker_drain_overdue. The traces and the logs of the receiver go through their own pipelines, traces and logs/otlp, so each record keeps the ServiceName of its sender. The metrics of the receiver join the scrape pipeline.

To make sure that the receiver forwards a trace, send one test span on the node. The command runs on the control node and needs the Vault session of scripts/dns-adhoc.sh:

Terminal window
scripts/dns-adhoc.sh <node> -m ansible.builtin.uri -a '{"url": "http://127.0.0.1:4318/v1/traces", "method": "POST", "body_format": "json", "body": {"resourceSpans": [{"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "otlp-receiver-test"}}]}, "scopeSpans": [{"spans": [{"traceId": "0af7651916cd43dd8448eb211c80319c", "spanId": "b7ad6b7169203331", "name": "otlp receiver test", "kind": 1, "startTimeUnixNano": "{{ (now().timestamp() * 1000000000) | int }}", "endTimeUnixNano": "{{ (now().timestamp() * 1000000000) | int }}"}]}]}]}}'

The result reads SUCCESS with "status": 200. Within a minute, this query returns one row for the node:

SELECT ResourceAttributes['host.name'] AS node, Timestamp
FROM default.otel_traces
WHERE ServiceName = 'otlp-receiver-test' AND Timestamp > now() - INTERVAL 10 MINUTE

To read the recent query rows of each node, run this query as the ClickHouse user app, the user that HyperDX reads with:

SELECT ResourceAttributes['host.name'] AS node, count(),
any(LogAttributes['dns.question.name']), any(LogAttributes['dns.policy.zone'])
FROM default.otel_logs
WHERE Timestamp > now() - INTERVAL 10 MINUTE AND ServiceName = 'dns-resolver'
GROUP BY node

What any other source sees on the main process

Section titled “What any other source sees on the main process”

A source on no line of legs-main.map receives NXDOMAIN, not NOERROR with an empty answer, for:

  • fzymgc.house and every name beneath it, for every query type. The withheld zone holds the apex and a wildcard under it as CNAME ., and an RPZ rule does not branch on question type.
  • every PTR under 168.192.in-addr.arpa and under 7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa, so internal hostnames cannot be walked in reverse.

No untrusted leg reaches the main process through a listener. The withheld zone is the fall-through for a source that the leg map does not name.

The one value an operator edits to change trust

Section titled “The one value an operator edits to change trust”

The trusted boolean on the leg’s own dns_vlan_interfaces entry, in both ansible/inventory/host_vars/sandiego.yml and ansible/inventory/host_vars/portland.yml. That is the estate’s single declaration of leg trust. It also selects the recursor process that listens on the leg, and the leg map that names the leg. It is not kea_served_legs. Serving DHCP on a leg and handing that leg the internal namespace are different claims.

Write a real YAML boolean, not a quoted string. trusted: "true" is a non-empty string and is truthy, so it grants trust while it reads like a value under review. Every entry must carry the key. An entry without it fails the render.

trusted does not decide DHCP service. A leg that declares kea_subnet_id gets a Kea subnet, trusted or not.

The template ansible/roles/pdns-recursor/templates/legs.map.j2 renders one map for each process. Each line holds one network and the list groups of its leg, separated by commas, or - for a leg with no group. A trusted leg gives two lines to legs-main.map, its IPv4 network and its ULA /64. An untrusted leg gives the same two lines to legs-public.map. The main map ends with the loopback line 127.0.0.1/32 and the tailnet line 100.64.0.0/10, both with dns_default_list_groups.

The map decides two things for a client of the main process: whether the withheld zone applies, and which feed zones apply. A client on no line gets both the withheld zone and the default groups. A leg that is in dns_vlan_interfaces and not in the map is a render fault, because the role derives the map from that structure. Read the map on the node:

Terminal window
# On a resolver node.
cat /etc/powerdns/legs-main.map /etc/powerdns/legs-public.map

The Lua script reads the map once, at load. After a change, the role restarts the process. A hand edit of the map on the node is lost on the next converge.

Both processes refuse the names on public blocklists with NXDOMAIN. Each list group is one feed zone, <group>.feed.rpz.internal., from the file /var/lib/pdns-recursor-feeds/<group>.feed.rpz on each node. Each leg declares its list groups in dns_list_groups on its dns_vlan_interfaces entry, in both host_vars files. The main process applies them on the trusted legs, and the public process on Guest and Telework. A client whose address is on no declared leg gets dns_default_list_groups. Until 2026-10-14 the attribute was blocky_list_groups.

Group Source Entries
hagezi HaGeZi Pro++, https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/wildcard/pro.plus.txt about 222,000
tif HaGeZi Threat Intelligence Feeds, medium tier, https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/wildcard/tif.medium.txt about 694,000
crypto the Block List Project crypto list, https://blocklistproject.github.io/Lists/alt-version/crypto-nl.txt about 1,300
domains an inline list of single domains and the two Private Relay names of Apple 5

dns_list_sources in ansible/roles/pdns-recursor/defaults/main.yml defines the groups, once for both processes. Write a domain as *.<domain> to block it and every subdomain. A plain entry blocks only that exact name. One rule maps an entry to records, for a line of a feed and for a list entry of dns-control: example.org becomes example.org CNAME ., and *.example.org becomes example.org CNAME . and *.example.org CNAME ..

The domains group blocks mask.icloud.com and mask-h2.icloud.com as plain names. When the resolver of a network answers NXDOMAIN for these two names, Apple devices turn off iCloud Private Relay on that network. While Private Relay is on, Safari sends its DNS queries through the relay of Apple, and no list of this resolver applies to them. Apple documents the method in Prepare your network or web server for iCloud Private Relay. To see which devices ask for Private Relay, search the query log of dns-control for mask.icloud.com.

The owned zones fzymgc.house, holomush.dev, fzymgc.net and hl-assets.dev are in dns_list_allowlist. The allow zone allow.rpz.internal. holds them as rpz-passthru. rules, and it comes before every entry and every feed, so no list blocks an owned name. The withheld zone comes before the allow zone, so the rules of The client view above do not change.

dns_list_excepted_names in the same file holds third-party names that no list blocks, each with every name under it. The allow zone holds them beside the owned zones. To except a name that a list blocks, add *.<name> there and converge --tags pdns-recursor one node at a time. An entry excepts only the name it says, so *.s1.sentry-cdn.com leaves the rest of sentry-cdn.com blocked.

The timer dns-feeds.timer starts dns-feeds.service once a day, at 03:30 on sandiego and at 04:30 on portland, so the two nodes never reload at the same time. The unit runs dns-node-worker feed for each group with a URL source. The command downloads the source, converts it to RPZ, makes sure that the new zone holds at least half the entries of the zone it replaces, and renames it into place. The role renders the zone of an inline group, such as domains. After the last group, the unit runs rec_control reload-yaml for each process, which reads every RPZ again, and then rec_control wipe-cache '.$', because a reload keeps the packet cache and a cached answer outlives a new block until its TTL ends. Each rec_control call waits up to 20 seconds, the value of pdns_recursor_control_timeout, because a reload of the main process reads every feed zone and can take longer than the default of 5 seconds. A failed download keeps the previous zone file in force, and the other groups still load. The unit then ends failed, and journalctl -u dns-feeds names the group.

To read the feed state on a node:

Terminal window
# On a resolver node.
systemctl list-timers dns-feeds.timer # the next and the last run
systemctl is-failed dns-feeds.service # want: inactive. failed means a group did not download
ls -l /var/lib/pdns-recursor-feeds/ # one .feed.rpz for each group, with its mtime
rec_control --config-dir=/etc/powerdns --config-name=main --socket-dir=/run/pdns-recursor-main rpzstatistics

rpzstatistics prints one block for each zone of the process with its serial and its record count. A feed zone with no records is a finding. To download the feeds now, run the action Refresh lists on Health, or the MCP tool dns_refresh_lists of dns-control. The node worker starts dns-feeds.service on each target node, the node that holds no VIP first, and waits up to 5 minutes until the unit ends. Each download stops after 60 seconds (pdns_recursor_feed_timeout), and each rec_control call after 20 seconds (pdns_recursor_control_timeout), so the unit ends inside that bound. A unit that ends failed fails the action, and the detail names the error of systemctl. The unit drops every cached answer, so dns-control reads the flush guard first and refuses the refresh while the upstream fails. Without dns-control, run sudo systemctl start dns-feeds.service on one node.

Each node has 4 GB of memory and runs both processes, and each process holds its own copy of every feed zone. Measure the memory of a new list before you add it. In the spike of the change dns-policy-as-rpz, both processes together with every feed took 415 to 434 MiB after the load and 1,103 to 1,151 MiB at the peak of a feed reload. Count about 370 bytes for each record in each process. The list of newly registered domains on the Firewalla, 6.1 million entries, was not moved for this reason.

To change what one leg blocks, edit its dns_list_groups in both host_vars files and converge --tags pdns-recursor one node at a time. An empty list turns list blocking off for that leg.

To find why a name was blocked, run this query as the ClickHouse user app:

SELECT Timestamp, ServiceName, LogAttributes['client.address'] AS client,
LogAttributes['dns.question.name'] AS name, LogAttributes['dns.policy.zone'] AS zone,
LogAttributes['dns.policy.trigger'] AS trigger
FROM default.otel_logs
WHERE Timestamp > now() - INTERVAL 1 HOUR
AND ServiceName IN ('dns-resolver', 'dns-resolver-public')
AND LogAttributes['dns.policy.action'] = 'nxdomain'
ORDER BY Timestamp DESC
LIMIT 50

dns.policy.zone names the zone that decided, for example hagezi.feed.rpz.internal., and dns.policy.trigger names the rule that matched. To let one owned name through, add it to dns_list_allowlist. To let a third-party name through, add it to dns_list_excepted_names. To let a name through on one leg only, remove the group from that leg.

The main process loads two kinds of zone from pdns-auth on 127.0.0.1:1053: the operator zone operator.rpz.internal., with every list entry of dns-control that names no scope, and one zone <scope>.scope.rpz.internal. for each scope of dns-control. The operator zone applies to every client of the main process: every trusted leg, the tailnet and a client on no declared leg. A scope zone applies to the members of the scope only. The public process loads no zone from pdns-auth, so Guest and Telework keep the public lists alone.

dns-control writes the zones through the pdns HTTP API on the writer node, and LightningStream copies them to the other node. A copied write sends no NOTIFY, so the node worker on each node sends one to its main process for the operator zone and the scope zones, and the process fetches the new serial by zone transfer. The main process keeps a copy of each zone in /var/lib/pdns-recursor-main/ and starts from it, so the entries stay in force while dns-control, k3s or pdns-auth is down. Change the entries in dns-control, not on the node. See List entries in the dns-control runbook.

The allow zone comes before the operator zone, so a block entry for an owned zone or an excepted name has no effect. The operator zone comes before every feed, so an allow entry lets through a name that a public list blocks. When an entry decides a query, dns.policy.zone names operator.rpz.internal. or the scope zone.

Until 2026-10-14 the entries reached the node as two files under /var/lib/dns-worker/lists/, which the main Blocky process read. Nothing reads the files now, and the retire play removes them. See Retiring Blocky.

To read what a node holds:

Terminal window
# On a resolver node.
rec_control --config-dir=/etc/powerdns --config-name=main --socket-dir=/run/pdns-recursor-main rpzstatistics | grep -A3 'operator.rpz.internal'
dig @127.0.0.1 -p 1053 operator.rpz.internal. AXFR | grep -c 'CNAME' # the rule count in pdns-auth

The serial of the zone in rpzstatistics and the serial of the SOA in pdns-auth are equal on a node that loaded the last change. If dns-control is down and an entry must change at once, there is no file to edit. Bring dns-control back and make the change there. Its next list change writes the whole zone set from its store.

What the pair emits onto an untrusted segment

Section titled “What the pair emits onto an untrusted segment”

:53 over both address families and both transports, ICMP echo, VRRP, RA carrying RDNSS, and DHCPv4 from Kea. Not a per-leg TSIG key, and not a line in legs-main.map.

Kea serves subnet 6 on Guest and subnet 7 on Telework, each with the pool .20 to .249. A lease on these legs follows three rules:

  • Kea sends no DNS update. ddns-send-updates is false for the subnet, so no client gets a name in the internal zone.
  • The lease carries no search domain. Option 119 and ddns-qualifying-suffix are absent.
  • The lease lasts 600 seconds, with a renew timer of 300 seconds. The trusted legs keep 43200.

A leg gets a Kea subnet and the udp/67 accept when it declares kea_subnet_id, not when it is trusted. trusted still decides DNS trust.

The gateway redirects no DNS query on these legs. Its zone policies block DNS from them to the gateway and all traffic from them to the trusted legs. So a client with a hand-set resolver gets no reply from .1, and a public answer from a public resolver such as 1.1.1.1. A client that uses the resolver from its lease is not affected. The keepalived sync group holds the trusted instances only (dns_vlan_interfaces | selectattr('trusted') | list), so VI_guest and VI_telework are not in the estate-wide group. Each of the two tracks chk_public_answer on its own. The check asks the public process at 127.0.0.3 for health.dns.invalid and requires NXDOMAIN, which the process answers from its identity zone dns.invalid. without an upstream query. A failure of the public process moves only those two VIPs.

The IPv6 half — what the pair answers, what the gateway sends, and the hazards

Section titled “The IPv6 half — what the pair answers, what the gateway sends, and the hazards”

All seven legs have their IPv6 clients pointed at the resolver pair through RDNSS, the DNS server option of a router advertisement. Two senders carry it on each leg: the RA of the pair and the RA of the gateway. No device serves DHCPv6. A v6 client asks the leg’s v6 VIP on-link.

What makes this possible:

  • Each recursor process listens on the ULA address and the ULA VIP of every leg it serves.
  • Every declared leg has a floating ULA VIP at ::5. See IPv6 ULA Addressing.

The list is dhcp_v6_server.dns_servers on the network in tf/unifi/networks.tf. The gateway runs no DHCPv6 service, and it sends the list as RDNSS in its router advertisement. Change the list through a pull request.

VLAN 802.1Q Value of dns_servers
Main 3000 fddb:f665:73f7:1::5
IoT 3020 fddb:f665:73f7:3::5
Core untagged fddb:f665:73f7:6::5
MainWireless 3001 fddb:f665:73f7:7::5
Lab 1000 fddb:f665:73f7:4::5
Guest 3100 fddb:f665:73f7:2::5
Telework 3200 fddb:f665:73f7:5::5

The ULA index and the node suffix are small integers in different hextets. Read a value from its own row, never from a neighbouring one.

Each list holds one address. A resolver node’s own address (::6 or ::7) bypasses the VIP and sits behind a health check that does not test IPv6. A public resolver sends internal queries out of the house and returns public NODATA for every .house name. A second address appears on the wire as a second RDNSS option, so the single entry is checkable.

Two IPv6-specific hazards on the floating addresses

Section titled “Two IPv6-specific hazards on the floating addresses”
  1. Every floating v6 address carries -nodad. Without it, Duplicate Address Detection can leave the VIP in the dadfailed state: keepalived reports the address as held, the interface lists it, and nothing answers on it.

    Terminal window
    ip -6 addr show | grep -i dadfailed
  2. keep_addr_on_down is 0 on every leg, so an admin ip link set <leg> down deletes the v6 VIP and not the v4 one. The IPv4 VIP comes back with the link. The IPv6 VIP is gone until keepalived reasserts it. An operator who bounces a leg has removed IPv6 service for that VLAN and left IPv4 working, which reads as a DNS problem.

A client can hold two resolvers after a change

Section titled “A client can hold two resolvers after a change”

A new RDNSS appears within the announcer’s interval. A displaced one is never withdrawn. The client ages it out on its own schedule, up to its advertised lifetime. Client behaviour differs: macOS replaces the per-interface resolver list from the newest advertisement, and NetworkManager accumulates. Read the client you have. If a displaced address is still present past the lifetime, a sender is advertising it again. Make sure that dhcp_v6_server.dns_servers of the network in tf/unifi/networks.tf holds the ULA VIP and nothing else.

Validating a resolver role off the estate — one command per role

Section titled “Validating a resolver role off the estate — one command per role”

Five roles carry a molecule scenario: dns-common, pdns, pdns-recursor, keepalived and kea. Each scenario’s converge runs that role’s own tasks/verify.yml against containers built from the trixie image the resolvers run. The scenarios link ansible/inventory/ in, so a converge reads the real group_vars and host_vars. lightningstream, dns-otel-collector and dns-tailscale have no scenario, because their verify.yml asserts nothing a converge can falsify.

Run these from a clone of this repository on your workstation, with Docker running, one scenario at a time. Platforms are named after the real resolvers, so two scenarios up at once collide on the container names. dns-common and pdns read from Vault on the control node, so export VAULT_TOKEN first. vault login -method=oidc writes ~/.vault-token but does not export it.

Terminal window
export VAULT_TOKEN="$(cat ~/.vault-token)"
mise //ansible:molecule-dns-common # want: PLAY RECAP failed=0 unreachable=0 for sandiego and portland
mise //ansible:molecule-pdns # want: the scenario stops at its recorded boundary, see below
mise //ansible:molecule-pdns-recursor # want: PLAY RECAP failed=0 unreachable=0 for sandiego
mise //ansible:molecule-keepalived # want: PLAY RECAP failed=0 unreachable=0 for sandiego and portland
mise //ansible:molecule-kea # want: PLAY RECAP failed=0 unreachable=0 for sandiego

Notes per scenario:

  • dns-common and keepalived need two platforms, because each dereferences hostvars[keepalived_peer_host]. A 403 on the ICA1 chain read is an absent or expired token, not a broken role. The dns-common scenario also proves the rate meter: portland sends twice the burst to port 53 of sandiego in under a second, and the counter of the meter rule on sandiego reads how many queries the meter dropped.
  • pdns stops at zone-set with a failed criterion, and mise exits with status 2. That red is the scenario’s boundary, not a defect: the role creates no zone, because dns-control applies every zone through the HTTP API. The boundary is recorded in the scenario’s converge.yml. Do not seed a zone and do not add a when:.
  • pdns-recursor needs no Vault session. Its prepare step installs pdns-auth with the policy zones of the scenario, a dnsmasq stub in place of dnscrypt-proxy on 127.0.0.1:5300, a local HTTP server for the list sources, and the node worker binary from a build with the Go toolchain of apps/dns-control. The converge runs the role with its guard on, and then the scenarios of the specs: a leg with no lists, an undeclared address, an internal name from Guest, rebinding, a scoped name in both orders, a restart with pdns-auth down, a failed source, the two health names, and a wipe of one name followed by a flush of .. The first feed run of the role ends failed by design, because one list group of the scenario points at an address that never answers.
  • keepalived runs with keepalived_enabled: false. At the real value every instance enters FAULT on a container without the legs, with the same roll-up.
  • kea needs no Vault session. The scenario skips the kea-certs tag. It writes a self-signed CA as the trust anchor, an HA bundle in place of the bundle of Vault Agent, and fixed test credentials at the paths tasks/certs.yml writes. It creates the user dns-worker and a reservation file for each trusted leg, so the seed reads no export. It loads the dns-common nftables ruleset and puts each leg’s address on a dummy link, so the daemon starts on every leg. The play fails if the role takes the reservation directory from dns-worker. The inventory names portland without a container, because the HA peers list reads its host_vars.

pdns holds 127.0.0.1:1053 on sandiego and on portland, and lightningstream runs on both. Every .house answer a client gets comes from PowerDNS, whichever node holds the floating addresses. There is no unbound on either node.

A read that tells you which state a node is in, without changing anything:

Terminal window
# On a resolver node. Reach it with:
# scripts/dns-adhoc.sh <node> -m ansible.builtin.shell -a '<the line below>'
# The resolver SSH key lives in Vault and the SSH agent is not used. The header of
# scripts/dns-adhoc.sh says why.
echo "$(hostname -s) vip=$(ip -br addr show | grep -c 192.168.20.5) pdns=$(systemctl is-active pdns) ls=$(systemctl is-active lightningstream)"
# want: <node> vip=<0 or 1> pdns=active ls=active
# vip= is the only field that differs between the nodes. Any other difference is a finding.

None of these is a defect. All of them look like breakage to somebody who meets them for the first time in an incident.

An unlisted RR type on a declared name returns NOERROR with no answer. This is the property that closes the type-65/HTTPS ECH leak:

Terminal window
dig @127.0.0.1 -p 1053 dns.main.fzymgc.house TYPE65
dig @127.0.0.1 -p 1053 dns.main.fzymgc.house HTTPS

Both answers carry NOERROR with an empty answer section and the apex SOA in AUTHORITY. That is NODATA, not NXDOMAIN.

An undeclared name returns NXDOMAIN with the apex SOA:

Terminal window
dig @127.0.0.1 -p 1053 nonexistent-name-for-the-gate.fzymgc.house A

The AUTHORITY SOA’s MINIMUM field reads 60. That is how long a client keeps believing this NXDOMAIN, and it is the dominant term in the convergence window below.

No .house query leaves the box, on two independent grounds. The main recursor process forwards the zone to pdns-auth with no fallback, and PowerDNS as configured cannot recurse or forward. The guard of the main process is the forward_zones entry for fzymgc.house with the forwarder 127.0.0.1:1053 in /etc/powerdns/recursor-main.yml. A forward_zones entry never falls through to forward_zones_recurse. Read it like this, and read the exit status rather than a || echo branch:

Terminal window
# On a resolver node. Reach it with:
# scripts/dns-adhoc.sh <node> -m ansible.builtin.shell -a '<the line below>'
grep -n -A1 'zone: fzymgc.house' /etc/powerdns/recursor-main.yml; echo "RC=$?"
# want: RC=0 and two lines, the zone and its forwarder 127.0.0.1:1053.
# RC=1 is an absence and a finding.

The apex itself answers:

Terminal window
dig @127.0.0.1 -p 1053 fzymgc.house SOA # NOERROR, the SOA in the answer section
dig @127.0.0.1 -p 1053 fzymgc.house A # NOERROR/NODATA. No A record is declared at the apex
dig @127.0.0.1 -p 1053 fzymgc.house NS # both nodes as NS, with their addresses in ADDITIONAL

PowerDNS runs multi-primary with no zone transfer. Each node owns a local LMDB, and LightningStream reconciles the two through S3. There is no primary to lose. What replaces it is a window during which the two nodes disagree.

Term Value What it is
lmdb_poll_interval 1s how long the writing node takes to notice its own change
the snapshot upload sub-second at this zone size pushing the snapshot to S3
storage_poll_interval 1s how long the peer takes to list the storage and see it
the download and merge sub-second at this zone size the peer pulling and merging it

Both intervals are in ansible/roles/lightningstream/defaults/main.yml. A record written on one node is answerable on the other in a few seconds. Grant metadata that dns-control writes through the writer node’s API takes longer to land on the peer, on the order of a minute. See The two latencies under The DDNS tier.

A client that asked the other node inside that window got NXDOMAIN and keeps believing it for the SOA minimum of 60 seconds, set on every zone that dns-control holds. The worst case an operator observes is about 65 seconds. Both polls are at their floor of one second, and the negative cache is the dominant term. Lowering the negative-cache TTL is a zone-wide decision, not an operational knob.

A record added to an existing zone is not delayed by the zone cache. A new zone takes up to zone-cache-refresh-interval (60 s, ansible/roles/pdns/defaults/main.yml) longer to become visible on the peer. That setting caches the zone list, not records.

Every zone is Native, because there is no transfer and no primary:

Zone Covers
168.192.in-addr.arpa. the whole of 192.168.0.0/16 outside the carved cuts. The curated PTRs
the per-leg /24 carves in dns_ddns_reverse_zones the DDNS-writable reverse space, one real /24 zone each. See The DDNS tier
7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa. the estate’s ULA fddb:f665:73f7::/48

The IPv4 zone makes the pair authoritative for address ranges that get no DNS update: Guest 192.168.79.0/24, Telework 192.168.217.0/24 and WireGuard 192.168.80.0/24. Reverse queries for those addresses return NXDOMAIN from the zone.

The 2600: GUA reverse is not attempted. The estate’s global IPv6 prefix is delegated by Verizon, and so is its ip6.arpa reverse. dns-control holds no zone for it. Do not file “IPv6 PTRs do not work for our global addresses” as a defect against this tier.

PTRs are not a side effect of the forward record. Each curated entry in dns-control carries an explicit ptr flag, so the canonical name for an address is one entry, and dns-control writes its PTR.

Record write paths, and the update boundary

Section titled “Record write paths, and the update boundary”

Three writers, and they do not overlap:

Writer What it writes How
dns-control the curated entries, the zones, the SOAs, the zone metadata with the TSIG grants, the control record sets one change workflow per change, against the pdns API on :8081
external-dns the Kubernetes service estate the dns-control webhook sidecar, one external-dns lane change per sync, TXT-registry ownership
Kea D2 DHCP-derived names in <leg>.lan.fzymgc.house and the carved reverse zones DNS UPDATE with TSIG

The update boundary is declared. Each leg’s child zone <leg>.lan.fzymgc.house and each carved reverse /24 carries a TSIG-ALLOW-DNSUPDATE grant in the zone metadata that dns-control holds. The parent lan.fzymgc.house, the covering /16 and the apex carry none, and dnsupdate-require-tsig=yes makes that absence a REFUSED. See Where the boundary is, and what enforces it under The DDNS tier.

dig TXT <managed-name> returns an ownership record that looks like junk. It is not junk. The TXT registry of external-dns runs with no --txt-prefix, the form that its old pdns provider needed, so it writes its ownership record at the same name as the record it owns. Deleting one by hand makes external-dns believe it no longer owns the record beside it.

Zone syntax is checked with pdnsutil zone check <zone>.

A name enters the zone through exactly one of three writers, decided by what the name is:

  • A device or host is one curated entry in dns-control. Sean decided on 2026-09-29 that the entry is data, not configuration (issue #2527): dns-control holds it, one workflow projects it onto pdns, onto the Kea reservation files and onto the export, and Terraform applies no curated record. Plan the entry on the Changes screen, through POST /api/changes/plan, or with the MCP tool dns_create_entry, read the diff, and apply it. The fields are key, hostname (a lowercase FQDN with a trailing dot), address, address6 (or empty), ptr, aliases, comment, and the optional mac. The AAAA of an entry is its address6, which its owner sets. The observed AAAA sweep writes nothing at the hostname or an alias of an entry. Write ptr: false unless the address is outside every zone in dns_ddns_reverse_zones (ansible/inventory/group_vars/dns_hosts.yml), because a curated PTR inside DDNS-writable space is parent-held under the carve and answers nothing. An entry that carries a mac is also a Kea reservation on the leg of its address. The workflow writes the reservation file of each trusted leg on both nodes and reloads Kea, so the device gets its address on its next request. The entry policy refuses the infrastructure names (dns.<leg>, the per-leg node names, the node apexes) and every name under lan.fzymgc.house. See Entry changes in the dns-control runbook.
  • A zone, its metadata or an infrastructure name is a zone or record set change in dns-control. The plan and the apply need the role dns-admin. Plan the change through POST /api/zones/plan, POST /api/zones/metadata/plan or POST /api/rrsets/plan, or with an MCP tool such as dns_create_rrset. Read the diff, and apply it. A record set in the control lane is not an entry, so it projects onto pdns only and writes no Kea reservation. See Zones and record sets in the dns-control runbook.
  • A Kubernetes service is a Host() matcher plus the external-dns.alpha.kubernetes.io/target annotation, and nothing under tf/. Read The two external-dns traps before adding one.
  • A DHCP client needs nothing done. Kea D2 writes <host>.<leg>.lan.fzymgc.house on the lease for the five zones in dns_ddns_forward_zones. The AAAA at that name comes from the observed AAAA sweep of dns-control. The sweep reads the client’s addresses in the UniFi client table and publishes the ones inside the ULA /64 of the leg, after two sweeps of 300 seconds. It never publishes a GUA. See Observed AAAA in the dns-control runbook.

The two apex writers are disjoint, and neither declares under lan.fzymgc.house. external-dns drops a lan. name through --exclude-domains=lan.fzymgc.house (argocd/app-configs/external-dns/values.yaml). A curated entry whose hostname sits under a delegated child is written into the apex by dns-control, is parent-held under the delegation, and answers NXDOMAIN with the child’s authority. No gate holds this rule. A name under no writer answers NXDOMAIN and is not forwarded.

The two external-dns traps, both of which fail silently

Section titled “The two external-dns traps, both of which fail silently”

external-dns runs with --policy=sync, so it deletes what it believes it no longer owns.

Trap 1: the annotation prefix is coupled to the image version. The pinned tag in argocd/app-configs/external-dns/values.yaml reads the external-dns.alpha.kubernetes.io/ spelling, and every manifest carries it. A version that changes the default prefix stops seeing the annotations, generates no endpoints, and deletes the records they produced. Its log says nothing. The symptom is names that worked yesterday returning NXDOMAIN with external-dns healthy. The annotationPrefix value in that file pins the spelling, so the image and the manifests can move on their own.

Trap 2: the traefik-proxy source builds targets from the target annotation and from nowhere else. Traefik CRDs do not expose a load balancer IP in their status, so an IngressRoute without the annotation produces no endpoint. Use the dual-stack value, which lists both Traefik VIPs with IPv4 first: external-dns.alpha.kubernetes.io/target: "192.168.250.145,fddb:f665:73f7:250::145". If the value holds only the IPv4 VIP, the name gets an A record and no AAAA record. See DNS records in the Lab runbook. The symptom is a newly added IngressRoute whose host never appears in DNS while every older one resolves.

LightningStream reconciles the two LMDBs through https://s3-direct.nas.fzymgc.house, the NAS’s own direct S3 address, with an estate-issued certificate. The credentials come from Vault at secret/fzymgc-house/infrastructure/nas/rustfs-s3.

This path is not in the serving path. PowerDNS answers every query from its own LMDB whether or not S3 is reachable. An S3 outage, a certificate expiry or a NAS reboot degrades convergence and nothing else. It is a ticket, not a page. There is no dedicated availability detector for this endpoint, because one would take a healthy node out of VRRP eligibility over a storage fault.

The certificate renews through Traefik’s DNS-01 resolver, which needs the WAN to renew and nothing to serve.

The endpoint name has no DNS record. roles/lightningstream pins it in /etc/hosts and in /etc/cloud/templates/hosts.debian.tmpl, because the nodes’ cloud-init sets manage_etc_hosts: true and re-renders /etc/hosts from that template at every boot. --tags lightningstream-verify asserts both. If replication has stopped and the journal shows lookup s3-direct.nas.fzymgc.house on 127.0.0.1:53: no such host, the pin is gone. Repair one node at a time:

Terminal window
scripts/dns-playbook.sh --limit portland --tags lightningstream-config
scripts/dns-playbook.sh --limit sandiego --tags lightningstream-config

Both nodes log Loaded remote update and startup phase completed successfully within seconds. The client symptom is a name that answers on one node and returns NXDOMAIN on the other. Nothing alerts on this.

Handing the authoritative socket to another daemon

Section titled “Handing the authoritative socket to another daemon”

These rules apply to any future handover of the authoritative tier.

Measure the name delta by querying, not by diffing configuration. For each name the tier being replaced serves, dig @127.0.0.1 -p 1053 <name> A +short on the successor. Empty output means the name is dark. A dark name is acceptable only if it is stale, and staleness is verified by asking the cluster:

Terminal window
kubectl get ingressroute,ingressroutetcp,ingress,httproute -A -o json | grep -c '<name>.fzymgc.house'

Traefik’s Host(...) rules use backticks. A pattern anchored on double quotes matches Ingress objects and misses every IngressRoute.

A dark name is an outage, because fallbackUpstream: false leaves no other tier to answer it.

Prove the rollback target before releasing the addresses. The rollback is sudo systemctl stop keepalived on the holder. Read the node you roll back to as healthy first: keepalived-check-house.sh rc=0 and both tiers active.

Two daemons cannot both hold 127.0.0.1:1053. The play that stops the incumbent runs before the play that starts the successor. ansible/dns-playbook.yml’s play order sequences that, so a single tagged run is correct and two runs are not. Run a handover on the node that holds no floating address.

LightningStream must push from a populated store before a peer pulls. A node whose pdns starts on an LMDB pulled from an empty store answers nothing for .house and takes chk_house_answer down. lightningstream_group: pdns names a group the pdns role creates, so converging LightningStream first fails with chgrp failed. The play order sequences both.

dns-control holds every zone, grant, record set and entry, and its export keeps a copy on the NAS and on B2. If both nodes lose their LMDB, converge the authoritative tier so that pdns runs on an empty store. Then rebuild the zones from the export:

Terminal window
kubectl -n dns-control exec deploy/dns-control -- dns-control restore --from b2 --dry-run
kubectl -n dns-control exec deploy/dns-control -- dns-control restore --from b2

The dry run reads create or change for each zone, record set and entry that pdns does not serve. The restore creates each zone with its SOA, its NS and its grant, and then writes its control record sets and its entries. external-dns writes its own lane on its next sync, and Kea D2 writes its names again on the next lease events. If B2 is not reachable, use --from nas. See Restore from the export in the dns-control runbook.

Both resolvers hang off switch-main. That is a single failure domain, and it is accepted: a failure of that switch takes the whole estate down and leaves a surviving resolver nothing to serve, and the switch does not need DNS to boot. Nothing in this repository re-checks the cabling. Read it live with one LLDP frame per node on eth0.

A rule that required the two nodes to sit on different PoE switches does not apply to this topology. Neither node is PoE-powered alone, and the reasoning above stands. Do not re-assert it.

Moving portland to a second switch is possible with one precondition: the new port needs the same native-VLAN-to-Core treatment as sandiego’s port, or that node’s Core leg drops. That is half the estate’s DNS. Do not do the move without the port work.

The surviving half of the old requirement is live: never change both resolver nodes in the same window.

Promotion — moving the floating addresses deliberately

Section titled “Promotion — moving the floating addresses deliberately”

Both instances start in the backup state with nopreempt (roles/keepalived/templates/keepalived.conf.j2). A node that is up and returns from a tracker FAULT does not take the addresses back. Promotion is an explicit act.

nopreempt does not survive a reboot of the higher-priority node. It suppresses the BACKUP to MASTER transition only for an instance that hears an existing master. On a reboot of sandiego (priority 150) its master-down timer expires before its VLAN sub-interfaces pass VRRP, so it reads the group as unowned and takes it. After a reboot of the priority-150 node, expect the group back on it. Read ownership. Do not assume the peer kept it.

Which node holds the addresses is a reading:

Terminal window
scripts/dns-vip-holder.py # prints the ACTIVE node
scripts/dns-vip-holder.py --passive # prints its peer
ssh 192.168.40.6 'ip -br addr | grep -c "192\.168\..*\.5/"'
ssh 192.168.40.7 'ip -br addr | grep -c "192\.168\..*\.5/"'
# want: one node reports 7 and the other 0. 5 and 2 follows a failure of the public recursor process, because the
# Guest and Telework VIPs are outside the sync group and move alone. A total above 7 is a split brain.

Verify on the node that is not holding the addresses, promote deliberately, then work on the other one:

Terminal window
# STEP 1: verify on the non-holder, where a mistake costs nothing
ssh <non-holder> 'systemctl is-active keepalived pdns-recursor@main pdns-recursor@public pdns'
ssh <non-holder> 'sudo /usr/local/sbin/keepalived-check-house.sh; echo $?' # expect 0
ssh <non-holder> 'sudo /usr/local/sbin/keepalived-check-public.sh; echo $?' # expect 0
ssh <non-holder> 'dig +short @127.0.0.1 dns-health.fzymgc.house A' # expect 192.168.x.x
# STEP 2: move the group
ssh <holder> 'sudo systemctl stop keepalived'
ssh 192.168.40.6 'ip -br addr'; ssh 192.168.40.7 'ip -br addr' # the peer now holds all seven
# STEP 3: work on the drained node, then return it as BACKUP
ssh <old-holder> 'sudo systemctl start keepalived'
# nopreempt: it returns as BACKUP and does not take the group back

systemctl stop keepalived is the role’s stated position. roles/keepalived/tasks/config.yml carries no stop branch, so dropping seven live floating addresses is an operator act.

dns-control drains a node through the node worker on the Pi. A drain is the move of STEP 2 above, with a check of the peer first and an audit row that names the person. Start a drain or an undrain from the card of the node on the Health screen of dns-control. The actions, the five peer checks and the roles are in the dns-control runbook, Operations actions.

A drain writes the drain marker /var/lib/dns-worker/drained and then stops keepalived. If the stop fails, the drain removes the marker again. If the drain activity ends during the stop, the stop can still complete, so the marker stays. An undrain clears that state. The marker is a JSON file:

Field Value
actor The person who drained the node
since The time of the drain
expected_minutes The expected duration of the drain
audit_id The id of the audit row of the drain

The keepalived role renders the systemd drop-in /etc/systemd/system/keepalived.service.d/10-drain-marker.conf with ConditionPathExists=!/var/lib/dns-worker/drained. While the marker exists, systemd does not start keepalived, at boot too. A start through systemd then exits 0 and leaves the unit inactive. The keepalived role reads the marker at each converge and reports it. It does not start keepalived while the marker exists, so a drained node stays enabled and inactive.

The undrain activity of the node worker removes the marker. If the node worker is lost, remove the marker by hand with The manual undrain when the node worker is lost. An undrain removes the marker, starts keepalived and makes sure that keepalived is active. The node returns as BACKUP, because of nopreempt. A marker that does not parse still counts as drained, and an undrain removes it. With no marker and keepalived not active, an undrain starts keepalived. An undrain therefore also finishes a drain that stopped half way.

Reload keepalived after the notify script lands

Section titled “Reload keepalived after the notify script lands”

The node worker reads the state of each keepalived instance from a file in /run/keepalived-state. The notify script /usr/local/sbin/keepalived-notify-state.sh writes that file at each state change. The keepalived role renders the script and the notify line in keepalived.conf, but the role never reloads keepalived. Until keepalived reads the new configuration, the state files do not exist. The worker then reads the keepalived state of the node as UNKNOWN, and the drain check keepalived_not_fault fails for that node.

After a converge that installs the notify script, do these steps on one node per window. Skip a drained node, because its keepalived is stopped.

  1. Reload keepalived:

    Terminal window
    scripts/dns-adhoc.sh <node> -b -m ansible.builtin.command -a 'systemctl reload keepalived.service'
  2. Make sure that /run/keepalived-state holds one VI_* file for each instance. The two numbers must be equal:

    Terminal window
    scripts/dns-adhoc.sh <node> -b -m ansible.builtin.shell -a 'ls /run/keepalived-state | grep -c "^VI_"; grep -c "^vrrp_instance " /etc/keepalived/keepalived.conf'
  3. If a file is missing, do not restart keepalived on the node that holds the VIPs. Move the VIPs first with the steps in the promotion section above.

The manual undrain when the node worker is lost

Section titled “The manual undrain when the node worker is lost”

If the node worker cannot run the undrain, undrain the node by hand. For example, the worker, its certificate or the Temporal gateway is down. Work on one node per window. Remove the marker first, then start keepalived, because the start does nothing while the marker exists. Replace <node> with sandiego or portland. Never use dns_hosts here.

  1. Make sure that the peer holds all seven VIPs. Expect 7 on the peer and 0 on the drained node:

    Terminal window
    ssh 192.168.40.6 'ip -br addr | grep -c "192\.168\..*\.5/"' # sandiego
    ssh 192.168.40.7 'ip -br addr | grep -c "192\.168\..*\.5/"' # portland
  2. Remove the marker as root:

    Terminal window
    scripts/dns-adhoc.sh <node> -b -m ansible.builtin.file -a 'path=/var/lib/dns-worker/drained state=absent'
  3. Start keepalived, and make sure that it is active:

    Terminal window
    scripts/dns-adhoc.sh <node> -b -m ansible.builtin.command -a 'systemctl start keepalived.service'
    scripts/dns-adhoc.sh <node> -m ansible.builtin.command -a 'systemctl is-active keepalived.service'
  4. Expect active. The node returns as BACKUP and does not take the VIPs back.

A manual undrain writes no audit row. Health shows the node as not drained at its next status reading, when the node worker answers again.

The Ansible role dns-node-worker installs the node worker and Vault Agent on a resolver node. Run it on one node per window:

Terminal window
scripts/dns-playbook.sh --limit <node> --tags dns-node-worker

The first task reads the AppRole role id of the node from Vault on the control node. If you run the role before the tf/vault apply, the AppRole does not exist yet. The run then stops at that first task with the error of vault read, and the node does not change.

The role writes an AppRole secret id to /etc/dns-node-worker/vault/secret_id only when the node has none. To rotate the secret id, remove that file on the node and run the role. Before it creates the new secret id, the role destroys every earlier secret id of the AppRole of that node. No earlier secret id then keeps working. A run where the file exists destroys nothing. Your Vault token on the control node must be allowed to list and destroy the secret id accessors of the AppRole.

The polkit rule of the role grants the worker two actions only: start and stop of keepalived.service. The worker cannot restart a recursor process or any other unit. It talks to the two recursor processes through their control sockets, which the pdns-recursor role gives the group dns-worker, and through their HTTP API with the key copy /etc/dns-node-worker/recursor-api.key.

The tag dns-node-worker-verify runs the read-only checks of the role alone. They make sure that the installed worker is the pinned release, and that both units are enabled exactly when dns_node_worker_service_enabled is true. When that value is true, the checks also make sure that:

  • dns-node-worker.service and dns-node-worker-pki.service are running.
  • The client certificate in the bundle has not expired. The worker user runs openssl x509 -noout -checkend 0 on the bundle.
  • The worker user can read its configuration, the API key copy and the CA file, and can run the two check scripts.
  • The worker user reaches the control socket of each recursor process, and the recursor user pdns reads the scope map, which is 0644 and owned by the worker.
  • The worker user can read the certificate bundle and cannot read the secret id.

A tracker-induced failover moves all five addresses together in about ten seconds: detection is check_interval 2 × fall 5, plus up to 3 × advert_int 1, plus the priority skew. The values are in roles/keepalived/defaults/main.yml. The group can be briefly unowned in transit.

With both nodes faulted the floating addresses go unowned, not duplicated. keepalived and pdns-recursor@main report active on both nodes, and both journals carry Entering FAULT STATE. The shape to recognise is nothing holds the addresses and every unit looks healthy.

keepalived sends a gratuitous ARP on entering the master state. That is what moves a client’s cached mapping for a floating address to the new node. If a class of client holds a stale mapping after a move: make sure that journalctl -u keepalived on the new master shows Entering MASTER STATE, check whether that client’s segment drops unsolicited ARP, and for unmanaged devices with long cache lifetimes accept a bounded window.

Breaking one node’s external DNS path on purpose

Section titled “Breaking one node’s external DNS path on purpose”

scripts/dns-break-external.sh induces real breakage on one node. Every public name of a node goes through dnscrypt-proxy on its loopback, and nothing else on the node speaks to the relays, so the script stops that unit. .house, the reverse zones, VRRP, SSH and tailscale are untouched, and a public name answers SERVFAIL in about 2 milliseconds:

Terminal window
scripts/dns-break-external.sh sandiego apply # systemctl stop dnscrypt-proxy
scripts/dns-break-external.sh sandiego status # unit state and the live-server line of its last start
scripts/dns-break-external.sh sandiego revert # start, and wait for a live route

The recursor serves a cached answer without the daemon, so the break does not make every lookup fail. Nothing consumes the result automatically. An operator reads the outcome. Before 2026-10-06 the script dropped TCP port 853 in an nftables table, the DoT egress to Control D.

Things that will bite you — read these before touching the pair

Section titled “Things that will bite you — read these before touching the pair”

A configuration change of the recursor is a rec_control reload or a restart of one process

Section titled “A configuration change of the recursor is a rec_control reload or a restart of one process”

A change to the RPZ list of a process, the include directory of the main process included, goes through rec_control reload-yaml. It reads every RPZ again and keeps the caches. A change to the scope map goes through rec_control reload-lua-script. Every other change to /etc/powerdns/recursor-<p>.yml, such as a listener or a forwarder, needs a restart of that one process:

Terminal window
# On a resolver node. <p> is main or public.
rec_control --config-dir=/etc/powerdns --config-name=<p> --socket-dir=/run/pdns-recursor-<p> --timeout=20 reload-yaml
sudo systemctl restart pdns-recursor@<p>

A reload of the main process reads every feed zone and can take longer than 5 seconds, the default wait of rec_control. Without --timeout, rec_control then prints Timeout waiting for control channel data and exits 1.

systemctl restart pdns-recursor@main interrupts the answers on the trusted legs for about two seconds. The keepalived check runs every 2 seconds and needs fall 5 failures to move the VIPs, so a restart does not move them. Restart one process at a time: each process binds its own addresses, so the other one keeps answering. Do not restart both processes of one node in one command, and never restart a process on both nodes in the same window.

Until 2026-10-14 the caution here read: never systemctl reload blocky. Blocky exited on SIGHUP and left the client tier down with a clean exit. The recursor has no such hazard. systemd refuses reload for the unit template, because it carries no ExecReload=.

networkctl reload strips foreign addresses. Never run it on the node holding the VIPs

Section titled “networkctl reload strips foreign addresses. Never run it on the node holding the VIPs”

keepalived’s floating addresses are foreign to systemd-networkd. networkctl reload removes foreign addresses from a link whose .network changed. On the VIP holder it pulls the live floating addresses while keepalived goes on reporting MASTER.

To add or change a leg:

  1. Converge the node that holds no floating address.
  2. networkctl reload that node.
  3. Move the group (systemctl stop keepalived on the holder).
  4. Repeat on the newly freed node.

A converge reports success and the machine does not change

Section titled “A converge reports success and the machine does not change”

All three bite whoever next edits dns_vlan_interfaces:

What The reading Consequence
keepalived has no path from a changed config to the running daemon after --tags keepalived reports changed=1, the new instances are on disk, the process uptime is unchanged, and the addresses held are unchanged the rendered config is adopted at the next restart or reboot, a VRRP topology change at an unplanned moment
Nothing reloads systemd-networkd, so a new leg never comes up after --tags dns-common reports changed=N, both files are rendered, the new eth0.<tag> is absent, and the nftables accepts that name it have reached the kernel the node admits DNS on an interface it does not have
Tag filtering silences notified handlers a converge reports failed=0 with no RUNNING HANDLER section, having deleted a file and written nothing in its place the machine is in a state neither configuration describes

After any change to dns_vlan_interfaces, read the machine, not the recap: ip -br addr for the leg, systemctl show -p ExecMainStartTimestamp keepalived against the config’s mtime, and the addresses held.

For the handler case: --skip-tags always drops notified handlers, --force-handlers does not rescue them, and --list-tasks cannot see them. Skip the two roles by name instead of filtering on always. The check that discriminates is the presence of the RUNNING HANDLER section in the output.

Anything you arm with systemd-run must pin AccuracySec

Section titled “Anything you arm with systemd-run must pin AccuracySec”

systemd-run --on-active defaults to AccuracySec=1min, so a --on-active=5 timer can fire about sixty seconds late. Pin --timer-property=AccuracySec=1s on every one.

Verifying a DHCP advertisement: select the packet by its own transaction id

Section titled “Verifying a DHCP advertisement: select the packet by its own transaction id”

The DHCP OFFER is what proves what a network advertises. A bare grep Domain-Name-Server over a shared capture attributes other clients’ packets to your probe, and matches a client’s Parameter-Request list as well as an OFFER. So:

  • Select the packet by the probe’s own transaction id, or its synthetic 02:00:00:* MAC.
  • Assert the message type is Offer, not Request.
  • Assert the lease beside option 6.
  • Never 2>/dev/null the probe injection. A probe that never went out must be a red.

Trigger the OFFER with a crafted DISCOVER. Tear a capture down by PID. pkill -f "tcpdump -i ..." matches the invoking ssh shell’s own command line and kills the session you work through.

tcpdump is declared for these nodes through dns_common_diagnostic_packages in ansible/inventory/group_vars/dns_hosts.yml. The role default is [].

The node’s own fallback resolver is the same daemon

Section titled “The node’s own fallback resolver is the same daemon”

/etc/resolv.conf on each node names 127.0.0.1 (the main recursor process) first and 127.0.0.2 second. The second address is a listener of the one dnscrypt-proxy of roles/dnscrypt-proxy, so a node whose client tier is down still resolves public names through the relays, including the tailnet name of the host an operator uses to fix it. Internal names come from pdns through the main process and have no fallback. The daemon refuses fzymgc.house and both reverse zones, so a node with the main process down gets REFUSED for an internal name and sends nothing to a relay.

Until 2026-10-06 that second listener was a separate packaged dnscrypt-proxy 2.1.8 with DoH to two static stamps, installed by roles/dns-common. One daemon replaced it, because the two shared a unit name and a configuration path. The cost is the double outage: a node with the main process down and every relay down resolves no public name. systemd-resolved with a DNSOverTLS=yes drop-in was never the home for this, because that unit is neither started nor masked by decision.

The glibc stub resolver advances to the second nameserver on a transport-level failure only. A main process that is up and answers NXDOMAIN for everything is, to the stub resolver, a working nameserver. The self-first /etc/resolv.conf recovers a node from a dead client tier, not from a misbehaving one.

An exception is a decision with a rationale. What follows is exposure with no detector and no owner.

The DHCPv4 server of the gateway is off on every leg (dhcp_server.enabled = false in tf/unifi/networks.tf). If a firmware update or an edit in the UniFi UI turns one back on, a second DHCPv4 server is on a live broadcast domain. Two DHCPv4 servers on one segment fight, and the client takes the first OFFER. No timer probes for a second server, and no alert fires on a foreign OFFER.

The discriminator is lease origin, never the address. Kea’s own lease4-get-all lists a Kea lease on both nodes. A lease from the gateway never appears there. See Reading Kea lease state from a workstation.

UniFi DHCP Guard on Main, Guest and Telework trusts only the leg’s .6, .7 and .5. Nobody has measured whether it drops an offer from the gateway itself.

Every name the pair serves has a declared source. That is narrower than “every name that resolves”, because search-suffix shadows are outside it. The DDNS children are outside the closure by construction: external-dns’s --exclude-domains=lan.fzymgc.house and D2’s per-leg grants put them there.

The closure is enforced in the chain of the main recursor process and PowerDNS. PowerDNS is authoritative for fzymgc.house, so an unlisted name is NXDOMAIN and an unlisted type is NODATA. The forward_zones entry of the main process (ansible/roles/pdns-recursor/templates/recursor-main.yml.j2) sends every .house query to pdns, and a forward_zones entry never retries an answer upstream.

There is no inventory variable that opens the closure. A change to the forward zone is an edit of the template and a converge of --tags pdns-recursor one node at a time. Such a change reopens the type-65/HTTPS leak.

Why does this name resolve when there is no Terraform entry for it?

Section titled “Why does this name resolve when there is no Terraform entry for it?”

A name answers, and nothing under tf/ declares it. The wrong conclusion is that DNS is lying. fzymgc.house has three writers, and none of them is declared under tf/:

# Writer Mechanism Declared in git?
1 dns-control writes each curated entry, each zone and each control record set through the pdns API on an apply, with the mark writer=dns-control No. The rows live in the dns-control database and in the export on the NAS and B2
2 The DDNS tier Kea’s D2 writes <host>.<leg>.lan.fzymgc.house from each lease No. See The DDNS tier below
3 external-dns writes a name for each Kubernetes Host() matcher that carries the external-dns.alpha.kubernetes.io/target annotation Yes, in the manifests under argocd/

Run this from a real client on the LAN.

Terminal window
for n in id llm grafana vault k8s-cluster; do echo -n "$n: "; dig $n.fzymgc.house A +short; done
dig id.fzymgc.house TYPE65 +short # want: empty (NODATA)
curl -sS -o/dev/null -w"%{http_code} %{remote_ip}\n" https://llm.fzymgc.house/v1/models # want: 401 via 192.168.250.145

This block does not tell you which node answered. Add the discriminator, and the address-family check:

Terminal window
dig whoami.dns.invalid +short # want: 192.168.40.6 or .7, a resolver node.
# EMPTY means something other than the pair answered, for
# example a hand-configured public nameserver.
# Which server is this client using? On macOS:
scutil --dns | grep nameserver | sort -u
# A fddb: entry is the pair's v6 VIP for that VLAN. On Linux: resolvectl status.

Two probes that manufacture a false incident

Section titled “Two probes that manufacture a false incident”

An NXDOMAIN for a name the zone does not curate is correct. Verify with a known curated name:

Terminal window
dig +short nas.fzymgc.house # 192.168.20.200. This is the liveness probe.

A workstation with tailnet-claimed routes over estate subnets is not a valid vantage point. If the machine you test from runs Tailscale accepting routes that overlap the estate, queries aimed at Core addresses transit the tunnel. What comes back is a non-authoritative answer no estate server sent, which reads like a resolver serving an empty zone.

Terminal window
# Before trusting any negative result, check that the query leaves on the LAN:
netstat -rn -f inet | grep -E '192\.168\.(20|40)' # macOS; want the LAN interface, not utun*

If a tunnel interface owns the route, move to a client that is on the VLAN before concluding anything.

Held in ansible/inventory/host_vars/{sandiego,portland}.yml. Seven legs per node, one address per leg. The .5 column is a floating address held by whichever node won the election.

Leg VLAN id Network Gateway VIP sandiego portland
core untagged native 192.168.40.0/22 .1 .5 .6 .7
lab 1000 192.168.200.0/22 .1 .5 .6 .7
main 3000 192.168.20.0/22 .1 .5 .6 .7
iot 3020 192.168.218.0/24 .1 .5 .6 .7
mainwireless 3001 192.168.24.0/22 .1 .5 .6 .7
guest 3100 192.168.79.0/24 .1 .5 .6 .7
telework 3200 192.168.217.0/24 .1 .5 .6 .7

Each leg also carries a ULA /64 (fddb:f665:73f7:N::/64) with the same ::5, ::6 and ::7 convention. mainwireless carries trusted wireless clients under the same trust policy as Main. Guest and Telework declare trusted: false, and Kea serves DHCPv4 on both.

The authoritative loopback listener is 127.0.0.1:1053, held by pdns. Loopback :53 is not available on these nodes: the main recursor process binds 127.0.0.1:53 for the health client, which is also why the systemd-resolved stub listener is taken down.

Role Owns
dns-common The node baseline: the tagged VLAN legs and their systemd-networkd units, the ip_forward=0 drop-in and its assert, the systemd-resolved stub-listener drop-in that frees :53, the nftables ruleset, the RA announcer and the RA-watch detector, the timed auto-revert, and the uv-built Python interpreter at /opt/dns-resolver/venv
pdns The authoritative tier for fzymgc.house and its children: pdns-auth on 127.0.0.1:1053, its LMDB backend, the DNS UPDATE boundary and the per-leg TSIG grants
lightningstream Multi-primary replication of that LMDB between the two nodes through S3
dnscrypt-proxy The anonymizing upstream of the client-facing tier: the pinned dnscrypt-proxy binary, proved by its minisign signature, its loopback listener on 127.0.0.1:5300, the route set that reaches every public resolver through a relay of another operator, and the signed resolver lists in its cache
pdns-recursor The client-facing tier: PowerDNS Recursor 5.4 from repo.powerdns.com as two processes, pdns-recursor@main for the trusted legs, the tailnet and the loopback, and pdns-recursor@public for Guest and Telework. The withheld, allow and rebinding zones from files, the operator zone and the scope zones from pdns-auth on loopback, one feed zone for each list group with dns-feeds.timer, the Lua script policy.lua with the leg map of each process, the API key, and DNS-collector as the protobuf reader dns-resolver-reader
dns-otel-collector Query-log and journal transport into ClickStack, including the unit list the RA-watch alert depends on, the parse of the reader rows under the ServiceName of each recursor process, dns-resolver and dns-resolver-public, the metrics scrape of both processes and LightningStream, the OTLP receiver of the node worker on 127.0.0.1:4318, and tlscheck.time_left of the three certificate bundles
vault-cert-agent Vault Agent as the user vault-cert-agent with the AppRole cert-agent-<node>. It renders and renews the bundle of the collector and the HA bundle of Kea
kea The DHCPv4 serving tier: kea-dhcp4 and its HA pair, the per-leg pools and option surface, kea-dhcp-ddns (D2), and the timer that restarts Kea after a renewal of its HA certificate
keepalived Per-VLAN floating-address failover: the VRRP sync group and the .house tracker, and the chk_public_answer tracker on each untrusted leg’s own instance
dns-tailscale The tailnet client on each resolver node, which makes remote .house resolution reach each node’s own 100.x address
dns-node-worker The dns-control node worker with the engine recursor: the locked-down user dns-worker, the polkit rule that allows it only the start and stop of keepalived, the Vault Agent that renews its client certificate, and the seed of the scope map that the main recursor process reads

The table follows ansible/dns-playbook.yml. The dynamic namespace is lan.fzymgc.house, populated by Kea’s D2. See The DDNS tier at the end of this page.

/opt/dns-resolver/venv stays because dns-tailscale runs from it. Its verify tasks run the venv’s Python interpreter to parse JSON. Remove a package from that venv with uv pip uninstall, never uv pip sync. A sync makes the venv equal the lockfile and removes every package that the lockfile does not name.

The role vault-cert-agent runs Vault Agent on each node with the AppRole cert-agent-<node>. The agent renders two bundles. Each bundle is the certificate, the CA chain and the key in one file, at 0640, owned by the user vault-cert-agent and the group in the table:

Bundle Vault PKI role Group Reader
/etc/otelcol-dns/certs/tls.pem otel-collector-dns-client on ICA1 otelcol-dns the collector, which reads it again every hour
/etc/kea-ha-tls/kea-ha.pem kea-ha on fzymgc-house/v1/kea-ha/v1 kea-ha-tls kea-dhcp4, which reads it only at start

Each certificate lives 30 days, and the agent renews it at about day 15. The node worker has its own Vault Agent and bundle, from the role dns-node-worker. The user vault-cert-agent is a member of otelcol-dns and kea-ha-tls only. The Kea credentials and the D2 TSIG secrets have the group _kea, so the agent cannot read them. kea-ha-tls has the member _kea, so Kea reads the HA bundle.

The Kea HA certificate carries the HA address of the node as its IP SAN. The sub-CA of its mount permits only the HA addresses and the HA names of the two nodes (docs/operations/certificates.md, The Kea HA issuer).

kea-ha-cert-reload.timer runs every 15 minutes on each node. Its service restarts isc-kea-dhcp4-server.service only when all of these conditions hold:

  1. The bundle is newer than the start of Kea.
  2. The node is the Kea standby, or it is the primary and the bundle is at least two hours old.
  3. Both nodes report hot-standby in status-get, the partner is in-touch, and communication-interrupted is false.

The two-hour wait puts the standby first when both certificates renew together, so the primary serves DHCP during the first restart. Kea keeps in-touch true after it loses the partner, so the service reads communication-interrupted as well. If a condition does not hold, the service exits 0 and prints the reason. After a restart, it waits up to five minutes for both nodes to report hot-standby. If they do not, the unit fails, and its journal row reaches ClickStack.

To read the next run of the timer and the last decisions on both nodes:

Terminal window
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
-a 'systemctl list-timers kea-ha-cert-reload.timer --no-pager; journalctl -u kea-ha-cert-reload.service -n 5 --no-pager </dev/null'

Do one node per window, the Kea standby first. Read the role of each node with status-get (Confirming the pair is peered).

  1. Run scripts/dns-playbook.sh --limit <node> --tags kea-install. It creates the group kea-ha-tls and adds _kea to it. The agent play needs that group.
  2. Run scripts/dns-playbook.sh --limit <node> --tags dns-otel-collector,vault-cert-agent,kea. The plays run in the order of ansible/dns-playbook.yml. The collector restarts on the new path at the end of its play and fails until the agent play writes its bundle. systemd starts it again every 10 seconds. Kea restarts on its bundle at the end of the Kea play.
  3. Make sure that status-get reports hot-standby on both nodes before you start the other node.

After the first converge, a run of the whole playbook in its order works on the node.

The nftables ruleset is policy drop. It permits udp and tcp destination port 53 on every declared leg, udp destination port 67 on every leg that declares kea_subnet_id (see The DHCPv4 accept), established and related, iif lo, VRRP, the tailnet accepts, and tcp destination port 22 on every leg the inventory marks trusted: true, in both address families. The rule is part of the ruleset with no flag of its own, so dns_common_nftables_enabled is the single switch. The role installs the include into /etc/nftables.conf below Debian’s flush ruleset, because an include above it is read and then wiped. Guest and Telework get no SSH path. Source-address matching is not an authentication control. SSH’s own authentication protects the host.

The IPv4 source set is dns_common_ssh_source_set_v4, because 192.168.16.0/20 covers both the Main and the MainWireless leg and nft refuses overlapping intervals in one anonymous set. The IPv6 set derives each trusted leg’s own ULA /64.

Leg v4 source entry v6 source entry
core 192.168.40.0/22 fddb:f665:73f7:6::/64
main, mainwireless 192.168.16.0/20 fddb:f665:73f7:1::/64, fddb:f665:73f7:7::/64
lab 192.168.200.0/22 fddb:f665:73f7:4::/64
iot 192.168.218.0/24 fddb:f665:73f7:3::/64

Reach a node at any trusted leg address. sandiego.core.fzymgc.house resolves to the Core address alone. ansible/inventory/hosts.yml gives each host its own Core address as ansible_host, never the .5 VIP.

The ruleset limits the queries of one client address to 100 a second, with a burst of 200, in two meter rules above the conntrack accept: dns_rate_v4 keyed on the whole IPv4 source address and dns_rate_v6 keyed on the whole IPv6 source address, so a /32 and a /128 each get their own bucket. Loopback is outside the meter, so the keepalived checks never hit it. The values are dns_common_dns_rate_limit and dns_common_dns_rate_burst in ansible/roles/dns-common/defaults/main.yml. Until 2026-10-14 the limit was the rateLimit of Blocky with the same values. The recursor has no limit per client, so the meter took its place.

The meter sits above the conntrack accept on purpose. A client that reuses one socket makes its flow established after the first answer, and a meter below that accept would never see its second query. The counter of each meter rule reads how many queries it dropped:

Terminal window
# On a resolver node.
sudo nft list chain inet dns_resolver input | grep dns_rate
# want: two rules. The `counter packets N` of each one is the number of dropped queries.

A counter that moves names a client above the limit. The nftables ruleset logs no address, so read the query rows of that minute in ClickStack to find the client.

Ping, neighbour discovery, and what ICMP the resolver answers

Section titled “Ping, neighbour discovery, and what ICMP the resolver answers”

echo-request is accepted on every declared leg, all seven, in both address families, with no source scope. The invariant is ping surface equals DNS surface, held by the two echo rules sharing the port-53 accepts’ loop expression in ansible/roles/dns-common/templates/nftables-dns.conf.j2. Echo is not accepted on the tailnet leg. SSH does not share this scope: echo on seven legs, SSH on the five trusted ones.

IPv6 Neighbour Discovery, MLD (both versions) and the ICMP error types are accepted above the invalid-state drop. NDP is not valid-new for nf_conntrack_icmpv6, so an accept placed below ct state invalid drop is dead code. Without these rules the node becomes IPv6-unreachable one RA lifetime after the ruleset lands. packet-too-big carries IPv6 PMTU discovery. The DHCPv6 client port (udp 546) is accepted from link-local sources only.

The dynamic namespace beneath lan.fzymgc.house carries A and PTR records that Kea’s D2 writes from leases. It carries no AAAA records. Nothing publishes the IPv6 address of a DHCP client. See The DDNS tier.

The upstream of the recursor is the daemon on loopback

Section titled “The upstream of the recursor is the daemon on loopback”

forward_zones_recurse of both processes carries one entry, the zone . with the forwarder 127.0.0.1:5300, which is dnscrypt-proxy on the same node. There is no second forwarder and no direct resolver, and dnssec.validation is off, so the recursor never asks a root server. The role reads the forwarder from dns_anonymizing_upstream_listen in ansible/inventory/group_vars/dns_hosts.yml, because a direct resolver would see the home address the first time the relays fail.

When the daemon has no live route, a public name answers SERVFAIL after about 6 seconds. When the daemon is stopped, it answers SERVFAIL in about 2 milliseconds. Both are the intended failure mode: availability before anonymity, never the other way round.

These SERVFAIL answers apply only to a name that is not in the cache. A name in the cache gets a stale answer, an expired record that the recursor serves when a fetch fails. Both processes set recordcache.serve_stale_extensions to 120, so a record stays in service for up to one hour, in 30-second steps. The recursor tries to fetch the record again at each step. Both processes also set recordcache.refresh_on_ttl_perc to 10. A record with 10 percent or less of its TTL left is fetched again in the background, so a client does not wait for that fetch. The two values are in ansible/roles/pdns-recursor/defaults/main.yml.

On 2026-10-19, four of the five resolvers dropped about one query in five for codeload.github.com, a name with a 60-second TTL. Before the two settings, each drop reached the client as SERVFAIL after the 1.5-second network timeout of the recursor.

Until 2026-10-06 the upstream was Control D over DoT, in two entries for the two address families. The ADR docs/adr/2026-10-06-pair-resolves-public-names-through-anonymized-relays.md records the change.

Ask the node itself, never the control node:

Terminal window
uv run scripts/dns-external-resolution.py \
--ssh fzymgc@192.168.40.6 --resolver 127.0.0.1 --decoy 192.168.40.99

The --decoy probe is mandatory. It aborts the run as GATE INVALID (exit 4) if anything answers for an address where nothing exists.

Each node runs one dnscrypt-proxy as the system user dnscrypt-proxy, on two loopback listeners, UDP and TCP: 127.0.0.1:5300 for both recursor processes, and 127.0.0.2:53 as the node’s own fallback resolver. The client tier forwards every public name to it since 2026-10-06, through the OpenSpec change pair-drops-control-d. The two addresses are the inventory values dns_anonymizing_upstream_listen and dns_node_fallback_listen, which the roles that use them read too.

The daemon speaks Anonymized DNSCrypt. It encrypts the query for the resolver and sends it to a relay. The relay sees the home address and no query. The resolver sees the query and the address of the relay. The route set is in ansible/roles/dnscrypt-proxy/defaults/main.yml:

Resolver Operator Relays Relay operator
cs-dc, cs-nyc CryptoStorm dnscry.pt-anon-ashburn-ipv4, dnscry.pt-anon-newyork-ipv4 dnscry.pt
dnscry.pt-ashburn-ipv4, dnscry.pt-newyork-ipv4 dnscry.pt anon-cs-dc, anon-cs-nyc CryptoStorm
plan9dns-nj plan9 anon-cs-dc, dnscry.pt-anon-ashburn-ipv4 both

Every resolver has a route, and the render fails for a resolver without one or for an empty server list. skip_incompatible is on, so a resolver that cannot be reached through its relays is dropped rather than reached directly, and direct_cert_fallback is off, so the certificate of a resolver is fetched through the relay too. When no route works, the daemon answers nothing, and the recursor answers SERVFAIL.

The daemon refuses the internal zones. /etc/dnscrypt-proxy/blocked-names.txt holds fzymgc.house, 168.192.in-addr.arpa and 7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa, rendered from dns_zone_domain and dns_reverse_zones. A bare zone matches the zone and every name under it. The daemon answers REFUSED before any upstream step, so no relay sees the name. The main recursor process never asks for these names, because it forwards them to pdns. The public process never asks either, because its withheld zone answers NXDOMAIN first. The node’s own fallback path does ask, and REFUSED is what it gets. The decision is dnscrypt-proxy refuses the internal zones.

The set holds no ODoH server. dnscrypt-proxy fetches the ODoH configuration of a target from the target itself, not through the relay, at each start and every 4 hours, so the target learns the home address. The route odoh-cloudflare through odohrelay-crypto-sx works from this network and stays out for that reason.

A security review of dnscrypt-proxy 2.1.18 on 2026-10-06 gave the verdict use with mitigations. The role applies each one: explicit server names, no DoH servers, skip_incompatible, no direct certificate fallback, require_nolog, classic DNSCrypt with a fresh client key for each query instead of the post-quantum mode (which reuses one ciphertext and lets a resolver group the queries of one client), no query log, no monitoring UI, an own hardened unit, and absolute cache paths in a cache that the daemon account owns and a configuration directory that it cannot write.

The daemon reads the resolvers and the relays from four signed lists in /var/cache/dnscrypt-proxy, and refreshes them every 72 hours. The converge seeds the lists, so a first start reads from disk. The few host names the daemon needs before a route is live, such as the ODoH relay, resolve through plain Quad9. Those two resolvers see those host names and no other query.

The daemon caches nothing and logs no query. The recursor does both in front of it. Its journal is in ClickStack under the ServiceName dnscrypt-proxy, through dns-otel-collector. The rows hold no client address, so the row filter of the two MCP users does not cover them.

Two HyperDX alerts read those rows, on the dashboard DNS — resolver alerts:

Alert Fires when Shape
DNS dnscrypt-proxy has no live route a node logs waiting for at least one server to be reachable in a 5-minute window Once at the start of an outage. The daemon writes that line one time
DNS dnscrypt-proxy resolvers timing out a node logs more than 60 resolver TIMEOUT lines in a 5-minute window In every window while the outage lasts. Five dead resolvers log about 140 lines in 5 minutes, one dead resolver about 30

Both were proved on 2026-10-06 with an nftables table on the passive node that dropped the UDP of the daemon account, and a restart of the daemon. The first alert fired in the window of the restart and cleared in the next. The second fired in each window until the table left. To repeat that break on a node that nothing uses yet:

Terminal window
# On a resolver node, as root. Reverse with `nft delete table inet dcp_break` and a restart.
nft add table inet dcp_break
nft add chain inet dcp_break out '{ type filter hook output priority 0; }'
nft add rule inet dcp_break out meta skuid "dnscrypt-proxy" meta l4proto udp counter drop
systemctl restart dnscrypt-proxy

While no route is live, a public name that is not in the cache answers SERVFAIL after 6 seconds on that node. A name in the cache gets a stale answer for up to one hour. A stopped daemon answers SERVFAIL in 2 milliseconds, so systemctl stop dnscrypt-proxy is the first move on a node whose routes are all dead.

The role removes the Debian package dnscrypt-proxy when it finds it, because the package shares the unit name and the configuration path. The role runs six criteria at the end of each converge: the daemon account reads the configuration (DCP-01), the daemon’s own parser accepts it (DCP-02), the unit is active as that account (DCP-03), the journal of the current start names a relay for each resolver and reports every resolver live (DCP-04), each loopback listener answers example.org (DCP-05), and each loopback listener refuses a name under each internal zone (DCP-06).

To read the route state on a node:

Terminal window
# On a resolver node.
sudo journalctl -u dnscrypt-proxy -b --no-pager | grep -E 'Anonymizing queries for|live servers|reached anonymously'

To change the route set, edit dnscrypt_proxy_servers and dnscrypt_proxy_routes in the role defaults and converge --tags dnscrypt-proxy one node at a time. Pick a relay from relays.md or odoh-relays.md in the cache, and never one that the operator of the resolver runs.

The release is proved by its minisign signature with the release key of the project, and the lists by the list key. The two keys differ, and both are in the role defaults. The release key is on the page Installation of the dnscrypt-proxy wiki.

Running a play, and how one-node-per-window is expressed on the command line

Section titled “Running a play, and how one-node-per-window is expressed on the command line”

Use the wrapper. It materializes credentials and pins the inventory:

Terminal window
scripts/dns-playbook.sh --limit sandiego --tags pdns-verify --check --diff # dry-run, one node
scripts/dns-playbook.sh --limit sandiego --tags pdns-recursor # one node, one tier
scripts/dns-playbook.sh --limit sandiego --tags pdns-recursor-verify # the read-only checks of that tier
scripts/dns-playbook.sh --limit portland --tags kea -vv # verbose

Ansible does not fail on an unmatched tag. It selects zero tasks and prints a green PLAY RECAP. Make sure that the tag is in ansible/dns-playbook.yml before you trust a green recap.

ansible/dns-playbook.yml is eleven separate plays, one role each, every role tagged: dns-common, pdns, lightningstream, dnscrypt-proxy, pdns-recursor, dns-otel-collector, vault-cert-agent, kea, keepalived, dns-tailscale and dns-node-worker, in that dependency order. Each role also exposes a -verify tag that selects only its read-only assertions, such as pdns-recursor-verify.

Never change both resolver nodes in the same window. The estate has exactly two, and a change applied to both at once leaves no surviving node. --limit sandiego --tags pdns-recursor is the expressible form of that rule. --limit reaches the node it names because ansible_host is each host’s own Core address, never the VIP.

The cut-over to the recursor, one node per window

Section titled “The cut-over to the recursor, one node per window”

The change dns-policy-as-rpz moved the client tier from Blocky to PowerDNS Recursor on 2026-10-14. The cut-over of one node is a converge of six tags, then their six -verify tags. Run it on the node that holds no VIP first. scripts/dns-vip-holder.py names the holder, and --passive names its peer.

  1. If this node holds a VIP, drain it, so its peer holds every VIP during the converge. Use the MCP tool dns_drain_node of dns-control, or follow Drain and undrain through dns-control. The node that holds no VIP needs no drain.

  2. Converge the six roles on this node:

    Terminal window
    scripts/dns-playbook.sh --limit <node> --tags dns-common,pdns,pdns-recursor,dns-node-worker,dns-otel-collector,keepalived

    The pdns-recursor role starts both recursor processes on the addresses that Blocky used. The role holds no Blocky task. On a node that still runs Blocky, stop and disable blocky.service and blocky-public.service before this step, because both tiers bind the same addresses.

  3. Make sure that every feed zone holds records. The converge reports the exit code of the first feed run, and the journal names each group with its count:

    Terminal window
    scripts/dns-adhoc.sh <node> -b -m ansible.builtin.shell -a 'journalctl -u dns-feeds -o cat | grep records='

    If a line reads unexpected arguments: feed, the worker binary on the node was older than release 0.23.0 when the pdns-recursor play ran, because that play comes before the dns-node-worker play. Start dns-feeds.service once more after the converge. If a group reads records=0, the source answered an error page with the status 200, and the CDN caches that page for hours. Copy the zone file of that group from the peer node into /var/lib/pdns-recursor-feeds, with the owner pdns, and run dns-feeds.service again. The unit reloads both processes and wipes their caches, so a name that the feed lists gets its block at once.

  4. Run the read-only checks of the same six roles:

    Terminal window
    scripts/dns-playbook.sh --limit <node> --tags dns-common-verify,pdns-verify,pdns-recursor-verify,dns-node-worker-verify,dns-otel-collector-verify,keepalived-verify
  5. If this node was drained, undrain it with dns_undrain_node.

  6. Take the readings of How to prove what an untrusted leg sees against this node, with --server <address> on the script of the skill.

Then do the same on the other node in the next window. Until the retire play runs, the Blocky units are the rollback of a node:

Terminal window
scripts/dns-adhoc.sh <node> -b -m ansible.builtin.shell -a 'systemctl disable --now pdns-recursor@main pdns-recursor@public && systemctl enable --now blocky blocky-public'

After both nodes ran the recursor for one day, the retire play ansible/blocky-retire.yml removes Blocky from a node. It is not in dns-playbook.yml. Run it by hand, one node per window, the node that holds no VIP first:

Terminal window
scripts/dns-playbook.sh --limit <node> --playbook=blocky-retire.yml

The play stops and disables blocky.service and blocky-public.service, and removes the binary /usr/local/bin/blocky, the two unit files, /etc/blocky, /var/cache/blocky and the operator files directory /var/lib/dns-worker/lists. It leaves /var/lib/dns-worker itself, which holds the drain marker, the scope map and the Kea reservation files, and it leaves the blocky user and group. A second run reports no change. The readback on the node:

Terminal window
# On a resolver node.
systemctl is-active blocky blocky-public # want: no line reads active
test -e /etc/blocky; echo "RC=$?" # want: RC=1

After the retire play, the rollback of a node is a revert of the cut-over pull request and a converge of --tags blocky on that node.

dns-common ships a dead-man switch. dns-common-revert.timer starts dns-common-revert.service a fixed interval after it is armed. That service unmasks and starts NetworkManager, masks and stops systemd-networkd, and stops nftables, which puts the node back on the network manager it was reachable through before the converge. Arm it before a change that can strand you. Disarm it once you can reach the node.

Run it against one node at a time. These commands are not read-only. The session that issues the hand-run apply in the recovery below does not survive it. Reconnect on the node’s Core address.

Terminal window
NODE=sandiego # the ONE node being worked on
NODE_IP=192.168.40.6 # that node's CORE address: sandiego .6, portland .7. Never the .5 VIP.

Arm it:

Terminal window
scripts/dns-playbook.sh --limit "$NODE" --tags dns-common \
--skip-tags dns-common-venv,dns-common-daemons \
-e dns_common_revert_enabled=true -e dns_common_revert_delay=600s
# want: PLAY RECAP ... changed=2 unreachable=0 failed=0
# The two changed tasks render the timer unit and set its state. revert-arm.yml is the last
# include, after verify.yml, so a converge that fails earlier never arms anything.
# 600s is the production delay. Pass both values as -e, never by editing dns_hosts.yml: an
# armed flag left true in a commit is an outage on somebody else's next converge.
sudo systemctl start dns-common-revert.timer
# A hand re-arm. OnActiveSec counts from THIS start.

Disarm it, in an already-open session, with the read and the stop in one invocation:

Terminal window
sudo systemctl stop dns-common-revert.timer
# A stop clears the runtime half only. The unit file stays enabled with its rendered delay, so a
# reboot re-arms it. A converge with dns_common_revert_enabled at its false default clears that.

Read it. Read SubState, never is-active:

Terminal window
sudo systemctl show dns-common-revert.timer \
-p ActiveState -p SubState -p LastTriggerUSec -p NextElapseUSecMonotonic
# want (armed): ActiveState=active SubState=waiting LastTriggerUSec= (empty)
# want (fired): ActiveState=active SubState=elapsed LastTriggerUSec=<a stamp>
# Keep the keys. --value strips them, and systemd emits properties in its own order.
# NextElapseUSecRealtime is empty on this timer, because OnActiveSec= is monotonic.
Timer state SubState LastTriggerUSec ActiveState
Armed, has not fired waiting empty active
Fired, not stopped elapsed set active
Fired then stopped dead empty inactive
Disarmed, then stopped dead empty inactive

The last two rows are identical. After any stop, journalctl -t dns-common-revert is the only instrument that discriminates, so capture it before you disarm. The timer sets no AccuracySec=, so the real window is OnActiveSec through OnActiveSec + 60s.

A complete fire writes one opening line, five ACTION-OK lines in manifest order (unmask and start NetworkManager, mask the networkd socket, mask the networkd service, stop nftables) and one DECISION=REVERT-COMPLETE line. Read the order by index. The exit status is 0 when every action returned 0, 64 for an unrecognised argument, 78 for an empty systemctl command, and otherwise the number of actions that did not succeed, each named under DECISION=REVERT-INCOMPLETE.

Stopping systemd-networkd does not release its addresses or destroy its VLAN netdevs. After a fire the static addresses and the VLAN netdevs remain present and UP, and NetworkManager adds one DHCP address of its own. A reverted node stays reachable at its inventory address.

A fired revert leaves the node with no packet filter. nftables.service is stopped, so the node is default-allow on every VLAN it holds a leg on until a converge restores the ruleset. Close it deliberately.

The converge refuses on a reverted node, and the refusal is the only detection surface a fired revert has. Do not add masked: false to that task:

Terminal window
scripts/dns-playbook.sh --limit "$NODE" --tags dns-common \
--skip-tags dns-common-venv,dns-common-daemons
# want, against a reverted node:
# TASK [dns-common : Set systemd-networkd's enabled state from its guard]
# Unable to enable service systemd-networkd.service: ... is masked
# PLAY RECAP ... unreachable=0 failed=1

Get the node back. Apply first, converge second. No Ansible verb on the converge path lifts a systemd mask:

Terminal window
# 1. Lift the masks by hand. Detached, because the session does not survive it.
ssh -o ControlPath=none fzymgc@"$NODE_IP" \
'sudo -n systemd-run --collect --unit=dns-common-network-apply-recover \
/usr/local/sbin/dns-common-network-apply.sh'
# want: rc=0 and `Running as unit: dns-common-network-apply-recover.service`
# 2. Reconnect on Core, then confirm both masks are gone. Count, never a bare ls.
ssh -o ControlPath=none fzymgc@"$NODE_IP" \
'ls -1 /etc/systemd/system/systemd-networkd.service /etc/systemd/system/systemd-networkd.socket \
2>/dev/null | wc -l'
# want: 0
# 3. Converge. This re-enables networkd, restarts nftables, restores OnActiveSec=600s and
# disarms the timer on both axes.
scripts/dns-playbook.sh --limit "$NODE" --tags dns-common \
--skip-tags dns-common-venv,dns-common-daemons
# want: PLAY RECAP ... unreachable=0 failed=0
# 4. Confirm the production delay is back.
ssh -o ControlPath=none fzymgc@"$NODE_IP" \
"systemctl cat dns-common-revert.timer | grep -E '^OnActiveSec='"
# want: OnActiveSec=600s

journalctl -t dns-common-network-apply logs every action in manifest order. A complete run ends DECISION=HANDOVER-COMPLETE.

A residual NetworkManager lease needs no step. It lands in Core on the same eth0 as the static .6 or .7 and expires on its own. If you chase odd source-address or firewall behaviour on a freshly recovered node, read it as a count:

Terminal window
ssh -o ControlPath=none fzymgc@"$NODE_IP" 'ip -o -4 addr show dev eth0 | wc -l'
# want: 1, the static Core address alone. 2 means the lease is held.

Three /dev/null mask symlinks in /etc/systemd/system/ is the correct steady state: NetworkManager.service, unbound-resolvconf.service and userconfig.service. Neither networkd unit is among them. To assert the directory is unchanged, compare a name-only digest (ls -1 /etc/systemd/system/ | LC_ALL=C sort | sha256sum).

ULA prefix announcement (the resolver pair)

Section titled “ULA prefix announcement (the resolver pair)”

Both resolvers advertise the ULA prefix on every leg, permanently, at a finite lifetime (dns_common_ra_announce_enabled: true in ansible/inventory/group_vars/dns_hosts.yml).

dns-common-ra-announce.service runs radvd as a standing daemon on both nodes, reading a role-owned config at /etc/dns-common/radvd-announce.conf rendered from dns_vlan_interfaces, one interface stanza per leg. Each leg advertises its own fddb:f665:73f7:N::/64. The values are in ansible/roles/dns-common/defaults/main.yml:

Directive Value What it buys
AdvOnLink on the client forms an on-link route to that leg’s ULA VIP
AdvAutonomous on the client can configure an address from the prefix. With it clear a client ignores the whole prefix-info option (RFC 4862 §5.5.3(a))
AdvPreferredLifetime 3600 finite
AdvValidLifetime 7200 finite. The ceiling on how long a withdrawal takes
AdvDefaultLifetime 0 neither resolver becomes a default router. On the wire, router_lifetime=0
MinRtrAdvInterval / MaxRtrAdvInterval 200 / 600 the unsolicited cadence. The join path is the solicitation
RDNSS the leg’s vip6, lifetime 1800 gated by dns_common_ra_announce_rdnss_enabled alone. The RA of the gateway carries the same VIP on every leg, so the two sources agree (RFC 8106 §5.3.1)

The pidfile is /run/dns-common-ra-announce.pid. See The pidfile below.

Without this announcement nothing on the trunk advertises the prefix the floating addresses live in, and a client handed a ULA VIP has no route to it.

A standing daemon rather than bursts, because only a listening daemon answers a Router Solicitation inside RFC 4861’s MAX_RA_DELAY_TIME. That is how a fresh client joins in under a second, and it is what makes scripts/solicit-ra.py --router a valid instrument. Both nodes, because prefix information is idempotent, and the estate survives one node down with no failover logic.

A zero exit from systemctl start is not the proof. The proof is ExecMainStartTimestamp in wall-clock epoch seconds, strictly later than the config file’s mtime:

Terminal window
# On a resolver node.
SVC=dns-common-ra-announce.service
CONF=/etc/dns-common/radvd-announce.conf
TS=$(systemctl show "$SVC" -p ExecMainStartTimestamp --value)
echo "START_EPOCH=$(date -d "$TS" +%s)"
echo "CONF_MTIME_EPOCH=$(stat -c %Y "$CONF")"
systemctl show "$SVC" -p ActiveState -p SubState -p MainPID
radvd -c -C "$CONF" -p /run/radvd-configtest.$$.pid ; echo "configtest=$?"

Want: START_EPOCH strictly greater than CONF_MTIME_EPOCH, active / running, configtest 0. Never compare ExecMainStartTimestampMonotonic against an mtime. It is microseconds since boot.

From the repository root, one dispatch per node, each attributed to the peer. A node does not receive its own multicast, so a node asked to attribute to itself reads UNINTERPRETABLE:

Terminal window
scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.script -a "../scripts/solicit-ra.py --router fe80::8aa2:9eff:fede:ae7c" # attributes to portland
scripts/dns-adhoc.sh portland -b -m ansible.builtin.script -a "../scripts/solicit-ra.py --router fe80::8aa2:9eff:fed4:6167" # attributes to sandiego

Those two link-locals are each node’s, identical across the legs, recorded beside dns_common_ra_watch_expected_lladdrs in ansible/inventory/group_vars/dns_hosts.yml.

Three dispatch traps:

  • The ../ prefix is required. ansible.builtin.script resolves a relative path against the playbook directory (ansible/), not the repository root.
  • -b is required. The script opens a raw ICMPv6 socket.
  • A non-zero exit prints the host FAILED in red, and that is the gate working. Exit 1 is measured and absent. Exit 2 is uninterpretable, and no verdict is claimed for the set.

Want, per node: rc 0 and SUMMARY: <n> advertising, 0 not advertising, 0 uninterpretable, of <n> legs, where <n> is the number of legs in dns_vlan_interfaces.

The exit status is a prefix-presence verdict only. A claim about the lifetimes must be read from the printed vlt= and plt= fields. Because both nodes announce, an unscoped count measures the link, not the node. Count only inside blocks whose printed source is the node under test:

Terminal window
# WRONG for a per-node claim: it counts the peer's frames too.
grep -o 'vlt=7200 plt=3600' sandiego.out | wc -l
# RIGHT: count only inside blocks whose printed source is the node under test.
awk -v tgt=fe80::8aa2:9eff:fed4:6167 '
/^ *from / { intgt = ($2 == tgt); next }
/^=== / { intgt = 0; next }
intgt && /vlt=7200 plt=3600/ { n++ }
END { print n+0 }
' sandiego.out

Want: one per leg. Keep a sanity count (grep -c 'vlt=') beside it. Also want, inside the target’s blocks: L=1 A=1 on every prefix-info line, router_lifetime=0 on every block, and an rdnss line carrying the leg’s vip6.

Terminal window
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell -a "systemctl is-enabled radvd.service ; ps -o pid,unit,cmd -C radvd --no-headers || echo 'no radvd process'"

Want: systemctl is-enabled radvd.service prints masked on both nodes (Debian’s own unit, masked whenever the package is present), and every row of ps shows a role unit in the UNIT column (dns-common-ra-announce.service or dns-common-ra-deprecate.service).

Two concurrent radvd is compliant: the standing announcer plus a bounded deprecation window. A check written as a process count equal to 1 is red on a correct estate. radvd forks, so even a single announcer shows two rows sharing one unit. Count by unit.

The announcer runs with -p /run/dns-common-ra-announce.pid. radvd takes an exclusive lock on its pidfile, and the bounded dead-prefix emitter passes no -p, so it takes the compiled-in default. If the announcer used that default, the emitter’s OnFailure=-triggered start would exit 255 and the dead-prefix withdrawal would never be sent.

Terminal window
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell -a "ls -l /run/dns-common-ra-announce.pid /run/radvd.pid /run/radvd/radvd.pid 2>&1 ; systemctl is-failed dns-common-ra-deprecate.service ; systemctl show dns-common-ra-deprecate.service -p Result -p ExecMainStatus"

Want: the announcer’s own pidfile present, the two compiled-in defaults absent, the emitter not failed, with Result=success and ExecMainStatus=0. inactive (dead) is the correct resting state for a bounded one-shot. The journal signature of a collision:

Terminal window
journalctl -u dns-common-ra-deprecate.service --no-pager | grep -i 'lock pid file'

The same lock is why radvd -c needs its own pidfile. Pass a scratch path, as the adoption read above does.

Set dns_common_ra_announce_enabled: false and converge, one node per window:

Terminal window
scripts/dns-playbook.sh --limit portland --tags dns-common-ra-announce
# verify, then the same command with --limit sandiego

The arm is two-sided. ra-announce-arm.yml derives both enabled and state from that one flag, so the converge disables and stops the unit. Keep the flag committed true: with it false, any routine --tags dns-common converge stops both announcers.

There is no withdrawal code. The prefix ages out within the advertised valid lifetime, at most two hours. radvd’s shutdown-deprecation prefix option is off and must stay off: it is safe only when a single router announces the prefix, and here two do.

This withdraws the recursive-DNS advertisement only. All three steps are required. A converge alone changes nothing on the wire:

Terminal window
# 1. Override the arm flag off, one node per window. Use the JSON form.
scripts/dns-playbook.sh --limit portland --tags dns-common-ra-announce \
-e '{"dns_common_ra_announce_rdnss_enabled": false}'
# 2. Confirm the rendered config lost the blocks (expect 0).
scripts/dns-adhoc.sh portland -b -m ansible.builtin.shell \
-a 'c=$(grep -c "^ *RDNSS " /etc/dns-common/radvd-announce.conf || true); echo "rdnss_block_count=$c"; test "$c" -eq 0'
# 3. Restart the announcer. Without this the wire does not change.
scripts/dns-adhoc.sh portland -b -m ansible.builtin.shell \
-a 'systemctl restart dns-common-ra-announce.service'

To restore, run the same three steps without the -e override and restart again. The restart is required in both directions: the conf-render task carries no notify:, the unit declares no ExecReload=, and no role handler names the announcer. -e key=value passes a string, and the guard applies | bool, so the JSON form is belt and braces.

The prefix survives the withdrawal. After the restart every leg reads rdnss (0): NONE with the prefix-info line unchanged. scripts/solicit-ra.py’s exit code covers the prefix, not the RDNSS. Read the printed option.

Plan-sequenced, never a role handler. The unit carries no Restart= policy, and the role installs no handler, following the same rule as keepalived: a converge must never be the thing that moves the serving plane. A radvd that has exited is a failed unit an operator reads.

Dead-prefix deprecation — when the ISP re-delegates

Section titled “Dead-prefix deprecation — when the ISP re-delegates”

A re-delegation can leave the old prefix live on every client. The Firewalla that was the gateway before 2026-10-01 did this. FireRouter’s dnsmasq stopped mentioning the old /64 instead of withdrawing it with preferred lifetime 0 (RFC 4862 §5.5.3). Nobody has measured whether the UCG-Fiber does the same, so the detector and the deprecation stay armed.

The failure to expect is a client holding a dead prefix for its remaining valid lifetime while every unit reports healthy. RFC 6724 source selection keeps choosing the dead prefix, so the client’s IPv6 fails while a probe forced onto the new prefix succeeds. Compare the default source address against one pinned to the current prefix before concluding anything about the gateway.

What detects, and how long it takes to notice

Section titled “What detects, and how long it takes to notice”

dns-common-ra-watch.service runs one capture per run (dns_common_ra_watch_window_seconds, 900 s) every dns_common_ra_watch_interval (6 h). It harvests prefix-information options from the frames of the gateway only, per VLAN. A gateway frame matches both the link-local fe80::76f9:2cff:fed8:fe0c (dns_common_ra_watch_appliance_lladdr) and the MAC 74:f9:2c:d8:fe:0c (dns_common_ra_watch_appliance_mac), in ansible/roles/dns-common/defaults/main.yml. The detector classifies each observed prefix by family: global under 2000::/3, unique-local under fc00::/7, RFC 3849 2001:db8::/32 as documentation, anything else other. Three journal literals carry it:

Literal Emitted when
RA-WATCH-PREFIX one per distinct observed prefix, with vlan, prefix and class
RA-WATCH-PREFIX-BASELINE first observation for a VLAN, no recorded state. Never an action
RA-WATCH-PREFIX-CHANGE an in-scope global prefix changed. This is the ClickStack alert’s subject

Detection latency is up to one full cadence plus one capture window. Shortening it is one role default away, and it is not done: a re-delegation is rare, and the capture runs tcpdump on the two nodes that are the entire DNS serving plane.

A ULA change is reported and never acted on. Only the global class participates in change detection.

dns_otel_journal_units in ansible/roles/dns-otel-collector/defaults/main.yml names dns-common-ra-watch.service. Removing it un-arms the alert: the detector’s output reaches default.otel_logs nowhere, and an alert over it is green forever. The collector does not re-read its configuration on its own. The role carries a restart handler. If you edit that configuration outside Ansible, restart the unit yourself.

dns-common-ra-deprecate.service runs dns-common-ra-deprecate.sh, which emits a bounded deprecating RA with radvd:

  • Preferred lifetime 0 is the whole fix. RFC 4862 §5.5.3(e) makes a host ignore a reduction of the valid lifetime below two hours unless the RA is authenticated, so the valid lifetime is set above the clamp and the preferred lifetime is the field that deprecates.
  • Router lifetime 0 on every stanza. It deprecates without claiming to be a default gateway.
  • Bounded. --announce runs under timeout for dns_common_ra_deprecate_duration_seconds and stops. timeout’s exit 124 is the expected end, and the unit declares it a success.
  • Never Debian’s own radvd.service, and never /etc/radvd.conf. Writing that path arms a second, unbounded, boot-persistent radvd. The converge masks the Debian unit whenever the package is present.
  • No recursive-DNS-server option. A resolver-list change on a transient burst would make a client’s DNS depend on whether it was listening during those minutes. The standing announcer carries RDNSS.

Expect our own frames in the peer’s detector output. While an announcement runs, the other resolver sees an RA source other than the gateway, with router lifetime 0s and no RDNSS. That lands in the RA-WATCH-OBSERVED bucket by membership of the expected-source set.

Flag False True
dns_common_ra_prefix_watch_enabled the detector prints RA-WATCH-PREFIX* and records prefix state. Pure observability the exit-3 path and the deprecation request file are live
dns_common_ra_deprecate_enabled no radvd installed. /etc/dns-common/ra-deprecate.armed is absent, so the emitter unit is skipped radvd installed, arm file present, the emitter can announce

The second is a file, not a unit state. dns-common-ra-watch.service carries OnFailure=dns-common-ra-deprecate.service, and that fires on every non-zero detector exit. An unsatisfied ConditionPathExists= is a skip rather than a failure, which is what makes that trigger safe.

One node per window, the non-holder first. Verify the first node before touching the second:

Terminal window
scripts/dns-playbook.sh --limit portland --tags dns-common-ra-watch --check --diff
scripts/dns-playbook.sh --limit portland --tags dns-common-ra-watch
# verify, then the same two commands with --limit sandiego

Both arms are two-sided. A converge with a flag false disarms rather than declining to arm.

systemctl start on a detector that is already running is a no-op, and a window that is already running keeps the old script. Before reading a triggered run as evidence for a script change, compare systemctl show dns-common-ra-watch.service -p ExecMainStartTimestamp --value against the script’s mtime.

--tags dns-common-ra-watch renders the whole detector tier: the detector script, both detector units, the emitter script and the emitter unit. --tags dns-common-ra-announce runs only the radvd install, the package stat and the mask of Debian’s unit. The announce tag does not deploy the detector or the emitter. If you changed a detector or emitter template, converge the detector tag.

The check that settles which tasks a tag reaches runs on the control node:

Terminal window
cd ansible && ansible-playbook -i inventory/hosts.yml dns-playbook.yml \
--limit portland --tags dns-common-ra-watch --check --diff

--list-tasks cannot expand include_tasks.

Every RA-WATCH-OBSERVED line carries expected=yes or expected=no, set by membership of dns_common_ra_watch_expected_lladdrs, which carries both resolvers’ link-local addresses. The membership test runs above the RDNSS test, because the pair advertises its own RDNSS on every leg.

  • An expected=yes line carrying an address in its rdnss= field is the pair announcing.
  • An expected=no source carrying an RDNSS on a trusted leg is a violation. It exits 2, lands dns-common-ra-watch.service in failed, and is reported on a RA-WATCH-VIOLATION line.
  • On an untrusted leg (Guest, Telework) the same frame is reported on a RA-WATCH-UNTRUSTED-RDNSS line and does not exit 2. Those are segments we do not police.
  • The observed bucket’s count is reported as non_appliance_non_violating= on the RA-WATCH-SUMMARY line.

The blind spot: the exemption keys on the source link-local alone, and a link-local is spoofable on-link. An attacker who spoofs one of the two pair addresses and advertises a hostile resolver is invisible to this detector. The detector reports and does not prevent. Do not widen the exemption.

The detector sees its own node’s advertisements, because tcpdump taps at the device layer. So dns_common_ra_watch_expected_lladdrs must carry both nodes’ link-locals. Do not narrow that list to a peer-only list.

The read is a cross-tab:

Terminal window
journalctl -u dns-common-ra-watch.service --since '<window start>' --until '<now>' --no-pager > /tmp/ra.txt
grep -o 'RA-WATCH-OBSERVED' /tmp/ra.txt | wc -l
sed -n 's/.*src=\([^ ]*\).*expected=\([^ ]*\).*/\1 \2/p' /tmp/ra.txt | sort | uniq -c

The cross-tab counts must sum to the RA-WATCH-OBSERVED count. If they do not, the detector on that node predates the field, and the fix is a --tags dns-common-ra-watch converge. No source appears on both sides. An expected=no source that is not one of the long-standing non-resolver sources is the change this annotation exists to surface.

The deprecating RA is observable on demand against an RFC 3849 documentation prefix no client can hold. The detector classes 2001:db8::/32 as documentation, so the rehearsal frame cannot reach the deprecation path.

Before you start, on both nodes: systemctl is-active dns-common-ra-watch.service (whether a capture is running), systemctl list-timers dns-common-ra-watch.timer --all --no-pager (when the next one opens), and cat /var/lib/dns-common/ra-prefix.state (whether a global prefix is recorded for the leg).

Do not run this rehearsal in a Kea cut window. To prove only that the self-deprecation applies preferred_lft 0, use the procedure below, which transmits nothing.

Terminal window
# 1. A scratch config for ONE leg. No recursive-DNS-server directive.
scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.shell -a 'cat > /tmp/probe.conf <<EOF
interface eth0
{
AdvSendAdvert on;
AdvDefaultLifetime 0;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 4;
prefix 2001:db8:dead:5f00::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvPreferredLifetime 0;
AdvValidLifetime 7200;
};
};
EOF'
# 2. Run radvd in the foreground for about 60 s and capture on the NAMED interface.
# Never -i any: cooked-mode offsets differ, so ip6[40] matches nothing.
scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.shell -a \
'(timeout 60 radvd --logmethod stderr -n -d 5 -C /tmp/probe.conf > /tmp/radvd.out 2>&1 &) ; sleep 3;
timeout 30 tcpdump -i eth0 -e -nn -v -s 0 -w /tmp/ra.pcap "icmp6 and ip6[40] == 134"'
# 3. Read it back with -v. Options only print under -v.
scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.shell -a 'tcpdump -e -nn -v -r /tmp/ra.pcap'

Assert all three on the frame sourced from this node’s own link-local: router lifetime 0s, a prefix info option for the dead /64 with Flags [onlink, auto] and pref. time 0s, and no RDNSS option on our frames. The frames of the gateway in the same capture carry one, which is the positive control.

The 30 s window is sufficient here because the 3/4 s interval is ours. A capture of gateway frames must run longer than the maximum RA interval of the gateway and be bounded by timeout, never by -c N.

Clean up: kill radvd, remove the scratch config, apt-get purge -y radvd, and re-read the forwarding sysctl.

This proves that --deprecate-local fires and stamps preferred_lft 0 on this node’s own address for a named prefix. It puts nothing on the wire.

The condition has to be manufactured. At rest the estate shows zero addresses under a dead prefix, because the same clock that made the prefix dead also removed the evidence. Two things make the reading mean anything:

  1. The antecedent. An address must exist under the named prefix, on the target leg, with a non-zero preferred lifetime, read before the invocation.
  2. The discriminator. Plant it with a preferred lifetime of an hour, invoke, and take the post-fire read within a couple of minutes. Expiry cannot explain preferred_lft 0sec observed a minute into a 3600-second lifetime.

Do not use the unit for this. systemctl start dns-common-ra-deprecate.service also runs --announce. Invoke the script directly. Run it on the non-holder. <LEG> is eth0.1000 (Lab), the lowest-traffic leg.

Terminal window
# 1. Pre-flight. Both counts must read 0. This output is also the recovery record.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; ip -6 addr show scope global ; \
ip -6 addr show scope global | grep -o 'preferred_lft 0sec' | wc -l ; \
ip -6 addr show scope global | grep -o '2001:db8:' | wc -l ; \
ls -l /var/lib/dns-common/ra-deprecate.request 2>&1 ; \
sha256sum /var/lib/dns-common/ra-deprecate.request 2>&1"
# 2. Plant the synthetic address with a known finite lifetime. Read it back and confirm a
# non-zero preferred_lft before going further.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; \
ip -6 addr add 2001:db8:04f3::1/64 dev <LEG> noprefixroute valid_lft 7500 preferred_lft 3600 ; \
ip -6 addr show dev <LEG> scope global"
# 3. The request, through the emitter's own file interface, with a fresh stamp. Then the safety
# pre-flight: the doc-prefix count must read exactly 1.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; \
printf 'requested_at=%s\nvlan=lab leg=<LEG> old_prefix=2001:db8:04f3::/64\n' \$(date +%s) \
> /var/lib/dns-common/ra-deprecate.request ; \
cat /var/lib/dns-common/ra-deprecate.request ; \
ip -6 addr show dev <LEG> scope global | grep -o 'inet6 2001:db8:4f3:' | wc -l ; \
ip -6 addr show dev <LEG> scope global | grep -o 'inet6 2600:' | wc -l ; \
ip -6 addr show dev <LEG> scope global | grep -o 'inet6 fddb:' | wc -l"
# 4. Fire. Directly, never via the unit. Read the printed rc.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; \
/usr/local/sbin/dns-common-ra-deprecate.sh --deprecate-local 2>&1 ; echo rc=\$?"
# 5. The reading, taken immediately. Record the elapsed seconds against step 2.
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; ip -6 addr show scope global ; \
ip -6 addr show scope global | grep -o 'preferred_lft 0sec' | wc -l"

The count in step 3 is anchored on 4f3, not 04f3. The kernel prints addresses in RFC 5952 canonical form, which strips leading zeros. A count anchored on the string you typed reads 0 on a leg that carries the address. The old_prefix= in the request file is unaffected, because the matcher compares hextet by hextet with zero padding.

The request-file interface is the way in. A first observation for a VLAN always emits RA-WATCH-PREFIX-BASELINE and never deprecates, so a hand-edited prefix-state file cannot drive this path. The file carries a staleness fence read from its own requested_at stamp, ceiling dns_common_ra_deprecate_request_max_age_seconds (3600 s). Give old_prefix an explicit /64.

How to read it, both sides:

  • Positive. The planted address reads preferred_lft 0sec with a finite, non-zero valid_lft, and the kernel adds its own deprecated token.
  • Collateral. The count of addresses reading preferred_lft 0sec on that node goes from 0 to exactly 1, and every fddb: address and every live 2600: address reads a non-zero preferred lifetime.

The kernel never clamps preferred_lft 0. It clamps valid_lft, which is why the valid lifetime is planted at 7500. ip -6 addr change replaces the whole attribute set, so a flag not named is a flag cleared. mngtmpaddr and noprefixroute are settable and are re-stated. dynamic and scope global are derived, and re-stating either is an iproute2 usage error. The emitter filters the observed flags to the settable subset.

The unique-local fence refuses a request naming an fddb: prefix before any address is read. Probe it against a prefix the node holds an address under:

Terminal window
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "\
printf 'requested_at=%s\nvlan=lab leg=<LEG> old_prefix=fddb:f665:73f7:4::/64\n' \$(date +%s) \
> /var/lib/dns-common/ra-deprecate.request ; \
/usr/local/sbin/dns-common-ra-deprecate.sh --deprecate-local 2>&1 ; echo rc=\$? ; \
ip -6 addr show dev <LEG> scope global"

Expect rc=1 with RA-DEPRECATE SKIP: the deprecation request names the unique-local prefix …, and every fddb: row unchanged. Exit 1 is the refusal. Exit 2 belongs to the strict-argv refusals only.

Clean up, and verify the cleanup:

Terminal window
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "\
ip -6 addr del 2001:db8:04f3::1/64 dev <LEG> ; \
rm -f /var/lib/dns-common/ra-deprecate.request ; \
ls -l /var/lib/dns-common/ra-deprecate.request 2>&1 ; \
ip -6 addr show scope global | grep -o '2001:db8:' | wc -l ; \
ip -6 addr show scope global | grep -o 'preferred_lft 0sec' | wc -l ; \
systemctl list-units --failed --no-pager"

Both counts must read 0 and the request file must be absent, or restored from the copy you took in step 1 and proved identical with sha256sum. A hand-written request left behind arms the emitter against the next detector failure. Finally, re-read the VIP line counts and both keepalived ExecMainStartTimestamp values on both nodes in one invocation and make sure that they are unchanged.

radvd reads /proc/sys/net/ipv6/conf/all/forwarding, which is 0 on these resolvers. It logs IPv6 forwarding seems to be disabled, but continuing anyway and keeps advertising. No sysctl is flipped anywhere in this design.

Reading Kea lease state from a workstation

Section titled “Reading Kea lease state from a workstation”

This is the supported way to read DHCP lease state off the resolver pair. It needs no interactive shell on either node:

Terminal window
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
-a 'kea-shell --host 127.0.0.1 --port 8000 --auth-user kea-ctrl --auth-password-file /etc/kea/creds/ctrl-basic-auth.pw lease4-get-all </dev/null'

Three details:

  • The outer quotes on -a are single. A double-quoted dispatch argument is expanded by the control shell.
  • -b is required. The credential file is 0640 root:_kea.
  • </dev/null is required. kea-shell reads the command’s JSON arguments from standard input and waits for EOF.

Replace dns_hosts with sandiego or portland to ask one node. The listener is bound to 127.0.0.1, so this command only works on the node. The control channel has no read-only mode.

The response is a JSON array with one object per answering service, on one line:

Field Carries Read it as
result Kea’s control result code 0 success, 1 error, 2 command unsupported, 3 success with an empty result
arguments.leases the lease payload [ ] means no leases
text a human sentence a summary, never the datum

"result": 3 is not an error. It is CONTROL_RESULT_EMPTY: the lease table was read and held nothing. "result": 1 with an empty leases array is a failed command, and text says why.

Each lease under arguments.leases carries client-id, cltt, hostname, hw-address, ip-address, subnet-id, valid-lft and state. Read state before concluding a client is present: 0 is assigned, 2 is expired-reclaimed.

Both nodes must return the identical payload. They are HA partners and lease updates replicate. A difference is a replication finding, which is why the documented dispatch targets dns_hosts.

The lease CSV (/var/lib/kea/kea-leases4.csv) is not the read surface. Kea’s lease file cleanup rewrites it on its own interval, so a reader of the file races that rewrite. Do not simplify this procedure into a cat of the CSV.

Terminal window
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
-a 'kea-shell --host 127.0.0.1 --port 8000 --auth-user kea-ctrl --auth-password-file /etc/kea/creds/ctrl-basic-auth.pw status-get </dev/null'

The high-availability state is at arguments.high-availability[0].ha-servers.local.state, and the partner’s last known state at …ha-servers.remote.last-state. A healthy pair reports hot-standby on both nodes, one as primary and the other as standby. Read the state field before the lease count. A zero reported without the state field beside it is not an observation.

Symptom Almost always
kea-shell: command not found isc-kea-admin is not installed. Re-run scripts/dns-playbook.sh --limit <node> --tags kea-install
Connection refused on 127.0.0.1:8000 the daemon is stopped, or kea_ctrl_http_enabled was overridden false
401 from the listener the credential file and the Vault entry disagree. Re-run --tags kea-certs
the dispatch hangs </dev/null was dropped
permission denied opening the credential -b was dropped

The DHCPv4 accept and the raw-socket asymmetry

Section titled “The DHCPv4 accept and the raw-socket asymmetry”

Kea hands out addresses on a leg whose firewall never admitted a single DHCP packet, and it stops renewing them hours later. That asymmetry is why table inet dns_resolver chain input carries a udp dport 67 accept on both resolver nodes.

kea-dhcp4 opens AF_PACKET raw sockets, because a client in INIT holds no address and its DISCOVER is a broadcast from an unspecified source. Raw sockets bypass the firewall, so DISCOVER and OFFER reach the daemon whether or not any accept exists. A chain missing this rule looks healthy, and every first acquisition succeeds.

The RENEWING path does not bypass the chain. A client at T1 unicasts its DHCPREQUEST to the server identifier, this node’s own leg address, and that is an ordinary UDP datagram to port 67 that meets policy drop. The client sits unanswered from T1 until T2, then falls through to a broadcast REBIND, which the raw socket answers. At the committed lease of kea_valid_lifetime (43200 s) that first shows up about six hours after the change that caused it.

The rule is derived from dns_vlan_interfaces like the dport 53 accepts beside it, so it renders the same before and after any leg is armed. It is not scoped to kea_served_legs, because that is a per-converge -e value, and scoping the firewall to it would let a routine converge withdraw the accept from a leg Kea serves. It would also render iifname { } under an empty list, which nft -f rejects.

Read the accept back off the live kernel:

Terminal window
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
-a 'nft list ruleset | grep "udp dport 67" | grep -c accept'
# want: 1 on each node.

The port match and the verdict are matched separately. The kernel prints a counter clause between them (udp dport 67 counter packets 0 bytes 0 accept), so a pattern anchoring on the contiguous string dport 67 accept matches zero. The rule is IPv4-only. meta nfproto ipv4 is explicit, there is no IPv6 sibling, and DHCPv6 is not served by this pair:

Terminal window
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
-a 'nft list ruleset | grep -c "nfproto ipv6 udp dport 67"'
# want: 0 on each node.

Cutting a leg over to Kea, and rolling it back

Section titled “Cutting a leg over to Kea, and rolling it back”

This is written for any leg. Substitute the leg name, its pool boundary and its host_vars entry. Every leg is cut, so the procedure is a rollback and rehearsal instrument. There is exactly one rollback, reached two ways: by an operator who decided to roll back, and by an operator who hit one of the abort triggers below. Both run the same numbered steps.

The other half of a cut is the DHCPv4 server of the gateway. It is dhcp_server.enabled on the network of the leg in tf/unifi/networks.tf, and it is false on every leg. You change it with a pull request that main-cluster-unifi applies. Nobody has rehearsed a rollback onto the UCG-Fiber, so read the plan of that pull request in full.

To cut, the DHCPv4 server of the gateway goes off first, then Kea is armed. To roll back, Kea is disarmed on both nodes first, then the DHCPv4 server of the gateway goes back on.

Direction Step 1 Step 2
Cut dhcp_server.enabled = false for that network arm Kea on the leg, both nodes
Roll back disarm Kea on both nodes dhcp_server.enabled = true for that network
Re-cut dhcp_server.enabled = false again arm Kea again

Two DHCPv4 servers on one broadcast domain fight, and the client takes the first OFFER, so there is no safe overlap window. An unserved leg for minutes is accepted.

A rollback that stops between the two halves leaves the leg with no DHCP server at all. If the gateway half does not apply, escalate.

Machine Reached how Does what
The instrumented client over a different leg from the one being cut, through its own wrapper (scripts/nas-adhoc.sh for the NAS) every client-side read: address, lease, forced renew, v6 address and resolvers
The resolver pair over the untagged Core leg, scripts/dns-adhoc.sh and scripts/dns-playbook.sh every converge, and the server-side lease read
The control-node workstation none runs both wrappers. It is on neither the cut leg nor the pair’s DHCP path

The instrument must outlive its subject. SSH is accepted on the five trusted legs. Guest and Telework have no SSH path by design. Do not propose one for the window.

Any one of them runs the rollback below and closes the window:

  • (a) No client on the leg holds an address from Kea’s pool within N minutes of arming. For a leg instrumented at a ten-minute lease, N is 15 minutes: the client’s own T2 plus one acquisition.
  • (b) A client that has just acquired an address holds an address outside Kea’s pool. This is the leaked-OFFER signal. Only a client force-renewed or rebooted after the change can be read against it, because a client that has not renewed still holds its old lease.
  • (c) The static management path to either resolver node is lost. The nodes’ leg addresses are static and have no DHCP dependency, so losing the path means something other than DHCP moved.

Read every leg’s boundary from that leg’s host_vars entry (kea_pool). On Lab, .200.140 to .200.159 below the pool is static (the k3s node Lab legs and free static addresses), so an address there in a lease is a misconfiguration. On iot, the upper bound of the pool stops below 192.168.218.224, which is a device’s second interface with no reservation. Do not widen that pool into .224.

Field Changes at the cut? Read it as
The DHCP server identifier yes primary evidence, which server answered
The gateway no correctness only. The UCG-Fiber stays the default router
The advertised resolver no correctness only. Kea advertises the leg VIP

A reading that concludes “DNS is correct, therefore Kea answered” has concluded nothing. Confirm by lease origin: the lease appears in Kea’s own lease4-get-all on both nodes, and its lifetime is Kea’s 43200. DHCP4_PACKET_SEND ... DHCPACK (type 5) from <node address>:67 in the isc-kea-dhcp4-server journal names the sending address.

Why the lease is shortened before every window

Section titled “Why the lease is shortened before every window”

A rollback is a cold acquisition. At T1 the client unicasts to a server identifier that no longer answers. It looks for another server at T2 (0.875 of the valid lifetime), when it broadcasts. So the unattended cost of a rollback is bounded below by T2: about 9 minutes at a 600 s window lease, and about 10.5 hours at the committed 43200 s. Shorten the lease before every leg’s window. A forced renew on one instrumented client measures how fast the restored server answers, not how long an unattended device takes to ask.

  • Do not run the RA rehearsal in a cut window. It has a path into deprecating the live GUA prefix.
  • This procedure never changes the IPv6 half. No device serves DHCPv6.
  1. The reservation inventory. dns-control holds the entries, and the node worker writes the reservation file of each leg on both nodes, so this is a read of dns-control entries export or of the Changes screen. Until the handoff of the curated records merges, Terraform still applies the DNS records of tf/dns-records/hosts.yml, and that file must equal the entry set.

  2. The lease shortening on the server that serves the leg now, and its lead time. The condition is that every DHCPv4 client on the leg holds the shortened lease, measured client-side. One full previous lease time after the shortening is the calendar proxy.

  3. The DHCP Guard check. UniFi DHCP Guard drops every DHCP server-side frame that is not from a trusted address. Kea can be correct while clients DISCOVER endlessly with zero DHCPREQUESTs. tf/unifi/networks.tf declares each network’s dhcp_guarding (an enabled flag and at most three trusted servers). A cut that brings a new DHCP server onto a guarded VLAN adds its address to that list first. Where the guard is on, prove it with a throwaway macvlan client and a known-good VLAN as the control, before touching the gateway:

    Terminal window
    ip link add dhcptest0 link eth0.<vlan> address 02:00:00:de:ad:01 type macvlan mode bridge
    ip link set dhcptest0 up
    busybox udhcpc -i dhcptest0 -s /bin/true -n -q -R -t 4 -T 5 -f
    ip link del dhcptest0

    A lease on the control VLAN and none on the target is the guard, not the daemon.

The ruleset converge runs before the window and is firewall-scoped only:

Terminal window
scripts/dns-playbook.sh --limit <node> --tags dns-common-firewall
# One node at a time, non-VIP-holder first.

The in-window converges are the Kea role tag only:

Terminal window
scripts/dns-playbook.sh --limit <node> --tags kea \
-e kea_service_enabled=true \
-e 'kea_served_legs=["lab"]' \
-e kea_valid_lifetime=600 \
-e kea_renew_timer=300

Never the bare dns-common role tag in the window. It sweeps the VLAN legs and the network-apply transaction and re-arms the RA announcer.

Pass the served-legs value in JSON list form. -e kea_served_legs=lab passes a string, and the template’s membership test becomes a substring test.

To disarm, pass the served-legs list explicitly. Dropping the -e falls back to the committed default, and every leg is committed to group_vars, so a bare converge re-arms:

Terminal window
scripts/dns-playbook.sh --limit <node> --tags kea \
-e kea_service_enabled=true \
-e 'kea_served_legs=["<every leg that stays served>"]'
# Rolling back the only served leg:
scripts/dns-playbook.sh --limit <node> --tags kea \
-e kea_service_enabled=false \
-e 'kea_served_legs=[]'

The three-part read that says a leg is served

Section titled “The three-part read that says a leg is served”

A leg is served when three things are true at once: Kea’s journal shows sockets open on the leg, one lease is allocated from that leg’s own pool, and one dynamic name resolves in that leg’s carved zone, forward and reverse. Any one alone has a healthy-looking failure mode.

  1. Sockets open on the leg:

    Terminal window
    # On a resolver node.
    grep -c '"<leg-interface>"' /etc/kea/kea-dhcp4.conf
    # want: 1 on each node.
    SINCE='<the arm time, as YYYY-MM-DD HH:MM:SS UTC>'
    sudo TZ=UTC journalctl --utc -u isc-kea-dhcp4-server --since "$SINCE" --no-pager -o cat -q \
    > /tmp/kea4.log
    echo "total=$(wc -l < /tmp/kea4.log)" # the denominator, first
    echo "nosockets=$(grep -c DHCPSRV_NO_SOCKETS_OPEN /tmp/kea4.log)" # the numerator
    # want: total greater than 0, and nosockets 0.

    A zero with no denominator beside it is not an observation. journalctl exits 0 and prints nothing whether the daemon is clean or the query was wrong. --since parses the node’s local time, so pin both ends with TZ=UTC and --utc. Never count a journal’s errors with a bare grep -i error: Kea’s LFC_*_STATS INFO lines carry errors: 0. Match ERROR with its spaces.

  2. One lease from that leg’s own pool. Read the leases back with the control-socket command in Reading Kea lease state from a workstation. The lease’s subnet-id is this leg’s id from host_vars and its ip-address falls inside this leg’s kea_pool. "result": 3 on a leg armed minutes ago is abort trigger (a)’s clock, not a pass.

  3. One dynamic name in that leg’s carved zone, from the lease in step 2:

    Terminal window
    scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
    -a 'dig +short <hostname>.<leg>.lan.fzymgc.house A @127.0.0.1 -p 1053'
    # want: the leased address, on both nodes.
    scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
    -a 'dig +short -x <the leased address> @127.0.0.1 -p 1053'
    # want: the same name, on both nodes.

    Ask the authoritative tier on @127.0.0.1 -p 1053, never on :53. The main recursor process holds :53 and answers from its cache. The reverse half fails on its own: D2 sends forward and reverse as one transaction but reports them separately, so a leg whose reverse /24s are not carved answers the forward query and not the reverse one. Reservations in a deliberately uncarved /24 (main’s .20, core’s .40) take a DHCP_DDNS_NO_REV_MATCH_ERROR on their reverse half and keep answering from the covering /16. That is the carve working.

Cut:

  1. Take the pre-cut client reads: v4 address, lease (address, lease time, server identifier, option surface), v6 address and resolvers. Verbatim.
  2. Set dhcp_server.enabled = false for that network in tf/unifi/networks.tf, and let main-cluster-unifi apply it.
  3. Prove the static management path with DHCP off on that leg: dispatch to both resolver nodes and confirm both answer. Its failure is abort trigger (c).
  4. Arm the non-VIP-holder, then the VIP holder, with the in-window command above.
  5. Confirm each daemon opened a socket for the leg’s interface and its journal carries no DHCPSRV_NO_SOCKETS_OPEN line.

Verify:

  1. Force a renew on the instrumented client and re-read the discriminating fields.
  2. Read the leases back from the pair through the control-socket command.
  3. Agreement between the client’s own view and the server’s lease is the evidence.
  4. Re-read the v6 address and resolvers. They must be unchanged.

Roll back, the same steps whether this is the drill or an abort:

  1. Disarm Kea on both nodes with the explicit served-legs form, and confirm both daemons are down before the gateway changes.
  2. On main, guest and telework, put the leg’s .1 on the trusted list of UniFi DHCP Guard for that network first, in tf/unifi/networks.tf. The list holds at most three addresses, so .1 replaces the VIP .5.
  3. Set dhcp_server.enabled = true for that network, and let main-cluster-unifi apply it.
  4. Force a renew on the instrumented client.
  5. Confirm by lease origin, not by address range. Read the valid-lft, and make sure that the lease no longer appears in Kea’s own lease4-get-all on both nodes.
  6. Re-read the v6 address and resolvers. The v6 half never changes, so anything that moved here is a finding.

Re-cut:

  1. Set dhcp_server.enabled = false again.
  2. Arm both nodes without the lease-instrument extra-vars, so the leg lands on the committed steady-state lease.
  3. Verify again, including that the on-node rendered lease time is the steady-state value.
  4. Commit the leg’s arm values to group_vars the same day, then run one bare converge with no extra-vars and confirm Kea is serving. That is the only reading that proves the steady state is self-sustaining.

The DDNS tier — how a DHCP lease becomes a name, and how each half fails

Section titled “The DDNS tier — how a DHCP lease becomes a name, and how each half fails”

Read this section before triaging a missing name. Kea D2 writes the A half and the PTR half as two separate updates to two different zones, and each half fails on its own.

The forward chain (A and PTR) is Kea’s, end to end. The A half and the PTR half are two separate updates, to two different zones, accepted by two different grants:

  1. kea-dhcp4 grants or renews a lease and builds an FQDN from the client’s hostname plus the leg’s own ddns-qualifying-suffix (lab.lan.fzymgc.house. for Lab, derived from the leg name so a client cannot mint a name outside its own VLAN).
  2. It emits a Name Change Request over the dhcp-ddns channel to 127.0.0.1:53001, this node’s own kea-dhcp-ddns (D2). The NCR channel never crosses the wire.
  3. D2 matches the FQDN against its configured forward domain, turns the request into RFC 2136 DNS UPDATE signed with that leg’s own kea-d2-<leg> TSIG key, and sends it to PowerDNS on 127.0.0.1:1053.
  4. The forward update targets <host>.<leg>.lan.fzymgc.house in the leg’s own child zone, which grants DNS UPDATE to that leg’s key alone. The parent lan.fzymgc.house grants nobody.
  5. The reverse update is a second update to a different zone, signed with the zone-wide kea-d2-lan key, and it lands in the leg’s own carved /24. Each carve is NS-delegated from the covering /16 and grants DNS UPDATE to kea-d2-lan alone.

168.192.in-addr.arpa is writable by nobody. The covering /16 carries the curated PTRs, so it carries no TSIG-ALLOW-DNSUPDATE and no ALLOW-DNSUPDATE-FROM. A reverse update that lands there is REFUSED, and that is correct. The carve boundary is drawn at the leg, so widening a pool needs no zone change. A leg whose pool spans a /24 that is not carved takes a DHCP_DDNS_NO_REV_MATCH_ERROR on every lease there, and those addresses keep answering in reverse from the covering /16.

If dig -x on a lease address returns nothing, first check whether the address is inside a carved zone at all:

Terminal window
# On a resolver node.
pdnsutil zone list-all | grep in-addr.arpa
pdnsutil metadata get 200.168.192.in-addr.arpa TSIG-ALLOW-DNSUPDATE
# want: TSIG-ALLOW-DNSUPDATE = the zone-wide kea-d2-lan key at the granted generation
pdnsutil metadata get 168.192.in-addr.arpa TSIG-ALLOW-DNSUPDATE
# want: EMPTY. The covering /16 grants nobody, and this read is the control for the one above.
journalctl -u isc-kea-dhcp-ddns-server | grep -E 'RCODE|Transaction outcome'

pdnsutil prints multi-valued metadata comma-joined on one line. Split on the comma before concluding a grant names one writer.

Reverse Change: yes on a DHCP_DDNS_REMOVE_SUCCEEDED line means the reverse half did not error. It does not mean a PTR was withdrawn, because deleting an absent RRset is a success in RFC 2136. The instrument is the reverse zone’s SOA serial, which PowerDNS advances only on an UPDATE that changes records:

Terminal window
pdnsutil zone list 200.168.192.in-addr.arpa | grep -E '[[:space:]]SOA[[:space:]]'
# want: the serial field moves only across an update that changed records

The carves are real /24 zones and not RFC 2317. RFC 2136 resolves the target zone by walking the ancestor labels of the name being updated, and an update never follows CNAME indirection. D2 has no RFC 2317 awareness. TSIG-ALLOW-DNSUPDATE is per zone, so a narrower boundary has to be a narrower zone.

No program writes AAAA records into a leg’s zone. Each leg’s zone has one writer and one key:

Terminal window
pdnsutil metadata get lab.lan.fzymgc.house TSIG-ALLOW-DNSUPDATE
# want: kea-d2-lab<gen>, the granted generation's spelling
pdnsutil metadata get lan.fzymgc.house TSIG-ALLOW-DNSUPDATE
# want: an empty value. The parent grants nobody.

An empty parent reading is the correct state. The grant governs RFC 2136 dynamic updates only, and dns-control and external-dns write over the HTTP API.

Where the boundary is, and what enforces it

Section titled “Where the boundary is, and what enforces it”

The curated apex fzymgc.house accepts no DNS UPDATE, and the reason is not that it carries no metadata:

Terminal window
pdnsutil metadata get fzymgc.house TSIG-ALLOW-DNSUPDATE
# want: an empty value

PowerDNS’s authorisation logic falls back to the global allow-dnsupdate-from range when a zone names no per-zone source. What closes the apex is configuration:

Terminal window
grep -E '^(dnsupdate|dnsupdate-require-tsig|allow-dnsupdate-from)' /etc/powerdns/pdns.conf
# want:
# dnsupdate=yes
# dnsupdate-require-tsig=yes
# allow-dnsupdate-from=127.0.0.0/8,::1

dnsupdate-require-tsig=yes makes the absence of a TSIG grant a REFUSED, regardless of source address. The global range is loopback-only as a second, independent layer. Drop the first and loopback processes can write the apex. Drop the second and the only thing between the wider network and a signed update is key custody.

Rotation is by key name, with a server-side overlap, never by replacing the secret behind a name the writers use. Every key carries a generation: the zone-wide kea-d2-lan and the five per-leg kea-d2-<leg> keys, six names at each generation. Generation 1 is the unsuffixed spelling (kea-d2-lab, Vault field secret_lab). A generation n > 1 renders -<n> on the name (kea-d2-lab-2) and _<n> on the Vault field (secret_lab_2). Two values choose the generation:

  • dns_tsig_key_generation in ansible/inventory/group_vars/dns_hosts.yml: the generation Kea D2 signs with.
  • dns_tsig_key_generations_granted in the same file, and the TSIG-ALLOW-DNSUPDATE grant of each dynamic zone in dns-control: every generation the PowerDNS store holds and every grant names. The two must be equal as sets.

dns-control zones grant --generations <list> sets the grant of every dynamic zone in one command. For each zone, the new grant names each Kea D2 key base of the zone at each generation in the list. A key that is not a Kea D2 key leaves the grant. The command plans and applies one change per zone that differs, through the change workflow, and prints no change for the others. --dry-run prints the same lines and writes nothing.

The Vault side is fields of one entry, fzymgc-house/infrastructure/dns/kea-d2-tsig: secret_<n> and secret_<leg>_<n>. No new path and no new policy grant.

This procedure gets its own window. A rotation gone wrong and a cut gone wrong have the same symptom, names that stop appearing.

The two latencies: zone-metadata-cache-ttl=60 is the documented upper bound for consumers that read metadata through the cache, and the RFC 2136 grant check does not. LightningStream replication is the latency that governs. dns-control writes a grant through the writer node’s API, and the other node receives it on the order of a minute later. A writer that signs against the VIP holder during that window is REFUSED on one node and accepted on the other. The role does not sleep for either. --tags pdns reads the grant with an until over pdnsutil metadata get on the node being converged (roles/pdns/tasks/verify.yml) until the parsed set equals the expected one.

The order:

  1. The Vault fields, from the control node. One value per field, minted on its own. Use vault kv patch, never vault kv put, which replaces the entry and drops the fields the current generation signs with:

    Terminal window
    for leg in core lab main iot mainwireless; do
    vault kv patch -mount=secret fzymgc-house/infrastructure/dns/kea-d2-tsig "secret_${leg}_<n>=$(openssl rand -base64 32)"
    done
    vault kv patch -mount=secret fzymgc-house/infrastructure/dns/kea-d2-tsig "secret_<n>=$(openssl rand -base64 32)"
    vault kv get -mount=secret -format=json fzymgc-house/infrastructure/dns/kea-d2-tsig | jq -r '.data.data | keys[]'

    The jq line reads field names only. Want the current generation’s fields beside the new ones.

  2. PR A, widen. Set dns_tsig_key_generation: <n> and dns_tsig_key_generations_granted: [<n-1>, <n>] in dns_hosts.yml, and merge it. Then widen the grants and converge the authoritative tier, both nodes in one run:

    Terminal window
    kubectl -n dns-control exec deploy/dns-control -- dns-control zones grant --generations <n-1>,<n> --dry-run
    kubectl -n dns-control exec deploy/dns-control -- dns-control zones grant --generations <n-1>,<n>
    scripts/dns-playbook.sh --tags pdns

    The dry run shows each grant gaining the -<n> names. Widen the grants before the converge, because the until of --tags pdns waits for the widened set. Use one converge and not two --limit runs. The store read and the import are run_once on the first host, and the grant reads are per host, so the until on the second host is what proves the peer received both the grant and the imported keys. Kea D2 must not converge until --tags pdns has passed on both resolvers.

  3. The writer, straight to <n>:

    Terminal window
    scripts/dns-playbook.sh --tags kea

    D2 restarts onto the new names in one converge, and its per-key counters start from zero under the new key objects. Take the reads in the next subsection before going on.

  4. PR B, narrow. Set dns_tsig_key_generations_granted: [<n>] and merge it. Then narrow the grants to the -<n> names alone and converge once more:

    Terminal window
    kubectl -n dns-control exec deploy/dns-control -- dns-control zones grant --generations <n>
    scripts/dns-playbook.sh --tags pdns

    The run_once pdnsutil tsigkey delete removes the previous generation’s six keys from the store, and d2-tsig-key-present proves the store equals the declared set. The previous generation’s .tsig files stay on the Kea nodes until the next --tags kea converge removes them.

Never narrow before the writer has converged. That is withdraw-then-converge: narrowing the grant while D2 signs with the previous generation REFUSES every one of its updates until it converges, and the refusal reads like a zone problem hours later with D2 reporting success.

Reading the keys and the grants on-box. pdnsutil tsigkey list prints the secret as its third field. Never paste its raw output anywhere:

Terminal window
# On a resolver node.
pdnsutil tsigkey list | cut -d " " -f1 | sort
# want (overlap): twelve names, six of them ending -<n>; want (narrowed): the six -<n> names
pdnsutil metadata get lab.lan.fzymgc.house TSIG-ALLOW-DNSUPDATE
# want (overlap): kea-d2-lab at <n-1>, kea-d2-lab-<n>
# want (narrowed): kea-d2-lab-<n>
pdnsutil metadata get 200.168.192.in-addr.arpa TSIG-ALLOW-DNSUPDATE
# want (overlap): kea-d2-lan at <n-1>, kea-d2-lan-<n>
# want (narrowed): kea-d2-lan-<n>
sudo /opt/dns-resolver/venv/bin/python -c 'import json, socket; s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM); s.connect("/run/kea/kea-ddns-ctrl-socket"); s.sendall(json.dumps({"command": "statistic-get-all"}).encode()); s.shutdown(socket.SHUT_WR); data = b"".join(iter(lambda: s.recv(65536), b"")); stats = json.loads(data)["arguments"]; print({k: v[0][0] for k, v in stats.items() if k.startswith("key[") and k.endswith("update-success")})'
# want (after step 2, once a lease event has happened): a positive count under a key[...-<n>...] name

The statistic is the reading that names the key. Neither PowerDNS’s success line nor D2’s Transaction outcome line names it. dns-control shows the stored grant of a zone with GET /api/zones/<name> or the MCP tool dns_get_zone.

The break-glass, same-name path is destructive and kept for the case the generation path cannot cover, when the secret behind a name must change without a generation bump:

Terminal window
pdnsutil tsigkey delete <name>

on one node only, then wait for replication, confirm absence on both nodes with the name-only list above, then converge --tags pdns. The absence guard fires and re-imports the name from whatever Vault holds. Every UPDATE signed with it is REFUSED for the whole interval, on both nodes, and nothing announces it.

“Refused” means DNS-refused, and never lease-refused

Section titled ““Refused” means DNS-refused, and never lease-refused”

A client that loses a DNS name conflict keeps its address, its gateway, its subnet mask and its resolvers. It has no forward record. DDNS sits downstream of leasing: a DNS failure of any kind degrades nothing about DHCP. Confirm DHCP health from the lease table, not from anything in this section.

A client that supplies a multi-label hostname gets no name

Section titled “A client that supplies a multi-label hostname gets no name”

Kea appends the leg’s ddns-qualifying-suffix only to a single-label hostname. A client that sends a hostname with dots is treated as fully qualified. The FQDN then falls outside lan.fzymgc.house, D2 matches no configured forward domain, and discards the request without contacting PowerDNS. A client sending nas.fzymgc.house produces DHCP_DDNS_NO_MATCH No DNS servers match FQDN nas.fzymgc.house. followed by DHCP_DDNS_NO_FWD_MATCH_ERROR.

What you look at What it says What is true
The lease (lease4-get-all) "fqdn-fwd": true Kea asked for a forward update. It says nothing about whether one happened
journalctl -u isc-kea-dhcp4-server clean The discard happens in D2, a different daemon in a different unit
--tags pdns-verify and kea-verify all green They assert configuration. Neither observes a published name
The zone no record Correct. The only error anywhere is in D2’s journal

This is a property of the naming policy. The same mechanism stops a Lab client asserting a name in the curated apex.

A name D2 missed comes back on the client’s next renewal

Section titled “A name D2 missed comes back on the client’s next renewal”

ddns-update-on-renew is true on both nodes. A renewal emits a Name Change Request even when neither the FQDN nor the DHCID changed, so a name D2 missed while it was down comes back on the client’s next renewal, within kea_renew_timer (21600 s in ansible/roles/kea/defaults/main.yml). A client release and re-acquire is the way to go faster. Three costs come with this:

  • A renewal queues two Name Change Requests, a CHG_REMOVE for the old lease and a CHG_ADD for the new one, so every renewing client’s name has a brief absence window once per renew interval. A resolver that queries inside it caches NXDOMAIN for up to the zone’s 60 s SOA minimum. This is accepted, because shortening the zone’s negative TTL would shorten every negative answer.
  • Record deletion is not durable. A name deleted from the zone by hand returns on the client’s next renewal. To make a name stay gone, take the lease away, not the record.
  • A claim refused by a seeded reserved label fails twice per renewal, per offending client. That is recurring noise in D2’s journal by design.

There is no reconciler. A reconciler scoped to everything would read a failed lease read as an empty desired state and reap the zone.

Reading the journal when a claim is refused. Both identifiers are LOG_ERROR, so both appear at the default kea_log_severity:

Terminal window
# On a resolver node.
journalctl -u isc-kea-dhcp-ddns-server --since '-1h' \
| grep -E 'DHCP_DDNS_REMOVE_FAILED|DHCP_DDNS_FORWARD_REPLACE_REJECTED'

An empty journal is a third answer. If nothing reached D2, Kea skipped the request before it was queued. It says so as DHCPSRV_QUEUE_NCR_SKIP, which is LOG_DEBUG and invisible at the default severity. Raise kea_log_severity on the node before concluding that nothing happened.

Read the total and the match in the same command, and treat a zero total as “the query was wrong”:

Terminal window
# On a resolver node. -o cat -q makes an empty result count 0 rather than 1 line of "-- No entries --".
SINCE='<the window start, as YYYY-MM-DD HH:MM:SS UTC>'
sudo TZ=UTC journalctl --utc -u isc-kea-dhcp-ddns-server --since "$SINCE" --no-pager -o cat -q \
> /tmp/d2.log
echo "total=$(wc -l < /tmp/d2.log)" # the denominator, first
echo "errors=$(grep -c DHCP_DDNS_NO_FWD_MATCH_ERROR /tmp/d2.log)" # the numerator

The unit is isc-kea-dhcp-ddns-server.service, and every line it emits is tagged kea-dhcp-ddns[<pid>]. journalctl -u kea-dhcp-ddns matches no unit and prints -- No entries -- with exit 0. --since parses the node’s local time, so pin both ends with TZ=UTC and --utc.

When a name is missing after a D2 outage: first confirm it is absent (pdnsutil zone list <leg>.lan.fzymgc.house on the node, and resolve the name at both daemons), and read D2’s journal for the transaction. Then decide whether you need it back sooner than the renewal cadence. If you do, have the client release and re-acquire its lease. A lease that never renews never heals. That is a device or IPAM question, not a carve defect.

The two verify tags, each answering a different question:

Terminal window
scripts/dns-playbook.sh --limit <node> --tags pdns-verify
scripts/dns-playbook.sh --limit <node> --tags kea-verify
Tag Answers Does not answer
pdns-verify Are the granted generations’ keys in this node’s key store? Do the carved zones grant exactly their writers, and do the parent dynamic zone, the covering /16 and the apex grant nobody? Is the running daemon armed for DNS UPDATE? Whether any update was ever accepted
kea-verify Is the D2 config 0640 root:_kea, is its TSIG secret readable as the daemon, does the config parse? Whether D2 is running, and whether a signed update is accepted

Neither tag observes a published name. To know whether names are being published, read the zone.

A configuration file is not a running process. PowerDNS reads its configuration only at start. A converge that writes pdns.conf and then fails before its handlers flush leaves the file correct and the process unchanged, and every later converge reports ok and never re-notifies the restart. The reading that tells a file from a process:

Terminal window
# On a resolver node.
pdns_control current-config | grep dnsupdate
systemctl show pdns -p ExecMainStartTimestamp
stat -c '%y %n' /etc/powerdns/pdns.conf

A commented # dnsupdate=no is how pdns reports a setting left at its default, and it means not armed. The armed reading is an uncommented dnsupdate=yes with an uncommented dnsupdate-require-tsig=yes. A process start timestamp older than the config file’s mtime means the running daemon is not running the file you read. The fix is systemctl restart pdns on that node. The running-dnsupdate-armed criterion in ansible/roles/pdns/tasks/verify.yml is the one criterion in the tier that reads the running daemon. When a gate is green and the live estate is red, the first hypothesis is the instrument.

The zone itself:

Terminal window
pdnsutil zone list lab.lan.fzymgc.house
pdnsutil zone list lab.lan.fzymgc.house | grep -c -E "[[:space:]]IN[[:space:]]+A[[:space:]]"

pdnsutil zone list emits tab-separated records. A pattern with a literal space matches nothing. Do not wrap a counting command in || true: grep -c exits 1 on zero matches, and that exit is the signal.

The key store prints secrets. Reduce to names on the box:

Terminal window
pdnsutil tsigkey list | awk '{print $1}'
# want: the six names of the granted generation

Is D2 alive and listening:

Terminal window
systemctl is-active isc-kea-dhcp-ddns-server
ss -lnup | grep 53001
journalctl -u isc-kea-dhcp-ddns-server --since '-1 hour' --no-pager

The unit is isc-kea-dhcp-ddns-server.service, and the package is isc-kea-dhcp-ddns. dpkg -l isc-kea-dhcp-ddns-server reports Installed: (none) on a node where D2 runs.

The silent degradation: D2’s queue limit

Section titled “The silent degradation: D2’s queue limit”

kea-dhcp-ddns accepts NCRs into a bounded queue, max-queue-size (kea_ddns_max_queue_size, 1024). When the queue is full, kea-dhcp4 suspends sending DDNS updates until the backlog clears. Leasing is unaffected. The only places this is visible are D2’s journal and the growing gap between leases and records.

A release carrying another client’s identifier can delete that client’s record. A DHCP Release is unauthenticated, so a host on the leg can send a Release with a victim’s client identifier and cause the victim’s name to be withdrawn. DHCID does not stop this, and no configuration available here does. The blast radius is a missing name for one client until its next renewal, and the attacker must already be on the leg. A name disappearing for no apparent reason has this candidate explanation.