Internal DNS (the resolver pair, and the gateway beside it)
The resolver pair sandiego and portland answers client DNS for every VLAN over IPv4 and over
IPv6. The UCG-Fiber gateway serves no DHCP. Its dnsmasq answers a client on a trusted leg that
carries a hand-set .1 nameserver, and forwards every query to the pair. It produces no curated
zone and redirects no DNS query. In an incident, read
Which tier answers each VLAN, and how to prove it first, then Things that will bite you.
A Firewalla appliance was the gateway until the cutover of 2026-10-01.
The pair has no ripgrep. Every command written to run on a resolver node uses grep. Commands that
run against this repository on a workstation use rg.
Which tier answers each VLAN, and how to prove it
Section titled “Which tier answers each VLAN, and how to prove it”| Client | Asks | Path | Answered by |
|---|---|---|---|
| Trusted VLAN (Core, Lab, Main, MainWireless, IoT), IPv4 | 192.168.<vlan>.5, the leg’s floating address |
on-link at layer 2 to the node that holds the VRRP sync group. The gateway is not in the path | the main recursor process pdns-recursor@main on that node. fzymgc.house goes to pdns at 127.0.0.1:1053, everything else to dnscrypt-proxy on 127.0.0.1:5300, which reaches a public resolver through a relay |
| Guest or Telework, IPv4 | 192.168.79.5 or 192.168.217.5 |
the same layer 2 path to the same node | the public recursor process pdns-recursor@public on that node, which answers every name from public DNS only. fzymgc.house and both reverse zones get NXDOMAIN from the withheld zone of the process, and no forwarder and no relay sees the name |
| Remote client over the tailnet, IPv4 | 100.102.168.37 (sandiego) or 100.124.42.55 (portland), each node’s own tailnet address |
no VIP and no subnet route in the path | the same main process on that node |
| Trusted VLAN (Core, Lab, Main, MainWireless, IoT), IPv6 | the leg’s v6 VIP on the resolver pair, learned as RDNSS from the RA of the gateway and from the RA of the pair | on-link | the same main process on that node |
| Guest or Telework, IPv6 | the leg’s v6 VIP, learned as RDNSS from the same two RAs | on-link | the public process on that node |
Trusted VLAN, a hand-set 192.168.<vlan>.1 |
the gateway | the gateway’s dnsmasq forwards the query to the Core VIP 192.168.40.5 |
the main process on the node that holds the Core VIP |
This table says nothing about DHCP. No device serves DHCPv6. Kea on the pair serves DHCPv4 on all seven legs.
The seven VLAN legs and their floating addresses
Section titled “The seven VLAN legs and their floating addresses”| VLAN | 802.1Q | Network | Floating address |
|---|---|---|---|
| Core | untagged native | 192.168.40.0/22 |
192.168.40.5 |
| Lab | 1000 | 192.168.200.0/22 |
192.168.200.5 |
| Main | 3000 | 192.168.20.0/22 |
192.168.20.5 |
| IoT | 3020 | 192.168.218.0/24 |
192.168.218.5 |
| MainWireless | 3001 | 192.168.24.0/22 |
192.168.24.5 |
| Guest | 3100 | 192.168.79.0/24 |
192.168.79.5 |
| Telework | 3200 | 192.168.217.0/24 |
192.168.217.5 |
The pair holds a floating address on all seven legs. ip -br addr on either node lists seven
VIPs, and this table accounts for all seven. Guest and Telework are dns_vlan_interfaces entries
that declare trusted: false, a kea_subnet_id and a kea_pool. Kea serves DHCPv4 on both. See
The one value an operator edits to change trust below.
Bringing a leg into service, and how to tell whether it is
Section titled “Bringing a leg into service, and how to tell whether it is”Declaring a leg in dns_vlan_interfaces does not serve it. Five consumers read that structure,
and each needs its own converge and its own adoption step. Check them one by one. A leg that is
short any one of them presents to a client as total DNS failure:
| Consumer | Check on the node | Adoption step |
|---|---|---|
| networkd legs | ip -br addr show | grep <vlan-id> |
networkctl reload. A converge writes the files only |
| nftables accepts | sudo nft list ruleset | grep -c '<iface>' |
the reload handler. Make sure that the dport 53 sets name every leg |
| recursor listeners | sudo ss -tlnup | grep '<leg-addr>:53'. On Guest or Telework the PID is systemctl show -p MainPID --value pdns-recursor@public |
a restart, through --tags pdns-recursor |
| keepalived VIPs | sudo grep -c '^vrrp_instance' /etc/keepalived/keepalived.conf |
never a converge alone. See Promotion |
| RA announcer | ActiveEnterTimestamp against the config mtime |
systemctl restart dns-common-ra-announce. There is no reload path |
Kea, both router advertisements and the gateway point at the VIP, not at a node address, so keepalived is the consumer that decides whether a client sees anything at all.
systemctl is-active cannot tell you that a daemon matches its configuration. It reports that a
process exists. Compare ActiveEnterTimestamp against the config file’s mtime. A unit can be active
while it runs a configuration older than the file on disk.
Probe an on-segment resolver from the segment. The UCG-Fiber redirects no DNS query, so a query
from another subnet is routed to the address it names. It then measures the routed path, the zone
firewall and the node ruleset, and not what a client on the segment sees. Keep a control query to
an address on that subnet that does not exist, and treat ip neigh show <addr> reading
INCOMPLETE as the negative.
How the gateway stays out of the path
Section titled “How the gateway stays out of the path”192.168.x.5 is on-link for a client on that VLAN, so a query to the floating address is switched
at layer 2 and never reaches the gateway. The pair is in the path because every leg advertises the
floating address as its DNS server: Kea’s option 6 over IPv4, and RDNSS over IPv6. The gateway
holds no DNS redirect, no ad blocking and no content filter.
Proving which tier answered, in one query
Section titled “Proving which tier answered, in one query”whoami.dns.invalid is the instrument. Each recursor process serves the zone dns.invalid. from
a file, and whoami in it holds the Core address of the node. The main process answers that
address, which shows which node answered. The public process answers NODATA, because its
rebinding zone drops the private address, so take this reading on a trusted leg. It does not show which address the
client asked, because the gateway forwards every name to the pair: a query to 192.168.x.1
returns the same node address as a query to the VIP.
Take the query from a host on that VLAN. A query from anywhere else is routed.
# From a host holding an address on that VLAN. On a resolver node, -b selects the leg.dig -b 192.168.24.7 @192.168.24.5 whoami.dns.invalid +short # want: the VIP holder's Core addressdig -b 192.168.24.7 @<unused address on the leg> whoami.dns.invalid +short +time=2 +tries=1 # CONTROL, want: a timeoutBoth readings are required. The control that times out is what makes the answer attributable.
The discriminator makes sure that the recursor answers a query from that source. It does not show what the network advertises. The DHCP OFFER shows the advertisement, and the discriminator shows that the recursor answers the floating address. Both are needed, and they are different claims.
Clients that ask the gateway
Section titled “Clients that ask the gateway”A client that carries a hand-configured 192.168.x.1 nameserver asks the gateway. The gateway’s
dnsmasq forwards every query to the Core VIP 192.168.40.5. A client on a trusted leg gets a
correct answer, with the internal view and the public lists of the gateway’s source address and
not of its own leg. The query row names that address as the client, not the device. The source
address is not measured. 192.168.40.1 is the likely one, because the VIP is on-link on Core. A
Guest or Telework client gets no reply from the gateway, because a zone policy blocks DNS from
each of the two legs to it.
The remedy is hand work: find each statically configured client and repoint it to the VIP of its leg. That is not repository work.
The tailnet path
Section titled “The tailnet path”Remote clients resolve .house at each node’s own tailnet address, not at a floating address:
sandiego 100.102.168.37, portland 100.124.42.55. The coordination server assigns them and
nothing in this repository guarantees them. They share no pattern. Do not tidy them into one.
Three artifacts carry this path. Removing any one of them takes remote .house resolution with it,
including the recovery path the auto-revert leans on:
| Artifact | Where | Scope |
|---|---|---|
| the nftables accept | roles/dns-common/templates/nftables-dns.conf.j2, guarded by dns_common_tailnet_dns_enabled |
iifname the tailnet device and ip saddr 100.64.0.0/10, udp and tcp, each with a counter |
| the client-tier listener | roles/pdns-recursor, derived from that host’s own dns_tailnet_address into the listen list of the main process |
one address per node, its own. Never the peer’s and never a wildcard |
| the tailnet policy | tailscale/policy.hujson, applied by its own GitHub Action |
grants port 53 on tag:ns and nothing wider |
The tailnet’s own access-control policy is the authorization tier. The node ruleset is a narrowing.
100.64.0.0/10 is every tailnet address there is, so the source scope excludes a spoofed
non-tailnet source on the tunnel and nothing more. Narrowing which peers reach port 53 is a
tailscale/policy.hujson change.
There is no subnet route in this path and no floating address. The tailnet device is not in
dns_vlan_interfaces, because that structure has five consumers and a sixth pseudo-leg would
produce a VLAN device, a floating address and a VRRP instance electing over a tunnel.
The client view: which clients see the internal fzymgc.house, and what the others see
Section titled “The client view: which clients see the internal fzymgc.house, and what the others see”Each resolver node runs two PowerDNS Recursor processes from one unit template,
pdns-recursor@.service. Until 2026-10-14 the tier was Blocky, in two processes with the same
split.
pdns-recursor@main.servicelistens on the trusted legs, the tailnet and127.0.0.1. It forwardsfzymgc.houseand both reverse zones to pdns-auth on127.0.0.1:1053, with no fallback, and every other name todnscrypt-proxyon127.0.0.1:5300. It answers NXDOMAIN for the whole offzymgc.houseand both reverse zones to a client on no declared leg. It blocks the public lists that each trusted leg declares. See The public blocklists. Its configuration is/etc/powerdns/recursor-main.ymland the include directory/etc/powerdns/recursor-main.d/.pdns-recursor@public.servicelistens on the untrusted legs (Guest and Telework) and127.0.0.3. It has no forward zone to pdns-auth and loads no zone from it. Its withheld zone answers NXDOMAIN forfzymgc.houseand both reverse zones before any forwarder sees the name, the public names of the zone included. Its rebinding zone drops any answer that carries a private address. Every other name goes todnscrypt-proxy. It blocks the public lists that Guest and Telework declare. Its configuration is/etc/powerdns/recursor-public.yml.
Every block rule and allow rule of both processes is a response policy zone (RPZ). An RPZ is a
DNS zone whose records are rules: the owner name is the trigger, and the data is the action. A
block is CNAME . and answers NXDOMAIN. An allow is CNAME rpz-passthru. and lets the name
resolve. A process consults its zones in a fixed order, and the first zone that matches a name
decides the answer. Which zones a client gets, and in which order below lists them.
The trusted attribute on each dns_vlan_interfaces entry selects the process. The decisions
are Guest and Telework resolve public DNS only
and, for the main process,
Untrusted clients lose the whole fzymgc.house apex.
To talk to a process, name its configuration and its socket directory:
# On a resolver node, as root or as the user dns-worker. <p> is main or public.rec_control --config-dir=/etc/powerdns --config-name=<p> --socket-dir=/run/pdns-recursor-<p> rpzstatisticsEach process also serves an HTTP API on loopback: 127.0.0.1:8082 for main and
127.0.0.1:8083 for public. The key is in /etc/powerdns/recursor-api.key (root, mode 0600),
with a copy for the node worker at /etc/dns-node-worker/recursor-api.key. /metrics on each
API port serves the Prometheus metrics of the process with no key.
Who is exempt, and why each member is a member
Section titled “Who is exempt, and why each member is a member”The exemption is by client source address. The Lua script /etc/powerdns/policy.lua reads the
leg map of its process, /etc/powerdns/legs-main.map or legs-public.map. The map holds one
line for each leg network, with the list groups of the leg. A client whose source address is in a
line of the map of the main process is on a declared leg, so the withheld zone does not apply to
it. A client on no line gets the withheld zone and the groups of dns_default_list_groups.
| Member | Lines in legs-main.map |
Why it is a member | Where the value comes from |
|---|---|---|---|
| Core leg | 192.168.40.0/22, fddb:f665:73f7:6::/64 |
the core entry declares trusted: true |
the trusted attribute on that entry in ansible/inventory/host_vars/{sandiego,portland}.yml |
| Lab leg | 192.168.200.0/22, fddb:f665:73f7:4::/64 |
the lab entry declares trusted: true |
same |
| Main leg | 192.168.20.0/22, fddb:f665:73f7:1::/64 |
the main entry declares trusted: true |
same |
| IoT leg | 192.168.218.0/24, fddb:f665:73f7:3::/64 |
the iot entry declares trusted: true |
same |
| MainWireless leg | 192.168.24.0/22, fddb:f665:73f7:7::/64 |
the mainwireless entry declares trusted: true |
same |
| keepalived health client | 127.0.0.1/32 |
dns-health.fzymgc.house sits under the withheld apex and is dug from loopback on each node. A withheld zone that catches it blackholes the health check on both nodes at once. The nftables rate meter also leaves loopback out, for the same reason |
pdns_recursor_health_client_address in ansible/roles/pdns-recursor/defaults/main.yml |
| the tailnet | 100.64.0.0/10 |
A tailnet client’s 100.x source is in no leg CIDR, so without this line the per-node tailnet :53 listener answers NXDOMAIN. This trusts every tailnet peer, the same posture the node ruleset takes. Narrowing it is a tailscale/policy.hujson change |
pdns_recursor_tailnet_cidr in ansible/roles/pdns-recursor/defaults/main.yml |
The loopback line and the tailnet line carry the groups of dns_default_list_groups.
Which zones a client gets, and in which order
Section titled “Which zones a client gets, and in which order”The zone list of a process is the precedence. The Lua script discards, for each query, every owned zone that is not in the zone set of the client, and the recursor applies the rest in this order:
| Order | Zone | Source | Holds | Who gets it |
|---|---|---|---|---|
| 1 | withheld.rpz.internal. |
a file that the role renders | fzymgc.house and both reverse zones, as NXDOMAIN |
every client of the public process, and a client of the main process on no declared leg |
| 2 | rebinding.rpz.internal. |
a file of the role | every private, ULA, loopback and link-local range, as NODATA | every client of the public process. The main process has no such zone |
| 3 | allow.rpz.internal. |
a file that the role renders | the owned zones of dns_list_allowlist and the names of dns_list_excepted_names, as rpz-passthru. |
every client of both processes |
| 4 | <scope>.scope.rpz.internal. |
pdns-auth on 127.0.0.1:1053, by zone transfer |
the entries of one scope of dns-control, in the order of the scopes | the members of the scope, on the main process |
| 5 | operator.rpz.internal. |
pdns-auth on 127.0.0.1:1053, by zone transfer |
the list entries of dns-control with no scope | every client of the main process |
| 6 | <group>.feed.rpz.internal. |
/var/lib/pdns-recursor-feeds/<group>.feed.rpz |
one zone for each public list group, in the order hagezi, tif, crypto, domains |
the clients of a leg that declares the group in dns_list_groups |
The order sets four rules. A client of the public process, and a client of the main process on no declared leg, never gets an internal name, and no such query leaves the process. An owned name or an excepted name always resolves for every other client. A scoped entry beats an entry with no scope. An operator entry beats every feed.
The main process spreads this list over three files, and the file order is the zone order:
| File | Holds | Written by |
|---|---|---|
/etc/powerdns/recursor-main.yml |
the withheld zone and the allow zone | the role |
/etc/powerdns/recursor-main.d/50-scopes.yml |
each scope zone | the node worker |
/etc/powerdns/recursor-main.d/90-tail.yml |
the operator zone and the feed zones | the role |
The scope map /var/lib/dns-worker/scopes.map names the members of each scope. The node worker
writes it: the line # version N first, then one <scope> <cidr> line for each member. The
public process has no scope map. The main process keeps a copy of each zone from pdns-auth in
/var/lib/pdns-recursor-main/, and starts from the copy when pdns-auth is down.
What Guest and Telework see
Section titled “What Guest and Telework see”Guest (3100) and Telework (3200) declare trusted: false, so the public process serves them. A
client on those legs gets what public DNS returns, with the internal zones withheld:
| Query | Answer |
|---|---|
fzymgc.house and every name under it, id.fzymgc.house included |
NXDOMAIN, from the withheld zone of the public process. The query never reaches dnscrypt-proxy |
a PTR under 168.192.in-addr.arpa or under 7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa |
NXDOMAIN, from the same zone |
whoami.dns.invalid |
NODATA. The identity zone holds the Core address of the node, and the rebinding zone drops the private answer |
health.dns.invalid |
NXDOMAIN, from the identity zone dns.invalid. of the process. The keepalived check asks for it |
| a public name whose answer is a private address | NODATA, from the rebinding zone |
| any other name | the public answer, through dnscrypt-proxy and a relay |
No internal address appears in an answer. If a guest device cannot reach an internal service by
name, that is working as designed. Until 2026-10-14 a Guest client got the public Cloudflare
addresses for id.fzymgc.house and REFUSED for an internal name, because Blocky forwarded the
zone to dnscrypt-proxy, which refused it.
The UCG-Fiber redirects no DNS query. A Guest or Telework query to the gateway gets no reply, because a zone policy blocks DNS from each of the two legs to the gateway. A query to a public resolver reaches that resolver and gets the public answer.
How to prove what an untrusted leg sees
Section titled “How to prove what an untrusted leg sees”Take each reading from a host on that leg. A reading from any other host is routed, and the zone firewall of the gateway decides it. Compare with DNS over HTTPS (DoH), which no device on the path can rewrite.
No workstation needs to be on the leg. The untrusted-leg-dns-check skill takes these readings from
a throwaway container that the nas-throwaway-lxc skill puts on the leg from the NAS. Both live
under .claude/skills/, and the scripts need a Vault session. The script of the skill also takes
--server <address>, which runs the answer readings against one server and skips the on-link
readings. The commands below are the same readings, for a host that is already on the leg.
# From a host on Guest. For Telework, replace 192.168.79 with 192.168.217 throughout.# Repeat the first and third lines against the v6 VIP, fddb:f665:73f7:2::5 (Telework ::5:5).dig @192.168.79.5 cloudflare.com A +short | sortcurl -s -H 'accept: application/dns-json' \ 'https://cloudflare-dns.com/dns-query?name=cloudflare.com&type=A' # want: the same addressesdig @192.168.79.5 id.fzymgc.house A +noall +comments | grep -c 'status: NXDOMAIN' # want: 1dig @192.168.79.5 dns-health.fzymgc.house A +short # want: no output, never a 192.168. addressdig @192.168.79.5 whoami.dns.invalid A +noall +comments +answer # want: NOERROR and no answer recorddig @<unused address on the leg> cloudflare.com +time=2 +tries=1 # decoy: want a timeoutdig @192.168.79.1 dns-health.fzymgc.house A +time=3 +tries=1 # hand-set gateway: want a timeout or no addressdig @1.1.1.1 dns-health.fzymgc.house A +short # hand-set public resolver: want no outputIf the decoy answers, something other than the pair answered the other readings. The reading for
whoami.dns.invalid tells the two processes apart. The main process answers the Core address of
the node, and the public process answers NODATA, because its rebinding zone drops that address.
An address in that answer means that the main process answered the leg, which is a finding.
The last two readings are for a client that sets its resolver by hand. The UCG-Fiber blocks DNS
from Guest and Telework to itself, so the query to .1 gets no reply, and the check script counts
that as a PASS. The query to 1.1.1.1 reaches the public resolver, which holds no address for the
name.
Public resolver failures
Section titled “Public resolver failures”This section covers the public recursor process, pdns-recursor@public. The alert message points
at this heading, so a rename of the heading also changes the message in
argocd/app-configs/clickstack-alerts/bootstrap-script.yaml.
The ClickStack alert DNS public resolver failing (Guest/Telework) fires when failed resolutions
reach 0.05 of the answers of the public process over 15 minutes. A failed resolution reaches the
client as SERVFAIL. The collector on each node scrapes pdns_recursor_servfail_answers and
pdns_recursor_questions from the API port of the public process, 127.0.0.1:8083, and copies
them to dns_resolver_errors_total and dns_resolver_responses_total under the ServiceName
dns-resolver-public. The alert reads the copies with the filter
ServiceName = 'dns-resolver-public', and the ratio sums both nodes.
The ratio grows with the duration of a failure, so a short upstream blip does not page. The
keepalived check on each node asks the public process for health.dns.invalid every 2 seconds,
and each answer counts in the denominator. So a stray error on a quiet leg does not move the
ratio. The keepalived check itself does not see an upstream failure, because the process answers
that name from its identity zone dns.invalid. with no upstream query. This alert is the signal
for an upstream failure on these legs.
To read the ratio, open the tile DNS public resolver failed resolutions per answer on the
Cluster — metric alerts dashboard in HyperDX.
To triage:
- Find the node that holds the untrusted VIPs:
ip -br addr | grep -E '192\.168\.(79|217)\.5/'. - On that node, read the SERVFAIL count of the public process:
rec_control --config-dir=/etc/powerdns --config-name=public --socket-dir=/run/pdns-recursor-public get servfail-answers. Read it twice, one minute apart. A count that moves is a live failure. - Compare with the main process. Run the same command with
mainin place ofpublic. If the main process fails too, the fault is the WAN,dnscrypt-proxyor its relays. Readjournalctl -u dnscrypt-proxyon that node and the alerts of The anonymizing upstream. - If only the public process on one node fails, restart it:
sudo systemctl restart pdns-recursor@public. keepalived moves the Guest and Telework VIPs only after 5 failed checks, 10 seconds, so a restart does not move them.
Query rows and pair metrics in ClickStack
Section titled “Query rows and pair metrics in ClickStack”Each recursor process sends one protobuf message for each answer to DNS-collector 3.1.0 on the
node, the unit dns-resolver-reader.service. The reader listens on 127.0.0.1:6000 for the main
process and 127.0.0.1:6001 for the public process. Its binary is /usr/local/bin/dnscollector
and its configuration is /etc/dns-resolver-reader/dnscollector.yml. It writes one JSON row for
each answer to its journal, and the collector otelcol-dns parses the row.
In default.otel_logs, a query row carries the ServiceName of the process that answered:
dns-resolver for the main process and dns-resolver-public for the public process. The name
comes from the field dnstap.identity of the row, which is the server_id of the process. The
daemon lines of pdns-recursor@main.service and pdns-recursor@public.service carry the same
two names, from the map dns_otel_recursor_log_service_names in
ansible/roles/dns-otel-collector/defaults/main.yml. The other lines of the reader carry
dns-resolver-reader. Every other unit keeps otelcol-dns. The node worker sends its own log
over OTLP with the ServiceName dns-node-worker, so the collector reads no journal of
dns-node-worker.service. Until 2026-10-14 the query rows carried the names blocky and
blocky-public.
The collector gives each query row the engine-neutral names that dns-control reads:
client.address, dns.question.name, dns.question.type, dns.response.code and
dns.answer. A row that a policy zone decided also gets dns.policy.zone, dns.policy.trigger,
dns.policy.hit and dns.policy.action. The zone is the policyName of the RPZ, such as
hagezi.feed.rpz.internal.. The trigger is the rule that matched, and the hit is the name that
matched it. The action is nxdomain for a block, nodata for the rebinding zone and passthru
for an allow. A row that no rule decided carries no policy attribute. The table in
ClickStack lists the names. The raw names of
the reader, such as dns.qname and powerdns.applied-policy, stay on the row beside them.
No row of dns-health.fzymgc.house or health.dns.invalid reaches ClickStack. The Lua script
of each process keeps the two names out of the protobuf, and the processor filter/health of the
collector drops a row that arrives anyway.
A row filter in argocd/app-configs/clickstack-chart/values.yaml hides the rows of these names
from mcp_readonly and mcp_readwrite. HyperDX and every other ClickHouse user read the rows.
If you change a name in the map, change the filter to the same name. Nothing checks that the two
are equal.
The collector scrapes three loopback endpoints every 60 seconds: the public process on
127.0.0.1:8083, the main process on 127.0.0.1:8082 and lightningstream on
127.0.0.1:8500. The job name is the metric ServiceName, so the two recursor jobs are
dns-resolver-public and dns-resolver. Each job passes only its own named series. The two
recursor series are pdns_recursor_questions, every query with the RD bit set, and
pdns_recursor_servfail_answers, every SERVFAIL the process gave. The collector copies them to
dns_resolver_responses_total and dns_resolver_errors_total under the same ServiceName.
dns-control and the ClickStack tiles read the copies.
The collector also takes OTLP over HTTP on 127.0.0.1:4318, for the node worker. Nothing off the
node can reach the port. The worker sends its traces, its logs and its metrics there, each with the
ServiceName dns-node-worker and the host.name of the node. Its three gauges are
dns_worker_client_cert_not_after_seconds, dns_worker_drained and dns_worker_drain_overdue. The traces and the logs of the receiver go through their own pipelines,
traces and logs/otlp, so each record keeps the ServiceName of its sender. The metrics of the
receiver join the scrape pipeline.
To make sure that the receiver forwards a trace, send one test span on the node. The command runs
on the control node and needs the Vault session of scripts/dns-adhoc.sh:
scripts/dns-adhoc.sh <node> -m ansible.builtin.uri -a '{"url": "http://127.0.0.1:4318/v1/traces", "method": "POST", "body_format": "json", "body": {"resourceSpans": [{"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "otlp-receiver-test"}}]}, "scopeSpans": [{"spans": [{"traceId": "0af7651916cd43dd8448eb211c80319c", "spanId": "b7ad6b7169203331", "name": "otlp receiver test", "kind": 1, "startTimeUnixNano": "{{ (now().timestamp() * 1000000000) | int }}", "endTimeUnixNano": "{{ (now().timestamp() * 1000000000) | int }}"}]}]}]}}'The result reads SUCCESS with "status": 200. Within a minute, this query returns one row for
the node:
SELECT ResourceAttributes['host.name'] AS node, TimestampFROM default.otel_tracesWHERE ServiceName = 'otlp-receiver-test' AND Timestamp > now() - INTERVAL 10 MINUTETo read the recent query rows of each node, run this query as the ClickHouse user app, the
user that HyperDX reads with:
SELECT ResourceAttributes['host.name'] AS node, count(), any(LogAttributes['dns.question.name']), any(LogAttributes['dns.policy.zone'])FROM default.otel_logsWHERE Timestamp > now() - INTERVAL 10 MINUTE AND ServiceName = 'dns-resolver'GROUP BY nodeWhat any other source sees on the main process
Section titled “What any other source sees on the main process”A source on no line of legs-main.map receives NXDOMAIN, not NOERROR with an empty answer, for:
fzymgc.houseand every name beneath it, for every query type. The withheld zone holds the apex and a wildcard under it asCNAME ., and an RPZ rule does not branch on question type.- every PTR under
168.192.in-addr.arpaand under7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa, so internal hostnames cannot be walked in reverse.
No untrusted leg reaches the main process through a listener. The withheld zone is the fall-through for a source that the leg map does not name.
The one value an operator edits to change trust
Section titled “The one value an operator edits to change trust”The trusted boolean on the leg’s own dns_vlan_interfaces entry, in both
ansible/inventory/host_vars/sandiego.yml and ansible/inventory/host_vars/portland.yml. That is
the estate’s single declaration of leg trust. It also selects the recursor process that listens
on the leg, and the leg map that names the leg. It is not kea_served_legs. Serving DHCP on a leg
and handing that leg the internal namespace are different claims.
Write a real YAML boolean, not a quoted string. trusted: "true" is a non-empty string and is
truthy, so it grants trust while it reads like a value under review. Every entry must carry the
key. An entry without it fails the render.
trusted does not decide DHCP service. A leg that declares kea_subnet_id gets a Kea subnet,
trusted or not.
The leg map, and how to read it
Section titled “The leg map, and how to read it”The template ansible/roles/pdns-recursor/templates/legs.map.j2 renders one map for each
process. Each line holds one network and the list groups of its leg, separated by commas, or -
for a leg with no group. A trusted leg gives two lines to legs-main.map, its IPv4 network and
its ULA /64. An untrusted leg gives the same two lines to legs-public.map. The main map ends
with the loopback line 127.0.0.1/32 and the tailnet line 100.64.0.0/10, both with
dns_default_list_groups.
The map decides two things for a client of the main process: whether the withheld zone applies,
and which feed zones apply. A client on no line gets both the withheld zone and the default
groups. A leg that is in dns_vlan_interfaces and not in the map is a render fault, because the
role derives the map from that structure. Read the map on the node:
# On a resolver node.cat /etc/powerdns/legs-main.map /etc/powerdns/legs-public.mapThe Lua script reads the map once, at load. After a change, the role restarts the process. A hand edit of the map on the node is lost on the next converge.
The public blocklists
Section titled “The public blocklists”Both processes refuse the names on public blocklists with NXDOMAIN. Each list group is one feed
zone, <group>.feed.rpz.internal., from the file /var/lib/pdns-recursor-feeds/<group>.feed.rpz
on each node. Each leg declares its list groups in dns_list_groups on its dns_vlan_interfaces
entry, in both host_vars files. The main process applies them on the trusted legs, and the public
process on Guest and Telework. A client whose address is on no declared leg gets
dns_default_list_groups. Until 2026-10-14 the attribute was blocky_list_groups.
| Group | Source | Entries |
|---|---|---|
hagezi |
HaGeZi Pro++, https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/wildcard/pro.plus.txt |
about 222,000 |
tif |
HaGeZi Threat Intelligence Feeds, medium tier, https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/wildcard/tif.medium.txt |
about 694,000 |
crypto |
the Block List Project crypto list, https://blocklistproject.github.io/Lists/alt-version/crypto-nl.txt |
about 1,300 |
domains |
an inline list of single domains and the two Private Relay names of Apple | 5 |
dns_list_sources in ansible/roles/pdns-recursor/defaults/main.yml defines the groups, once
for both processes. Write a domain as *.<domain> to block it and every subdomain. A plain entry
blocks only that exact name. One rule maps an entry to records, for a line of a feed and for a
list entry of dns-control: example.org becomes example.org CNAME ., and *.example.org
becomes example.org CNAME . and *.example.org CNAME ..
The domains group blocks mask.icloud.com and mask-h2.icloud.com as plain names. When the
resolver of a network answers NXDOMAIN for these two names, Apple devices turn off iCloud Private
Relay on that network. While Private Relay is on, Safari sends its DNS queries through the relay
of Apple, and no list of this resolver applies to them. Apple documents the method in Prepare
your network or web server for iCloud Private Relay. To see which devices ask for Private Relay,
search the query log of dns-control for mask.icloud.com.
The owned zones fzymgc.house, holomush.dev, fzymgc.net and hl-assets.dev are in
dns_list_allowlist. The allow zone allow.rpz.internal. holds them as rpz-passthru. rules,
and it comes before every entry and every feed, so no list blocks an owned name. The withheld
zone comes before the allow zone, so the rules of The client view above do not change.
dns_list_excepted_names in the same file holds third-party names that no list blocks, each with
every name under it. The allow zone holds them beside the owned zones. To except a name that a
list blocks, add *.<name> there and converge --tags pdns-recursor one node at a time. An
entry excepts only the name it says, so *.s1.sentry-cdn.com leaves the rest of
sentry-cdn.com blocked.
The timer dns-feeds.timer starts dns-feeds.service once a day, at 03:30 on sandiego and at
04:30 on portland, so the two nodes never reload at the same time. The unit runs
dns-node-worker feed for each group with a URL source. The command downloads the source,
converts it to RPZ, makes sure that the new zone holds at least half the entries of the zone it
replaces, and renames it into place. The role renders the zone of an inline group, such as
domains. After the last group, the unit runs rec_control reload-yaml for each process, which
reads every RPZ again, and then rec_control wipe-cache '.$', because a reload keeps the packet
cache and a cached answer outlives a new block until its TTL ends. Each rec_control call waits up
to 20 seconds, the value of pdns_recursor_control_timeout, because a reload of the main process
reads every feed zone and can take longer than the default of 5 seconds. A failed download keeps the
previous zone file in force, and the other groups still load. The unit then ends failed, and
journalctl -u dns-feeds names the group.
To read the feed state on a node:
# On a resolver node.systemctl list-timers dns-feeds.timer # the next and the last runsystemctl is-failed dns-feeds.service # want: inactive. failed means a group did not downloadls -l /var/lib/pdns-recursor-feeds/ # one .feed.rpz for each group, with its mtimerec_control --config-dir=/etc/powerdns --config-name=main --socket-dir=/run/pdns-recursor-main rpzstatisticsrpzstatistics prints one block for each zone of the process with its serial and its record
count. A feed zone with no records is a finding. To download the feeds now, run the action
Refresh lists on Health, or the MCP tool dns_refresh_lists of dns-control. The node worker starts
dns-feeds.service on each target node, the node that holds no VIP first, and waits up to 5
minutes until the unit ends. Each download stops after 60 seconds (pdns_recursor_feed_timeout),
and each rec_control call after 20 seconds (pdns_recursor_control_timeout), so the unit ends
inside that bound. A unit that ends failed fails
the action, and the detail names the error of systemctl. The unit drops every cached answer, so
dns-control reads the flush guard first and refuses the refresh while the upstream fails. Without
dns-control, run sudo systemctl start dns-feeds.service on one node.
Each node has 4 GB of memory and runs both processes, and each process holds its own copy of
every feed zone. Measure the memory of a new list before you add it. In the spike of the change
dns-policy-as-rpz, both processes together with every feed took 415 to 434 MiB after the load
and 1,103 to 1,151 MiB at the peak of a feed reload. Count about 370 bytes for each record in
each process. The list of newly registered domains on the Firewalla, 6.1 million entries, was not
moved for this reason.
To change what one leg blocks, edit its dns_list_groups in both host_vars files and converge
--tags pdns-recursor one node at a time. An empty list turns list blocking off for that leg.
To find why a name was blocked, run this query as the ClickHouse user app:
SELECT Timestamp, ServiceName, LogAttributes['client.address'] AS client, LogAttributes['dns.question.name'] AS name, LogAttributes['dns.policy.zone'] AS zone, LogAttributes['dns.policy.trigger'] AS triggerFROM default.otel_logsWHERE Timestamp > now() - INTERVAL 1 HOUR AND ServiceName IN ('dns-resolver', 'dns-resolver-public') AND LogAttributes['dns.policy.action'] = 'nxdomain'ORDER BY Timestamp DESCLIMIT 50dns.policy.zone names the zone that decided, for example hagezi.feed.rpz.internal., and
dns.policy.trigger names the rule that matched. To let one owned name through, add it to
dns_list_allowlist. To let a third-party name through, add it to dns_list_excepted_names. To
let a name through on one leg only, remove the group from that leg.
The list entries of dns-control
Section titled “The list entries of dns-control”The main process loads two kinds of zone from pdns-auth on 127.0.0.1:1053: the operator zone
operator.rpz.internal., with every list entry of dns-control that names no scope, and one zone
<scope>.scope.rpz.internal. for each scope of dns-control. The operator zone applies to every
client of the main process: every trusted leg, the tailnet and a client on no declared leg. A
scope zone applies to the members of the scope only. The public process loads no zone from
pdns-auth, so Guest and Telework keep the public lists alone.
dns-control writes the zones through the pdns HTTP API on the writer node, and LightningStream
copies them to the other node. A copied write sends no NOTIFY, so the node worker on each node
sends one to its main process for the operator zone and the scope zones, and the process fetches
the new serial by zone transfer. The main process keeps a copy of each zone in
/var/lib/pdns-recursor-main/ and starts from it, so the entries stay in force while
dns-control, k3s or pdns-auth is down. Change the entries in dns-control, not on the node. See
List entries in the dns-control runbook.
The allow zone comes before the operator zone, so a block entry for an owned zone or an excepted
name has no effect. The operator zone comes before every feed, so an allow entry lets through a
name that a public list blocks. When an entry decides a query, dns.policy.zone names
operator.rpz.internal. or the scope zone.
Until 2026-10-14 the entries reached the node as two files under /var/lib/dns-worker/lists/,
which the main Blocky process read. Nothing reads the files now, and the retire play removes them.
See Retiring Blocky.
To read what a node holds:
# On a resolver node.rec_control --config-dir=/etc/powerdns --config-name=main --socket-dir=/run/pdns-recursor-main rpzstatistics | grep -A3 'operator.rpz.internal'dig @127.0.0.1 -p 1053 operator.rpz.internal. AXFR | grep -c 'CNAME' # the rule count in pdns-authThe serial of the zone in rpzstatistics and the serial of the SOA in pdns-auth are equal on a
node that loaded the last change. If dns-control is down and an entry must change at once, there
is no file to edit. Bring dns-control back and make the change there. Its next list change writes
the whole zone set from its store.
What the pair emits onto an untrusted segment
Section titled “What the pair emits onto an untrusted segment”:53 over both address families and both transports, ICMP echo, VRRP, RA carrying RDNSS, and
DHCPv4 from Kea. Not a per-leg TSIG key, and not a line in legs-main.map.
Kea serves subnet 6 on Guest and subnet 7 on Telework, each with the pool .20 to .249. A lease
on these legs follows three rules:
- Kea sends no DNS update.
ddns-send-updatesisfalsefor the subnet, so no client gets a name in the internal zone. - The lease carries no search domain. Option 119 and
ddns-qualifying-suffixare absent. - The lease lasts 600 seconds, with a renew timer of 300 seconds. The trusted legs keep 43200.
A leg gets a Kea subnet and the udp/67 accept when it declares kea_subnet_id, not when it is
trusted. trusted still decides DNS trust.
The gateway redirects no DNS query on these legs. Its zone policies block DNS from them to the
gateway and all traffic from them to the trusted legs. So a client with a hand-set resolver gets
no reply from .1, and a public answer from a public resolver such as 1.1.1.1. A client that
uses the resolver from its lease is not affected.
The keepalived sync group holds the trusted instances only
(dns_vlan_interfaces | selectattr('trusted') | list), so VI_guest and VI_telework are not in
the estate-wide group. Each of the two tracks chk_public_answer on its own. The check asks the
public process at 127.0.0.3 for health.dns.invalid and requires NXDOMAIN, which the process
answers from its identity zone dns.invalid. without an upstream query. A failure of the public
process moves only those two VIPs.
The IPv6 half — what the pair answers, what the gateway sends, and the hazards
Section titled “The IPv6 half — what the pair answers, what the gateway sends, and the hazards”All seven legs have their IPv6 clients pointed at the resolver pair through RDNSS, the DNS server option of a router advertisement. Two senders carry it on each leg: the RA of the pair and the RA of the gateway. No device serves DHCPv6. A v6 client asks the leg’s v6 VIP on-link.
What makes this possible:
- Each recursor process listens on the ULA address and the ULA VIP of every leg it serves.
- Every declared leg has a floating ULA VIP at
::5. See IPv6 ULA Addressing.
The gateway’s RDNSS list
Section titled “The gateway’s RDNSS list”The list is dhcp_v6_server.dns_servers on the network in tf/unifi/networks.tf. The gateway
runs no DHCPv6 service, and it sends the list as RDNSS in its router advertisement. Change the
list through a pull request.
| VLAN | 802.1Q | Value of dns_servers |
|---|---|---|
| Main | 3000 | fddb:f665:73f7:1::5 |
| IoT | 3020 | fddb:f665:73f7:3::5 |
| Core | untagged | fddb:f665:73f7:6::5 |
| MainWireless | 3001 | fddb:f665:73f7:7::5 |
| Lab | 1000 | fddb:f665:73f7:4::5 |
| Guest | 3100 | fddb:f665:73f7:2::5 |
| Telework | 3200 | fddb:f665:73f7:5::5 |
The ULA index and the node suffix are small integers in different hextets. Read a value from its own row, never from a neighbouring one.
Each list holds one address. A resolver node’s own address (::6 or ::7) bypasses the VIP and
sits behind a health check that does not test IPv6. A public resolver sends internal queries out
of the house and returns public NODATA for every .house name. A second address appears on the
wire as a second RDNSS option, so the single entry is checkable.
Two IPv6-specific hazards on the floating addresses
Section titled “Two IPv6-specific hazards on the floating addresses”-
Every floating v6 address carries
-nodad. Without it, Duplicate Address Detection can leave the VIP in thedadfailedstate: keepalived reports the address as held, the interface lists it, and nothing answers on it.Terminal window ip -6 addr show | grep -i dadfailed -
keep_addr_on_downis 0 on every leg, so an adminip link set <leg> downdeletes the v6 VIP and not the v4 one. The IPv4 VIP comes back with the link. The IPv6 VIP is gone until keepalived reasserts it. An operator who bounces a leg has removed IPv6 service for that VLAN and left IPv4 working, which reads as a DNS problem.
A client can hold two resolvers after a change
Section titled “A client can hold two resolvers after a change”A new RDNSS appears within the announcer’s interval. A displaced one is never withdrawn. The
client ages it out on its own schedule, up to its advertised lifetime. Client behaviour differs:
macOS replaces the per-interface resolver list from the newest advertisement, and NetworkManager
accumulates. Read the client you have. If a displaced address is still present past the
lifetime, a sender is advertising it again. Make sure that dhcp_v6_server.dns_servers of the
network in tf/unifi/networks.tf holds the ULA VIP and nothing else.
Validating a resolver role off the estate — one command per role
Section titled “Validating a resolver role off the estate — one command per role”Five roles carry a molecule scenario: dns-common, pdns, pdns-recursor, keepalived and
kea. Each scenario’s converge runs that role’s own tasks/verify.yml against containers built
from the trixie image the resolvers run. The scenarios link ansible/inventory/ in, so a converge reads the
real group_vars and host_vars. lightningstream, dns-otel-collector and dns-tailscale
have no scenario, because their verify.yml asserts nothing a converge can
falsify.
Run these from a clone of this repository on your workstation, with Docker running, one scenario
at a time. Platforms are named after the real resolvers, so two scenarios up at once collide on the
container names. dns-common and pdns read from Vault on the control node, so export
VAULT_TOKEN first. vault login -method=oidc writes ~/.vault-token but does not export it.
export VAULT_TOKEN="$(cat ~/.vault-token)"mise //ansible:molecule-dns-common # want: PLAY RECAP failed=0 unreachable=0 for sandiego and portlandmise //ansible:molecule-pdns # want: the scenario stops at its recorded boundary, see belowmise //ansible:molecule-pdns-recursor # want: PLAY RECAP failed=0 unreachable=0 for sandiegomise //ansible:molecule-keepalived # want: PLAY RECAP failed=0 unreachable=0 for sandiego and portlandmise //ansible:molecule-kea # want: PLAY RECAP failed=0 unreachable=0 for sandiegoNotes per scenario:
dns-commonandkeepalivedneed two platforms, because each dereferenceshostvars[keepalived_peer_host]. A 403 on the ICA1 chain read is an absent or expired token, not a broken role. Thedns-commonscenario also proves the rate meter: portland sends twice the burst to port 53 of sandiego in under a second, and the counter of the meter rule on sandiego reads how many queries the meter dropped.pdnsstops atzone-setwith a failed criterion, and mise exits with status 2. That red is the scenario’s boundary, not a defect: the role creates no zone, because dns-control applies every zone through the HTTP API. The boundary is recorded in the scenario’sconverge.yml. Do not seed a zone and do not add awhen:.pdns-recursorneeds no Vault session. Its prepare step installs pdns-auth with the policy zones of the scenario, a dnsmasq stub in place of dnscrypt-proxy on127.0.0.1:5300, a local HTTP server for the list sources, and the node worker binary from a build with the Go toolchain ofapps/dns-control. The converge runs the role with its guard on, and then the scenarios of the specs: a leg with no lists, an undeclared address, an internal name from Guest, rebinding, a scoped name in both orders, a restart with pdns-auth down, a failed source, the two health names, and a wipe of one name followed by a flush of.. The first feed run of the role endsfailedby design, because one list group of the scenario points at an address that never answers.keepalivedruns withkeepalived_enabled: false. At the real value every instance enters FAULT on a container without the legs, with the same roll-up.keaneeds no Vault session. The scenario skips thekea-certstag. It writes a self-signed CA as the trust anchor, an HA bundle in place of the bundle of Vault Agent, and fixed test credentials at the pathstasks/certs.ymlwrites. It creates the userdns-workerand a reservation file for each trusted leg, so the seed reads no export. It loads thedns-commonnftables ruleset and puts each leg’s address on a dummy link, so the daemon starts on every leg. The play fails if the role takes the reservation directory fromdns-worker. The inventory namesportlandwithout a container, because the HA peers list reads its host_vars.
The PowerDNS authoritative tier
Section titled “The PowerDNS authoritative tier”pdns holds 127.0.0.1:1053 on sandiego and on portland, and lightningstream runs on both.
Every .house answer a client gets comes from PowerDNS, whichever node holds the floating
addresses. There is no unbound on either node.
A read that tells you which state a node is in, without changing anything:
# On a resolver node. Reach it with:# scripts/dns-adhoc.sh <node> -m ansible.builtin.shell -a '<the line below>'# The resolver SSH key lives in Vault and the SSH agent is not used. The header of# scripts/dns-adhoc.sh says why.echo "$(hostname -s) vip=$(ip -br addr show | grep -c 192.168.20.5) pdns=$(systemctl is-active pdns) ls=$(systemctl is-active lightningstream)"# want: <node> vip=<0 or 1> pdns=active ls=active# vip= is the only field that differs between the nodes. Any other difference is a finding.The apex, and what it answers
Section titled “The apex, and what it answers”None of these is a defect. All of them look like breakage to somebody who meets them for the first time in an incident.
An unlisted RR type on a declared name returns NOERROR with no answer. This is the property that closes the type-65/HTTPS ECH leak:
dig @127.0.0.1 -p 1053 dns.main.fzymgc.house TYPE65dig @127.0.0.1 -p 1053 dns.main.fzymgc.house HTTPSBoth answers carry NOERROR with an empty answer section and the apex SOA in AUTHORITY. That is
NODATA, not NXDOMAIN.
An undeclared name returns NXDOMAIN with the apex SOA:
dig @127.0.0.1 -p 1053 nonexistent-name-for-the-gate.fzymgc.house AThe AUTHORITY SOA’s MINIMUM field reads 60. That is how long a client keeps believing this
NXDOMAIN, and it is the dominant term in the convergence window below.
No .house query leaves the box, on two independent grounds. The main recursor process forwards
the zone to pdns-auth with no fallback, and PowerDNS as configured cannot recurse or forward. The
guard of the main process is the forward_zones entry for fzymgc.house with the forwarder
127.0.0.1:1053 in /etc/powerdns/recursor-main.yml. A forward_zones entry never falls
through to forward_zones_recurse. Read it like this, and read the exit status rather than a
|| echo branch:
# On a resolver node. Reach it with:# scripts/dns-adhoc.sh <node> -m ansible.builtin.shell -a '<the line below>'grep -n -A1 'zone: fzymgc.house' /etc/powerdns/recursor-main.yml; echo "RC=$?"# want: RC=0 and two lines, the zone and its forwarder 127.0.0.1:1053.# RC=1 is an absence and a finding.The apex itself answers:
dig @127.0.0.1 -p 1053 fzymgc.house SOA # NOERROR, the SOA in the answer sectiondig @127.0.0.1 -p 1053 fzymgc.house A # NOERROR/NODATA. No A record is declared at the apexdig @127.0.0.1 -p 1053 fzymgc.house NS # both nodes as NS, with their addresses in ADDITIONALConvergence lag
Section titled “Convergence lag”PowerDNS runs multi-primary with no zone transfer. Each node owns a local LMDB, and LightningStream reconciles the two through S3. There is no primary to lose. What replaces it is a window during which the two nodes disagree.
| Term | Value | What it is |
|---|---|---|
lmdb_poll_interval |
1s |
how long the writing node takes to notice its own change |
| the snapshot upload | sub-second at this zone size | pushing the snapshot to S3 |
storage_poll_interval |
1s |
how long the peer takes to list the storage and see it |
| the download and merge | sub-second at this zone size | the peer pulling and merging it |
Both intervals are in ansible/roles/lightningstream/defaults/main.yml. A record written on one
node is answerable on the other in a few seconds. Grant metadata that dns-control writes through
the writer node’s API takes longer to land on the peer, on the order of a minute. See The two latencies
under The DDNS tier.
A client that asked the other node inside that window got NXDOMAIN and keeps believing it for the
SOA minimum of 60 seconds, set on every zone that dns-control holds. The worst case an operator
observes is about 65 seconds. Both polls are at their floor of one second, and the negative cache
is the dominant term. Lowering the negative-cache TTL is a zone-wide decision, not an operational
knob.
A record added to an existing zone is not delayed by the zone cache. A new zone takes up to
zone-cache-refresh-interval (60 s, ansible/roles/pdns/defaults/main.yml) longer to become
visible on the peer. That setting caches the zone list, not records.
Reverse resolution
Section titled “Reverse resolution”Every zone is Native, because there is no transfer and no primary:
| Zone | Covers |
|---|---|
168.192.in-addr.arpa. |
the whole of 192.168.0.0/16 outside the carved cuts. The curated PTRs |
the per-leg /24 carves in dns_ddns_reverse_zones |
the DDNS-writable reverse space, one real /24 zone each. See The DDNS tier |
7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa. |
the estate’s ULA fddb:f665:73f7::/48 |
The IPv4 zone makes the pair authoritative for address ranges that get no DNS update: Guest
192.168.79.0/24, Telework 192.168.217.0/24 and WireGuard 192.168.80.0/24. Reverse queries
for those addresses return NXDOMAIN from the zone.
The 2600: GUA reverse is not attempted. The estate’s global IPv6 prefix is delegated by Verizon,
and so is its ip6.arpa reverse. dns-control holds no zone for it. Do not file “IPv6
PTRs do not work for our global addresses” as a defect against this tier.
PTRs are not a side effect of the forward record. Each curated entry in dns-control carries an
explicit ptr flag, so the canonical name for an address is one entry, and dns-control writes its
PTR.
Record write paths, and the update boundary
Section titled “Record write paths, and the update boundary”Three writers, and they do not overlap:
| Writer | What it writes | How |
|---|---|---|
| dns-control | the curated entries, the zones, the SOAs, the zone metadata with the TSIG grants, the control record sets |
one change workflow per change, against the pdns API on :8081 |
| external-dns | the Kubernetes service estate | the dns-control webhook sidecar, one external-dns lane change per sync, TXT-registry ownership |
| Kea D2 | DHCP-derived names in <leg>.lan.fzymgc.house and the carved reverse zones |
DNS UPDATE with TSIG |
The update boundary is declared. Each leg’s child zone <leg>.lan.fzymgc.house and each carved
reverse /24 carries a TSIG-ALLOW-DNSUPDATE grant in the zone metadata that dns-control holds. The parent
lan.fzymgc.house, the covering /16 and the apex carry none, and dnsupdate-require-tsig=yes
makes that absence a REFUSED. See Where the boundary is, and what enforces it under The DDNS
tier.
dig TXT <managed-name> returns an ownership record that looks like junk. It is not junk.
The TXT registry of external-dns runs with no --txt-prefix, the form that its old pdns provider
needed, so it writes its ownership record at the same name as the record it owns. Deleting one by hand makes external-dns
believe it no longer owns the record beside it.
Zone syntax is checked with pdnsutil zone check <zone>.
Adding a name
Section titled “Adding a name”A name enters the zone through exactly one of three writers, decided by what the name is:
- A device or host is one curated entry in dns-control. Sean decided on 2026-09-29 that the entry
is data, not configuration (issue #2527): dns-control holds it, one workflow projects it onto
pdns, onto the Kea reservation files and onto the export, and Terraform applies no curated
record. Plan the entry on the Changes screen, through
POST /api/changes/plan, or with the MCP tooldns_create_entry, read the diff, and apply it. The fields arekey,hostname(a lowercase FQDN with a trailing dot),address,address6(or empty),ptr,aliases,comment, and the optionalmac. The AAAA of an entry is itsaddress6, which its owner sets. The observed AAAA sweep writes nothing at the hostname or an alias of an entry. Writeptr: falseunless the address is outside every zone indns_ddns_reverse_zones(ansible/inventory/group_vars/dns_hosts.yml), because a curated PTR inside DDNS-writable space is parent-held under the carve and answers nothing. An entry that carries amacis also a Kea reservation on the leg of its address. The workflow writes the reservation file of each trusted leg on both nodes and reloads Kea, so the device gets its address on its next request. The entry policy refuses the infrastructure names (dns.<leg>, the per-leg node names, the node apexes) and every name underlan.fzymgc.house. See Entry changes in the dns-control runbook. - A zone, its metadata or an infrastructure name is a zone or record set change in dns-control.
The plan and the apply need the role
dns-admin. Plan the change throughPOST /api/zones/plan,POST /api/zones/metadata/planorPOST /api/rrsets/plan, or with an MCP tool such asdns_create_rrset. Read the diff, and apply it. A record set in thecontrollane is not an entry, so it projects onto pdns only and writes no Kea reservation. See Zones and record sets in the dns-control runbook. - A Kubernetes service is a
Host()matcher plus theexternal-dns.alpha.kubernetes.io/targetannotation, and nothing undertf/. Read The two external-dns traps before adding one. - A DHCP client needs nothing done. Kea D2 writes
<host>.<leg>.lan.fzymgc.houseon the lease for the five zones indns_ddns_forward_zones. The AAAA at that name comes from the observed AAAA sweep of dns-control. The sweep reads the client’s addresses in the UniFi client table and publishes the ones inside the ULA /64 of the leg, after two sweeps of 300 seconds. It never publishes a GUA. See Observed AAAA in the dns-control runbook.
The two apex writers are disjoint, and neither declares under lan.fzymgc.house. external-dns
drops a lan. name through --exclude-domains=lan.fzymgc.house
(argocd/app-configs/external-dns/values.yaml). A curated entry whose hostname sits under a
delegated child is written into the apex by dns-control, is parent-held under the delegation, and
answers NXDOMAIN with the child’s authority. No gate holds this rule. A name under no writer
answers NXDOMAIN and is not forwarded.
The two external-dns traps, both of which fail silently
Section titled “The two external-dns traps, both of which fail silently”external-dns runs with --policy=sync, so it deletes what it believes it no longer owns.
Trap 1: the annotation prefix is coupled to the image version. The pinned tag in
argocd/app-configs/external-dns/values.yaml reads the external-dns.alpha.kubernetes.io/
spelling, and every manifest carries it. A version that changes the default prefix stops seeing the
annotations, generates no endpoints, and deletes the records they produced. Its log says nothing.
The symptom is names that worked yesterday returning NXDOMAIN with external-dns healthy. The
annotationPrefix value in that file pins the spelling, so the image and the manifests can move
on their own.
Trap 2: the traefik-proxy source builds targets from the target annotation and from nowhere else.
Traefik CRDs do not expose a load balancer IP in their status, so an IngressRoute without the
annotation produces no endpoint. Use the dual-stack value, which lists both Traefik VIPs with
IPv4 first:
external-dns.alpha.kubernetes.io/target: "192.168.250.145,fddb:f665:73f7:250::145".
If the value holds only the IPv4 VIP, the name gets an A record and no AAAA record. See
DNS records in the Lab runbook. The symptom is a
newly added IngressRoute whose host never appears in DNS while every older one resolves.
The S3 endpoint and its certificate
Section titled “The S3 endpoint and its certificate”LightningStream reconciles the two LMDBs through https://s3-direct.nas.fzymgc.house, the NAS’s
own direct S3 address, with an estate-issued certificate. The credentials come from Vault at
secret/fzymgc-house/infrastructure/nas/rustfs-s3.
This path is not in the serving path. PowerDNS answers every query from its own LMDB whether or not S3 is reachable. An S3 outage, a certificate expiry or a NAS reboot degrades convergence and nothing else. It is a ticket, not a page. There is no dedicated availability detector for this endpoint, because one would take a healthy node out of VRRP eligibility over a storage fault.
The certificate renews through Traefik’s DNS-01 resolver, which needs the WAN to renew and nothing to serve.
The endpoint name has no DNS record. roles/lightningstream pins it in /etc/hosts and in
/etc/cloud/templates/hosts.debian.tmpl, because the nodes’ cloud-init sets manage_etc_hosts: true and re-renders /etc/hosts from that template at every boot.
--tags lightningstream-verify asserts both. If replication has stopped and the journal shows
lookup s3-direct.nas.fzymgc.house on 127.0.0.1:53: no such host, the pin is gone. Repair one node
at a time:
scripts/dns-playbook.sh --limit portland --tags lightningstream-configscripts/dns-playbook.sh --limit sandiego --tags lightningstream-configBoth nodes log Loaded remote update and startup phase completed successfully within seconds.
The client symptom is a name that answers on one node and returns NXDOMAIN on the other. Nothing
alerts on this.
Handing the authoritative socket to another daemon
Section titled “Handing the authoritative socket to another daemon”These rules apply to any future handover of the authoritative tier.
Measure the name delta by querying, not by diffing configuration. For each name the tier being
replaced serves, dig @127.0.0.1 -p 1053 <name> A +short on the successor. Empty output means the
name is dark. A dark name is acceptable only if it is stale, and staleness is verified by asking
the cluster:
kubectl get ingressroute,ingressroutetcp,ingress,httproute -A -o json | grep -c '<name>.fzymgc.house'Traefik’s Host(...) rules use backticks. A pattern anchored on double quotes matches Ingress
objects and misses every IngressRoute.
A dark name is an outage, because fallbackUpstream: false leaves no other tier to answer it.
Prove the rollback target before releasing the addresses. The rollback is
sudo systemctl stop keepalived on the holder. Read the node you roll back to as healthy first:
keepalived-check-house.sh rc=0 and both tiers active.
Two daemons cannot both hold 127.0.0.1:1053. The play that stops the incumbent runs before the
play that starts the successor. ansible/dns-playbook.yml’s play order sequences that, so a single
tagged run is correct and two runs are not. Run a handover on the node that holds no floating
address.
LightningStream must push from a populated store before a peer pulls. A node whose pdns starts on
an LMDB pulled from an empty store answers nothing for .house and takes chk_house_answer down.
lightningstream_group: pdns names a group the pdns role creates, so converging LightningStream
first fails with chgrp failed. The play order sequences both.
When both nodes lose the zones
Section titled “When both nodes lose the zones”dns-control holds every zone, grant, record set and entry, and its export keeps a copy on the NAS and on B2. If both nodes lose their LMDB, converge the authoritative tier so that pdns runs on an empty store. Then rebuild the zones from the export:
kubectl -n dns-control exec deploy/dns-control -- dns-control restore --from b2 --dry-runkubectl -n dns-control exec deploy/dns-control -- dns-control restore --from b2The dry run reads create or change for each zone, record set and entry that pdns does not
serve. The restore creates each zone with its SOA, its NS and its grant, and then writes its
control record sets and its entries. external-dns writes its own lane on its next sync, and Kea D2
writes its names again on the next lease events. If B2 is not reachable, use --from nas. See
Restore from the export in the dns-control
runbook.
Where the resolver pair is cabled
Section titled “Where the resolver pair is cabled”Both resolvers hang off switch-main. That is a single failure domain, and it is accepted: a
failure of that switch takes the whole estate down and leaves a surviving resolver nothing to serve,
and the switch does not need DNS to boot. Nothing in this repository re-checks the cabling. Read it
live with one LLDP frame per node on eth0.
A rule that required the two nodes to sit on different PoE switches does not apply to this topology. Neither node is PoE-powered alone, and the reasoning above stands. Do not re-assert it.
Moving portland to a second switch is possible with one precondition: the new port needs the same native-VLAN-to-Core treatment as sandiego’s port, or that node’s Core leg drops. That is half the estate’s DNS. Do not do the move without the port work.
The surviving half of the old requirement is live: never change both resolver nodes in the same window.
Promotion — moving the floating addresses deliberately
Section titled “Promotion — moving the floating addresses deliberately”Both instances start in the backup state with nopreempt (roles/keepalived/templates/keepalived.conf.j2).
A node that is up and returns from a tracker FAULT does not take the addresses back. Promotion is
an explicit act.
nopreempt does not survive a reboot of the higher-priority node. It suppresses the BACKUP to
MASTER transition only for an instance that hears an existing master. On a reboot of sandiego
(priority 150) its master-down timer expires before its VLAN sub-interfaces pass VRRP, so it reads
the group as unowned and takes it. After a reboot of the priority-150 node, expect the group back
on it. Read ownership. Do not assume the peer kept it.
Which node holds the addresses is a reading:
scripts/dns-vip-holder.py # prints the ACTIVE nodescripts/dns-vip-holder.py --passive # prints its peerssh 192.168.40.6 'ip -br addr | grep -c "192\.168\..*\.5/"'ssh 192.168.40.7 'ip -br addr | grep -c "192\.168\..*\.5/"'# want: one node reports 7 and the other 0. 5 and 2 follows a failure of the public recursor process, because the# Guest and Telework VIPs are outside the sync group and move alone. A total above 7 is a split brain.Verify on the node that is not holding the addresses, promote deliberately, then work on the other one:
# STEP 1: verify on the non-holder, where a mistake costs nothingssh <non-holder> 'systemctl is-active keepalived pdns-recursor@main pdns-recursor@public pdns'ssh <non-holder> 'sudo /usr/local/sbin/keepalived-check-house.sh; echo $?' # expect 0ssh <non-holder> 'sudo /usr/local/sbin/keepalived-check-public.sh; echo $?' # expect 0ssh <non-holder> 'dig +short @127.0.0.1 dns-health.fzymgc.house A' # expect 192.168.x.x
# STEP 2: move the groupssh <holder> 'sudo systemctl stop keepalived'ssh 192.168.40.6 'ip -br addr'; ssh 192.168.40.7 'ip -br addr' # the peer now holds all seven
# STEP 3: work on the drained node, then return it as BACKUPssh <old-holder> 'sudo systemctl start keepalived'# nopreempt: it returns as BACKUP and does not take the group backsystemctl stop keepalived is the role’s stated position. roles/keepalived/tasks/config.yml
carries no stop branch, so dropping seven live floating addresses is an operator act.
Drain and undrain through dns-control
Section titled “Drain and undrain through dns-control”dns-control drains a node through the node worker on the Pi. A drain is the move of STEP 2 above, with a check of the peer first and an audit row that names the person. Start a drain or an undrain from the card of the node on the Health screen of dns-control. The actions, the five peer checks and the roles are in the dns-control runbook, Operations actions.
A drain writes the drain marker /var/lib/dns-worker/drained and then stops keepalived. If the
stop fails, the drain removes the marker again. If the drain activity ends during the stop, the
stop can still complete, so the marker stays. An undrain clears that state. The marker is a JSON
file:
| Field | Value |
|---|---|
actor |
The person who drained the node |
since |
The time of the drain |
expected_minutes |
The expected duration of the drain |
audit_id |
The id of the audit row of the drain |
The keepalived role renders the systemd drop-in
/etc/systemd/system/keepalived.service.d/10-drain-marker.conf with
ConditionPathExists=!/var/lib/dns-worker/drained. While the marker exists, systemd does not start
keepalived, at boot too. A start through systemd then exits 0 and leaves the unit inactive. The
keepalived role reads the marker at each converge and reports it. It does not start keepalived
while the marker exists, so a drained node stays enabled and inactive.
The undrain activity of the node worker removes the marker. If the node worker is lost, remove the
marker by hand with
The manual undrain when the node worker is lost.
An undrain removes the marker, starts keepalived and makes sure that keepalived is active. The
node returns as BACKUP, because of nopreempt. A marker that does not parse still counts as
drained, and an undrain removes it. With no marker and keepalived not active, an undrain starts
keepalived. An undrain therefore also finishes a drain that stopped half way.
Reload keepalived after the notify script lands
Section titled “Reload keepalived after the notify script lands”The node worker reads the state of each keepalived instance from a file in
/run/keepalived-state. The notify script /usr/local/sbin/keepalived-notify-state.sh writes
that file at each state change. The keepalived role renders the script and the notify line in
keepalived.conf, but the role never reloads keepalived. Until keepalived reads the new
configuration, the state files do not exist. The worker then reads the keepalived state of the
node as UNKNOWN, and the drain check keepalived_not_fault fails for that node.
After a converge that installs the notify script, do these steps on one node per window. Skip a drained node, because its keepalived is stopped.
-
Reload keepalived:
Terminal window scripts/dns-adhoc.sh <node> -b -m ansible.builtin.command -a 'systemctl reload keepalived.service' -
Make sure that
/run/keepalived-stateholds oneVI_*file for each instance. The two numbers must be equal:Terminal window scripts/dns-adhoc.sh <node> -b -m ansible.builtin.shell -a 'ls /run/keepalived-state | grep -c "^VI_"; grep -c "^vrrp_instance " /etc/keepalived/keepalived.conf' -
If a file is missing, do not restart keepalived on the node that holds the VIPs. Move the VIPs first with the steps in the promotion section above.
The manual undrain when the node worker is lost
Section titled “The manual undrain when the node worker is lost”If the node worker cannot run the undrain, undrain the node by hand. For example, the worker, its
certificate or the Temporal gateway is down. Work on one node per window. Remove the marker first,
then start keepalived, because the start does nothing while the marker exists. Replace <node>
with sandiego or portland. Never use dns_hosts here.
-
Make sure that the peer holds all seven VIPs. Expect 7 on the peer and 0 on the drained node:
Terminal window ssh 192.168.40.6 'ip -br addr | grep -c "192\.168\..*\.5/"' # sandiegossh 192.168.40.7 'ip -br addr | grep -c "192\.168\..*\.5/"' # portland -
Remove the marker as root:
Terminal window scripts/dns-adhoc.sh <node> -b -m ansible.builtin.file -a 'path=/var/lib/dns-worker/drained state=absent' -
Start keepalived, and make sure that it is active:
Terminal window scripts/dns-adhoc.sh <node> -b -m ansible.builtin.command -a 'systemctl start keepalived.service'scripts/dns-adhoc.sh <node> -m ansible.builtin.command -a 'systemctl is-active keepalived.service' -
Expect
active. The node returns as BACKUP and does not take the VIPs back.
A manual undrain writes no audit row. Health shows the node as not drained at its next status reading, when the node worker answers again.
The node worker role
Section titled “The node worker role”The Ansible role dns-node-worker installs the node worker and Vault Agent on a resolver node.
Run it on one node per window:
scripts/dns-playbook.sh --limit <node> --tags dns-node-workerThe first task reads the AppRole role id of the node from Vault on the control node. If you run
the role before the tf/vault apply, the AppRole does not exist yet. The run then stops at that
first task with the error of vault read, and the node does not change.
The role writes an AppRole secret id to /etc/dns-node-worker/vault/secret_id only when the node
has none. To rotate the secret id, remove that file on the node and run the role. Before it
creates the new secret id, the role destroys every earlier secret id of the AppRole of that node.
No earlier secret id then keeps working. A run where the file exists destroys nothing. Your Vault
token on the control node must be allowed to list and destroy the secret id accessors of the
AppRole.
The polkit rule of the role grants the worker two actions only: start and stop of
keepalived.service. The worker cannot restart a recursor process or any other unit. It talks to
the two recursor processes through their control sockets, which the pdns-recursor role gives
the group dns-worker, and through their HTTP API with the key copy
/etc/dns-node-worker/recursor-api.key.
The tag dns-node-worker-verify runs the read-only checks of the role alone. They make sure that
the installed worker is the pinned release, and that both units are enabled exactly when
dns_node_worker_service_enabled is true. When that value is true, the checks also make sure
that:
dns-node-worker.serviceanddns-node-worker-pki.serviceare running.- The client certificate in the bundle has not expired. The worker user runs
openssl x509 -noout -checkend 0on the bundle. - The worker user can read its configuration, the API key copy and the CA file, and can run the two check scripts.
- The worker user reaches the control socket of each recursor process, and the recursor user
pdnsreads the scope map, which is 0644 and owned by the worker. - The worker user can read the certificate bundle and cannot read the secret id.
What a failover looks like
Section titled “What a failover looks like”A tracker-induced failover moves all five addresses together in about ten seconds: detection is
check_interval 2 × fall 5, plus up to 3 × advert_int 1, plus the priority skew. The values are
in roles/keepalived/defaults/main.yml. The group can be briefly unowned in transit.
With both nodes faulted the floating addresses go unowned, not duplicated. keepalived and
pdns-recursor@main report active on both nodes, and both journals carry Entering FAULT STATE. The shape
to recognise is nothing holds the addresses and every unit looks healthy.
keepalived sends a gratuitous ARP on entering the master state. That is what moves a client’s
cached mapping for a floating address to the new node. If a class of client holds a stale mapping
after a move: make sure that journalctl -u keepalived on the new master shows Entering MASTER STATE, check whether that client’s segment drops unsolicited ARP, and for unmanaged devices with
long cache lifetimes accept a bounded window.
Breaking one node’s external DNS path on purpose
Section titled “Breaking one node’s external DNS path on purpose”scripts/dns-break-external.sh induces real breakage on one node. Every public name of a node
goes through dnscrypt-proxy on its loopback, and nothing else on the node speaks to the relays,
so the script stops that unit. .house, the reverse zones, VRRP, SSH and tailscale are untouched,
and a public name answers SERVFAIL in about 2 milliseconds:
scripts/dns-break-external.sh sandiego apply # systemctl stop dnscrypt-proxyscripts/dns-break-external.sh sandiego status # unit state and the live-server line of its last startscripts/dns-break-external.sh sandiego revert # start, and wait for a live routeThe recursor serves a cached answer without the daemon, so the break does not make every lookup fail. Nothing consumes the result automatically. An operator reads the outcome. Before 2026-10-06 the script dropped TCP port 853 in an nftables table, the DoT egress to Control D.
Things that will bite you — read these before touching the pair
Section titled “Things that will bite you — read these before touching the pair”A configuration change of the recursor is a rec_control reload or a restart of one process
Section titled “A configuration change of the recursor is a rec_control reload or a restart of one process”A change to the RPZ list of a process, the include directory of the main process included, goes
through rec_control reload-yaml. It reads every RPZ again and keeps the caches. A change to the
scope map goes through rec_control reload-lua-script. Every other change to
/etc/powerdns/recursor-<p>.yml, such as a listener or a forwarder, needs a restart of that one
process:
# On a resolver node. <p> is main or public.rec_control --config-dir=/etc/powerdns --config-name=<p> --socket-dir=/run/pdns-recursor-<p> --timeout=20 reload-yamlsudo systemctl restart pdns-recursor@<p>A reload of the main process reads every feed zone and can take longer than 5 seconds, the default
wait of rec_control. Without --timeout, rec_control then prints Timeout waiting for control channel data and exits 1.
systemctl restart pdns-recursor@main interrupts the answers on the trusted legs for about two
seconds. The keepalived check runs every 2 seconds and needs fall 5 failures to move the VIPs,
so a restart does not move them. Restart one process at a time: each process binds its own
addresses, so the other one keeps answering. Do not restart both processes of one node in one
command, and never restart a process on both nodes in the same window.
Until 2026-10-14 the caution here read: never systemctl reload blocky. Blocky exited on SIGHUP
and left the client tier down with a clean exit. The recursor has no such hazard. systemd refuses
reload for the unit template, because it carries no ExecReload=.
networkctl reload strips foreign addresses. Never run it on the node holding the VIPs
Section titled “networkctl reload strips foreign addresses. Never run it on the node holding the VIPs”keepalived’s floating addresses are foreign to systemd-networkd. networkctl reload removes
foreign addresses from a link whose .network changed. On the VIP holder it pulls the live
floating addresses while keepalived goes on reporting MASTER.
To add or change a leg:
- Converge the node that holds no floating address.
networkctl reloadthat node.- Move the group (
systemctl stop keepalivedon the holder). - Repeat on the newly freed node.
A converge reports success and the machine does not change
Section titled “A converge reports success and the machine does not change”All three bite whoever next edits dns_vlan_interfaces:
| What | The reading | Consequence |
|---|---|---|
keepalived has no path from a changed config to the running daemon |
after --tags keepalived reports changed=1, the new instances are on disk, the process uptime is unchanged, and the addresses held are unchanged |
the rendered config is adopted at the next restart or reboot, a VRRP topology change at an unplanned moment |
Nothing reloads systemd-networkd, so a new leg never comes up |
after --tags dns-common reports changed=N, both files are rendered, the new eth0.<tag> is absent, and the nftables accepts that name it have reached the kernel |
the node admits DNS on an interface it does not have |
| Tag filtering silences notified handlers | a converge reports failed=0 with no RUNNING HANDLER section, having deleted a file and written nothing in its place |
the machine is in a state neither configuration describes |
After any change to dns_vlan_interfaces, read the machine, not the recap: ip -br addr for the
leg, systemctl show -p ExecMainStartTimestamp keepalived against the config’s mtime, and the
addresses held.
For the handler case: --skip-tags always drops notified handlers, --force-handlers does not
rescue them, and --list-tasks cannot see them. Skip the two roles by name instead of filtering on
always. The check that discriminates is the presence of the RUNNING HANDLER section in the
output.
Anything you arm with systemd-run must pin AccuracySec
Section titled “Anything you arm with systemd-run must pin AccuracySec”systemd-run --on-active defaults to AccuracySec=1min, so a --on-active=5 timer can fire about
sixty seconds late. Pin --timer-property=AccuracySec=1s on every one.
Verifying a DHCP advertisement: select the packet by its own transaction id
Section titled “Verifying a DHCP advertisement: select the packet by its own transaction id”The DHCP OFFER is what proves what a network advertises. A bare grep Domain-Name-Server over a
shared capture attributes other clients’ packets to your probe, and matches a client’s
Parameter-Request list as well as an OFFER. So:
- Select the packet by the probe’s own transaction id, or its synthetic
02:00:00:*MAC. - Assert the message type is
Offer, notRequest. - Assert the lease beside option 6.
- Never
2>/dev/nullthe probe injection. A probe that never went out must be a red.
Trigger the OFFER with a crafted DISCOVER. Tear a capture down by PID. pkill -f "tcpdump -i ..."
matches the invoking ssh shell’s own command line and kills the session you work through.
tcpdump is declared for these nodes through dns_common_diagnostic_packages in
ansible/inventory/group_vars/dns_hosts.yml. The role default is [].
The node’s own fallback resolver is the same daemon
Section titled “The node’s own fallback resolver is the same daemon”/etc/resolv.conf on each node names 127.0.0.1 (the main recursor process) first and
127.0.0.2 second. The second address is a listener of the one dnscrypt-proxy of
roles/dnscrypt-proxy, so a node whose client tier is down still resolves public names through
the relays, including the tailnet name of the host an operator uses to fix it. Internal names come
from pdns through the main process and have no fallback. The daemon refuses fzymgc.house and
both reverse zones, so a node with the main process down gets REFUSED for an internal name and
sends nothing to a relay.
Until 2026-10-06 that second listener was a separate packaged dnscrypt-proxy 2.1.8 with DoH to
two static stamps, installed by roles/dns-common. One daemon replaced it, because the two shared
a unit name and a configuration path. The cost is the double outage: a node with the main process
down and every relay down resolves no public name. systemd-resolved with a DNSOverTLS=yes
drop-in was never the home for this, because that unit is neither started nor masked by decision.
The glibc stub resolver advances to the second nameserver on a transport-level failure only. A
main process that is up and answers NXDOMAIN for everything is, to the stub resolver, a working
nameserver. The self-first /etc/resolv.conf recovers a node from a dead client tier, not from a
misbehaving one.
Known risks nobody owns
Section titled “Known risks nobody owns”An exception is a decision with a rationale. What follows is exposure with no detector and no owner.
A second DHCPv4 server on a live leg
Section titled “A second DHCPv4 server on a live leg”The DHCPv4 server of the gateway is off on every leg (dhcp_server.enabled = false in
tf/unifi/networks.tf). If a firmware update or an edit in the UniFi UI turns one back on, a second
DHCPv4 server is on a live broadcast domain. Two DHCPv4 servers on one segment fight, and the
client takes the first OFFER. No timer probes for a second server, and no alert fires on a foreign
OFFER.
The discriminator is lease origin, never the address. Kea’s own lease4-get-all lists a Kea lease
on both nodes. A lease from the gateway never appears there. See Reading Kea lease state from a
workstation.
UniFi DHCP Guard on Main, Guest and Telework trusts only the leg’s .6, .7 and .5. Nobody has
measured whether it drops an offer from the gateway itself.
What the namespace closure covers
Section titled “What the namespace closure covers”Every name the pair serves has a declared source. That is narrower than “every name that
resolves”, because search-suffix shadows are outside it. The DDNS
children are outside the closure by construction: external-dns’s
--exclude-domains=lan.fzymgc.house and D2’s per-leg grants put them there.
The closure is enforced in the chain of the main recursor process and PowerDNS. PowerDNS is
authoritative for fzymgc.house, so an unlisted name is NXDOMAIN and an unlisted type is NODATA.
The forward_zones entry of the main process
(ansible/roles/pdns-recursor/templates/recursor-main.yml.j2) sends every .house query to
pdns, and a forward_zones entry never retries an answer upstream.
There is no inventory variable that opens the closure. A change to the forward zone is an edit of
the template and a converge of --tags pdns-recursor one node at a time. Such a change reopens
the type-65/HTTPS leak.
Why does this name resolve when there is no Terraform entry for it?
Section titled “Why does this name resolve when there is no Terraform entry for it?”A name answers, and nothing under tf/ declares it. The wrong conclusion is that DNS is lying.
fzymgc.house has three writers, and none of them is declared under tf/:
| # | Writer | Mechanism | Declared in git? |
|---|---|---|---|
| 1 | dns-control | writes each curated entry, each zone and each control record set through the pdns API on an apply, with the mark writer=dns-control |
No. The rows live in the dns-control database and in the export on the NAS and B2 |
| 2 | The DDNS tier | Kea’s D2 writes <host>.<leg>.lan.fzymgc.house from each lease |
No. See The DDNS tier below |
| 3 | external-dns | writes a name for each Kubernetes Host() matcher that carries the external-dns.alpha.kubernetes.io/target annotation |
Yes, in the manifests under argocd/ |
Client-side verification runbook
Section titled “Client-side verification runbook”Run this from a real client on the LAN.
for n in id llm grafana vault k8s-cluster; do echo -n "$n: "; dig $n.fzymgc.house A +short; donedig id.fzymgc.house TYPE65 +short # want: empty (NODATA)curl -sS -o/dev/null -w"%{http_code} %{remote_ip}\n" https://llm.fzymgc.house/v1/models # want: 401 via 192.168.250.145This block does not tell you which node answered. Add the discriminator, and the address-family check:
dig whoami.dns.invalid +short # want: 192.168.40.6 or .7, a resolver node. # EMPTY means something other than the pair answered, for # example a hand-configured public nameserver.
# Which server is this client using? On macOS:scutil --dns | grep nameserver | sort -u# A fddb: entry is the pair's v6 VIP for that VLAN. On Linux: resolvectl status.Two probes that manufacture a false incident
Section titled “Two probes that manufacture a false incident”An NXDOMAIN for a name the zone does not curate is correct. Verify with a known curated name:
dig +short nas.fzymgc.house # 192.168.20.200. This is the liveness probe.A workstation with tailnet-claimed routes over estate subnets is not a valid vantage point. If the machine you test from runs Tailscale accepting routes that overlap the estate, queries aimed at Core addresses transit the tunnel. What comes back is a non-authoritative answer no estate server sent, which reads like a resolver serving an empty zone.
# Before trusting any negative result, check that the query leaves on the LAN:netstat -rn -f inet | grep -E '192\.168\.(20|40)' # macOS; want the LAN interface, not utun*If a tunnel interface owns the route, move to a client that is on the VLAN before concluding anything.
The resolver pair
Section titled “The resolver pair”The two nodes and their addresses
Section titled “The two nodes and their addresses”Held in ansible/inventory/host_vars/{sandiego,portland}.yml. Seven legs per node, one address
per leg. The .5 column is a floating address held by whichever node won the election.
| Leg | VLAN id | Network | Gateway | VIP | sandiego |
portland |
|---|---|---|---|---|---|---|
core |
untagged native | 192.168.40.0/22 |
.1 |
.5 |
.6 |
.7 |
lab |
1000 | 192.168.200.0/22 |
.1 |
.5 |
.6 |
.7 |
main |
3000 | 192.168.20.0/22 |
.1 |
.5 |
.6 |
.7 |
iot |
3020 | 192.168.218.0/24 |
.1 |
.5 |
.6 |
.7 |
mainwireless |
3001 | 192.168.24.0/22 |
.1 |
.5 |
.6 |
.7 |
guest |
3100 | 192.168.79.0/24 |
.1 |
.5 |
.6 |
.7 |
telework |
3200 | 192.168.217.0/24 |
.1 |
.5 |
.6 |
.7 |
Each leg also carries a ULA /64 (fddb:f665:73f7:N::/64) with the same ::5, ::6 and ::7
convention. mainwireless carries trusted wireless clients under the same trust policy as Main.
Guest and Telework declare trusted: false, and Kea serves DHCPv4 on both.
The authoritative loopback listener is 127.0.0.1:1053, held by pdns. Loopback :53 is not
available on these nodes: the main recursor process binds 127.0.0.1:53 for the health client,
which is also why the systemd-resolved stub listener is taken down.
The eleven roles and what each one owns
Section titled “The eleven roles and what each one owns”| Role | Owns |
|---|---|
dns-common |
The node baseline: the tagged VLAN legs and their systemd-networkd units, the ip_forward=0 drop-in and its assert, the systemd-resolved stub-listener drop-in that frees :53, the nftables ruleset, the RA announcer and the RA-watch detector, the timed auto-revert, and the uv-built Python interpreter at /opt/dns-resolver/venv |
pdns |
The authoritative tier for fzymgc.house and its children: pdns-auth on 127.0.0.1:1053, its LMDB backend, the DNS UPDATE boundary and the per-leg TSIG grants |
lightningstream |
Multi-primary replication of that LMDB between the two nodes through S3 |
dnscrypt-proxy |
The anonymizing upstream of the client-facing tier: the pinned dnscrypt-proxy binary, proved by its minisign signature, its loopback listener on 127.0.0.1:5300, the route set that reaches every public resolver through a relay of another operator, and the signed resolver lists in its cache |
pdns-recursor |
The client-facing tier: PowerDNS Recursor 5.4 from repo.powerdns.com as two processes, pdns-recursor@main for the trusted legs, the tailnet and the loopback, and pdns-recursor@public for Guest and Telework. The withheld, allow and rebinding zones from files, the operator zone and the scope zones from pdns-auth on loopback, one feed zone for each list group with dns-feeds.timer, the Lua script policy.lua with the leg map of each process, the API key, and DNS-collector as the protobuf reader dns-resolver-reader |
dns-otel-collector |
Query-log and journal transport into ClickStack, including the unit list the RA-watch alert depends on, the parse of the reader rows under the ServiceName of each recursor process, dns-resolver and dns-resolver-public, the metrics scrape of both processes and LightningStream, the OTLP receiver of the node worker on 127.0.0.1:4318, and tlscheck.time_left of the three certificate bundles |
vault-cert-agent |
Vault Agent as the user vault-cert-agent with the AppRole cert-agent-<node>. It renders and renews the bundle of the collector and the HA bundle of Kea |
kea |
The DHCPv4 serving tier: kea-dhcp4 and its HA pair, the per-leg pools and option surface, kea-dhcp-ddns (D2), and the timer that restarts Kea after a renewal of its HA certificate |
keepalived |
Per-VLAN floating-address failover: the VRRP sync group and the .house tracker, and the chk_public_answer tracker on each untrusted leg’s own instance |
dns-tailscale |
The tailnet client on each resolver node, which makes remote .house resolution reach each node’s own 100.x address |
dns-node-worker |
The dns-control node worker with the engine recursor: the locked-down user dns-worker, the polkit rule that allows it only the start and stop of keepalived, the Vault Agent that renews its client certificate, and the seed of the scope map that the main recursor process reads |
The table follows ansible/dns-playbook.yml. The dynamic namespace is lan.fzymgc.house,
populated by Kea’s D2. See The DDNS tier at the end of this page.
/opt/dns-resolver/venv stays because dns-tailscale runs from it. Its verify tasks run the venv’s
Python interpreter to parse JSON. Remove a package from that venv with uv pip uninstall, never
uv pip sync. A sync makes the venv equal the lockfile and removes every package that the lockfile
does not name.
The certificates of the pair
Section titled “The certificates of the pair”The role vault-cert-agent runs Vault Agent on each node with the AppRole cert-agent-<node>. The
agent renders two bundles. Each bundle is the certificate, the CA chain and the key in one file, at
0640, owned by the user vault-cert-agent and the group in the table:
| Bundle | Vault PKI role | Group | Reader |
|---|---|---|---|
/etc/otelcol-dns/certs/tls.pem |
otel-collector-dns-client on ICA1 |
otelcol-dns |
the collector, which reads it again every hour |
/etc/kea-ha-tls/kea-ha.pem |
kea-ha on fzymgc-house/v1/kea-ha/v1 |
kea-ha-tls |
kea-dhcp4, which reads it only at start |
Each certificate lives 30 days, and the agent renews it at about day 15. The node worker has its
own Vault Agent and bundle, from the role dns-node-worker. The user vault-cert-agent is a
member of otelcol-dns and kea-ha-tls only. The Kea credentials and the D2 TSIG secrets have the
group _kea, so the agent cannot read them. kea-ha-tls has the member _kea, so Kea reads the
HA bundle.
The Kea HA certificate carries the HA address of the node as its IP SAN. The sub-CA of its mount
permits only the HA addresses and the HA names of the two nodes
(docs/operations/certificates.md, The Kea HA issuer).
The restart of Kea after a renewal
Section titled “The restart of Kea after a renewal”kea-ha-cert-reload.timer runs every 15 minutes on each node. Its service restarts
isc-kea-dhcp4-server.service only when all of these conditions hold:
- The bundle is newer than the start of Kea.
- The node is the Kea standby, or it is the primary and the bundle is at least two hours old.
- Both nodes report
hot-standbyinstatus-get, the partner isin-touch, andcommunication-interruptedis false.
The two-hour wait puts the standby first when both certificates renew together, so the primary
serves DHCP during the first restart. Kea keeps in-touch true after it loses the partner, so the
service reads communication-interrupted as well. If a condition does not hold, the service exits
0 and prints the reason. After a restart, it waits up to five minutes for both nodes to report
hot-standby. If they do not, the unit fails, and its journal row reaches ClickStack.
To read the next run of the timer and the last decisions on both nodes:
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \ -a 'systemctl list-timers kea-ha-cert-reload.timer --no-pager; journalctl -u kea-ha-cert-reload.service -n 5 --no-pager </dev/null'The first converge of a node on the agent
Section titled “The first converge of a node on the agent”Do one node per window, the Kea standby first. Read the role of each node with status-get
(Confirming the pair is peered).
- Run
scripts/dns-playbook.sh --limit <node> --tags kea-install. It creates the groupkea-ha-tlsand adds_keato it. The agent play needs that group. - Run
scripts/dns-playbook.sh --limit <node> --tags dns-otel-collector,vault-cert-agent,kea. The plays run in the order ofansible/dns-playbook.yml. The collector restarts on the new path at the end of its play and fails until the agent play writes its bundle. systemd starts it again every 10 seconds. Kea restarts on its bundle at the end of the Kea play. - Make sure that
status-getreportshot-standbyon both nodes before you start the other node.
After the first converge, a run of the whole playbook in its order works on the node.
The ruleset, and where SSH is accepted
Section titled “The ruleset, and where SSH is accepted”The nftables ruleset is policy drop. It permits udp and tcp destination port 53 on every
declared leg, udp destination port 67 on every leg that declares kea_subnet_id (see The DHCPv4 accept),
established and related, iif lo, VRRP, the tailnet accepts, and tcp destination port 22 on
every leg the inventory marks trusted: true, in both address families. The rule is part of the
ruleset with no flag of its own, so dns_common_nftables_enabled is the single switch. The role
installs the include into /etc/nftables.conf below Debian’s flush ruleset, because an include
above it is read and then wiped. Guest and Telework get no SSH path. Source-address matching is
not an authentication control. SSH’s own authentication protects the host.
The IPv4 source set is dns_common_ssh_source_set_v4, because 192.168.16.0/20 covers both the
Main and the MainWireless leg and nft refuses overlapping intervals in one anonymous set. The
IPv6 set derives each trusted leg’s own ULA /64.
| Leg | v4 source entry | v6 source entry |
|---|---|---|
| core | 192.168.40.0/22 |
fddb:f665:73f7:6::/64 |
| main, mainwireless | 192.168.16.0/20 |
fddb:f665:73f7:1::/64, fddb:f665:73f7:7::/64 |
| lab | 192.168.200.0/22 |
fddb:f665:73f7:4::/64 |
| iot | 192.168.218.0/24 |
fddb:f665:73f7:3::/64 |
Reach a node at any trusted leg address. sandiego.core.fzymgc.house resolves to the Core address
alone. ansible/inventory/hosts.yml gives each host its own Core address as ansible_host, never
the .5 VIP.
The rate limit is an nftables meter
Section titled “The rate limit is an nftables meter”The ruleset limits the queries of one client address to 100 a second, with a burst of 200, in
two meter rules above the conntrack accept: dns_rate_v4 keyed on the whole IPv4 source address
and dns_rate_v6 keyed on the whole IPv6 source address, so a /32 and a /128 each get their
own bucket. Loopback is outside the meter, so the keepalived checks never hit it. The values are
dns_common_dns_rate_limit and dns_common_dns_rate_burst in
ansible/roles/dns-common/defaults/main.yml. Until 2026-10-14 the limit was the rateLimit of
Blocky with the same values. The recursor has no limit per client, so the meter took its place.
The meter sits above the conntrack accept on purpose. A client that reuses one socket makes its
flow established after the first answer, and a meter below that accept would never see its
second query. The counter of each meter rule reads how many queries it dropped:
# On a resolver node.sudo nft list chain inet dns_resolver input | grep dns_rate# want: two rules. The `counter packets N` of each one is the number of dropped queries.A counter that moves names a client above the limit. The nftables ruleset logs no address, so read the query rows of that minute in ClickStack to find the client.
Ping, neighbour discovery, and what ICMP the resolver answers
Section titled “Ping, neighbour discovery, and what ICMP the resolver answers”echo-request is accepted on every declared leg, all seven, in both address families, with no
source scope. The invariant is ping surface equals DNS surface, held by the two echo rules sharing
the port-53 accepts’ loop expression in ansible/roles/dns-common/templates/nftables-dns.conf.j2.
Echo is not accepted on the tailnet leg. SSH does not share this scope: echo on seven legs, SSH on
the five trusted ones.
IPv6 Neighbour Discovery, MLD (both versions) and the ICMP error types are accepted above the
invalid-state drop. NDP is not valid-new for nf_conntrack_icmpv6, so an accept placed below
ct state invalid drop is dead code. Without these rules the node becomes IPv6-unreachable one RA
lifetime after the ruleset lands. packet-too-big carries IPv6 PMTU discovery. The DHCPv6 client
port (udp 546) is accepted from link-local sources only.
What the dynamic namespace publishes
Section titled “What the dynamic namespace publishes”The dynamic namespace beneath lan.fzymgc.house carries A and PTR records that Kea’s D2 writes
from leases. It carries no AAAA records. Nothing publishes the IPv6 address of a DHCP client. See
The DDNS tier.
The upstream of the recursor is the daemon on loopback
Section titled “The upstream of the recursor is the daemon on loopback”forward_zones_recurse of both processes carries one entry, the zone . with the forwarder
127.0.0.1:5300, which is dnscrypt-proxy on the same node. There is no second forwarder and no
direct resolver, and dnssec.validation is off, so the recursor never asks a root server. The
role reads the forwarder from dns_anonymizing_upstream_listen in
ansible/inventory/group_vars/dns_hosts.yml, because a direct resolver would see the home
address the first time the relays fail.
When the daemon has no live route, a public name answers SERVFAIL after about 6 seconds. When
the daemon is stopped, it answers SERVFAIL in about 2 milliseconds. Both are the intended
failure mode: availability before anonymity, never the other way round.
These SERVFAIL answers apply only to a name that is not in the cache. A name in the cache gets
a stale answer, an expired record that the recursor serves when a fetch fails. Both processes set
recordcache.serve_stale_extensions to 120, so a record stays in service for up to one hour, in
30-second steps. The recursor tries to fetch the record again at each step. Both processes also
set recordcache.refresh_on_ttl_perc to 10. A record with 10 percent or less of its TTL left is
fetched again in the background, so a client does not wait for that fetch. The two values are in
ansible/roles/pdns-recursor/defaults/main.yml.
On 2026-10-19, four of the five resolvers dropped about one query in five for
codeload.github.com, a name with a 60-second TTL. Before the two settings, each drop reached
the client as SERVFAIL after the 1.5-second network timeout of the recursor.
Until 2026-10-06 the upstream was Control D over DoT, in two entries for the two address
families. The ADR docs/adr/2026-10-06-pair-resolves-public-names-through-anonymized-relays.md
records the change.
Ask the node itself, never the control node:
uv run scripts/dns-external-resolution.py \ --ssh fzymgc@192.168.40.6 --resolver 127.0.0.1 --decoy 192.168.40.99The --decoy probe is mandatory. It aborts the run as GATE INVALID (exit 4) if anything answers
for an address where nothing exists.
The anonymizing upstream, dnscrypt-proxy
Section titled “The anonymizing upstream, dnscrypt-proxy”Each node runs one dnscrypt-proxy as the system user dnscrypt-proxy, on two loopback
listeners, UDP and TCP: 127.0.0.1:5300 for both recursor processes, and 127.0.0.2:53 as the
node’s own fallback resolver. The client tier forwards every public name to it since
2026-10-06, through the OpenSpec change pair-drops-control-d. The two addresses are the
inventory values dns_anonymizing_upstream_listen and dns_node_fallback_listen, which the
roles that use them read too.
The daemon speaks Anonymized DNSCrypt. It encrypts the query for the resolver and sends it to a
relay. The relay sees the home address and no query. The resolver sees the query and the address
of the relay. The route set is in ansible/roles/dnscrypt-proxy/defaults/main.yml:
| Resolver | Operator | Relays | Relay operator |
|---|---|---|---|
cs-dc, cs-nyc |
CryptoStorm | dnscry.pt-anon-ashburn-ipv4, dnscry.pt-anon-newyork-ipv4 |
dnscry.pt |
dnscry.pt-ashburn-ipv4, dnscry.pt-newyork-ipv4 |
dnscry.pt | anon-cs-dc, anon-cs-nyc |
CryptoStorm |
plan9dns-nj |
plan9 | anon-cs-dc, dnscry.pt-anon-ashburn-ipv4 |
both |
Every resolver has a route, and the render fails for a resolver without one or for an empty
server list. skip_incompatible is on, so a resolver that cannot be reached through its relays
is dropped rather than reached directly, and direct_cert_fallback is off, so the certificate
of a resolver is fetched through the relay too. When no route works, the daemon answers nothing,
and the recursor answers SERVFAIL.
The daemon refuses the internal zones. /etc/dnscrypt-proxy/blocked-names.txt holds
fzymgc.house, 168.192.in-addr.arpa and 7.f.3.7.5.6.6.f.b.d.d.f.ip6.arpa, rendered from
dns_zone_domain and dns_reverse_zones. A bare zone matches the zone and every name under it.
The daemon answers REFUSED before any upstream step, so no relay sees the name. The main
recursor process never asks for these names, because it forwards them to pdns. The public process
never asks either, because its withheld zone answers NXDOMAIN first. The node’s own fallback path
does ask, and REFUSED is what it gets. The decision is
dnscrypt-proxy refuses the internal zones.
The set holds no ODoH server. dnscrypt-proxy fetches the ODoH configuration of a target from the
target itself, not through the relay, at each start and every 4 hours, so the target learns the
home address. The route odoh-cloudflare through odohrelay-crypto-sx works from this network
and stays out for that reason.
A security review of dnscrypt-proxy 2.1.18 on 2026-10-06 gave the verdict use with mitigations.
The role applies each one: explicit server names, no DoH servers, skip_incompatible, no direct
certificate fallback, require_nolog, classic DNSCrypt with a fresh client key for each query
instead of the post-quantum mode (which reuses one ciphertext and lets a resolver group the
queries of one client), no query log, no monitoring UI, an own hardened unit, and absolute cache
paths in a cache that the daemon account owns and a configuration directory that it cannot
write.
The daemon reads the resolvers and the relays from four signed lists in
/var/cache/dnscrypt-proxy, and refreshes them every 72 hours. The converge seeds the lists, so
a first start reads from disk. The few host names the daemon needs before a route is live, such
as the ODoH relay, resolve through plain Quad9. Those two resolvers see those host names and no
other query.
The daemon caches nothing and logs no query. The recursor does both in front of it. Its journal is in
ClickStack under the ServiceName dnscrypt-proxy, through dns-otel-collector. The rows hold
no client address, so the row filter of the two MCP users does not cover them.
Two HyperDX alerts read those rows, on the dashboard DNS — resolver alerts:
| Alert | Fires when | Shape |
|---|---|---|
DNS dnscrypt-proxy has no live route |
a node logs waiting for at least one server to be reachable in a 5-minute window |
Once at the start of an outage. The daemon writes that line one time |
DNS dnscrypt-proxy resolvers timing out |
a node logs more than 60 resolver TIMEOUT lines in a 5-minute window |
In every window while the outage lasts. Five dead resolvers log about 140 lines in 5 minutes, one dead resolver about 30 |
Both were proved on 2026-10-06 with an nftables table on the passive node that dropped the UDP of the daemon account, and a restart of the daemon. The first alert fired in the window of the restart and cleared in the next. The second fired in each window until the table left. To repeat that break on a node that nothing uses yet:
# On a resolver node, as root. Reverse with `nft delete table inet dcp_break` and a restart.nft add table inet dcp_breaknft add chain inet dcp_break out '{ type filter hook output priority 0; }'nft add rule inet dcp_break out meta skuid "dnscrypt-proxy" meta l4proto udp counter dropsystemctl restart dnscrypt-proxyWhile no route is live, a public name that is not in the cache answers SERVFAIL after 6 seconds
on that node. A name in the cache gets a stale answer for up to one hour. A
stopped daemon answers SERVFAIL in 2 milliseconds, so systemctl stop dnscrypt-proxy is the
first move on a node whose routes are all dead.
The role removes the Debian package dnscrypt-proxy when it finds it, because the package shares
the unit name and the configuration path. The role runs six criteria at the end of each converge: the daemon account reads the
configuration (DCP-01), the daemon’s own parser accepts it (DCP-02), the unit is active as that
account (DCP-03), the journal of the current start names a relay for each resolver and reports
every resolver live (DCP-04), each loopback listener answers example.org (DCP-05), and each
loopback listener refuses a name under each internal zone (DCP-06).
To read the route state on a node:
# On a resolver node.sudo journalctl -u dnscrypt-proxy -b --no-pager | grep -E 'Anonymizing queries for|live servers|reached anonymously'To change the route set, edit dnscrypt_proxy_servers and dnscrypt_proxy_routes in the role
defaults and converge --tags dnscrypt-proxy one node at a time. Pick a relay from
relays.md or odoh-relays.md in the cache, and never one that the operator of the resolver
runs.
The release is proved by its minisign signature with the release key of the project, and the lists by the list key. The two keys differ, and both are in the role defaults. The release key is on the page Installation of the dnscrypt-proxy wiki.
Running a play, and how one-node-per-window is expressed on the command line
Section titled “Running a play, and how one-node-per-window is expressed on the command line”Use the wrapper. It materializes credentials and pins the inventory:
scripts/dns-playbook.sh --limit sandiego --tags pdns-verify --check --diff # dry-run, one nodescripts/dns-playbook.sh --limit sandiego --tags pdns-recursor # one node, one tierscripts/dns-playbook.sh --limit sandiego --tags pdns-recursor-verify # the read-only checks of that tierscripts/dns-playbook.sh --limit portland --tags kea -vv # verboseAnsible does not fail on an unmatched tag. It selects zero tasks and prints a green PLAY RECAP.
Make sure that the tag is in ansible/dns-playbook.yml before you trust a green recap.
ansible/dns-playbook.yml is eleven separate plays, one role each, every role tagged: dns-common,
pdns, lightningstream, dnscrypt-proxy, pdns-recursor, dns-otel-collector,
vault-cert-agent, kea, keepalived, dns-tailscale and dns-node-worker, in that dependency
order. Each role also
exposes a -verify tag that selects only its read-only assertions, such as
pdns-recursor-verify.
Never change both resolver nodes in the same window. The estate has exactly two, and a change
applied to both at once leaves no surviving node. --limit sandiego --tags pdns-recursor is the
expressible form of that rule. --limit reaches the node it names because ansible_host is each
host’s own Core address, never the VIP.
The cut-over to the recursor, one node per window
Section titled “The cut-over to the recursor, one node per window”The change dns-policy-as-rpz moved the client tier from Blocky to PowerDNS Recursor on
2026-10-14. The cut-over of one node is a converge of six tags, then their six -verify tags.
Run it on the node that holds no VIP first. scripts/dns-vip-holder.py names the holder, and
--passive names its peer.
-
If this node holds a VIP, drain it, so its peer holds every VIP during the converge. Use the MCP tool
dns_drain_nodeof dns-control, or follow Drain and undrain through dns-control. The node that holds no VIP needs no drain. -
Converge the six roles on this node:
Terminal window scripts/dns-playbook.sh --limit <node> --tags dns-common,pdns,pdns-recursor,dns-node-worker,dns-otel-collector,keepalivedThe
pdns-recursorrole starts both recursor processes on the addresses that Blocky used. The role holds no Blocky task. On a node that still runs Blocky, stop and disableblocky.serviceandblocky-public.servicebefore this step, because both tiers bind the same addresses. -
Make sure that every feed zone holds records. The converge reports the exit code of the first feed run, and the journal names each group with its count:
Terminal window scripts/dns-adhoc.sh <node> -b -m ansible.builtin.shell -a 'journalctl -u dns-feeds -o cat | grep records='If a line reads
unexpected arguments: feed, the worker binary on the node was older than release 0.23.0 when thepdns-recursorplay ran, because that play comes before thedns-node-workerplay. Startdns-feeds.serviceonce more after the converge. If a group readsrecords=0, the source answered an error page with the status 200, and the CDN caches that page for hours. Copy the zone file of that group from the peer node into/var/lib/pdns-recursor-feeds, with the ownerpdns, and rundns-feeds.serviceagain. The unit reloads both processes and wipes their caches, so a name that the feed lists gets its block at once. -
Run the read-only checks of the same six roles:
Terminal window scripts/dns-playbook.sh --limit <node> --tags dns-common-verify,pdns-verify,pdns-recursor-verify,dns-node-worker-verify,dns-otel-collector-verify,keepalived-verify -
If this node was drained, undrain it with
dns_undrain_node. -
Take the readings of How to prove what an untrusted leg sees against this node, with
--server <address>on the script of the skill.
Then do the same on the other node in the next window. Until the retire play runs, the Blocky units are the rollback of a node:
scripts/dns-adhoc.sh <node> -b -m ansible.builtin.shell -a 'systemctl disable --now pdns-recursor@main pdns-recursor@public && systemctl enable --now blocky blocky-public'Retiring Blocky
Section titled “Retiring Blocky”After both nodes ran the recursor for one day, the retire play ansible/blocky-retire.yml removes
Blocky from a node. It is not in dns-playbook.yml. Run it by hand, one node per window, the node
that holds no VIP first:
scripts/dns-playbook.sh --limit <node> --playbook=blocky-retire.ymlThe play stops and disables blocky.service and blocky-public.service, and removes the binary
/usr/local/bin/blocky, the two unit files, /etc/blocky, /var/cache/blocky and the operator
files directory /var/lib/dns-worker/lists. It leaves /var/lib/dns-worker itself, which holds
the drain marker, the scope map and the Kea reservation files, and it leaves the blocky user
and group. A second run reports no change. The readback on the node:
# On a resolver node.systemctl is-active blocky blocky-public # want: no line reads activetest -e /etc/blocky; echo "RC=$?" # want: RC=1After the retire play, the rollback of a node is a revert of the cut-over pull request and a
converge of --tags blocky on that node.
The timed auto-revert
Section titled “The timed auto-revert”dns-common ships a dead-man switch. dns-common-revert.timer starts dns-common-revert.service
a fixed interval after it is armed. That service unmasks and starts NetworkManager, masks and stops
systemd-networkd, and stops nftables, which puts the node back on the network manager it was
reachable through before the converge. Arm it before a change that can strand you. Disarm it once
you can reach the node.
Run it against one node at a time. These commands are not read-only. The session that issues the hand-run apply in the recovery below does not survive it. Reconnect on the node’s Core address.
NODE=sandiego # the ONE node being worked onNODE_IP=192.168.40.6 # that node's CORE address: sandiego .6, portland .7. Never the .5 VIP.Arm it:
scripts/dns-playbook.sh --limit "$NODE" --tags dns-common \ --skip-tags dns-common-venv,dns-common-daemons \ -e dns_common_revert_enabled=true -e dns_common_revert_delay=600s# want: PLAY RECAP ... changed=2 unreachable=0 failed=0# The two changed tasks render the timer unit and set its state. revert-arm.yml is the last# include, after verify.yml, so a converge that fails earlier never arms anything.# 600s is the production delay. Pass both values as -e, never by editing dns_hosts.yml: an# armed flag left true in a commit is an outage on somebody else's next converge.
sudo systemctl start dns-common-revert.timer# A hand re-arm. OnActiveSec counts from THIS start.Disarm it, in an already-open session, with the read and the stop in one invocation:
sudo systemctl stop dns-common-revert.timer# A stop clears the runtime half only. The unit file stays enabled with its rendered delay, so a# reboot re-arms it. A converge with dns_common_revert_enabled at its false default clears that.Read it. Read SubState, never is-active:
sudo systemctl show dns-common-revert.timer \ -p ActiveState -p SubState -p LastTriggerUSec -p NextElapseUSecMonotonic# want (armed): ActiveState=active SubState=waiting LastTriggerUSec= (empty)# want (fired): ActiveState=active SubState=elapsed LastTriggerUSec=<a stamp># Keep the keys. --value strips them, and systemd emits properties in its own order.# NextElapseUSecRealtime is empty on this timer, because OnActiveSec= is monotonic.| Timer state | SubState |
LastTriggerUSec |
ActiveState |
|---|---|---|---|
| Armed, has not fired | waiting |
empty | active |
| Fired, not stopped | elapsed |
set | active |
| Fired then stopped | dead |
empty | inactive |
| Disarmed, then stopped | dead |
empty | inactive |
The last two rows are identical. After any stop, journalctl -t dns-common-revert is the only
instrument that discriminates, so capture it before you disarm. The timer sets no AccuracySec=,
so the real window is OnActiveSec through OnActiveSec + 60s.
A complete fire writes one opening line, five ACTION-OK lines in manifest order (unmask and start
NetworkManager, mask the networkd socket, mask the networkd service, stop nftables) and one
DECISION=REVERT-COMPLETE line. Read the order by index. The exit status is 0 when every action
returned 0, 64 for an unrecognised argument, 78 for an empty systemctl command, and otherwise
the number of actions that did not succeed, each named under DECISION=REVERT-INCOMPLETE.
Stopping systemd-networkd does not release its addresses or destroy its VLAN netdevs. After a
fire the static addresses and the VLAN netdevs remain present and UP, and NetworkManager adds one
DHCP address of its own. A reverted node stays reachable at its inventory address.
A fired revert leaves the node with no packet filter. nftables.service is stopped, so the node is
default-allow on every VLAN it holds a leg on until a converge restores the ruleset. Close it
deliberately.
The converge refuses on a reverted node, and the refusal is the only detection surface a fired
revert has. Do not add masked: false to that task:
scripts/dns-playbook.sh --limit "$NODE" --tags dns-common \ --skip-tags dns-common-venv,dns-common-daemons# want, against a reverted node:# TASK [dns-common : Set systemd-networkd's enabled state from its guard]# Unable to enable service systemd-networkd.service: ... is masked# PLAY RECAP ... unreachable=0 failed=1Get the node back. Apply first, converge second. No Ansible verb on the converge path lifts a systemd mask:
# 1. Lift the masks by hand. Detached, because the session does not survive it.ssh -o ControlPath=none fzymgc@"$NODE_IP" \ 'sudo -n systemd-run --collect --unit=dns-common-network-apply-recover \ /usr/local/sbin/dns-common-network-apply.sh'# want: rc=0 and `Running as unit: dns-common-network-apply-recover.service`
# 2. Reconnect on Core, then confirm both masks are gone. Count, never a bare ls.ssh -o ControlPath=none fzymgc@"$NODE_IP" \ 'ls -1 /etc/systemd/system/systemd-networkd.service /etc/systemd/system/systemd-networkd.socket \ 2>/dev/null | wc -l'# want: 0
# 3. Converge. This re-enables networkd, restarts nftables, restores OnActiveSec=600s and# disarms the timer on both axes.scripts/dns-playbook.sh --limit "$NODE" --tags dns-common \ --skip-tags dns-common-venv,dns-common-daemons# want: PLAY RECAP ... unreachable=0 failed=0
# 4. Confirm the production delay is back.ssh -o ControlPath=none fzymgc@"$NODE_IP" \ "systemctl cat dns-common-revert.timer | grep -E '^OnActiveSec='"# want: OnActiveSec=600sjournalctl -t dns-common-network-apply logs every action in manifest order. A complete run ends
DECISION=HANDOVER-COMPLETE.
A residual NetworkManager lease needs no step. It lands in Core on the same eth0 as the static
.6 or .7 and expires on its own. If you chase odd source-address or firewall behaviour on a
freshly recovered node, read it as a count:
ssh -o ControlPath=none fzymgc@"$NODE_IP" 'ip -o -4 addr show dev eth0 | wc -l'# want: 1, the static Core address alone. 2 means the lease is held.Three /dev/null mask symlinks in /etc/systemd/system/ is the correct steady state:
NetworkManager.service, unbound-resolvconf.service and userconfig.service. Neither networkd
unit is among them. To assert the directory is unchanged, compare a name-only digest
(ls -1 /etc/systemd/system/ | LC_ALL=C sort | sha256sum).
ULA prefix announcement (the resolver pair)
Section titled “ULA prefix announcement (the resolver pair)”Both resolvers advertise the ULA prefix on every leg, permanently, at a finite lifetime
(dns_common_ra_announce_enabled: true in ansible/inventory/group_vars/dns_hosts.yml).
What it is
Section titled “What it is”dns-common-ra-announce.service runs radvd as a standing daemon on both nodes, reading a
role-owned config at /etc/dns-common/radvd-announce.conf rendered from dns_vlan_interfaces, one
interface stanza per leg. Each leg advertises its own fddb:f665:73f7:N::/64. The values are in
ansible/roles/dns-common/defaults/main.yml:
| Directive | Value | What it buys |
|---|---|---|
AdvOnLink |
on |
the client forms an on-link route to that leg’s ULA VIP |
AdvAutonomous |
on |
the client can configure an address from the prefix. With it clear a client ignores the whole prefix-info option (RFC 4862 §5.5.3(a)) |
AdvPreferredLifetime |
3600 |
finite |
AdvValidLifetime |
7200 |
finite. The ceiling on how long a withdrawal takes |
AdvDefaultLifetime |
0 |
neither resolver becomes a default router. On the wire, router_lifetime=0 |
MinRtrAdvInterval / MaxRtrAdvInterval |
200 / 600 |
the unsolicited cadence. The join path is the solicitation |
RDNSS |
the leg’s vip6, lifetime 1800 |
gated by dns_common_ra_announce_rdnss_enabled alone. The RA of the gateway carries the same VIP on every leg, so the two sources agree (RFC 8106 §5.3.1) |
The pidfile is /run/dns-common-ra-announce.pid. See The pidfile below.
Without this announcement nothing on the trunk advertises the prefix the floating addresses live in, and a client handed a ULA VIP has no route to it.
A standing daemon rather than bursts, because only a listening daemon answers a Router Solicitation
inside RFC 4861’s MAX_RA_DELAY_TIME. That is how a fresh client joins in under a second, and it is
what makes scripts/solicit-ra.py --router a valid instrument. Both nodes, because prefix
information is idempotent, and the estate survives one node down with no failover logic.
How to read whether it adopted
Section titled “How to read whether it adopted”A zero exit from systemctl start is not the proof. The proof is ExecMainStartTimestamp in
wall-clock epoch seconds, strictly later than the config file’s mtime:
# On a resolver node.SVC=dns-common-ra-announce.serviceCONF=/etc/dns-common/radvd-announce.confTS=$(systemctl show "$SVC" -p ExecMainStartTimestamp --value)echo "START_EPOCH=$(date -d "$TS" +%s)"echo "CONF_MTIME_EPOCH=$(stat -c %Y "$CONF")"systemctl show "$SVC" -p ActiveState -p SubState -p MainPIDradvd -c -C "$CONF" -p /run/radvd-configtest.$$.pid ; echo "configtest=$?"Want: START_EPOCH strictly greater than CONF_MTIME_EPOCH, active / running, configtest 0.
Never compare ExecMainStartTimestampMonotonic against an mtime. It is microseconds since boot.
How to read it on the wire
Section titled “How to read it on the wire”From the repository root, one dispatch per node, each attributed to the peer. A node does not receive its own multicast, so a node asked to attribute to itself reads UNINTERPRETABLE:
scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.script -a "../scripts/solicit-ra.py --router fe80::8aa2:9eff:fede:ae7c" # attributes to portlandscripts/dns-adhoc.sh portland -b -m ansible.builtin.script -a "../scripts/solicit-ra.py --router fe80::8aa2:9eff:fed4:6167" # attributes to sandiegoThose two link-locals are each node’s, identical across the legs, recorded beside
dns_common_ra_watch_expected_lladdrs in ansible/inventory/group_vars/dns_hosts.yml.
Three dispatch traps:
- The
../prefix is required.ansible.builtin.scriptresolves a relative path against the playbook directory (ansible/), not the repository root. -bis required. The script opens a raw ICMPv6 socket.- A non-zero exit prints the host FAILED in red, and that is the gate working. Exit
1is measured and absent. Exit2is uninterpretable, and no verdict is claimed for the set.
Want, per node: rc 0 and SUMMARY: <n> advertising, 0 not advertising, 0 uninterpretable, of <n> legs, where <n> is the number of legs in dns_vlan_interfaces.
The exit status is a prefix-presence verdict only. A claim about the lifetimes must be read from
the printed vlt= and plt= fields. Because both nodes announce, an unscoped count measures the
link, not the node. Count only inside blocks whose printed source is the node under test:
# WRONG for a per-node claim: it counts the peer's frames too.grep -o 'vlt=7200 plt=3600' sandiego.out | wc -l
# RIGHT: count only inside blocks whose printed source is the node under test.awk -v tgt=fe80::8aa2:9eff:fed4:6167 ' /^ *from / { intgt = ($2 == tgt); next } /^=== / { intgt = 0; next } intgt && /vlt=7200 plt=3600/ { n++ } END { print n+0 }' sandiego.outWant: one per leg. Keep a sanity count (grep -c 'vlt=') beside it. Also want, inside the
target’s blocks: L=1 A=1 on every prefix-info line, router_lifetime=0 on every block, and an
rdnss line carrying the leg’s vip6.
How to check containment
Section titled “How to check containment”scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell -a "systemctl is-enabled radvd.service ; ps -o pid,unit,cmd -C radvd --no-headers || echo 'no radvd process'"Want: systemctl is-enabled radvd.service prints masked on both nodes (Debian’s own unit, masked
whenever the package is present), and every row of ps shows a role unit in the UNIT column
(dns-common-ra-announce.service or dns-common-ra-deprecate.service).
Two concurrent radvd is compliant: the standing announcer plus a bounded deprecation window. A
check written as a process count equal to 1 is red on a correct estate. radvd forks, so even a
single announcer shows two rows sharing one unit. Count by unit.
The pidfile
Section titled “The pidfile”The announcer runs with -p /run/dns-common-ra-announce.pid. radvd takes an exclusive lock on its
pidfile, and the bounded dead-prefix emitter passes no -p, so it takes the compiled-in default.
If the announcer used that default, the emitter’s OnFailure=-triggered start would exit 255 and
the dead-prefix withdrawal would never be sent.
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell -a "ls -l /run/dns-common-ra-announce.pid /run/radvd.pid /run/radvd/radvd.pid 2>&1 ; systemctl is-failed dns-common-ra-deprecate.service ; systemctl show dns-common-ra-deprecate.service -p Result -p ExecMainStatus"Want: the announcer’s own pidfile present, the two compiled-in defaults absent, the emitter not
failed, with Result=success and ExecMainStatus=0. inactive (dead) is the correct resting
state for a bounded one-shot. The journal signature of a collision:
journalctl -u dns-common-ra-deprecate.service --no-pager | grep -i 'lock pid file'The same lock is why radvd -c needs its own pidfile. Pass a scratch path, as the adoption read
above does.
How to disarm, and what withdrawal means
Section titled “How to disarm, and what withdrawal means”Set dns_common_ra_announce_enabled: false and converge, one node per window:
scripts/dns-playbook.sh --limit portland --tags dns-common-ra-announce# verify, then the same command with --limit sandiegoThe arm is two-sided. ra-announce-arm.yml derives both enabled and state from that one flag,
so the converge disables and stops the unit. Keep the flag committed true: with it false, any
routine --tags dns-common converge stops both announcers.
There is no withdrawal code. The prefix ages out within the advertised valid lifetime, at most two hours. radvd’s shutdown-deprecation prefix option is off and must stay off: it is safe only when a single router announces the prefix, and here two do.
How to withdraw the RDNSS advertisement
Section titled “How to withdraw the RDNSS advertisement”This withdraws the recursive-DNS advertisement only. All three steps are required. A converge alone changes nothing on the wire:
# 1. Override the arm flag off, one node per window. Use the JSON form.scripts/dns-playbook.sh --limit portland --tags dns-common-ra-announce \ -e '{"dns_common_ra_announce_rdnss_enabled": false}'
# 2. Confirm the rendered config lost the blocks (expect 0).scripts/dns-adhoc.sh portland -b -m ansible.builtin.shell \ -a 'c=$(grep -c "^ *RDNSS " /etc/dns-common/radvd-announce.conf || true); echo "rdnss_block_count=$c"; test "$c" -eq 0'
# 3. Restart the announcer. Without this the wire does not change.scripts/dns-adhoc.sh portland -b -m ansible.builtin.shell \ -a 'systemctl restart dns-common-ra-announce.service'To restore, run the same three steps without the -e override and restart again. The restart is
required in both directions: the conf-render task carries no notify:, the unit declares no
ExecReload=, and no role handler names the announcer. -e key=value passes a string, and the
guard applies | bool, so the JSON form is belt and braces.
The prefix survives the withdrawal. After the restart every leg reads rdnss (0): NONE with the
prefix-info line unchanged. scripts/solicit-ra.py’s exit code covers the prefix, not the RDNSS.
Read the printed option.
How to restart
Section titled “How to restart”Plan-sequenced, never a role handler. The unit carries no Restart= policy, and the role installs
no handler, following the same rule as keepalived: a converge must never be the thing that moves
the serving plane. A radvd that has exited is a failed unit an operator reads.
Dead-prefix deprecation — when the ISP re-delegates
Section titled “Dead-prefix deprecation — when the ISP re-delegates”A re-delegation can leave the old prefix live on every client. The Firewalla that was the gateway before 2026-10-01 did this. FireRouter’s dnsmasq stopped mentioning the old /64 instead of withdrawing it with preferred lifetime 0 (RFC 4862 §5.5.3). Nobody has measured whether the UCG-Fiber does the same, so the detector and the deprecation stay armed.
The failure to expect is a client holding a dead prefix for its remaining valid lifetime while every unit reports healthy. RFC 6724 source selection keeps choosing the dead prefix, so the client’s IPv6 fails while a probe forced onto the new prefix succeeds. Compare the default source address against one pinned to the current prefix before concluding anything about the gateway.
What detects, and how long it takes to notice
Section titled “What detects, and how long it takes to notice”dns-common-ra-watch.service runs one capture per run (dns_common_ra_watch_window_seconds, 900 s)
every dns_common_ra_watch_interval (6 h). It harvests prefix-information options from the frames
of the gateway only, per VLAN. A gateway frame matches both the link-local
fe80::76f9:2cff:fed8:fe0c (dns_common_ra_watch_appliance_lladdr) and the MAC
74:f9:2c:d8:fe:0c (dns_common_ra_watch_appliance_mac), in
ansible/roles/dns-common/defaults/main.yml. The detector classifies each observed prefix by
family: global under 2000::/3,
unique-local under fc00::/7, RFC 3849 2001:db8::/32 as documentation, anything else other.
Three journal literals carry it:
| Literal | Emitted when |
|---|---|
RA-WATCH-PREFIX |
one per distinct observed prefix, with vlan, prefix and class |
RA-WATCH-PREFIX-BASELINE |
first observation for a VLAN, no recorded state. Never an action |
RA-WATCH-PREFIX-CHANGE |
an in-scope global prefix changed. This is the ClickStack alert’s subject |
Detection latency is up to one full cadence plus one capture window. Shortening it is one role
default away, and it is not done: a re-delegation is rare, and the capture runs tcpdump on the
two nodes that are the entire DNS serving plane.
A ULA change is reported and never acted on. Only the global class participates in change detection.
dns_otel_journal_units in ansible/roles/dns-otel-collector/defaults/main.yml names
dns-common-ra-watch.service. Removing it un-arms the alert: the detector’s output reaches
default.otel_logs nowhere, and an alert over it is green forever. The collector does not re-read
its configuration on its own. The role carries a restart handler. If you edit that configuration
outside Ansible, restart the unit yourself.
What announces
Section titled “What announces”dns-common-ra-deprecate.service runs dns-common-ra-deprecate.sh, which emits a bounded
deprecating RA with radvd:
- Preferred lifetime 0 is the whole fix. RFC 4862 §5.5.3(e) makes a host ignore a reduction of the valid lifetime below two hours unless the RA is authenticated, so the valid lifetime is set above the clamp and the preferred lifetime is the field that deprecates.
- Router lifetime 0 on every stanza. It deprecates without claiming to be a default gateway.
- Bounded.
--announceruns undertimeoutfordns_common_ra_deprecate_duration_secondsand stops.timeout’s exit 124 is the expected end, and the unit declares it a success. - Never Debian’s own
radvd.service, and never/etc/radvd.conf. Writing that path arms a second, unbounded, boot-persistent radvd. The converge masks the Debian unit whenever the package is present. - No recursive-DNS-server option. A resolver-list change on a transient burst would make a client’s DNS depend on whether it was listening during those minutes. The standing announcer carries RDNSS.
Expect our own frames in the peer’s detector output. While an announcement runs, the other resolver
sees an RA source other than the gateway, with router lifetime 0s and no RDNSS. That lands in the
RA-WATCH-OBSERVED bucket by membership of the expected-source set.
The two flags
Section titled “The two flags”| Flag | False | True |
|---|---|---|
dns_common_ra_prefix_watch_enabled |
the detector prints RA-WATCH-PREFIX* and records prefix state. Pure observability |
the exit-3 path and the deprecation request file are live |
dns_common_ra_deprecate_enabled |
no radvd installed. /etc/dns-common/ra-deprecate.armed is absent, so the emitter unit is skipped |
radvd installed, arm file present, the emitter can announce |
The second is a file, not a unit state. dns-common-ra-watch.service carries
OnFailure=dns-common-ra-deprecate.service, and that fires on every non-zero detector exit. An
unsatisfied ConditionPathExists= is a skip rather than a failure, which is what makes that
trigger safe.
Bring-up order
Section titled “Bring-up order”One node per window, the non-holder first. Verify the first node before touching the second:
scripts/dns-playbook.sh --limit portland --tags dns-common-ra-watch --check --diffscripts/dns-playbook.sh --limit portland --tags dns-common-ra-watch# verify, then the same two commands with --limit sandiegoBoth arms are two-sided. A converge with a flag false disarms rather than declining to arm.
systemctl start on a detector that is already running is a no-op, and a window that is already
running keeps the old script. Before reading a triggered run as evidence for a script change,
compare systemctl show dns-common-ra-watch.service -p ExecMainStartTimestamp --value against the
script’s mtime.
Which tag deploys what
Section titled “Which tag deploys what”--tags dns-common-ra-watch renders the whole detector tier: the detector script, both detector
units, the emitter script and the emitter unit. --tags dns-common-ra-announce runs only the radvd
install, the package stat and the mask of Debian’s unit. The announce tag does not deploy the
detector or the emitter. If you changed a detector or emitter template, converge the detector tag.
The check that settles which tasks a tag reaches runs on the control node:
cd ansible && ansible-playbook -i inventory/hosts.yml dns-playbook.yml \ --limit portland --tags dns-common-ra-watch --check --diff--list-tasks cannot expand include_tasks.
The expected-source set
Section titled “The expected-source set”Every RA-WATCH-OBSERVED line carries expected=yes or expected=no, set by membership of
dns_common_ra_watch_expected_lladdrs, which carries both resolvers’ link-local addresses. The
membership test runs above the RDNSS test, because the pair advertises its own RDNSS on every leg.
- An
expected=yesline carrying an address in itsrdnss=field is the pair announcing. - An
expected=nosource carrying an RDNSS on a trusted leg is a violation. It exits 2, landsdns-common-ra-watch.serviceinfailed, and is reported on aRA-WATCH-VIOLATIONline. - On an untrusted leg (Guest, Telework) the same frame is reported on a
RA-WATCH-UNTRUSTED-RDNSSline and does not exit 2. Those are segments we do not police. - The observed bucket’s count is reported as
non_appliance_non_violating=on theRA-WATCH-SUMMARYline.
The blind spot: the exemption keys on the source link-local alone, and a link-local is spoofable on-link. An attacker who spoofs one of the two pair addresses and advertises a hostile resolver is invisible to this detector. The detector reports and does not prevent. Do not widen the exemption.
The detector sees its own node’s advertisements, because tcpdump taps at the device layer. So
dns_common_ra_watch_expected_lladdrs must carry both nodes’ link-locals. Do not narrow that list
to a peer-only list.
The read is a cross-tab:
journalctl -u dns-common-ra-watch.service --since '<window start>' --until '<now>' --no-pager > /tmp/ra.txtgrep -o 'RA-WATCH-OBSERVED' /tmp/ra.txt | wc -lsed -n 's/.*src=\([^ ]*\).*expected=\([^ ]*\).*/\1 \2/p' /tmp/ra.txt | sort | uniq -cThe cross-tab counts must sum to the RA-WATCH-OBSERVED count. If they do not, the detector on
that node predates the field, and the fix is a --tags dns-common-ra-watch converge. No source
appears on both sides. An expected=no source that is not one of the long-standing non-resolver
sources is the change this annotation exists to surface.
Proving the announcement on demand
Section titled “Proving the announcement on demand”The deprecating RA is observable on demand against an RFC 3849 documentation prefix no client can
hold. The detector classes 2001:db8::/32 as documentation, so the rehearsal frame cannot reach
the deprecation path.
Before you start, on both nodes: systemctl is-active dns-common-ra-watch.service (whether a
capture is running), systemctl list-timers dns-common-ra-watch.timer --all --no-pager (when the
next one opens), and cat /var/lib/dns-common/ra-prefix.state (whether a global prefix is recorded
for the leg).
Do not run this rehearsal in a Kea cut window. To prove only that the self-deprecation applies
preferred_lft 0, use the procedure below, which transmits nothing.
# 1. A scratch config for ONE leg. No recursive-DNS-server directive.scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.shell -a 'cat > /tmp/probe.conf <<EOFinterface eth0{ AdvSendAdvert on; AdvDefaultLifetime 0; MinRtrAdvInterval 3; MaxRtrAdvInterval 4; prefix 2001:db8:dead:5f00::/64 { AdvOnLink on; AdvAutonomous on; AdvPreferredLifetime 0; AdvValidLifetime 7200; };};EOF'
# 2. Run radvd in the foreground for about 60 s and capture on the NAMED interface.# Never -i any: cooked-mode offsets differ, so ip6[40] matches nothing.scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.shell -a \ '(timeout 60 radvd --logmethod stderr -n -d 5 -C /tmp/probe.conf > /tmp/radvd.out 2>&1 &) ; sleep 3; timeout 30 tcpdump -i eth0 -e -nn -v -s 0 -w /tmp/ra.pcap "icmp6 and ip6[40] == 134"'
# 3. Read it back with -v. Options only print under -v.scripts/dns-adhoc.sh sandiego -b -m ansible.builtin.shell -a 'tcpdump -e -nn -v -r /tmp/ra.pcap'Assert all three on the frame sourced from this node’s own link-local: router lifetime 0s, a
prefix info option for the dead /64 with Flags [onlink, auto] and pref. time 0s, and no RDNSS
option on our frames. The frames of the gateway in the same capture carry one, which is the
positive control.
The 30 s window is sufficient here because the 3/4 s interval is ours. A capture of gateway frames
must run longer than the maximum RA interval of the gateway and be bounded by timeout, never by
-c N.
Clean up: kill radvd, remove the scratch config, apt-get purge -y radvd, and re-read the
forwarding sysctl.
Proving the self-deprecation on demand
Section titled “Proving the self-deprecation on demand”This proves that --deprecate-local fires and stamps preferred_lft 0 on this node’s own address
for a named prefix. It puts nothing on the wire.
The condition has to be manufactured. At rest the estate shows zero addresses under a dead prefix, because the same clock that made the prefix dead also removed the evidence. Two things make the reading mean anything:
- The antecedent. An address must exist under the named prefix, on the target leg, with a non-zero preferred lifetime, read before the invocation.
- The discriminator. Plant it with a preferred lifetime of an hour, invoke, and take the post-fire
read within a couple of minutes. Expiry cannot explain
preferred_lft 0secobserved a minute into a 3600-second lifetime.
Do not use the unit for this. systemctl start dns-common-ra-deprecate.service also runs
--announce. Invoke the script directly. Run it on the non-holder. <LEG> is eth0.1000 (Lab),
the lowest-traffic leg.
# 1. Pre-flight. Both counts must read 0. This output is also the recovery record.scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; ip -6 addr show scope global ; \ ip -6 addr show scope global | grep -o 'preferred_lft 0sec' | wc -l ; \ ip -6 addr show scope global | grep -o '2001:db8:' | wc -l ; \ ls -l /var/lib/dns-common/ra-deprecate.request 2>&1 ; \ sha256sum /var/lib/dns-common/ra-deprecate.request 2>&1"
# 2. Plant the synthetic address with a known finite lifetime. Read it back and confirm a# non-zero preferred_lft before going further.scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; \ ip -6 addr add 2001:db8:04f3::1/64 dev <LEG> noprefixroute valid_lft 7500 preferred_lft 3600 ; \ ip -6 addr show dev <LEG> scope global"
# 3. The request, through the emitter's own file interface, with a fresh stamp. Then the safety# pre-flight: the doc-prefix count must read exactly 1.scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; \ printf 'requested_at=%s\nvlan=lab leg=<LEG> old_prefix=2001:db8:04f3::/64\n' \$(date +%s) \ > /var/lib/dns-common/ra-deprecate.request ; \ cat /var/lib/dns-common/ra-deprecate.request ; \ ip -6 addr show dev <LEG> scope global | grep -o 'inet6 2001:db8:4f3:' | wc -l ; \ ip -6 addr show dev <LEG> scope global | grep -o 'inet6 2600:' | wc -l ; \ ip -6 addr show dev <LEG> scope global | grep -o 'inet6 fddb:' | wc -l"
# 4. Fire. Directly, never via the unit. Read the printed rc.scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; \ /usr/local/sbin/dns-common-ra-deprecate.sh --deprecate-local 2>&1 ; echo rc=\$?"
# 5. The reading, taken immediately. Record the elapsed seconds against step 2.scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "date -Is ; ip -6 addr show scope global ; \ ip -6 addr show scope global | grep -o 'preferred_lft 0sec' | wc -l"The count in step 3 is anchored on 4f3, not 04f3. The kernel prints addresses in RFC 5952
canonical form, which strips leading zeros. A count anchored on the string you typed reads 0 on a
leg that carries the address. The old_prefix= in the request file is unaffected, because the
matcher compares hextet by hextet with zero padding.
The request-file interface is the way in. A first observation for a VLAN always emits
RA-WATCH-PREFIX-BASELINE and never deprecates, so a hand-edited prefix-state file cannot drive
this path. The file carries a staleness fence read from its own requested_at stamp, ceiling
dns_common_ra_deprecate_request_max_age_seconds (3600 s). Give old_prefix an explicit /64.
How to read it, both sides:
- Positive. The planted address reads
preferred_lft 0secwith a finite, non-zerovalid_lft, and the kernel adds its owndeprecatedtoken. - Collateral. The count of addresses reading
preferred_lft 0secon that node goes from 0 to exactly 1, and everyfddb:address and every live2600:address reads a non-zero preferred lifetime.
The kernel never clamps preferred_lft 0. It clamps valid_lft, which is why the valid lifetime
is planted at 7500. ip -6 addr change replaces the whole attribute set, so a flag not named is a flag cleared.
mngtmpaddr and noprefixroute are settable and are re-stated. dynamic and scope global are
derived, and re-stating either is an iproute2 usage error. The emitter filters the observed flags
to the settable subset.
The unique-local fence refuses a request naming an fddb: prefix before any address is read. Probe
it against a prefix the node holds an address under:
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "\ printf 'requested_at=%s\nvlan=lab leg=<LEG> old_prefix=fddb:f665:73f7:4::/64\n' \$(date +%s) \ > /var/lib/dns-common/ra-deprecate.request ; \ /usr/local/sbin/dns-common-ra-deprecate.sh --deprecate-local 2>&1 ; echo rc=\$? ; \ ip -6 addr show dev <LEG> scope global"Expect rc=1 with RA-DEPRECATE SKIP: the deprecation request names the unique-local prefix …,
and every fddb: row unchanged. Exit 1 is the refusal. Exit 2 belongs to the strict-argv refusals
only.
Clean up, and verify the cleanup:
scripts/dns-adhoc.sh <NODE> -b -m ansible.builtin.shell -a "\ ip -6 addr del 2001:db8:04f3::1/64 dev <LEG> ; \ rm -f /var/lib/dns-common/ra-deprecate.request ; \ ls -l /var/lib/dns-common/ra-deprecate.request 2>&1 ; \ ip -6 addr show scope global | grep -o '2001:db8:' | wc -l ; \ ip -6 addr show scope global | grep -o 'preferred_lft 0sec' | wc -l ; \ systemctl list-units --failed --no-pager"Both counts must read 0 and the request file must be absent, or restored from the copy you took in
step 1 and proved identical with sha256sum. A hand-written request left behind arms the emitter
against the next detector failure. Finally, re-read the VIP line counts and both keepalived
ExecMainStartTimestamp values on both nodes in one invocation and make sure that they are
unchanged.
IPv6 forwarding is not touched
Section titled “IPv6 forwarding is not touched”radvd reads /proc/sys/net/ipv6/conf/all/forwarding, which is 0 on these resolvers. It logs
IPv6 forwarding seems to be disabled, but continuing anyway and keeps advertising. No sysctl is
flipped anywhere in this design.
Reading Kea lease state from a workstation
Section titled “Reading Kea lease state from a workstation”This is the supported way to read DHCP lease state off the resolver pair. It needs no interactive shell on either node:
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \ -a 'kea-shell --host 127.0.0.1 --port 8000 --auth-user kea-ctrl --auth-password-file /etc/kea/creds/ctrl-basic-auth.pw lease4-get-all </dev/null'Three details:
- The outer quotes on
-aare single. A double-quoted dispatch argument is expanded by the control shell. -bis required. The credential file is0640 root:_kea.</dev/nullis required.kea-shellreads the command’s JSON arguments from standard input and waits for EOF.
Replace dns_hosts with sandiego or portland to ask one node. The listener is bound to
127.0.0.1, so this command only works on the node. The control channel has no read-only mode.
What it returns
Section titled “What it returns”The response is a JSON array with one object per answering service, on one line:
| Field | Carries | Read it as |
|---|---|---|
result |
Kea’s control result code | 0 success, 1 error, 2 command unsupported, 3 success with an empty result |
arguments.leases |
the lease payload | [ ] means no leases |
text |
a human sentence | a summary, never the datum |
"result": 3 is not an error. It is CONTROL_RESULT_EMPTY: the lease table was read and held
nothing. "result": 1 with an empty leases array is a failed command, and text says why.
Each lease under arguments.leases carries client-id, cltt, hostname, hw-address,
ip-address, subnet-id, valid-lft and state. Read state before concluding a client is
present: 0 is assigned, 2 is expired-reclaimed.
Both nodes must return the identical payload. They are HA partners and lease updates replicate. A
difference is a replication finding, which is why the documented dispatch targets dns_hosts.
The lease CSV (/var/lib/kea/kea-leases4.csv) is not the read surface. Kea’s lease file cleanup
rewrites it on its own interval, so a reader of the file races that rewrite. Do not simplify this
procedure into a cat of the CSV.
Confirming the pair is peered
Section titled “Confirming the pair is peered”scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \ -a 'kea-shell --host 127.0.0.1 --port 8000 --auth-user kea-ctrl --auth-password-file /etc/kea/creds/ctrl-basic-auth.pw status-get </dev/null'The high-availability state is at arguments.high-availability[0].ha-servers.local.state, and the
partner’s last known state at …ha-servers.remote.last-state. A healthy pair reports hot-standby
on both nodes, one as primary and the other as standby. Read the state field before the lease
count. A zero reported without the state field beside it is not an observation.
If it does not work
Section titled “If it does not work”| Symptom | Almost always |
|---|---|
kea-shell: command not found |
isc-kea-admin is not installed. Re-run scripts/dns-playbook.sh --limit <node> --tags kea-install |
Connection refused on 127.0.0.1:8000 |
the daemon is stopped, or kea_ctrl_http_enabled was overridden false |
401 from the listener |
the credential file and the Vault entry disagree. Re-run --tags kea-certs |
| the dispatch hangs | </dev/null was dropped |
| permission denied opening the credential | -b was dropped |
The DHCPv4 accept and the raw-socket asymmetry
Section titled “The DHCPv4 accept and the raw-socket asymmetry”Kea hands out addresses on a leg whose firewall never admitted a single DHCP packet, and it stops
renewing them hours later. That asymmetry is why table inet dns_resolver chain input carries a
udp dport 67 accept on both resolver nodes.
kea-dhcp4 opens AF_PACKET raw sockets, because a client in INIT holds no address and its DISCOVER
is a broadcast from an unspecified source. Raw sockets bypass the firewall, so DISCOVER and OFFER
reach the daemon whether or not any accept exists. A chain missing this rule looks healthy, and
every first acquisition succeeds.
The RENEWING path does not bypass the chain. A client at T1 unicasts its DHCPREQUEST to the server
identifier, this node’s own leg address, and that is an ordinary UDP datagram to port 67 that meets
policy drop. The client sits unanswered from T1 until T2, then falls through to a broadcast
REBIND, which the raw socket answers. At the committed lease of kea_valid_lifetime (43200 s) that
first shows up about six hours after the change that caused it.
The rule is derived from dns_vlan_interfaces like the dport 53 accepts beside it, so it renders
the same before and after any leg is armed. It is not scoped to kea_served_legs, because that is
a per-converge -e value, and scoping the firewall to it would let a routine converge withdraw the
accept from a leg Kea serves. It would also render iifname { } under an empty list, which
nft -f rejects.
Read the accept back off the live kernel:
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \ -a 'nft list ruleset | grep "udp dport 67" | grep -c accept'# want: 1 on each node.The port match and the verdict are matched separately. The kernel prints a counter clause between
them (udp dport 67 counter packets 0 bytes 0 accept), so a pattern anchoring on the contiguous
string dport 67 accept matches zero. The rule is IPv4-only. meta nfproto ipv4 is explicit,
there is no IPv6 sibling, and DHCPv6 is not served by this pair:
scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \ -a 'nft list ruleset | grep -c "nfproto ipv6 udp dport 67"'# want: 0 on each node.Cutting a leg over to Kea, and rolling it back
Section titled “Cutting a leg over to Kea, and rolling it back”This is written for any leg. Substitute the leg name, its pool boundary and its host_vars entry.
Every leg is cut, so the procedure is a rollback and rehearsal instrument. There is exactly
one rollback, reached two ways: by an operator who decided to roll back, and by an operator who
hit one of the abort triggers below. Both run the same numbered steps.
The other half of a cut is the DHCPv4 server of the gateway. It is dhcp_server.enabled on the
network of the leg in tf/unifi/networks.tf, and it is false on every leg. You change it with a
pull request that main-cluster-unifi applies. Nobody has rehearsed a rollback onto the UCG-Fiber,
so read the plan of that pull request in full.
The ordering rule, in both directions
Section titled “The ordering rule, in both directions”To cut, the DHCPv4 server of the gateway goes off first, then Kea is armed. To roll back, Kea is disarmed on both nodes first, then the DHCPv4 server of the gateway goes back on.
| Direction | Step 1 | Step 2 |
|---|---|---|
| Cut | dhcp_server.enabled = false for that network |
arm Kea on the leg, both nodes |
| Roll back | disarm Kea on both nodes | dhcp_server.enabled = true for that network |
| Re-cut | dhcp_server.enabled = false again |
arm Kea again |
Two DHCPv4 servers on one broadcast domain fight, and the client takes the first OFFER, so there is no safe overlap window. An unserved leg for minutes is accepted.
A rollback that stops between the two halves leaves the leg with no DHCP server at all. If the gateway half does not apply, escalate.
Three machines, three roles
Section titled “Three machines, three roles”| Machine | Reached how | Does what |
|---|---|---|
| The instrumented client | over a different leg from the one being cut, through its own wrapper (scripts/nas-adhoc.sh for the NAS) |
every client-side read: address, lease, forced renew, v6 address and resolvers |
| The resolver pair | over the untagged Core leg, scripts/dns-adhoc.sh and scripts/dns-playbook.sh |
every converge, and the server-side lease read |
| The control-node workstation | none | runs both wrappers. It is on neither the cut leg nor the pair’s DHCP path |
The instrument must outlive its subject. SSH is accepted on the five trusted legs. Guest and Telework have no SSH path by design. Do not propose one for the window.
Abort triggers
Section titled “Abort triggers”Any one of them runs the rollback below and closes the window:
- (a) No client on the leg holds an address from Kea’s pool within N minutes of arming. For a leg instrumented at a ten-minute lease, N is 15 minutes: the client’s own T2 plus one acquisition.
- (b) A client that has just acquired an address holds an address outside Kea’s pool. This is the leaked-OFFER signal. Only a client force-renewed or rebooted after the change can be read against it, because a client that has not renewed still holds its old lease.
- (c) The static management path to either resolver node is lost. The nodes’ leg addresses are static and have no DHCP dependency, so losing the path means something other than DHCP moved.
The pool boundary
Section titled “The pool boundary”Read every leg’s boundary from that leg’s host_vars entry (kea_pool). On Lab, .200.140 to
.200.159 below the pool is static (the k3s node Lab legs and
free static addresses), so an address there in a lease is a misconfiguration. On iot, the
upper bound of the pool stops below 192.168.218.224, which is a device’s second interface with no
reservation. Do not widen that pool into .224.
Which client-side fields discriminate
Section titled “Which client-side fields discriminate”| Field | Changes at the cut? | Read it as |
|---|---|---|
| The DHCP server identifier | yes | primary evidence, which server answered |
| The gateway | no | correctness only. The UCG-Fiber stays the default router |
| The advertised resolver | no | correctness only. Kea advertises the leg VIP |
A reading that concludes “DNS is correct, therefore Kea answered” has concluded nothing. Confirm by
lease origin: the lease appears in Kea’s own lease4-get-all on both nodes, and its lifetime is
Kea’s 43200. DHCP4_PACKET_SEND ... DHCPACK (type 5) from <node address>:67 in the isc-kea-dhcp4-server journal names the sending address.
Why the lease is shortened before every window
Section titled “Why the lease is shortened before every window”A rollback is a cold acquisition. At T1 the client unicasts to a server identifier that no longer
answers. It looks for another server at T2 (0.875 of the valid lifetime), when it broadcasts. So
the unattended cost of a rollback is bounded below by T2: about 9 minutes at a 600 s window
lease, and about 10.5 hours at the committed 43200 s. Shorten the lease before every leg’s
window. A forced renew on one instrumented client measures how fast the restored server answers,
not how long an unattended device takes to ask.
Standing cautions
Section titled “Standing cautions”- Do not run the RA rehearsal in a cut window. It has a path into deprecating the live GUA prefix.
- This procedure never changes the IPv6 half. No device serves DHCPv6.
Preparation
Section titled “Preparation”-
The reservation inventory. dns-control holds the entries, and the node worker writes the reservation file of each leg on both nodes, so this is a read of
dns-control entries exportor of the Changes screen. Until the handoff of the curated records merges, Terraform still applies the DNS records oftf/dns-records/hosts.yml, and that file must equal the entry set. -
The lease shortening on the server that serves the leg now, and its lead time. The condition is that every DHCPv4 client on the leg holds the shortened lease, measured client-side. One full previous lease time after the shortening is the calendar proxy.
-
The DHCP Guard check. UniFi DHCP Guard drops every DHCP server-side frame that is not from a trusted address. Kea can be correct while clients DISCOVER endlessly with zero DHCPREQUESTs.
tf/unifi/networks.tfdeclares each network’sdhcp_guarding(an enabled flag and at most three trusted servers). A cut that brings a new DHCP server onto a guarded VLAN adds its address to that list first. Where the guard is on, prove it with a throwaway macvlan client and a known-good VLAN as the control, before touching the gateway:Terminal window ip link add dhcptest0 link eth0.<vlan> address 02:00:00:de:ad:01 type macvlan mode bridgeip link set dhcptest0 upbusybox udhcpc -i dhcptest0 -s /bin/true -n -q -R -t 4 -T 5 -fip link del dhcptest0A lease on the control VLAN and none on the target is the guard, not the daemon.
The converge commands
Section titled “The converge commands”The ruleset converge runs before the window and is firewall-scoped only:
scripts/dns-playbook.sh --limit <node> --tags dns-common-firewall# One node at a time, non-VIP-holder first.The in-window converges are the Kea role tag only:
scripts/dns-playbook.sh --limit <node> --tags kea \ -e kea_service_enabled=true \ -e 'kea_served_legs=["lab"]' \ -e kea_valid_lifetime=600 \ -e kea_renew_timer=300Never the bare dns-common role tag in the window. It sweeps the VLAN legs and the network-apply
transaction and re-arms the RA announcer.
Pass the served-legs value in JSON list form. -e kea_served_legs=lab passes a string, and the
template’s membership test becomes a substring test.
To disarm, pass the served-legs list explicitly. Dropping the -e falls back to the committed
default, and every leg is committed to group_vars, so a bare converge re-arms:
scripts/dns-playbook.sh --limit <node> --tags kea \ -e kea_service_enabled=true \ -e 'kea_served_legs=["<every leg that stays served>"]'
# Rolling back the only served leg:scripts/dns-playbook.sh --limit <node> --tags kea \ -e kea_service_enabled=false \ -e 'kea_served_legs=[]'The three-part read that says a leg is served
Section titled “The three-part read that says a leg is served”A leg is served when three things are true at once: Kea’s journal shows sockets open on the leg, one lease is allocated from that leg’s own pool, and one dynamic name resolves in that leg’s carved zone, forward and reverse. Any one alone has a healthy-looking failure mode.
-
Sockets open on the leg:
Terminal window # On a resolver node.grep -c '"<leg-interface>"' /etc/kea/kea-dhcp4.conf# want: 1 on each node.SINCE='<the arm time, as YYYY-MM-DD HH:MM:SS UTC>'sudo TZ=UTC journalctl --utc -u isc-kea-dhcp4-server --since "$SINCE" --no-pager -o cat -q \> /tmp/kea4.logecho "total=$(wc -l < /tmp/kea4.log)" # the denominator, firstecho "nosockets=$(grep -c DHCPSRV_NO_SOCKETS_OPEN /tmp/kea4.log)" # the numerator# want: total greater than 0, and nosockets 0.A zero with no denominator beside it is not an observation.
journalctlexits 0 and prints nothing whether the daemon is clean or the query was wrong.--sinceparses the node’s local time, so pin both ends withTZ=UTCand--utc. Never count a journal’s errors with a baregrep -i error: Kea’sLFC_*_STATSINFO lines carryerrors: 0. MatchERRORwith its spaces. -
One lease from that leg’s own pool. Read the leases back with the control-socket command in Reading Kea lease state from a workstation. The lease’s
subnet-idis this leg’s id fromhost_varsand itsip-addressfalls inside this leg’skea_pool."result": 3on a leg armed minutes ago is abort trigger (a)’s clock, not a pass. -
One dynamic name in that leg’s carved zone, from the lease in step 2:
Terminal window scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \-a 'dig +short <hostname>.<leg>.lan.fzymgc.house A @127.0.0.1 -p 1053'# want: the leased address, on both nodes.scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \-a 'dig +short -x <the leased address> @127.0.0.1 -p 1053'# want: the same name, on both nodes.Ask the authoritative tier on
@127.0.0.1 -p 1053, never on:53. The main recursor process holds:53and answers from its cache. The reverse half fails on its own: D2 sends forward and reverse as one transaction but reports them separately, so a leg whose reverse/24s are not carved answers the forward query and not the reverse one. Reservations in a deliberately uncarved/24(main’s.20,core’s.40) take aDHCP_DDNS_NO_REV_MATCH_ERRORon their reverse half and keep answering from the covering/16. That is the carve working.
The procedure
Section titled “The procedure”Cut:
- Take the pre-cut client reads: v4 address, lease (address, lease time, server identifier, option surface), v6 address and resolvers. Verbatim.
- Set
dhcp_server.enabled = falsefor that network intf/unifi/networks.tf, and letmain-cluster-unifiapply it. - Prove the static management path with DHCP off on that leg: dispatch to both resolver nodes and confirm both answer. Its failure is abort trigger (c).
- Arm the non-VIP-holder, then the VIP holder, with the in-window command above.
- Confirm each daemon opened a socket for the leg’s interface and its journal carries no
DHCPSRV_NO_SOCKETS_OPENline.
Verify:
- Force a renew on the instrumented client and re-read the discriminating fields.
- Read the leases back from the pair through the control-socket command.
- Agreement between the client’s own view and the server’s lease is the evidence.
- Re-read the v6 address and resolvers. They must be unchanged.
Roll back, the same steps whether this is the drill or an abort:
- Disarm Kea on both nodes with the explicit served-legs form, and confirm both daemons are down before the gateway changes.
- On
main,guestandtelework, put the leg’s.1on the trusted list of UniFi DHCP Guard for that network first, intf/unifi/networks.tf. The list holds at most three addresses, so.1replaces the VIP.5. - Set
dhcp_server.enabled = truefor that network, and letmain-cluster-unifiapply it. - Force a renew on the instrumented client.
- Confirm by lease origin, not by address range. Read the
valid-lft, and make sure that the lease no longer appears in Kea’s ownlease4-get-allon both nodes. - Re-read the v6 address and resolvers. The v6 half never changes, so anything that moved here is a finding.
Re-cut:
- Set
dhcp_server.enabled = falseagain. - Arm both nodes without the lease-instrument extra-vars, so the leg lands on the committed steady-state lease.
- Verify again, including that the on-node rendered lease time is the steady-state value.
- Commit the leg’s arm values to
group_varsthe same day, then run one bare converge with no extra-vars and confirm Kea is serving. That is the only reading that proves the steady state is self-sustaining.
The DDNS tier — how a DHCP lease becomes a name, and how each half fails
Section titled “The DDNS tier — how a DHCP lease becomes a name, and how each half fails”Read this section before triaging a missing name. Kea D2 writes the A half and the PTR half as two separate updates to two different zones, and each half fails on its own.
What publishes what
Section titled “What publishes what”The forward chain (A and PTR) is Kea’s, end to end. The A half and the PTR half are two separate updates, to two different zones, accepted by two different grants:
kea-dhcp4grants or renews a lease and builds an FQDN from the client’s hostname plus the leg’s ownddns-qualifying-suffix(lab.lan.fzymgc.house.for Lab, derived from the leg name so a client cannot mint a name outside its own VLAN).- It emits a Name Change Request over the
dhcp-ddnschannel to127.0.0.1:53001, this node’s ownkea-dhcp-ddns(D2). The NCR channel never crosses the wire. - D2 matches the FQDN against its configured forward domain, turns the request into RFC 2136 DNS
UPDATE signed with that leg’s own
kea-d2-<leg>TSIG key, and sends it to PowerDNS on127.0.0.1:1053. - The forward update targets
<host>.<leg>.lan.fzymgc.housein the leg’s own child zone, which grants DNS UPDATE to that leg’s key alone. The parentlan.fzymgc.housegrants nobody. - The reverse update is a second update to a different zone, signed with the zone-wide
kea-d2-lankey, and it lands in the leg’s own carved/24. Each carve is NS-delegated from the covering/16and grants DNS UPDATE tokea-d2-lanalone.
168.192.in-addr.arpa is writable by nobody. The covering /16 carries the curated PTRs, so it
carries no TSIG-ALLOW-DNSUPDATE and no ALLOW-DNSUPDATE-FROM. A reverse update that lands there
is REFUSED, and that is correct. The carve boundary is drawn at the leg, so widening a pool needs
no zone change. A leg whose pool spans a /24 that is not carved takes a
DHCP_DDNS_NO_REV_MATCH_ERROR on every lease there, and those addresses keep answering in reverse
from the covering /16.
If dig -x on a lease address returns nothing, first check whether the address is inside a carved
zone at all:
# On a resolver node.pdnsutil zone list-all | grep in-addr.arpapdnsutil metadata get 200.168.192.in-addr.arpa TSIG-ALLOW-DNSUPDATE# want: TSIG-ALLOW-DNSUPDATE = the zone-wide kea-d2-lan key at the granted generationpdnsutil metadata get 168.192.in-addr.arpa TSIG-ALLOW-DNSUPDATE# want: EMPTY. The covering /16 grants nobody, and this read is the control for the one above.journalctl -u isc-kea-dhcp-ddns-server | grep -E 'RCODE|Transaction outcome'pdnsutil prints multi-valued metadata comma-joined on one line. Split on the comma before
concluding a grant names one writer.
Reverse Change: yes on a DHCP_DDNS_REMOVE_SUCCEEDED line means the reverse half did not error.
It does not mean a PTR was withdrawn, because deleting an absent RRset is a success in RFC 2136.
The instrument is the reverse zone’s SOA serial, which PowerDNS advances only on an UPDATE that
changes records:
pdnsutil zone list 200.168.192.in-addr.arpa | grep -E '[[:space:]]SOA[[:space:]]'# want: the serial field moves only across an update that changed recordsThe carves are real /24 zones and not RFC 2317. RFC 2136 resolves the target zone by walking the
ancestor labels of the name being updated, and an update never follows CNAME indirection. D2 has no
RFC 2317 awareness. TSIG-ALLOW-DNSUPDATE is per zone, so a narrower boundary has to be a narrower
zone.
No program writes AAAA records into a leg’s zone. Each leg’s zone has one writer and one key:
pdnsutil metadata get lab.lan.fzymgc.house TSIG-ALLOW-DNSUPDATE# want: kea-d2-lab<gen>, the granted generation's spellingpdnsutil metadata get lan.fzymgc.house TSIG-ALLOW-DNSUPDATE# want: an empty value. The parent grants nobody.An empty parent reading is the correct state. The grant governs RFC 2136 dynamic updates only, and dns-control and external-dns write over the HTTP API.
Where the boundary is, and what enforces it
Section titled “Where the boundary is, and what enforces it”The curated apex fzymgc.house accepts no DNS UPDATE, and the reason is not that it carries no
metadata:
pdnsutil metadata get fzymgc.house TSIG-ALLOW-DNSUPDATE# want: an empty valuePowerDNS’s authorisation logic falls back to the global allow-dnsupdate-from range when a zone
names no per-zone source. What closes the apex is configuration:
grep -E '^(dnsupdate|dnsupdate-require-tsig|allow-dnsupdate-from)' /etc/powerdns/pdns.conf# want:# dnsupdate=yes# dnsupdate-require-tsig=yes# allow-dnsupdate-from=127.0.0.0/8,::1dnsupdate-require-tsig=yes makes the absence of a TSIG grant a REFUSED, regardless of source
address. The global range is loopback-only as a second, independent layer. Drop the first and
loopback processes can write the apex. Drop the second and the only thing between the wider network
and a signed update is key custody.
TSIG key rotation
Section titled “TSIG key rotation”Rotation is by key name, with a server-side overlap, never by replacing the secret behind a name the
writers use. Every key carries a generation: the zone-wide kea-d2-lan and the five per-leg
kea-d2-<leg> keys, six names at each generation. Generation 1 is the unsuffixed spelling
(kea-d2-lab, Vault field secret_lab). A generation n > 1 renders -<n> on the name
(kea-d2-lab-2) and _<n> on the Vault field (secret_lab_2). Two values choose the generation:
dns_tsig_key_generationinansible/inventory/group_vars/dns_hosts.yml: the generation Kea D2 signs with.dns_tsig_key_generations_grantedin the same file, and theTSIG-ALLOW-DNSUPDATEgrant of each dynamic zone in dns-control: every generation the PowerDNS store holds and every grant names. The two must be equal as sets.
dns-control zones grant --generations <list> sets the grant of every dynamic zone in one command.
For each zone, the new grant names each Kea D2 key base of the zone at each generation in the list.
A key that is not a Kea D2 key leaves the grant. The command plans and applies one change per zone
that differs, through the change workflow, and prints no change for the others. --dry-run
prints the same lines and writes nothing.
The Vault side is fields of one entry, fzymgc-house/infrastructure/dns/kea-d2-tsig: secret_<n>
and secret_<leg>_<n>. No new path and no new policy grant.
This procedure gets its own window. A rotation gone wrong and a cut gone wrong have the same symptom, names that stop appearing.
The two latencies: zone-metadata-cache-ttl=60 is the documented upper bound for consumers that
read metadata through the cache, and the RFC 2136 grant check does not. LightningStream replication
is the latency that governs. dns-control writes a grant through the writer node’s API, and the other
node receives it on the order of a minute later. A writer that signs against the VIP holder during
that window is REFUSED on one node and accepted on the other. The role does not sleep for either.
--tags pdns reads the grant with an until over pdnsutil metadata get on the node being
converged (roles/pdns/tasks/verify.yml) until the parsed set equals the expected one.
The order:
-
The Vault fields, from the control node. One value per field, minted on its own. Use
vault kv patch, nevervault kv put, which replaces the entry and drops the fields the current generation signs with:Terminal window for leg in core lab main iot mainwireless; dovault kv patch -mount=secret fzymgc-house/infrastructure/dns/kea-d2-tsig "secret_${leg}_<n>=$(openssl rand -base64 32)"donevault kv patch -mount=secret fzymgc-house/infrastructure/dns/kea-d2-tsig "secret_<n>=$(openssl rand -base64 32)"vault kv get -mount=secret -format=json fzymgc-house/infrastructure/dns/kea-d2-tsig | jq -r '.data.data | keys[]'The
jqline reads field names only. Want the current generation’s fields beside the new ones. -
PR A, widen. Set
dns_tsig_key_generation: <n>anddns_tsig_key_generations_granted: [<n-1>, <n>]indns_hosts.yml, and merge it. Then widen the grants and converge the authoritative tier, both nodes in one run:Terminal window kubectl -n dns-control exec deploy/dns-control -- dns-control zones grant --generations <n-1>,<n> --dry-runkubectl -n dns-control exec deploy/dns-control -- dns-control zones grant --generations <n-1>,<n>scripts/dns-playbook.sh --tags pdnsThe dry run shows each grant gaining the
-<n>names. Widen the grants before the converge, because theuntilof--tags pdnswaits for the widened set. Use one converge and not two--limitruns. The store read and the import arerun_onceon the first host, and the grant reads are per host, so theuntilon the second host is what proves the peer received both the grant and the imported keys. Kea D2 must not converge until--tags pdnshas passed on both resolvers. -
The writer, straight to
<n>:Terminal window scripts/dns-playbook.sh --tags keaD2 restarts onto the new names in one converge, and its per-key counters start from zero under the new key objects. Take the reads in the next subsection before going on.
-
PR B, narrow. Set
dns_tsig_key_generations_granted: [<n>]and merge it. Then narrow the grants to the-<n>names alone and converge once more:Terminal window kubectl -n dns-control exec deploy/dns-control -- dns-control zones grant --generations <n>scripts/dns-playbook.sh --tags pdnsThe
run_oncepdnsutil tsigkey deleteremoves the previous generation’s six keys from the store, andd2-tsig-key-presentproves the store equals the declared set. The previous generation’s.tsigfiles stay on the Kea nodes until the next--tags keaconverge removes them.
Never narrow before the writer has converged. That is withdraw-then-converge: narrowing the grant while D2 signs with the previous generation REFUSES every one of its updates until it converges, and the refusal reads like a zone problem hours later with D2 reporting success.
Reading the keys and the grants on-box. pdnsutil tsigkey list prints the secret as its third
field. Never paste its raw output anywhere:
# On a resolver node.pdnsutil tsigkey list | cut -d " " -f1 | sort# want (overlap): twelve names, six of them ending -<n>; want (narrowed): the six -<n> namespdnsutil metadata get lab.lan.fzymgc.house TSIG-ALLOW-DNSUPDATE# want (overlap): kea-d2-lab at <n-1>, kea-d2-lab-<n># want (narrowed): kea-d2-lab-<n>pdnsutil metadata get 200.168.192.in-addr.arpa TSIG-ALLOW-DNSUPDATE# want (overlap): kea-d2-lan at <n-1>, kea-d2-lan-<n># want (narrowed): kea-d2-lan-<n>sudo /opt/dns-resolver/venv/bin/python -c 'import json, socket; s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM); s.connect("/run/kea/kea-ddns-ctrl-socket"); s.sendall(json.dumps({"command": "statistic-get-all"}).encode()); s.shutdown(socket.SHUT_WR); data = b"".join(iter(lambda: s.recv(65536), b"")); stats = json.loads(data)["arguments"]; print({k: v[0][0] for k, v in stats.items() if k.startswith("key[") and k.endswith("update-success")})'# want (after step 2, once a lease event has happened): a positive count under a key[...-<n>...] nameThe statistic is the reading that names the key. Neither PowerDNS’s success line nor D2’s
Transaction outcome line names it. dns-control shows the stored grant of a zone with
GET /api/zones/<name> or the MCP tool dns_get_zone.
The break-glass, same-name path is destructive and kept for the case the generation path cannot cover, when the secret behind a name must change without a generation bump:
pdnsutil tsigkey delete <name>on one node only, then wait for replication, confirm absence on both nodes with the name-only list
above, then converge --tags pdns. The absence guard fires and re-imports the name from whatever
Vault holds. Every UPDATE signed with it is REFUSED for the whole interval, on both nodes, and
nothing announces it.
“Refused” means DNS-refused, and never lease-refused
Section titled ““Refused” means DNS-refused, and never lease-refused”A client that loses a DNS name conflict keeps its address, its gateway, its subnet mask and its resolvers. It has no forward record. DDNS sits downstream of leasing: a DNS failure of any kind degrades nothing about DHCP. Confirm DHCP health from the lease table, not from anything in this section.
A client that supplies a multi-label hostname gets no name
Section titled “A client that supplies a multi-label hostname gets no name”Kea appends the leg’s ddns-qualifying-suffix only to a single-label hostname. A client that sends
a hostname with dots is treated as fully qualified. The FQDN then falls outside lan.fzymgc.house,
D2 matches no configured forward domain, and discards the request without contacting PowerDNS. A
client sending nas.fzymgc.house produces DHCP_DDNS_NO_MATCH No DNS servers match FQDN nas.fzymgc.house. followed by DHCP_DDNS_NO_FWD_MATCH_ERROR.
| What you look at | What it says | What is true |
|---|---|---|
The lease (lease4-get-all) |
"fqdn-fwd": true |
Kea asked for a forward update. It says nothing about whether one happened |
journalctl -u isc-kea-dhcp4-server |
clean | The discard happens in D2, a different daemon in a different unit |
--tags pdns-verify and kea-verify |
all green | They assert configuration. Neither observes a published name |
| The zone | no record | Correct. The only error anywhere is in D2’s journal |
This is a property of the naming policy. The same mechanism stops a Lab client asserting a name in the curated apex.
A name D2 missed comes back on the client’s next renewal
Section titled “A name D2 missed comes back on the client’s next renewal”ddns-update-on-renew is true on both nodes. A renewal emits a Name Change Request even when
neither the FQDN nor the DHCID changed, so a name D2 missed while it was down comes back on the
client’s next renewal, within kea_renew_timer (21600 s in ansible/roles/kea/defaults/main.yml).
A client release and re-acquire is the way to go faster. Three costs come with this:
- A renewal queues two Name Change Requests, a
CHG_REMOVEfor the old lease and aCHG_ADDfor the new one, so every renewing client’s name has a brief absence window once per renew interval. A resolver that queries inside it caches NXDOMAIN for up to the zone’s 60 s SOA minimum. This is accepted, because shortening the zone’s negative TTL would shorten every negative answer. - Record deletion is not durable. A name deleted from the zone by hand returns on the client’s next renewal. To make a name stay gone, take the lease away, not the record.
- A claim refused by a seeded reserved label fails twice per renewal, per offending client. That is recurring noise in D2’s journal by design.
There is no reconciler. A reconciler scoped to everything would read a failed lease read as an empty desired state and reap the zone.
Reading the journal when a claim is refused. Both identifiers are LOG_ERROR, so both appear at
the default kea_log_severity:
# On a resolver node.journalctl -u isc-kea-dhcp-ddns-server --since '-1h' \ | grep -E 'DHCP_DDNS_REMOVE_FAILED|DHCP_DDNS_FORWARD_REPLACE_REJECTED'An empty journal is a third answer. If nothing reached D2, Kea skipped the request before it was
queued. It says so as DHCPSRV_QUEUE_NCR_SKIP, which is LOG_DEBUG and invisible at the default
severity. Raise kea_log_severity on the node before concluding that nothing happened.
Read the total and the match in the same command, and treat a zero total as “the query was wrong”:
# On a resolver node. -o cat -q makes an empty result count 0 rather than 1 line of "-- No entries --".SINCE='<the window start, as YYYY-MM-DD HH:MM:SS UTC>'sudo TZ=UTC journalctl --utc -u isc-kea-dhcp-ddns-server --since "$SINCE" --no-pager -o cat -q \ > /tmp/d2.logecho "total=$(wc -l < /tmp/d2.log)" # the denominator, firstecho "errors=$(grep -c DHCP_DDNS_NO_FWD_MATCH_ERROR /tmp/d2.log)" # the numeratorThe unit is isc-kea-dhcp-ddns-server.service, and every line it emits is tagged
kea-dhcp-ddns[<pid>]. journalctl -u kea-dhcp-ddns matches no unit and prints -- No entries --
with exit 0. --since parses the node’s local time, so pin both ends with TZ=UTC and --utc.
When a name is missing after a D2 outage: first confirm it is absent (pdnsutil zone list <leg>.lan.fzymgc.house on the node, and resolve the name at both daemons), and read D2’s journal
for the transaction. Then decide whether you need it back sooner than the renewal cadence. If you
do, have the client release and re-acquire its lease. A lease that never renews never heals. That
is a device or IPAM question, not a carve defect.
How to read each tier
Section titled “How to read each tier”The two verify tags, each answering a different question:
scripts/dns-playbook.sh --limit <node> --tags pdns-verifyscripts/dns-playbook.sh --limit <node> --tags kea-verify| Tag | Answers | Does not answer |
|---|---|---|
pdns-verify |
Are the granted generations’ keys in this node’s key store? Do the carved zones grant exactly their writers, and do the parent dynamic zone, the covering /16 and the apex grant nobody? Is the running daemon armed for DNS UPDATE? |
Whether any update was ever accepted |
kea-verify |
Is the D2 config 0640 root:_kea, is its TSIG secret readable as the daemon, does the config parse? |
Whether D2 is running, and whether a signed update is accepted |
Neither tag observes a published name. To know whether names are being published, read the zone.
A configuration file is not a running process. PowerDNS reads its configuration only at start. A
converge that writes pdns.conf and then fails before its handlers flush leaves the file correct
and the process unchanged, and every later converge reports ok and never re-notifies the restart.
The reading that tells a file from a process:
# On a resolver node.pdns_control current-config | grep dnsupdatesystemctl show pdns -p ExecMainStartTimestampstat -c '%y %n' /etc/powerdns/pdns.confA commented # dnsupdate=no is how pdns reports a setting left at its default, and it means not
armed. The armed reading is an uncommented dnsupdate=yes with an uncommented
dnsupdate-require-tsig=yes. A process start timestamp older than the config file’s mtime means
the running daemon is not running the file you read. The fix is systemctl restart pdns on that
node. The running-dnsupdate-armed criterion in ansible/roles/pdns/tasks/verify.yml is the one
criterion in the tier that reads the running daemon. When a gate is green and the live estate is
red, the first hypothesis is the instrument.
The zone itself:
pdnsutil zone list lab.lan.fzymgc.housepdnsutil zone list lab.lan.fzymgc.house | grep -c -E "[[:space:]]IN[[:space:]]+A[[:space:]]"pdnsutil zone list emits tab-separated records. A pattern with a literal space matches nothing.
Do not wrap a counting command in || true: grep -c exits 1 on zero matches, and that exit is
the signal.
The key store prints secrets. Reduce to names on the box:
pdnsutil tsigkey list | awk '{print $1}'# want: the six names of the granted generationIs D2 alive and listening:
systemctl is-active isc-kea-dhcp-ddns-serverss -lnup | grep 53001journalctl -u isc-kea-dhcp-ddns-server --since '-1 hour' --no-pagerThe unit is isc-kea-dhcp-ddns-server.service, and the package is isc-kea-dhcp-ddns.
dpkg -l isc-kea-dhcp-ddns-server reports Installed: (none) on a node where D2 runs.
The silent degradation: D2’s queue limit
Section titled “The silent degradation: D2’s queue limit”kea-dhcp-ddns accepts NCRs into a bounded queue, max-queue-size (kea_ddns_max_queue_size,
1024). When the queue is full, kea-dhcp4 suspends sending DDNS updates until the backlog clears.
Leasing is unaffected. The only places this is visible are D2’s journal and the growing gap between
leases and records.
One residual risk, accepted
Section titled “One residual risk, accepted”A release carrying another client’s identifier can delete that client’s record. A DHCP Release is unauthenticated, so a host on the leg can send a Release with a victim’s client identifier and cause the victim’s name to be withdrawn. DHCID does not stop this, and no configuration available here does. The blast radius is a missing name for one client until its next renewal, and the attacker must already be on the leg. A name disappearing for no apparent reason has this candidate explanation.