Design: DNS policy as RPZ
Context
Section titled “Context”See openspec/changes/dns-policy-as-rpz/proposal.md for the reason and the scope. The specs of
this change are under openspec/changes/dns-policy-as-rpz/specs/.
RPZ (response policy zone) is a DNS zone whose records are rules. The owner name of a record is
the trigger, and its data is the action. example.org CNAME . answers NXDOMAIN for
example.org, and example.org CNAME rpz-passthru. lets the name resolve. A resolver consults
its policy zones in a fixed order, and the first zone that matches decides.
The parts of the estate that this design replaces or touches:
- Each resolver node runs two Blocky processes from one binary.
blockyserves the trusted legs, the tailnet and127.0.0.1.blocky-publicserves Guest, Telework and127.0.0.3, and its configuration has noconditionaland nocustomDNS, so no query of it reaches pdns-auth. Neither has a DoT or DoH listener. - Four list groups block names.
hageziis HaGeZi Pro++,tifis TIF medium,cryptois the Block List Project crypto list, anddomainsholds three names written inline. Each leg declares its groups inblocky_list_groupson itsdns_vlan_interfacesentry, and today every leg declares all four. A client on no declared leg getsblocky_default_list_groups. - The main process withholds
fzymgc.houseand the reverse zones from a client on no declared leg. The groupinternal-namespaceanswers NXDOMAIN for them. The tailnet and127.0.0.1are outside that group. blocky-publicforwards a name underfzymgc.houseto dnscrypt-proxy, which answers REFUSED, and Blocky passes REFUSED to the client. The recursor treats a REFUSED from its forwarder as a failed server and answers SERVFAIL.- Both processes forward every public name to dnscrypt-proxy on
127.0.0.1:5300. The main process sendsfzymgc.houseand the reverse zones to pdns-auth on127.0.0.1:1053, withfallbackUpstream: false. - The memory on 2026-10-09: each node has 4,049 MiB and about 3,000 MiB available. The two Blocky processes use 211 to 237 MiB each. HaGeZi Pro++ holds 221,621 entries, TIF medium 693,959 and the crypto list about 1,300.
- keepalived tracks two scripts.
chk_house_answerasks127.0.0.1fordns-health.fzymgc.houseand passes on an address under192.168..chk_public_answerasks127.0.0.3forhealth.dns.invalidand passes only on NXDOMAIN, which Blocky answers from its special-use table with no upstream query. - dns-control writes pdns through
internal/pdnson the writer node, portland athttp://192.168.20.7:8081. LightningStream copies the LMDB of pdns-auth to sandiego through S3 on the NAS. A copied write sends no NOTIFY.zone-cache-refresh-intervalis 60 seconds, so a new zone appears on the other node up to 60 seconds after LightningStream copies it. A changed record appears at once. - The lineage index (
internal/refresh) and the converge (internal/change/converge.go) read every zone that pdns-auth lists. internal/querylogreads the rows of the ServiceNamesblockyandblocky-publicfromdefault.otel_logs.health.goand the error guard readblocky_response_totalandblocky_error_total.explain.goparses the Blocky reasonBLOCKED (group: rule).- The ClickHouse row filter of both MCP users hides every row with the ServiceName
blockyorblocky-public. Onlydns_control_readerreads those rows. - The node worker writes
/var/lib/dns-worker/lists/operator-denylist.txtandoperator-allowlist.txt, and posts/api/lists/refreshand/api/cache/flushto Blocky.nodeOrderputs the node that holds no VIP first.
docs/reference/network.md is the authority for every address in this design: the Main VIP
192.168.20.5, sandiego .20.6, portland .20.7, the Guest VIP 192.168.79.5 and the Lab VIP
192.168.200.5. The recursor listens on the addresses that Blocky listens on today. This change
allocates no address and no prefix. No Terraform resource changes.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- Every rule set is an RPZ zone, and the zone order is the whole precedence.
- A list entry change and a scope change are in force on both nodes within one minute, with no restart of a resolver process.
- Every answer gives one row in ClickStack that names the client, the question, the answer, and the zone and the rule that decided it.
- dns-control reads engine-neutral names only.
- The entries, the scopes and the feeds hold while dns-control, k3s, Temporal or the NAS is down.
- Every Blocky feature in use today has a named equivalent. The parity table under Decision 9 lists them.
Non-Goals:
- A scope on Guest or Telework. The public process loads no zone of dns-control.
- Moving the excepted names, the owned zones or the feed list into dns-control.
- Client identity by MAC address or Kea reservation. A scope member is an address, a CIDR or a leg. A later change can add a device member.
- DNSSEC validation in this tier. The tier does not validate today, and dnscrypt-proxy stays its only public upstream.
- A DoT or DoH listener.
Decisions
Section titled “Decisions”Decision 1: PowerDNS Recursor is the engine
Section titled “Decision 1: PowerDNS Recursor is the engine”On 2026-10-09 Sean chose PowerDNS Recursor 5.4 from the suite trixie-rec-54 of
repo.powerdns.com. The role installs it from the repository that the pdns role already
trusts, with the same key and the same pinned Debian version string. Spike 011 installed
5.4.7 (5.4.7-1pdns.debian13), the newest stable release. trixie-rec-55 holds only
5.5.0~alpha1, so the role pins 5.4.7.
| Engine | RPZ input | Scope by client | Reason for each query | Why not |
|---|---|---|---|---|
| PowerDNS Recursor 5.4 | IXFR or AXFR from a primary, or a file | Lua hooks gettag_ffi and prerpz |
protobuf appliedPolicy, appliedPolicyTrigger, appliedPolicyHit, appliedPolicyKind |
chosen |
| Knot Resolver 6.5 | files only | declarative views, from subnet to tags |
RPZ match lines at notice level, and dnstap without the policy | no reason for each query, and no zone transfer |
| Unbound 1.23 | AXFR or url: |
declarative tags |
rpz-log text lines |
users report several GB for a 1M-entry RPZ, and fast_reload is experimental |
| CoreDNS | none in-tree | view plugin |
log plugin, with no reason |
blocking needs a third-party plugin and our own build |
| AdGuard Home | its own rule syntax | $client rules |
its own JSON log | it rewrites its own YAML file, and it has no metrics without an exporter |
| Blocky 0.35 | none | static groups | JSON line with the list | a scope needs a restart, and there is no reload upstream |
The maintainer of DNS-collector states that no DNS server except PowerDNS Recursor exports the RPZ hit for each query. That property and the zone transfer decide the choice. Knot Resolver is the fallback. Decisions 2, 3 and 8 hold the same for it, so a later change of engine changes one role and the scope mechanism only.
Decision 2: The zones, their names and their order
Section titled “Decision 2: The zones, their names and their order”The owned zones live under rpz.internal.. .internal is the top-level domain that ICANN
reserved for private use in 2024. The recursor forwards no name under it to pdns-auth, so no
client reaches a policy zone through the resolver.
The main process consults these zones in this order:
| Order | Zone | Source | Holds |
|---|---|---|---|
| 1 | withheld.rpz.internal. |
a file that the role renders | fzymgc.house and the reverse zones, as NXDOMAIN. It applies to a client on no declared leg only |
| 2 | allow.rpz.internal. |
a file that the role renders | the owned zones and the excepted names, as rpz-passthru. |
| 3 | <scope>.scope.rpz.internal. |
pdns-auth | the entries of one scope, in the order of the scopes |
| 4 | operator.rpz.internal. |
pdns-auth | the entries with no scope |
| 5 | <group>.feed.rpz.internal. |
a file on each node | one zone for each list group that the leg of the client declares, in the order hagezi, tif, crypto, domains |
The public process consults the withheld zone for every client, then rebinding.rpz.internal.,
then the allow zone, then the feed zones of the leg of the client. The withheld zone answers
NXDOMAIN for an internal name before any forwarder sees it, so a Guest client gets NXDOMAIN in
place of the REFUSED of today. rebinding.rpz.internal. holds rpz-ip triggers for every
private, ULA, loopback and link-local range, with the action NODATA. It replaces
rebindingProtection of blocky-public, which answers an empty NOERROR.
The order sets four rules:
- A client of the public process, and a client of the main process on no declared leg, never gets an internal name. No such query leaves the process.
- An owned name or an excepted name always resolves for every other client.
- A scoped entry beats an entry with no scope. A scoped allow unblocks a name that the operator zone blocks, and a scoped block blocks a name that the operator zone allows.
- An operator entry beats every feed.
One rule maps an entry to records, for a list entry and for a line of a feed. A plain name
example.org becomes example.org CNAME ., which matches that name only. A wildcard name
*.example.org becomes example.org CNAME . and *.example.org CNAME ., which match the name
and every name under it. An allow entry uses rpz-passthru. in place of .. These are the
matches of a Blocky entry today. The policy TTL of the owned zones is 30 seconds, which is
blockTTL today.
The configuration of the main process spreads this list over three files, and spike 011 showed the layout. A sequence in a file of the include directory is appended to the same sequence of the files before it, so the order of the files is the order of the zones:
| File | Holds |
|---|---|
/etc/powerdns/recursor-main.yml |
the withheld zone and the allow zone |
recursor-main.d/50-scopes.yml |
each scope zone, and its name in incoming.allow_notify_for |
recursor-main.d/90-tail.yml |
the operator zone and the feed zones |
Every RPZ sets policyName to its zone name with the trailing dot, such as
hagezi.feed.rpz.internal.. Without it, a zone from a file has the name rpzFile, and the Lua
script and the protobuf rows cannot tell the feeds apart.
Alternative: one zone for every operator entry, with the scope held in Lua. That gives up the standard format for exactly the rules that change most. The protobuf then names one zone for every scoped decision.
Decision 3: The owned zones are a projection of the list store
Section titled “Decision 3: The owned zones are a projection of the list store”The list store and the scope store of dns-control are the source of truth. The zones in
pdns-auth are their projection, as the operator files are today. The projection creates
operator.rpz.internal. once, creates a scope zone when a scope is created, and deletes it when
the scope is deleted. A projection writes the whole set again, as the requirement “A projection
writes the whole set again” of dns-operator-list says today.
The zones are kind Native with SOA-EDIT-API set, so every API write raises the serial and
pdns-auth sends no NOTIFY. The lineage index and the converge skip every zone under
rpz.internal.. One predicate in internal/pdns holds that suffix. The list projection keeps
its own drift read: it compares each zone with the store.
The role pdns renders allow-axfr-ips=127.0.0.1/32,::1/128 explicitly, because its template
renders every setting and inherits none.
Alternative: the policy zones as curated zones in the zone store. Then a curated-zone screen,
the export of zones.json and the drift view show records that no person writes.
Decision 4: How a change reaches a node
Section titled “Decision 4: How a change reaches a node”There are three kinds of change, and each has its own path:
| Change | How often | Path on each node, the node that holds no VIP first |
|---|---|---|
| A list entry | often | The worker waits until pdns-auth on loopback serves the new serial. It sends a NOTIFY for the zone to the main process, and reads the loaded serial from rpzstatistics of the recursor API. Then it runs the wipe action of Decision 12 with subtree set for each name of the change |
| The members of a scope | sometimes | The worker writes the scope map file with a temporary file and a rename, runs rec_control reload-lua-script, and reads back the map version with rec_control get scope-map-version |
| A new scope | seldom | The worker writes the scope map that names the new scope and runs reload-lua-script. Then it writes 50-scopes.yml and runs rec_control reload-yaml and rec_control reload-zones. Then it takes the list-entry path |
| A deleted scope | seldom | The worker writes 50-scopes.yml without the scope and runs reload-yaml and reload-zones. Then it deletes the dump file of the zone and writes the scope map without the scope |
Spike 011 sets the order of these steps:
- Recursor 5.4 does not wipe the packet cache when an RPZ loads, so a cached answer outlives a new entry until the step wipes it. 5.5 wipes the packet cache on a load, and the wipe step of a list entry can go with an upgrade to a stable 5.5.
- The Lua script discards only the scopes that the scope map names. A scope zone that is loaded before the map names it applies to every client. So the map comes first for a new scope and last for a deleted one.
incoming.allow_notify_forfrom the include takes effect only withreload-zones. Before it, the main process ignores a NOTIFY for the new zone. A parent name such asrpz.internaldoes not cover the zones under it.- The recursor seeds a zone from its dump file. A scope that is deleted and created again on
the same day gets the same date serial from pdns-auth, and the recursor keeps the old entries
from the dump file.
rpzstatisticsthen shows the serial of pdns-auth, so only the delete of the dump file prevents this. reload-yamlexits 1 on an include with invalid YAML, and the process keeps its zones. The step fails on that exit code.
In spike 011 the main process loaded a new serial 10 ms after the NOTIFY, with a full transfer.
The serial wait covers LightningStream and the 60-second zone cache. It times out after 120
seconds. A timeout fails the step. A failed step sends no NOTIFY to the next node, and the change
ends failed, as a failed node.list_write does today. Each RPZ in the configuration also has
refresh: 300, so a node picks up a change within five minutes when dns-control is down after a
write.
This changes one property of today. pdns-auth on the next node already holds the change, so its
main process loads the change by refresh within five minutes, also after a failed step. The
operator files of Blocky kept the change off the second node. A list entry is data and not
configuration, so the canary value of that property is small. The failed change still tells the
operator which node did not load the change in time.
Each step records the serial of each owned zone that the change wrote and the serial that its
main process loaded. The recursor keeps a copy of each transferred zone in its dumpFile and
starts from it as seedFile. A restart of the recursor with pdns-auth down still starts with
every entry.
Alternative: rely on refresh alone, with a short interval. Then both nodes take a change at
about the same time, the passive-first order goes away, and a step cannot read back what its
node holds.
Decision 5: Scopes, and the zones that apply to a client
Section titled “Decision 5: Scopes, and the zones that apply to a client”A scope has a name, an order and members. The name is a DNS label of 1 to 40 characters from
a-z, 0-9 and -, because it becomes a label of the zone name. The order is a unique integer
that sets the place of the scope zone in Decision 2. A member is an address, a CIDR or a trusted
leg. dns-control resolves a leg to its IPv4 prefix and its ULA /64 from internal/legs, and
refuses Guest and Telework. A list entry names zero or one scope, and the store holds one entry
for each name in each scope. A scope change is a change of the subject block, as a list change
is, so the audit rows, the change metric and its alerts cover it.
The Lua script of each process decides which zones apply to a client. It reads two maps:
- The leg map, which the role renders from
dns_vlan_interfaces. Each line holds a CIDR and the list groups of its leg. The map of the main process also names the tailnet and127.0.0.1. - The scope map
/var/lib/dns-worker/scopes.map, which the worker writes. Each line holds one scope name and one CIDR. The public process has no scope map.
The script does two things:
gettag_ffifinds the zone set of the client: the withheld zone when no line of the leg map holds the address, the feed zones of its leg or ofdns_default_list_groups, and every scope whose members hold the address. In the public process, the set always holds the withheld zone. It sets one packet cache tag for each distinct zone set. The tag is a hash of the sorted zone names, so a reload of the script keeps each tag on the same set. It also callspdns_ffi_param_set_log_response(obj, false)fordns-health.fzymgc.houseandhealth.dns.invalid, so the recursor exports no row for them.prerpzfinds the same set again fromdq.remoteaddr, and callsdq:discardPolicyfor every owned zone that is not in the set.
The script reads the line # version <n> of the scope map into the custom metric
scope-map-version, which rec_control get scope-map-version prints. The prototype of spike 011
is .planning/spikes/011-recursor-rpz-policy/harness/policy.lua.
The tag partitions the packet cache. Without it, a scoped NXDOMAIN in the packet cache answers a client outside the scope. The PowerDNS mailing list shows this failure in April 2022. Today every leg declares the same list groups, so every client on a declared leg in no scope shares one partition. The cache cost grows with the number of distinct zone sets, not with the number of clients.
The script is the one piece of code in the data plane that this repository owns. It stays small, and the molecule scenario tests it against the real recursor.
Decision 6: The feeds
Section titled “Decision 6: The feeds”Each list group becomes one zone file on each node. The groups, their sources and the leg
declarations stay in Ansible and move to the new role as dns_list_sources,
dns_default_list_groups and the leg attribute dns_list_groups.
A systemd timer on each node starts a oneshot unit once a day. For each group with a URL source,
the unit runs dns-node-worker feed. The command downloads the source to a temporary file and
writes the zone with the mapping rule of Decision 2. It makes sure that the new zone holds at
least half the entries of the zone that it replaces, and then it renames the zone into place.
After the last group, the unit runs rec_control reload-yaml for each process. The role renders
the zone of an inline group, such as domains. The timer runs at 03:30 on sandiego and at 04:30
on portland, so the two nodes never reload at the same time.
A failed download keeps the last zone, so a source that cannot be reached never empties a feed.
That keeps the requirement “A failed list source does not stop the resolver” true. A failed
group does not stop the other groups. The unit ends failed when any group failed, and the
journal names the group.
The converter is code of this repository. It shares the mapping rule with the projection of
Decision 3, and the unit tests of internal/lists cover both. HaGeZi also publishes an RPZ
edition, but the crypto list has none, and one converter for every source keeps one rule for
every match.
The feeds never enter pdns-auth. They are about 917,000 entries of external data that change every day, and the LMDB of pdns-auth and its LightningStream copies through the NAS have no use for them.
Decision 7: Two processes on each node
Section titled “Decision 7: Two processes on each node”The package ships the unit template pdns-recursor@.service. The role runs pdns-recursor@main
and pdns-recursor@public as the user pdns. --config-name makes each one read its own file,
/etc/powerdns/recursor-main.yml or /etc/powerdns/recursor-public.yml, and each has its own
API socket on loopback. The public process has no forward zone to pdns-auth and no zone from
pdns-auth. Its forward_zones_recurse names only dnscrypt-proxy. It runs the Lua script of
Decision 5 with its own leg map and no scope map. The isolation of the untrusted legs stays a
process boundary, as it is today.
Decision 8: The decision rows and the engine-neutral names
Section titled “Decision 8: The decision rows and the engine-neutral names”Each process sends protobuf to a reader on loopback. The reader writes one JSON line for each
answer to the journal, and the OTel collector parses it as it parses the Blocky rows today.
Spike 011 chose DNS-collector 3.1.0: its powerdns input on one port for each process, and its
stdout output in flat-json mode. Each process sends responses only (logQueries: false), and
sets recursor.server_id to its ServiceName. A row reaches the reader about one second after the
answer. The fields that the collector maps:
| Reader field | Name |
|---|---|
dnstap.identity |
ServiceName |
network.query-ip |
client.address |
dns.qname, dns.qtype |
dns.question.name, dns.question.type |
dns.rcode, dns.resource-records.an.rdatas |
dns.response.code, dns.answer |
powerdns.applied-policy |
dns.policy.zone, empty when no rule decided |
powerdns.applied-policy-trigger, powerdns.applied-policy-hit |
dns.policy.trigger, dns.policy.hit |
powerdns.applied-policy-kind |
dns.policy.action: NXDOMAIN is nxdomain, NODATA is nodata, and NoAction with a zone is passthru |
A row with no rule also has the kind NoAction, so the policy object of dns-control is null
exactly when the zone is empty. The reader filters nothing. DNS-collector 3.1.0 holds back the
rows before a row that a transform drops, until the next row arrives, so the Lua script keeps
the health names out of the export (Decision 5).
The rows use these names. Where an OpenTelemetry semantic convention exists, the name follows it:
| Name | Holds |
|---|---|
ServiceName dns-resolver and dns-resolver-public |
the process that answered |
client.address |
the client address |
dns.question.name, dns.question.type |
the question |
dns.response.code, dns.answer |
the answer |
dns.policy.zone, dns.policy.trigger, dns.policy.hit, dns.policy.action |
the rule that decided, or empty |
The collector also writes the rows of Blocky with these attribute names from pull request 3, so
dns-control reads one shape across the cut-over. The ServiceName of a Blocky row stays blocky
or blocky-public, because it names the unit that wrote the row. dns-control reads all four
ServiceNames until the cleanup. The health and the error guard read
dns_resolver_responses_total and dns_resolver_errors_total. The collector maps the Blocky
counters onto these names first, and the recursor counters at the cut-over. No row of
dns-health.fzymgc.house or health.dns.invalid reaches the collector: Blocky leaves them out
with queryLog.ignore, and the recursor with the Lua script.
The row filter of both MCP users names all four ServiceNames before any row carries a new name. Without that order, the MCP users read the query names of every client.
Decision 9: Parity with the Blocky features in use
Section titled “Decision 9: Parity with the Blocky features in use”| Blocky today | The recursor | Proof |
|---|---|---|
One listener per address, freeBind for the VIPs |
incoming.listen with each address, incoming.non_local_bind |
molecule |
| dnscrypt-proxy as the only upstream | recursor.forward_zones_recurse for . to 127.0.0.1:5300 |
molecule, and the spec dns-upstream-anonymity |
conditional for fzymgc.house and the reverse zones |
recursor.forward_zones to 127.0.0.1:1053, with no fallback |
molecule. Spike 011: serve_rfc1918 does not hide a forwarded reverse zone |
customDNS for whoami.dns.invalid and the special-use table for health.dns.invalid |
recursor.auth_zones with the zone dns.invalid. from a file for each node. The rebinding zone also applies to it, so the public process answers whoami.dns.invalid with NODATA, as blocky-public answers no customDNS name today |
molecule, and both keepalived scripts unchanged |
blockType: nxDomain, blockTTL: 30s |
CNAME ., policy TTL 30 |
molecule |
blocky_list_sources, with a URL or inline text for each group |
the feed unit, dns-node-worker feed and the zones that the role renders |
unit tests of the converter, and molecule |
blocky_list_groups on each leg, blocky_default_list_groups |
the leg map of the Lua script | molecule, and spike S3 |
the group internal-namespace for a client on no declared leg |
withheld.rpz.internal. |
molecule, and spike S3 |
REFUSED from dnscrypt-proxy for an internal name, passed on by blocky-public |
withheld.rpz.internal. in the public process, with NXDOMAIN |
molecule, and the spec dns-upstream-anonymity |
rebindingProtection on the public process |
rebinding.rpz.internal., before the allow zone |
spike S6 |
rateLimit: 100 a second, burst 200, per /32 and /128 |
an nftables meter in dns-common, with the same values and the same allowlist |
molecule of dns-common |
caching: TTL as given, negative 30 seconds |
recordcache with max_negative_ttl: 30 |
molecule |
caching.exclude for fzymgc.house |
none. See Decision 10 | none |
/api/cache/flush, for the whole cache only |
the wipe action of Decision 12, for one name, a name with its subtree, or . |
molecule |
Prometheus on 127.0.0.1:4000 and :4001 |
the webserver of each process on loopback | the scrape job |
queryLog to the journal |
Decision 8 | spike S4 |
Decision 10: The cache and the owned names
Section titled “Decision 10: The cache and the owned names”Blocky never caches an answer for fzymgc.house. The recursor caches it for its TTL, which is
300 seconds for a dynamic name. For a record write through dns-control, which is a change of an
entry, a zone or an RRset, a step on each node runs the wipe action of Decision 12 with subtree
set for each name that the change wrote. A name that Kea D2
writes keeps its TTL in the cache. A client caches the same answer for the same TTL anyway.
Decision 11: The node worker serves both engines until the cleanup
Section titled “Decision 11: The node worker serves both engines until the cleanup”The dns-node-worker role renders the engine of its node into the worker configuration. The
worker reports it in Status, and the workflow picks the steps of each node from it. Between
pull request 6 and pull request 10, one node can run Blocky while the other runs the recursor.
Status.blocky (field 8) and Status.blocky_config (field 11) become reserved in the cleanup,
and a new ResolverReading takes their place.
The cleanup deletes the Blocky messages and reserves the number and the name of both fields. The
proto gate checks the category WIRE_JSON from then on. FILE protects generated code for
callers outside the repository, and it refuses every deleted field and message, also a reserved
one. The worker and dns-control are the only callers, and both live in this repository.
The ops action refresh stays. Its node activity node.list_refresh starts dns-feeds.service
and waits until the unit ends, so an operator can load the feeds at once. A run takes 7 to 9
seconds on each node. The action takes one node after the other, so the two nodes never reload
at the same time. The unit drops every cached answer, so a refresh reads the flush guard and takes
no force. Each download of the unit stops after 60 seconds, and the activity waits 5 minutes.
The worker decodes its configuration file strictly, and so does dns-control. The pin of the new
worker removes the two Blocky keys of the worker configuration in the same pull request. The
updater pull request of dns-control removes policy.scopes_enabled on its branch.
Decision 12: The operator wipes a name from the caches
Section titled “Decision 12: The operator wipes a name from the caches”The recursor caches a public answer for its TTL, as Blocky does. An operator who sees a stale answer in the query log needs to remove that one name. Blocky can only drop its whole cache. The recursor removes one name, or a name and every name under it, from its record cache, its negative cache and its packet cache.
The node worker has one wipe action, node.cache_wipe. Its input holds the names and the flag
subtree. For each name, the worker calls
PUT /api/v1/servers/localhost/cache/flush?domain=<name>&subtree=<flag> on the API of each
process. It records the count that each process returns. The worker already reads
rpzstatistics through this API with the key file of the role (Decision 4), so the wipe needs no
new access.
Three callers use the action:
| Caller | Names | subtree |
Guard |
|---|---|---|---|
| A record write or a list entry (Decisions 4 and 10) | each name of the change | set | none |
The operator wipe: dns_wipe_name, POST /api/ops/wipe and the Query log |
one name | the choice of the operator | none |
The flush of a recursor node: dns_flush_cache |
. |
set | the error guard, as today |
The operator wipe is the ops action wipe. It takes the same lock as the other ops actions. It
runs on both nodes, the node that holds no VIP first, and writes one audit row with the name and
the flag. It needs the role dns-operator, and an agent identity can run it. It reads no guard
and takes no force. The guard exists because a flush during an upstream outage turns every
cached answer into SERVFAIL. A wipe of one name loses the answers of that name only.
Each wipe reaches both processes. A Guest client asks the public process, so a stale public answer for that client is in the cache of the public process. A process that holds no entry for the name answers with the count 0.
If one node runs Blocky, the wipe refuses before it reaches any node, with the reason
engine_cannot_wipe. A whole flush of the Blocky node in its place drops every cached answer to
remove one. The flush of a Blocky node keeps /api/cache/flush until the cleanup.
Alternative: rec_control wipe-cache <name>$, which spike 011 ran. The runner of the worker runs
a closed set of commands, each with a fixed argument list, and a name breaks that rule. The
command also prints its counts as text, and the API returns them as JSON.
The spike runs in a throwaway container with Debian 13 on arm64, with the recursor and a pdns-auth with the LMDB backend. It uses the newest stable suite of the recursor that repo.powerdns.com publishes for trixie, and it records that version. If the suite is 5.5 or later, Decision 1 takes that version. Each item ends with a command that passes or fails.
| Item | Question | Pass |
|---|---|---|
| S1 | Memory: both processes with every feed, after the load and at the peak of a feed reload | The two processes together stay under 1,500 MiB of RSS. The node keeps 1.5 GiB available |
| S2 | Feed reload: rec_control reload-yaml after a new feed file |
A loop of dig during the reload gets no timeout and no SERVFAIL, and the record cache keeps its entries |
| S3 | Zone sets: two clients, one name, one scope, and a client on no declared leg | The scoped client gets NXDOMAIN and the other gets an address, in both orders of the first query. The client on no declared leg gets NXDOMAIN for dns-health.fzymgc.house |
| S4 | Rows: one block and one pass-through give one JSON line each in the journal | Each line names the zone, the trigger and the action, and the health names give no line |
| S5 | Transfer: AXFR of an owned zone from pdns-auth on loopback, a NOTIFY from 127.0.0.1, and rpzstatistics |
The new serial is loaded within 5 seconds of the NOTIFY. A name that the process answered before the new block entry gets NXDOMAIN right after the load |
| S6 | Rebinding: an owned name whose public answer is a private address, from the public process | The answer is NODATA, with the allow zone in place |
| S7 | Include and reload: a new RPZ in a file of the include directory, then rec_control reload-yaml |
The new zone is in rpzstatistics with no restart |
If S1, S3 or S7 fails, the design stops at the spike and comes back with the result. Knot Resolver is then the next candidate, and Decisions 2, 3 and 8 stay.
Results
Section titled “Results”Spike 011 (.planning/spikes/011-recursor-rpz-policy/) passed all seven items in four runs on
2026-10-09 and 2026-10-10 UTC, with Recursor 5.4.7, pdns-auth 5.1.4 on LMDB and DNS-collector 3.1.0. run.sh <scratch> in that
directory runs them again.
| Item | Reading |
|---|---|
| S1 | Each process holds 1,555,410 to 1,635,440 policy records. Both processes take 415 to 434 MiB after the load and 1,103 to 1,151 MiB at the peak. After the third reload they stay at the peak |
| S2 | 24,436 to 29,356 queries during reload-yaml of both processes: no SERVFAIL and no timeout. A reload takes 1.1 to 1.6 s for each process |
| S3 | The scoped answer reaches the member only, in both orders. The control without the tag leaks in both orders |
| S4 | One row for each answer, with the zone, the trigger and the action. No row for the health names |
| S5 | The new serial loaded 10 to 11 ms after the NOTIFY. The cached name needed wipe-cache |
| S6 | NOERROR with no record from the public process, also for a name in the allow zone |
| S7 | The new scope zone loaded with reload-yaml, and the process kept its PID and its start time |
The memory after the change, from a read-only reading of both nodes on 2026-10-09: sandiego keeps 3,100 + 413 − 1,151 = 2,362 MiB available, and portland 3,058 + 438 − 1,151 = 2,345 MiB. The second number in each sum is the RSS of the two Blocky processes. The steady state is about 370 bytes for each record in each process, so 1,500 MiB holds about 2.1 million records in each process. The harness ran 4 KiB pages, and both nodes run 16 KiB pages, so task 9.13 reads the real RSS on the first node.
The results change Decisions 2, 4, 5 and 8 and the parity table of Decision 9. They touch these
tasks: 4.6 and 4.9 (the order of a scope write and the dump file), 4.8 (wipe-cache after the
load), 8.2 (the three configuration files and policyName), 8.3 (the health names in the
script), 8.4 (a record limit for the feeds), 8.5 (the reader), 9.5 (no drop of the health rows in the collector) and 9.13 (the RSS
reading).
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [The memory of a feed reload] → S1 measured a peak of 1,103 to 1,151 MiB for both processes, and the same value after three reloads. A feed reload runs once a day, at a time that the timer sets apart on the two nodes.
- [The nodes run 16 KiB pages, and S1 ran 4 KiB pages] → After the converge on the node that holds
no VIP and one
reload-yamlof each process, task 9.13 reads the RSS withps -o rss= -C pdns_recursor. The sum must stay under 1,500 MiB. - [The feeds grow] → TIF medium grew by 80,000 records between two runs of S1 on consecutive days. The feeds of the second run use 77 percent of the 2.1 million records that 1,500 MiB holds. Task 8.4 decides whether the feed unit refuses a set of feeds above a record limit.
- [A scope that is deleted and created again keeps its old entries from the dump file] → The worker deletes the dump file of the zone when it deletes the scope (Decision 4).
- [DNS-collector holds back rows before a dropped row] → The reader drops nothing, and the Lua script keeps the health names out of the export (Decision 8).
- [A wrong Lua script leaks a scoped answer through the packet cache] → S3 and the molecule scenario test both orders of the first query.
- [
reload-yamlreloads every RPZ of the process, also the feeds] → A new scope is seldom. S2 and S7 measure the cost. - [LightningStream or the NAS is down] → A write reaches only portland. The serial wait times out on sandiego and the change fails, as a record change does today. The next projection repairs it.
- [The protobuf reader is one more daemon on each node] → It runs under systemd with no state.
The recursor queues up to
maxQueuedEntriesmessages while the reader restarts and answers queries without it. - [A rename of a ServiceName or an attribute breaks a ClickStack alert] → Pull request 2 changes the alert and the row filter before any name changes.
- [The recursor caches owned names] → Decision 10.
- [A failed first node no longer keeps a list change off the second node] → Decision 4. The step order still gives the node that holds no VIP the NOTIFY first.
- [A cached answer outlives a new block entry] → S5 showed that 5.4 keeps the packet cache entry after a load of the RPZ. Only 5.5 clears it. The step wipes each changed name (Decisions 4 and 12).
- [The API refuses the root name
.for the flush] → The molecule scenario flushes.through the API. If the API refuses it, the flush runsrec_control wipe-cache .$, which has a fixed argument list. - [A wipe misses the packet cache partition of another zone set] → The molecule scenario caches one name for two zone sets, wipes it, and makes sure that a second wipe counts zero.
Migration Plan
Section titled “Migration Plan”The pull requests follow the split of proposal.md. Standing constraint C3 holds for every
converge: one node per window, and the node that holds no VIP first.
scripts/dns-vip-holder.py names the holder.
-
Pull requests 2 and 7 are ArgoCD syncs. Rollback: revert the pull request.
-
Pull requests 3 and 6 are converges of one role. Rollback: revert the pull request and converge the same tag on each node, for example
scripts/dns-playbook.sh --limit sandiego --tags dns-otel-collector. -
Pull request 9 is the cut-over. On each node, the play stops and disables
blockyandblocky-publicand leaves their files in place. It then starts both recursor processes and convergeskeepalived,dns-otel-collector,dns-commonanddns-node-worker. Before the second node,dns_drain_nodemoves the VIPs off it. After both nodes run for one day, the retire playblocky-retire.ymlremoves the binary, the units, the configuration and the operator files, throughscripts/dns-playbook.sh --playbook=blocky-retire.yml. Rollback on a node before the retire play:Terminal window scripts/dns-adhoc.sh <node> -b -m ansible.builtin.shell -a 'systemctl disable --now pdns-recursor@main pdns-recursor@public && systemctl enable --now blocky blocky-public'After the retire play, the rollback is a revert of pull request 9 and a converge of
--tags blockyon each node. -
Pull request 10 is a release and two pins, with the rollback of each pin.
Open Questions
Section titled “Open Questions”None. Spike 011 settled the version of the recursor (5.4.7) and the protobuf reader (DNS-collector 3.1.0).