Skip to content

Design: DNS policy as RPZ

See openspec/changes/dns-policy-as-rpz/proposal.md for the reason and the scope. The specs of this change are under openspec/changes/dns-policy-as-rpz/specs/.

RPZ (response policy zone) is a DNS zone whose records are rules. The owner name of a record is the trigger, and its data is the action. example.org CNAME . answers NXDOMAIN for example.org, and example.org CNAME rpz-passthru. lets the name resolve. A resolver consults its policy zones in a fixed order, and the first zone that matches decides.

The parts of the estate that this design replaces or touches:

  • Each resolver node runs two Blocky processes from one binary. blocky serves the trusted legs, the tailnet and 127.0.0.1. blocky-public serves Guest, Telework and 127.0.0.3, and its configuration has no conditional and no customDNS, so no query of it reaches pdns-auth. Neither has a DoT or DoH listener.
  • Four list groups block names. hagezi is HaGeZi Pro++, tif is TIF medium, crypto is the Block List Project crypto list, and domains holds three names written inline. Each leg declares its groups in blocky_list_groups on its dns_vlan_interfaces entry, and today every leg declares all four. A client on no declared leg gets blocky_default_list_groups.
  • The main process withholds fzymgc.house and the reverse zones from a client on no declared leg. The group internal-namespace answers NXDOMAIN for them. The tailnet and 127.0.0.1 are outside that group.
  • blocky-public forwards a name under fzymgc.house to dnscrypt-proxy, which answers REFUSED, and Blocky passes REFUSED to the client. The recursor treats a REFUSED from its forwarder as a failed server and answers SERVFAIL.
  • Both processes forward every public name to dnscrypt-proxy on 127.0.0.1:5300. The main process sends fzymgc.house and the reverse zones to pdns-auth on 127.0.0.1:1053, with fallbackUpstream: false.
  • The memory on 2026-10-09: each node has 4,049 MiB and about 3,000 MiB available. The two Blocky processes use 211 to 237 MiB each. HaGeZi Pro++ holds 221,621 entries, TIF medium 693,959 and the crypto list about 1,300.
  • keepalived tracks two scripts. chk_house_answer asks 127.0.0.1 for dns-health.fzymgc.house and passes on an address under 192.168.. chk_public_answer asks 127.0.0.3 for health.dns.invalid and passes only on NXDOMAIN, which Blocky answers from its special-use table with no upstream query.
  • dns-control writes pdns through internal/pdns on the writer node, portland at http://192.168.20.7:8081. LightningStream copies the LMDB of pdns-auth to sandiego through S3 on the NAS. A copied write sends no NOTIFY. zone-cache-refresh-interval is 60 seconds, so a new zone appears on the other node up to 60 seconds after LightningStream copies it. A changed record appears at once.
  • The lineage index (internal/refresh) and the converge (internal/change/converge.go) read every zone that pdns-auth lists.
  • internal/querylog reads the rows of the ServiceNames blocky and blocky-public from default.otel_logs. health.go and the error guard read blocky_response_total and blocky_error_total. explain.go parses the Blocky reason BLOCKED (group: rule).
  • The ClickHouse row filter of both MCP users hides every row with the ServiceName blocky or blocky-public. Only dns_control_reader reads those rows.
  • The node worker writes /var/lib/dns-worker/lists/operator-denylist.txt and operator-allowlist.txt, and posts /api/lists/refresh and /api/cache/flush to Blocky. nodeOrder puts the node that holds no VIP first.

docs/reference/network.md is the authority for every address in this design: the Main VIP 192.168.20.5, sandiego .20.6, portland .20.7, the Guest VIP 192.168.79.5 and the Lab VIP 192.168.200.5. The recursor listens on the addresses that Blocky listens on today. This change allocates no address and no prefix. No Terraform resource changes.

Goals:

  • Every rule set is an RPZ zone, and the zone order is the whole precedence.
  • A list entry change and a scope change are in force on both nodes within one minute, with no restart of a resolver process.
  • Every answer gives one row in ClickStack that names the client, the question, the answer, and the zone and the rule that decided it.
  • dns-control reads engine-neutral names only.
  • The entries, the scopes and the feeds hold while dns-control, k3s, Temporal or the NAS is down.
  • Every Blocky feature in use today has a named equivalent. The parity table under Decision 9 lists them.

Non-Goals:

  • A scope on Guest or Telework. The public process loads no zone of dns-control.
  • Moving the excepted names, the owned zones or the feed list into dns-control.
  • Client identity by MAC address or Kea reservation. A scope member is an address, a CIDR or a leg. A later change can add a device member.
  • DNSSEC validation in this tier. The tier does not validate today, and dnscrypt-proxy stays its only public upstream.
  • A DoT or DoH listener.

Decision 1: PowerDNS Recursor is the engine

Section titled “Decision 1: PowerDNS Recursor is the engine”

On 2026-10-09 Sean chose PowerDNS Recursor 5.4 from the suite trixie-rec-54 of repo.powerdns.com. The role installs it from the repository that the pdns role already trusts, with the same key and the same pinned Debian version string. Spike 011 installed 5.4.7 (5.4.7-1pdns.debian13), the newest stable release. trixie-rec-55 holds only 5.5.0~alpha1, so the role pins 5.4.7.

Engine RPZ input Scope by client Reason for each query Why not
PowerDNS Recursor 5.4 IXFR or AXFR from a primary, or a file Lua hooks gettag_ffi and prerpz protobuf appliedPolicy, appliedPolicyTrigger, appliedPolicyHit, appliedPolicyKind chosen
Knot Resolver 6.5 files only declarative views, from subnet to tags RPZ match lines at notice level, and dnstap without the policy no reason for each query, and no zone transfer
Unbound 1.23 AXFR or url: declarative tags rpz-log text lines users report several GB for a 1M-entry RPZ, and fast_reload is experimental
CoreDNS none in-tree view plugin log plugin, with no reason blocking needs a third-party plugin and our own build
AdGuard Home its own rule syntax $client rules its own JSON log it rewrites its own YAML file, and it has no metrics without an exporter
Blocky 0.35 none static groups JSON line with the list a scope needs a restart, and there is no reload upstream

The maintainer of DNS-collector states that no DNS server except PowerDNS Recursor exports the RPZ hit for each query. That property and the zone transfer decide the choice. Knot Resolver is the fallback. Decisions 2, 3 and 8 hold the same for it, so a later change of engine changes one role and the scope mechanism only.

Decision 2: The zones, their names and their order

Section titled “Decision 2: The zones, their names and their order”

The owned zones live under rpz.internal.. .internal is the top-level domain that ICANN reserved for private use in 2024. The recursor forwards no name under it to pdns-auth, so no client reaches a policy zone through the resolver.

The main process consults these zones in this order:

Order Zone Source Holds
1 withheld.rpz.internal. a file that the role renders fzymgc.house and the reverse zones, as NXDOMAIN. It applies to a client on no declared leg only
2 allow.rpz.internal. a file that the role renders the owned zones and the excepted names, as rpz-passthru.
3 <scope>.scope.rpz.internal. pdns-auth the entries of one scope, in the order of the scopes
4 operator.rpz.internal. pdns-auth the entries with no scope
5 <group>.feed.rpz.internal. a file on each node one zone for each list group that the leg of the client declares, in the order hagezi, tif, crypto, domains

The public process consults the withheld zone for every client, then rebinding.rpz.internal., then the allow zone, then the feed zones of the leg of the client. The withheld zone answers NXDOMAIN for an internal name before any forwarder sees it, so a Guest client gets NXDOMAIN in place of the REFUSED of today. rebinding.rpz.internal. holds rpz-ip triggers for every private, ULA, loopback and link-local range, with the action NODATA. It replaces rebindingProtection of blocky-public, which answers an empty NOERROR.

The order sets four rules:

  • A client of the public process, and a client of the main process on no declared leg, never gets an internal name. No such query leaves the process.
  • An owned name or an excepted name always resolves for every other client.
  • A scoped entry beats an entry with no scope. A scoped allow unblocks a name that the operator zone blocks, and a scoped block blocks a name that the operator zone allows.
  • An operator entry beats every feed.

One rule maps an entry to records, for a list entry and for a line of a feed. A plain name example.org becomes example.org CNAME ., which matches that name only. A wildcard name *.example.org becomes example.org CNAME . and *.example.org CNAME ., which match the name and every name under it. An allow entry uses rpz-passthru. in place of .. These are the matches of a Blocky entry today. The policy TTL of the owned zones is 30 seconds, which is blockTTL today.

The configuration of the main process spreads this list over three files, and spike 011 showed the layout. A sequence in a file of the include directory is appended to the same sequence of the files before it, so the order of the files is the order of the zones:

File Holds
/etc/powerdns/recursor-main.yml the withheld zone and the allow zone
recursor-main.d/50-scopes.yml each scope zone, and its name in incoming.allow_notify_for
recursor-main.d/90-tail.yml the operator zone and the feed zones

Every RPZ sets policyName to its zone name with the trailing dot, such as hagezi.feed.rpz.internal.. Without it, a zone from a file has the name rpzFile, and the Lua script and the protobuf rows cannot tell the feeds apart.

Alternative: one zone for every operator entry, with the scope held in Lua. That gives up the standard format for exactly the rules that change most. The protobuf then names one zone for every scoped decision.

Decision 3: The owned zones are a projection of the list store

Section titled “Decision 3: The owned zones are a projection of the list store”

The list store and the scope store of dns-control are the source of truth. The zones in pdns-auth are their projection, as the operator files are today. The projection creates operator.rpz.internal. once, creates a scope zone when a scope is created, and deletes it when the scope is deleted. A projection writes the whole set again, as the requirement “A projection writes the whole set again” of dns-operator-list says today.

The zones are kind Native with SOA-EDIT-API set, so every API write raises the serial and pdns-auth sends no NOTIFY. The lineage index and the converge skip every zone under rpz.internal.. One predicate in internal/pdns holds that suffix. The list projection keeps its own drift read: it compares each zone with the store.

The role pdns renders allow-axfr-ips=127.0.0.1/32,::1/128 explicitly, because its template renders every setting and inherits none.

Alternative: the policy zones as curated zones in the zone store. Then a curated-zone screen, the export of zones.json and the drift view show records that no person writes.

There are three kinds of change, and each has its own path:

Change How often Path on each node, the node that holds no VIP first
A list entry often The worker waits until pdns-auth on loopback serves the new serial. It sends a NOTIFY for the zone to the main process, and reads the loaded serial from rpzstatistics of the recursor API. Then it runs the wipe action of Decision 12 with subtree set for each name of the change
The members of a scope sometimes The worker writes the scope map file with a temporary file and a rename, runs rec_control reload-lua-script, and reads back the map version with rec_control get scope-map-version
A new scope seldom The worker writes the scope map that names the new scope and runs reload-lua-script. Then it writes 50-scopes.yml and runs rec_control reload-yaml and rec_control reload-zones. Then it takes the list-entry path
A deleted scope seldom The worker writes 50-scopes.yml without the scope and runs reload-yaml and reload-zones. Then it deletes the dump file of the zone and writes the scope map without the scope

Spike 011 sets the order of these steps:

  • Recursor 5.4 does not wipe the packet cache when an RPZ loads, so a cached answer outlives a new entry until the step wipes it. 5.5 wipes the packet cache on a load, and the wipe step of a list entry can go with an upgrade to a stable 5.5.
  • The Lua script discards only the scopes that the scope map names. A scope zone that is loaded before the map names it applies to every client. So the map comes first for a new scope and last for a deleted one.
  • incoming.allow_notify_for from the include takes effect only with reload-zones. Before it, the main process ignores a NOTIFY for the new zone. A parent name such as rpz.internal does not cover the zones under it.
  • The recursor seeds a zone from its dump file. A scope that is deleted and created again on the same day gets the same date serial from pdns-auth, and the recursor keeps the old entries from the dump file. rpzstatistics then shows the serial of pdns-auth, so only the delete of the dump file prevents this.
  • reload-yaml exits 1 on an include with invalid YAML, and the process keeps its zones. The step fails on that exit code.

In spike 011 the main process loaded a new serial 10 ms after the NOTIFY, with a full transfer. The serial wait covers LightningStream and the 60-second zone cache. It times out after 120 seconds. A timeout fails the step. A failed step sends no NOTIFY to the next node, and the change ends failed, as a failed node.list_write does today. Each RPZ in the configuration also has refresh: 300, so a node picks up a change within five minutes when dns-control is down after a write.

This changes one property of today. pdns-auth on the next node already holds the change, so its main process loads the change by refresh within five minutes, also after a failed step. The operator files of Blocky kept the change off the second node. A list entry is data and not configuration, so the canary value of that property is small. The failed change still tells the operator which node did not load the change in time.

Each step records the serial of each owned zone that the change wrote and the serial that its main process loaded. The recursor keeps a copy of each transferred zone in its dumpFile and starts from it as seedFile. A restart of the recursor with pdns-auth down still starts with every entry.

Alternative: rely on refresh alone, with a short interval. Then both nodes take a change at about the same time, the passive-first order goes away, and a step cannot read back what its node holds.

Decision 5: Scopes, and the zones that apply to a client

Section titled “Decision 5: Scopes, and the zones that apply to a client”

A scope has a name, an order and members. The name is a DNS label of 1 to 40 characters from a-z, 0-9 and -, because it becomes a label of the zone name. The order is a unique integer that sets the place of the scope zone in Decision 2. A member is an address, a CIDR or a trusted leg. dns-control resolves a leg to its IPv4 prefix and its ULA /64 from internal/legs, and refuses Guest and Telework. A list entry names zero or one scope, and the store holds one entry for each name in each scope. A scope change is a change of the subject block, as a list change is, so the audit rows, the change metric and its alerts cover it.

The Lua script of each process decides which zones apply to a client. It reads two maps:

  • The leg map, which the role renders from dns_vlan_interfaces. Each line holds a CIDR and the list groups of its leg. The map of the main process also names the tailnet and 127.0.0.1.
  • The scope map /var/lib/dns-worker/scopes.map, which the worker writes. Each line holds one scope name and one CIDR. The public process has no scope map.

The script does two things:

  • gettag_ffi finds the zone set of the client: the withheld zone when no line of the leg map holds the address, the feed zones of its leg or of dns_default_list_groups, and every scope whose members hold the address. In the public process, the set always holds the withheld zone. It sets one packet cache tag for each distinct zone set. The tag is a hash of the sorted zone names, so a reload of the script keeps each tag on the same set. It also calls pdns_ffi_param_set_log_response(obj, false) for dns-health.fzymgc.house and health.dns.invalid, so the recursor exports no row for them.
  • prerpz finds the same set again from dq.remoteaddr, and calls dq:discardPolicy for every owned zone that is not in the set.

The script reads the line # version <n> of the scope map into the custom metric scope-map-version, which rec_control get scope-map-version prints. The prototype of spike 011 is .planning/spikes/011-recursor-rpz-policy/harness/policy.lua.

The tag partitions the packet cache. Without it, a scoped NXDOMAIN in the packet cache answers a client outside the scope. The PowerDNS mailing list shows this failure in April 2022. Today every leg declares the same list groups, so every client on a declared leg in no scope shares one partition. The cache cost grows with the number of distinct zone sets, not with the number of clients.

The script is the one piece of code in the data plane that this repository owns. It stays small, and the molecule scenario tests it against the real recursor.

Each list group becomes one zone file on each node. The groups, their sources and the leg declarations stay in Ansible and move to the new role as dns_list_sources, dns_default_list_groups and the leg attribute dns_list_groups.

A systemd timer on each node starts a oneshot unit once a day. For each group with a URL source, the unit runs dns-node-worker feed. The command downloads the source to a temporary file and writes the zone with the mapping rule of Decision 2. It makes sure that the new zone holds at least half the entries of the zone that it replaces, and then it renames the zone into place. After the last group, the unit runs rec_control reload-yaml for each process. The role renders the zone of an inline group, such as domains. The timer runs at 03:30 on sandiego and at 04:30 on portland, so the two nodes never reload at the same time.

A failed download keeps the last zone, so a source that cannot be reached never empties a feed. That keeps the requirement “A failed list source does not stop the resolver” true. A failed group does not stop the other groups. The unit ends failed when any group failed, and the journal names the group.

The converter is code of this repository. It shares the mapping rule with the projection of Decision 3, and the unit tests of internal/lists cover both. HaGeZi also publishes an RPZ edition, but the crypto list has none, and one converter for every source keeps one rule for every match.

The feeds never enter pdns-auth. They are about 917,000 entries of external data that change every day, and the LMDB of pdns-auth and its LightningStream copies through the NAS have no use for them.

The package ships the unit template pdns-recursor@.service. The role runs pdns-recursor@main and pdns-recursor@public as the user pdns. --config-name makes each one read its own file, /etc/powerdns/recursor-main.yml or /etc/powerdns/recursor-public.yml, and each has its own API socket on loopback. The public process has no forward zone to pdns-auth and no zone from pdns-auth. Its forward_zones_recurse names only dnscrypt-proxy. It runs the Lua script of Decision 5 with its own leg map and no scope map. The isolation of the untrusted legs stays a process boundary, as it is today.

Decision 8: The decision rows and the engine-neutral names

Section titled “Decision 8: The decision rows and the engine-neutral names”

Each process sends protobuf to a reader on loopback. The reader writes one JSON line for each answer to the journal, and the OTel collector parses it as it parses the Blocky rows today. Spike 011 chose DNS-collector 3.1.0: its powerdns input on one port for each process, and its stdout output in flat-json mode. Each process sends responses only (logQueries: false), and sets recursor.server_id to its ServiceName. A row reaches the reader about one second after the answer. The fields that the collector maps:

Reader field Name
dnstap.identity ServiceName
network.query-ip client.address
dns.qname, dns.qtype dns.question.name, dns.question.type
dns.rcode, dns.resource-records.an.rdatas dns.response.code, dns.answer
powerdns.applied-policy dns.policy.zone, empty when no rule decided
powerdns.applied-policy-trigger, powerdns.applied-policy-hit dns.policy.trigger, dns.policy.hit
powerdns.applied-policy-kind dns.policy.action: NXDOMAIN is nxdomain, NODATA is nodata, and NoAction with a zone is passthru

A row with no rule also has the kind NoAction, so the policy object of dns-control is null exactly when the zone is empty. The reader filters nothing. DNS-collector 3.1.0 holds back the rows before a row that a transform drops, until the next row arrives, so the Lua script keeps the health names out of the export (Decision 5).

The rows use these names. Where an OpenTelemetry semantic convention exists, the name follows it:

Name Holds
ServiceName dns-resolver and dns-resolver-public the process that answered
client.address the client address
dns.question.name, dns.question.type the question
dns.response.code, dns.answer the answer
dns.policy.zone, dns.policy.trigger, dns.policy.hit, dns.policy.action the rule that decided, or empty

The collector also writes the rows of Blocky with these attribute names from pull request 3, so dns-control reads one shape across the cut-over. The ServiceName of a Blocky row stays blocky or blocky-public, because it names the unit that wrote the row. dns-control reads all four ServiceNames until the cleanup. The health and the error guard read dns_resolver_responses_total and dns_resolver_errors_total. The collector maps the Blocky counters onto these names first, and the recursor counters at the cut-over. No row of dns-health.fzymgc.house or health.dns.invalid reaches the collector: Blocky leaves them out with queryLog.ignore, and the recursor with the Lua script.

The row filter of both MCP users names all four ServiceNames before any row carries a new name. Without that order, the MCP users read the query names of every client.

Decision 9: Parity with the Blocky features in use

Section titled “Decision 9: Parity with the Blocky features in use”
Blocky today The recursor Proof
One listener per address, freeBind for the VIPs incoming.listen with each address, incoming.non_local_bind molecule
dnscrypt-proxy as the only upstream recursor.forward_zones_recurse for . to 127.0.0.1:5300 molecule, and the spec dns-upstream-anonymity
conditional for fzymgc.house and the reverse zones recursor.forward_zones to 127.0.0.1:1053, with no fallback molecule. Spike 011: serve_rfc1918 does not hide a forwarded reverse zone
customDNS for whoami.dns.invalid and the special-use table for health.dns.invalid recursor.auth_zones with the zone dns.invalid. from a file for each node. The rebinding zone also applies to it, so the public process answers whoami.dns.invalid with NODATA, as blocky-public answers no customDNS name today molecule, and both keepalived scripts unchanged
blockType: nxDomain, blockTTL: 30s CNAME ., policy TTL 30 molecule
blocky_list_sources, with a URL or inline text for each group the feed unit, dns-node-worker feed and the zones that the role renders unit tests of the converter, and molecule
blocky_list_groups on each leg, blocky_default_list_groups the leg map of the Lua script molecule, and spike S3
the group internal-namespace for a client on no declared leg withheld.rpz.internal. molecule, and spike S3
REFUSED from dnscrypt-proxy for an internal name, passed on by blocky-public withheld.rpz.internal. in the public process, with NXDOMAIN molecule, and the spec dns-upstream-anonymity
rebindingProtection on the public process rebinding.rpz.internal., before the allow zone spike S6
rateLimit: 100 a second, burst 200, per /32 and /128 an nftables meter in dns-common, with the same values and the same allowlist molecule of dns-common
caching: TTL as given, negative 30 seconds recordcache with max_negative_ttl: 30 molecule
caching.exclude for fzymgc.house none. See Decision 10 none
/api/cache/flush, for the whole cache only the wipe action of Decision 12, for one name, a name with its subtree, or . molecule
Prometheus on 127.0.0.1:4000 and :4001 the webserver of each process on loopback the scrape job
queryLog to the journal Decision 8 spike S4

Decision 10: The cache and the owned names

Section titled “Decision 10: The cache and the owned names”

Blocky never caches an answer for fzymgc.house. The recursor caches it for its TTL, which is 300 seconds for a dynamic name. For a record write through dns-control, which is a change of an entry, a zone or an RRset, a step on each node runs the wipe action of Decision 12 with subtree set for each name that the change wrote. A name that Kea D2 writes keeps its TTL in the cache. A client caches the same answer for the same TTL anyway.

Decision 11: The node worker serves both engines until the cleanup

Section titled “Decision 11: The node worker serves both engines until the cleanup”

The dns-node-worker role renders the engine of its node into the worker configuration. The worker reports it in Status, and the workflow picks the steps of each node from it. Between pull request 6 and pull request 10, one node can run Blocky while the other runs the recursor. Status.blocky (field 8) and Status.blocky_config (field 11) become reserved in the cleanup, and a new ResolverReading takes their place.

The cleanup deletes the Blocky messages and reserves the number and the name of both fields. The proto gate checks the category WIRE_JSON from then on. FILE protects generated code for callers outside the repository, and it refuses every deleted field and message, also a reserved one. The worker and dns-control are the only callers, and both live in this repository.

The ops action refresh stays. Its node activity node.list_refresh starts dns-feeds.service and waits until the unit ends, so an operator can load the feeds at once. A run takes 7 to 9 seconds on each node. The action takes one node after the other, so the two nodes never reload at the same time. The unit drops every cached answer, so a refresh reads the flush guard and takes no force. Each download of the unit stops after 60 seconds, and the activity waits 5 minutes.

The worker decodes its configuration file strictly, and so does dns-control. The pin of the new worker removes the two Blocky keys of the worker configuration in the same pull request. The updater pull request of dns-control removes policy.scopes_enabled on its branch.

Decision 12: The operator wipes a name from the caches

Section titled “Decision 12: The operator wipes a name from the caches”

The recursor caches a public answer for its TTL, as Blocky does. An operator who sees a stale answer in the query log needs to remove that one name. Blocky can only drop its whole cache. The recursor removes one name, or a name and every name under it, from its record cache, its negative cache and its packet cache.

The node worker has one wipe action, node.cache_wipe. Its input holds the names and the flag subtree. For each name, the worker calls PUT /api/v1/servers/localhost/cache/flush?domain=<name>&subtree=<flag> on the API of each process. It records the count that each process returns. The worker already reads rpzstatistics through this API with the key file of the role (Decision 4), so the wipe needs no new access.

Three callers use the action:

Caller Names subtree Guard
A record write or a list entry (Decisions 4 and 10) each name of the change set none
The operator wipe: dns_wipe_name, POST /api/ops/wipe and the Query log one name the choice of the operator none
The flush of a recursor node: dns_flush_cache . set the error guard, as today

The operator wipe is the ops action wipe. It takes the same lock as the other ops actions. It runs on both nodes, the node that holds no VIP first, and writes one audit row with the name and the flag. It needs the role dns-operator, and an agent identity can run it. It reads no guard and takes no force. The guard exists because a flush during an upstream outage turns every cached answer into SERVFAIL. A wipe of one name loses the answers of that name only.

Each wipe reaches both processes. A Guest client asks the public process, so a stale public answer for that client is in the cache of the public process. A process that holds no entry for the name answers with the count 0.

If one node runs Blocky, the wipe refuses before it reaches any node, with the reason engine_cannot_wipe. A whole flush of the Blocky node in its place drops every cached answer to remove one. The flush of a Blocky node keeps /api/cache/flush until the cleanup.

Alternative: rec_control wipe-cache <name>$, which spike 011 ran. The runner of the worker runs a closed set of commands, each with a fixed argument list, and a name breaks that rule. The command also prints its counts as text, and the API returns them as JSON.

The spike runs in a throwaway container with Debian 13 on arm64, with the recursor and a pdns-auth with the LMDB backend. It uses the newest stable suite of the recursor that repo.powerdns.com publishes for trixie, and it records that version. If the suite is 5.5 or later, Decision 1 takes that version. Each item ends with a command that passes or fails.

Item Question Pass
S1 Memory: both processes with every feed, after the load and at the peak of a feed reload The two processes together stay under 1,500 MiB of RSS. The node keeps 1.5 GiB available
S2 Feed reload: rec_control reload-yaml after a new feed file A loop of dig during the reload gets no timeout and no SERVFAIL, and the record cache keeps its entries
S3 Zone sets: two clients, one name, one scope, and a client on no declared leg The scoped client gets NXDOMAIN and the other gets an address, in both orders of the first query. The client on no declared leg gets NXDOMAIN for dns-health.fzymgc.house
S4 Rows: one block and one pass-through give one JSON line each in the journal Each line names the zone, the trigger and the action, and the health names give no line
S5 Transfer: AXFR of an owned zone from pdns-auth on loopback, a NOTIFY from 127.0.0.1, and rpzstatistics The new serial is loaded within 5 seconds of the NOTIFY. A name that the process answered before the new block entry gets NXDOMAIN right after the load
S6 Rebinding: an owned name whose public answer is a private address, from the public process The answer is NODATA, with the allow zone in place
S7 Include and reload: a new RPZ in a file of the include directory, then rec_control reload-yaml The new zone is in rpzstatistics with no restart

If S1, S3 or S7 fails, the design stops at the spike and comes back with the result. Knot Resolver is then the next candidate, and Decisions 2, 3 and 8 stay.

Spike 011 (.planning/spikes/011-recursor-rpz-policy/) passed all seven items in four runs on 2026-10-09 and 2026-10-10 UTC, with Recursor 5.4.7, pdns-auth 5.1.4 on LMDB and DNS-collector 3.1.0. run.sh <scratch> in that directory runs them again.

Item Reading
S1 Each process holds 1,555,410 to 1,635,440 policy records. Both processes take 415 to 434 MiB after the load and 1,103 to 1,151 MiB at the peak. After the third reload they stay at the peak
S2 24,436 to 29,356 queries during reload-yaml of both processes: no SERVFAIL and no timeout. A reload takes 1.1 to 1.6 s for each process
S3 The scoped answer reaches the member only, in both orders. The control without the tag leaks in both orders
S4 One row for each answer, with the zone, the trigger and the action. No row for the health names
S5 The new serial loaded 10 to 11 ms after the NOTIFY. The cached name needed wipe-cache
S6 NOERROR with no record from the public process, also for a name in the allow zone
S7 The new scope zone loaded with reload-yaml, and the process kept its PID and its start time

The memory after the change, from a read-only reading of both nodes on 2026-10-09: sandiego keeps 3,100 + 413 − 1,151 = 2,362 MiB available, and portland 3,058 + 438 − 1,151 = 2,345 MiB. The second number in each sum is the RSS of the two Blocky processes. The steady state is about 370 bytes for each record in each process, so 1,500 MiB holds about 2.1 million records in each process. The harness ran 4 KiB pages, and both nodes run 16 KiB pages, so task 9.13 reads the real RSS on the first node.

The results change Decisions 2, 4, 5 and 8 and the parity table of Decision 9. They touch these tasks: 4.6 and 4.9 (the order of a scope write and the dump file), 4.8 (wipe-cache after the load), 8.2 (the three configuration files and policyName), 8.3 (the health names in the script), 8.4 (a record limit for the feeds), 8.5 (the reader), 9.5 (no drop of the health rows in the collector) and 9.13 (the RSS reading).

  • [The memory of a feed reload] → S1 measured a peak of 1,103 to 1,151 MiB for both processes, and the same value after three reloads. A feed reload runs once a day, at a time that the timer sets apart on the two nodes.
  • [The nodes run 16 KiB pages, and S1 ran 4 KiB pages] → After the converge on the node that holds no VIP and one reload-yaml of each process, task 9.13 reads the RSS with ps -o rss= -C pdns_recursor. The sum must stay under 1,500 MiB.
  • [The feeds grow] → TIF medium grew by 80,000 records between two runs of S1 on consecutive days. The feeds of the second run use 77 percent of the 2.1 million records that 1,500 MiB holds. Task 8.4 decides whether the feed unit refuses a set of feeds above a record limit.
  • [A scope that is deleted and created again keeps its old entries from the dump file] → The worker deletes the dump file of the zone when it deletes the scope (Decision 4).
  • [DNS-collector holds back rows before a dropped row] → The reader drops nothing, and the Lua script keeps the health names out of the export (Decision 8).
  • [A wrong Lua script leaks a scoped answer through the packet cache] → S3 and the molecule scenario test both orders of the first query.
  • [reload-yaml reloads every RPZ of the process, also the feeds] → A new scope is seldom. S2 and S7 measure the cost.
  • [LightningStream or the NAS is down] → A write reaches only portland. The serial wait times out on sandiego and the change fails, as a record change does today. The next projection repairs it.
  • [The protobuf reader is one more daemon on each node] → It runs under systemd with no state. The recursor queues up to maxQueuedEntries messages while the reader restarts and answers queries without it.
  • [A rename of a ServiceName or an attribute breaks a ClickStack alert] → Pull request 2 changes the alert and the row filter before any name changes.
  • [The recursor caches owned names] → Decision 10.
  • [A failed first node no longer keeps a list change off the second node] → Decision 4. The step order still gives the node that holds no VIP the NOTIFY first.
  • [A cached answer outlives a new block entry] → S5 showed that 5.4 keeps the packet cache entry after a load of the RPZ. Only 5.5 clears it. The step wipes each changed name (Decisions 4 and 12).
  • [The API refuses the root name . for the flush] → The molecule scenario flushes . through the API. If the API refuses it, the flush runs rec_control wipe-cache .$, which has a fixed argument list.
  • [A wipe misses the packet cache partition of another zone set] → The molecule scenario caches one name for two zone sets, wipes it, and makes sure that a second wipe counts zero.

The pull requests follow the split of proposal.md. Standing constraint C3 holds for every converge: one node per window, and the node that holds no VIP first. scripts/dns-vip-holder.py names the holder.

  • Pull requests 2 and 7 are ArgoCD syncs. Rollback: revert the pull request.

  • Pull requests 3 and 6 are converges of one role. Rollback: revert the pull request and converge the same tag on each node, for example scripts/dns-playbook.sh --limit sandiego --tags dns-otel-collector.

  • Pull request 9 is the cut-over. On each node, the play stops and disables blocky and blocky-public and leaves their files in place. It then starts both recursor processes and converges keepalived, dns-otel-collector, dns-common and dns-node-worker. Before the second node, dns_drain_node moves the VIPs off it. After both nodes run for one day, the retire play blocky-retire.yml removes the binary, the units, the configuration and the operator files, through scripts/dns-playbook.sh --playbook=blocky-retire.yml. Rollback on a node before the retire play:

    Terminal window
    scripts/dns-adhoc.sh <node> -b -m ansible.builtin.shell -a 'systemctl disable --now pdns-recursor@main pdns-recursor@public && systemctl enable --now blocky blocky-public'

    After the retire play, the rollback is a revert of pull request 9 and a converge of --tags blocky on each node.

  • Pull request 10 is a release and two pins, with the rollback of each pin.

None. Spike 011 settled the version of the recursor (5.4.7) and the protobuf reader (DNS-collector 3.1.0).