Design: the pair drops Control D
Context
Section titled “Context”See openspec/changes/archive/2026-10-06-pair-drops-control-d/proposal.md for the reason, the two spikes and the decisions of 2026-10-05.
The parts of the estate that this design touches:
- Both Blocky instances on each node forward every public name to
tcp-tls:{{ dns_upstream_host }}:853and to the anycast literaldns_upstream_v4_addresswith the same TLS name. The two values live inansible/inventory/group_vars/dns_hosts.yml, andansible/roles/blocky/tasks/converge.ymlrefuses a render when either one is empty. - The role
blockyinstalls a release binary with the checksum manifest of the release, runs it as the userblockyunder a hardened unit, and runs named criteria against the node at the end of each converge. - The role
dns-otel-collectorreads the journals of the units indns_otel_journal_units. Every unit outsidedns_otel_blocky_log_service_namesships under the ServiceNameotelcol-dns. - The node ruleset of
dns-commonfilters input only and acceptsiif lo. There is no output rule to change. scripts/dns-break-external.shbreaks the external path of one node by dropping TCP port 853 in a table of its own.scripts/dns-external-resolution.pyproves that a node still answers names that need the WAN.- The ClickStack alert
DNS blocky-public failing (Guest/Telework)is a HyperDX tile alert on theCluster — metric alertsdashboard. An alert of this change is a second tile there. - Since 2026-10-05 the lists of Blocky block what Control D blocked, so the upstream does no blocking that a client needs.
This change adds no address and no prefix. The daemon listens on a loopback port, and
docs/reference/network.md stays as it is.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- Every public query of the pair leaves the house through a relay, and no resolver sees the home address.
- A route that stops working costs availability and never anonymity.
- The daemon is as pinned, as contained and as observable as Blocky.
Non-Goals:
- Anonymity from the relays. A relay sees the home address and the time of each query.
- A route for every leg. Both Blocky instances share one daemon and one route set.
- A query log of the daemon. Blocky logs each query with its client, and the daemon logs none.
- The end of the Control D account.
Decisions
Section titled “Decisions”1. A new role dnscrypt-proxy, built like blocky
Section titled “1. A new role dnscrypt-proxy, built like blocky”The role installs the release tarball of dnscrypt-proxy 2.1.18 for linux_arm64, runs it as the
system user dnscrypt-proxy under a hardened unit, and runs named criteria against the node at
the end of the converge. Its files mirror the role blocky: defaults, tasks/install.yml, tasks/config.yml,
tasks/verify.yml, templates/dnscrypt-proxy.toml.j2, templates/dnscrypt-proxy.service.j2 and
a handler that restarts the unit.
A Debian package was set aside. The estate pins Blocky from its release for the same reason: the version in the repository is the version on the node.
2. The release is verified with minisign, on the node
Section titled “2. The release is verified with minisign, on the node”dnscrypt-proxy publishes no checksum manifest. Each release asset carries a .minisig signature
from the release key RWTk1xXqcTODeYttYMCMLo0YJHaFEHn7a3akqHlb/7QvIQXHVPxKbjB5, which the page
Installation of the project wiki documents. The resolver lists carry signatures from a second
key, RWQf6LRCGA9i53mlYecO4IzT51TGPpvWucNSCh1CBM0QTaLn73Y7GFO3. The role holds both. It installs
the Debian package minisign, downloads the tarball and its signature, and runs
minisign -Vm <tarball> -P <release key> before it unpacks anything. A version bump then carries
its own proof, as the checksum manifest does for Blocky. A test on 2026-10-06 showed that the
list key rejects a release asset, so the two keys are not interchangeable.
A sha256 literal in a variable was set aside for the reason that roles/blocky/tasks/install.yml
states: it goes stale at the first bump, and a stale digest gets deleted rather than fixed.
3. One daemon for each node, shared by both Blocky instances and by the node itself
Section titled “3. One daemon for each node, shared by both Blocky instances and by the node itself”The daemon listens on two loopback addresses, UDP and TCP. 127.0.0.1:5300 is the only upstream
of both Blocky instances. 127.0.0.2:53 is the node’s own fallback resolver, the second
nameserver of /etc/resolv.conf after Blocky. Both addresses are inventory values,
dns_anonymizing_upstream_listen and dns_node_fallback_listen, and the roles that use them
read the same values. The daemon holds no internal name and sees no client address, so sharing
it leaks nothing from one client to another.
Two daemons, one for each instance, was set aside. It doubles the routes, the list downloads and the alerts for no gain in isolation.
The second listener was not in the first version of this design. roles/dns-common already ran
the Debian package dnscrypt-proxy 2.1.8 as the node’s fallback, socket-activated on
127.0.0.2:53, with DoH to two static stamps. The first pull request of this change reused the
same unit name and configuration path and shadowed it, found on 2026-10-06 during the proof of
the stopped daemon. Sean decided the same day that one daemon serves both. The role removes the
package, and roles/dns-common keeps only the resolver order. The cost is the double outage, in
the risks below. The gain is one binary at one version, one configuration, one set of alerts, and
anonymity for the node’s own queries, which reached Cloudflare and Quad9 directly before.
4. The route set
Section titled “4. The route set”The configuration names each resolver and each relay, with skip_incompatible = true and no
wildcard. Each resolver goes through relays of a different operator.
server_names = ['cs-dc', 'cs-nyc', 'dnscry.pt-ashburn-ipv4', 'dnscry.pt-newyork-ipv4', 'plan9dns-nj']
[anonymized_dns]routes = [ { server_name = 'cs-dc', via = ['dnscry.pt-anon-ashburn-ipv4', 'dnscry.pt-anon-newyork-ipv4'] }, { server_name = 'cs-nyc', via = ['dnscry.pt-anon-newyork-ipv4', 'dnscry.pt-anon-ashburn-ipv4'] }, { server_name = 'dnscry.pt-ashburn-ipv4', via = ['anon-cs-dc', 'anon-cs-nyc'] }, { server_name = 'dnscry.pt-newyork-ipv4', via = ['anon-cs-nyc', 'anon-cs-dc'] }, { server_name = 'plan9dns-nj', via = ['anon-cs-dc', 'dnscry.pt-anon-ashburn-ipv4'] },]skip_incompatible = truedirect_cert_fallback = falseThe role holds the set as two variables, dnscrypt_proxy_servers (a list) and
dnscrypt_proxy_routes (a map from resolver to relays), and the template renders both. A route
whose resolver is not in the server list fails the render, and so does an empty server list,
because dnscrypt-proxy treats an empty server_names as every resolver of the lists, reached
directly.
Five resolvers over two relay operators. The spike of 2026-10-05 measured each route at 5 to 17 ms median.
Sean decided on 2026-10-05 to add odoh-cloudflare through odohrelay-crypto-sx. The security
review of 2026-10-06 showed that dnscrypt-proxy fetches the ODoH configuration of a target from
the target itself, not through the relay, at each start and every cert_refresh_delay. The
target then learns the home address, which the requirement of this change does not allow. The
role ships without the route until Sean decides again. The route itself works from this network,
at 21 to 25 ms median, and odohrelay-numa added more than 100 ms.
direct_cert_fallback = false keeps the certificate fetch of a resolver behind the relay. The
review showed that the default fetches it directly when a probe decides that the relay drops
large packets.
5. The lists come signed from the public sources, and the role seeds the cache
Section titled “5. The lists come signed from the public sources, and the role seeds the cache”The daemon reads the four signed lists public-resolvers, relays, odoh-servers and
odoh-relays from download.dnscrypt.info, with refresh_delay = 72 hours and the minisign key
above. It makes sure that each list carries a valid signature before use. The cache lives in /var/cache/dnscrypt-proxy.
At each converge the role downloads the four lists and their .minisig files into the cache, so
a first start reads the lists from disk. The daemon still needs a few host names before any
route is live: the ODoH relay and the ODoH target carry a host name in their stamps, and the list
host at a refresh. bootstrap_resolvers names plain Quad9, 9.9.9.9:53 and 149.112.112.112:53,
for those names only. Blocky cannot play that role: after the swap its only upstream is this
daemon, and at a start the route that would answer does not exist yet. A test on 2026-10-06 with
Blocky as the bootstrap showed Bootstrap resolvers didn't respond. Quad9 sees those host names
and no other query. ignore_system_dns = true.
Fixed DNS stamps in the role were set aside. A stamp holds the public key and the address of a resolver, and both change over time. The signed list follows such a change, and a pinned stamp fails without a word.
6. The daemon caches nothing and logs no query
Section titled “6. The daemon caches nothing and logs no query”cache = false. Blocky caches every answer in front of the daemon, and a second cache gives a
second TTL to reason about. No query log is enabled, because Blocky already logs each query with
its client, and the daemon’s log would hold the same names a second time with no client.
log_level = 2 keeps the route lines, the certificate lines and the errors in the journal.
pqdnscrypt = false and dnscrypt_ephemeral_keys = true. The post-quantum mode of dnscrypt-proxy
reuses one encapsulation and chains resumption tickets that the resolver issues, so the resolver
can group every query of this node under one pseudonym. Classic DNSCrypt with a fresh client key
for each query gives each query its own identity. The trade is the post-quantum protection of
the transport, and the queries stay encrypted.
require_nolog = true. A named resolver that does not declare no-log is dropped, and the verify
criterion DCP-04 then fails on the live count.
7. The unit
Section titled “7. The unit”The unit runs /usr/local/bin/dnscrypt-proxy -config /etc/dnscrypt-proxy/dnscrypt-proxy.toml
as User=dnscrypt-proxy, with CacheDirectory=dnscrypt-proxy, ProtectSystem=strict,
NoNewPrivileges=true and no capability. Port 5300 needs none. The configuration is 0640,
owned root:dnscrypt-proxy, and the verify criteria read it as the daemon’s user. The role
asserts the mode at the write and reads the file back as the privilege-dropped user.
8. Blocky forwards to the daemon, and the Control D variables leave
Section titled “8. Blocky forwards to the daemon, and the Control D variables leave”Both Blocky templates render upstreams.groups.default: ["127.0.0.1:5300"]. The variables
dns_upstream_host, dns_upstream_v4_address, blocky_upstream_host, blocky_upstream_port and
blocky_upstream_v4_address leave, with the assertions in roles/blocky/tasks/converge.yml and
the drift-check comment in the inventory. A new assertion in the same place refuses a render when
the upstream is not a loopback address, so a fallback to a direct resolver cannot enter by an
edit.
Blocky keeps its upstream timeout. The spike measured a SERVFAIL after 6 seconds when the
daemon runs with no live route, and 2 ms when the daemon is stopped. The alert of decision 10
covers the first case, and a shorter timeout was set aside until a measurement shows a need.
9. The two scripts follow the new path
Section titled “9. The two scripts follow the new path”scripts/dns-break-external.sh stops the unit dnscrypt-proxy on the named node for apply,
starts it for revert, and prints systemctl is-active for status. The daemon is the only
thing on the node that speaks to the relays, so stopping it is the external break with no other
effect. The digest subcommand and its nftables table leave.
scripts/dns-external-resolution.py keeps its four probes. Its comments name the new path.
10. The journal goes to ClickStack, and one alert watches for a dead route
Section titled “10. The journal goes to ClickStack, and one alert watches for a dead route”dns_otel_journal_units gains dnscrypt-proxy.service, and a new map
dns_otel_daemon_log_service_names gives it the ServiceName dnscrypt-proxy, beside the Blocky
map. The rows hold no client address, so the row filter of clickstack-chart does not change.
Two HyperDX tile alerts read the rows of ServiceName = 'dnscrypt-proxy', grouped by
host.name, on a new dashboard DNS — resolver alerts. DNS dnscrypt-proxy has no live route
fires on a count of waiting for at least one server to be reachable above 0 in a five-minute
window. The daemon writes that line one time per outage, so the alert marks the start. DNS dnscrypt-proxy resolvers timing out fires on a count of resolver TIMEOUT lines above 60 in a
five-minute window, and stays in alert while the outage lasts. Five dead resolvers log about 140
lines in that window, and one dead resolver about 30.
The tiles live on their own dashboard. A tile of the Cluster — metric alerts dashboard carries a
field that the dashboard API rejects on a full save, so adding a tile there risks that tile.
11. The runbook, the ADR and the skills
Section titled “11. The runbook, the ADR and the skills”docs/operations/dns.md gets a section for the daemon: the path, the route set, the loopback
port, the lists, the alert, the break procedure and the proof commands of the spec. A new ADR,
docs/adr/2026-10-06-pair-resolves-public-names-through-anonymized-relays.md, records the
decision and marks hl-p5ck as superseded. The ADR of 2026-09-25 keeps its text, and the runbook
is the place that names the upstream. ansible/CLAUDE.md gets the row for the new role.
A copy of this design lands under
docs/engineering/specs/2026-10-06-pair-drops-control-d-design.md, because the change adds a
daemon to the path of every public name.
12. A security review before the role
Section titled “12. A security review before the role”The skill fable-security-review reviewed dnscrypt-proxy 2.1.18 on 2026-10-06. The verdict is
use with mitigations. The release tarball matched its minisign signature, every vendored module
matched the Go module proxy, and the binary holds no telemetry. The medium findings are the
direct fallback of Anonymized DNSCrypt and of the certificate fetch, unrelayed DoH servers in a
mixed set, the ODoH configuration fetch from the target, and the query linkage of the
post-quantum mode. Decisions 4 and 6 apply the mitigation for each one. The verdict and the
findings are in the description of pull request 1 and in the runbook section of the daemon.
13. No molecule scenario for the new role
Section titled “13. No molecule scenario for the new role”The role’s verify criteria run on the node at each converge, as for blocky. They make sure
that the unit is active, that the journal names a relay for each resolver and a live server count
equal to the server count, that a query for example.org to 127.0.0.1:5300 answers, and that the
configuration is readable as the daemon’s user. A molecule scenario was set aside: the behaviour
that matters is the route through the WAN, and a container on the workstation does not measure
it.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [Both relay operators fail at once] → Every public name answers
SERVFAILuntil a route returns. That is the chosen failure mode. The ODoH route to Cloudflare would have stayed up, and it is out for the reason in decision 4. - [A client waits 6 seconds for each
SERVFAILwhile the daemon runs with no live route] → The alert of decision 10 fires after five minutes. The operator stops the unit, which turns the 6-secondSERVFAILinto a 2-millisecond one, and repairs the route. - [A relay or a resolver leaves the public list] → The daemon logs the missing name at the next list refresh and keeps the other routes. The verify criteria of the next converge fail on the server count, and the operator edits the route set.
- [The gateway starts to drop UDP fragments after a firmware change] → Every route stops with
couldn't be reached anonymously, and the alert fires. The runbook names the log line. - [The CryptoStorm relays and resolvers are run by one VPN company] → Each route pairs its resolver with relays of the other operator, so one company never holds the address and the query of one request. Sean accepted this on 2026-10-05.
- [A list refresh fails for 72 hours or more] → The daemon keeps the cached lists and the certificates it holds. Each converge seeds the cache again.
- [The first start after a converge is slow] → The daemon probes each resolver through each
relay before it reports
live servers. The spike measured 2 to 25 seconds. Blocky answersSERVFAILfor public names in that window, and the converge takes the passive node first. - [The bootstrap resolvers do not answer at a start] → The DNSCrypt routes come up anyway, because their stamps carry addresses. The cached lists stay in force.
- [Blocky is down and every relay is down on one node at the same time] → The node resolves no public name until one of them returns. Internal names were never covered by the fallback. The separate fallback that covered this case showed the home address to Cloudflare and Quad9, and shared a unit name with this daemon.
- [A converge removes the Debian package while Blocky forwards to the daemon] → The package’s
removal stops the shared unit for the seconds until the role starts it again, and the start
probes the routes. Public names on that node answer
SERVFAILfor about half a minute. The converge takes the passive node first.
Migration Plan
Section titled “Migration Plan”Pull request 1 and 2 change nothing that a client sees. Pull request 3 is the one live change.
- Pull request 1: the security review, the role, the play in
ansible/dns-playbook.ymlbefore the Blocky play, the runbook section and theansible/CLAUDE.mdrow. One converge for each node with--tags dnscrypt-proxy. The proof is the verify criteria of the role and a query forexample.orgto127.0.0.1:5300on each node. The nodes have nodig, so the verify task sends the query withpython3. - Pull request 2: the collector unit list and the ServiceName map. One converge for each node
with
--tags dns-otel-collector. Then the HyperDX alert, saved through the ClickStack MCP, and its row on the dashboard. The proof is a row withServiceName = 'dnscrypt-proxy'indefault.otel_logsfrom each node. - Pull request 3: the Blocky upstream, the removal of the Control D variables, the two scripts,
the ADR and the runbook. Find the passive node with
scripts/dns-vip-holder.py, thenscripts/dns-playbook.sh --limit <passive node> --tags blocky, the proof commands of the spec against the Main address of that node, and the same for the other node.
The rollback of pull request 3 is a pull request that reverts its merge commit, and then one
converge for each node from main:
git revert <merge commit> # on a branch, for the pull requestscripts/dns-playbook.sh --limit <passive node> --tags blockyscripts/dns-playbook.sh --limit <other node> --tags blockyThe revert restores the Control D entries, and the Control D profile exists until the renewal on
2027-01-08. Pull requests 1 and 2 need no rollback: a daemon that nothing uses and a journal that
nobody reads do no harm, and systemctl disable --now dnscrypt-proxy on a node is the fast stop.
Open Questions
Section titled “Open Questions”- Whether
plan9dns-njstays in the set after a month of the alert and the route log. It is the slowest DNSCrypt route and the only one with a single maintainer. The route set is two variables, so the answer changes no spec and no task.