Skip to content

Design: the pair drops Control D

See openspec/changes/archive/2026-10-06-pair-drops-control-d/proposal.md for the reason, the two spikes and the decisions of 2026-10-05.

The parts of the estate that this design touches:

  • Both Blocky instances on each node forward every public name to tcp-tls:{{ dns_upstream_host }}:853 and to the anycast literal dns_upstream_v4_address with the same TLS name. The two values live in ansible/inventory/group_vars/dns_hosts.yml, and ansible/roles/blocky/tasks/converge.yml refuses a render when either one is empty.
  • The role blocky installs a release binary with the checksum manifest of the release, runs it as the user blocky under a hardened unit, and runs named criteria against the node at the end of each converge.
  • The role dns-otel-collector reads the journals of the units in dns_otel_journal_units. Every unit outside dns_otel_blocky_log_service_names ships under the ServiceName otelcol-dns.
  • The node ruleset of dns-common filters input only and accepts iif lo. There is no output rule to change.
  • scripts/dns-break-external.sh breaks the external path of one node by dropping TCP port 853 in a table of its own. scripts/dns-external-resolution.py proves that a node still answers names that need the WAN.
  • The ClickStack alert DNS blocky-public failing (Guest/Telework) is a HyperDX tile alert on the Cluster — metric alerts dashboard. An alert of this change is a second tile there.
  • Since 2026-10-05 the lists of Blocky block what Control D blocked, so the upstream does no blocking that a client needs.

This change adds no address and no prefix. The daemon listens on a loopback port, and docs/reference/network.md stays as it is.

Goals:

  • Every public query of the pair leaves the house through a relay, and no resolver sees the home address.
  • A route that stops working costs availability and never anonymity.
  • The daemon is as pinned, as contained and as observable as Blocky.

Non-Goals:

  • Anonymity from the relays. A relay sees the home address and the time of each query.
  • A route for every leg. Both Blocky instances share one daemon and one route set.
  • A query log of the daemon. Blocky logs each query with its client, and the daemon logs none.
  • The end of the Control D account.

1. A new role dnscrypt-proxy, built like blocky

Section titled “1. A new role dnscrypt-proxy, built like blocky”

The role installs the release tarball of dnscrypt-proxy 2.1.18 for linux_arm64, runs it as the system user dnscrypt-proxy under a hardened unit, and runs named criteria against the node at the end of the converge. Its files mirror the role blocky: defaults, tasks/install.yml, tasks/config.yml, tasks/verify.yml, templates/dnscrypt-proxy.toml.j2, templates/dnscrypt-proxy.service.j2 and a handler that restarts the unit.

A Debian package was set aside. The estate pins Blocky from its release for the same reason: the version in the repository is the version on the node.

2. The release is verified with minisign, on the node

Section titled “2. The release is verified with minisign, on the node”

dnscrypt-proxy publishes no checksum manifest. Each release asset carries a .minisig signature from the release key RWTk1xXqcTODeYttYMCMLo0YJHaFEHn7a3akqHlb/7QvIQXHVPxKbjB5, which the page Installation of the project wiki documents. The resolver lists carry signatures from a second key, RWQf6LRCGA9i53mlYecO4IzT51TGPpvWucNSCh1CBM0QTaLn73Y7GFO3. The role holds both. It installs the Debian package minisign, downloads the tarball and its signature, and runs minisign -Vm <tarball> -P <release key> before it unpacks anything. A version bump then carries its own proof, as the checksum manifest does for Blocky. A test on 2026-10-06 showed that the list key rejects a release asset, so the two keys are not interchangeable.

A sha256 literal in a variable was set aside for the reason that roles/blocky/tasks/install.yml states: it goes stale at the first bump, and a stale digest gets deleted rather than fixed.

3. One daemon for each node, shared by both Blocky instances and by the node itself

Section titled “3. One daemon for each node, shared by both Blocky instances and by the node itself”

The daemon listens on two loopback addresses, UDP and TCP. 127.0.0.1:5300 is the only upstream of both Blocky instances. 127.0.0.2:53 is the node’s own fallback resolver, the second nameserver of /etc/resolv.conf after Blocky. Both addresses are inventory values, dns_anonymizing_upstream_listen and dns_node_fallback_listen, and the roles that use them read the same values. The daemon holds no internal name and sees no client address, so sharing it leaks nothing from one client to another.

Two daemons, one for each instance, was set aside. It doubles the routes, the list downloads and the alerts for no gain in isolation.

The second listener was not in the first version of this design. roles/dns-common already ran the Debian package dnscrypt-proxy 2.1.8 as the node’s fallback, socket-activated on 127.0.0.2:53, with DoH to two static stamps. The first pull request of this change reused the same unit name and configuration path and shadowed it, found on 2026-10-06 during the proof of the stopped daemon. Sean decided the same day that one daemon serves both. The role removes the package, and roles/dns-common keeps only the resolver order. The cost is the double outage, in the risks below. The gain is one binary at one version, one configuration, one set of alerts, and anonymity for the node’s own queries, which reached Cloudflare and Quad9 directly before.

The configuration names each resolver and each relay, with skip_incompatible = true and no wildcard. Each resolver goes through relays of a different operator.

server_names = ['cs-dc', 'cs-nyc', 'dnscry.pt-ashburn-ipv4', 'dnscry.pt-newyork-ipv4', 'plan9dns-nj']
[anonymized_dns]
routes = [
{ server_name = 'cs-dc', via = ['dnscry.pt-anon-ashburn-ipv4', 'dnscry.pt-anon-newyork-ipv4'] },
{ server_name = 'cs-nyc', via = ['dnscry.pt-anon-newyork-ipv4', 'dnscry.pt-anon-ashburn-ipv4'] },
{ server_name = 'dnscry.pt-ashburn-ipv4', via = ['anon-cs-dc', 'anon-cs-nyc'] },
{ server_name = 'dnscry.pt-newyork-ipv4', via = ['anon-cs-nyc', 'anon-cs-dc'] },
{ server_name = 'plan9dns-nj', via = ['anon-cs-dc', 'dnscry.pt-anon-ashburn-ipv4'] },
]
skip_incompatible = true
direct_cert_fallback = false

The role holds the set as two variables, dnscrypt_proxy_servers (a list) and dnscrypt_proxy_routes (a map from resolver to relays), and the template renders both. A route whose resolver is not in the server list fails the render, and so does an empty server list, because dnscrypt-proxy treats an empty server_names as every resolver of the lists, reached directly.

Five resolvers over two relay operators. The spike of 2026-10-05 measured each route at 5 to 17 ms median.

Sean decided on 2026-10-05 to add odoh-cloudflare through odohrelay-crypto-sx. The security review of 2026-10-06 showed that dnscrypt-proxy fetches the ODoH configuration of a target from the target itself, not through the relay, at each start and every cert_refresh_delay. The target then learns the home address, which the requirement of this change does not allow. The role ships without the route until Sean decides again. The route itself works from this network, at 21 to 25 ms median, and odohrelay-numa added more than 100 ms.

direct_cert_fallback = false keeps the certificate fetch of a resolver behind the relay. The review showed that the default fetches it directly when a probe decides that the relay drops large packets.

5. The lists come signed from the public sources, and the role seeds the cache

Section titled “5. The lists come signed from the public sources, and the role seeds the cache”

The daemon reads the four signed lists public-resolvers, relays, odoh-servers and odoh-relays from download.dnscrypt.info, with refresh_delay = 72 hours and the minisign key above. It makes sure that each list carries a valid signature before use. The cache lives in /var/cache/dnscrypt-proxy.

At each converge the role downloads the four lists and their .minisig files into the cache, so a first start reads the lists from disk. The daemon still needs a few host names before any route is live: the ODoH relay and the ODoH target carry a host name in their stamps, and the list host at a refresh. bootstrap_resolvers names plain Quad9, 9.9.9.9:53 and 149.112.112.112:53, for those names only. Blocky cannot play that role: after the swap its only upstream is this daemon, and at a start the route that would answer does not exist yet. A test on 2026-10-06 with Blocky as the bootstrap showed Bootstrap resolvers didn't respond. Quad9 sees those host names and no other query. ignore_system_dns = true.

Fixed DNS stamps in the role were set aside. A stamp holds the public key and the address of a resolver, and both change over time. The signed list follows such a change, and a pinned stamp fails without a word.

6. The daemon caches nothing and logs no query

Section titled “6. The daemon caches nothing and logs no query”

cache = false. Blocky caches every answer in front of the daemon, and a second cache gives a second TTL to reason about. No query log is enabled, because Blocky already logs each query with its client, and the daemon’s log would hold the same names a second time with no client.

log_level = 2 keeps the route lines, the certificate lines and the errors in the journal.

pqdnscrypt = false and dnscrypt_ephemeral_keys = true. The post-quantum mode of dnscrypt-proxy reuses one encapsulation and chains resumption tickets that the resolver issues, so the resolver can group every query of this node under one pseudonym. Classic DNSCrypt with a fresh client key for each query gives each query its own identity. The trade is the post-quantum protection of the transport, and the queries stay encrypted.

require_nolog = true. A named resolver that does not declare no-log is dropped, and the verify criterion DCP-04 then fails on the live count.

The unit runs /usr/local/bin/dnscrypt-proxy -config /etc/dnscrypt-proxy/dnscrypt-proxy.toml as User=dnscrypt-proxy, with CacheDirectory=dnscrypt-proxy, ProtectSystem=strict, NoNewPrivileges=true and no capability. Port 5300 needs none. The configuration is 0640, owned root:dnscrypt-proxy, and the verify criteria read it as the daemon’s user. The role asserts the mode at the write and reads the file back as the privilege-dropped user.

8. Blocky forwards to the daemon, and the Control D variables leave

Section titled “8. Blocky forwards to the daemon, and the Control D variables leave”

Both Blocky templates render upstreams.groups.default: ["127.0.0.1:5300"]. The variables dns_upstream_host, dns_upstream_v4_address, blocky_upstream_host, blocky_upstream_port and blocky_upstream_v4_address leave, with the assertions in roles/blocky/tasks/converge.yml and the drift-check comment in the inventory. A new assertion in the same place refuses a render when the upstream is not a loopback address, so a fallback to a direct resolver cannot enter by an edit.

Blocky keeps its upstream timeout. The spike measured a SERVFAIL after 6 seconds when the daemon runs with no live route, and 2 ms when the daemon is stopped. The alert of decision 10 covers the first case, and a shorter timeout was set aside until a measurement shows a need.

scripts/dns-break-external.sh stops the unit dnscrypt-proxy on the named node for apply, starts it for revert, and prints systemctl is-active for status. The daemon is the only thing on the node that speaks to the relays, so stopping it is the external break with no other effect. The digest subcommand and its nftables table leave.

scripts/dns-external-resolution.py keeps its four probes. Its comments name the new path.

10. The journal goes to ClickStack, and one alert watches for a dead route

Section titled “10. The journal goes to ClickStack, and one alert watches for a dead route”

dns_otel_journal_units gains dnscrypt-proxy.service, and a new map dns_otel_daemon_log_service_names gives it the ServiceName dnscrypt-proxy, beside the Blocky map. The rows hold no client address, so the row filter of clickstack-chart does not change.

Two HyperDX tile alerts read the rows of ServiceName = 'dnscrypt-proxy', grouped by host.name, on a new dashboard DNS — resolver alerts. DNS dnscrypt-proxy has no live route fires on a count of waiting for at least one server to be reachable above 0 in a five-minute window. The daemon writes that line one time per outage, so the alert marks the start. DNS dnscrypt-proxy resolvers timing out fires on a count of resolver TIMEOUT lines above 60 in a five-minute window, and stays in alert while the outage lasts. Five dead resolvers log about 140 lines in that window, and one dead resolver about 30.

The tiles live on their own dashboard. A tile of the Cluster — metric alerts dashboard carries a field that the dashboard API rejects on a full save, so adding a tile there risks that tile.

docs/operations/dns.md gets a section for the daemon: the path, the route set, the loopback port, the lists, the alert, the break procedure and the proof commands of the spec. A new ADR, docs/adr/2026-10-06-pair-resolves-public-names-through-anonymized-relays.md, records the decision and marks hl-p5ck as superseded. The ADR of 2026-09-25 keeps its text, and the runbook is the place that names the upstream. ansible/CLAUDE.md gets the row for the new role.

A copy of this design lands under docs/engineering/specs/2026-10-06-pair-drops-control-d-design.md, because the change adds a daemon to the path of every public name.

The skill fable-security-review reviewed dnscrypt-proxy 2.1.18 on 2026-10-06. The verdict is use with mitigations. The release tarball matched its minisign signature, every vendored module matched the Go module proxy, and the binary holds no telemetry. The medium findings are the direct fallback of Anonymized DNSCrypt and of the certificate fetch, unrelayed DoH servers in a mixed set, the ODoH configuration fetch from the target, and the query linkage of the post-quantum mode. Decisions 4 and 6 apply the mitigation for each one. The verdict and the findings are in the description of pull request 1 and in the runbook section of the daemon.

The role’s verify criteria run on the node at each converge, as for blocky. They make sure that the unit is active, that the journal names a relay for each resolver and a live server count equal to the server count, that a query for example.org to 127.0.0.1:5300 answers, and that the configuration is readable as the daemon’s user. A molecule scenario was set aside: the behaviour that matters is the route through the WAN, and a container on the workstation does not measure it.

  • [Both relay operators fail at once] → Every public name answers SERVFAIL until a route returns. That is the chosen failure mode. The ODoH route to Cloudflare would have stayed up, and it is out for the reason in decision 4.
  • [A client waits 6 seconds for each SERVFAIL while the daemon runs with no live route] → The alert of decision 10 fires after five minutes. The operator stops the unit, which turns the 6-second SERVFAIL into a 2-millisecond one, and repairs the route.
  • [A relay or a resolver leaves the public list] → The daemon logs the missing name at the next list refresh and keeps the other routes. The verify criteria of the next converge fail on the server count, and the operator edits the route set.
  • [The gateway starts to drop UDP fragments after a firmware change] → Every route stops with couldn't be reached anonymously, and the alert fires. The runbook names the log line.
  • [The CryptoStorm relays and resolvers are run by one VPN company] → Each route pairs its resolver with relays of the other operator, so one company never holds the address and the query of one request. Sean accepted this on 2026-10-05.
  • [A list refresh fails for 72 hours or more] → The daemon keeps the cached lists and the certificates it holds. Each converge seeds the cache again.
  • [The first start after a converge is slow] → The daemon probes each resolver through each relay before it reports live servers. The spike measured 2 to 25 seconds. Blocky answers SERVFAIL for public names in that window, and the converge takes the passive node first.
  • [The bootstrap resolvers do not answer at a start] → The DNSCrypt routes come up anyway, because their stamps carry addresses. The cached lists stay in force.
  • [Blocky is down and every relay is down on one node at the same time] → The node resolves no public name until one of them returns. Internal names were never covered by the fallback. The separate fallback that covered this case showed the home address to Cloudflare and Quad9, and shared a unit name with this daemon.
  • [A converge removes the Debian package while Blocky forwards to the daemon] → The package’s removal stops the shared unit for the seconds until the role starts it again, and the start probes the routes. Public names on that node answer SERVFAIL for about half a minute. The converge takes the passive node first.

Pull request 1 and 2 change nothing that a client sees. Pull request 3 is the one live change.

  1. Pull request 1: the security review, the role, the play in ansible/dns-playbook.yml before the Blocky play, the runbook section and the ansible/CLAUDE.md row. One converge for each node with --tags dnscrypt-proxy. The proof is the verify criteria of the role and a query for example.org to 127.0.0.1:5300 on each node. The nodes have no dig, so the verify task sends the query with python3.
  2. Pull request 2: the collector unit list and the ServiceName map. One converge for each node with --tags dns-otel-collector. Then the HyperDX alert, saved through the ClickStack MCP, and its row on the dashboard. The proof is a row with ServiceName = 'dnscrypt-proxy' in default.otel_logs from each node.
  3. Pull request 3: the Blocky upstream, the removal of the Control D variables, the two scripts, the ADR and the runbook. Find the passive node with scripts/dns-vip-holder.py, then scripts/dns-playbook.sh --limit <passive node> --tags blocky, the proof commands of the spec against the Main address of that node, and the same for the other node.

The rollback of pull request 3 is a pull request that reverts its merge commit, and then one converge for each node from main:

Terminal window
git revert <merge commit> # on a branch, for the pull request
scripts/dns-playbook.sh --limit <passive node> --tags blocky
scripts/dns-playbook.sh --limit <other node> --tags blocky

The revert restores the Control D entries, and the Control D profile exists until the renewal on 2027-01-08. Pull requests 1 and 2 need no rollback: a daemon that nothing uses and a journal that nobody reads do no harm, and systemctl disable --now dnscrypt-proxy on a node is the fast stop.

  • Whether plan9dns-nj stays in the set after a month of the alert and the route log. It is the slowest DNSCrypt route and the only one with a single maintainer. The route set is two variables, so the answer changes no spec and no task.