Skip to content

Design: dns-control is the only DNS writer

See openspec/changes/archive/2026-10-03-dns-control-sole-writer/proposal.md for the motivation. The pieces this design builds on already exist:

  • apps/dns-control stores curated host entries in the entries table on the CNPG main cluster. A change is a stored plan in change_plans, an apply, a ChangeWorkflow and one append-only row in change_audit. The unique index change_audit_one_pending lets one change run at a time. All three tables are keyed by the entry key and the hostname today.
  • The pdns writer client sends one call: a PATCH of RRsets with REPLACE or DELETE. It cannot create a zone or read or write zone metadata. The readonly transport of the reader client allows GET and HEAD only and stays as it is.
  • The lineage index attributes Terraform from the state, external-dns from its txt registry records, kea-d2 from DHCID, dns-aaaa-publish from the TSIG-ALLOW-DNSUPDATE grants in the Terraform state, and dns-control from its ledger. dns-control reads no state, so 155 RRsets read as orphans.
  • tf/dns-records declares 23 zones, 23 SOA records, 50 RRsets (apex NS, delegation NS, the per-leg resolver addresses and the reserved-label seeds) and 19 metadata entries. Its addresses come from local.dns_legs in inventory.tf, a copy of docs/reference/network.md, which stays the addressing authority. This change allocates no address or prefix.
  • external-dns v0.22.0 uses the pdns provider with a txt registry, policy: sync, the domain filter fzymgc.house and the exclusion lan.fzymgc.house. It holds the pdns API key.
  • A dns-control static key carries the reader role only. Service writes use a Keycloak client-credentials token.
  • The pdns API admits calls from the k8s node addresses only (webserver-allow-from).

Goals:

  • One model that a person can hold in their head: dns-control writes every static record, the fenced Kea D2 writes lease records under grants that dns-control writes, and nothing else writes pdns.
  • Every write path that remains has a break-glass path that does not depend on dns-control.

Non-Goals:

  • No change to Kea D2, LightningStream or the Pi serving plane, except the removal of dns-aaaa-publish (D-11).
  • No change to the curated entry model. Host entries keep their table and their projection.
  • No observed AAAA. The change observed-aaaa-every-leg adds it on top of these lanes.

D-1. Two new stores: zones and record sets

Section titled “D-1. Two new stores: zones and record sets”

A zones table holds one row per zone: the name, the kind, the SOA values (mname, rname, refresh, retry, expire, minimum, ttl), the NS set and its TTL, and a map of metadata kind to value list. A rrsets table holds one row per static record set: the zone, the name, the type, the TTL, the records and the lane.

A record set is the right shape for what moves. The node apex record (sandiego.fzymgc.house) has one address per leg, a delegation is an NS set, and a seed is a TXT. None of those is a host entry. The alternative was to stretch the entry with a list of addresses and a free type. That would change the meaning of an entry. Sean ruled on 2026-08-18 that a device’s reservation and its A record are one literal, and an entry stays that literal.

The SOA serial is not stored. Every zone keeps soa_edit_api = DEFAULT, so pdns raises the serial on each API write. The lineage index already skips the SOA serial when it compares.

change_plans and change_audit gain subject (entry, zone or rrset) and subject_key. Existing rows read entry. The key and hostname columns stay for entries. The one-change lock, the workflow, the export and the audit trigger are shared, so every subject is audited and exported the same way. A zone or record set change runs the same ChangeWorkflow with the reservation step skipped, because only an entry projects a reservation.

The alternative was a second workflow and a second audit table. Two audit tables would make “what changed at 14:02” a join, and the lock would no longer cover every write.

D-3. New pdns writer calls, each a full replace of what it names

Section titled “D-3. New pdns writer calls, each a full replace of what it names”

The writer gains three calls, all on the writer node, which LightningStream copies to the other:

Call Wire shape What it replaces
Create a zone POST /api/v1/servers/localhost/zones with name, kind, soa_edit_api and the SOA and NS RRsets Nothing. The call fails if the zone exists
Delete a zone DELETE /api/v1/servers/localhost/zones/{zone} The whole zone
Write a metadata kind PUT /api/v1/servers/localhost/zones/{zone}/metadata/{kind} with {"metadata": [...]} Every value of that one kind

The metadata PUT is a full replace of one kind. The store models the whole value list of each kind it manages, so a write never drops a value. dns-control manages TSIG-ALLOW-DNSUPDATE only. The import records every other kind that pdns holds, and dns-control never writes those.

SOA and NS changes use the existing RRset PATCH.

A record set has a lane: control or external-dns. The write policy decides who can write a name:

Writer May write
A dns-writer caller Curated entries, as today
A dns-admin caller Zones, metadata and control record sets
The external-dns webhook external-dns record sets only

A name belongs to one lane. A plan that names a record set held by another lane, or by an entry, is refused with policy_other_lane. The dynamic zones stay closed to every lane except the control seeds, as the policy_dynamic_zone rule does today.

D-5. The import reads live pdns, not the Terraform state

Section titled “D-5. The import reads live pdns, not the Terraform state”

dns-control zones import reads every zone, its metadata and its RRsets from the pdns API, and the lineage index of the same moment. It sorts each RRset by its current writer:

Current writer Goes to
none (the Terraform RRsets) and SOA or apex NS the zone row
none, any other type a control record set
external-dns, including its registry TXT records an external-dns record set
dns-control (curated entries), kea-d2, dns-aaaa-publish nothing, the import skips it

The import writes the rows and one finalised audit row per subject with the outcome imported, the same as entries import. No workflow runs and pdns does not change. --dry-run prints what it would store.

Live pdns is what the pair serves, both nodes agree, and the read needs no HCP credential. The Terraform state would need the token that Sean found cannot be scoped. The proof pull request compares the import counts with terraform state list run by the operator.

The subcommand dns-control webhook runs as a sidecar in the external-dns pod, from the dns-control image. The webhook container of the external-dns chart sets no command, so the sidecar passes webhook as its argument. The record API listens on 127.0.0.1:8888 and serves the four webhook routes. A second listener answers GET /healthz on :8080 for the probes of the chart.

  • GET / answers the domain filter fzymgc.house with the exclusion lan.fzymgc.house.
  • GET /records answers the record sets of the external-dns lane.
  • POST /records turns the batch into one plan and one apply on dns-control and waits for the workflow. A busy lock answers 503, and external-dns tries again.
  • POST /adjustendpoints answers the endpoints unchanged.

A batch is one change of the subject batch: a list of record sets, each with its row before and after. One workflow writes them all. The webhook client of external-dns waits 15 s by default and exits on a timeout, so the sidecar answers within 12 s. dns-control waits up to 10 s for the workflow and then answers pending, the sidecar answers 503, and external-dns tries again. The retry finds the records in the store and needs no write. The start of the workflow does not follow the cancellation of the request, so a caller that hangs up cannot leave a pending change with no workflow.

The pdns step writes each record set with its own PATCHes, the deletes of a zone first, so a record that pdns refuses fails alone. When a batch does not succeed, its finalize puts each record set that pdns does not serve back to its before side in the same transaction. The store then matches pdns, and the next sync of external-dns sends those records again. A registry TXT that claims a name that another lane or an entry holds is refused with that name. The audit row lists every name of the batch, so a lookup of the changes by name finds a record inside a batch. dns-control places each record in the longest zone of the store that holds its name. A record that the lane policy refuses leaves the batch, and dns-control logs it, so one Ingress that claims a curated name does not hold back the lane. The plan derives create, change or delete from the store, not from the old side that external-dns sends, so a batch that external-dns sends again after a timeout changes nothing.

The sidecar presents each record set the way the pdns provider of external-dns did, so the first sync after the cutover is empty. The name has no trailing dot. A target loses its trailing dot, except for TXT, NAPTR and SRV, so a registry TXT keeps its quotes. A disabled record is left out, and ALIAS reads as CNAME. On a write, the sidecar adds the trailing dot back to the types of the provider, writes a CNAME at a zone apex as ALIAS, and writes a TTL of 0 as 300.

The sidecar authenticates to dns-control with a Keycloak client-credentials token for a new client dns-control-external-dns that holds the role dns-lane-external-dns. Terraform writes the client secret to the Terraform-owned subpath fzymgc-house/cluster/external-dns/oidc, and an ExternalSecret supplies it. The allow-list of the ESO policy on cluster/* already covers that path. dns-control serves the lane on a new route group /api/lanes/external-dns/ that accepts that bearer token. The session-only opsRoute stays as it is.

The txt registry stays. Its TXT records are ordinary record sets of the lane, so the cutover changes no record and external-dns keeps its ownership logic. The view of one lane is the real fence. The registry is now redundant, and dropping it later is one values change with no risk to other lanes.

D-7. Lineage reads only dns-control’s stores

Section titled “D-7. Lineage reads only dns-control’s stores”

The index builds every claim from the stores: the entry ledger, the zones (SOA and NS), the control and external-dns record sets. kea-d2 stays attributed from DHCID. The dns-aaaa-publish writer leaves with D-11. An external-dns lane record set shows the writer external-dns with the mark of dns-control. internal/tfstate, HCP_TERRAFORM_TOKEN and the “Terraform state not read” alert leave in the teardown pull request.

The export writes zones.json beside entries.json: the version, every zone row and every record set row, sorted, with no change id or timestamp, so the same state gives the same bytes. The Kea seed keeps reading entries.json only.

dns-control restore --from <bucket> reads both objects, plans every zone first, then every record set, then every entry, and applies them one at a time. --dry-run prints no change, create or change per subject.

tf/keycloak gains the client role dns-admin on the dns-control client and the client dns-control-external-dns with the role dns-lane-external-dns. Sean holds dns-admin.

Terraform writes the client secret of dns-control-external-dns to fzymgc-house/cluster/external-dns/oidc, never to the bare cluster/external-dns path. The paired grant on that subpath in tf/vault/policy-keycloak.tf lands in its own pull request and applies first, because the Keycloak write answers 403 without it. Sean decided this on 2026-10-02.

The rotation keeps its widen and narrow pull requests. The Ansible half stays. The Terraform half, var.tsig_key_generations_granted, becomes a dns-control change to the TSIG-ALLOW-DNSUPDATE value of each granted zone. dns-control zones grant --generations <n-1>,<n> plans that change for every dynamic zone at once. The grants name the Kea D2 keys only, once D-11 removes the publisher’s key.

The publisher reads the resolver’s own IPv6 neighbour table. On 2026-10-01 that table held 0 to 3 in-scope clients per leg, and Lab, the one leg the publisher covers, held 0 AAAA records. The UniFi client table held a ULA for 27 clients on the same day. The change observed-aaaa-every-leg moves the work into dns-control, so the publisher has no job left.

The teardown runs in this order. A dns-control change drops every generation of the publisher’s key from the TSIG-ALLOW-DNSUPDATE grant of each dynamic zone. A retire playbook, following ansible/unbound-retire.yml, stops and removes the timer, the service, the key file and the publisher’s distribution in the shared resolver venv on each Pi, the passive node first. The venv stays, because dns-tailscale also runs from it. The same pull request deletes the role, the tool, the Vault policy and every reference. The reserved-label list in dns_hosts.yml has no reader after that, so it goes too, and the seed record sets in dns-control are the only copy.

  • [The import misreads a writer and moves a record into the wrong lane] → the proof pull request compares the dry run with terraform state list and with the external-dns registry count before the handoff stores anything.
  • [external-dns waits on a busy lock and a new Ingress name is late] → the wait is one interval (30 s) per busy lock, and a change runs in seconds.
  • [dns-control or its database is down and a name must change now] → break glass: write the pdns API directly from a k8s node with the key at fzymgc-house/infrastructure/dns/pdns-api, then adopt the change once dns-control is back.
  • [The export is lost along with the database] → both buckets hold the export, and the B2 copy is off site. pdns itself also keeps serving, and an import from live pdns rebuilds the stores.
  • [AAAA records stop for the time between the teardown and observed-aaaa-every-leg] → the publisher writes no record today, so nothing changes for a client.

Each pull request in the proposal’s split lands in order. Rollback per live step:

  1. The handoff (the removed blocks, the import, the project run). The marks are the only pdns change, and the records keep their content. Roll back the store with dns-control zones delete --imported and revert the pull request. To give Terraform the resources back, add import {} blocks generated by dns-control zones export --format tf-import.
  2. The external-dns cutover. Revert the values change. external-dns returns to the pdns provider, because the pdns key is removed only in the teardown pull request.
  3. The teardown. Revert the pull request. The workspace and the Vault role return through Terraform, and the removed blocks keep the workspace from touching pdns.
  • Resolved: the first pull request ships the API, the CLI and the MCP tools, and no zone editor in the UI. The Changes screen lists zone and record set changes by name.