Design: dns-control is the only DNS writer
Context
Section titled “Context”See openspec/changes/archive/2026-10-03-dns-control-sole-writer/proposal.md for the motivation. The pieces this
design builds on already exist:
apps/dns-controlstores curated host entries in theentriestable on the CNPGmaincluster. A change is a stored plan inchange_plans, an apply, aChangeWorkflowand one append-only row inchange_audit. The unique indexchange_audit_one_pendinglets one change run at a time. All three tables are keyed by the entry key and the hostname today.- The pdns writer client sends one call: a PATCH of RRsets with
REPLACEorDELETE. It cannot create a zone or read or write zone metadata. The readonly transport of the reader client allows GET and HEAD only and stays as it is. - The lineage index attributes Terraform from the state, external-dns from its
txtregistry records, kea-d2 from DHCID,dns-aaaa-publishfrom theTSIG-ALLOW-DNSUPDATEgrants in the Terraform state, and dns-control from its ledger. dns-control reads no state, so 155 RRsets read as orphans. tf/dns-recordsdeclares 23 zones, 23 SOA records, 50 RRsets (apex NS, delegation NS, the per-leg resolver addresses and the reserved-label seeds) and 19 metadata entries. Its addresses come fromlocal.dns_legsininventory.tf, a copy ofdocs/reference/network.md, which stays the addressing authority. This change allocates no address or prefix.- external-dns v0.22.0 uses the pdns provider with a
txtregistry,policy: sync, the domain filterfzymgc.houseand the exclusionlan.fzymgc.house. It holds the pdns API key. - A dns-control static key carries the reader role only. Service writes use a Keycloak client-credentials token.
- The pdns API admits calls from the k8s node addresses only (
webserver-allow-from).
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- One model that a person can hold in their head: dns-control writes every static record, the fenced Kea D2 writes lease records under grants that dns-control writes, and nothing else writes pdns.
- Every write path that remains has a break-glass path that does not depend on dns-control.
Non-Goals:
- No change to Kea D2, LightningStream or the Pi serving plane, except the removal of
dns-aaaa-publish(D-11). - No change to the curated entry model. Host entries keep their table and their projection.
- No observed AAAA. The change
observed-aaaa-every-legadds it on top of these lanes.
Decisions
Section titled “Decisions”D-1. Two new stores: zones and record sets
Section titled “D-1. Two new stores: zones and record sets”A zones table holds one row per zone: the name, the kind, the SOA values (mname, rname,
refresh, retry, expire, minimum, ttl), the NS set and its TTL, and a map of metadata
kind to value list. A rrsets table holds one row per static record set: the zone, the name, the
type, the TTL, the records and the lane.
A record set is the right shape for what moves. The node apex record (sandiego.fzymgc.house)
has one address per leg, a delegation is an NS set, and a seed is a TXT. None of those is a host
entry. The alternative was to stretch the entry with a list of addresses and a free type. That
would change the meaning of an entry. Sean ruled on 2026-08-18 that a device’s reservation and
its A record are one literal, and an entry stays that literal.
The SOA serial is not stored. Every zone keeps soa_edit_api = DEFAULT, so pdns raises the
serial on each API write. The lineage index already skips the SOA serial when it compares.
D-2. One change model with a subject
Section titled “D-2. One change model with a subject”change_plans and change_audit gain subject (entry, zone or rrset) and subject_key.
Existing rows read entry. The key and hostname columns stay for entries. The one-change lock,
the workflow, the export and the audit trigger are shared, so every subject is audited and
exported the same way. A zone or record set change runs the same ChangeWorkflow with the
reservation step skipped, because only an entry projects a reservation.
The alternative was a second workflow and a second audit table. Two audit tables would make “what changed at 14:02” a join, and the lock would no longer cover every write.
D-3. New pdns writer calls, each a full replace of what it names
Section titled “D-3. New pdns writer calls, each a full replace of what it names”The writer gains three calls, all on the writer node, which LightningStream copies to the other:
| Call | Wire shape | What it replaces |
|---|---|---|
| Create a zone | POST /api/v1/servers/localhost/zones with name, kind, soa_edit_api and the SOA and NS RRsets |
Nothing. The call fails if the zone exists |
| Delete a zone | DELETE /api/v1/servers/localhost/zones/{zone} |
The whole zone |
| Write a metadata kind | PUT /api/v1/servers/localhost/zones/{zone}/metadata/{kind} with {"metadata": [...]} |
Every value of that one kind |
The metadata PUT is a full replace of one kind. The store models the whole value list of each
kind it manages, so a write never drops a value. dns-control manages TSIG-ALLOW-DNSUPDATE only.
The import records every other kind that pdns holds, and dns-control never writes those.
SOA and NS changes use the existing RRset PATCH.
D-4. Lanes
Section titled “D-4. Lanes”A record set has a lane: control or external-dns. The write policy decides who can write a
name:
| Writer | May write |
|---|---|
A dns-writer caller |
Curated entries, as today |
A dns-admin caller |
Zones, metadata and control record sets |
| The external-dns webhook | external-dns record sets only |
A name belongs to one lane. A plan that names a record set held by another lane, or by an entry,
is refused with policy_other_lane. The dynamic zones stay closed to every lane except the
control seeds, as the policy_dynamic_zone rule does today.
D-5. The import reads live pdns, not the Terraform state
Section titled “D-5. The import reads live pdns, not the Terraform state”dns-control zones import reads every zone, its metadata and its RRsets from the pdns API, and
the lineage index of the same moment. It sorts each RRset by its current writer:
| Current writer | Goes to |
|---|---|
| none (the Terraform RRsets) and SOA or apex NS | the zone row |
| none, any other type | a control record set |
| external-dns, including its registry TXT records | an external-dns record set |
dns-control (curated entries), kea-d2, dns-aaaa-publish |
nothing, the import skips it |
The import writes the rows and one finalised audit row per subject with the outcome imported,
the same as entries import. No workflow runs and pdns does not change. --dry-run prints what
it would store.
Live pdns is what the pair serves, both nodes agree, and the read needs no HCP credential. The
Terraform state would need the token that Sean found cannot be scoped. The proof pull request
compares the import counts with terraform state list run by the operator.
D-6. The external-dns webhook sidecar
Section titled “D-6. The external-dns webhook sidecar”The subcommand dns-control webhook runs as a sidecar in the external-dns pod, from the
dns-control image. The webhook container of the external-dns chart sets no command, so the sidecar
passes webhook as its argument. The record API listens on 127.0.0.1:8888 and serves the four
webhook routes. A second listener answers GET /healthz on :8080 for the probes of the chart.
GET /answers the domain filterfzymgc.housewith the exclusionlan.fzymgc.house.GET /recordsanswers the record sets of theexternal-dnslane.POST /recordsturns the batch into one plan and one apply on dns-control and waits for the workflow. A busy lock answers 503, and external-dns tries again.POST /adjustendpointsanswers the endpoints unchanged.
A batch is one change of the subject batch: a list of record sets, each with its row before and
after. One workflow writes them all. The webhook client of external-dns waits 15 s by default and
exits on a timeout, so the sidecar answers within 12 s. dns-control waits up to 10 s for the
workflow and then answers pending, the sidecar answers 503, and external-dns tries again. The
retry finds the records in the store and needs no write. The start of the workflow does not follow
the cancellation of the request, so a caller that hangs up cannot leave a pending change with no
workflow.
The pdns step writes each record set with its own PATCHes, the deletes of a zone first, so a record that pdns refuses fails alone. When a batch does not succeed, its finalize puts each record set that pdns does not serve back to its before side in the same transaction. The store then matches pdns, and the next sync of external-dns sends those records again. A registry TXT that claims a name that another lane or an entry holds is refused with that name. The audit row lists every name of the batch, so a lookup of the changes by name finds a record inside a batch. dns-control places each record in the longest zone of the store that holds its name. A record that the lane policy refuses leaves the batch, and dns-control logs it, so one Ingress that claims a curated name does not hold back the lane. The plan derives create, change or delete from the store, not from the old side that external-dns sends, so a batch that external-dns sends again after a timeout changes nothing.
The sidecar presents each record set the way the pdns provider of external-dns did, so the first sync after the cutover is empty. The name has no trailing dot. A target loses its trailing dot, except for TXT, NAPTR and SRV, so a registry TXT keeps its quotes. A disabled record is left out, and ALIAS reads as CNAME. On a write, the sidecar adds the trailing dot back to the types of the provider, writes a CNAME at a zone apex as ALIAS, and writes a TTL of 0 as 300.
The sidecar authenticates to dns-control with a Keycloak client-credentials token for a new
client dns-control-external-dns that holds the role dns-lane-external-dns. Terraform writes
the client secret to the Terraform-owned subpath fzymgc-house/cluster/external-dns/oidc, and an
ExternalSecret supplies it. The allow-list of the ESO policy on cluster/* already covers that
path. dns-control serves the lane on a new route group
/api/lanes/external-dns/ that accepts that bearer token. The session-only opsRoute stays as it
is.
The txt registry stays. Its TXT records are ordinary record sets of the lane, so the cutover
changes no record and external-dns keeps its ownership logic. The view of one lane is the real
fence. The registry is now redundant, and dropping it later is one values change with no risk to
other lanes.
D-7. Lineage reads only dns-control’s stores
Section titled “D-7. Lineage reads only dns-control’s stores”The index builds every claim from the stores: the entry ledger, the zones (SOA and NS), the
control and external-dns record sets. kea-d2 stays attributed from DHCID. The
dns-aaaa-publish writer leaves with D-11. An external-dns lane record set shows the
writer external-dns with the mark of dns-control. internal/tfstate, HCP_TERRAFORM_TOKEN and
the “Terraform state not read” alert leave in the teardown pull request.
D-8. The export and the restore
Section titled “D-8. The export and the restore”The export writes zones.json beside entries.json: the version, every zone row and every record
set row, sorted, with no change id or timestamp, so the same state gives the same bytes. The Kea
seed keeps reading entries.json only.
dns-control restore --from <bucket> reads both objects, plans every zone first, then every record
set, then every entry, and applies them one at a time. --dry-run prints no change, create or
change per subject.
D-9. The role
Section titled “D-9. The role”tf/keycloak gains the client role dns-admin on the dns-control client and the client
dns-control-external-dns with the role dns-lane-external-dns. Sean holds dns-admin.
Terraform writes the client secret of dns-control-external-dns to
fzymgc-house/cluster/external-dns/oidc, never to the bare cluster/external-dns path. The
paired grant on that subpath in tf/vault/policy-keycloak.tf lands in its own pull request and
applies first, because the Keycloak write answers 403 without it. Sean decided this on
2026-10-02.
D-10. TSIG key rotation
Section titled “D-10. TSIG key rotation”The rotation keeps its widen and narrow pull requests. The Ansible half stays. The Terraform half,
var.tsig_key_generations_granted, becomes a dns-control change to the TSIG-ALLOW-DNSUPDATE
value of each granted zone. dns-control zones grant --generations <n-1>,<n> plans that change for
every dynamic zone at once. The grants name the Kea D2 keys only, once D-11 removes the
publisher’s key.
D-11. Retire dns-aaaa-publish
Section titled “D-11. Retire dns-aaaa-publish”The publisher reads the resolver’s own IPv6 neighbour table. On 2026-10-01 that table held 0 to 3
in-scope clients per leg, and Lab, the one leg the publisher covers, held 0 AAAA records. The
UniFi client table held a ULA for 27 clients on the same day. The change observed-aaaa-every-leg
moves the work into dns-control, so the publisher has no job left.
The teardown runs in this order. A dns-control change drops every generation of the publisher’s
key from the TSIG-ALLOW-DNSUPDATE grant of each dynamic zone. A retire playbook, following
ansible/unbound-retire.yml, stops and removes the timer, the service, the key file and the
publisher’s distribution in the shared resolver venv on each Pi, the passive node first. The venv
stays, because dns-tailscale also runs from it. The same pull request deletes the role, the tool, the Vault
policy and every reference. The reserved-label list in dns_hosts.yml has no reader after that,
so it goes too, and the seed record sets in dns-control are the only copy.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [The import misreads a writer and moves a record into the wrong lane] → the proof pull request
compares the dry run with
terraform state listand with the external-dns registry count before the handoff stores anything. - [external-dns waits on a busy lock and a new Ingress name is late] → the wait is one interval (30 s) per busy lock, and a change runs in seconds.
- [dns-control or its database is down and a name must change now] → break glass: write the pdns
API directly from a k8s node with the key at
fzymgc-house/infrastructure/dns/pdns-api, then adopt the change once dns-control is back. - [The export is lost along with the database] → both buckets hold the export, and the B2 copy is off site. pdns itself also keeps serving, and an import from live pdns rebuilds the stores.
- [AAAA records stop for the time between the teardown and
observed-aaaa-every-leg] → the publisher writes no record today, so nothing changes for a client.
Migration Plan
Section titled “Migration Plan”Each pull request in the proposal’s split lands in order. Rollback per live step:
- The handoff (the
removedblocks, the import, the project run). The marks are the only pdns change, and the records keep their content. Roll back the store withdns-control zones delete --importedand revert the pull request. To give Terraform the resources back, addimport {}blocks generated bydns-control zones export --format tf-import. - The external-dns cutover. Revert the values change. external-dns returns to the pdns provider, because the pdns key is removed only in the teardown pull request.
- The teardown. Revert the pull request. The workspace and the Vault role return through
Terraform, and the
removedblocks keep the workspace from touching pdns.
Open Questions
Section titled “Open Questions”- Resolved: the first pull request ships the API, the CLI and the MCP tools, and no zone editor in the UI. The Changes screen lists zone and record set changes by name.