Skip to content

Curated DNS records and Kea reservations as control-plane data

The curated host entries of fzymgc.house are data that dns-control holds, not configuration that Terraform applies. One Temporal workflow projects each change onto pdns, onto the Kea reservation files of both resolver nodes and onto two S3 sinks. Sean decided the model on 2026-09-29 after issue #2527, which explored why a live write followed by a capture pull request gave one fact two appliers. This design records the decisions and their alternatives.

See proposal.md for the motivation. The pieces this design builds on already exist:

  • apps/dns-control runs in k3s with a database on the shared CNPG main cluster, an ops_audit table with an append-only trigger, a lineage index that reads pdns RRset comments as writer marks, a leg file at /config/legs.json, and an OpsWorkflow that orders the two nodes from that leg file (Phase 4 D-17).
  • cmd/dns-node-worker runs on each Pi as user dns-worker with StateDirectory /var/lib/dns-worker, registers fixed activities on its own node queue, and holds a list write activity that writes a file with a temporary file and a rename. Every payload across the Temporal boundary is a protobuf message built with buf and checked by protovalidate (Phase 4 D-25).
  • Kea 3.2 on each Pi runs in hot-standby with a unix control socket that only root and _kea reach, and an HTTP control listener on 127.0.0.1:8000 with basic authentication. The Kea role renders subnet4[].reservations inline from tf/dns-records/hosts.yml by address arithmetic.
  • tf/dns-records reads hosts.yml (37 entries, 21 with a MAC) in hosts.tf and derives the A, AAAA, CNAME and PTR resources. Terraform 1.15 and the mmianl/powerdns 2.5.0 provider. The provider keeps a foreign comment when comments is unset.
  • tf/rustfs manages one bucket on the NAS RustFS through the aminueza/minio provider. tf/backblaze manages buckets and application keys and stores each key in Vault. The NAS S3 endpoint is 192.168.20.203, named s3-direct.nas.fzymgc.house in /etc/hosts on the Pis and in no DNS zone. docs/reference/network.md is the addressing authority. This change allocates no new address or prefix.
  • The dns-control network policy names every egress by namespace or address and forbids the Cilium DNS-name key. It holds no world entity today.

Goals:

  • One writer for a curated record and its reservation, and one store that both project from.
  • A change that completes with no pull request, no merge and no second applier.
  • A rebuild path for a Pi and for the entry set that needs neither the cluster nor git.
  • The smallest change to the Kea role and to the node worker that gives the above.

Non-Goals:

  • Moving legs, listeners, client groups or any other topology into the control plane. SEED-001 stays open for those.
  • Moving the records.tf families, the DDNS resources or the zones out of Terraform.
  • A hosts database for Kea, or Kea’s reservation-add command. Both put a database on the read path of a Pi.
  • Batching several changes into one export. The queue is serial and an unchanged set writes nothing, so one export per change is enough until the audit rows show bursts.
  • A review step after a change is live. The audit objects and the Changes screen are the record.

D-1: The entry is the unit of truth, and pdns and Kea are projections

Section titled “D-1: The entry is the unit of truth, and pdns and Kea are projections”

Sean ruled on 2026-08-18 that a device’s reservation and its A record are one literal. Under the old model the literal was a line in hosts.yml. Under this model it is an entry row. pdns holds the RRsets and the Kea fragment holds the reservation, and both are derived from the row by the workflow. The lineage index compares pdns with the row and reports drift.

Alternative: pdns as the truth for records and the database as the truth for MACs only. That keeps the invariant in the workflow instead of the data, and it gives the reconcile no target. Rejected.

The write API gains an entry shape beside the RRset shape of WRT-01. An entry write produces the A, AAAA, CNAME and PTR RRsets and the reservation in one plan. The RRset shape stays for TXT and CNAME names that are not host entries.

D-2: One ChangeWorkflow on the existing serial queue

Section titled “D-2: One ChangeWorkflow on the existing serial queue”

The workflow runs five activities in order: pdns write, reservation write on the passive node, reservation write on the active node, export to each sink, and audit finalise plus event. It reuses the readOrder of OpsWorkflow for the node order and the single-worker task queue of WRT-05. Each activity has its own retry policy. The pdns write and the reservation writes are idempotent by content, so a retry after a crash is safe. A failure in the reservation write does not roll back pdns: the audit row records the partial state, the Changes screen shows it, and a retry of the workflow finishes it. Temporal’s history is the record of a half-done change.

Alternative: one activity that does everything. Rejected, because a retry would repeat the pdns write and the export for a Kea failure.

D-3: The reservation write is a file per leg and a config-reload over the HTTP listener

Section titled “D-3: The reservation write is a file per leg and a config-reload over the HTTP listener”

The activity input is ReservationWriteInput { repeated LegReservations legs }, where each leg carries its name and repeated Reservation { hw_address, ip_address, hostname }. dns-control places an entry in a leg from ipv4_cidr in the leg file, so the worker does no arithmetic and the Kea subnet id never reaches dns-control. The worker writes /var/lib/dns-worker/kea/reservations-<leg>.json for every leg in the input, with the writeAtomic helper of the list write, and then posts config-reload to 127.0.0.1:8000 with the kea-ctrl basic-authentication credential. /etc/kea is 0750 root:_kea, so the worker cannot read the copy that the Kea role writes. The worker role reads the same Vault entry (secret/fzymgc-house/infrastructure/dns/kea-ha, fields ctrl_auth_user and ctrl_auth_password) on the control node and writes its own copy under /etc/dns-node-worker/kea/ at 0640 root:dns-worker, with the mode set at the write and a verify task that reads it back as the worker user.

The template renders "reservations": <?include "/var/lib/dns-worker/kea/reservations-<leg>.json"?> in each subnet block. The Phase 3 comment in the template that rejects an include file argues against a generated intermediate between two readers of one file. There is now one writer and no second reader, so the objection no longer applies, and the comment is rewritten to say why.

Alternative: the unix control socket. It needs root or _kea, and _kea in the worker’s SupplementaryGroups gives the worker every Kea command. The HTTP listener exists for kea-shell and the credential is already on the box. Chosen.

Alternative: config-set with the whole configuration. That makes the worker the owner of the Kea configuration and collides with Ansible. Rejected.

Rollback for a reservation change: apply the previous plan of the entry, or on a node run scripts/dns-playbook.sh --limit <node> --tags kea after removing the file, which reseeds it from the export.

D-4: The export is a dual write to a NAS bucket and a B2 bucket, decided by Sean on 2026-09-29

Section titled “D-4: The export is a dual write to a NAS bucket and a B2 bucket, decided by Sean on 2026-09-29”

One activity, change.export_sink, runs once per sink inside its own workflow change.export, each run with its own retry policy and each idempotent: the activity reads the current object’s sha256 from its metadata and skips the write when it matches. The object entries.json carries the sha256 as user metadata. The change row stays immutable after it is final, so the result of each sink lives in the side table change_exports, keyed by change and sink. A sink with no row is pending, and the Changes screen reads it as pending: <sink>. The export workflow is a child of the change workflow with the parent close policy abandon, so a sink that is down never holds the change lock.

Sean rejected git as the sink on 2026-09-29: the ruleset on main requires a pull request and its bypass only covers the pull request rule, so a bot cannot land on main, and a bot branch or a second repository adds a GitHub App for nothing the audit rows do not already give. Sean rejected RustFS tiered storage on 2026-09-29 because a tiered object is unreadable while the NAS is down. Sean rejected RustFS S3 Tables for the audit on 2026-09-29 because the append-only trigger in CNPG is the audit’s guarantee and Iceberg cannot give it.

The buckets: tf/rustfs adds minio_s3_bucket dns-control-export with prevent_destroy and minio_s3_bucket_versioning Enabled, plus a minio_iam_user, a minio_iam_policy scoped to that bucket and a vault_kv_secret_v2 for the key, per the RustFS runbook. tf/backblaze adds a b2_bucket with versioning and a b2_application_key scoped to it, stored in Vault beside cluster_backups. The dns-control ExternalSecret adds both keys.

Network: s3-direct.nas.fzymgc.house has no DNS record by design, so the Deployment gets a hostAliases entry for 192.168.20.203, and the network policy gets toCIDRSet for that address on 443. B2’s addresses are not fixed and the policy forbids the DNS-name key, so the policy gets toEntities: [world] on TCP 443, the shape the ClickStack policy uses for its S3 backups. The header comment of the policy is rewritten to name the two reasons.

D-5: The Kea role seeds from the export and never overwrites

Section titled “D-5: The Kea role seeds from the export and never overwrites”

The role reads entries.json on the control node with the aws command and the scoped key of each sink from Vault, the NAS bucket first and B2 when the NAS does not answer, checks the sha256 of the bytes against the object metadata, slices the set per trusted leg by the address arithmetic of the subnet block, and writes each file with force: false. A node with a file keeps it. A fresh node gets the export. kea_reservations_path and the include_vars task go away. kea-dhcp4 -t in the role’s verify stays and now covers the includes.

Alternative: seed empty arrays and let the worker fill them. Rejected because a reserved device on a fresh active node would get a pool address until the first change.

D-6: The Terraform handoff is removed with destroy = false and a mark adoption run

Section titled “D-6: The Terraform handoff is removed with destroy = false and a mark adoption run”

Terraform 1.15 supports removed { from = ...; lifecycle { destroy = false } }. The handoff pull request adds one removed block per resource family (hosts, hosts_aaaa, host_aliases, the PTR resources of reverse.tf), deletes hosts.yml and hosts.tf, and cuts the host_entries and canonical_ptr_map outputs and the canonical_ptr read in reverse-ddns.tf. The apply removes the resources from state and touches no zone. Before the merge, the store already holds the import of hosts.yml and the export diff is empty. After the apply, one workflow run marks every curated RRset with writer=dns-control change=<uuid> and writes nothing else, so the dig readings before and after are equal.

Rollback: git revert of the handoff pull request. The apply then imports nothing, because the resources are gone from state, so the rollback is terraform import of each resource from the export, with the import commands generated by dns-control entries export --format tf-import. That command ships in the same pull request as the removed blocks.

The mark grammar becomes writer=dns-control change=<uuid>. The parser accepts the old grammar with pr= until no RRset carries it, then the acceptance is removed.

D-7: Placement and validation live in dns-control, not in the worker

Section titled “D-7: Placement and validation live in dns-control, not in the worker”

The worker validates only the shape of each reservation. dns-control validates that an address sits in exactly one trusted leg, that a MAC is unique across entries, and that the write-policy filter accepts the hostname and every alias. The worker never reads the leg file.

  • [A half-done change: pdns written, reservation not] → The audit row and the Changes screen show the state, the workflow retries, and dns_list_drift reports the name until Kea matches.
  • [Kea refuses the reload because one included file is malformed] → The worker validates each reservation before the write, and a refused write leaves the old file. kea-dhcp4 -t runs in the role’s verify.
  • [The credential of the HTTP listener widens: the worker can send any command] → The restrict-commands list of that listener is checked in the design task, and config-reload, status-get and the lease commands are the set it allows.
  • [B2 egress opens world:443 for dns-control] → The policy comment names the reason, and the only world call in the binary is the B2 client.
  • [The NAS object and the B2 object disagree] → Both carry the sha256, the audit row holds the expected one, and the seed task refuses a file whose sha256 does not match the audit object it reads beside it.
  • [A person edits pdns by hand] → The lineage reconcile reports the name as drift. Sean ruled on 2026-09-14 against a drift detector for Terraform-owned inventory. This reconcile is different: it compares two stores the repository owns, which is the case the ruling allowed.
  • [The lineage index attributes fewer names to Terraform] → This is a gain. Issue #2413 shrinks to the records.tf families.
  1. Merge the buckets, credentials, ExternalSecret, hostAliases and network policy. Nothing reads them yet.
  2. Merge dns-control with the store, the workflow and the export activities. The store is empty.
  3. Merge the node worker and the Kea role. Converge the passive node, then the active node. The seed reads an export the store produced from a dry-run import of hosts.yml, so Kea renders the same reservations as before.
  4. Import hosts.yml into the store for real. Run the export diff. Run the lineage reconcile and make sure that no curated name shows drift.
  5. Merge the handoff. Read the plan: every curated record removed from state, 0 to destroy. After the apply, run the mark adoption, then the dig readings.
  6. Merge the docs and the planning amendments.

Rollback per step is in the decision that introduces it. Steps 1 to 4 are reversible by revert with no live effect.

  • The name of the B2 bucket and its region. tf/backblaze fixes the convention at implementation time, and nothing in the specs depends on the name.
  • Whether the Changes screen gets a “retry export now” button or the operator waits for the workflow’s retry. The retry policy decides the wait, and the button can come later.

This file is the copy of the design of the OpenSpec change curated-records-as-control-plane-data. The change directory holds the proposal, the specs and the tasks.