Design: observed AAAA on every leg
Context
Section titled “Context”See proposal.md for the motivation. The pieces this design builds on:
- The change
dns-control-sole-writergives dns-control a record set store with lanes, one change model for every subject, and one change lock. It retiresdns-aaaa-publish. This change adds a lane to that store and starts after its pull request 1. docs/reference/network.mdis the addressing authority. The leg file of dns-control (argocd/app-configs/dns-control/config.yaml) mirrors it and holdsula_cidrfor each leg. The five trusted legs are Core (fddb:f665:73f7:6::/64), Lab (:4::/64), Main (:1::/64), IoT (:3::/64) and MainWireless (:7::/64). This change allocates no address or prefix.- Kea D2 writes
<host>.<leg>.lan.fzymgc.houseA records with a DHCID, usingddns-conflict-resolution-mode: check-with-dhcidandddns-update-on-renew: true. Its ADD fails at a name that holds any record, and its REPLACE needs a matching DHCID (tf/dns-records/reserved-seeds.tfrecords the mechanism, read off the Kea 3.2.0 source). - The UniFi controller on the UCG-Fiber answers at
https://unifi.fzymgc.house(192.168.40.1) with a Let’s Encrypt certificate for the name only. A UniFi OS login isPOST /api/auth/login, and the controller answers HTTP 429 to many logins in a short time (docs/operations/unifi.md). - A read on 2026-10-01 of
GET /proxy/network/api/s/default/stat/stareturned 78 clients. Each record holdsipv6_addresses(the current set),last_ipv6(a superset that keeps addresses that are no longer current),network,ip,macandis_wired. 27 clients held a ULA, each inside its own leg/64, and none held more than one ULA. - No Temporal Schedule exists in the repository today.
apps/dns-control/internal/configis a hand-written loader: a pure function over the environment map, withtime.ParseDurationfor each duration. The Deployment carries the Reloader annotation, so a change to its ConfigMap or Secret restarts the pod.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- An AAAA record for every named trusted-leg client that uses a ULA, with no component on the resolver nodes.
- A lane that cannot damage another lane or block a Kea D2 claim.
Non-Goals:
- No PTR records for observed addresses. The ULA reverse zone holds curated PTRs only.
- No AAAA on Guest and Telework. They have no internal names.
- No event trigger from UniFi. The design leaves room for one (D-3).
Decisions
Section titled “Decisions”D-1. The source is stat/sta and its ipv6_addresses
Section titled “D-1. The source is stat/sta and its ipv6_addresses”The sweep reads GET /proxy/network/api/s/default/stat/sta and uses ipv6_addresses. It never
uses last_ipv6, which on 2026-10-01 held stale addresses on 13 of 78 clients. The v2 endpoint
/proxy/network/v2/api/site/default/clients/active carries the same list as ipv6_address. The
older endpoint is chosen because its shape has been stable for many releases.
The client’s network names its leg. A map in the leg file ties each UniFi network name to a leg
name, so a rename in UniFi fails loudly instead of matching nothing. The map is unifi_networks, and
an empty leg name marks a network with no leg, such as WAN2. A client of a network that the map
does not name fails the read.
A read on 2026-10-03 showed that ipv6_addresses is absent, not empty, on a client without IPv6,
and that ip is absent on some clients. The reader treats both as normal. It fails on a record
without mac, network or is_wired, and on a table where no record carries ipv6_addresses,
because that is a renamed field. The base URL of the controller is the key unifi.url.
The reader keeps one session cookie across sweeps and logs in again only on HTTP 401. On HTTP 429 it waits for the next sweep. It connects by name, so the certificate check passes.
D-2. dns-control polls, on a Temporal Schedule
Section titled “D-2. dns-control polls, on a Temporal Schedule”Sean decided on 2026-10-01 that dns-control does the poll. The alternatives were:
| Option | Why not |
|---|---|
| A. The publisher on each resolver node polls UniFi and writes with RFC 2136 | Two fenced writers instead of one, a UniFi credential on the serving plane, and the node has no view of the fact that the cluster lacks. Its advantages: AAAA keeps updating with the cluster down, and pdns enforces its fence |
| A hybrid: dns-control schedules, the node worker polls and writes | It carries the cluster dependency of B and the credential on the node of A |
The cost of the choice: with the cluster down, AAAA records stay in pdns but stop following their clients. A records keep working through Kea D2.
D-3. The Schedule
Section titled “D-3. The Schedule”dns-control creates the Schedule observed-aaaa-sweep at start. On “already exists” it updates
it, so the configured interval always wins at start. The Schedule starts the workflow
ObservedAAAASweep on the dns-control task queue at observed_aaaa.interval (D-11). The default
is 300 seconds, the same as Kea’s ddns-ttl and the record TTL. A shorter interval publishes a new
client sooner and costs one more controller read per interval. The record TTL stays 300 seconds,
because it is the cache time for a client and not the sweep time. The overlap policy is skip, and
the catch-up window is one interval. observed_aaaa.enabled: false creates it paused, and the
update at start sets the paused state too. A run with dry_run plans from its one read, as if every
address in it were published, and writes nothing, so a person can read the join before the enable. The workflow
input is a protobuf message, SweepInput { dry_run }, through the existing data converter, as for
every other payload across the Temporal boundary.
A later event trigger signals the same workflow with the same input. The sweep stays the reconciliation, because an event can be lost and an address that disappears raises no event.
D-4. The name comes from Kea D2, joined on IPv4
Section titled “D-4. The name comes from Kea D2, joined on IPv4”The sweep takes the current lineage index in process. For each trusted dynamic zone, it maps each
IPv4 address to the name whose A record has the writer kea-d2. It joins each client on its ip.
A client with no such name gets no AAAA.
Several clients can ask Kea for one hostname, and only the first gets the name. On 2026-10-01,
iphone.mainwireless.lan.fzymgc.house pointed at 192.168.26.233, and a second iPhone at
192.168.24.232 held no name. The join on IPv4 gives the AAAA to the client that owns the name
and to no other.
The join also binds the MAC, because a sighting keeps the IPv4 address of its last read for up to the retire time. If the current read shows another MAC at that IPv4 address, the sightings of the old client are not published, because the Kea D2 name there belongs to the new client. Of two MACs whose sightings name one IPv4 address, only the MAC seen last keeps its addresses. The read keeps each client of a trusted leg that has an IPv4 address, including a client without a ULA, so the join knows the holder of each address.
A curated entry lives in the apex and has its own IPv6 field. The lane writes only in the
<leg>.lan zones, and the policy refuses a name that is the hostname or an alias of an entry.
D-5. The filter
Section titled “D-5. The filter”The sweep keeps an address only when it is inside the ula_cidr of the client’s leg. It drops GUAs,
because the ISP re-delegates the prefix (docs/operations/dns.md, the scope of AAAA publication),
link-local addresses, and foreign ULAs. A foreign fd99:: ULA, most likely a Thread border
router’s prefix, showed in one client’s history on 2026-10-01.
D-6. Sighting state in Postgres
Section titled “D-6. Sighting state in Postgres”A table observed_addresses holds one row per client and address: the MAC, the leg, the address,
the first and last sighting, and the count of sweeps that saw it. The row also keeps the IPv4 address
of the client at the last sighting, and the join reads it there. A client that leaves the table then
keeps its AAAA until the retire time passes, as the spec asks. An address is published once
observed_aaaa.publish_after sweeps have seen it, two by default. It is removed once
observed_aaaa.retire_after passes without a sighting, 30 minutes by default, and its row goes with
it. Both are times and counts, not sweep numbers, so a change of the interval keeps their meaning.
The state lives in Postgres and not in the workflow. A Schedule starts a fresh run each time, and the Health screen reads the table. A restart of dns-control keeps every sighting, so it neither delays nor removes an address. The in-memory debouncer of the lineage index lacks exactly this, as issue #2614 records.
D-7. Coexistence with Kea D2
Section titled “D-7. Coexistence with Kea D2”Four rules keep the lane from blocking a Kea D2 claim or leaving a record behind. If a delete of rule 3 or rule 4 fails, the change lists the AAAA as failed, the store keeps its row, and the next sweep removes the AAAA again and retries both:
- The lane writes an AAAA only at a name where Kea D2 holds the A record and its DHCID.
- The first sweep that finds no
kea-d2A record at a name removes the lane’s AAAA there. - If that name then holds only a DHCID, the same change removes the DHCID as well.
- Before it removes that DHCID, the change removes the PTR and the DHCID at each reverse name whose DHCID equals it and whose PTR points at the name.
RFC 4703 §5.5 says that a client’s DHCID goes only when no other address record remains at the
name. The proof pull request measured Kea 3.2 on 2026-10-03 with two throwaway clients on Main.
When a lease ends while the lane’s AAAA is present, Kea D2 removes the A record and keeps the
DHCID. Its log reads DHCP_DDNS_FORWARD_REMOVE_RRS_REJECTED with RCODE 7, then
DHCP_DDNS_REMOVE_FAILED. Rule 2 removes the AAAA, and the DHCID alone then blocks every other
client from the name. The second client’s claim failed with DHCP_DDNS_FORWARD_REPLACE_REJECTED
until rule 3 removed the DHCID. Rule 3 removes that leftover, which only the lane’s own record
caused.
Kea D2 does not try the reverse removal after its forward removal fails. The PTR and the DHCID at the reverse name of the old IPv4 address stay, and the PTR names a name that the client no longer holds. Rule 4 removes them. Kea D2 writes the same DHCID at the forward and the reverse name, so the match is exact. If Kea D2 has since leased the address to another client, the reverse name holds that client’s DHCID, and rule 4 leaves it. Rule 4 runs before rule 3, because the lone forward DHCID is what makes the next sweep act again after a failed reverse delete.
A residual window remains: for up to one sweep after a lease ends, a new client that asks for the
same name fails its claim. It gets the name at its next renewal that is not a reused lease, because
ddns-update-on-renew is on. Kea caches leases, so a renewal soon after an allocation reuses the
lease and sends no update. On Main the first renewal comes after 6 hours, at T1 of a 43200-second
lease.
D-8. Failure safety
Section titled “D-8. Failure safety”A sweep that cannot read the client table, or reads zero clients, changes nothing and records the
error for the Health screen. A sweep that would remove more than half of the lane’s record sets at
once writes nothing, records the refusal and raises an alert. The alert is a log line at ERROR and
the status failing on the Health screen and in dns_get_health. The guard counts only the removals
at names where Kea D2 still holds the A record, because only those depend on the read, and it holds
back no run with two removals or fewer. A removal after Kea D2 removed its A record goes through,
so a small lane can still free a released name (rule 2 of D-7). A partial read must not empty the
lane.
D-9. Audit and export
Section titled “D-9. Audit and export”A sweep that finds a difference plans and applies one change: subject batch, because one sweep
can change several names, lane observed-aaaa, actor system:observed-aaaa. The caller kind and
the surface are both system. Migration 00006 adds both values, the lane, and the two sweep tables. A sweep with no difference writes nothing. When the
change lock is busy, the sweep writes nothing and the next sweep tries again. The lane is left out
of zones.json, because the next sweep rebuilds it from the client table.
D-10. The credential
Section titled “D-10. The credential”A UniFi local account dns-control-observer with the read-only Network role. Sean decided on
2026-10-01 that automation accounts keep a password and use no API key. The username and the
password live at fzymgc-house/cluster/dns-control as unifi_username and unifi_password, which
the existing ExternalSecret policy already reads. The network policy of dns-control gains egress
to 192.168.40.1/32 on TCP 443. The name unifi.fzymgc.house resolves through cluster DNS.
D-11. The configuration moves onto koanf
Section titled “D-11. The configuration moves onto koanf”internal/config moves onto koanf v2 in its own pull request, before any observed AAAA code. The
layers load in this order, and each later layer wins:
- Defaults, through the
confmapprovider. - The YAML file
/config/dns-control.yaml, a new key in the existing ConfigMap. A missing file is not an error, so a pod without the key starts on defaults and environment. - The environment, through the
env/v2provider with the prefixDNS_CONTROL_.
Every variable that dns-control reads gets the prefix DNS_CONTROL_, so its names cannot collide
with names that the platform or a library sets. The three OpenTelemetry names
(OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME) keep their
standard names, because the OpenTelemetry SDK defines and reads them.
One rule maps a variable to a key: strip the prefix, change to lower case, and read __ as one
level of nesting. DNS_CONTROL_OIDC_CLIENT_ID is oidc_client_id, and
DNS_CONTROL_OBSERVED_AAAA__INTERVAL is observed_aaaa.interval. So every key in the file can be
set from the environment, and no table of names can drift from the struct. The alternative was an
explicit table that kept today’s names. It would keep the old names but add a list to maintain.
The decode is strict. A key that the struct does not hold stops dns-control at start, and the error
names the key, so a misspelt variable fails loudly. That needs enableServiceLinks: false on the
pod. Kubernetes otherwise sets DNS_CONTROL_SERVICE_HOST, DNS_CONTROL_PORT and more for the
Service dns-control, and the strict decode would refuse them. Nothing in the pod reads those
variables.
The CLI reads its configuration through the same loader. Today cmd/dns-control/entries.go reads
DATABASE_URL and LEGS_FILE with os.Getenv. The external-dns sidecar dns-control webhook
runs in the external-dns pod. It loads its own configuration the same way, with the prefix
DNS_CONTROL_WEBHOOK_, the same rule and a strict decode, and it has no file layer. Sean decided on
2026-10-03 that the sidecar moves in the same pull request.
The rename lands in the pin pull request of the release that carries the loader, together with the
image digest and enableServiceLinks: false. The same pull request renames the env entries of the
sidecar in argocd/app-configs/external-dns/values.yaml. The old image keeps the old names until that pull
request merges, so no release reads both forms, and a revert of the pin restores both together. A
test loads today’s environment map under the old names through the old loader, and the same values
under the new names through the new loader, and makes sure that the two configurations are equal.
The struct carries koanf tags, and koanf’s default decoder turns "30s" into a time.Duration.
Validation runs after the unmarshal and names the key in each error. Load stays a pure function: it
takes the environment entries in the form of os.Environ and the file bytes, and passes the entries
to the EnvironFunc of the env/v2 provider. The tests need no process environment and no file.
The new keys:
| Key | Environment | Default | Bounds |
|---|---|---|---|
observed_aaaa.enabled |
DNS_CONTROL_OBSERVED_AAAA__ENABLED |
false |
|
observed_aaaa.interval |
DNS_CONTROL_OBSERVED_AAAA__INTERVAL |
300s |
30 s to 1 h |
observed_aaaa.publish_after |
DNS_CONTROL_OBSERVED_AAAA__PUBLISH_AFTER |
2 |
1 to 10 sightings |
observed_aaaa.retire_after |
DNS_CONTROL_OBSERVED_AAAA__RETIRE_AFTER |
30m |
at least two intervals |
unifi.username |
DNS_CONTROL_UNIFI__USERNAME |
none | required when enabled |
unifi.password |
DNS_CONTROL_UNIFI__PASSWORD |
none | required when enabled |
A change of a key takes effect on the next start. A change to the ConfigMap restarts the pod through Reloader, and the start updates the Schedule. A live reload through a koanf file watch was the alternative. It would add a second path that changes a running sweep, and the restart already applies a change within a minute.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [The cluster is down, and AAAA records point at addresses that clients no longer use] → each record keeps its 300 second TTL, ULAs on these legs did not rotate on 2026-10-01, and the first sweep after the outage corrects every record.
- [A UniFi upgrade changes the shape of
stat/sta] → the reader fails on a missing field instead of reading an empty list, and D-8 then changes nothing. - [A client never shows a ULA in UniFi, for example a device that has IPv6 but never uses its ULA] → it gets no AAAA. On 2026-10-01, 17 clients held IPv6 but no ULA. DNS cannot publish an address that the device does not use.
- [Kea 3.2 removal behaves unlike RFC 4703] → the proof pull request measured it before the enable. Kea 3.2 follows RFC 4703, and it also skips the reverse removal after a failed forward removal. Rule 4 of D-7 covers the reverse records.
Migration Plan
Section titled “Migration Plan”The pull requests land in the order of the proposal. The enable is the only live step. Roll back by
pausing the Schedule with temporal schedule toggle --pause --schedule-id observed-aaaa-sweep, then
plan and apply dns-control lane purge observed-aaaa, which removes every record set of the lane in
one change. Set observed_aaaa.enabled: false in the same pull request that reverts the enable, so a restart
keeps the Schedule paused. The koanf pull request rolls back with git revert, because it changes
no behaviour.