Skip to content

Design: observed AAAA on every leg

See proposal.md for the motivation. The pieces this design builds on:

  • The change dns-control-sole-writer gives dns-control a record set store with lanes, one change model for every subject, and one change lock. It retires dns-aaaa-publish. This change adds a lane to that store and starts after its pull request 1.
  • docs/reference/network.md is the addressing authority. The leg file of dns-control (argocd/app-configs/dns-control/config.yaml) mirrors it and holds ula_cidr for each leg. The five trusted legs are Core (fddb:f665:73f7:6::/64), Lab (:4::/64), Main (:1::/64), IoT (:3::/64) and MainWireless (:7::/64). This change allocates no address or prefix.
  • Kea D2 writes <host>.<leg>.lan.fzymgc.house A records with a DHCID, using ddns-conflict-resolution-mode: check-with-dhcid and ddns-update-on-renew: true. Its ADD fails at a name that holds any record, and its REPLACE needs a matching DHCID (tf/dns-records/reserved-seeds.tf records the mechanism, read off the Kea 3.2.0 source).
  • The UniFi controller on the UCG-Fiber answers at https://unifi.fzymgc.house (192.168.40.1) with a Let’s Encrypt certificate for the name only. A UniFi OS login is POST /api/auth/login, and the controller answers HTTP 429 to many logins in a short time (docs/operations/unifi.md).
  • A read on 2026-10-01 of GET /proxy/network/api/s/default/stat/sta returned 78 clients. Each record holds ipv6_addresses (the current set), last_ipv6 (a superset that keeps addresses that are no longer current), network, ip, mac and is_wired. 27 clients held a ULA, each inside its own leg /64, and none held more than one ULA.
  • No Temporal Schedule exists in the repository today.
  • apps/dns-control/internal/config is a hand-written loader: a pure function over the environment map, with time.ParseDuration for each duration. The Deployment carries the Reloader annotation, so a change to its ConfigMap or Secret restarts the pod.

Goals:

  • An AAAA record for every named trusted-leg client that uses a ULA, with no component on the resolver nodes.
  • A lane that cannot damage another lane or block a Kea D2 claim.

Non-Goals:

  • No PTR records for observed addresses. The ULA reverse zone holds curated PTRs only.
  • No AAAA on Guest and Telework. They have no internal names.
  • No event trigger from UniFi. The design leaves room for one (D-3).

D-1. The source is stat/sta and its ipv6_addresses

Section titled “D-1. The source is stat/sta and its ipv6_addresses”

The sweep reads GET /proxy/network/api/s/default/stat/sta and uses ipv6_addresses. It never uses last_ipv6, which on 2026-10-01 held stale addresses on 13 of 78 clients. The v2 endpoint /proxy/network/v2/api/site/default/clients/active carries the same list as ipv6_address. The older endpoint is chosen because its shape has been stable for many releases.

The client’s network names its leg. A map in the leg file ties each UniFi network name to a leg name, so a rename in UniFi fails loudly instead of matching nothing. The map is unifi_networks, and an empty leg name marks a network with no leg, such as WAN2. A client of a network that the map does not name fails the read.

A read on 2026-10-03 showed that ipv6_addresses is absent, not empty, on a client without IPv6, and that ip is absent on some clients. The reader treats both as normal. It fails on a record without mac, network or is_wired, and on a table where no record carries ipv6_addresses, because that is a renamed field. The base URL of the controller is the key unifi.url.

The reader keeps one session cookie across sweeps and logs in again only on HTTP 401. On HTTP 429 it waits for the next sweep. It connects by name, so the certificate check passes.

D-2. dns-control polls, on a Temporal Schedule

Section titled “D-2. dns-control polls, on a Temporal Schedule”

Sean decided on 2026-10-01 that dns-control does the poll. The alternatives were:

Option Why not
A. The publisher on each resolver node polls UniFi and writes with RFC 2136 Two fenced writers instead of one, a UniFi credential on the serving plane, and the node has no view of the fact that the cluster lacks. Its advantages: AAAA keeps updating with the cluster down, and pdns enforces its fence
A hybrid: dns-control schedules, the node worker polls and writes It carries the cluster dependency of B and the credential on the node of A

The cost of the choice: with the cluster down, AAAA records stay in pdns but stop following their clients. A records keep working through Kea D2.

dns-control creates the Schedule observed-aaaa-sweep at start. On “already exists” it updates it, so the configured interval always wins at start. The Schedule starts the workflow ObservedAAAASweep on the dns-control task queue at observed_aaaa.interval (D-11). The default is 300 seconds, the same as Kea’s ddns-ttl and the record TTL. A shorter interval publishes a new client sooner and costs one more controller read per interval. The record TTL stays 300 seconds, because it is the cache time for a client and not the sweep time. The overlap policy is skip, and the catch-up window is one interval. observed_aaaa.enabled: false creates it paused, and the update at start sets the paused state too. A run with dry_run plans from its one read, as if every address in it were published, and writes nothing, so a person can read the join before the enable. The workflow input is a protobuf message, SweepInput { dry_run }, through the existing data converter, as for every other payload across the Temporal boundary.

A later event trigger signals the same workflow with the same input. The sweep stays the reconciliation, because an event can be lost and an address that disappears raises no event.

D-4. The name comes from Kea D2, joined on IPv4

Section titled “D-4. The name comes from Kea D2, joined on IPv4”

The sweep takes the current lineage index in process. For each trusted dynamic zone, it maps each IPv4 address to the name whose A record has the writer kea-d2. It joins each client on its ip. A client with no such name gets no AAAA.

Several clients can ask Kea for one hostname, and only the first gets the name. On 2026-10-01, iphone.mainwireless.lan.fzymgc.house pointed at 192.168.26.233, and a second iPhone at 192.168.24.232 held no name. The join on IPv4 gives the AAAA to the client that owns the name and to no other.

The join also binds the MAC, because a sighting keeps the IPv4 address of its last read for up to the retire time. If the current read shows another MAC at that IPv4 address, the sightings of the old client are not published, because the Kea D2 name there belongs to the new client. Of two MACs whose sightings name one IPv4 address, only the MAC seen last keeps its addresses. The read keeps each client of a trusted leg that has an IPv4 address, including a client without a ULA, so the join knows the holder of each address.

A curated entry lives in the apex and has its own IPv6 field. The lane writes only in the <leg>.lan zones, and the policy refuses a name that is the hostname or an alias of an entry.

The sweep keeps an address only when it is inside the ula_cidr of the client’s leg. It drops GUAs, because the ISP re-delegates the prefix (docs/operations/dns.md, the scope of AAAA publication), link-local addresses, and foreign ULAs. A foreign fd99:: ULA, most likely a Thread border router’s prefix, showed in one client’s history on 2026-10-01.

A table observed_addresses holds one row per client and address: the MAC, the leg, the address, the first and last sighting, and the count of sweeps that saw it. The row also keeps the IPv4 address of the client at the last sighting, and the join reads it there. A client that leaves the table then keeps its AAAA until the retire time passes, as the spec asks. An address is published once observed_aaaa.publish_after sweeps have seen it, two by default. It is removed once observed_aaaa.retire_after passes without a sighting, 30 minutes by default, and its row goes with it. Both are times and counts, not sweep numbers, so a change of the interval keeps their meaning.

The state lives in Postgres and not in the workflow. A Schedule starts a fresh run each time, and the Health screen reads the table. A restart of dns-control keeps every sighting, so it neither delays nor removes an address. The in-memory debouncer of the lineage index lacks exactly this, as issue #2614 records.

Four rules keep the lane from blocking a Kea D2 claim or leaving a record behind. If a delete of rule 3 or rule 4 fails, the change lists the AAAA as failed, the store keeps its row, and the next sweep removes the AAAA again and retries both:

  1. The lane writes an AAAA only at a name where Kea D2 holds the A record and its DHCID.
  2. The first sweep that finds no kea-d2 A record at a name removes the lane’s AAAA there.
  3. If that name then holds only a DHCID, the same change removes the DHCID as well.
  4. Before it removes that DHCID, the change removes the PTR and the DHCID at each reverse name whose DHCID equals it and whose PTR points at the name.

RFC 4703 §5.5 says that a client’s DHCID goes only when no other address record remains at the name. The proof pull request measured Kea 3.2 on 2026-10-03 with two throwaway clients on Main. When a lease ends while the lane’s AAAA is present, Kea D2 removes the A record and keeps the DHCID. Its log reads DHCP_DDNS_FORWARD_REMOVE_RRS_REJECTED with RCODE 7, then DHCP_DDNS_REMOVE_FAILED. Rule 2 removes the AAAA, and the DHCID alone then blocks every other client from the name. The second client’s claim failed with DHCP_DDNS_FORWARD_REPLACE_REJECTED until rule 3 removed the DHCID. Rule 3 removes that leftover, which only the lane’s own record caused.

Kea D2 does not try the reverse removal after its forward removal fails. The PTR and the DHCID at the reverse name of the old IPv4 address stay, and the PTR names a name that the client no longer holds. Rule 4 removes them. Kea D2 writes the same DHCID at the forward and the reverse name, so the match is exact. If Kea D2 has since leased the address to another client, the reverse name holds that client’s DHCID, and rule 4 leaves it. Rule 4 runs before rule 3, because the lone forward DHCID is what makes the next sweep act again after a failed reverse delete.

A residual window remains: for up to one sweep after a lease ends, a new client that asks for the same name fails its claim. It gets the name at its next renewal that is not a reused lease, because ddns-update-on-renew is on. Kea caches leases, so a renewal soon after an allocation reuses the lease and sends no update. On Main the first renewal comes after 6 hours, at T1 of a 43200-second lease.

A sweep that cannot read the client table, or reads zero clients, changes nothing and records the error for the Health screen. A sweep that would remove more than half of the lane’s record sets at once writes nothing, records the refusal and raises an alert. The alert is a log line at ERROR and the status failing on the Health screen and in dns_get_health. The guard counts only the removals at names where Kea D2 still holds the A record, because only those depend on the read, and it holds back no run with two removals or fewer. A removal after Kea D2 removed its A record goes through, so a small lane can still free a released name (rule 2 of D-7). A partial read must not empty the lane.

A sweep that finds a difference plans and applies one change: subject batch, because one sweep can change several names, lane observed-aaaa, actor system:observed-aaaa. The caller kind and the surface are both system. Migration 00006 adds both values, the lane, and the two sweep tables. A sweep with no difference writes nothing. When the change lock is busy, the sweep writes nothing and the next sweep tries again. The lane is left out of zones.json, because the next sweep rebuilds it from the client table.

A UniFi local account dns-control-observer with the read-only Network role. Sean decided on 2026-10-01 that automation accounts keep a password and use no API key. The username and the password live at fzymgc-house/cluster/dns-control as unifi_username and unifi_password, which the existing ExternalSecret policy already reads. The network policy of dns-control gains egress to 192.168.40.1/32 on TCP 443. The name unifi.fzymgc.house resolves through cluster DNS.

internal/config moves onto koanf v2 in its own pull request, before any observed AAAA code. The layers load in this order, and each later layer wins:

  1. Defaults, through the confmap provider.
  2. The YAML file /config/dns-control.yaml, a new key in the existing ConfigMap. A missing file is not an error, so a pod without the key starts on defaults and environment.
  3. The environment, through the env/v2 provider with the prefix DNS_CONTROL_.

Every variable that dns-control reads gets the prefix DNS_CONTROL_, so its names cannot collide with names that the platform or a library sets. The three OpenTelemetry names (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME) keep their standard names, because the OpenTelemetry SDK defines and reads them.

One rule maps a variable to a key: strip the prefix, change to lower case, and read __ as one level of nesting. DNS_CONTROL_OIDC_CLIENT_ID is oidc_client_id, and DNS_CONTROL_OBSERVED_AAAA__INTERVAL is observed_aaaa.interval. So every key in the file can be set from the environment, and no table of names can drift from the struct. The alternative was an explicit table that kept today’s names. It would keep the old names but add a list to maintain.

The decode is strict. A key that the struct does not hold stops dns-control at start, and the error names the key, so a misspelt variable fails loudly. That needs enableServiceLinks: false on the pod. Kubernetes otherwise sets DNS_CONTROL_SERVICE_HOST, DNS_CONTROL_PORT and more for the Service dns-control, and the strict decode would refuse them. Nothing in the pod reads those variables.

The CLI reads its configuration through the same loader. Today cmd/dns-control/entries.go reads DATABASE_URL and LEGS_FILE with os.Getenv. The external-dns sidecar dns-control webhook runs in the external-dns pod. It loads its own configuration the same way, with the prefix DNS_CONTROL_WEBHOOK_, the same rule and a strict decode, and it has no file layer. Sean decided on 2026-10-03 that the sidecar moves in the same pull request.

The rename lands in the pin pull request of the release that carries the loader, together with the image digest and enableServiceLinks: false. The same pull request renames the env entries of the sidecar in argocd/app-configs/external-dns/values.yaml. The old image keeps the old names until that pull request merges, so no release reads both forms, and a revert of the pin restores both together. A test loads today’s environment map under the old names through the old loader, and the same values under the new names through the new loader, and makes sure that the two configurations are equal.

The struct carries koanf tags, and koanf’s default decoder turns "30s" into a time.Duration. Validation runs after the unmarshal and names the key in each error. Load stays a pure function: it takes the environment entries in the form of os.Environ and the file bytes, and passes the entries to the EnvironFunc of the env/v2 provider. The tests need no process environment and no file.

The new keys:

Key Environment Default Bounds
observed_aaaa.enabled DNS_CONTROL_OBSERVED_AAAA__ENABLED false
observed_aaaa.interval DNS_CONTROL_OBSERVED_AAAA__INTERVAL 300s 30 s to 1 h
observed_aaaa.publish_after DNS_CONTROL_OBSERVED_AAAA__PUBLISH_AFTER 2 1 to 10 sightings
observed_aaaa.retire_after DNS_CONTROL_OBSERVED_AAAA__RETIRE_AFTER 30m at least two intervals
unifi.username DNS_CONTROL_UNIFI__USERNAME none required when enabled
unifi.password DNS_CONTROL_UNIFI__PASSWORD none required when enabled

A change of a key takes effect on the next start. A change to the ConfigMap restarts the pod through Reloader, and the start updates the Schedule. A live reload through a koanf file watch was the alternative. It would add a second path that changes a running sweep, and the restart already applies a change within a minute.

  • [The cluster is down, and AAAA records point at addresses that clients no longer use] → each record keeps its 300 second TTL, ULAs on these legs did not rotate on 2026-10-01, and the first sweep after the outage corrects every record.
  • [A UniFi upgrade changes the shape of stat/sta] → the reader fails on a missing field instead of reading an empty list, and D-8 then changes nothing.
  • [A client never shows a ULA in UniFi, for example a device that has IPv6 but never uses its ULA] → it gets no AAAA. On 2026-10-01, 17 clients held IPv6 but no ULA. DNS cannot publish an address that the device does not use.
  • [Kea 3.2 removal behaves unlike RFC 4703] → the proof pull request measured it before the enable. Kea 3.2 follows RFC 4703, and it also skips the reverse removal after a failed forward removal. Rule 4 of D-7 covers the reverse records.

The pull requests land in the order of the proposal. The enable is the only live step. Roll back by pausing the Schedule with temporal schedule toggle --pause --schedule-id observed-aaaa-sweep, then plan and apply dns-control lane purge observed-aaaa, which removes every record set of the lane in one change. Set observed_aaaa.enabled: false in the same pull request that reverts the enable, so a restart keeps the Schedule paused. The koanf pull request rolls back with git revert, because it changes no behaviour.