Skip to content

dns-control

Operational guide for dns-control, the control plane over the resolver pair.

dns-control serves four screens at https://dns-control.fzymgc.house: Search, Name detail, Query log and Health. It shows which writer owns each record on the pdns nodes sandiego and portland, and whether the two nodes agree. dns-control changes no record. It has no button that changes a record.

dns-control also runs four operations actions on the resolver nodes: a cache flush, a list refresh, a drain and an undrain. See Operations actions.

It reads both pdns nodes, over the pdns HTTP API on port 8081, by IP address only. It reads ClickHouse, as the read-only user dns_control_reader, for the Query log and the Health metrics.

The index of records lives in memory. dns-control builds it from the pdns nodes at start, so a restart loses nothing.

Property Value
URL https://dns-control.fzymgc.house
Namespace dns-control
Workload Deployment/dns-control, one replica, Service/dns-control on port 8080 (http) and port 8081 (mcp)
Image ghcr.io/fzymgc-house/dns-control, pinned by version tag in the kustomization.yaml of the app. Argo CD Image Updater moves the pin
Source apps/dns-control/
Keycloak clients dns-control, in tf/keycloak/dns_control.tf. mcp-dns and dns-agent-triage for the MCP path. dns-control-external-dns for the external-dns lane
MCP URL https://llm.fzymgc.house/dns/mcp, through the LiteLLM route dns
Vault paths fzymgc-house/cluster/dns-control/oidc, fzymgc-house/cluster/dns-control, fzymgc-house/infrastructure/dns/pdns-api, fzymgc-house/cluster/clickstack, fzymgc-house/cluster/postgres/users/main-dns-control. The ExternalSecret dns-control-db renders the database URL from the Postgres path, and the Deployment mounts the property nats_creds at /etc/dns-control/nats/dns-control.creds
Manifests argocd/app-configs/dns-control/
Logs ClickStack, ServiceName = 'dns-control'
Audit store The database dns_control on the CNPG cluster main, table ops_audit
Temporal The namespace dns-control, the task queue dns-control for the workflows, and dns-node-<node> for each node worker
Events The JetStream stream DNS_EVENTS in the NATS account DNS

dns-control loads its configuration from three layers. Each later layer replaces the values of the layers before it:

  1. The defaults in apps/dns-control/internal/config.
  2. The YAML file /config/dns-control.yaml, the key dns-control.yaml of the ConfigMap dns-control-legs. A missing or empty file is not an error.
  3. The environment variables that start with DNS_CONTROL_.

One rule maps a variable to a key. Remove the prefix, change the rest to lower case, and read a double underscore (__) as one level of nesting. DNS_CONTROL_OIDC_CLIENT_ID sets oidc_client_id, and DNS_CONTROL_OBSERVED_AAAA__INTERVAL sets observed_aaaa.interval. An empty variable counts as unset. OTEL_EXPORTER_OTLP_ENDPOINT keeps its standard name, because the OpenTelemetry SDK defines it. The SDK reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_RESOURCE_ATTRIBUTES itself. Each process sets its service name in code, so OTEL_SERVICE_NAME changes nothing (see Telemetry).

The decode is strict. If a key is not known, dns-control does not start, and the error names the key, for example 'config.Config' has invalid keys: http_adrr. A duration is a Go duration string such as 30s, and a bare number is refused. The pod sets enableServiceLinks: false. Without it, Kubernetes sets DNS_CONTROL_SERVICE_HOST and other variables for the Service dns-control, and the strict decode refuses them.

A change takes effect at the next start. Reloader restarts the pod when the ConfigMap or the Secret dns-control changes.

Key Variable Default and limits
http_addr DNS_CONTROL_HTTP_ADDR :8080
request_timeout DNS_CONTROL_REQUEST_TIMEOUT 30s
otel_exporter_otlp_endpoint OTEL_EXPORTER_OTLP_ENDPOINT Empty, which turns telemetry off
oidc_issuer_url DNS_CONTROL_OIDC_ISSUER_URL https://id.fzymgc.house/realms/fzymgc
oidc_client_id DNS_CONTROL_OIDC_CLIENT_ID dns-control
oidc_client_secret DNS_CONTROL_OIDC_CLIENT_SECRET Required
oidc_redirect_url DNS_CONTROL_OIDC_REDIRECT_URL https://dns-control.fzymgc.house/auth/callback
oidc_post_logout_redirect_url DNS_CONTROL_OIDC_POST_LOGOUT_REDIRECT_URL https://dns-control.fzymgc.house/
extra_ca_file DNS_CONTROL_EXTRA_CA_FILE Empty, which means the system roots only
pdns_api_key DNS_CONTROL_PDNS_API_KEY Required
legs_file DNS_CONTROL_LEGS_FILE /config/legs.json
clickhouse_url DNS_CONTROL_CLICKHOUSE_URL http://cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc.cluster.local:8123
clickhouse_user DNS_CONTROL_CLICKHOUSE_USER dns_control_reader
clickhouse_password DNS_CONTROL_CLICKHOUSE_PASSWORD Required
zone_refresh_interval DNS_CONTROL_ZONE_REFRESH_INTERVAL 30s
mcp_addr DNS_CONTROL_MCP_ADDR :8081
mcp_audience DNS_CONTROL_MCP_AUDIENCE https://llm.fzymgc.house/dns/mcp
mcp_resource_metadata_url DNS_CONTROL_MCP_RESOURCE_METADATA_URL https://llm.fzymgc.house/.well-known/oauth-protected-resource/dns/mcp
mcp_static_key_hashes DNS_CONTROL_MCP_STATIC_KEY_HASHES Empty, which accepts no static key
ui_base_url DNS_CONTROL_UI_BASE_URL https://dns-control.fzymgc.house
temporal_addr DNS_CONTROL_TEMPORAL_ADDR temporal-frontend.temporal.svc.cluster.local:7233
temporal_namespace DNS_CONTROL_TEMPORAL_NAMESPACE dns-control
database_url DNS_CONTROL_DATABASE_URL Required
nats_url DNS_CONTROL_NATS_URL nats://nats.nats.svc.cluster.local:4222
nats_creds_file DNS_CONTROL_NATS_CREDS_FILE /etc/dns-control/nats/dns-control.creds
nats_ca_file DNS_CONTROL_NATS_CA_FILE /etc/fzymgc-ca/ca.crt
drain_default DNS_CONTROL_DRAIN_DEFAULT 30m, from 1m to 1440m
flush_guard_threshold DNS_CONTROL_FLUSH_GUARD_THRESHOLD 0.05, greater than 0 and less than 1
ops_wait DNS_CONTROL_OPS_WAIT 120s, less than 150s
export_nas.endpoint, export_nas.bucket, export_nas.access_key, export_nas.secret_key DNS_CONTROL_EXPORT_NAS__ENDPOINT and so on Required
export_nas.region DNS_CONTROL_EXPORT_NAS__REGION Empty
export_b2.endpoint, export_b2.region, export_b2.bucket, export_b2.access_key, export_b2.secret_key DNS_CONTROL_EXPORT_B2__ENDPOINT and so on Required
observed_aaaa.enabled DNS_CONTROL_OBSERVED_AAAA__ENABLED false
observed_aaaa.interval DNS_CONTROL_OBSERVED_AAAA__INTERVAL 300s, from 30s to 1h
observed_aaaa.publish_after DNS_CONTROL_OBSERVED_AAAA__PUBLISH_AFTER 2, from 1 to 10
observed_aaaa.retire_after DNS_CONTROL_OBSERVED_AAAA__RETIRE_AFTER 30m, at least two intervals
unifi.url DNS_CONTROL_UNIFI__URL https://unifi.fzymgc.house
unifi.username, unifi.password DNS_CONTROL_UNIFI__USERNAME, DNS_CONTROL_UNIFI__PASSWORD Required when observed_aaaa.enabled is true
unifi.writer_username, unifi.writer_password DNS_CONTROL_UNIFI__WRITER_USERNAME, DNS_CONTROL_UNIFI__WRITER_PASSWORD Required when unifi_alias.enabled is true
unifi_alias.enabled DNS_CONTROL_UNIFI_ALIAS__ENABLED false
unifi_alias.interval DNS_CONTROL_UNIFI_ALIAS__INTERVAL 15m, from 1m to 1h

The subcommands zones and entries load the same configuration, so they run inside the pod with kubectl exec.

The subcommand dns-control webhook has its own configuration, with the prefix DNS_CONTROL_WEBHOOK_ and the same rule and the same strict decode. It has no file layer.

Key Variable Default and limits
addr DNS_CONTROL_WEBHOOK_ADDR 127.0.0.1:8888, a loopback address only
health_addr DNS_CONTROL_WEBHOOK_HEALTH_ADDR :8080
dns_control_url DNS_CONTROL_WEBHOOK_DNS_CONTROL_URL Required
oidc_token_url DNS_CONTROL_WEBHOOK_OIDC_TOKEN_URL Required
oidc_client_id DNS_CONTROL_WEBHOOK_OIDC_CLIENT_ID dns-control-external-dns
oidc_client_secret DNS_CONTROL_WEBHOOK_OIDC_CLIENT_SECRET Required
extra_ca_file DNS_CONTROL_WEBHOOK_EXTRA_CA_FILE Empty, which means the system roots only
domain_filter DNS_CONTROL_WEBHOOK_DOMAIN_FILTER fzymgc.house, a comma-separated list
domain_exclude DNS_CONTROL_WEBHOOK_DOMAIN_EXCLUDE lan.fzymgc.house, a comma-separated list
budget DNS_CONTROL_WEBHOOK_BUDGET 12s, greater than zero

dns-control checks the Keycloak sign-in itself. No proxy sits in front of it. The browser signs in at https://id.fzymgc.house with the authorization code flow and PKCE. PKCE is a proof that the code goes back to the client that asked for it. The browser then holds only a session cookie.

The Keycloak client dns-control has four client roles for people. One group grants each role:

Group Role What the role allows today
dns-readers dns-reader Every screen and every /api/* route
dns-writers dns-writer The changes of curated entries: plan, apply and project
dns-operators dns-operator The operations actions. See Who can run what
dns-admins dns-admin Zones, their grants and the control lane. See Zones and record sets

dns-control reads roles from resource_access["dns-control"].roles in the access token only. A realm role or the role of another client does not count. The access token must also carry dns-control in its aud claim. The audience mapper in tf/keycloak/dns_control.tf adds it.

After the sign-in, the sidebar lists the screens: Search, Query log, Health, Changes and Rules. The foot of the sidebar shows the name of the person, the roles from the token, Keys and Sign out. At 1024px and wider, the sidebar shows its labels, and its toggle button narrows it to icons. From 640px to 1023px, it shows icons only, and each label is the accessible name of its icon. Below 640px, the menu button at the top of the screen opens the sidebar.

Terraform writes the client secret to fzymgc-house/cluster/dns-control/oidc. The ExternalSecret dns-control reads oidc_client_secret from that path. Do not put this secret by hand. The next apply of the main-cluster-keycloak workspace replaces it.

On the first rollout, the ExternalSecret dns-control can report SecretSynced before the Keycloak apply writes the OIDC secret. The pod then waits in CreateContainerConfigError. The next refresh of the ExternalSecret, 5 minutes later at most, fills the key. The pod then starts with no restart.

To give a person access:

  1. Add a keycloak_user_groups resource for the person to tf/keycloak/dns_control.tf, in the shape of operator_dns_control. Use the group keycloak_group.dns_control["reader"]. For the operations actions, add keycloak_group.dns_control["operator"] too. For the zone and record set changes, add keycloak_group.dns_control["admin"].
  2. Merge the pull request. The main-cluster-keycloak workspace applies it.
  3. Tell the person to sign out of dns-control and sign in again. A token carries the roles that were current when Keycloak issued it.

If a person sees “You do not have access to dns-control.”, the person is not in dns-readers.

Each tab of the app holds one event stream on /api/events, in the format of server-sent events. An event names what changed, and the screen that shows it reads its own route again. The stream checks the session every 15 seconds and ends when the session ends. dns-control keeps the sessions in the memory of its process, so a restart of dns-control signs every tab out. The tab then shows the sign-in link within 30 seconds.

Every screen takes the keys in this table. A key does nothing while a field, a menu or a dialog has the focus. No key uses Ctrl, Cmd or Alt, so the keys of the browser stay as they are. Ctrl+B or Cmd+B still narrows the sidebar.

Key What it does
g then s Opens Search
g then q Opens Query log
g then h Opens Health
g then c Opens Changes
g then r Opens Rules
/ Moves the focus to the search field of Search, Query log or Rules
n Moves the focus to the name field of the add form on Rules, for a person with dns-writer
? Opens the list of keys
Escape Closes the list of keys, or moves the focus out of a search field and keeps its text

After g, press the second key within 1.5 seconds. Any other key ends the wait.

The Keys button in the foot of the sidebar opens the same list. The list holds a switch that turns every key off for this browser, as WCAG 2.1 success criterion 2.1.4 requires for a key of one character. The browser keeps the value in localStorage under dns-control.keys. If the browser blocks localStorage, the keys stay on. The Keys button opens the list also while the keys are off.

The Search screen finds record sets in the lineage index by a name pattern. The pattern input has a toggle with two modes, glob and regex. Glob is the default.

Mode Example How it matches
glob *.lab.fzymgc.house. * matches any run of characters, dots included. ? matches one character. The pattern matches the whole name.
regex ^(nas|calgary)\. Go regular expression syntax, not case-sensitive. The pattern matches anywhere in the name.

Every name in the index ends with a dot. In glob mode, the screen adds the trailing dot to a pattern that ends with neither . nor *, so *.lab.fzymgc.house searches for *.lab.fzymgc.house.. In regex mode, the screen sends the pattern as typed. Anchor a regex with ^ and $, and escape a dot as \.. The input colors the operators of the pattern while you type. In regex mode, the input turns red and reads “Pattern does not compile” when the browser cannot compile the pattern. The server compiles the pattern with Go regexp and refuses a pattern that does not compile with invalid_filter for the field name.

The Type, Zone, Writer and State headers of the table each open a menu of the values that the last answer offered. A choice sets the filter and searches at once. A set filter fills its header icon and shows as a chip above the table. Select the chip to remove that filter. Below 768px, each record set is a card, and the four filters open from the Filters button.

The URL holds the pattern, its mode and every filter, so Back, Forward and a copied link open the same view. A regex search adds mode=regex to the URL.

The search route takes the same mode parameter. A request with no mode is a glob search. A mode other than glob or regex answers invalid_filter for the field mode:

Terminal window
curl -fsS -H "Authorization: Bearer $TOKEN" 'https://dns-control.fzymgc.house/api/search?name=%5Enas%5C.&mode=regex&limit=5'

The status bar is the last row of the list. It shows the count, the page range, the pagination and the age of the index and of each node read. Changes and Query log have the same bar.

The Query log screen shows the queries that the resolver pair answered, newest first.

The name pattern takes the glob and regex modes of Search. In glob mode, the screen adds the trailing dot that the resolver logs with every name. The column headers hold the other filters: Time, Result, Resolver and Node, Client, and Type. Each choice reads the log at once. A set filter marks its header and shows as a chip above the table. Select the chip to remove the filter, or select Clear all. Below 768px, the filters open from the Filters button.

The time range is a duration in whole minutes from 1m to 24h, such as 90m or 2h30m. The default is 5m. The Time menu offers 5m, 15m, 1h, 6h and 24h, and a field for any other duration. A link from Name detail or from an MCP tool escapes the glob operators of the name, so a wildcard name such as *.lab.fzymgc.house. shows its own queries only.

The first read asks for about two screens of rows. When you scroll near the end of the rows, the screen reads the next older rows, until the start of the range. The status bar then says “More rows load as you scroll” while the range holds older rows. The list holds at most 1,000 rows. At that count the status bar names the bound, and a filter or a shorter range shows older rows.

New rows of the current filters appear above the rows on screen with no click. While at least one tab holds an event stream, dns-control reads the time of the newest query log row from ClickStack every 15 seconds. When that time advances, it sends a querylog event. The screen reads the rows after its newest row 30 seconds after the first event, so two of those reads are at least 30 seconds apart. If you scrolled down, the rows on screen stay where they are. If more rows are new than one read holds, the new rows replace the list. Rows older than the start of the range leave the list. In system.query_log of ClickHouse, each read of the newest row carries the log_comment dns_control_querylog_newest. Select Pause to keep the rows as they are. The status bar then reads Paused. Select Resume to read the new rows once and follow them again. A submit of the filters reads the log from its newest row, also while the screen is paused.

The Client cell names the client that asked. It shows the first label of the name of the client and then its address, on one line. The name is the PTR record that pdns serves for the address. If pdns holds no PTR, the cell shows the first label of the alias that UniFi holds for the device. With neither, the cell shows the address alone. Guest and Telework clients have no PTR, because Kea sends no DNS update on those legs.

Rest the pointer on the Client cell, or move focus to it with Tab, to open its card. The card shows these values:

  • The full name, with a badge for each writer of its PTR. A Kea D2 badge marks a name that the device chose when it asked for its DHCP lease.
  • The address.
  • The device that the last good read of the UniFi client table holds at the address: the alias, the hostname, the vendor, the MAC address, the network, and wired or Wi-Fi. The card labels the hostname “Hostname (from the device)”, because the device reports it.
  • The time of that read. The device is the one that held the address at that time, which is not always the one that sent the query.

The device values come from the observed AAAA sweep. With observed_aaaa.enabled false, the card shows the name and the address only.

Each row names its resolver process by a neutral name: dns-resolver is the process of the trusted legs and the tailnet, and dns-resolver-public is the process of Guest and Telework. The Resolver filter takes these two names. A row that a policy zone decided carries a policy object with the zone, the trigger that matched and the action: nxdomain, nodata or passthru. A row that no policy zone decided has policy null. The Result label reads Blocked for nxdomain and nodata, Allowed for passthru, Resolved for NOERROR with no policy, and else the response code. The Result filter takes blocked, passthru or a response code such as SERVFAIL.

Each card of a row opens from its own cell, so one card shows at a time. Rest the pointer on the Result label to open its card with the response code and, for a row with a policy, the zone, the trigger and the action. A person with dns-operator also finds Wipe from cache in that card, see Wipe one name. A name that ends in an ellipsis opens a card with the full name.

dns-control serves MCP on a second listener, port 8081. MCP (Model Context Protocol) is the protocol that AI agents use to call tools. Only the LiteLLM proxy pods reach port 8081. The network policy dns-control-default admits the litellm namespace on TCP 8081, and Traefik has no rule on that port. The UI host does not serve MCP. https://dns-control.fzymgc.house/mcp answers 404.

Agents call https://llm.fzymgc.house/dns/mcp. The LiteLLM route dns relays each call to http://dns-control.dns-control.svc.cluster.local:8081/mcp. Each call carries two headers:

Header Carries Who reads it
x-litellm-api-key: Bearer <virtual key> The LiteLLM virtual key LiteLLM, for admission, limits and spend
Authorization: Bearer <credential> A Keycloak access token or a static key dns-control

LiteLLM relays Authorization unchanged and checks nothing in it. Never put the virtual key in Authorization. dns-control refuses a bearer that starts with sk- and logs the prefix only, but the key already left LiteLLM. If a virtual key reached dns-control, rotate it. The route table and the two-header contract of every route are in docs/operations/mcp-gateway-clients.md.

Three kinds of caller exist:

Caller Credential Where the credential lives
A person in Claude Code A Keycloak access token from the public client mcp-dns, with PKCE and the scope mcp-dns Claude Code holds the token. The Keycloak group dns-readers grants the role
The triage agent A client_credentials token from the Keycloak client dns-agent-triage The client id and secret at secret/fzymgc-house/cluster/dns-agent-triage/oidc. The virtual key at secret/fzymgc-house/cluster/litellm-keys/dns-agent-triage, property api_key
A service that is not an agent A static key that starts with dnsk_ The digest in the property mcp_static_key_hashes at secret/fzymgc-house/cluster/dns-control. The caller holds the plaintext

A client_credentials token is a token that a client gets with its own id and secret, with no person in the flow. Terraform creates both Keycloak clients, in tf/keycloak/mcp_dns.tf and tf/keycloak/dns_agent_triage.tf, and writes the secret of dns-agent-triage to Vault.

By default, Claude Code does not use the client mcp-dns. It registers its own client through Keycloak DCR (dynamic client registration), and Keycloak creates that client with full scope turned off. A role then reaches the token only through a role scope mapping of a client scope. The scope mcp-dns therefore maps dns-reader, dns-writer and dns-operator and emits preferred_username. The clients mcp-dns and dns-agent-triage have full scope turned off too. A token carries only the mapped roles that the user or the service account holds.

Every caller needs the dns-reader role. dns-control verifies each Keycloak token itself. It makes sure that the token has:

  1. The issuer https://id.fzymgc.house/realms/fzymgc and a valid signature from the realm keys.
  2. The audience https://llm.fzymgc.house/dns/mcp. The scope mcp-dns adds it.
  3. The dns-reader role in resource_access["dns-control"].roles.
  4. An expiry in the future.

A token for another audience gets 401. A static key gives the identity key:<name> with the dns-reader role. dns-control reads no forwarded identity header. The identity comes from the verified credential only.

An MCP client finds Keycloak through the document at https://llm.fzymgc.house/.well-known/oauth-protected-resource/dns/mcp. dns-control serves the document on port 8081, and LiteLLM relays it. The document names Keycloak as the authorization server and lists the scopes mcp-dns and offline_access.

dns-control has eleven tools. The six tools in this table are read-only. The five operations tools are in The MCP tools. No tool changes a record. LiteLLM joins the route name and the tool name with -, so a client sees dns-dns_lookup_name and so on.

Tool Relayed name What it answers
dns_lookup_name dns-dns_lookup_name One name: its RRsets, writers, lineage state, node agreement and what each leg sees
dns_search_records dns-dns_search_records The RRsets that match a name pattern, zone, writer, type or lineage state
dns_query_log dns-dns_query_log The query log of the resolver pair, newest first, by name, client and time span, with the policy that decided each row. Each row names its client: the PTR name of the address with the writers of that PTR, and the UniFi device from the station copy
dns_get_health dns-dns_get_health The health of the resolver pair and the freshness of the index, the node worker readings, the VIP holder, the recent actions of each node, the audit status and the observed AAAA sweep
dns_list_drift dns-dns_list_drift The RRsets in an orphan, conflict, content drift or dangling state
dns_explain_answer dns-dns_explain_answer Why a client got the answer it got for a name, step by step, with the evidence

Each answer is JSON Lines text in the content of the tool result. A list answer is one header line, then one JSON line for each row. The header line holds the total and the paging values. A single answer is one JSON line. dns-control removes the null and empty members. The limit argument has the default 25 and the maximum 100. Each answer carries the link to the same view in dns-control, in the member url.

dns_search_records takes an optional mode argument, glob or regex, with the default glob. It has the same meaning as the mode parameter of the search route. See Search. A regex that does not compile, or a mode other than the two, answers invalid_filter with the field name. The url of a regex answer carries mode=regex.

Each tool call writes one log line with the message mcp tool call. The line holds these fields:

Field Value
tool The tool name without the dns- prefix
sub The subject of the token, or key:<name> for a static key
azp The Keycloak client that got the token. It is empty for a static key
preferred_username The person, the service account of the agent, or the name of the static key
kind keycloak or static
args The arguments of the call, as JSON

The line never holds a token or a key. The fields are log attributes, and the row carries the trace id of the tool span. Read the calls in ClickStack:

SELECT Timestamp, LogAttributes['tool'] AS tool, LogAttributes['preferred_username'] AS caller,
LogAttributes['args'] AS args, TraceId
FROM default.otel_logs
WHERE ServiceName = 'dns-control' AND Body = 'mcp tool call'
AND Timestamp > now() - INTERVAL 1 HOUR
ORDER BY Timestamp DESC
LIMIT 100

A static key is dnsk_ and 43 base64url characters from 32 random bytes. The Vault property mcp_static_key_hashes holds one line <name>:<sha256 hex> for each key. The digest covers the whole key, the prefix included. A name has 1 to 32 characters from a-z, 0-9 and -. The pod holds the digests only. If a line is malformed, dns-control stops at start.

The plaintext of the smoke key lives beside the digests, as the property mcp_smoke_key. It never leaves Vault. The ExternalSecret dns-control names mcp_static_key_hashes only.

To add a key:

  1. Generate the key and its digest in shell variables. Replace <name> with the name of the caller.

    Terminal window
    NAME='<name>'
    KEY="dnsk_$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=')"
    HASH="$(printf '%s' "$KEY" | shasum -a 256 | cut -d' ' -f1)"
  2. Add the line to the property. vault kv patch changes the named property only. Do not use vault kv put, because it replaces every property at the path.

    Terminal window
    CUR="$(vault kv get -mount=secret -field=mcp_static_key_hashes fzymgc-house/cluster/dns-control)"
    printf '%s\n%s:%s\n' "$CUR" "$NAME" "$HASH" \
    | vault kv patch -mount=secret fzymgc-house/cluster/dns-control mcp_static_key_hashes=-
  3. Wait for the ExternalSecret to refresh, 5 minutes at most. Reloader then restarts the pod.

  4. Give the plaintext in $KEY to the caller out of band. Do not write it to a file or a ticket.

To revoke a key, remove its line:

Terminal window
vault kv get -mount=secret -field=mcp_static_key_hashes fzymgc-house/cluster/dns-control \
| rg -v '^<name>:' \
| vault kv patch -mount=secret fzymgc-house/cluster/dns-control mcp_static_key_hashes=-

After the refresh and the restart, the key gets 401.

Run these readings once after a deploy that changes the MCP path. They are not standing gates. Run them from a workstation with Vault access. The commands read each secret from Vault, so you paste no secret into the shell.

  1. Mint a token for the triage agent and read its claims. Expect aud to hold https://llm.fzymgc.house/dns/mcp and not dns-control (D-08), azp to be dns-agent-triage, and resource_access.dns-control.roles to hold dns-reader.

    Terminal window
    CSECRET="$(vault kv get -field=oidc_client_secret secret/fzymgc-house/cluster/dns-agent-triage/oidc)"
    VKEY="$(vault kv get -field=api_key secret/fzymgc-house/cluster/litellm-keys/dns-agent-triage)"
    TOKEN="$(curl -sS https://id.fzymgc.house/realms/fzymgc/protocol/openid-connect/token \
    -u "dns-agent-triage:$CSECRET" -d grant_type=client_credentials | jq -r .access_token)"
    printf '%s' "$TOKEN" | cut -d. -f2 | tr '_-' '/+' \
    | jq -R '@base64d | fromjson | {aud, azp, typ, preferred_username, resource_access}'

    Then send the token to the REST API of dns-control. Expect 401, because an MCP token does not open the REST API.

    Terminal window
    curl -sS -o /dev/null -w '%{http_code}\n' https://dns-control.fzymgc.house/api/health \
    -H "Authorization: Bearer $TOKEN"
  2. Send initialize through the route. Expect HTTP/2 200.

    Terminal window
    curl -sS -D - -o /dev/null -X POST https://llm.fzymgc.house/dns/mcp \
    -H "x-litellm-api-key: Bearer $VKEY" -H "Authorization: Bearer $TOKEN" \
    -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
    -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}' \
    | rg -i '^HTTP/|mcp-session-id'
  3. Send tools/list. Expect the eleven dns- tools. The log line of a later tool call shows azp dns-agent-triage. If the answer to initialize had an mcp-session-id header, send it on this call too.

    Terminal window
    curl -sS -X POST https://llm.fzymgc.house/dns/mcp \
    -H "x-litellm-api-key: Bearer $VKEY" -H "Authorization: Bearer $TOKEN" \
    -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
    -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | jq -r '.result.tools[].name'
  4. Send tools/list with the smoke key in place of the token. Expect the same eleven tools.

    Terminal window
    SKEY="$(vault kv get -mount=secret -field=mcp_smoke_key fzymgc-house/cluster/dns-control)"
    curl -sS -X POST https://llm.fzymgc.house/dns/mcp \
    -H "x-litellm-api-key: Bearer $VKEY" -H "Authorization: Bearer $SKEY" \
    -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
    -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | jq -r '.result.tools[].name'
  5. Add the dns entry to your Claude Code with your own virtual key. Then open /mcp in Claude Code, select dns, and sign in through Keycloak in the browser. Call dns-dns_explain_answer once.

    Terminal window
    claude mcp add --transport http --scope user --callback-port 51339 \
    dns https://llm.fzymgc.house/dns/mcp \
    --header "x-litellm-api-key: Bearer <your-virtual-key>"

If Keycloak refuses the dynamic client registration, add --client-id mcp-dns to the command. Then set oauth.scopes of the dns entry in ~/.claude.json to mcp-dns offline_access, so the token carries the audience. Do not put an Authorization header on the entry, because Claude Code then turns off its sign-in for the server.

dns-control runs five actions on the resolver nodes. An action changes the state of a node. It never changes a record.

Action What the node does Target
Flush The node wipes the root name and every name under it from the caches of both recursor processes, through the API of each process. Until 2026-10-14 a node ran Blocky, and the flush dropped the whole cache of blocky and then of blocky-public One node or both
Refresh The node worker starts dns-feeds.service and waits up to 5 minutes until the unit ends. The unit downloads every list source with a URL again, writes the feed zones, reloads both recursor processes and drops every cached answer. dns-feeds.timer also runs it once a day One node or both
Drain The node writes the drain marker and stops keepalived, so the peer takes every VIP One node
Undrain The node removes the drain marker and starts keepalived One node
Wipe Both resolver processes remove one name, or the name and every name under it, from their caches. See Wipe one name Both nodes

A person starts an action from the card of a node on the Health screen. The card has the buttons Flush cache, Refresh lists, and Drain node or Undrain node. The screen also has Flush cache on both nodes and Refresh lists on both nodes. An agent starts an action through an MCP tool, and a script starts one through the REST API. All three surfaces call one operations service, so the role check, the flush guard and the audit row exist once.

The node worker on each Pi runs the action. The node worker is a Temporal worker that polls the task queue of its own node, for example dns-node-sandiego. See the Temporal runbook, The node workers. For each action, dns-control starts the Temporal workflow ops.action on its own task queue dns-control. The workflow reads the status of the node before and after the action, and the audit row keeps both readings.

One action runs at a time. The workflow id dns-ops is the lock. A second action while one runs gets action_in_progress and a refused audit row. Each node activity fails after 10 seconds when no worker polls the queue of the node, and each action runs once with no retry. Temporal ends a run of ops.action after 40 minutes, and the end of the run releases the lock. This limit is longer than the 30 minutes for which the workflow retries the audit store and the publish.

dns-control writes the pending row and then starts the workflow. The start runs apart from the request, with its own limit of 10 seconds. If the client closes the connection, the start goes on, and the row does not become failed.

dns-control then waits at most 120 seconds for the result (ops_wait). If the wait ends first, the answer holds the pending row, and the workflow goes on. dns-control then logs no workflow result within the wait at Warn with the audit_id. ops_wait must be less than 150 seconds, the HTTP write timeout of both listeners. With a larger value, dns-control does not start.

If the workflow fails, dns-control logs the action workflow failed at Error with the audit_id. The answer is 200 with the row as it stands, and that row is often still pending. When the pod gets SIGTERM, dns-control ends every wait for a result. Each caller then gets the row as it stands.

If the start fails and Postgres fails at the same moment, the row stays pending. dns-control then logs the audit row of a lost start stays pending at Error.

The REST routes need a session cookie or a bearer token for the client dns-control.

Route Body or query
POST /api/ops/flush {"target":"sandiego","force":false}. The target is a node or both
POST /api/ops/refresh {"target":"both"}
POST /api/ops/drain {"node":"portland","expected_minutes":60}. With no expected_minutes, the drain takes the default
POST /api/ops/undrain {"node":"portland"}
POST /api/ops/wipe {"name":"www.example.com","subtree":false}. The wipe runs on both nodes and takes no target
GET /api/ops/preflight ?action=drain&target=portland. It answers the guard readings of a flush or a refresh, or the five peer checks of a drain. It needs dns-operator, and a drain or an undrain preflight also needs a person. It writes no audit row, for a refusal too

A route reads at most 4096 bytes of one JSON object and refuses an unknown field. A flush or a refresh body names target, a drain or an undrain body names node, and a wipe body names name and subtree. The caller always comes from the verified credential and never from the body.

If dns-control accepts the request, the answer is 200 with the audit row. A guard refusal and a failed node action also answer 200, and the row holds the outcome and the reason. The other answers are:

Code HTTP status Meaning Audit row
invalid_action 400 A field of the request is not valid. The answer names the field None
role_required 403 The caller does not hold dns-operator Refused
identity_not_allowed 403 The caller is not a person, and the action needs one Refused
origin_mismatch 403 The Origin header of an action POST does not equal UI_BASE_URL None
action_in_progress 409 Another action runs Refused
operations_unavailable 503 The audit store or Temporal did not answer None
internal_error 500 Any other error None

A browser request must carry an Origin header equal to UI_BASE_URL. If UI_BASE_URL is empty, dns-control refuses every POST from a session. A bearer request from a script carries no Origin. If it carries one, the value must equal UI_BASE_URL too. Each of these refusals answers origin_mismatch. Every answer of the routes carries Cache-Control: no-store.

GET /api/health and the MCP tool dns_get_health carry the operations readings too:

Field Where Value
worker Each node The newest node worker reading: keepalived state, VIP count, drain marker with overdue, the engine of the node (recursor on both nodes), one reading per resolver process in resolvers, check scripts and certificate window. answering is false when the last read failed. Then only last_good_read_at and poller carry a value
recent_actions Each node The five newest audit rows that name the node
vip_holder Top level one_holder with the node, none, split or not_read
audit_status Top level ok or audit_unavailable

dns-control reads the status of each node every 30 seconds. Health never shows a drain state or a VIP count from an older reading.

The node worker leaves the VIP count out of a reading when it cannot read the interface addresses. vips_held is then null. dns-control never reads a missing count as 0. A flush or a refresh of both nodes then refuses with vip_holder_not_one, and the drain counts that peer reading as a failed reading. If the worker cannot read the drain marker, drain is null and drain_unknown is true. The card of the node then shows No reading for the drain state. The worker reads the state of each keepalived instance from its file in /run/keepalived-state. If a file does not read, the keepalived state of an active unit is UNKNOWN. A stopped unit reads STOPPED.

Action Role Caller
Flush and refresh dns-operator A person or a client
Wipe dns-operator A person or a client
Forced flush dns-operator A person only
Drain and undrain dns-operator A person only
Read the history with dns_list_operations or on Health dns-reader Any caller

dns-control reads the kind of each caller from the verified credential:

  • person is a token that a person got through a sign-in, in the browser or in Claude Code.
  • client is a client_credentials token, for example the token of the triage agent.
  • static is a static key that starts with dnsk_.

A static key carries dns-reader only, so it gets role_required on every action. The triage service account holds dns-reader only, so it also gets role_required. A client token that holds dns-operator gets identity_not_allowed on a drain, an undrain or a forced flush. Each refusal writes an audit row with the caller: sub, azp, preferred_username and the kind.

To give a person the actions, add the person to the group dns-operators. See Sign-in and roles.

Health opens with four tiles: Node agreement, VIP holder, Node workers and Record index. Each tile shows one value and one line of detail. Its left edge has the worst tone of the facts that it sums up. A pending difference between the nodes is warn, and a settled one is destructive. The status bar under the tiles shows the age of the readings and the Refresh readings button.

Health reads /api/health when it opens and after each health, ops and index event of the event stream. dns-control sends health after each read of the node workers, every 30 seconds. It sends ops when an action ends, and index when the content of the record index changes. A hidden tab reads once when it is visible again. Refresh readings reads at once. The header shows the state of the stream: Live while it is open, and Reconnecting while it is down.

Each node card has one health badge, in one of three tones:

Tone Badge text When
ok Healthy The worker answers, both recursor processes run, and both resolver checks pass
warn Drained, No reading The node is drained, or the drain marker, a resolver process or a check has no reading
destructive Not answering, Resolver stopped, Check failing The worker does not answer, a resolver process does not run, or a check fails

The readings of a node card sit under tabs: Worker, pdns, Resolvers, LightningStream and Recent actions. Worker opens first. The action buttons sit at the foot of the card, under the tabs. Below 640px, each node card is one row with the name, the keepalived state and the health badge. A tap on the row opens the Worker readings and the action buttons.

Health reads the roles of the person from /api/me. If that request fails, Health shows no role line and no action buttons. Health sends the request again at each read of /api/health.

The Last action panel shows the newest action of the tab. If the answer holds a pending row, the action still runs. The panel then keeps the Running badge, the spinner and the running line, with Started HH:MM:SS. Running for N seconds. Every action button stays disabled. The ops event of the end of the action makes Health read again. That read finds the final row with the same id in the recent actions of a node. The panel then shows that row, and the buttons become active again.

The pending row can drop out of the five newest recent actions of every node before it finishes. The buttons then stay disabled until you reload the page. This is a known limit.

A guard refusal of a flush runs on no node. The panel then shows the reason and the button Review forced flush. The button opens the forced flush dialog with the node and the ratio of the guard line.

If dns-control answers origin_mismatch, the dialog shows the error “dns-control refused the action from this page address.” Its text tells the person to open dns-control again from its configured address.

Before a flush, dns-control reads the upstream error ratio of each target node from ClickStack. The ratio is dns_resolver_errors_total over dns_resolver_responses_total in the last 15 minutes, for the processes dns-resolver and dns-resolver-public. The collector on each node copies the two counters of each recursor process under these names. These are the series of the alert DNS public resolver failing (Guest/Telework). The worst process of a node decides.

Reading of a target node Without force With force, by a person
Below 0.05 The flush runs The flush runs with forced true and no override
0.05 or above Refused with guard_above_limit The flush runs with the override guard_above_limit
No reading Refused with guard_no_reading The flush runs with the override guard_no_reading

The comparison is inclusive. A ratio of exactly 0.05 refuses the flush, and the alert fires at exactly 0.05 too. The threshold comes from FLUSH_GUARD_THRESHOLD, with the default 0.05. A flush clears the cache of both processes, so a reading needs responses in the window from both dns-resolver and dns-resolver-public. A node has no reading when one of its processes has no responses in the window, or when ClickStack does not answer.

A flush with no reading runs only when a person forces it. On the Health screen, a forced flush takes a separate typed confirmation. An agent never forces. A client token with force true gets identity_not_allowed. The audit row keeps the guard reading of each node, forced and the override.

A refresh reads the same guard, because the feed unit drops every cached answer as a flush does. A refresh takes no force. It runs only when every target node reads below the limit, and it is refused with guard_above_limit or guard_no_reading otherwise, for a person too. To load the feeds while the guard refuses, run sudo systemctl start dns-feeds.service on the node.

In the result of a flush, the audit row holds the count of entries that each process removed, as process and count.

A flush or a refresh with the target both runs on one node at a time. The node that does not hold the VIPs goes first, and the VIP holder goes second. Each node gets its own readings before and after the action, and the result reports each node on its own.

If the first node does not succeed, dns-control does not touch the second node. A first node that gives no result counts as a node that did not succeed. The entry of the second node reads refused with previous_node_failed. If the VIPs are split across the nodes, or no node holds them, the action refuses with vip_holder_not_one and runs on no node. If a node does not answer, the action refuses with node_not_answering.

A wipe removes one name from the record cache, the negative cache and the packet cache of both resolver processes on both nodes. With subtree, it also removes every name under the name. The next query for the name goes upstream again. Use it when a client holds a stale answer for one name and the whole cache must stay. The MCP tool dns_wipe_name, the route POST /api/ops/wipe and the Result card of the Query log start a wipe.

A wipe refuses the root name ., a name that holds * and a name that is not a valid DNS name. The refusal is invalid_action on the field name, and no audit row is written. The flush covers the root name, and a wildcard is a pattern of the Query log, not a name. dns-control stores the name in lower case with a trailing dot.

A wipe runs on both nodes, the node that holds no VIP first, like a flush of both nodes. It reads no flush guard and takes no force. An agent identity with dns-operator can run it.

Every wipe writes one audit row with the action wipe, the columns wipe_name and wipe_subtree, and one result per node. The result of a node holds wiped: the number of entries that each process removed, as process (main or public) and count. A second wipe of the same name counts zero on each process. The audit row of a flush holds the same counts, because a flush is a wipe of the root name with the subtree. The guard, force and the role of the flush stay as they are.

On the Query log, a person with dns-operator opens the Result card of a row and selects Wipe from cache. The dialog names the query name of the row and the subtree choice. After the confirm, it shows the outcome and the count of each process on each node, or the reason of a refusal. A person without dns-operator sees no wipe control.

A drain takes a node out of service. The node writes the drain marker /var/lib/dns-worker/drained and stops keepalived, and the peer takes all seven VIPs. keepalived stays stopped until an undrain, across a reboot too. The resolver runbook docs/operations/dns.md describes the marker and the manual undrain, in Drain and undrain through dns-control.

The node checks its peer at the time of the drain. It refuses the drain with peer_reading_failed and the id of a check that fails:

Check id The peer passes when
house_check keepalived-check-house.sh exits 0
public_check keepalived-check-public.sh exits 0
keepalived_active keepalived is active
keepalived_not_fault The summary state and every instance state are MASTER or BACKUP. A state in FAULT, UNKNOWN or a value that the node does not know fails the check
peer_not_drained The peer is not drained. If the worker of the peer cannot read its drain marker, the check fails with the value unknown

A peer reading that is too old refuses with peer_reading_stale, and a node that is drained already refuses with already_drained. A peer reading that names no node, or names the node itself, refuses with peer_reading_failed.

The drain dialog on Health shows the five checks before the person confirms the drain. The person types the node name to confirm. The preflight reports the node as drained only when the node answered its last status reading.

After the stop, the workflow reads the peer every 2 seconds, 10 times at most. If the peer does not hold all its VIPs, the workflow undrains the node and records failed with vips_not_moved. If the workflow is cancelled during these reads, it still runs the undrain, the last reading of the node, the finish of the audit row and the publish.

The drain activity can give no result, because it failed or ran out of time. The node can still have written the marker or stopped keepalived. The workflow then reads the node and then the peer, and records the VIP count of the peer when the reading has one. If the node shows a drain marker or keepalived is not active, the workflow runs an undrain at once. The detail of the node entry is then rollback undrain ran or rollback undrain failed. The outcome stays failed with node_not_answering.

On the node, a failed stop of keepalived removes the marker again. If the activity ends during the stop, the stop can still complete, so the node keeps the marker. An undrain clears that state.

Each drain has an expected duration. The default is 30 minutes (DRAIN_DEFAULT), and it is also the shortest value. The longest value is 1440 minutes. DRAIN_DEFAULT must lie between 1 and 1440 minutes. With another value, dns-control does not start. When a drain passes its expected duration, dns-control shows it in four places:

  • The card of the node on Health shows a warning, Drain overdue.
  • The drain reading in /api/health and dns_get_health has overdue true.
  • dns-control publishes the event dns.ops.drain.overdue once for the drain.
  • The node worker sets the metric dns_worker_drain_overdue to 1.

An undrain removes the drain marker, starts keepalived and makes sure that keepalived is active. The node refuses the undrain with not_drained only when it has no marker and keepalived is active. With no marker and keepalived not active, the undrain starts keepalived. An undrain therefore finishes a drain or an undrain that stopped half way. If the start of keepalived fails, run the undrain again.

The detail of a failed node action names the step and the cause:

  • A command that exited with an error reads like keepalived stop: exit 1: <standard error>.
  • A command that did not run to its end reads like keepalived stop: <error>.
  • A failed call to the API of a recursor process names the process that failed, main or public, and the error.
  • A failed read, write or remove of the drain marker carries the error text.

The worker keeps the last 512 bytes of standard error at most, and it clips the whole detail to 512 characters.

Each failed or refused action also writes the log line node action did not succeed at WARN on the node. The line holds action, node, outcome, reason, failed_check and detail. Read it in ClickStack with the journal query in A node worker does not answer.

Every action attempt writes one row in the table ops_audit of the database dns_control. The database is on the CNPG cluster main in the namespace postgres. A success, a failure, a guard refusal, a precondition refusal and a role refusal each write a row. A read writes no row.

A row is pending while its workflow runs. The workflow then writes the outcome: succeeded, failed or refused. The trigger ops_audit_append_only refuses every delete, and every change to a row that is not pending. dns-control does not prune the rows.

Three activities in dns-control load the request of the row, finish the row and publish the event. Each retries with no limit on the attempts for at most 30 minutes. The pause between two attempts is 30 seconds at most. A fault of the audit store is retried. Only a missing row, or a row that finished with another outcome, ends the activity at once with not_pending. If Postgres answers again within 30 minutes, the row of an action that ran on a node still finishes.

dns-control connects as the Postgres user dns_control through DATABASE_URL. The password is in secret/fzymgc-house/cluster/postgres/users/main-dns-control. The ExternalSecret dns-control-db in argocd/app-configs/dns-control/external-secret-db.yaml renders the URL from that path with sslmode=verify-full and sslrootcert=/etc/fzymgc-ca/ca.crt. The CNPG server presents a chain of three certificates, and that file holds the top one, fzymgc Intermediate CA1, so the client verifies the chain and the name main-rw.postgres.svc.cluster.local. At start, dns-control runs its migrations and tries again every 10 seconds until they succeed. Until then, every action answers operations_unavailable.

Read the last rows on the CNPG primary:

Terminal window
P=$(kubectl get pod -n postgres -l cnpg.io/cluster=main,cnpg.io/instanceRole=primary -o name | head -1)
kubectl exec -n postgres $P -c postgres -- psql -U postgres -d dns_control -c \
"SELECT started_at, action, target, outcome, reason, forced, override, caller_username, caller_kind, surface FROM ops_audit ORDER BY started_at DESC LIMIT 20"

Each write of a row also sends one OTel log event, dns_control.ops_audit, with the same audit id. An accepted action writes twice, once for the pending row and once for the outcome. Read the copy in ClickStack:

SELECT Timestamp,
LogAttributes['dns_control.action'] AS action,
LogAttributes['dns_control.outcome'] AS outcome,
LogAttributes['dns_control.reason'] AS reason,
LogAttributes['dns_control.caller_username'] AS caller
FROM default.otel_logs
WHERE ServiceName = 'dns-control' AND EventName = 'dns_control.ops_audit'
AND Timestamp > now() - INTERVAL 1 DAY
ORDER BY Timestamp DESC
LIMIT 50

default.otel_logs keeps a row for 14 days (see ClickStack). The Postgres table is the record, and the ClickStack copy is for search and charts.

dns-control publishes events on the JetStream stream DNS_EVENTS in the NATS account DNS. The stream carries events only. The audit store is the record.

Subject When
dns.ops.<action>.<outcome>, for example dns.ops.flush.succeeded The workflow finished the audit row of the action
dns.ops.drain.overdue A drain passed its expected duration

The header Nats-Msg-Id of an action event is the audit id. For an overdue drain it is drain-overdue:<node>:<drain start in unix seconds>. The stream drops a repeat of the same id inside its duplicate window of 2 minutes. The payload is JSON with audit_id, action, target_nodes, outcome, reason, surface, caller_username, caller_azp and at.

A refusal that dns-control writes before the workflow starts gets no event. This covers a role refusal, an identity refusal, a guard refusal and action_in_progress. The workflow retries a publish for at most 30 minutes. After that, it logs the failure and keeps the outcome. If NATS is down, the actions still run. A consumer that misses an event reads the audit store. The stream and the credentials of dns-control are in the NATS runbook, NATS.

Tool Role Caller destructiveHint
dns_flush_cache dns-operator A person or a client. force needs a person false
dns_refresh_lists dns-operator A person or a client false
dns_drain_node dns-operator A person only true
dns_undrain_node dns-operator A person only false
dns_wipe_name dns-operator A person or a client false
dns_list_operations dns-reader Any caller Read-only

The five action tools call the same operations service as the REST routes, with the surface mcp. dns_wipe_name takes name and subtree, see Wipe one name. Each action tool answers one JSON line: the audit row with the readings before and after. A refusal answers the code and the message of the REST answer. The descriptions of dns_drain_node and dns_undrain_node name the refusals: a static key gets role_required, and a client token gets identity_not_allowed. The four action tools other than the drain set destructiveHint to false on purpose, because MCP reads a missing value as true.

dns_list_operations answers the audit rows as JSON Lines, newest first. It filters by node and by action, wipe included. The limit argument has the default 25 and the maximum 100.

Every action answers action_in_progress, and no action finishes. An action normally ends within a few minutes. Temporal ends a run of dns-ops after 40 minutes, and the end releases the lock. Follow these steps only when a workflow dns-ops runs for more than 40 minutes.

  1. Read the workflow in the admin tools pod:

    Terminal window
    kubectl exec -n temporal deploy/temporal-admintools -- \
    temporal workflow describe --workflow-id dns-ops --namespace dns-control
  2. Read the node readings on Health, or on the nodes. Make sure that no action runs on a node. A drained node has the marker, and a drain that runs has keepalived stopping or stopped:

    Terminal window
    scripts/dns-adhoc.sh dns_hosts -m ansible.builtin.stat -a 'path=/var/lib/dns-worker/drained'
    scripts/dns-adhoc.sh dns_hosts -m ansible.builtin.command -a 'systemctl is-active keepalived'
  3. Only when the readings show that no action runs, end the workflow with a reason that names you and the cause:

    Terminal window
    kubectl exec -n temporal deploy/temporal-admintools -- \
    temporal workflow terminate --workflow-id dns-ops --namespace dns-control \
    --reason "<your name>: stuck action lock, no action running"

The audit row of the terminated action stays pending. A run that Temporal ends at the limit also leaves its row pending. Do not change it by hand. The pending row is the record that the action did not finish. If a node shows a drain that nobody wanted, undrain it from Health.

Every action answers operations_unavailable, and Health shows audit_unavailable. The read screens and the read tools keep serving.

  1. Read the dns-control log lines in ClickStack for the Postgres error, with the query in When a screen shows a data alert.

  2. Make sure that the CNPG cluster and the database are ready:

    Terminal window
    kubectl get cluster main -n postgres
    kubectl get database dns-control -n postgres
  3. If the error is an authentication error, make sure that the managed role dns_control and the variable DATABASE_URL of dns-control read the same password from secret/fzymgc-house/cluster/postgres/users/main-dns-control.

When Postgres answers again, dns-control connects with no restart.

The card of the node on Health shows that the worker does not answer, and the actions on that node refuse with node_not_answering.

  1. Read the poller of the node queue in the admin tools pod. The output lists a poller when the worker polls:

    Terminal window
    kubectl exec -n temporal deploy/temporal-admintools -- \
    temporal task-queue describe --task-queue dns-node-<node> --task-queue-type activity -n dns-control
  2. Read the log of the worker and the journal of its certificate unit in ClickStack. The worker sends its log over OTLP. The collector reads the journal of dns-node-worker-pki.service:

    SELECT Timestamp, ResourceAttributes['host.name'] AS node, ServiceName,
    LogAttributes['_SYSTEMD_UNIT'] AS unit, Body
    FROM default.otel_logs
    WHERE (ServiceName = 'dns-node-worker'
    OR (ServiceName = 'otelcol-dns' AND LogAttributes['_SYSTEMD_UNIT'] = 'dns-node-worker-pki.service'))
    AND Timestamp > now() - INTERVAL 1 HOUR
    ORDER BY Timestamp DESC
    LIMIT 100
  3. Read the state of both units on the Pi:

    Terminal window
    scripts/dns-adhoc.sh <node> -m ansible.builtin.command -a 'systemctl status dns-node-worker.service dns-node-worker-pki.service --no-pager'
  4. Read the decision log of the gateway. OPA writes each decision to the log of its container. A decision names the principal, the method, the namespace, the task queue and the result. It carries no other part of the request body:

    Terminal window
    kubectl logs -n temporal deploy/temporal-node-gateway -c opa --tail=100
    kubectl logs -n temporal deploy/temporal-node-gateway -c envoy --tail=100
  5. If the certificate expired, Health shows its window and the metric dns_worker_client_cert_not_after_seconds in default.otel_metrics_gauge shows the end. Vault Agent in dns-node-worker-pki.service renews it. The worker loads the new bundle at its next handshake with no restart.

  6. If the Envoy log shows a failed handshake, the client certificate can lack the URI SAN of the node. Follow Roll out the node identity.

  7. If the journal of the worker shows vips[<i>]: "<address>" repeats vips[<j>], the worker configuration names one VIP twice, also in the IPv4-mapped form. The worker does not start with that configuration. Correct vip in dns_vlan_interfaces and converge the node.

A node worker that does not answer does not change DNS. The resolvers answer with or without it.

dns-control holds the curated host entries. An entry is one device or service: its key, its hostname, its IPv4 address, an optional IPv6 address, the PTR flag, its aliases and an optional MAC. The entry is the truth. The pdns records, the Kea reservation and the export are projections of it. Sean decided this model on 2026-09-29, after issue #2527. Terraform no longer applies a curated record, so no second applier can put an old value back.

A change runs in two steps. Plan stores the change with a diff and writes nothing. Apply writes the entry and starts one workflow. The plan expires after 30 minutes. A repeated apply of one plan changes nothing and returns the same change.

Step Route Tool Role
Plan a create, a change or a delete POST /api/changes/plan dns_create_entry, dns_change_entry, dns_delete_entry dns-writer
Apply a plan POST /api/changes/apply dns_apply_change dns-writer
Project every entry unchanged POST /api/changes/project dns_project_entries dns-writer
Read the entries GET /api/entries, GET /api/entries/{key} dns_list_entries dns-reader
Read the changes GET /api/changes, GET /api/changes/{id} dns_list_changes, dns_get_change dns-reader

The plan body carries op, key and, for a create or a change, entry with hostname, address, address6, ptr, aliases, mac and comment. The apply body carries plan_id. The UI has the same flow on the Changes screen and on the Name screen of a curated name.

The write policy refuses a name outside the apex zone, a name in lan.fzymgc.house, a name of the reserved infrastructure families (dns.<leg>, <node>.<leg> and <node> under the apex), and a name that external-dns or Kea D2 owns. It also refuses a name that a record set of the store holds, with the reason policy_other_lane. The answer is write_policy with the name. An entry with a MAC must sit on a trusted leg, or the answer is write_policy too.

Apply writes the entry row and a pending change row in one transaction, then starts the Temporal workflow change.apply on the task queue dns-control under the workflow id dns-change-<id>. The pending change row is the lock, so one change runs at a time. The unique index change_audit_one_pending allows one pending row only. A second apply while a change is pending gets change_in_progress and writes nothing.

If the workflow of a pending change did not start, for example because Temporal was down, the next apply starts it and gets change_in_progress. Send that apply again when the pending change is final.

The workflow runs these steps in order:

  1. pdns_write: replace each RRset of the entry on the writer node with its records and the comment writer=dns-control change=<id>. For a delete, clear the comment and then remove the RRset. LightningStream replicates the write to the other node.
  2. reservation_write, when the change adds, moves or removes a reservation: send the full reservation set of every trusted leg to the node worker of the node that holds no VIP, then to the other node. The worker writes one file per leg under /var/lib/dns-worker/kea/ and runs config-reload on Kea. A node that fails stops the write to the next node.
  3. Finish the change row with the outcome and each step.
  4. Start the workflow change.export for the change and publish the event dns.change.<op>.<outcome>.

The export workflow writes entries.json, the whole entry set, zones.json, every zone and record set, and one audit object audit/<time>-<id>.json to the NAS bucket and to the B2 bucket. Each sink has its own retries for up to seven days. A sink that already holds the same bytes of an object does not get that object again. The change row shows exported when both sinks hold it, and pending: <sink> until then. The export runs as its own workflow, so a sink that is down does not hold the change lock.

The lineage index reads the entry, zone and record set stores. A live RRset that the store projects is attributed to dns-control. A record set of the external-dns lane is the one exception: it reads as external-dns and carries the mark of dns-control. A live RRset whose content differs from its row reads content_drift, and a row that pdns does not serve reads as dangling. The drift comparison of an SOA leaves out the serial. dns_list_drift and the Health screen show both. A hand edit of pdns therefore shows up on the next refresh.

The entry set can be read back from either sink and imported into an empty store:

Terminal window
kubectl exec -n dns-control deploy/dns-control -- dns-control entries export --format json > entries.json
kubectl exec -n dns-control deploy/dns-control -- dns-control entries export --format hosts-yml > hosts.yml
kubectl exec -i -n dns-control deploy/dns-control -- dns-control entries import --from /dev/stdin < hosts.yml

entries import reads the hosts.yml form, the form that tf/dns-records/hosts.yml had, and writes one change per entry in the surface cli. It starts no workflow. After an import, run POST /api/changes/project with an empty keys list, or the tool dns_project_entries, to project the set onto pdns, Kea and the sinks. A projection applies each entry unchanged, one at a time, and reloads Kea on both nodes for each entry with a MAC. entries import --dry-run prints the parsed entries in hosts.yml form and writes nothing. entries export --format tf-import prints one terraform import line per record, which is the rollback of the Terraform handoff.

dns-control holds the list entries of the resolver pair. A list entry is a block entry or an allow entry for one name. A block entry makes the trusted resolver process answer NXDOMAIN for the name. An allow entry lets the name resolve when a public list blocks it. An entry with no scope applies to every client of the trusted legs and the tailnet. An entry with a scope applies to the members of that scope only, see Scopes. No entry applies on Guest or Telework, which the public process serves.

The entries reach the nodes as response policy zones (RPZ): zones with one rule per name, which pdns-auth serves and the main recursor process of each node loads, see The list entries of dns-control. The entries with no scope form the zone operator.rpz.internal., and the entries of a scope form the zone <scope>.scope.rpz.internal.. A block entry is an NXDOMAIN rule, and an allow entry is a passthru rule. A plain entry and a wildcard entry for one name give one rule each, so a plain entry wins over the base rule of a wildcard for the same name. Until 2026-10-14 the nodes ran Blocky, and the entries reached a node as two operator list files that the node worker wrote. The retire play of the DNS runbook removes those files.

Name What it matches
example.org That name only
*.example.org example.org and every name under it

A name holds one entry in each scope and one entry with no scope. To change the kind or the reason of an entry, delete it and create it again. The UI calls the screen Rules.

Rules reads every entry in one answer, and the browser searches, sorts and filters the list. The search field finds the text in the name, the scope, the reason or the actor, in any case. Each column header sorts its column, and a second selection reverses the order. The Kind, By and Time headers also hold a filter on the kind, the actor, and the age of the last change. The Scope column shows the scope of each entry, or No scope. Below 768px, the sort and the filters are in the sheet behind the Filters button. The Scopes section above the search lists every scope with its name, order, members, actor and time.

The address holds the view in five keys: q, kind, actor, since and sort. A reload or a copied link opens the same view. A sort or a filter adds a history entry, so Back restores the view before it. Typing in the search field adds no history entry.

Key Value
q The search text
kind block or allow
actor The name of the actor
since 24h, 7d or 30d
sort name, kind, scope, reason, actor or time, with a leading - for the reverse order. The default is name

For example, #/rules?q=ads&kind=block&sort=-time shows the block entries that hold ads, newest first. An unknown value of kind or since reads as no filter, and an unknown sort reads as name.

Step Route Tool Role
Plan a create or a delete POST /api/list-entries/plan dns_create_list_entry, dns_delete_list_entry dns-writer
Apply a plan POST /api/changes/apply dns_apply_change dns-writer
Write every entry to both nodes again POST /api/changes/project with subject: block dns_project_entries with the subject block dns-writer
Read the entries GET /api/list-entries dns_list_list_entries dns-reader
Plan a create, a change or a delete of a scope POST /api/scopes/plan dns_create_scope, dns_change_scope, dns_delete_scope dns-writer
Read the scopes GET /api/scopes dns_list_scopes dns-reader

The plan body carries op (create or delete) and name. A create also carries kind (block or allow) and reason, which holds 1 to 256 characters. An optional scope names the scope of the entry, and a delete with a scope deletes the entry in that scope. The diff of the plan names the entry with its scope and its kind before and after. A change of a list entry or of a scope has the subject block, also for an allow entry.

On Rules, a person with dns-writer creates an entry in the add form between the header and the list. The form holds the name, the kind (Block by default), the scope (No scope by default) and the reason. Plan shows the diff under the form, and Apply applies the plan. The Delete button of a row opens the delete dialog, which plans and applies the delete with the same steps. A person without dns-writer sees no form and no Delete button.

A scope is a named set of clients. A list entry in a scope applies to those clients only, and it decides before an entry with no scope. A scope has these fields:

Field Rule
name One DNS label of 1 to 40 characters from a-z, 0-9 and -. It becomes a label of the zone <name>.scope.rpz.internal.
order A whole number from 1 to 65535 that no other scope holds. A client in two scopes takes the scope with the lower order
members 1 to 64 members: an IP address, a CIDR prefix with no host bits, or the name of a trusted leg such as main or lab. Guest, Telework and an IPv4-mapped IPv6 prefix are refused

The plan body of a scope carries op (create, change or delete) and name. A create and a change also carry order and members, the whole new member list. A plan refuses a name that a scope holds already, an order that another scope holds, and the delete of a scope that a list entry names, each as invalid_entry with the field.

On Rules, the Scopes section holds New scope, and each scope has Change and Delete. Each opens the scope dialog, which plans, shows the diff of the order and the members, and applies. A person without dns-writer sees the scopes and no control.

On each node, the node worker writes the scopes as the map file /var/lib/dns-worker/scopes.map and the include /etc/powerdns/recursor-main.d/50-scopes.yml. The Lua script of the main process reads the map, and the include names the scope zones. A new scope writes the map first, and a deleted scope writes the include first and deletes the dump file of its zone, so a scope zone never applies to every client.

A plan refuses these names and stores nothing:

Name Answer
Not a host name of 2 to 127 labels, with an optional leading *. invalid_entry on the field name
A name that holds an entry invalid_entry on the field name
The 10,001st entry of one kind invalid_entry on the field name
A name in a zone of the zone store, or a wildcard whose base is above such a zone write_policy with the zone and the reason policy_list_zone

The zone rule protects the pair. A block entry under fzymgc.house would fail the keepalived health check of dns-health.fzymgc.house on both nodes at once. An allow entry there would unblock the internal namespace for a client on no trusted leg.

dns-control does not hold the owned public zones (holomush.dev, fzymgc.net, hl-assets.dev) or the excepted names of the pdns-recursor role. The allow zone allow.rpz.internal. holds those names and comes before the operator zone, so a block entry for one of them is stored and has no effect.

Apply writes the list entry or the scope and a pending change row in one transaction, as for a host entry. The pending row is the lock, so one change runs at a time. The workflow reads the status of each node from its worker to find the node that holds no VIP, and runs these steps:

  1. policy_write in dns-control. It writes every policy zone to pdns-auth on the writer node and records the serial of each zone it wrote in written. The policy zones never show in the zone lists of dns-control, and a zone change refuses a name under rpz.internal..
  2. The steps of the node that holds no VIP, then the steps of the other node. A node that fails stops the steps of the next node, which read previous_node_failed.
  3. Finish the change row, export and publish the event.

Each node takes up to three steps:

Step What the node worker does What the step carries
scope_write, only when the change touched a scope Writes the scope map and the include, reloads the Lua script and the YAML of the main process, and reads the map version back map_version and map_sha256
list_write Waits until pdns-auth on the node serves each written serial, sends a NOTIFY for each zone to the main process, and waits until the process loaded the serial. It gives up after 120 seconds written and loaded, the serials. A change that wrote no zone reads no zone to load
cache_wipe, only for a change that names a client Removes each name of the change and every name under it from the caches of both processes wiped, the count of each process

A list change writes nothing to Kea. The export writes blocks.json beside entries.json and zones.json, to both sinks. It holds the scopes sorted by order, each with its name, order and members, and the name, the kind, the scope and the reason of each entry, sorted by name. The actor and the time are in the audit object of each change.

The entry stays in the store when a list change fails, so the store holds the truth. A projection writes the whole set to both nodes again and changes no entry. Run it with dns_project_entries and the subject block, because the Rules screen has no control for a projection.

The step reads What happened What to do
list_refresh_failed on list_write The zone load of the node failed. pdns-auth on the node did not serve a written serial, the NOTIFY failed, or the main process did not load the serial within 120 seconds. The detail names the zone. Each zone has refresh: 300, so the process loads the change within five minutes on its own Read rec_control --config-name=main rpzstatistics on the node. If the serial stays old, apply a projection
pdns_write_failed on policy_write pdns-auth on the writer node did not take a policy zone. No node ran a step Read the pdns-auth logs on the writer node, then apply a projection
scope_write_failed The node did not write the scope map or the include, or the main process did not reload. The rules in a scope do not apply on that node yet Read journalctl -u pdns-recursor@main on the node. Fix the cause, then apply a projection
node_not_answering The node worker did not take the activity Bring the node or its worker back, then apply a projection
previous_node_failed The first node failed, so the second node was not written Fix the first node, then apply a projection

After a node is rebuilt, the dns-node-worker role seeds an empty scope map. Apply a projection after the converge, so the node holds the scopes again.

If dns-control is down and an entry must change at once, edit the files on the nodes. See The list entries of dns-control. The next list change or projection writes the files from the store again.

dns-control also holds every zone and every static record set as data. A zone row holds the kind, the SOA values, the apex NS set and the zone metadata. The SOA serial is not stored, because pdns raises it on each API write. A record set row holds one name and one type in a zone, with its TTL, its records and its lane. The SOA, the apex NS, the metadata and each record set in pdns are projections of these rows. Sean decided on 2026-10-01 that dns-control is the only writer of what the resolver pair serves. Kea D2 is the one other writer, and it writes only inside its grants.

A lane names the one writer that can change a record set. A name belongs to one lane.

Writer Role Can write
A person or an agent dns-writer Curated entries. See Entry changes
A person or an agent dns-admin Zones, their TSIG-ALLOW-DNSUPDATE grant, the control lane
The external-dns webhook dns-lane-external-dns The record sets of the external-dns lane
The observed AAAA sweep dns-admin, as the caller system:observed-aaaa The AAAA record sets of the observed-aaaa lane. See Observed AAAA

A plan at a name that another lane or a curated entry holds gets write_policy with the reason policy_other_lane. A name that Kea D2 writes gets policy_other_writer. A name inside a child zone of the store gets policy_zone. The NS or DS record set of a delegation at the apex of the child is the exception. The external-dns lane also stays out of lan.fzymgc.house and off the reserved names. The apply checks the lane rule again inside its transaction, because another change can land between the plan and the apply.

The observed-aaaa lane writes beside Kea D2 at its names, inside a narrow scope. A plan of another type gets policy_type. A zone that is not the dynamic zone of a trusted leg gets policy_zone. A create or a change at a name with no Kea D2 A record gets policy_no_kea_name. An address outside the ULA /64 of the leg gets policy_address. A delete needs no Kea D2 name.

The group dns-admins grants dns-admin, and the operator is in it. The MCP scope mcp-dns maps dns-admin, so the MCP token of a member carries it.

The external-dns lane has its own route group, /api/lanes/external-dns/. GET on rrsets lists the record sets of the lane. POST on apply applies one batch: the records of one external-dns sync as one change of the subject batch. A lookup of the changes by name finds each record inside a batch.

Only a client token with dns-lane-external-dns reaches the group. A session, a static key, a person and any other client get 401 or 403. Each body is a protobuf message of dnscontrol.lane.v1 in protojson, and both sides validate it.

The external-dns sidecar calls the group. It is the subcommand dns-control webhook of the dns-control image. It serves the webhook API of external-dns on 127.0.0.1:8888, and it answers the kubelet probes with GET /healthz on :8080. It takes a client-credentials token of the Keycloak client dns-control-external-dns, whose secret is at fzymgc-house/cluster/external-dns/oidc. argocd/app-configs/external-dns/values.yaml sets the provider webhook and runs the sidecar from the dns-control image. The sidecar reads the one pin of the image, as The image and its pin describes. The external-dns namespace holds no pdns API key.

In a batch, a record that the lane policy refuses leaves the batch, and dns-control logs it with the line external-dns record refused. The other records still apply. A registry TXT of external-dns that claims a name that another lane or an entry holds is refused too. The plan reads create, change or delete from the store, so a batch that external-dns sends again changes nothing.

The pdns step writes each record set of a batch with its own PATCHes, so a record that pdns refuses fails alone. The step lists each record set that it did not write in failed. When a batch does not succeed, its finalize puts each such record set back to its before side, so the store matches pdns. external-dns then sees those records as missing and sends them again on its next sync.

external-dns waits 15 s for the sidecar and exits on a timeout. The sidecar therefore gives each call to dns-control 12 s, set by budget, and the apply route of dns-control waits 10 s for the workflow. A slower change answers pending and then 503, and the workflow goes on. Every failure of dns-control answers 503, because external-dns retries a 5xx and exits on any other error. A refused token or a refused body logs at ERROR in the sidecar, because it does not clear by itself.

Step Route Tool Role
Plan a zone create, change or delete POST /api/zones/plan dns_create_zone, dns_change_zone, dns_delete_zone dns-admin
Plan a grant change POST /api/zones/metadata/plan dns_change_zone_metadata dns-admin
Plan a record set create, change or delete POST /api/rrsets/plan dns_create_rrset, dns_change_rrset, dns_delete_rrset dns-admin
Apply a plan POST /api/changes/apply dns_apply_change dns-admin
Project the zones or the control lane unchanged POST /api/changes/project with subject dns_project_entries with subject dns-admin
Read the zones GET /api/zones, GET /api/zones/{name} dns_list_zones, dns_get_zone dns-reader
Read the record sets GET /api/rrsets?zone=&lane= dns_list_rrsets dns-reader
Read the changes GET /api/changes?subject=zone, GET /api/changes?subject=rrset dns_list_changes dns-reader

The zone plan body carries op and zone with name, kind, soa, ns, ns_ttl and metadata. The soa object carries mname, rname, refresh, retry, expire, minimum and ttl. A delete carries op and name only. A change can alter the SOA values, the NS set and TSIG-ALLOW-DNSUPDATE. The kind and every other metadata kind keep their values. A change with no metadata key keeps the metadata of the row, so a change of the SOA keeps the grant.

A zone delete is refused for the apex and the two reverse zones of the curated entries. It is also refused while the zone holds record sets, and while pdns serves any RRset in the zone besides its SOA and apex NS, such as a Kea D2 lease. The pdns step makes the same comparison again before it deletes the zone.

The grant body carries zone, kind and values. An empty values list removes the grant. The record set plan body carries op and rrset with zone, name, type, ttl and records. A TXT record is quoted. The SOA and the apex NS belong to the zone, so a record set plan refuses them. A plan is also refused for a CNAME at a zone apex, a CNAME beside another type at its name, and another type beside a CNAME, because pdns refuses each one. A record set plan and a zone delete wait for the first build of the lineage index. The UI has no zone editor. The Changes screen lists zone and record set changes by name.

A zone or record set change runs the same change.apply workflow as an entry, under the same lock, with no reservation step. The pdns_write step reads what the writer node serves and writes only what differs:

  1. If pdns does not serve the zone, the step creates it with soa_edit_api DEFAULT, its SOA and its apex NS. Then it writes each metadata kind of the row.
  2. If pdns serves the zone, the step replaces the SOA or the NS set when its values, its TTL or its comment differ. It sends the live serial, and pdns raises it. If pdns serves the zone with another kind or another soa_edit_api, the step fails and writes nothing.
  3. The step writes each metadata kind that differs. pdns loses a TSIG-ALLOW-DNSUPDATE grant that the row does not hold.
  4. The step replaces a record set with its records, its TTL and the comment writer=dns-control change=<id>. A disabled record in pdns counts as not served, so the step writes the RRset again.
  5. A delete clears every comment first and then removes the RRset or the zone. The lmdb backend keeps the comments of a deleted RRset or zone for the next one with that name.

A retry of the step after a partial write repeats no write that landed.

Terminal window
kubectl exec -n dns-control deploy/dns-control -- dns-control zones import --dry-run
kubectl exec -n dns-control deploy/dns-control -- dns-control zones import

zones import reads every zone, its metadata and its RRsets from the writer node. It builds the lineage index from the reads of both nodes and the store. Then the import sorts each RRset by its writer:

Writer in the index Goes to
None, for the SOA or the apex NS The zone row
None, for any other type A record set of the control lane
external-dns, its registry TXT records included A record set of the external-dns lane
dns-control or Kea D2 Nothing: skip writer_<writer>
Two writers Nothing: skip conflict

The import also skips a record that pdns holds disabled. Every zone must have soa_edit_api DEFAULT, or the import stops with the name of the zone. Each zone keeps every metadata kind that the pdns API can write. A kind that the API cannot write, such as API-RECTIFY or TSIG-ALLOW-AXFR, gets the line skip protected_metadata with the type METADATA <kind>. The command prints one line per zone, record set and skip, then the counts as one JSON line. --dry-run stops there.

The import writes each row that the store does not hold as one change in the surface cli, with the reason imported. It starts no workflow, and pdns does not change. If the store holds a row with other content, the import does not write it, names it, and ends with an error. Run adopt for each such row.

To roll the import back, run dns-control zones delete --imported. The command removes every row that an import wrote, each as one change with the reason import_removed, and pdns does not change. If a zone holds a record set from another source, the zone stays and the command stops with its name.

dns-control zones export --format tf-import prints one Terraform import block per zone, SOA, apex NS, metadata kind and control record set, in the import id forms of mmianl/powerdns. No store holds the old Terraform addresses, so each block names a generated address such as powerdns_zone.z_lab_lan_fzymgc_house. Run terraform plan -generate-config-out=generated.tf to write the resource blocks. --format json prints the zones.json form of the store.

If dns-control or its database is down and a name must change now, write the pdns API directly from a k8s node with the key at fzymgc-house/infrastructure/dns/pdns-api. pdns keeps serving while dns-control is down, so only changes wait. The lineage index shows the edit as content_drift. When dns-control is back, adopt the live content:

Terminal window
kubectl exec -n dns-control deploy/dns-control -- dns-control adopt dns.core.fzymgc.house A
kubectl exec -n dns-control deploy/dns-control -- dns-control adopt dns.core.fzymgc.house A --apply

adopt plans a change that stores what pdns serves at the name and type, and prints the plan. --apply applies it, and the RRset then carries the mark of the new change. The SOA and the apex NS of a zone adopt into the zone row. An RRset that pdns no longer serves adopts as a delete. --zone names the zone when the longest zone of the store does not hold the name. A record set of the external-dns lane is not adopted, because external-dns writes it again on its next sync. A disabled record is not adopted, because an apply writes every record enabled. adopt --apply ends with an error when the change does not succeed.

Terminal window
kubectl exec -n dns-control deploy/dns-control -- dns-control zones grant --generations 2,3 --dry-run

zones grant sets the TSIG-ALLOW-DNSUPDATE grant of every dynamic zone in one command. For each zone, the new grant names each Kea D2 key base of the zone, such as kea-d2-lab, at each generation in --generations. Generation 1 is the base alone, and a generation n > 1 adds -<n>. A key that is not a Kea D2 key leaves the grant, and a zone whose grant names no Kea D2 key stops the command. Without --dry-run, it plans and applies one change per zone that differs, through the change workflow. The TSIG key rotation in the DNS runbook uses it to widen and then narrow the grants.

Terminal window
kubectl exec -n dns-control deploy/dns-control -- dns-control restore --from s3://dns-control-export --dry-run

restore reads entries.json and zones.json from one sink. --from takes nas, b2, or the s3://<bucket> URL of one of them. The command compares each zone, then each record set, then each entry with the store and with what the writer node serves. It prints one line per subject:

Action When
no change The store holds the subject equal and pdns serves it
create The store does not hold the subject
change The store holds other content, or pdns does not serve the subject
skip external-dns The record set is in the external-dns lane, and external-dns writes it again

Without --dry-run, the restore plans and applies each subject that is not no change, one at a time, through the change workflow. It stops at the first change that does not succeed. A row that the store holds and the export does not is left alone. On an empty pdns, the restore creates each zone and then writes its control record sets and its entries. external-dns then writes its own lane on its next sync. A grant that pdns holds and the export does not reads change.

Terminal window
kubectl exec -n dns-control deploy/dns-control -- dns-control drift --state orphan --count

drift builds the lineage index from the reads of both nodes and the store. It prints the RRsets in one state with their writers, and --count prints the count only. The states are orphan, conflict, content_drift, attributed and dangling.

Each command of this section reads the configuration of the server from the environment, so it runs inside the pod.

dns-control publishes an AAAA record for each named client of a trusted leg that uses an address of its leg ULA. The record sits at the Kea D2 name of the client, <host>.<leg>.lan.fzymgc.house, in the observed-aaaa lane.

The sweep reads the client table of the UniFi controller as the read-only account dns-control-observer. The request is GET /proxy/network/api/s/default/stat/sta on unifi.url. The sweep reads ipv6_addresses and never last_ipv6, which keeps addresses that a client no longer uses. It keeps an address only when it sits in the ULA /64 of the client’s leg. It publishes no GUA, no link-local address and no foreign ULA.

The key unifi_networks of the leg file maps each UniFi network name to a leg. A client of a network that the map does not name fails the read. A rename in UniFi then stops the sweep, and it does not match nothing. The controller lists some clients with no network. Such a client is on no leg, so the sweep skips it, and the Query log shows no network for it.

The sweep joins each client to a name by its IPv4 address. The name is the one whose Kea D2 A record holds that address in the dynamic zone of the client’s leg. A client with no Kea D2 name gets no AAAA. A curated entry gets no observed AAAA, and its owner sets the IPv6 address of the entry.

The join also binds the MAC of the client. When a client leaves and its IPv4 address goes to another client, the read shows the new MAC at that address, and the addresses of the old client are not published at the name. Of two MACs whose sightings name one IPv4 address, the MAC seen last wins.

The table observed_addresses keeps one row for each address of each client. The row holds the first and the last sighting, the number of sweeps that saw the address, and the leg and the IPv4 address of the client at the last sighting.

  • An address is published after observed_aaaa.publish_after sweeps saw it, 2 by default.
  • An address leaves after observed_aaaa.retire_after without a sighting, 30 minutes by default.
  • A restart keeps every row, so it neither delays nor removes a record.

When Kea D2 removes the A record at a name, the next sweep removes the AAAA there. While the AAAA is there, Kea D2 keeps the DHCID at the name, and it does not remove the PTR and the DHCID of the released address either. The same change removes these leftovers:

  1. The PTR and the DHCID at each reverse name that holds only a PTR to the name and the same DHCID. A reverse name that Kea D2 gave to another client since then holds another DHCID, and the change leaves it.
  2. The DHCID at the name, when it is the last record there, so the next client can claim the name.

If a delete fails, the change fails, the store keeps the AAAA, and the next sweep tries again. The reverse records go first, because the DHCID at the name is what finds them again on that retry.

Each lease that ends under a lane AAAA logs two ERROR lines in the D2 journal of both resolver nodes (journalctl -u isc-kea-dhcp-ddns-server). They are expected, and the next sweep removes what they leave:

DHCP_DDNS_FORWARD_REMOVE_RRS_REJECTED ... rejected a DNS update request to remove forward RR entries for FQDN, <name>, with an RCODE: 7
DHCP_DDNS_REMOVE_FAILED ... Transaction outcome: Status: Failed, Event: UPDATE_FAILED_EVT, Forward change: failed, Reverse change: failed

Each read that holds at least one client also replaces the table observed_stations in one transaction. The table holds every station of the read, every leg included. A row holds the MAC, the IPv4 and IPv6 addresses, the network, whether the station is wired, the alias that an admin set in UniFi, the hostname that the device reports, the vendor, the Wi-Fi network, the time UniFi last saw the station and the time of the read.

The Query log and the MCP tool dns_query_log read this table to show the device at a client address. A Query log request never reaches the controller. The stations never enter the payload of the sweep workflow, which Temporal keeps in its history.

  • A failed read and an empty table leave the table as it was.
  • A sweep that the removal guard refuses still replaces the table. The guard protects the lane, and the table is for display. A device that the partial read missed shows no device until the next sweep.
  • A failed write of the table logs replacing the observed stations and does not fail the sweep.
  • With observed_aaaa.enabled false, the Query log reads no device, even from an older table.

dns-control creates or updates the Temporal Schedule observed-aaaa-sweep at each start, with the interval observed_aaaa.interval, 300 seconds by default. A run that overlaps the one before it is skipped, and a missed run is caught up for one interval. With observed_aaaa.enabled set to false, the Schedule is paused. A change in the key dns-control.yaml of the ConfigMap restarts the pod through Reloader, and the start applies it.

Terminal window
temporal schedule describe --namespace dns-control --schedule-id observed-aaaa-sweep

A sweep that finds a difference applies one batch change of the lane, as the caller system:observed-aaaa on the surface system. A sweep with no difference writes nothing. The lane is not in zones.json, because the next sweep builds it again.

A run of the workflow observed.sweep with the input dry_run set to true plans from that one read, as if every address in it were published. It writes no sighting, no health reading and no change, and its result lists the planned record sets.

A sweep changes no record when one of these is true:

  • The read of the client table fails, for example on HTTP 429 or a timeout.
  • The client table holds no client.
  • A client sits on a UniFi network that the leg file does not name.
  • The sweep would remove more than two record sets, and more than half of the lane, at names where Kea D2 still holds the A record. Those removals depend on the read, and a partial read must not empty the lane. A removal after Kea D2 removed its A record always goes through.

The sweep records the error, and the Health screen and dns_get_health show the time of the last good read, the time and the text of the last error, and the lane size. The status reads failing until a later sweep reads the table again. A sweep that meets a pending change writes nothing, and the next sweep tries again.

To stop the sweep, set observed_aaaa.enabled: false in the key dns-control.yaml of the ConfigMap in argocd/app-configs/dns-control/config.yaml, and merge. The restart pauses the Schedule. To remove every record set of the lane, pause the sweep first, then run the purge inside the pod:

Terminal window
kubectl exec -n dns-control deploy/dns-control -- dns-control lane purge observed-aaaa --dry-run
kubectl exec -n dns-control deploy/dns-control -- dns-control lane purge observed-aaaa
kubectl exec -n dns-control deploy/dns-control -- dns-control rrsets list --lane observed-aaaa --count

The purge applies one change of the lane, and the last command prints 0 after it.

dns-control sets the alias and the note of each UniFi client whose MAC a curated entry holds. The client table of the controller then shows the name of each curated device. The OpenSpec change unifi-client-alias holds the design, and spike 012 (.planning/spikes/012-unifi-client-rename/) measured the write.

A run reads every entry from the store, and every client that the controller knows from GET /proxy/network/api/s/default/rest/user, as the account dns-control-writer. It joins the two on the MAC. For each client that an entry holds, the alias is the first label of the hostname of the entry. The note is the mark writer=dns-control entry=<key>. The key names the entry, so a change of the hostname keeps the mark.

  • A client whose alias and note already agree gets no request.
  • A client with another alias or another note gets PUT rest/user/<_id> with name and note alone. The controller merges the body, so every other key of the client stays as it was.
  • A client whose note carries the mark and whose MAC no entry holds gets name and note set to "". That is the case after a delete of the entry, or after its MAC goes.
  • A client with no mark and no entry gets no request. An alias that a person set there stays.

Each write logs one line, UniFi client alias written, with the MAC, the client id, the entry, and the old and new alias and note. The controller also stores a cleared alias or note as "", and noted reads false after a clear.

Change the hostname of its entry. A rename in the UniFi UI lasts only until the next run, which puts the alias of the entry back. The note of a curated client belongs to dns-control in the same way, so describe the device in the comment of the entry.

The Health screen and dns_get_health list the key of each entry whose MAC no client of the controller holds, in unifi_alias.unknown. The usual causes:

  • The device uses a private Wi-Fi address, so the controller sees another MAC. Turn the private address off for the network on the device, or put the MAC that the controller sees in the entry.
  • The entry holds another MAC of the device. The entry nas holds 3c:ec:ef:fa:66:af, and the NAS answers on 3c:ec:ef:fa:66:ae.
  • The device never connected through the UniFi fabric, such as a container behind a host bridge.

dns-control creates or updates the Temporal Schedule unifi-client-alias at each start, with the interval unifi_alias.interval, 15 minutes by default. The interval lies between 1 minute and 1 hour. The Schedule skips a run that overlaps the one before it. With unifi_alias.enabled set to false, dns-control pauses the Schedule. dns-control refuses to start with the comparison on and no unifi.writer_username or unifi.writer_password.

Terminal window
temporal schedule describe --namespace dns-control --schedule-id unifi-client-alias

A run writes nothing when one of these is true:

  • The configuration holds no writer account.
  • The read of the entries fails, or the store holds no entry with a MAC. This guard keeps a fault in the read of the store from clearing every alias that dns-control wrote.
  • The login or the read of the client table fails, for example on HTTP 429.
  • The client table holds no client.

The first failed write ends the run, and the next run sends the remaining writes again. The run records the error, and the Health screen and dns_get_health show unifi_alias.status failing with the time and the text of the error. No alert pages for a failing run, because an old alias asks no person to act at once.

dns-control-writer holds the role Network Admin, because the console offers no narrower role that can rename a client. The controller therefore does not confine the account, and dns-control does. The client has one write, to rest/user/<_id>, and the test TestRunSendsOnlyTheLoginTheReadAndTheWrites fails on any other request of a run. The admin activity log of the controller names the account for each change, by its display name DnsControl Writer. Each change by that account must have the change key CLIENT and the collection user. The probe of spike 001 saves the log as JSON with the View Only account, and jq counts the changes by account:

Terminal window
.planning/spikes/001-networkconf-guard-fields/run.sh probe_admin_activity.py <scratch>/activity.json <since, ISO 8601> > /dev/null
jq -r '.data[] | [.admin.name, .change_key, (.meta.collection // "-")] | @tsv' <scratch>/activity.json | sort | uniq -c

Every line that names DnsControl Writer must read CLIENT and user.

After each upgrade of the Network application, make sure that a write still keeps the other keys of a client. Write the alias and the note that the client of pdu already holds. The probe diffs the client before and after the write, and it must print diff: no key changed. A controller that replaces the record deletes every other key, and the diff then lists them.

Terminal window
.planning/spikes/012-unifi-client-rename/run.sh writer rename.py <scratch> 00:0c:15:41:30:15 \
--name pdu --note 'writer=dns-control entry=pdu'

To stop the comparison, set unifi_alias.enabled: false in the key dns-control.yaml of the ConfigMap in argocd/app-configs/dns-control/config.yaml, and merge. The restart pauses the Schedule. The aliases stay, and they still name the right devices. To remove them, clear the alias and the note of each client in the UniFi UI. These commands list the clients that carry the mark:

Terminal window
scripts/unifi-readonly.sh scripts/unifi-inventory.py <scratch> > /dev/null
jq -r '.[] | select((.note // "") | test("^writer=dns-control entry=")) | [.mac, .name] | @tsv' <scratch>/user.json

The file apps/dns-control/VERSION holds the version, and changie writes it on a release. The release workflow (.github/workflows/release.yml) runs when a release pull request merges. GoReleaser Pro builds the amd64 and arm64 images from apps/dns-control/.goreleaser.yaml, pushes the multi-architecture index, and signs it with a keyless cosign signature. The same release holds the node worker tarball. App Releases describes the release.

apps/dns-control/Dockerfile keeps the gated build that pr-gates runs. The release image comes from apps/dns-control/Dockerfile.release, which copies the binary that GoReleaser built onto the same Debian base.

If the version tag exists in GHCR, the release workflow fails before GoReleaser runs, so a version tag never moves. The summary of the dns-control job shows the published image in one line, for example:

published: ghcr.io/fzymgc-house/dns-control:0.1.0@sha256:b9ade42aac443c09478b12841e6891d1650c7dfe91ef5d1a77541e9b787df2a1

dns-control has one pin, and Argo CD Image Updater moves it. The pin is newTag in the kustomize Component argocd/app-configs/dns-control-image/kustomization.yaml. The kustomization of dns-control includes the Component for the Deployment dns-control, and the kustomization of external-dns includes it for the webhook. The pin holds a version tag and no digest. A release opens one pull request, and it waits for a review.

Image Updater checks a cosign signature only against a key pair, and dns-control signs keyless. Before you merge an updater pull request, make sure that the signature of the new version comes from the release workflow on main:

Terminal window
cosign verify ghcr.io/fzymgc-house/dns-control:<version> \
--certificate-identity 'https://github.com/fzymgc-house/selfhosted-cluster/.github/workflows/release.yml@refs/heads/main' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com

A version from 0.4.0 onward stops at start without the nine EXPORT_* variables of the two export sinks, and reaches the NAS sink only with the hostAliases pin and the two egress rules of the network policy. The manifests in argocd/app-configs/dns-control/ carry all of them.

A version from 0.3.0 onward needs three changes. The manifests in argocd/app-configs/dns-control/ carry them:

  • The variable DATABASE_URL, from the key database_url of the Secret dns-control-db. dns-control stops at start without it.
  • The mount of the NATS credentials file from the property nats_creds of the Secret dns-control, at /etc/dns-control/nats/dns-control.creds with mode 0400.
  • Egress to the Temporal frontend on TCP 7233, to the CNPG pods of the cluster main on TCP 5432 and to the NATS servers on TCP 4222. Each rule names a namespace and a label. No rule names a DNS name.

To release a new version:

  1. Open the release pull request with scripts/release.sh dns-control, as in Release an app.
  2. Merge the release pull request. The release workflow tags dns-control/v<version>, and builds, signs and publishes the image.
  3. Within five minutes, Image Updater opens the two pull requests that move the pins. Review and merge them as After a dns-control release describes.

Every value that crosses the Temporal boundary is a protobuf message. The node worker on the Pis and the dns-control image ship on their own, so an older worker can meet a newer dns-control. The field numbers of the messages keep the two sides compatible. The paths below are relative to apps/dns-control/.

Package Files What it carries
dnscontrol.nodeops.v1 proto/dnscontrol/nodeops/v1/actions.proto, proto/dnscontrol/nodeops/v1/status.proto The inputs and results of the six node worker activities, and the status reading of a node
dnscontrol.ops.v1 proto/dnscontrol/ops/v1/ops.proto The inputs and results of the operations workflows and of their control activities in dns-control

The two packages change for different reasons. dnscontrol.nodeops.v1 crosses the network to the Pis, so a change to it needs a worker release on both Pis. dnscontrol.ops.v1 stays inside dns-control. Temporal keeps the workflow history across releases of dns-control, so its field numbers are part of the contract too.

The generated Go code is in internal/gen/dnscontrol/nodeops/v1 and internal/gen/dnscontrol/ops/v1. Do not edit it by hand. The gate generates it again and fails on any difference.

Each message carries protovalidate rules. Protovalidate is the rule language of Buf for protobuf fields. The Temporal client of the node worker and the client of dns-control both carry wire.DataConverter() and wire.Interceptor():

  • The converter sends each message as binary protobuf. Its last step accepts only a bool, an integer, a float or a string, so a hand-written Go struct fails at every Temporal call site.
  • The interceptor checks each payload against its rules in the client, the workflow and the activity. A payload that fails its rules, or a nil message, fails with a non-retryable error of the type invalid_payload.

The node worker therefore refuses an invalid input before the activity code runs. dns-control refuses an invalid result, and the operations workflow records the action as failed with the reason action_failed.

The tasks need Docker. Each step runs in the builder image of apps/dns-control/Dockerfile, and buf v1.73.0 and protoc-gen-go come from the tools module tools/go.mod. No shipped binary links the tools module.

  1. Edit the .proto file under proto/dnscontrol/.

  2. Write the generated code:

    Terminal window
    mise //apps/dns-control:proto:generate
  3. Run the gate before the push. Fetch main first, because the breaking check compares with origin/main. The last line of a pass is PROTO_GATE=PASS.

    Terminal window
    git fetch origin main
    mise //apps/dns-control:proto:check
  4. Commit the .proto file and the generated code together.

The gate tools/proto.sh check prints one line for each step: MODULES, NO_DEPS, LINT, VENDOR_PIN, BREAKING, DRIFT and OFFLINE. The pr-gates member dns-control-proto in .github/workflows/pr-gates.yml fetches main and runs the same gate with BUF_BREAKING_BASE=origin/main. It runs when a pull request changes a file under apps/dns-control/, and in the nightly pr-gates run. buf breaking uses the rule set FILE. If main has no apps/dns-control/buf.yaml, the gate prints BREAKING=NO_BASELINE and skips the breaking step.

The Buf Schema Registry is the hosted schema service at buf.build. No build or CI step reads it.

  • buf.yaml names no dependency. The gate refuses a deps entry and a buf.lock by name.
  • The one dependency schema, proto/buf/validate/validate.proto, is vendored into the repository.
  • The step MODULES downloads the Go modules through proxy.golang.org only, with no direct fallback.
  • Every later step runs in a container with --network none and GOPROXY=off. A step that tries to reach buf.build or any other host fails there.

The Go runtime of protovalidate imports the module buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go. That module comes from proxy.golang.org, and go.sum and sum.golang.org pin it. No step contacts buf.build for it.

The vendored validate.proto comes from bufbuild/protovalidate at the commit ec950f2039c7cf07895f1f83ab565d7fc57628be, with the sha256 39f6496bef0a622be188703bd3db2ed23b3e4b5c2da27b121b88947f311d3448. tools/proto.sh holds both values as PROTOVALIDATE_COMMIT and PROTOVALIDATE_SHA256, and the step VENDOR_PIN compares the file with the sha256. To move the pin:

  1. Change both values in tools/proto.sh.

  2. Download the file at the new commit. The task refuses a file whose sha256 differs:

    Terminal window
    mise //apps/dns-control:proto:vendor
  3. Run mise //apps/dns-control:proto:generate and mise //apps/dns-control:proto:check.

  1. A new field takes a new number. Never use a number again.
  2. A removed field keeps its number and its name as reserved.
  3. Every enum field refuses the value 0.
  4. An enum that a receiver acts on, Outcome and Action, carries defined_only: true. An older receiver refuses a value that it does not know with invalid_payload, so a new value ships in the receiving side first. For a value that dns-control sends, release the node worker on both Pis first. For a value that the node worker sends, release dns-control first.
  5. An enum that a receiver only shows or compares, KeepalivedState and ListSourceKind, carries no defined_only. An older receiver accepts a new value. It reads an unknown keepalived state as UNKNOWN, which fails keepalived_not_fault, and an unknown list source kind as no kind.
  6. A receiver ignores a field that it does not know, in binary and in JSON payloads.
  7. A new reason id ships in the receiving side first too. The reason pattern ^[a-z_]{0,64}$ accepts an id that the receiver does not know, but an older UI shows the generic failure copy for it.

After dns-node-worker-v0.3.0, a change to dnscontrol.nodeops.v1 needs a worker release on both Pis. Open the release pull request with scripts/release.sh dns-control and merge it. The release dns-control/v<version> holds the worker tarball. Then set dns_node_worker_version in ansible/roles/dns-node-worker/defaults/main.yml and converge one node per window.

The ConfigMap dns-control-legs in argocd/app-configs/dns-control/config.yaml holds legs.json. A leg is one network segment that the resolvers serve, for example main or guest. The file lists the two pdns nodes, the conditional zones, the seven legs and the tailnet. dns-control does not start without a valid leg file.

The file is a copy. The source of each value is in Ansible:

Value Source
The legs (VLAN, IPv4 subnet, trust) and the pdns node addresses dns_vlan_interfaces in ansible/inventory/host_vars/sandiego.yml and ansible/inventory/host_vars/portland.yml
The ULA /64 of each leg ula_site_prefix and ula_leg_index in ansible/inventory/group_vars/all.yml
The pdns API port, the forward zone and the reverse zones pdns_api_port, dns_zone_domain and dns_reverse_zones in ansible/inventory/group_vars/dns_hosts.yml

No check compares the copy with Ansible. If you change a leg, a node address or a zone in Ansible, change config.yaml in the same pull request.

Each node name must equal the Ansible inventory hostname. The resolver collectors send that name as host.name. If a name differs, every Health metric of that node shows no value.

dns-control and Keycloak both have names that the resolver pair serves. If the pair is down, a client on the LAN cannot resolve either name. The sign-in also goes through https://id.fzymgc.house, so a name for dns-control alone does not help.

Traefik serves both names on the address 192.168.250.145. To reach dns-control when the pair is down, add these two lines to the hosts file of your workstation (/etc/hosts on macOS and Linux):

192.168.250.145 dns-control.fzymgc.house
192.168.250.145 id.fzymgc.house

Remove the lines when the pair is back. If the lines stay, a later change of the Traefik address breaks both names on your workstation.

Issue #2386 proposed tailnet names for Keycloak and dns-control. It closed as not planned. A second Keycloak hostname changes the iss claim of the tokens, and dns-control accepts one issuer only. The hosts-file lines stay the fallback. A break-glass surface on the Pis is future work (FUT-03 in .planning/REQUIREMENTS.md).

Each process of the binary sends traces, metrics and logs over OTLP to cs-otel-collector in the namespace clickstack. internal/observe builds the three providers from OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS. If the endpoint is empty, the process sends nothing. A collector that is down or refuses the data never stops a request. Each process flushes its telemetry for at most 5 seconds before it exits.

Process Service name Local output
The server dns-control JSON on stdout, with trace_id and span_id
dns-control webhook, the container webhook in the external-dns pod dns-control-webhook JSON on stdout, with trace_id and span_id
An operator command through kubectl exec dns-control-cli Text on stderr at the level Warn
The node worker on sandiego and portland dns-node-worker JSON on stdout, with trace_id and span_id, in the journal of the unit

Each process sets its service name in code. An operator command inherits the environment of the server, so a name from OTEL_SERVICE_NAME labels its spans as the server. The node worker sends to the collector on its own node at 127.0.0.1:4318, which forwards through otel-gateway.fzymgc.house. It sends the three gauges dns_worker_client_cert_not_after_seconds, dns_worker_drained and dns_worker_drain_overdue, the Temporal SDK metrics and the Go runtime metrics, with the host.name of the node. The journal receiver of the collector reads dns-node-worker-pki.service and not the worker unit. The pod annotation vector.dev/exclude-containers keeps Vector off the containers dns-control and webhook, so each log line reaches default.otel_logs once. The OTLP log records start at the level Info.

One trace follows a request from its first seam to its last. Trace context crosses each boundary in W3C form: HTTP headers, gRPC metadata, Temporal headers, and the traceparent header of each NATS message.

Seam Span
A request to the REST, MCP or webhook listener A server span named <method> <route>, such as POST /mcp
A request to pdns, Keycloak, the UniFi controller, an S3 sink, ClickHouse or the lane API A client span
A Temporal workflow or activity RunWorkflow:<type> and RunActivity:<type>
A Postgres query A client span with the SQL text and no parameter
A NATS publish publish <subject>
An MCP tool call A span named after the tool, as a child of the server span
An apply of a change change.apply, with dns_control.surface, dns_control.subject, dns_control.caller_kind and dns_control.caller_azp
A run of a background loop One root span: lineage.refresh, ops.poll, sessions.sweep or ledger.reload
An operator command One root span: dns-control <command>

The server sends the Temporal SDK metrics, the Go runtime metrics and eight application metrics. The meter exports every 60 seconds. Each gauge sends every value of its attributes, with 0 where nothing matches, so an alert reads 0 and not an empty bucket. No metric carries caller_azp, because each dynamic client registration gives a new client id.

The counters are delta. The Deployment sets OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to delta, so each counter point holds the increase since the previous export. A counter sends a point only for an attribute set that changed in that export. Add the points with sum(Value). HyperDX skips the first point of a cumulative series, and each pod starts new series. A cumulative counter therefore hides the first event of each kind after a restart.

Metric Kind Attributes Value
dns_control_change_applies_total counter surface, subject, op, outcome, caller_kind One for each change whose outcome is final
dns_control_change_refusals_total counter surface, step, reason One for each refused plan or apply
dns_control_export_pending_age_seconds gauge sink The age of the oldest succeeded change that the sink does not hold yet, or 0. An imported change has no workflow and never counts
dns_control_lineage_rrsets gauge state, finding The RRsets of the published lineage index in each state and settle status
dns_control_lineage_refresh_timestamp_seconds gauge none The Unix time of the last published index
dns_control_observed_sweep_timestamp_seconds gauge none last_good_read_at of the table observed_sweep
dns_control_lineage_node_disagreements gauge result, finding The RRsets on which the pdns nodes do not agree
dns_control_vip_holder_status gauge status 1 for the current status of the VIP holder and 0 for the other three
Attribute Values
surface ui, rest, mcp, cli, system
subject entry, zone, rrset, batch
op create, change, delete. A batch has change
outcome succeeded, failed
caller_kind person, client, static, system
step plan, apply
reason role_required, index_not_ready, invalid_entry, plan_not_found, plan_expired, entry_changed, subject_changed, change_in_progress and the policy_* codes of the write policy
sink nas, b2
state attributed, orphan, conflict, content_drift, dangling
result differ, missing, unreadable
finding pending, settled, and none for attributed
status one_holder, none, split, not_read

The two gauges that read Postgres skip their point when the read fails, and they log the failure. The lineage gauges and the refresh time send nothing before the first index.

Run these in clickhouse-client on a ClickHouse pod (see Useful queries).

The trace of the last call of a tool, with the services and the spans that it holds:

SELECT countDistinct(ServiceName) AS services, groupUniqArray(SpanName) AS spans
FROM default.otel_traces
WHERE TraceId = (SELECT TraceId FROM default.otel_traces
WHERE ServiceName = 'dns-control' AND SpanName = 'dns_get_health'
ORDER BY Timestamp DESC LIMIT 1)

The surfaces of the changes in the last 10 minutes:

SELECT Attributes['surface'] AS surface, Attributes['outcome'] AS outcome, sum(Value) AS total
FROM default.otel_metrics_sum
WHERE ServiceName = 'dns-control' AND MetricName = 'dns_control_change_applies_total'
AND TimeUnix > now() - INTERVAL 10 MINUTE
GROUP BY surface, outcome

A log line that reaches ClickStack twice. The answer must hold no row:

SELECT TraceId, SpanId, count() AS copies
FROM default.otel_logs
WHERE ServiceName = 'dns-control' AND Body = 'mcp tool call'
AND Timestamp > now() - INTERVAL 1 HOUR
GROUP BY TraceId, SpanId
HAVING copies > 1

The spans of the operator commands in the last hour:

SELECT Timestamp, SpanName, StatusCode
FROM default.otel_traces
WHERE ServiceName = 'dns-control-cli' AND Timestamp > now() - INTERVAL 1 HOUR
ORDER BY Timestamp DESC

A credential in a span or a log record. Do not send the value to ClickHouse, because system.query_log keeps the text and the settings of each query. Hash the value on your machine instead. For the pdns API key, run this on the control node with a Vault session. It prints the hash only:

Terminal window
vault kv get -mount=secret -field=api_key fzymgc-house/infrastructure/dns/pdns-api \
| tr -d '\n' | shasum -a 256 | cut -d' ' -f1

The query hashes each run of letters and digits in each span and each log record, and it compares each hash with the hash of the value. It finds a value that holds only letters and digits, such as the pdns API key. Put the hash in secret_hash. The answer must be 0 in secret_rows for both sources. To make sure that the query works, put the hash of a word that a recent log record holds, such as the key of an entry that you projected, in control_hash. control_rows must then be greater than 0:

WITH '<sha256 of the value>' AS secret_hash, '<sha256 of a known word>' AS control_hash
SELECT 'trace' AS source, count() AS rows_searched,
countIf(arrayExists(t -> lower(hex(SHA256(t))) = secret_hash, tokens)) AS secret_rows,
countIf(arrayExists(t -> lower(hex(SHA256(t))) = control_hash, tokens)) AS control_rows
FROM (SELECT extractAll(concat(toString(SpanAttributes), ' ', toString(`Events.Attributes`), ' ',
toString(ResourceAttributes), ' ', StatusMessage), '[A-Za-z0-9]+') AS tokens
FROM default.otel_traces
WHERE Timestamp > now() - INTERVAL 1 DAY AND ServiceName LIKE 'dns-%')
UNION ALL
SELECT 'log', count(),
countIf(arrayExists(t -> lower(hex(SHA256(t))) = secret_hash, tokens)),
countIf(arrayExists(t -> lower(hex(SHA256(t))) = control_hash, tokens))
FROM (SELECT extractAll(concat(Body, ' ', toString(LogAttributes), ' ', toString(ResourceAttributes)),
'[A-Za-z0-9]+') AS tokens
FROM default.otel_logs
WHERE Timestamp > now() - INTERVAL 1 DAY AND ServiceName LIKE 'dns-%')

Two systems page the operator through Pushover about dns-control. ClickStack alerts read the telemetry of dns-control and of the node workers. Uptime Kuma monitors check dns-control and the resolver VIPs from outside, so they keep working when ClickStack is down. Each alert message and each monitor description names its entry in this section.

tf/uptime-kuma/monitors-dns.tf holds the three monitors. Each monitor runs a check every 60 seconds. After a failed check, it tries 3 more times, 60 seconds apart. If each try fails, the monitor reads DOWN and pages through the notification Pushover (cluster-edge).

Each check starts in the Uptime Kuma pod. If the reading from a workstation passes and the monitor reads DOWN, the fault is on the path from the pod. dns-control and the resolvers are then not the cause.

The monitor requests https://dns-control.fzymgc.house/readyz and expects the status 200. The request goes through the public name, Traefik and the certificate dns-control-public-tls to the pod. The pod answers 200 when its configuration parsed and the first record index exists.

  1. On a workstation, run curl -s -o /dev/null -w '%{http_code}\n' https://dns-control.fzymgc.house/readyz.
  2. If the status is 503, the pod is not Ready, and Traefik has no pod for the request. Follow The record index does not finish.
  3. If the status is 404, Traefik has no route for the name. Make sure that the IngressRoute dns-control exists and that ArgoCD shows the Application dns-control as Synced.
  4. If curl reports a certificate error, make sure that the Certificate dns-control-public-tls in the namespace dns-control is Ready.

The usual causes are these:

  • The pod restarted. The first index takes about 30 seconds, and the retries of the monitor cover that time.
  • The database or a pdns node was down when the pod started, so the first index did not finish.
  • A sync of the Application dns-control failed.

The monitor sends an A query for nas.fzymgc.house to the Core VIP 192.168.40.5 on port 53. It passes when the answer holds an A record. The main recursor process on the node that holds the VIPs answers, with the record from pdns. nas is a curated entry of dns-control. The monitor does not compare the address in the answer.

  1. On a workstation, run dig @192.168.40.5 nas.fzymgc.house A.
  2. If the query times out, no node answers on the VIP. Read the VIP holder on the Health screen of dns-control.
  3. If the VIP holder is none, follow What a failover looks like.
  4. If the status is NXDOMAIN, or the answer holds no A record, search for nas on the Search screen of dns-control.

The usual causes are these:

  • Both resolver nodes are in the keepalived state FAULT, so no node holds the VIPs.
  • The main recursor process pdns-recursor@main stopped on the node that holds the VIPs.
  • A change deleted or renamed the entry nas.

This monitor is the same as DNS resolver VIP (Core), with the Lab VIP 192.168.200.5. The Lab VIP is the resolver for the clients on the Lab VLAN. The same node holds all the VIPs, so a fault on the holder usually fails both monitors.

  1. On a workstation, run dig @192.168.200.5 nas.fzymgc.house A.
  2. If the Core monitor reads UP and this monitor reads DOWN, the Lab leg of the node that holds the VIPs is the fault. On that node, run ip -br addr and make sure that 192.168.200.5 is on the Lab interface.
  3. If both monitors read DOWN, follow the steps of DNS resolver VIP (Core).

The usual causes are the causes of the Core monitor, and a fault on the Lab leg of the holder.

argocd/app-configs/clickstack-alerts/bootstrap-script.yaml holds the HyperDX dashboard DNS control plane and its 13 tile alerts. Each alert pages through the webhook Pushover (cluster-infra), and its page links to its tile. The tile is the first reading. The interval of each alert is in Per-alert interval map.

Do not re-sync the Application clickstack-alerts while you read an alert that fires. A sync sets each alert back to OK. The counter tiles add delta points, as Metrics describes.

The alert reads the sum of dns_control_change_applies_total with surface mcp. It fires when more than 20 changes with a final outcome were applied through MCP in 15 minutes. An apply through the UI, REST, the CLI or the system surface does not count.

  1. Run the query of What a call logs with AND LogAttributes['tool'] = 'dns_apply_change'. The column caller names the agent or the key.
  2. If the calls are not expected, stop the caller. Revoke a static key with Add or revoke a static key. For a Keycloak caller, take the role dns-writer from its user or its service account.

The usual causes are these:

  • An agent works through a large task that a person asked for.
  • An agent repeats one change in a loop.

The alert reads the sum of dns_control_change_applies_total with surface mcp and op delete. It fires on each delete of an entry, a zone or a record set through MCP, in a window of 5 minutes. A delete that a person asked an agent for also pages.

  1. Read the newest changes on the Changes screen, or with the tool dns_list_changes. The change row names the subject that the change deleted.
  2. Run the query of What a call logs with AND LogAttributes['tool'] = 'dns_apply_change' to find the caller.
  3. If nobody asked for the delete, read the subject before the change in before, zone_before or rrset_before of the change row. Plan a create with those values and apply it.

The usual causes are these:

  • A person asked an agent to delete a name. The page is expected.
  • An agent chose a delete that nobody asked for.

The alert reads the sum of dns_control_change_refusals_total with surface mcp. It fires when dns-control refused more than 10 plans or applies through MCP in 15 minutes.

  1. Read the refusals by step and reason:

    SELECT Attributes['step'] AS step, Attributes['reason'] AS reason, sum(Value) AS refusals
    FROM default.otel_metrics_sum
    WHERE MetricName = 'dns_control_change_refusals_total' AND Attributes['surface'] = 'mcp'
    AND TimeUnix > now() - INTERVAL 1 HOUR
    GROUP BY step, reason
    ORDER BY refusals DESC
  2. Run the query of What a call logs to find the caller.

The usual causes are these:

  • role_required: a caller without dns-writer tries to plan or apply.
  • A policy_* reason: an agent tries names that the write policy refuses.
  • change_in_progress: an agent applies again while a change is pending.
  • plan_expired: an agent applies a plan after its 30 minutes.

The alert reads the sum of dns_control_change_applies_total with outcome failed, from every surface. It fires on each change that ended with the outcome failed, in a window of 15 minutes.

  1. Read the newest failed change on the Changes screen, or with the tool dns_get_change. The change row lists each step of The workflow and its result.
  2. If pdns_write failed, follow Node unreadable for the writer node.
  3. If reservation_write failed, follow A node worker does not answer.

The usual causes are these:

  • The writer pdns node refused the write or did not answer.
  • A node worker did not answer during the reservation write.

The alert reads the maximum of dns_control_export_pending_age_seconds for each sink. It fires when a sink has not held a succeeded change for more than one hour. The tile shows the sink, nas or b2. Each sink retries for up to seven days, so an alert means a sink that is down, not a retry.

  1. Read the sink on the tile.
  2. Read the dns-control log lines of the export in ClickStack. Filter on Body ILIKE '%export%' in the query of When a screen shows a data alert.
  3. For the sink nas, make sure that RustFS on the NAS answers. See the RustFS runbook.
  4. For the sink b2, make sure that the application key dns-control-export of tf/backblaze/dns_control_export.tf can still write the bucket fzymgc-dns-control-export.

The usual causes are these:

  • RustFS on the NAS is down.
  • The B2 key expired, or a change removed its access to the bucket.

The alert reads the sum of dns_control_lineage_rrsets in the states orphan, conflict, content_drift and dangling with the finding settled. A finding settles after two readings and five minutes. The alert fires on any settled finding, in a window of 30 minutes, so an adopt inside that time pages nobody.

  1. Run dns-control drift --state <state> for each state, as Drift from the command line describes. The tool dns_list_drift and the Health screen show the same findings.
  2. If an RRset reads content_drift after a direct write to pdns, follow Adopt a break-glass edit.
  3. If an RRset reads orphan, no row of the store projects it. If dns-control must hold it, adopt it with the same adopt command.

The usual causes are these:

  • An operator wrote pdns directly and did not adopt the edit.
  • A writer outside dns-control wrote pdns.
  • A pdns write of a change failed, so a row of the store reads dangling.

The alert reads the sum of dns_control_lineage_node_disagreements with the finding settled. It fires when the pdns nodes sandiego and portland hold a settled disagreement on an RRset, in a window of 15 minutes. LightningStream copies a write between the nodes in seconds, so a copy in flight never settles.

  1. Read the node agreement on the Health screen, or with the tool dns_get_health.
  2. If the result is unreadable, follow Node unreadable.
  3. If the result is differ or missing, follow Convergence lag in the DNS runbook.

The usual causes are these:

  • LightningStream stopped on one node.
  • RustFS on the NAS is down, so LightningStream cannot copy the change.
  • A person wrote the pdns node that is not the writer.

The alert reads the maximum of dns_control_vip_holder_status for the status none or split. The operations poller reads the holder from the node workers. The alert fires when the VIPs have no holder or two holders, in a window of 5 minutes. The status not_read does not page.

  1. Read the VIP holder and the keepalived state of each node on the Health screen.
  2. If the status is none, follow What a failover looks like. The Uptime Kuma VIP monitors usually fire too.
  3. If the status is split, both nodes claim the VIPs. On each node, run journalctl -u keepalived -n 50 and read the state changes.

The usual causes are these:

  • Both nodes are in the keepalived state FAULT.
  • The two nodes do not receive the VRRP adverts of each other, so each node holds the VIPs.

The alert reads the maximum increase of dns_control_lineage_refresh_timestamp_seconds in the window. The refresh loop publishes an index every 30 seconds. The alert fires when the timestamp did not move in 15 minutes, and also when dns-control sends no metric.

  1. Look at the monitor dns-control readiness. If it reads DOWN, follow its entry.
  2. Read the spans lineage.refresh of dns-control in ClickStack. A span with the status Error names the read that failed. Then read the log lines with the query of When a screen shows a data alert.

The usual causes are these:

  • The dns-control pod is down, or it does not reach the collector.
  • The refresh loop stopped in a running pod.
  • dns-control started after an outage near the end of a window. The pod sent too few points to show a change, and the alert clears at its next evaluation.

The alert reads the maximum increase of dns_control_observed_sweep_timestamp_seconds, the time of the last good read of the UniFi client table. The sweep runs every 5 minutes. The alert fires when no sweep read the table in one hour.

A paused sweep fires this alert. If you paused the sweep with Pause and purge, the alert fires after one hour and stays in ALERT until the sweep runs again.

  1. Read the observed AAAA sweep on the Health screen, or with the tool dns_get_health. It shows the last good read, the last error and the lane size.
  2. If observed_aaaa.enabled is false in argocd/app-configs/dns-control/config.yaml, the alert is expected.
  3. Otherwise, follow When a sweep fails.

The usual causes are these:

  • An operator paused the sweep.
  • The read of the UniFi client table fails, for example on HTTP 429 or a timeout.

The alert reads the count of the series of dns_worker_drained. It fires when fewer than two node workers sent their gauges in 15 minutes.

The threshold counts one series for each worker. HyperDX counts a series, not a data point, and the resource of the worker holds no process id. Both workers therefore read 2, and a restart of a worker keeps its series. A process id or another value that changes at each start, added to the resource of the worker, would break this threshold.

  1. Find the node that is missing:

    SELECT ResourceAttributes['host.name'] AS node, max(TimeUnix) AS last
    FROM default.otel_metrics_gauge
    WHERE MetricName = 'dns_worker_drained' AND TimeUnix > now() - INTERVAL 1 HOUR
    GROUP BY node
  2. Follow A node worker does not answer for that node.

The usual causes are these:

  • The worker stopped on the node.
  • The collector on the node stopped, so the gauges of a running worker do not arrive.
  • The node is down.

The alert reads the maximum of dns_worker_drain_overdue. The worker sets it to 1 when a drain outlives its expected duration, 30 minutes by default. The alert fires in a window of 5 minutes and stays in ALERT until the undrain.

  1. Read the card of the node on the Health screen. It shows Drain overdue.
  2. If the work on the node is complete, follow The undrain.

The usual causes are these:

  • An operator did not undrain the node after the work.
  • The expected duration of the drain was shorter than the work.

The alert reads the count of the server spans of dns-control and dns-control-webhook with the status Error. A server span is one request to the REST, MCP or webhook listener, and it has the status Error on a 5xx answer. The alert fires on more than 10 such spans in 15 minutes.

  1. Count the error spans by service and route:

    SELECT ServiceName, SpanName, count() AS errors, any(TraceId) AS trace
    FROM default.otel_traces
    WHERE ServiceName IN ('dns-control', 'dns-control-webhook') AND SpanKind = 'Server'
    AND StatusCode = 'Error' AND Timestamp > now() - INTERVAL 1 HOUR
    GROUP BY ServiceName, SpanName
    ORDER BY errors DESC
  2. Open the trace of the busiest route in HyperDX. The client span with an error names the dependency that failed.

The usual causes are these:

  • The CNPG cluster main is down, so each route that reads the database fails.
  • A pdns node does not answer.
  • The webhook cannot reach dns-control, because the dns-control pod is down.

Start each investigation in ClickStack. The dns-control log lines are in default.otel_logs:

SELECT Timestamp, SeverityText, Body
FROM default.otel_logs
WHERE ServiceName = 'dns-control' AND Timestamp > now() - INTERVAL 1 HOUR
ORDER BY Timestamp DESC
LIMIT 100

Use the Kubernetes API only for the current state of the pod and its events. The binary reads its Secrets once at start. The Deployment carries the Reloader annotation, so Reloader restarts the pod when a Secret or ConfigMap that it uses changes.

The alert reads “dns-control failed to read sandiego.” or the same for portland. The data for that node on each screen is from the last good read.

  1. Read the dns-control log lines in ClickStack for the node name and the error.
  2. Make sure that the pdns API on the node answers on port 8081. From a k8s node, run curl -s -o /dev/null -w '%{http_code}\n' http://192.168.20.6:8081/api/v1/servers for sandiego or 192.168.20.7 for portland. Any HTTP status means that the port answers. A timeout means that a firewall or the pdns service stops the call.
  3. Make sure that the network policy dns-control-default allows egress to the node address on TCP 8081.
  4. The resolver firewall admits the pdns API from pdns_api_source_set only. Ansible derives that set from the k8s node addresses in ansible/inventory/group_vars/dns_hosts.yml. If a k8s node address changed, run the dns-common role on both resolvers with scripts/dns-playbook.sh.
  5. If the log shows 401, the pdns API key is wrong. dns-control reads pdns_api_key from fzymgc-house/infrastructure/dns/pdns-api. After a rotation of that key, the ExternalSecret refreshes within 5 minutes, and Reloader then restarts the pod.

The alert reads “dns-control failed to read the query log from ClickStack.” Search and Name detail still work.

  1. Read the dns-control log lines in ClickStack for the ClickHouse error.
  2. If the error is an authentication error, the password of dns_control_reader differs between dns-control and ClickHouse. Both read dns_control_reader_password from fzymgc-house/cluster/clickstack. See the ClickStack runbook for the seed of that key.
  3. Make sure that the network policy clickstack-default admits the dns-control namespace on TCP 8123.

A pull request that adds a key to the ExternalSecret clickstack-secret can fail to reach the cluster. ArgoCD reports clickstack as Synced, but the live ExternalSecret keeps the old key list, and a ClickHouse replica does not start. Issue #2412 tracks the cause. Until it closes, compare the key list of the live clickstack-secret with git after each such merge. If they differ, follow the unstick procedure in the ArgoCD runbook. The documented command removes the argocd.argoproj.io/tracking-id annotation, so add that annotation to the manifest before the apply.

The screen “dns-control is building the record index.” is normal for about 30 seconds after a start. The screen tries again every 5 seconds. The pod is not Ready until the first index exists, because /readyz waits for it. Until then, the Service sends no request to the pod, and each read of the index answers index_not_ready.

The first index waits for two inputs. First, the entry ledger reads the entries and the recent changes from the database. The ledger is the record of what dns-control wrote. Then the builder reads the pdns nodes. An index built without the ledger reads every curated RRset as an orphan, so dns-control publishes none.

If the screen stays, or the pod does not become Ready:

  1. Read the dns-control log lines in ClickStack.
  2. If the log shows waiting for the entry ledger before the first index build, and then entry ledger not reloaded every 10 seconds, the database read fails. The error of the second line names the cause. Make sure that the CNPG cluster main is up and that the ExternalSecret dns-control-db rendered DATABASE_URL.
  3. If the log shows a pdns node error, follow Node unreadable for the node that the log names.
  4. If the log has no lines, the pod did not start. Look at the pod events. A missing leg file or an empty Secret value stops the binary at start.
  • ClickStack: the dns_control_reader user and its password.
  • ArgoCD: the unstick procedure for an ExternalSecret array.
  • DNS: the resolver pair, pdns, the Ansible roles and the manual undrain.
  • Temporal: the namespace dns-control, the node gateway and the node workers.
  • NATS: the stream DNS_EVENTS and the dns-control user.
  • MCP gateway clients: the LiteLLM routes, the two headers and the Claude Code entries.
  • LiteLLM: the virtual keys and the key of dns-agent-triage.
  • Services Catalog and Secrets.