dns-control
Operational guide for dns-control, the control plane over the resolver pair.
What this is
Section titled “What this is”dns-control serves four screens at https://dns-control.fzymgc.house: Search, Name detail, Query
log and Health. It shows which writer owns each record on the pdns nodes sandiego and
portland, and whether the two nodes agree. dns-control changes no record. It has no button that
changes a record.
dns-control also runs four operations actions on the resolver nodes: a cache flush, a list refresh, a drain and an undrain. See Operations actions.
It reads both pdns nodes, over the pdns HTTP API on port 8081, by IP address only. It reads
ClickHouse, as the read-only user dns_control_reader, for the Query log and the Health metrics.
The index of records lives in memory. dns-control builds it from the pdns nodes at start, so a restart loses nothing.
| Property | Value |
|---|---|
| URL | https://dns-control.fzymgc.house |
| Namespace | dns-control |
| Workload | Deployment/dns-control, one replica, Service/dns-control on port 8080 (http) and port 8081 (mcp) |
| Image | ghcr.io/fzymgc-house/dns-control, pinned by version tag in the kustomization.yaml of the app. Argo CD Image Updater moves the pin |
| Source | apps/dns-control/ |
| Keycloak clients | dns-control, in tf/keycloak/dns_control.tf. mcp-dns and dns-agent-triage for the MCP path. dns-control-external-dns for the external-dns lane |
| MCP URL | https://llm.fzymgc.house/dns/mcp, through the LiteLLM route dns |
| Vault paths | fzymgc-house/cluster/dns-control/oidc, fzymgc-house/cluster/dns-control, fzymgc-house/infrastructure/dns/pdns-api, fzymgc-house/cluster/clickstack, fzymgc-house/cluster/postgres/users/main-dns-control. The ExternalSecret dns-control-db renders the database URL from the Postgres path, and the Deployment mounts the property nats_creds at /etc/dns-control/nats/dns-control.creds |
| Manifests | argocd/app-configs/dns-control/ |
| Logs | ClickStack, ServiceName = 'dns-control' |
| Audit store | The database dns_control on the CNPG cluster main, table ops_audit |
| Temporal | The namespace dns-control, the task queue dns-control for the workflows, and dns-node-<node> for each node worker |
| Events | The JetStream stream DNS_EVENTS in the NATS account DNS |
Configuration
Section titled “Configuration”dns-control loads its configuration from three layers. Each later layer replaces the values of the layers before it:
- The defaults in
apps/dns-control/internal/config. - The YAML file
/config/dns-control.yaml, the keydns-control.yamlof the ConfigMapdns-control-legs. A missing or empty file is not an error. - The environment variables that start with
DNS_CONTROL_.
One rule maps a variable to a key. Remove the prefix, change the rest to lower case, and read a
double underscore (__) as one level of nesting. DNS_CONTROL_OIDC_CLIENT_ID sets
oidc_client_id, and DNS_CONTROL_OBSERVED_AAAA__INTERVAL sets observed_aaaa.interval. An empty
variable counts as unset. OTEL_EXPORTER_OTLP_ENDPOINT keeps its standard name, because the
OpenTelemetry SDK defines it. The SDK reads OTEL_EXPORTER_OTLP_HEADERS and
OTEL_RESOURCE_ATTRIBUTES itself. Each process sets its service name in code, so
OTEL_SERVICE_NAME changes nothing (see Telemetry).
The decode is strict. If a key is not known, dns-control does not start, and the error names the
key, for example 'config.Config' has invalid keys: http_adrr. A duration is a Go duration string
such as 30s, and a bare number is refused. The pod sets enableServiceLinks: false. Without it,
Kubernetes sets DNS_CONTROL_SERVICE_HOST and other variables for the Service dns-control, and
the strict decode refuses them.
A change takes effect at the next start. Reloader restarts the pod when the ConfigMap or the Secret
dns-control changes.
| Key | Variable | Default and limits |
|---|---|---|
http_addr |
DNS_CONTROL_HTTP_ADDR |
:8080 |
request_timeout |
DNS_CONTROL_REQUEST_TIMEOUT |
30s |
otel_exporter_otlp_endpoint |
OTEL_EXPORTER_OTLP_ENDPOINT |
Empty, which turns telemetry off |
oidc_issuer_url |
DNS_CONTROL_OIDC_ISSUER_URL |
https://id.fzymgc.house/realms/fzymgc |
oidc_client_id |
DNS_CONTROL_OIDC_CLIENT_ID |
dns-control |
oidc_client_secret |
DNS_CONTROL_OIDC_CLIENT_SECRET |
Required |
oidc_redirect_url |
DNS_CONTROL_OIDC_REDIRECT_URL |
https://dns-control.fzymgc.house/auth/callback |
oidc_post_logout_redirect_url |
DNS_CONTROL_OIDC_POST_LOGOUT_REDIRECT_URL |
https://dns-control.fzymgc.house/ |
extra_ca_file |
DNS_CONTROL_EXTRA_CA_FILE |
Empty, which means the system roots only |
pdns_api_key |
DNS_CONTROL_PDNS_API_KEY |
Required |
legs_file |
DNS_CONTROL_LEGS_FILE |
/config/legs.json |
clickhouse_url |
DNS_CONTROL_CLICKHOUSE_URL |
http://cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc.cluster.local:8123 |
clickhouse_user |
DNS_CONTROL_CLICKHOUSE_USER |
dns_control_reader |
clickhouse_password |
DNS_CONTROL_CLICKHOUSE_PASSWORD |
Required |
zone_refresh_interval |
DNS_CONTROL_ZONE_REFRESH_INTERVAL |
30s |
mcp_addr |
DNS_CONTROL_MCP_ADDR |
:8081 |
mcp_audience |
DNS_CONTROL_MCP_AUDIENCE |
https://llm.fzymgc.house/dns/mcp |
mcp_resource_metadata_url |
DNS_CONTROL_MCP_RESOURCE_METADATA_URL |
https://llm.fzymgc.house/.well-known/oauth-protected-resource/dns/mcp |
mcp_static_key_hashes |
DNS_CONTROL_MCP_STATIC_KEY_HASHES |
Empty, which accepts no static key |
ui_base_url |
DNS_CONTROL_UI_BASE_URL |
https://dns-control.fzymgc.house |
temporal_addr |
DNS_CONTROL_TEMPORAL_ADDR |
temporal-frontend.temporal.svc.cluster.local:7233 |
temporal_namespace |
DNS_CONTROL_TEMPORAL_NAMESPACE |
dns-control |
database_url |
DNS_CONTROL_DATABASE_URL |
Required |
nats_url |
DNS_CONTROL_NATS_URL |
nats://nats.nats.svc.cluster.local:4222 |
nats_creds_file |
DNS_CONTROL_NATS_CREDS_FILE |
/etc/dns-control/nats/dns-control.creds |
nats_ca_file |
DNS_CONTROL_NATS_CA_FILE |
/etc/fzymgc-ca/ca.crt |
drain_default |
DNS_CONTROL_DRAIN_DEFAULT |
30m, from 1m to 1440m |
flush_guard_threshold |
DNS_CONTROL_FLUSH_GUARD_THRESHOLD |
0.05, greater than 0 and less than 1 |
ops_wait |
DNS_CONTROL_OPS_WAIT |
120s, less than 150s |
export_nas.endpoint, export_nas.bucket, export_nas.access_key, export_nas.secret_key |
DNS_CONTROL_EXPORT_NAS__ENDPOINT and so on |
Required |
export_nas.region |
DNS_CONTROL_EXPORT_NAS__REGION |
Empty |
export_b2.endpoint, export_b2.region, export_b2.bucket, export_b2.access_key, export_b2.secret_key |
DNS_CONTROL_EXPORT_B2__ENDPOINT and so on |
Required |
observed_aaaa.enabled |
DNS_CONTROL_OBSERVED_AAAA__ENABLED |
false |
observed_aaaa.interval |
DNS_CONTROL_OBSERVED_AAAA__INTERVAL |
300s, from 30s to 1h |
observed_aaaa.publish_after |
DNS_CONTROL_OBSERVED_AAAA__PUBLISH_AFTER |
2, from 1 to 10 |
observed_aaaa.retire_after |
DNS_CONTROL_OBSERVED_AAAA__RETIRE_AFTER |
30m, at least two intervals |
unifi.url |
DNS_CONTROL_UNIFI__URL |
https://unifi.fzymgc.house |
unifi.username, unifi.password |
DNS_CONTROL_UNIFI__USERNAME, DNS_CONTROL_UNIFI__PASSWORD |
Required when observed_aaaa.enabled is true |
unifi.writer_username, unifi.writer_password |
DNS_CONTROL_UNIFI__WRITER_USERNAME, DNS_CONTROL_UNIFI__WRITER_PASSWORD |
Required when unifi_alias.enabled is true |
unifi_alias.enabled |
DNS_CONTROL_UNIFI_ALIAS__ENABLED |
false |
unifi_alias.interval |
DNS_CONTROL_UNIFI_ALIAS__INTERVAL |
15m, from 1m to 1h |
The subcommands zones and entries load the same configuration, so they run inside the pod with
kubectl exec.
The webhook sidecar
Section titled “The webhook sidecar”The subcommand dns-control webhook has its own configuration, with the prefix
DNS_CONTROL_WEBHOOK_ and the same rule and the same strict decode. It has no file layer.
| Key | Variable | Default and limits |
|---|---|---|
addr |
DNS_CONTROL_WEBHOOK_ADDR |
127.0.0.1:8888, a loopback address only |
health_addr |
DNS_CONTROL_WEBHOOK_HEALTH_ADDR |
:8080 |
dns_control_url |
DNS_CONTROL_WEBHOOK_DNS_CONTROL_URL |
Required |
oidc_token_url |
DNS_CONTROL_WEBHOOK_OIDC_TOKEN_URL |
Required |
oidc_client_id |
DNS_CONTROL_WEBHOOK_OIDC_CLIENT_ID |
dns-control-external-dns |
oidc_client_secret |
DNS_CONTROL_WEBHOOK_OIDC_CLIENT_SECRET |
Required |
extra_ca_file |
DNS_CONTROL_WEBHOOK_EXTRA_CA_FILE |
Empty, which means the system roots only |
domain_filter |
DNS_CONTROL_WEBHOOK_DOMAIN_FILTER |
fzymgc.house, a comma-separated list |
domain_exclude |
DNS_CONTROL_WEBHOOK_DOMAIN_EXCLUDE |
lan.fzymgc.house, a comma-separated list |
budget |
DNS_CONTROL_WEBHOOK_BUDGET |
12s, greater than zero |
Sign-in and roles
Section titled “Sign-in and roles”dns-control checks the Keycloak sign-in itself. No proxy sits in front of it. The browser signs
in at https://id.fzymgc.house with the authorization code flow and PKCE. PKCE is a proof that
the code goes back to the client that asked for it. The browser then holds only a session cookie.
The Keycloak client dns-control has four client roles for people. One group grants each role:
| Group | Role | What the role allows today |
|---|---|---|
dns-readers |
dns-reader |
Every screen and every /api/* route |
dns-writers |
dns-writer |
The changes of curated entries: plan, apply and project |
dns-operators |
dns-operator |
The operations actions. See Who can run what |
dns-admins |
dns-admin |
Zones, their grants and the control lane. See Zones and record sets |
dns-control reads roles from resource_access["dns-control"].roles in the access token only.
A realm role or the role of another client does not count. The access token must also carry
dns-control in its aud claim. The audience mapper in tf/keycloak/dns_control.tf adds it.
After the sign-in, the sidebar lists the screens: Search, Query log, Health, Changes and Rules. The foot of the sidebar shows the name of the person, the roles from the token, Keys and Sign out. At 1024px and wider, the sidebar shows its labels, and its toggle button narrows it to icons. From 640px to 1023px, it shows icons only, and each label is the accessible name of its icon. Below 640px, the menu button at the top of the screen opens the sidebar.
Terraform writes the client secret to fzymgc-house/cluster/dns-control/oidc. The ExternalSecret
dns-control reads oidc_client_secret from that path. Do not put this secret by hand. The next
apply of the main-cluster-keycloak workspace replaces it.
On the first rollout, the ExternalSecret dns-control can report SecretSynced before the
Keycloak apply writes the OIDC secret. The pod then waits in CreateContainerConfigError. The
next refresh of the ExternalSecret, 5 minutes later at most, fills the key. The pod then starts
with no restart.
To give a person access:
- Add a
keycloak_user_groupsresource for the person totf/keycloak/dns_control.tf, in the shape ofoperator_dns_control. Use the groupkeycloak_group.dns_control["reader"]. For the operations actions, addkeycloak_group.dns_control["operator"]too. For the zone and record set changes, addkeycloak_group.dns_control["admin"]. - Merge the pull request. The
main-cluster-keycloakworkspace applies it. - Tell the person to sign out of dns-control and sign in again. A token carries the roles that were current when Keycloak issued it.
If a person sees “You do not have access to dns-control.”, the person is not in dns-readers.
Each tab of the app holds one event stream on /api/events, in the format of server-sent events.
An event names what changed, and the screen that shows it reads its own route again. The stream
checks the session every 15 seconds and ends when the session ends. dns-control keeps the
sessions in the memory of its process, so a restart of dns-control signs every tab out. The tab
then shows the sign-in link within 30 seconds.
Every screen takes the keys in this table. A key does nothing while a field, a menu or a dialog has the focus. No key uses Ctrl, Cmd or Alt, so the keys of the browser stay as they are. Ctrl+B or Cmd+B still narrows the sidebar.
| Key | What it does |
|---|---|
g then s |
Opens Search |
g then q |
Opens Query log |
g then h |
Opens Health |
g then c |
Opens Changes |
g then r |
Opens Rules |
/ |
Moves the focus to the search field of Search, Query log or Rules |
n |
Moves the focus to the name field of the add form on Rules, for a person with dns-writer |
? |
Opens the list of keys |
| Escape | Closes the list of keys, or moves the focus out of a search field and keeps its text |
After g, press the second key within 1.5 seconds. Any other key ends the wait.
The Keys button in the foot of the sidebar opens the same list. The list holds a switch that turns
every key off for this browser, as WCAG 2.1 success criterion 2.1.4 requires for a key of one
character. The browser keeps the value in localStorage under dns-control.keys. If the browser
blocks localStorage, the keys stay on. The Keys button opens the list also while the keys are
off.
Search
Section titled “Search”The Search screen finds record sets in the lineage index by a name pattern. The pattern input has a toggle with two modes, glob and regex. Glob is the default.
| Mode | Example | How it matches |
|---|---|---|
| glob | *.lab.fzymgc.house. |
* matches any run of characters, dots included. ? matches one character. The pattern matches the whole name. |
| regex | ^(nas|calgary)\. |
Go regular expression syntax, not case-sensitive. The pattern matches anywhere in the name. |
Every name in the index ends with a dot. In glob mode, the screen adds the trailing dot to a
pattern that ends with neither . nor *, so *.lab.fzymgc.house searches for
*.lab.fzymgc.house.. In regex mode, the screen sends the pattern as typed. Anchor a regex with
^ and $, and escape a dot as \.. The input colors the operators of the pattern while you
type. In regex mode, the input turns red and reads “Pattern does not compile” when the browser
cannot compile the pattern. The server compiles the pattern with Go regexp and refuses a
pattern that does not compile with invalid_filter for the field name.
The Type, Zone, Writer and State headers of the table each open a menu of the values that the last answer offered. A choice sets the filter and searches at once. A set filter fills its header icon and shows as a chip above the table. Select the chip to remove that filter. Below 768px, each record set is a card, and the four filters open from the Filters button.
The URL holds the pattern, its mode and every filter, so Back, Forward and a copied link open
the same view. A regex search adds mode=regex to the URL.
The search route takes the same mode parameter. A request with no mode is a glob search. A
mode other than glob or regex answers invalid_filter for the field mode:
curl -fsS -H "Authorization: Bearer $TOKEN" 'https://dns-control.fzymgc.house/api/search?name=%5Enas%5C.&mode=regex&limit=5'The status bar is the last row of the list. It shows the count, the page range, the pagination and the age of the index and of each node read. Changes and Query log have the same bar.
Query log
Section titled “Query log”The Query log screen shows the queries that the resolver pair answered, newest first.
The name pattern takes the glob and regex modes of Search. In glob mode, the screen adds the trailing dot that the resolver logs with every name. The column headers hold the other filters: Time, Result, Resolver and Node, Client, and Type. Each choice reads the log at once. A set filter marks its header and shows as a chip above the table. Select the chip to remove the filter, or select Clear all. Below 768px, the filters open from the Filters button.
The time range is a duration in whole minutes from 1m to 24h, such as 90m or 2h30m. The
default is 5m. The Time menu offers 5m, 15m, 1h, 6h and 24h, and a field for any other duration.
A link from Name detail or from an MCP tool escapes the glob operators of the name, so a wildcard
name such as *.lab.fzymgc.house. shows its own queries only.
The first read asks for about two screens of rows. When you scroll near the end of the rows, the screen reads the next older rows, until the start of the range. The status bar then says “More rows load as you scroll” while the range holds older rows. The list holds at most 1,000 rows. At that count the status bar names the bound, and a filter or a shorter range shows older rows.
New rows of the current filters appear above the rows on screen with no click. While at least one
tab holds an event stream, dns-control reads the time of the newest query log row from ClickStack
every 15 seconds. When that time advances, it sends a querylog event. The screen reads the rows
after its newest row 30 seconds after the first event, so two of those reads are at least 30
seconds apart. If you scrolled down, the rows on screen stay where they are. If more rows are new
than one read holds, the new rows replace the list. Rows older than the start of the range leave
the list. In system.query_log of ClickHouse, each read of the newest row carries the
log_comment dns_control_querylog_newest. Select Pause to keep the rows as they are. The status
bar then reads Paused. Select Resume to read the new rows once and follow them again. A submit of
the filters reads the log from its newest row, also while the screen is paused.
The Client cell names the client that asked. It shows the first label of the name of the client and then its address, on one line. The name is the PTR record that pdns serves for the address. If pdns holds no PTR, the cell shows the first label of the alias that UniFi holds for the device. With neither, the cell shows the address alone. Guest and Telework clients have no PTR, because Kea sends no DNS update on those legs.
Rest the pointer on the Client cell, or move focus to it with Tab, to open its card. The card shows these values:
- The full name, with a badge for each writer of its PTR. A Kea D2 badge marks a name that the device chose when it asked for its DHCP lease.
- The address.
- The device that the last good read of the UniFi client table holds at the address: the alias, the hostname, the vendor, the MAC address, the network, and wired or Wi-Fi. The card labels the hostname “Hostname (from the device)”, because the device reports it.
- The time of that read. The device is the one that held the address at that time, which is not always the one that sent the query.
The device values come from the observed AAAA sweep. With observed_aaaa.enabled false, the card
shows the name and the address only.
Each row names its resolver process by a neutral name: dns-resolver is the
process of the trusted legs and the tailnet, and dns-resolver-public is the process of Guest and
Telework. The Resolver filter takes these two names. A row that a policy zone decided carries a
policy object with the zone, the trigger that matched and the action: nxdomain,
nodata or passthru. A row that no policy zone decided has policy null. The Result label
reads Blocked for nxdomain and nodata, Allowed for passthru, Resolved for NOERROR with no
policy, and else the response code. The Result filter takes blocked, passthru or a response
code such as SERVFAIL.
Each card of a row opens from its own cell, so one card shows at a time. Rest the pointer on the
Result label to open its card with the response code and, for a row with a policy, the zone, the
trigger and the action. A person with dns-operator also finds Wipe from cache in that card, see
Wipe one name. A name that ends in an ellipsis opens a card with the full name.
The MCP path
Section titled “The MCP path”dns-control serves MCP on a second listener, port 8081. MCP (Model Context Protocol) is the
protocol that AI agents use to call tools. Only the LiteLLM proxy pods reach port 8081. The network
policy dns-control-default admits the litellm namespace on TCP 8081, and Traefik has no rule
on that port. The UI host does not serve MCP. https://dns-control.fzymgc.house/mcp answers 404.
Agents call https://llm.fzymgc.house/dns/mcp. The LiteLLM route dns relays each call to
http://dns-control.dns-control.svc.cluster.local:8081/mcp. Each call carries two headers:
| Header | Carries | Who reads it |
|---|---|---|
x-litellm-api-key: Bearer <virtual key> |
The LiteLLM virtual key | LiteLLM, for admission, limits and spend |
Authorization: Bearer <credential> |
A Keycloak access token or a static key | dns-control |
LiteLLM relays Authorization unchanged and checks nothing in it. Never put the virtual key in
Authorization. dns-control refuses a bearer that starts with sk- and logs the prefix only, but
the key already left LiteLLM. If a virtual key reached dns-control, rotate it. The route table and
the two-header contract of every route are in docs/operations/mcp-gateway-clients.md.
Who can call
Section titled “Who can call”Three kinds of caller exist:
| Caller | Credential | Where the credential lives |
|---|---|---|
| A person in Claude Code | A Keycloak access token from the public client mcp-dns, with PKCE and the scope mcp-dns |
Claude Code holds the token. The Keycloak group dns-readers grants the role |
| The triage agent | A client_credentials token from the Keycloak client dns-agent-triage |
The client id and secret at secret/fzymgc-house/cluster/dns-agent-triage/oidc. The virtual key at secret/fzymgc-house/cluster/litellm-keys/dns-agent-triage, property api_key |
| A service that is not an agent | A static key that starts with dnsk_ |
The digest in the property mcp_static_key_hashes at secret/fzymgc-house/cluster/dns-control. The caller holds the plaintext |
A client_credentials token is a token that a client gets with its own id and secret, with no
person in the flow. Terraform creates both Keycloak clients, in tf/keycloak/mcp_dns.tf and
tf/keycloak/dns_agent_triage.tf, and writes the secret of dns-agent-triage to Vault.
By default, Claude Code does not use the client mcp-dns. It registers its own client through
Keycloak DCR (dynamic client registration), and Keycloak creates that client with full scope
turned off. A role then reaches the token only through a role scope mapping of a client scope.
The scope mcp-dns therefore maps dns-reader, dns-writer and dns-operator and emits
preferred_username. The clients mcp-dns and dns-agent-triage have full scope turned off too. A token carries only the
mapped roles that the user or the service account holds.
Every caller needs the dns-reader role. dns-control verifies each Keycloak token itself. It
makes sure that the token has:
- The issuer
https://id.fzymgc.house/realms/fzymgcand a valid signature from the realm keys. - The audience
https://llm.fzymgc.house/dns/mcp. The scopemcp-dnsadds it. - The
dns-readerrole inresource_access["dns-control"].roles. - An expiry in the future.
A token for another audience gets 401. A static key gives the identity key:<name> with the
dns-reader role. dns-control reads no forwarded identity header. The identity comes from the
verified credential only.
An MCP client finds Keycloak through the document at
https://llm.fzymgc.house/.well-known/oauth-protected-resource/dns/mcp. dns-control serves the
document on port 8081, and LiteLLM relays it. The document names Keycloak as the authorization
server and lists the scopes mcp-dns and offline_access.
The tools
Section titled “The tools”dns-control has eleven tools. The six tools in this table are read-only. The five operations
tools are in The MCP tools. No tool changes a record. LiteLLM joins the route
name and the tool name with -, so a client sees dns-dns_lookup_name and so on.
| Tool | Relayed name | What it answers |
|---|---|---|
dns_lookup_name |
dns-dns_lookup_name |
One name: its RRsets, writers, lineage state, node agreement and what each leg sees |
dns_search_records |
dns-dns_search_records |
The RRsets that match a name pattern, zone, writer, type or lineage state |
dns_query_log |
dns-dns_query_log |
The query log of the resolver pair, newest first, by name, client and time span, with the policy that decided each row. Each row names its client: the PTR name of the address with the writers of that PTR, and the UniFi device from the station copy |
dns_get_health |
dns-dns_get_health |
The health of the resolver pair and the freshness of the index, the node worker readings, the VIP holder, the recent actions of each node, the audit status and the observed AAAA sweep |
dns_list_drift |
dns-dns_list_drift |
The RRsets in an orphan, conflict, content drift or dangling state |
dns_explain_answer |
dns-dns_explain_answer |
Why a client got the answer it got for a name, step by step, with the evidence |
Each answer is JSON Lines text in the content of the tool result. A list answer is one header
line, then one JSON line for each row. The header line holds the total and the paging values. A
single answer is one JSON line. dns-control removes the null and empty members. The limit
argument has the default 25 and the maximum 100. Each answer carries the link to the same view in
dns-control, in the member url.
dns_search_records takes an optional mode argument, glob or regex, with the default
glob. It has the same meaning as the mode parameter of the search route. See
Search. A regex that does not compile, or a mode other than the two, answers
invalid_filter with the field name. The url of a regex answer carries mode=regex.
What a call logs
Section titled “What a call logs”Each tool call writes one log line with the message mcp tool call. The line holds these fields:
| Field | Value |
|---|---|
tool |
The tool name without the dns- prefix |
sub |
The subject of the token, or key:<name> for a static key |
azp |
The Keycloak client that got the token. It is empty for a static key |
preferred_username |
The person, the service account of the agent, or the name of the static key |
kind |
keycloak or static |
args |
The arguments of the call, as JSON |
The line never holds a token or a key. The fields are log attributes, and the row carries the trace id of the tool span. Read the calls in ClickStack:
SELECT Timestamp, LogAttributes['tool'] AS tool, LogAttributes['preferred_username'] AS caller, LogAttributes['args'] AS args, TraceIdFROM default.otel_logsWHERE ServiceName = 'dns-control' AND Body = 'mcp tool call' AND Timestamp > now() - INTERVAL 1 HOURORDER BY Timestamp DESCLIMIT 100Add or revoke a static key
Section titled “Add or revoke a static key”A static key is dnsk_ and 43 base64url characters from 32 random bytes. The Vault property
mcp_static_key_hashes holds one line <name>:<sha256 hex> for each key. The digest covers the
whole key, the prefix included. A name has 1 to 32 characters from a-z, 0-9 and -. The pod
holds the digests only. If a line is malformed, dns-control stops at start.
The plaintext of the smoke key lives beside the digests, as the property mcp_smoke_key. It never
leaves Vault. The ExternalSecret dns-control names mcp_static_key_hashes only.
To add a key:
-
Generate the key and its digest in shell variables. Replace
<name>with the name of the caller.Terminal window NAME='<name>'KEY="dnsk_$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=')"HASH="$(printf '%s' "$KEY" | shasum -a 256 | cut -d' ' -f1)" -
Add the line to the property.
vault kv patchchanges the named property only. Do not usevault kv put, because it replaces every property at the path.Terminal window CUR="$(vault kv get -mount=secret -field=mcp_static_key_hashes fzymgc-house/cluster/dns-control)"printf '%s\n%s:%s\n' "$CUR" "$NAME" "$HASH" \| vault kv patch -mount=secret fzymgc-house/cluster/dns-control mcp_static_key_hashes=- -
Wait for the ExternalSecret to refresh, 5 minutes at most. Reloader then restarts the pod.
-
Give the plaintext in
$KEYto the caller out of band. Do not write it to a file or a ticket.
To revoke a key, remove its line:
vault kv get -mount=secret -field=mcp_static_key_hashes fzymgc-house/cluster/dns-control \ | rg -v '^<name>:' \ | vault kv patch -mount=secret fzymgc-house/cluster/dns-control mcp_static_key_hashes=-After the refresh and the restart, the key gets 401.
The one-time live readings
Section titled “The one-time live readings”Run these readings once after a deploy that changes the MCP path. They are not standing gates. Run them from a workstation with Vault access. The commands read each secret from Vault, so you paste no secret into the shell.
-
Mint a token for the triage agent and read its claims. Expect
audto holdhttps://llm.fzymgc.house/dns/mcpand notdns-control(D-08),azpto bedns-agent-triage, andresource_access.dns-control.rolesto holddns-reader.Terminal window CSECRET="$(vault kv get -field=oidc_client_secret secret/fzymgc-house/cluster/dns-agent-triage/oidc)"VKEY="$(vault kv get -field=api_key secret/fzymgc-house/cluster/litellm-keys/dns-agent-triage)"TOKEN="$(curl -sS https://id.fzymgc.house/realms/fzymgc/protocol/openid-connect/token \-u "dns-agent-triage:$CSECRET" -d grant_type=client_credentials | jq -r .access_token)"printf '%s' "$TOKEN" | cut -d. -f2 | tr '_-' '/+' \| jq -R '@base64d | fromjson | {aud, azp, typ, preferred_username, resource_access}'Then send the token to the REST API of dns-control. Expect
401, because an MCP token does not open the REST API.Terminal window curl -sS -o /dev/null -w '%{http_code}\n' https://dns-control.fzymgc.house/api/health \-H "Authorization: Bearer $TOKEN" -
Send
initializethrough the route. ExpectHTTP/2 200.Terminal window curl -sS -D - -o /dev/null -X POST https://llm.fzymgc.house/dns/mcp \-H "x-litellm-api-key: Bearer $VKEY" -H "Authorization: Bearer $TOKEN" \-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}' \| rg -i '^HTTP/|mcp-session-id' -
Send
tools/list. Expect the elevendns-tools. The log line of a later tool call showsazpdns-agent-triage. If the answer toinitializehad anmcp-session-idheader, send it on this call too.Terminal window curl -sS -X POST https://llm.fzymgc.house/dns/mcp \-H "x-litellm-api-key: Bearer $VKEY" -H "Authorization: Bearer $TOKEN" \-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \-d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | jq -r '.result.tools[].name' -
Send
tools/listwith the smoke key in place of the token. Expect the same eleven tools.Terminal window SKEY="$(vault kv get -mount=secret -field=mcp_smoke_key fzymgc-house/cluster/dns-control)"curl -sS -X POST https://llm.fzymgc.house/dns/mcp \-H "x-litellm-api-key: Bearer $VKEY" -H "Authorization: Bearer $SKEY" \-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \-d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | jq -r '.result.tools[].name' -
Add the
dnsentry to your Claude Code with your own virtual key. Then open/mcpin Claude Code, selectdns, and sign in through Keycloak in the browser. Calldns-dns_explain_answeronce.Terminal window claude mcp add --transport http --scope user --callback-port 51339 \dns https://llm.fzymgc.house/dns/mcp \--header "x-litellm-api-key: Bearer <your-virtual-key>"
If Keycloak refuses the dynamic client registration, add --client-id mcp-dns to the command.
Then set oauth.scopes of the dns entry in ~/.claude.json to mcp-dns offline_access, so the
token carries the audience. Do not put an Authorization header on the entry, because Claude Code
then turns off its sign-in for the server.
Operations actions
Section titled “Operations actions”dns-control runs five actions on the resolver nodes. An action changes the state of a node. It never changes a record.
| Action | What the node does | Target |
|---|---|---|
| Flush | The node wipes the root name and every name under it from the caches of both recursor processes, through the API of each process. Until 2026-10-14 a node ran Blocky, and the flush dropped the whole cache of blocky and then of blocky-public |
One node or both |
| Refresh | The node worker starts dns-feeds.service and waits up to 5 minutes until the unit ends. The unit downloads every list source with a URL again, writes the feed zones, reloads both recursor processes and drops every cached answer. dns-feeds.timer also runs it once a day |
One node or both |
| Drain | The node writes the drain marker and stops keepalived, so the peer takes every VIP | One node |
| Undrain | The node removes the drain marker and starts keepalived | One node |
| Wipe | Both resolver processes remove one name, or the name and every name under it, from their caches. See Wipe one name | Both nodes |
A person starts an action from the card of a node on the Health screen. The card has the buttons Flush cache, Refresh lists, and Drain node or Undrain node. The screen also has Flush cache on both nodes and Refresh lists on both nodes. An agent starts an action through an MCP tool, and a script starts one through the REST API. All three surfaces call one operations service, so the role check, the flush guard and the audit row exist once.
The node worker on each Pi runs the action. The node worker is a Temporal worker that polls the
task queue of its own node, for example dns-node-sandiego. See the Temporal runbook,
The node workers. For each action, dns-control starts
the Temporal workflow ops.action on its own task queue dns-control. The workflow reads the
status of the node before and after the action, and the audit row keeps both readings.
One action runs at a time. The workflow id dns-ops is the lock. A second action while one runs
gets action_in_progress and a refused audit row. Each node activity fails after 10 seconds when
no worker polls the queue of the node, and each action runs once with no retry. Temporal ends a
run of ops.action after 40 minutes, and the end of the run releases the lock. This limit is
longer than the 30 minutes for which the workflow retries the audit store and the publish.
The start and the wait
Section titled “The start and the wait”dns-control writes the pending row and then starts the workflow. The start runs apart from the
request, with its own limit of 10 seconds. If the client closes the connection, the start goes on,
and the row does not become failed.
dns-control then waits at most 120 seconds for the result (ops_wait). If the wait ends first, the
answer holds the pending row, and the workflow goes on. dns-control then logs
no workflow result within the wait at Warn with the audit_id. ops_wait must be less than 150
seconds, the HTTP write timeout of both listeners. With a larger value, dns-control does not start.
If the workflow fails, dns-control logs the action workflow failed at Error with the audit_id.
The answer is 200 with the row as it stands, and that row is often still pending. When the pod
gets SIGTERM, dns-control ends every wait for a result. Each caller then gets the row as it stands.
If the start fails and Postgres fails at the same moment, the row stays pending. dns-control then
logs the audit row of a lost start stays pending at Error.
The routes
Section titled “The routes”The REST routes need a session cookie or a bearer token for the client dns-control.
| Route | Body or query |
|---|---|
POST /api/ops/flush |
{"target":"sandiego","force":false}. The target is a node or both |
POST /api/ops/refresh |
{"target":"both"} |
POST /api/ops/drain |
{"node":"portland","expected_minutes":60}. With no expected_minutes, the drain takes the default |
POST /api/ops/undrain |
{"node":"portland"} |
POST /api/ops/wipe |
{"name":"www.example.com","subtree":false}. The wipe runs on both nodes and takes no target |
GET /api/ops/preflight |
?action=drain&target=portland. It answers the guard readings of a flush or a refresh, or the five peer checks of a drain. It needs dns-operator, and a drain or an undrain preflight also needs a person. It writes no audit row, for a refusal too |
A route reads at most 4096 bytes of one JSON object and refuses an unknown field. A flush or a
refresh body names target, a drain or an undrain body names node, and a wipe body names name
and subtree. The caller always comes from the verified credential and never from the body.
If dns-control accepts the request, the answer is 200 with the audit row. A guard refusal and a failed node action also answer 200, and the row holds the outcome and the reason. The other answers are:
| Code | HTTP status | Meaning | Audit row |
|---|---|---|---|
invalid_action |
400 | A field of the request is not valid. The answer names the field | None |
role_required |
403 | The caller does not hold dns-operator |
Refused |
identity_not_allowed |
403 | The caller is not a person, and the action needs one | Refused |
origin_mismatch |
403 | The Origin header of an action POST does not equal UI_BASE_URL |
None |
action_in_progress |
409 | Another action runs | Refused |
operations_unavailable |
503 | The audit store or Temporal did not answer | None |
internal_error |
500 | Any other error | None |
A browser request must carry an Origin header equal to UI_BASE_URL. If UI_BASE_URL is
empty, dns-control refuses every POST from a session. A bearer request from a script carries no
Origin. If it carries one, the value must equal UI_BASE_URL too. Each of these refusals answers
origin_mismatch. Every answer of the routes carries Cache-Control: no-store.
GET /api/health and the MCP tool dns_get_health carry the operations readings too:
| Field | Where | Value |
|---|---|---|
worker |
Each node | The newest node worker reading: keepalived state, VIP count, drain marker with overdue, the engine of the node (recursor on both nodes), one reading per resolver process in resolvers, check scripts and certificate window. answering is false when the last read failed. Then only last_good_read_at and poller carry a value |
recent_actions |
Each node | The five newest audit rows that name the node |
vip_holder |
Top level | one_holder with the node, none, split or not_read |
audit_status |
Top level | ok or audit_unavailable |
dns-control reads the status of each node every 30 seconds. Health never shows a drain state or a VIP count from an older reading.
The node worker leaves the VIP count out of a reading when it cannot read the interface addresses.
vips_held is then null. dns-control never reads a missing count as 0. A flush or a refresh of
both nodes then refuses with vip_holder_not_one, and the drain counts that peer reading as a
failed reading. If the worker cannot read the drain marker, drain is null and drain_unknown is
true. The card of the node then shows No reading for the drain state. The worker reads
the state of each keepalived instance from its file in /run/keepalived-state. If a file does not
read, the keepalived state of an active unit is UNKNOWN. A stopped unit reads STOPPED.
Who can run what
Section titled “Who can run what”| Action | Role | Caller |
|---|---|---|
| Flush and refresh | dns-operator |
A person or a client |
| Wipe | dns-operator |
A person or a client |
| Forced flush | dns-operator |
A person only |
| Drain and undrain | dns-operator |
A person only |
Read the history with dns_list_operations or on Health |
dns-reader |
Any caller |
dns-control reads the kind of each caller from the verified credential:
personis a token that a person got through a sign-in, in the browser or in Claude Code.clientis aclient_credentialstoken, for example the token of the triage agent.staticis a static key that starts withdnsk_.
A static key carries dns-reader only, so it gets role_required on every action. The triage
service account holds dns-reader only, so it also gets role_required. A client token that
holds dns-operator gets identity_not_allowed on a drain, an undrain or a forced flush. Each
refusal writes an audit row with the caller: sub, azp, preferred_username and the kind.
To give a person the actions, add the person to the group dns-operators. See
Sign-in and roles.
The Health screen during an action
Section titled “The Health screen during an action”Health opens with four tiles: Node agreement, VIP holder, Node workers and Record index. Each tile shows one value and one line of detail. Its left edge has the worst tone of the facts that it sums up. A pending difference between the nodes is warn, and a settled one is destructive. The status bar under the tiles shows the age of the readings and the Refresh readings button.
Health reads /api/health when it opens and after each health, ops and index event of the
event stream. dns-control sends health after each read of the node workers, every 30 seconds.
It sends ops when an action ends, and index when the content of the record index changes. A
hidden tab reads once when it is visible again. Refresh readings reads at once. The header shows
the state of the stream: Live while it is open, and Reconnecting while it is down.
Each node card has one health badge, in one of three tones:
| Tone | Badge text | When |
|---|---|---|
| ok | Healthy | The worker answers, both recursor processes run, and both resolver checks pass |
| warn | Drained, No reading | The node is drained, or the drain marker, a resolver process or a check has no reading |
| destructive | Not answering, Resolver stopped, Check failing | The worker does not answer, a resolver process does not run, or a check fails |
The readings of a node card sit under tabs: Worker, pdns, Resolvers, LightningStream and Recent actions. Worker opens first. The action buttons sit at the foot of the card, under the tabs. Below 640px, each node card is one row with the name, the keepalived state and the health badge. A tap on the row opens the Worker readings and the action buttons.
Health reads the roles of the person from /api/me. If that request fails, Health shows no role
line and no action buttons. Health sends the request again at each read of /api/health.
The Last action panel shows the newest action of the tab. If the answer holds a pending row, the
action still runs. The panel then keeps the Running badge, the spinner and the running line, with
Started HH:MM:SS. Running for N seconds. Every action button stays disabled. The ops event of
the end of the action makes Health read again. That read finds the final row with the same id in
the recent actions of a node. The panel then shows that row, and the buttons become active again.
The pending row can drop out of the five newest recent actions of every node before it finishes. The buttons then stay disabled until you reload the page. This is a known limit.
A guard refusal of a flush runs on no node. The panel then shows the reason and the button Review forced flush. The button opens the forced flush dialog with the node and the ratio of the guard line.
If dns-control answers origin_mismatch, the dialog shows the error “dns-control refused the
action from this page address.” Its text tells the person to open dns-control again from its
configured address.
The flush guard
Section titled “The flush guard”Before a flush, dns-control reads the upstream error ratio of each target node from ClickStack.
The ratio is dns_resolver_errors_total over dns_resolver_responses_total in the last 15
minutes, for the processes dns-resolver and dns-resolver-public. The collector on each node
copies the two counters of each recursor process under these names. These are the series of the
alert DNS public resolver failing (Guest/Telework). The worst process of a node decides.
| Reading of a target node | Without force | With force, by a person |
|---|---|---|
| Below 0.05 | The flush runs | The flush runs with forced true and no override |
| 0.05 or above | Refused with guard_above_limit |
The flush runs with the override guard_above_limit |
| No reading | Refused with guard_no_reading |
The flush runs with the override guard_no_reading |
The comparison is inclusive. A ratio of exactly 0.05 refuses the flush, and the alert fires at
exactly 0.05 too. The threshold comes from FLUSH_GUARD_THRESHOLD, with the default 0.05. A
flush clears the cache of both processes, so a reading needs responses in the window from both
dns-resolver and dns-resolver-public. A node has no reading when one of its processes has no
responses in the window, or when ClickStack does not answer.
A flush with no reading runs only when a person forces it. On the Health screen, a forced flush
takes a separate typed confirmation. An agent never forces. A client token with force true gets
identity_not_allowed. The audit row keeps the guard reading of each node, forced and the
override.
A refresh reads the same guard, because the feed unit drops every cached answer as a flush does.
A refresh takes no force. It runs only when every target node reads below the limit, and it is
refused with guard_above_limit or guard_no_reading otherwise, for a person too. To load the
feeds while the guard refuses, run sudo systemctl start dns-feeds.service on the node.
In the result of a flush, the audit row holds the count of entries that each process removed, as
process and count.
Both nodes
Section titled “Both nodes”A flush or a refresh with the target both runs on one node at a time. The node that does not
hold the VIPs goes first, and the VIP holder goes second. Each node gets its own readings before
and after the action, and the result reports each node on its own.
If the first node does not succeed, dns-control does not touch the second node. A first node that
gives no result counts as a node that did not succeed. The entry of the second node reads
refused with previous_node_failed. If the VIPs are split across the nodes,
or no node holds them, the action refuses with vip_holder_not_one and runs on no node. If a node
does not answer, the action refuses with node_not_answering.
Wipe one name
Section titled “Wipe one name”A wipe removes one name from the record cache, the negative cache and the packet cache of both
resolver processes on both nodes. With subtree, it also removes every name under the name. The
next query for the name goes upstream again. Use it when a client holds a stale answer for one
name and the whole cache must stay. The MCP tool dns_wipe_name, the route POST /api/ops/wipe
and the Result card of the Query log start a wipe.
A wipe refuses the root name ., a name that holds * and a name that is not a valid DNS name.
The refusal is invalid_action on the field name, and no audit row is written. The flush covers
the root name, and a wildcard is a pattern of the Query log, not a name. dns-control stores the
name in lower case with a trailing dot.
A wipe runs on both nodes, the node that holds no VIP first, like a flush of both nodes. It reads
no flush guard and takes no force. An agent identity with dns-operator can run it.
Every wipe writes one audit row with the action wipe, the columns wipe_name and
wipe_subtree, and one result per node. The result of a node holds wiped: the number of entries
that each process removed, as process (main or public) and count. A second wipe of the
same name counts zero on each process. The audit row of a flush holds the same counts, because a
flush is a wipe of the root name with the subtree. The guard,
force and the role of the flush stay as they are.
On the Query log, a person with dns-operator opens the Result card of a row and selects Wipe
from cache. The dialog names the query name of the row and the subtree choice. After the confirm,
it shows the outcome and the count of each process on each node, or the reason of a refusal. A
person without dns-operator sees no wipe control.
The drain
Section titled “The drain”A drain takes a node out of service. The node writes the drain marker
/var/lib/dns-worker/drained and stops keepalived, and the peer takes all seven VIPs. keepalived
stays stopped until an undrain, across a reboot too. The resolver runbook docs/operations/dns.md
describes the marker and the manual undrain, in
Drain and undrain through dns-control.
The node checks its peer at the time of the drain. It refuses the drain with
peer_reading_failed and the id of a check that fails:
| Check id | The peer passes when |
|---|---|
house_check |
keepalived-check-house.sh exits 0 |
public_check |
keepalived-check-public.sh exits 0 |
keepalived_active |
keepalived is active |
keepalived_not_fault |
The summary state and every instance state are MASTER or BACKUP. A state in FAULT, UNKNOWN or a value that the node does not know fails the check |
peer_not_drained |
The peer is not drained. If the worker of the peer cannot read its drain marker, the check fails with the value unknown |
A peer reading that is too old refuses with peer_reading_stale, and a node that is drained
already refuses with already_drained. A peer reading that names no node, or names the node
itself, refuses with peer_reading_failed.
The drain dialog on Health shows the five checks before the person confirms the drain. The person types the node name to confirm. The preflight reports the node as drained only when the node answered its last status reading.
After the stop, the workflow reads the peer every 2 seconds, 10 times at most. If the peer does
not hold all its VIPs, the workflow undrains the node and records failed with vips_not_moved.
If the workflow is cancelled during these reads, it still runs the undrain, the last reading of
the node, the finish of the audit row and the publish.
The drain activity can give no result, because it failed or ran out of time. The node can still
have written the marker or stopped keepalived. The workflow then reads the node and then the peer,
and records the VIP count of the peer when the reading has one. If the node shows a drain marker
or keepalived is not active, the workflow runs an undrain at once. The detail of the node entry is
then rollback undrain ran or rollback undrain failed. The outcome stays failed with
node_not_answering.
On the node, a failed stop of keepalived removes the marker again. If the activity ends during the stop, the stop can still complete, so the node keeps the marker. An undrain clears that state.
Each drain has an expected duration. The default is 30 minutes (DRAIN_DEFAULT), and it is also
the shortest value. The longest value is 1440 minutes. DRAIN_DEFAULT must lie between 1 and 1440
minutes. With another value, dns-control does not start. When a drain passes its expected
duration, dns-control shows it in four places:
- The card of the node on Health shows a warning, Drain overdue.
- The drain reading in
/api/healthanddns_get_healthhasoverduetrue. - dns-control publishes the event
dns.ops.drain.overdueonce for the drain. - The node worker sets the metric
dns_worker_drain_overdueto 1.
The undrain
Section titled “The undrain”An undrain removes the drain marker, starts keepalived and makes sure that keepalived is active.
The node refuses the undrain with not_drained only when it has no marker and keepalived is
active. With no marker and keepalived not active, the undrain starts keepalived. An undrain
therefore finishes a drain or an undrain that stopped half way. If the start of keepalived fails,
run the undrain again.
A failed node action
Section titled “A failed node action”The detail of a failed node action names the step and the cause:
- A command that exited with an error reads like
keepalived stop: exit 1: <standard error>. - A command that did not run to its end reads like
keepalived stop: <error>. - A failed call to the API of a recursor process names the process that failed,
mainorpublic, and the error. - A failed read, write or remove of the drain marker carries the error text.
The worker keeps the last 512 bytes of standard error at most, and it clips the whole detail to 512 characters.
Each failed or refused action also writes the log line node action did not succeed at WARN on
the node. The line holds action, node, outcome, reason, failed_check and detail. Read
it in ClickStack with the journal query in
A node worker does not answer.
The audit store
Section titled “The audit store”Every action attempt writes one row in the table ops_audit of the database dns_control. The
database is on the CNPG cluster main in the namespace postgres. A success, a failure, a guard
refusal, a precondition refusal and a role refusal each write a row. A read writes no row.
A row is pending while its workflow runs. The workflow then writes the outcome: succeeded,
failed or refused. The trigger ops_audit_append_only refuses every delete, and every change
to a row that is not pending. dns-control does not prune the rows.
Three activities in dns-control load the request of the row, finish the row and publish the
event. Each retries with no limit on the attempts for at most 30 minutes. The pause between two
attempts is 30 seconds at most. A fault of the audit store is retried. Only a missing row, or a row
that finished with another outcome, ends the activity at once with not_pending. If Postgres
answers again within 30 minutes, the row of an action that ran on a node still finishes.
dns-control connects as the Postgres user dns_control through DATABASE_URL. The password is in
secret/fzymgc-house/cluster/postgres/users/main-dns-control. The ExternalSecret dns-control-db
in argocd/app-configs/dns-control/external-secret-db.yaml renders the URL from that path with
sslmode=verify-full and sslrootcert=/etc/fzymgc-ca/ca.crt. The CNPG server presents a chain of
three certificates, and that file holds the top one, fzymgc Intermediate CA1, so the client
verifies the chain and the name main-rw.postgres.svc.cluster.local. At start, dns-control runs
its migrations and tries again every 10 seconds until they succeed. Until then, every action
answers operations_unavailable.
Read the last rows on the CNPG primary:
P=$(kubectl get pod -n postgres -l cnpg.io/cluster=main,cnpg.io/instanceRole=primary -o name | head -1)kubectl exec -n postgres $P -c postgres -- psql -U postgres -d dns_control -c \ "SELECT started_at, action, target, outcome, reason, forced, override, caller_username, caller_kind, surface FROM ops_audit ORDER BY started_at DESC LIMIT 20"Each write of a row also sends one OTel log event, dns_control.ops_audit, with the same audit id.
An accepted action writes twice, once for the pending row and once for the outcome. Read the copy
in ClickStack:
SELECT Timestamp, LogAttributes['dns_control.action'] AS action, LogAttributes['dns_control.outcome'] AS outcome, LogAttributes['dns_control.reason'] AS reason, LogAttributes['dns_control.caller_username'] AS callerFROM default.otel_logsWHERE ServiceName = 'dns-control' AND EventName = 'dns_control.ops_audit' AND Timestamp > now() - INTERVAL 1 DAYORDER BY Timestamp DESCLIMIT 50default.otel_logs keeps a row for 14 days (see ClickStack). The
Postgres table is the record, and the ClickStack copy is for search and charts.
The events
Section titled “The events”dns-control publishes events on the JetStream stream DNS_EVENTS in the NATS account DNS. The
stream carries events only. The audit store is the record.
| Subject | When |
|---|---|
dns.ops.<action>.<outcome>, for example dns.ops.flush.succeeded |
The workflow finished the audit row of the action |
dns.ops.drain.overdue |
A drain passed its expected duration |
The header Nats-Msg-Id of an action event is the audit id. For an overdue drain it is
drain-overdue:<node>:<drain start in unix seconds>. The stream drops a repeat of the same id
inside its duplicate window of 2 minutes. The payload is JSON with audit_id, action,
target_nodes, outcome, reason, surface, caller_username, caller_azp and at.
A refusal that dns-control writes before the workflow starts gets no event. This covers a role
refusal, an identity refusal, a guard refusal and action_in_progress. The workflow retries a
publish for at most 30 minutes. After that, it logs the failure and keeps the outcome. If NATS is
down, the actions still run. A consumer that misses an event reads the audit store. The stream and the
credentials of dns-control are in the NATS runbook, NATS.
The MCP tools
Section titled “The MCP tools”| Tool | Role | Caller | destructiveHint |
|---|---|---|---|
dns_flush_cache |
dns-operator |
A person or a client. force needs a person |
false |
dns_refresh_lists |
dns-operator |
A person or a client | false |
dns_drain_node |
dns-operator |
A person only | true |
dns_undrain_node |
dns-operator |
A person only | false |
dns_wipe_name |
dns-operator |
A person or a client | false |
dns_list_operations |
dns-reader |
Any caller | Read-only |
The five action tools call the same operations service as the REST routes, with the surface
mcp. dns_wipe_name takes name and subtree, see Wipe one name. Each action tool answers one JSON line: the audit row with the readings before and after.
A refusal answers the code and the message of the REST answer. The descriptions of
dns_drain_node and dns_undrain_node name the refusals: a static key gets role_required, and
a client token gets identity_not_allowed. The four action tools other than
the drain set destructiveHint to false on purpose, because MCP reads a missing value as true.
dns_list_operations answers the audit rows as JSON Lines, newest first. It filters by node
and by action, wipe included. The limit argument has the default 25 and the maximum 100.
Recovery
Section titled “Recovery”A stuck action lock
Section titled “A stuck action lock”Every action answers action_in_progress, and no action finishes. An action normally ends within
a few minutes. Temporal ends a run of dns-ops after 40 minutes, and the end releases the lock.
Follow these steps only when a workflow dns-ops runs for more than 40 minutes.
-
Read the workflow in the admin tools pod:
Terminal window kubectl exec -n temporal deploy/temporal-admintools -- \temporal workflow describe --workflow-id dns-ops --namespace dns-control -
Read the node readings on Health, or on the nodes. Make sure that no action runs on a node. A drained node has the marker, and a drain that runs has keepalived stopping or stopped:
Terminal window scripts/dns-adhoc.sh dns_hosts -m ansible.builtin.stat -a 'path=/var/lib/dns-worker/drained'scripts/dns-adhoc.sh dns_hosts -m ansible.builtin.command -a 'systemctl is-active keepalived' -
Only when the readings show that no action runs, end the workflow with a reason that names you and the cause:
Terminal window kubectl exec -n temporal deploy/temporal-admintools -- \temporal workflow terminate --workflow-id dns-ops --namespace dns-control \--reason "<your name>: stuck action lock, no action running"
The audit row of the terminated action stays pending. A run that Temporal ends at the limit also
leaves its row pending. Do not change it by hand. The pending row is the record that the action
did not finish. If a node shows a drain that nobody wanted, undrain
it from Health.
The audit store is down
Section titled “The audit store is down”Every action answers operations_unavailable, and Health shows audit_unavailable. The read
screens and the read tools keep serving.
-
Read the dns-control log lines in ClickStack for the Postgres error, with the query in When a screen shows a data alert.
-
Make sure that the CNPG cluster and the database are ready:
Terminal window kubectl get cluster main -n postgreskubectl get database dns-control -n postgres -
If the error is an authentication error, make sure that the managed role
dns_controland the variableDATABASE_URLof dns-control read the same password fromsecret/fzymgc-house/cluster/postgres/users/main-dns-control.
When Postgres answers again, dns-control connects with no restart.
A node worker does not answer
Section titled “A node worker does not answer”The card of the node on Health shows that the worker does not answer, and the actions on that
node refuse with node_not_answering.
-
Read the poller of the node queue in the admin tools pod. The output lists a poller when the worker polls:
Terminal window kubectl exec -n temporal deploy/temporal-admintools -- \temporal task-queue describe --task-queue dns-node-<node> --task-queue-type activity -n dns-control -
Read the log of the worker and the journal of its certificate unit in ClickStack. The worker sends its log over OTLP. The collector reads the journal of
dns-node-worker-pki.service:SELECT Timestamp, ResourceAttributes['host.name'] AS node, ServiceName,LogAttributes['_SYSTEMD_UNIT'] AS unit, BodyFROM default.otel_logsWHERE (ServiceName = 'dns-node-worker'OR (ServiceName = 'otelcol-dns' AND LogAttributes['_SYSTEMD_UNIT'] = 'dns-node-worker-pki.service'))AND Timestamp > now() - INTERVAL 1 HOURORDER BY Timestamp DESCLIMIT 100 -
Read the state of both units on the Pi:
Terminal window scripts/dns-adhoc.sh <node> -m ansible.builtin.command -a 'systemctl status dns-node-worker.service dns-node-worker-pki.service --no-pager' -
Read the decision log of the gateway. OPA writes each decision to the log of its container. A decision names the principal, the method, the namespace, the task queue and the result. It carries no other part of the request body:
Terminal window kubectl logs -n temporal deploy/temporal-node-gateway -c opa --tail=100kubectl logs -n temporal deploy/temporal-node-gateway -c envoy --tail=100 -
If the certificate expired, Health shows its window and the metric
dns_worker_client_cert_not_after_secondsindefault.otel_metrics_gaugeshows the end. Vault Agent indns-node-worker-pki.servicerenews it. The worker loads the new bundle at its next handshake with no restart. -
If the Envoy log shows a failed handshake, the client certificate can lack the URI SAN of the node. Follow Roll out the node identity.
-
If the journal of the worker shows
vips[<i>]: "<address>" repeats vips[<j>], the worker configuration names one VIP twice, also in the IPv4-mapped form. The worker does not start with that configuration. Correctvipindns_vlan_interfacesand converge the node.
A node worker that does not answer does not change DNS. The resolvers answer with or without it.
Entry changes
Section titled “Entry changes”dns-control holds the curated host entries. An entry is one device or service: its key, its hostname, its IPv4 address, an optional IPv6 address, the PTR flag, its aliases and an optional MAC. The entry is the truth. The pdns records, the Kea reservation and the export are projections of it. Sean decided this model on 2026-09-29, after issue #2527. Terraform no longer applies a curated record, so no second applier can put an old value back.
Plan then apply
Section titled “Plan then apply”A change runs in two steps. Plan stores the change with a diff and writes nothing. Apply writes the entry and starts one workflow. The plan expires after 30 minutes. A repeated apply of one plan changes nothing and returns the same change.
| Step | Route | Tool | Role |
|---|---|---|---|
| Plan a create, a change or a delete | POST /api/changes/plan |
dns_create_entry, dns_change_entry, dns_delete_entry |
dns-writer |
| Apply a plan | POST /api/changes/apply |
dns_apply_change |
dns-writer |
| Project every entry unchanged | POST /api/changes/project |
dns_project_entries |
dns-writer |
| Read the entries | GET /api/entries, GET /api/entries/{key} |
dns_list_entries |
dns-reader |
| Read the changes | GET /api/changes, GET /api/changes/{id} |
dns_list_changes, dns_get_change |
dns-reader |
The plan body carries op, key and, for a create or a change, entry with hostname,
address, address6, ptr, aliases, mac and comment. The apply body carries plan_id.
The UI has the same flow on the Changes screen and on the Name screen of a curated name.
The write policy refuses a name outside the apex zone, a name in lan.fzymgc.house, a name of
the reserved infrastructure families (dns.<leg>, <node>.<leg> and <node> under the apex),
and a name that external-dns or Kea D2 owns. It also refuses a name that a record set of the
store holds, with the reason policy_other_lane. The answer is write_policy with the
name. An entry with a MAC must sit on a trusted leg, or the answer is write_policy too.
The workflow
Section titled “The workflow”Apply writes the entry row and a pending change row in one transaction, then starts the Temporal
workflow change.apply on the task queue dns-control under the workflow id dns-change-<id>.
The pending change row is the lock, so one change runs at a time. The unique index
change_audit_one_pending allows one pending row only. A second apply while a change is pending
gets change_in_progress and writes nothing.
If the workflow of a pending change did not start, for example because Temporal was down, the next
apply starts it and gets change_in_progress. Send that apply again when the pending change is
final.
The workflow runs these steps in order:
pdns_write: replace each RRset of the entry on the writer node with its records and the commentwriter=dns-control change=<id>. For a delete, clear the comment and then remove the RRset. LightningStream replicates the write to the other node.reservation_write, when the change adds, moves or removes a reservation: send the full reservation set of every trusted leg to the node worker of the node that holds no VIP, then to the other node. The worker writes one file per leg under/var/lib/dns-worker/kea/and runsconfig-reloadon Kea. A node that fails stops the write to the next node.- Finish the change row with the outcome and each step.
- Start the workflow
change.exportfor the change and publish the eventdns.change.<op>.<outcome>.
The export workflow writes entries.json, the whole entry set, zones.json, every zone and record
set, and one audit object audit/<time>-<id>.json to the NAS bucket and to the B2 bucket. Each
sink has its own retries for up to seven days. A sink that already holds the same bytes of an
object does not get that object again. The change row shows
exported when both sinks hold it, and pending: <sink> until then. The export runs as its own
workflow, so a sink that is down does not hold the change lock.
The lineage index and drift
Section titled “The lineage index and drift”The lineage index reads the entry, zone and record set stores. A live RRset that the store projects
is attributed to dns-control. A record set of the external-dns lane is the one exception: it
reads as external-dns and carries the mark of dns-control. A live RRset whose content differs
from its row reads content_drift, and a row that pdns does not serve reads as dangling. The drift
comparison of an SOA leaves out the serial. dns_list_drift and the Health screen show both. A
hand edit of pdns therefore shows up on the next refresh.
Recovery
Section titled “Recovery”The entry set can be read back from either sink and imported into an empty store:
kubectl exec -n dns-control deploy/dns-control -- dns-control entries export --format json > entries.jsonkubectl exec -n dns-control deploy/dns-control -- dns-control entries export --format hosts-yml > hosts.ymlkubectl exec -i -n dns-control deploy/dns-control -- dns-control entries import --from /dev/stdin < hosts.ymlentries import reads the hosts.yml form, the form that tf/dns-records/hosts.yml had, and
writes one change per entry in the surface cli. It starts no workflow. After an import, run
POST /api/changes/project with an empty keys list, or the tool dns_project_entries, to
project the set onto pdns, Kea and the sinks. A projection applies each entry unchanged, one at a
time, and reloads Kea on both nodes for each entry with a MAC.
entries import --dry-run prints the parsed entries in hosts.yml form and writes nothing.
entries export --format tf-import prints one terraform import line per record, which is the
rollback of the Terraform handoff.
List entries
Section titled “List entries”dns-control holds the list entries of the resolver pair. A list entry is a block entry or an allow entry for one name. A block entry makes the trusted resolver process answer NXDOMAIN for the name. An allow entry lets the name resolve when a public list blocks it. An entry with no scope applies to every client of the trusted legs and the tailnet. An entry with a scope applies to the members of that scope only, see Scopes. No entry applies on Guest or Telework, which the public process serves.
The entries reach the nodes as response policy zones (RPZ): zones with one rule per name, which
pdns-auth serves and the main recursor process of each node loads, see
The list entries of dns-control. The entries
with no scope form the zone operator.rpz.internal., and the entries of a scope form the zone
<scope>.scope.rpz.internal.. A block entry is an NXDOMAIN rule, and an allow entry is a passthru
rule. A plain entry and a wildcard entry for one name give one rule each, so a plain entry wins
over the base rule of a wildcard for the same name. Until 2026-10-14 the nodes ran Blocky, and
the entries reached a node as two operator list files that the node worker wrote. The retire play
of the DNS runbook removes those files.
| Name | What it matches |
|---|---|
example.org |
That name only |
*.example.org |
example.org and every name under it |
A name holds one entry in each scope and one entry with no scope. To change the kind or the reason of an entry, delete it and create it again. The UI calls the screen Rules.
Rules reads every entry in one answer, and the browser searches, sorts and filters the list. The search field finds the text in the name, the scope, the reason or the actor, in any case. Each column header sorts its column, and a second selection reverses the order. The Kind, By and Time headers also hold a filter on the kind, the actor, and the age of the last change. The Scope column shows the scope of each entry, or No scope. Below 768px, the sort and the filters are in the sheet behind the Filters button. The Scopes section above the search lists every scope with its name, order, members, actor and time.
The address holds the view in five keys: q, kind, actor, since and sort. A reload or a
copied link opens the same view. A sort or a filter adds a history entry, so Back restores the view
before it. Typing in the search field adds no history entry.
| Key | Value |
|---|---|
q |
The search text |
kind |
block or allow |
actor |
The name of the actor |
since |
24h, 7d or 30d |
sort |
name, kind, scope, reason, actor or time, with a leading - for the reverse order. The default is name |
For example, #/rules?q=ads&kind=block&sort=-time shows the block entries that hold ads, newest
first. An unknown value of kind or since reads as no filter, and an unknown sort reads as
name.
Plan then apply a list entry
Section titled “Plan then apply a list entry”| Step | Route | Tool | Role |
|---|---|---|---|
| Plan a create or a delete | POST /api/list-entries/plan |
dns_create_list_entry, dns_delete_list_entry |
dns-writer |
| Apply a plan | POST /api/changes/apply |
dns_apply_change |
dns-writer |
| Write every entry to both nodes again | POST /api/changes/project with subject: block |
dns_project_entries with the subject block |
dns-writer |
| Read the entries | GET /api/list-entries |
dns_list_list_entries |
dns-reader |
| Plan a create, a change or a delete of a scope | POST /api/scopes/plan |
dns_create_scope, dns_change_scope, dns_delete_scope |
dns-writer |
| Read the scopes | GET /api/scopes |
dns_list_scopes |
dns-reader |
The plan body carries op (create or delete) and name. A create also carries kind
(block or allow) and reason, which holds 1 to 256 characters. An optional scope names the
scope of the entry, and a delete with a scope deletes the entry in that scope. The diff of the
plan names the entry with its scope and its kind before and after. A change of a list entry or of a
scope has the subject block, also for an allow entry.
On Rules, a person with dns-writer creates an entry in the add form between the header and the
list. The form holds the name, the kind (Block by default), the scope (No scope by default) and the
reason. Plan shows the diff under the form, and Apply applies the plan. The Delete button of a row
opens the delete dialog, which plans and applies the delete with the same steps. A person without
dns-writer sees no form and no Delete button.
Scopes
Section titled “Scopes”A scope is a named set of clients. A list entry in a scope applies to those clients only, and it decides before an entry with no scope. A scope has these fields:
| Field | Rule |
|---|---|
name |
One DNS label of 1 to 40 characters from a-z, 0-9 and -. It becomes a label of the zone <name>.scope.rpz.internal. |
order |
A whole number from 1 to 65535 that no other scope holds. A client in two scopes takes the scope with the lower order |
members |
1 to 64 members: an IP address, a CIDR prefix with no host bits, or the name of a trusted leg such as main or lab. Guest, Telework and an IPv4-mapped IPv6 prefix are refused |
The plan body of a scope carries op (create, change or delete) and name. A create and a
change also carry order and members, the whole new member list. A plan refuses a name that a
scope holds already, an order that another scope holds, and the delete of a scope that a list
entry names, each as invalid_entry with the field.
On Rules, the Scopes section holds New scope, and each scope has Change and Delete. Each opens the
scope dialog, which plans, shows the diff of the order and the members, and applies. A person
without dns-writer sees the scopes and no control.
On each node, the node worker writes the scopes as the map file
/var/lib/dns-worker/scopes.map and the include /etc/powerdns/recursor-main.d/50-scopes.yml. The
Lua script of the main process reads the map, and the include names the scope zones. A new scope
writes the map first, and a deleted scope writes the include first and deletes the dump file of
its zone, so a scope zone never applies to every client.
A plan refuses these names and stores nothing:
| Name | Answer |
|---|---|
Not a host name of 2 to 127 labels, with an optional leading *. |
invalid_entry on the field name |
| A name that holds an entry | invalid_entry on the field name |
| The 10,001st entry of one kind | invalid_entry on the field name |
| A name in a zone of the zone store, or a wildcard whose base is above such a zone | write_policy with the zone and the reason policy_list_zone |
The zone rule protects the pair. A block entry under fzymgc.house would fail the keepalived
health check of dns-health.fzymgc.house on both nodes at once. An allow entry there would unblock
the internal namespace for a client on no trusted leg.
dns-control does not hold the owned public zones (holomush.dev, fzymgc.net, hl-assets.dev) or
the excepted names of the pdns-recursor role. The allow zone allow.rpz.internal. holds those
names and comes before the operator zone, so a block entry for one of them is stored and has no
effect.
The workflow of a list change
Section titled “The workflow of a list change”Apply writes the list entry or the scope and a pending change row in one transaction, as for a host entry. The pending row is the lock, so one change runs at a time. The workflow reads the status of each node from its worker to find the node that holds no VIP, and runs these steps:
policy_writein dns-control. It writes every policy zone to pdns-auth on the writer node and records the serial of each zone it wrote inwritten. The policy zones never show in the zone lists of dns-control, and a zone change refuses a name underrpz.internal..- The steps of the node that holds no VIP, then the steps of the other node. A node that fails
stops the steps of the next node, which read
previous_node_failed. - Finish the change row, export and publish the event.
Each node takes up to three steps:
| Step | What the node worker does | What the step carries |
|---|---|---|
scope_write, only when the change touched a scope |
Writes the scope map and the include, reloads the Lua script and the YAML of the main process, and reads the map version back | map_version and map_sha256 |
list_write |
Waits until pdns-auth on the node serves each written serial, sends a NOTIFY for each zone to the main process, and waits until the process loaded the serial. It gives up after 120 seconds | written and loaded, the serials. A change that wrote no zone reads no zone to load |
cache_wipe, only for a change that names a client |
Removes each name of the change and every name under it from the caches of both processes | wiped, the count of each process |
A list change writes nothing to Kea. The export writes blocks.json beside entries.json and
zones.json, to both sinks. It holds the scopes sorted by order, each with its name, order and
members, and the name, the kind, the scope and the reason of each entry, sorted by name. The actor
and the time are in the audit object of each change.
A failed list change
Section titled “A failed list change”The entry stays in the store when a list change fails, so the store holds the truth. A projection
writes the whole set to both nodes again and changes no entry. Run it with dns_project_entries
and the subject block, because the Rules screen has no control for a projection.
| The step reads | What happened | What to do |
|---|---|---|
list_refresh_failed on list_write |
The zone load of the node failed. pdns-auth on the node did not serve a written serial, the NOTIFY failed, or the main process did not load the serial within 120 seconds. The detail names the zone. Each zone has refresh: 300, so the process loads the change within five minutes on its own |
Read rec_control --config-name=main rpzstatistics on the node. If the serial stays old, apply a projection |
pdns_write_failed on policy_write |
pdns-auth on the writer node did not take a policy zone. No node ran a step | Read the pdns-auth logs on the writer node, then apply a projection |
scope_write_failed |
The node did not write the scope map or the include, or the main process did not reload. The rules in a scope do not apply on that node yet | Read journalctl -u pdns-recursor@main on the node. Fix the cause, then apply a projection |
node_not_answering |
The node worker did not take the activity | Bring the node or its worker back, then apply a projection |
previous_node_failed |
The first node failed, so the second node was not written | Fix the first node, then apply a projection |
After a node is rebuilt, the dns-node-worker role seeds an empty scope map. Apply a projection
after the converge, so the node holds the scopes again.
If dns-control is down and an entry must change at once, edit the files on the nodes. See The list entries of dns-control. The next list change or projection writes the files from the store again.
Zones and record sets
Section titled “Zones and record sets”dns-control also holds every zone and every static record set as data. A zone row holds the kind, the SOA values, the apex NS set and the zone metadata. The SOA serial is not stored, because pdns raises it on each API write. A record set row holds one name and one type in a zone, with its TTL, its records and its lane. The SOA, the apex NS, the metadata and each record set in pdns are projections of these rows. Sean decided on 2026-10-01 that dns-control is the only writer of what the resolver pair serves. Kea D2 is the one other writer, and it writes only inside its grants.
A lane names the one writer that can change a record set. A name belongs to one lane.
| Writer | Role | Can write |
|---|---|---|
| A person or an agent | dns-writer |
Curated entries. See Entry changes |
| A person or an agent | dns-admin |
Zones, their TSIG-ALLOW-DNSUPDATE grant, the control lane |
| The external-dns webhook | dns-lane-external-dns |
The record sets of the external-dns lane |
| The observed AAAA sweep | dns-admin, as the caller system:observed-aaaa |
The AAAA record sets of the observed-aaaa lane. See Observed AAAA |
A plan at a name that another lane or a curated entry holds gets write_policy with the reason
policy_other_lane. A name that Kea D2 writes gets policy_other_writer. A
name inside a child zone of the store gets policy_zone. The NS or DS record set of a delegation
at the apex of the child is the exception. The external-dns lane also stays out of
lan.fzymgc.house and off the reserved names. The apply checks the lane rule again inside its
transaction, because another change can land between the plan and the apply.
The observed-aaaa lane writes beside Kea D2 at its names, inside a narrow scope. A plan of another
type gets policy_type. A zone that is not the dynamic zone of a trusted leg gets policy_zone. A
create or a change at a name with no Kea D2 A record gets policy_no_kea_name. An address outside
the ULA /64 of the leg gets policy_address. A delete needs no Kea D2 name.
The group dns-admins grants dns-admin, and the operator is in it. The MCP scope mcp-dns maps
dns-admin, so the MCP token of a member carries it.
The external-dns lane routes
Section titled “The external-dns lane routes”The external-dns lane has its own route group, /api/lanes/external-dns/. GET on rrsets
lists the record sets of the lane. POST on apply applies one batch: the records of one
external-dns sync as one change of the subject batch. A lookup of the changes by name finds each
record inside a batch.
Only a client token with dns-lane-external-dns reaches the group. A session, a static key, a
person and any other client get 401 or 403. Each body is a protobuf message of dnscontrol.lane.v1
in protojson, and both sides validate it.
The external-dns sidecar calls the group. It is the subcommand dns-control webhook of the
dns-control image. It serves the webhook API of external-dns on 127.0.0.1:8888, and it answers the
kubelet probes with GET /healthz on :8080. It takes a client-credentials
token of the Keycloak client dns-control-external-dns, whose secret is at
fzymgc-house/cluster/external-dns/oidc. argocd/app-configs/external-dns/values.yaml sets the
provider webhook and runs the sidecar from the dns-control image. The sidecar reads the one pin of
the image, as The image and its pin describes. The external-dns namespace holds no pdns
API key.
In a batch, a record that the lane policy refuses leaves the batch, and dns-control logs it with
the line external-dns record refused. The other records still apply. A registry TXT of
external-dns that claims a name that another lane or an entry holds is refused too. The plan reads
create, change or delete from the store, so a batch that external-dns sends again changes nothing.
The pdns step writes each record set of a batch with its own PATCHes, so a record that pdns refuses
fails alone. The step lists each record set that it did not write in failed. When a batch does
not succeed, its finalize puts each such record set back to its before side, so the store matches
pdns. external-dns then sees those records as missing and sends them again on its next sync.
external-dns waits 15 s for the sidecar and exits on a timeout. The sidecar therefore gives each
call to dns-control 12 s, set by budget, and the apply route of dns-control waits 10 s for
the workflow. A slower change answers pending and then 503, and the workflow goes on. Every
failure of dns-control answers 503, because external-dns retries a 5xx and exits on any other
error. A refused token or a refused body logs at ERROR in the sidecar, because it does not clear by
itself.
Plan and apply a zone or a record set
Section titled “Plan and apply a zone or a record set”| Step | Route | Tool | Role |
|---|---|---|---|
| Plan a zone create, change or delete | POST /api/zones/plan |
dns_create_zone, dns_change_zone, dns_delete_zone |
dns-admin |
| Plan a grant change | POST /api/zones/metadata/plan |
dns_change_zone_metadata |
dns-admin |
| Plan a record set create, change or delete | POST /api/rrsets/plan |
dns_create_rrset, dns_change_rrset, dns_delete_rrset |
dns-admin |
| Apply a plan | POST /api/changes/apply |
dns_apply_change |
dns-admin |
Project the zones or the control lane unchanged |
POST /api/changes/project with subject |
dns_project_entries with subject |
dns-admin |
| Read the zones | GET /api/zones, GET /api/zones/{name} |
dns_list_zones, dns_get_zone |
dns-reader |
| Read the record sets | GET /api/rrsets?zone=&lane= |
dns_list_rrsets |
dns-reader |
| Read the changes | GET /api/changes?subject=zone, GET /api/changes?subject=rrset |
dns_list_changes |
dns-reader |
The zone plan body carries op and zone with name, kind, soa, ns, ns_ttl and
metadata. The soa object carries mname, rname, refresh, retry, expire, minimum and
ttl. A delete carries op and name only. A change can alter the SOA values, the NS set and
TSIG-ALLOW-DNSUPDATE. The kind and every other metadata kind keep their values. A change with no
metadata key keeps the metadata of the row, so a change of the SOA keeps the grant.
A zone delete is refused for the apex and the two reverse zones of the curated entries. It is also refused while the zone holds record sets, and while pdns serves any RRset in the zone besides its SOA and apex NS, such as a Kea D2 lease. The pdns step makes the same comparison again before it deletes the zone.
The grant body carries zone, kind and values. An empty values list removes the grant. The
record set plan body carries op and rrset with zone, name, type, ttl and records. A
TXT record is quoted. The SOA and the apex NS belong to the zone, so a record set plan refuses
them. A plan is also refused for a CNAME at a zone apex, a CNAME beside another type at its name,
and another type beside a CNAME, because pdns refuses each one. A record set plan and a zone delete
wait for the first build of the lineage index. The UI has no zone editor. The Changes screen lists zone and record set changes by name.
The workflow of a zone or a record set
Section titled “The workflow of a zone or a record set”A zone or record set change runs the same change.apply workflow as an entry, under the same
lock, with no reservation step. The pdns_write step reads what the writer node serves and writes
only what differs:
- If pdns does not serve the zone, the step creates it with
soa_edit_apiDEFAULT, its SOA and its apex NS. Then it writes each metadata kind of the row. - If pdns serves the zone, the step replaces the SOA or the NS set when its values, its TTL or its
comment differ. It sends the live serial, and pdns raises it. If pdns serves the zone with
another kind or another
soa_edit_api, the step fails and writes nothing. - The step writes each metadata kind that differs. pdns loses a
TSIG-ALLOW-DNSUPDATEgrant that the row does not hold. - The step replaces a record set with its records, its TTL and the comment
writer=dns-control change=<id>. A disabled record in pdns counts as not served, so the step writes the RRset again. - A delete clears every comment first and then removes the RRset or the zone. The lmdb backend keeps the comments of a deleted RRset or zone for the next one with that name.
A retry of the step after a partial write repeats no write that landed.
Import from live pdns
Section titled “Import from live pdns”kubectl exec -n dns-control deploy/dns-control -- dns-control zones import --dry-runkubectl exec -n dns-control deploy/dns-control -- dns-control zones importzones import reads every zone, its metadata and its RRsets from the writer node. It builds the
lineage index from the reads of both nodes and the store. Then the import sorts each RRset by its
writer:
| Writer in the index | Goes to |
|---|---|
| None, for the SOA or the apex NS | The zone row |
| None, for any other type | A record set of the control lane |
| external-dns, its registry TXT records included | A record set of the external-dns lane |
| dns-control or Kea D2 | Nothing: skip writer_<writer> |
| Two writers | Nothing: skip conflict |
The import also skips a record that pdns holds disabled. Every zone must have soa_edit_api
DEFAULT, or the import stops with the name of the zone. Each zone keeps every metadata kind that
the pdns API can write. A kind that the API cannot write, such as API-RECTIFY or
TSIG-ALLOW-AXFR, gets the line skip protected_metadata with the type METADATA <kind>. The
command prints one line per zone, record set and skip, then the counts as one JSON line.
--dry-run stops there.
The import writes each row that the store does not hold as one change in the surface cli, with
the reason imported. It starts no workflow, and pdns does not change. If the store holds a row
with other content, the import does not write it, names it, and ends with an error. Run adopt
for each such row.
To roll the import back, run dns-control zones delete --imported. The command removes every row
that an import wrote, each as one change with the reason import_removed, and pdns does not
change. If a zone holds a record set from another source, the zone stays and the command stops
with its name.
dns-control zones export --format tf-import prints one Terraform import block per zone, SOA,
apex NS, metadata kind and control record set, in the import id forms of mmianl/powerdns. No
store holds the old Terraform addresses, so each block names a generated address such as
powerdns_zone.z_lab_lan_fzymgc_house. Run terraform plan -generate-config-out=generated.tf to
write the resource blocks. --format json prints the zones.json form of the store.
Adopt a break-glass edit
Section titled “Adopt a break-glass edit”If dns-control or its database is down and a name must change now, write the pdns API directly
from a k8s node with the key at fzymgc-house/infrastructure/dns/pdns-api. pdns keeps serving
while dns-control is down, so only changes wait. The lineage index shows the edit as
content_drift. When dns-control is back, adopt the live content:
kubectl exec -n dns-control deploy/dns-control -- dns-control adopt dns.core.fzymgc.house Akubectl exec -n dns-control deploy/dns-control -- dns-control adopt dns.core.fzymgc.house A --applyadopt plans a change that stores what pdns serves at the name and type, and prints the plan.
--apply applies it, and the RRset then carries the mark of the new change. The SOA and the apex
NS of a zone adopt into the zone row. An RRset that pdns no longer serves adopts as a delete.
--zone names the zone when the longest zone of the store does not hold the name. A record set of
the external-dns lane is not adopted, because external-dns writes it again on its next sync. A
disabled record is not adopted, because an apply writes every record enabled. adopt --apply ends
with an error when the change does not succeed.
Grants from the command line
Section titled “Grants from the command line”kubectl exec -n dns-control deploy/dns-control -- dns-control zones grant --generations 2,3 --dry-runzones grant sets the TSIG-ALLOW-DNSUPDATE grant of every dynamic zone in one command. For each
zone, the new grant names each Kea D2 key base of the zone, such as kea-d2-lab, at each generation
in --generations. Generation 1 is the base alone, and a generation n > 1 adds -<n>. A key that
is not a Kea D2 key leaves the grant, and a zone whose grant names no Kea D2 key stops the command.
Without --dry-run, it plans and applies one change per zone that differs, through the change
workflow. The TSIG key rotation in the DNS runbook uses it
to widen and then narrow the grants.
Restore from the export
Section titled “Restore from the export”kubectl exec -n dns-control deploy/dns-control -- dns-control restore --from s3://dns-control-export --dry-runrestore reads entries.json and zones.json from one sink. --from takes nas, b2, or the
s3://<bucket> URL of one of them. The command compares each zone, then each record set, then
each entry with the store and with what the writer node serves. It prints one line per subject:
| Action | When |
|---|---|
no change |
The store holds the subject equal and pdns serves it |
create |
The store does not hold the subject |
change |
The store holds other content, or pdns does not serve the subject |
skip external-dns |
The record set is in the external-dns lane, and external-dns writes it again |
Without --dry-run, the restore plans and applies each subject that is not no change, one at a
time, through the change workflow. It stops at the first change that does not succeed. A row that
the store holds and the export does not is left alone. On an empty pdns, the restore creates each
zone and then writes its control record sets and its entries. external-dns then writes its own
lane on its next sync. A grant that pdns holds and the export does not reads change.
Drift from the command line
Section titled “Drift from the command line”kubectl exec -n dns-control deploy/dns-control -- dns-control drift --state orphan --countdrift builds the lineage index from the reads of both nodes and the store. It prints the RRsets in one state with their writers, and --count prints the count only.
The states are orphan, conflict, content_drift, attributed and dangling.
Each command of this section reads the configuration of the server from the environment, so it runs inside the pod.
Observed AAAA
Section titled “Observed AAAA”dns-control publishes an AAAA record for each named client of a trusted leg that uses an address of
its leg ULA. The record sits at the Kea D2 name of the client, <host>.<leg>.lan.fzymgc.house, in
the observed-aaaa lane.
Where the address comes from
Section titled “Where the address comes from”The sweep reads the client table of the UniFi controller as the read-only account
dns-control-observer. The request is GET /proxy/network/api/s/default/stat/sta on unifi.url.
The sweep reads ipv6_addresses and never last_ipv6, which keeps addresses that a client no longer
uses. It keeps an address only when it sits in the ULA /64 of the client’s leg. It publishes no GUA,
no link-local address and no foreign ULA.
The key unifi_networks of the leg file maps each UniFi network name to a leg. A client of a
network that the map does not name fails the read. A rename in UniFi then stops the sweep, and it
does not match nothing. The controller lists some clients with no network. Such a client is on no
leg, so the sweep skips it, and the Query log shows no network for it.
The sweep joins each client to a name by its IPv4 address. The name is the one whose Kea D2 A record holds that address in the dynamic zone of the client’s leg. A client with no Kea D2 name gets no AAAA. A curated entry gets no observed AAAA, and its owner sets the IPv6 address of the entry.
The join also binds the MAC of the client. When a client leaves and its IPv4 address goes to another client, the read shows the new MAC at that address, and the addresses of the old client are not published at the name. Of two MACs whose sightings name one IPv4 address, the MAC seen last wins.
Sightings
Section titled “Sightings”The table observed_addresses keeps one row for each address of each client. The row holds the
first and the last sighting, the number of sweeps that saw the address, and the leg and the IPv4
address of the client at the last sighting.
- An address is published after
observed_aaaa.publish_aftersweeps saw it, 2 by default. - An address leaves after
observed_aaaa.retire_afterwithout a sighting, 30 minutes by default. - A restart keeps every row, so it neither delays nor removes a record.
When Kea D2 removes the A record at a name, the next sweep removes the AAAA there. While the AAAA is there, Kea D2 keeps the DHCID at the name, and it does not remove the PTR and the DHCID of the released address either. The same change removes these leftovers:
- The PTR and the DHCID at each reverse name that holds only a PTR to the name and the same DHCID. A reverse name that Kea D2 gave to another client since then holds another DHCID, and the change leaves it.
- The DHCID at the name, when it is the last record there, so the next client can claim the name.
If a delete fails, the change fails, the store keeps the AAAA, and the next sweep tries again. The reverse records go first, because the DHCID at the name is what finds them again on that retry.
Each lease that ends under a lane AAAA logs two ERROR lines in the D2 journal of both resolver nodes
(journalctl -u isc-kea-dhcp-ddns-server). They are expected, and the next sweep removes what they
leave:
DHCP_DDNS_FORWARD_REMOVE_RRS_REJECTED ... rejected a DNS update request to remove forward RR entries for FQDN, <name>, with an RCODE: 7DHCP_DDNS_REMOVE_FAILED ... Transaction outcome: Status: Failed, Event: UPDATE_FAILED_EVT, Forward change: failed, Reverse change: failedThe station copy for the Query log
Section titled “The station copy for the Query log”Each read that holds at least one client also replaces the table observed_stations in one
transaction. The table holds every station of the read, every leg included. A row holds the MAC,
the IPv4 and IPv6 addresses, the network, whether the station is wired, the alias that an admin set
in UniFi, the hostname that the device reports, the vendor, the Wi-Fi network, the time UniFi last
saw the station and the time of the read.
The Query log and the MCP tool dns_query_log read this table to show the device at a client
address. A Query log request never reaches the controller. The stations never enter the payload of
the sweep workflow, which Temporal keeps in its history.
- A failed read and an empty table leave the table as it was.
- A sweep that the removal guard refuses still replaces the table. The guard protects the lane, and the table is for display. A device that the partial read missed shows no device until the next sweep.
- A failed write of the table logs
replacing the observed stationsand does not fail the sweep. - With
observed_aaaa.enabledfalse, the Query log reads no device, even from an older table.
The Schedule
Section titled “The Schedule”dns-control creates or updates the Temporal Schedule observed-aaaa-sweep at each start, with the
interval observed_aaaa.interval, 300 seconds by default. A run that overlaps the one before it is
skipped, and a missed run is caught up for one interval. With observed_aaaa.enabled set to
false, the Schedule is paused. A change in the key dns-control.yaml of the ConfigMap restarts the
pod through Reloader, and the start applies it.
temporal schedule describe --namespace dns-control --schedule-id observed-aaaa-sweepA sweep that finds a difference applies one batch change of the lane, as the caller
system:observed-aaaa on the surface system. A sweep with no difference writes nothing. The lane
is not in zones.json, because the next sweep builds it again.
A run of the workflow observed.sweep with the input dry_run set to true plans from that one read,
as if every address in it were published. It writes no sighting, no health reading and no change, and
its result lists the planned record sets.
When a sweep fails
Section titled “When a sweep fails”A sweep changes no record when one of these is true:
- The read of the client table fails, for example on HTTP 429 or a timeout.
- The client table holds no client.
- A client sits on a UniFi network that the leg file does not name.
- The sweep would remove more than two record sets, and more than half of the lane, at names where Kea D2 still holds the A record. Those removals depend on the read, and a partial read must not empty the lane. A removal after Kea D2 removed its A record always goes through.
The sweep records the error, and the Health screen and dns_get_health show the time of the last
good read, the time and the text of the last error, and the lane size. The status reads failing
until a later sweep reads the table again. A sweep that meets a pending change writes nothing, and
the next sweep tries again.
Pause and purge
Section titled “Pause and purge”To stop the sweep, set observed_aaaa.enabled: false in the key dns-control.yaml of the
ConfigMap in argocd/app-configs/dns-control/config.yaml, and merge. The restart pauses the
Schedule. To remove every record set of the lane, pause the sweep first, then run the purge inside
the pod:
kubectl exec -n dns-control deploy/dns-control -- dns-control lane purge observed-aaaa --dry-runkubectl exec -n dns-control deploy/dns-control -- dns-control lane purge observed-aaaakubectl exec -n dns-control deploy/dns-control -- dns-control rrsets list --lane observed-aaaa --countThe purge applies one change of the lane, and the last command prints 0 after it.
UniFi client alias
Section titled “UniFi client alias”dns-control sets the alias and the note of each UniFi client whose MAC a curated entry holds. The
client table of the controller then shows the name of each curated device. The OpenSpec change
unifi-client-alias holds the design, and spike 012 (.planning/spikes/012-unifi-client-rename/)
measured the write.
What a run does
Section titled “What a run does”A run reads every entry from the store, and every client that the controller knows from
GET /proxy/network/api/s/default/rest/user, as the account dns-control-writer. It joins the two
on the MAC. For each client that an entry holds, the alias is the first label of the hostname of
the entry. The note is the mark writer=dns-control entry=<key>. The key names the entry, so a
change of the hostname keeps the mark.
- A client whose alias and note already agree gets no request.
- A client with another alias or another note gets
PUT rest/user/<_id>withnameandnotealone. The controller merges the body, so every other key of the client stays as it was. - A client whose note carries the mark and whose MAC no entry holds gets
nameandnoteset to"". That is the case after a delete of the entry, or after its MAC goes. - A client with no mark and no entry gets no request. An alias that a person set there stays.
Each write logs one line, UniFi client alias written, with the MAC, the client id, the entry,
and the old and new alias and note. The controller also stores a cleared alias or note as "",
and noted reads false after a clear.
Rename a curated device
Section titled “Rename a curated device”Change the hostname of its entry. A rename in the UniFi UI lasts only until the next run, which puts the alias of the entry back. The note of a curated client belongs to dns-control in the same way, so describe the device in the comment of the entry.
Entries that the controller does not know
Section titled “Entries that the controller does not know”The Health screen and dns_get_health list the key of each entry whose MAC no client of the
controller holds, in unifi_alias.unknown. The usual causes:
- The device uses a private Wi-Fi address, so the controller sees another MAC. Turn the private address off for the network on the device, or put the MAC that the controller sees in the entry.
- The entry holds another MAC of the device. The entry
nasholds3c:ec:ef:fa:66:af, and the NAS answers on3c:ec:ef:fa:66:ae. - The device never connected through the UniFi fabric, such as a container behind a host bridge.
The Schedule
Section titled “The Schedule”dns-control creates or updates the Temporal Schedule unifi-client-alias at each start, with the
interval unifi_alias.interval, 15 minutes by default. The interval lies between 1 minute and 1
hour. The Schedule skips a run that overlaps the one before it. With unifi_alias.enabled set to
false, dns-control pauses the Schedule. dns-control refuses to start with the comparison on and no
unifi.writer_username or unifi.writer_password.
temporal schedule describe --namespace dns-control --schedule-id unifi-client-aliasWhen a run fails
Section titled “When a run fails”A run writes nothing when one of these is true:
- The configuration holds no writer account.
- The read of the entries fails, or the store holds no entry with a MAC. This guard keeps a fault in the read of the store from clearing every alias that dns-control wrote.
- The login or the read of the client table fails, for example on HTTP 429.
- The client table holds no client.
The first failed write ends the run, and the next run sends the remaining writes again. The run
records the error, and the Health screen and dns_get_health show unifi_alias.status failing
with the time and the text of the error. No alert pages for a failing run, because an old alias
asks no person to act at once.
The writer account
Section titled “The writer account”dns-control-writer holds the role Network Admin, because the console offers no narrower role that
can rename a client. The controller therefore does not confine the account, and dns-control does.
The client has one write, to rest/user/<_id>, and the test
TestRunSendsOnlyTheLoginTheReadAndTheWrites fails on any other request of a run. The admin
activity log of the controller names the account for each change, by its display name
DnsControl Writer. Each change by that account must have the change key CLIENT and the
collection user. The probe of spike 001 saves the log as JSON with the View Only account, and
jq counts the changes by account:
.planning/spikes/001-networkconf-guard-fields/run.sh probe_admin_activity.py <scratch>/activity.json <since, ISO 8601> > /dev/nulljq -r '.data[] | [.admin.name, .change_key, (.meta.collection // "-")] | @tsv' <scratch>/activity.json | sort | uniq -cEvery line that names DnsControl Writer must read CLIENT and user.
After a Network upgrade
Section titled “After a Network upgrade”After each upgrade of the Network application, make sure that a write still keeps the other keys
of a client. Write the alias and the note that the client of pdu already holds. The probe diffs
the client before and after the write, and it must print diff: no key changed. A controller that
replaces the record deletes every other key, and the diff then lists them.
.planning/spikes/012-unifi-client-rename/run.sh writer rename.py <scratch> 00:0c:15:41:30:15 \ --name pdu --note 'writer=dns-control entry=pdu'Pause, and remove the aliases
Section titled “Pause, and remove the aliases”To stop the comparison, set unifi_alias.enabled: false in the key dns-control.yaml of the
ConfigMap in argocd/app-configs/dns-control/config.yaml, and merge. The restart pauses the
Schedule. The aliases stay, and they still name the right devices. To remove them, clear the alias
and the note of each client in the UniFi UI. These commands list the clients that carry the mark:
scripts/unifi-readonly.sh scripts/unifi-inventory.py <scratch> > /dev/nulljq -r '.[] | select((.note // "") | test("^writer=dns-control entry=")) | [.mac, .name] | @tsv' <scratch>/user.jsonThe image and its pin
Section titled “The image and its pin”The file apps/dns-control/VERSION holds the version, and changie writes it on a release. The
release workflow (.github/workflows/release.yml) runs when a release pull request merges.
GoReleaser Pro builds the amd64 and arm64 images from apps/dns-control/.goreleaser.yaml, pushes
the multi-architecture index, and signs it with a keyless cosign signature. The same release holds
the node worker tarball. App Releases describes the release.
apps/dns-control/Dockerfile keeps the gated build that pr-gates runs. The release image comes from
apps/dns-control/Dockerfile.release, which copies the binary that GoReleaser built onto the same
Debian base.
If the version tag exists in GHCR, the release workflow fails before GoReleaser runs, so a version
tag never moves. The summary of the dns-control job shows the published image in one line, for
example:
published: ghcr.io/fzymgc-house/dns-control:0.1.0@sha256:b9ade42aac443c09478b12841e6891d1650c7dfe91ef5d1a77541e9b787df2a1dns-control has one pin, and Argo CD Image Updater moves it.
The pin is newTag in the kustomize Component argocd/app-configs/dns-control-image/kustomization.yaml.
The kustomization of dns-control includes the Component for the Deployment dns-control, and the
kustomization of external-dns includes it for the webhook. The pin holds a version tag and no
digest. A release opens one pull request, and it waits for a review.
Image Updater checks a cosign signature only against a key pair, and dns-control signs keyless. Before
you merge an updater pull request, make sure that the signature of the new version comes from the
release workflow on main:
cosign verify ghcr.io/fzymgc-house/dns-control:<version> \ --certificate-identity 'https://github.com/fzymgc-house/selfhosted-cluster/.github/workflows/release.yml@refs/heads/main' \ --certificate-oidc-issuer https://token.actions.githubusercontent.comA version from 0.4.0 onward stops at start without the nine EXPORT_* variables of the two
export sinks, and reaches the NAS sink only with the hostAliases pin and the two egress rules
of the network policy. The manifests in argocd/app-configs/dns-control/ carry all of them.
A version from 0.3.0 onward needs three changes. The manifests in
argocd/app-configs/dns-control/ carry them:
- The variable
DATABASE_URL, from the keydatabase_urlof the Secretdns-control-db. dns-control stops at start without it. - The mount of the NATS credentials file from the property
nats_credsof the Secretdns-control, at/etc/dns-control/nats/dns-control.credswith mode 0400. - Egress to the Temporal frontend on TCP 7233, to the CNPG pods of the cluster
mainon TCP 5432 and to the NATS servers on TCP 4222. Each rule names a namespace and a label. No rule names a DNS name.
To release a new version:
- Open the release pull request with
scripts/release.sh dns-control, as in Release an app. - Merge the release pull request. The
releaseworkflow tagsdns-control/v<version>, and builds, signs and publishes the image. - Within five minutes, Image Updater opens the two pull requests that move the pins. Review and merge them as After a dns-control release describes.
The wire contract
Section titled “The wire contract”Every value that crosses the Temporal boundary is a protobuf message. The node worker on the Pis
and the dns-control image ship on their own, so an older worker can meet a newer dns-control. The
field numbers of the messages keep the two sides compatible. The paths below are relative to
apps/dns-control/.
| Package | Files | What it carries |
|---|---|---|
dnscontrol.nodeops.v1 |
proto/dnscontrol/nodeops/v1/actions.proto, proto/dnscontrol/nodeops/v1/status.proto |
The inputs and results of the six node worker activities, and the status reading of a node |
dnscontrol.ops.v1 |
proto/dnscontrol/ops/v1/ops.proto |
The inputs and results of the operations workflows and of their control activities in dns-control |
The two packages change for different reasons. dnscontrol.nodeops.v1 crosses the network to the
Pis, so a change to it needs a worker release on both Pis. dnscontrol.ops.v1 stays inside
dns-control. Temporal keeps the workflow history across releases of dns-control, so its field
numbers are part of the contract too.
The generated Go code is in internal/gen/dnscontrol/nodeops/v1 and
internal/gen/dnscontrol/ops/v1. Do not edit it by hand. The gate generates it again and fails on
any difference.
Validation on both sides
Section titled “Validation on both sides”Each message carries protovalidate rules. Protovalidate is the rule language of Buf for protobuf
fields. The Temporal client of the node worker and the client of dns-control both carry
wire.DataConverter() and wire.Interceptor():
- The converter sends each message as binary protobuf. Its last step accepts only a bool, an integer, a float or a string, so a hand-written Go struct fails at every Temporal call site.
- The interceptor checks each payload against its rules in the client, the workflow and the
activity. A payload that fails its rules, or a nil message, fails with a non-retryable error of
the type
invalid_payload.
The node worker therefore refuses an invalid input before the activity code runs. dns-control
refuses an invalid result, and the operations workflow records the action as failed with the
reason action_failed.
Change the schema
Section titled “Change the schema”The tasks need Docker. Each step runs in the builder image of apps/dns-control/Dockerfile, and
buf v1.73.0 and protoc-gen-go come from the tools module tools/go.mod. No shipped binary links
the tools module.
-
Edit the
.protofile underproto/dnscontrol/. -
Write the generated code:
Terminal window mise //apps/dns-control:proto:generate -
Run the gate before the push. Fetch
mainfirst, because the breaking check compares withorigin/main. The last line of a pass isPROTO_GATE=PASS.Terminal window git fetch origin mainmise //apps/dns-control:proto:check -
Commit the
.protofile and the generated code together.
The gate tools/proto.sh check prints one line for each step: MODULES, NO_DEPS, LINT,
VENDOR_PIN, BREAKING, DRIFT and OFFLINE. The pr-gates member dns-control-proto in
.github/workflows/pr-gates.yml fetches main and runs the same gate with
BUF_BREAKING_BASE=origin/main. It runs when a pull request changes a file under
apps/dns-control/, and in the nightly pr-gates run. buf breaking uses the rule set FILE. If
main has no apps/dns-control/buf.yaml, the gate prints BREAKING=NO_BASELINE and skips the
breaking step.
No step reads the Buf Schema Registry
Section titled “No step reads the Buf Schema Registry”The Buf Schema Registry is the hosted schema service at buf.build. No build or CI step reads it.
buf.yamlnames no dependency. The gate refuses adepsentry and abuf.lockby name.- The one dependency schema,
proto/buf/validate/validate.proto, is vendored into the repository. - The step
MODULESdownloads the Go modules throughproxy.golang.orgonly, with no direct fallback. - Every later step runs in a container with
--network noneandGOPROXY=off. A step that tries to reachbuf.buildor any other host fails there.
The Go runtime of protovalidate imports the module
buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go. That module comes from
proxy.golang.org, and go.sum and sum.golang.org pin it. No step contacts buf.build for it.
The vendored validate.proto comes from bufbuild/protovalidate at the commit
ec950f2039c7cf07895f1f83ab565d7fc57628be, with the sha256
39f6496bef0a622be188703bd3db2ed23b3e4b5c2da27b121b88947f311d3448. tools/proto.sh holds both
values as PROTOVALIDATE_COMMIT and PROTOVALIDATE_SHA256, and the step VENDOR_PIN compares
the file with the sha256. To move the pin:
-
Change both values in
tools/proto.sh. -
Download the file at the new commit. The task refuses a file whose sha256 differs:
Terminal window mise //apps/dns-control:proto:vendor -
Run
mise //apps/dns-control:proto:generateandmise //apps/dns-control:proto:check.
Evolution rules
Section titled “Evolution rules”- A new field takes a new number. Never use a number again.
- A removed field keeps its number and its name as
reserved. - Every enum field refuses the value 0.
- An enum that a receiver acts on,
OutcomeandAction, carriesdefined_only: true. An older receiver refuses a value that it does not know withinvalid_payload, so a new value ships in the receiving side first. For a value that dns-control sends, release the node worker on both Pis first. For a value that the node worker sends, release dns-control first. - An enum that a receiver only shows or compares,
KeepalivedStateandListSourceKind, carries nodefined_only. An older receiver accepts a new value. It reads an unknown keepalived state asUNKNOWN, which failskeepalived_not_fault, and an unknown list source kind as no kind. - A receiver ignores a field that it does not know, in binary and in JSON payloads.
- A new reason id ships in the receiving side first too. The reason pattern
^[a-z_]{0,64}$accepts an id that the receiver does not know, but an older UI shows the generic failure copy for it.
After dns-node-worker-v0.3.0, a change to dnscontrol.nodeops.v1 needs a worker release on both
Pis. Open the release pull request with scripts/release.sh dns-control and merge it. The release
dns-control/v<version> holds the worker tarball. Then set dns_node_worker_version in
ansible/roles/dns-node-worker/defaults/main.yml and converge one node per window.
The leg file
Section titled “The leg file”The ConfigMap dns-control-legs in argocd/app-configs/dns-control/config.yaml holds
legs.json. A leg is one network segment that the resolvers serve, for example main or
guest. The file lists the two pdns nodes, the conditional zones, the seven legs and the tailnet.
dns-control does not start without a valid leg file.
The file is a copy. The source of each value is in Ansible:
| Value | Source |
|---|---|
| The legs (VLAN, IPv4 subnet, trust) and the pdns node addresses | dns_vlan_interfaces in ansible/inventory/host_vars/sandiego.yml and ansible/inventory/host_vars/portland.yml |
| The ULA /64 of each leg | ula_site_prefix and ula_leg_index in ansible/inventory/group_vars/all.yml |
| The pdns API port, the forward zone and the reverse zones | pdns_api_port, dns_zone_domain and dns_reverse_zones in ansible/inventory/group_vars/dns_hosts.yml |
No check compares the copy with Ansible. If you change a leg, a node address or a zone in
Ansible, change config.yaml in the same pull request.
Each node name must equal the Ansible inventory hostname. The resolver collectors send that name
as host.name. If a name differs, every Health metric of that node shows no value.
Reach while the resolver pair is down
Section titled “Reach while the resolver pair is down”dns-control and Keycloak both have names that the resolver pair serves. If the pair is down, a
client on the LAN cannot resolve either name. The sign-in also goes through
https://id.fzymgc.house, so a name for dns-control alone does not help.
Traefik serves both names on the address 192.168.250.145. To reach dns-control when the pair is
down, add these two lines to the hosts file of your workstation (/etc/hosts on macOS and Linux):
192.168.250.145 dns-control.fzymgc.house192.168.250.145 id.fzymgc.houseRemove the lines when the pair is back. If the lines stay, a later change of the Traefik address breaks both names on your workstation.
Issue #2386 proposed tailnet names for Keycloak and dns-control. It closed as not planned. A
second Keycloak hostname changes the iss claim of the tokens, and dns-control accepts one issuer
only. The hosts-file lines stay the fallback. A break-glass surface on the Pis is future work
(FUT-03 in .planning/REQUIREMENTS.md).
Telemetry
Section titled “Telemetry”Each process of the binary sends traces, metrics and logs over OTLP to cs-otel-collector in the
namespace clickstack. internal/observe builds the three providers from
OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS. If the endpoint is empty, the
process sends nothing. A collector that is down or refuses the data never stops a request. Each
process flushes its telemetry for at most 5 seconds before it exits.
| Process | Service name | Local output |
|---|---|---|
| The server | dns-control |
JSON on stdout, with trace_id and span_id |
dns-control webhook, the container webhook in the external-dns pod |
dns-control-webhook |
JSON on stdout, with trace_id and span_id |
An operator command through kubectl exec |
dns-control-cli |
Text on stderr at the level Warn |
| The node worker on sandiego and portland | dns-node-worker |
JSON on stdout, with trace_id and span_id, in the journal of the unit |
Each process sets its service name in code. An operator command inherits the environment of the
server, so a name from OTEL_SERVICE_NAME labels its spans as the server. The node worker sends
to the collector on its own node at 127.0.0.1:4318, which forwards through
otel-gateway.fzymgc.house. It sends the three gauges dns_worker_client_cert_not_after_seconds,
dns_worker_drained and dns_worker_drain_overdue, the Temporal SDK metrics and the Go runtime
metrics, with the host.name of the node. The journal receiver of the collector reads
dns-node-worker-pki.service and not the worker unit. The pod annotation
vector.dev/exclude-containers keeps Vector off the containers dns-control and webhook, so
each log line reaches default.otel_logs once. The OTLP log records start at the level Info.
Traces
Section titled “Traces”One trace follows a request from its first seam to its last. Trace context crosses each boundary
in W3C form: HTTP headers, gRPC metadata, Temporal headers, and the traceparent header of each
NATS message.
| Seam | Span |
|---|---|
| A request to the REST, MCP or webhook listener | A server span named <method> <route>, such as POST /mcp |
| A request to pdns, Keycloak, the UniFi controller, an S3 sink, ClickHouse or the lane API | A client span |
| A Temporal workflow or activity | RunWorkflow:<type> and RunActivity:<type> |
| A Postgres query | A client span with the SQL text and no parameter |
| A NATS publish | publish <subject> |
| An MCP tool call | A span named after the tool, as a child of the server span |
| An apply of a change | change.apply, with dns_control.surface, dns_control.subject, dns_control.caller_kind and dns_control.caller_azp |
| A run of a background loop | One root span: lineage.refresh, ops.poll, sessions.sweep or ledger.reload |
| An operator command | One root span: dns-control <command> |
Metrics
Section titled “Metrics”The server sends the Temporal SDK metrics, the Go runtime metrics and eight application metrics.
The meter exports every 60 seconds. Each gauge sends every value of its attributes, with 0 where
nothing matches, so an alert reads 0 and not an empty bucket. No metric carries caller_azp,
because each dynamic client registration gives a new client id.
The counters are delta. The Deployment sets OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to
delta, so each counter point holds the increase since the previous export. A counter sends a
point only for an attribute set that changed in that export. Add the points with sum(Value).
HyperDX skips the first point of a cumulative series, and each pod starts new series. A cumulative
counter therefore hides the first event of each kind after a restart.
| Metric | Kind | Attributes | Value |
|---|---|---|---|
dns_control_change_applies_total |
counter | surface, subject, op, outcome, caller_kind |
One for each change whose outcome is final |
dns_control_change_refusals_total |
counter | surface, step, reason |
One for each refused plan or apply |
dns_control_export_pending_age_seconds |
gauge | sink |
The age of the oldest succeeded change that the sink does not hold yet, or 0. An imported change has no workflow and never counts |
dns_control_lineage_rrsets |
gauge | state, finding |
The RRsets of the published lineage index in each state and settle status |
dns_control_lineage_refresh_timestamp_seconds |
gauge | none | The Unix time of the last published index |
dns_control_observed_sweep_timestamp_seconds |
gauge | none | last_good_read_at of the table observed_sweep |
dns_control_lineage_node_disagreements |
gauge | result, finding |
The RRsets on which the pdns nodes do not agree |
dns_control_vip_holder_status |
gauge | status |
1 for the current status of the VIP holder and 0 for the other three |
| Attribute | Values |
|---|---|
surface |
ui, rest, mcp, cli, system |
subject |
entry, zone, rrset, batch |
op |
create, change, delete. A batch has change |
outcome |
succeeded, failed |
caller_kind |
person, client, static, system |
step |
plan, apply |
reason |
role_required, index_not_ready, invalid_entry, plan_not_found, plan_expired, entry_changed, subject_changed, change_in_progress and the policy_* codes of the write policy |
sink |
nas, b2 |
state |
attributed, orphan, conflict, content_drift, dangling |
result |
differ, missing, unreadable |
finding |
pending, settled, and none for attributed |
status |
one_holder, none, split, not_read |
The two gauges that read Postgres skip their point when the read fails, and they log the failure. The lineage gauges and the refresh time send nothing before the first index.
Queries
Section titled “Queries”Run these in clickhouse-client on a ClickHouse pod (see
Useful queries).
The trace of the last call of a tool, with the services and the spans that it holds:
SELECT countDistinct(ServiceName) AS services, groupUniqArray(SpanName) AS spansFROM default.otel_tracesWHERE TraceId = (SELECT TraceId FROM default.otel_traces WHERE ServiceName = 'dns-control' AND SpanName = 'dns_get_health' ORDER BY Timestamp DESC LIMIT 1)The surfaces of the changes in the last 10 minutes:
SELECT Attributes['surface'] AS surface, Attributes['outcome'] AS outcome, sum(Value) AS totalFROM default.otel_metrics_sumWHERE ServiceName = 'dns-control' AND MetricName = 'dns_control_change_applies_total' AND TimeUnix > now() - INTERVAL 10 MINUTEGROUP BY surface, outcomeA log line that reaches ClickStack twice. The answer must hold no row:
SELECT TraceId, SpanId, count() AS copiesFROM default.otel_logsWHERE ServiceName = 'dns-control' AND Body = 'mcp tool call' AND Timestamp > now() - INTERVAL 1 HOURGROUP BY TraceId, SpanIdHAVING copies > 1The spans of the operator commands in the last hour:
SELECT Timestamp, SpanName, StatusCodeFROM default.otel_tracesWHERE ServiceName = 'dns-control-cli' AND Timestamp > now() - INTERVAL 1 HOURORDER BY Timestamp DESCA credential in a span or a log record. Do not send the value to ClickHouse, because
system.query_log keeps the text and the settings of each query. Hash the value on your machine
instead. For the pdns API key, run this on the control node with a Vault session. It prints the
hash only:
vault kv get -mount=secret -field=api_key fzymgc-house/infrastructure/dns/pdns-api \ | tr -d '\n' | shasum -a 256 | cut -d' ' -f1The query hashes each run of letters and digits in each span and each log record, and it compares
each hash with the hash of the value. It finds a value that holds only letters and digits, such as
the pdns API key. Put the hash in secret_hash. The answer must be 0 in secret_rows for both
sources. To make sure that the query works, put the hash of a word that a recent log record holds,
such as the key of an entry that you projected, in control_hash. control_rows must then be
greater than 0:
WITH '<sha256 of the value>' AS secret_hash, '<sha256 of a known word>' AS control_hashSELECT 'trace' AS source, count() AS rows_searched, countIf(arrayExists(t -> lower(hex(SHA256(t))) = secret_hash, tokens)) AS secret_rows, countIf(arrayExists(t -> lower(hex(SHA256(t))) = control_hash, tokens)) AS control_rowsFROM (SELECT extractAll(concat(toString(SpanAttributes), ' ', toString(`Events.Attributes`), ' ', toString(ResourceAttributes), ' ', StatusMessage), '[A-Za-z0-9]+') AS tokens FROM default.otel_traces WHERE Timestamp > now() - INTERVAL 1 DAY AND ServiceName LIKE 'dns-%')UNION ALLSELECT 'log', count(), countIf(arrayExists(t -> lower(hex(SHA256(t))) = secret_hash, tokens)), countIf(arrayExists(t -> lower(hex(SHA256(t))) = control_hash, tokens))FROM (SELECT extractAll(concat(Body, ' ', toString(LogAttributes), ' ', toString(ResourceAttributes)), '[A-Za-z0-9]+') AS tokens FROM default.otel_logs WHERE Timestamp > now() - INTERVAL 1 DAY AND ServiceName LIKE 'dns-%')Alerts and monitors
Section titled “Alerts and monitors”Two systems page the operator through Pushover about dns-control. ClickStack alerts read the telemetry of dns-control and of the node workers. Uptime Kuma monitors check dns-control and the resolver VIPs from outside, so they keep working when ClickStack is down. Each alert message and each monitor description names its entry in this section.
Uptime Kuma monitors
Section titled “Uptime Kuma monitors”tf/uptime-kuma/monitors-dns.tf holds the three monitors. Each monitor runs a check every 60
seconds. After a failed check, it tries 3 more times, 60 seconds apart. If each try fails, the
monitor reads DOWN and pages through the notification Pushover (cluster-edge).
Each check starts in the Uptime Kuma pod. If the reading from a workstation passes and the monitor reads DOWN, the fault is on the path from the pod. dns-control and the resolvers are then not the cause.
dns-control readiness
Section titled “dns-control readiness”The monitor requests https://dns-control.fzymgc.house/readyz and expects the status 200. The
request goes through the public name, Traefik and the certificate dns-control-public-tls to the
pod. The pod answers 200 when its configuration parsed and the first record index exists.
- On a workstation, run
curl -s -o /dev/null -w '%{http_code}\n' https://dns-control.fzymgc.house/readyz. - If the status is 503, the pod is not Ready, and Traefik has no pod for the request. Follow The record index does not finish.
- If the status is 404, Traefik has no route for the name. Make sure that the IngressRoute
dns-controlexists and that ArgoCD shows the Applicationdns-controlas Synced. - If curl reports a certificate error, make sure that the Certificate
dns-control-public-tlsin the namespacedns-controlis Ready.
The usual causes are these:
- The pod restarted. The first index takes about 30 seconds, and the retries of the monitor cover that time.
- The database or a pdns node was down when the pod started, so the first index did not finish.
- A sync of the Application
dns-controlfailed.
DNS resolver VIP (Core)
Section titled “DNS resolver VIP (Core)”The monitor sends an A query for nas.fzymgc.house to the Core VIP 192.168.40.5 on port 53. It
passes when the answer holds an A record. The main recursor process on the node that holds the
VIPs answers, with the record from pdns. nas is a curated entry of dns-control. The monitor does not compare the address
in the answer.
- On a workstation, run
dig @192.168.40.5 nas.fzymgc.house A. - If the query times out, no node answers on the VIP. Read the VIP holder on the Health screen of dns-control.
- If the VIP holder is
none, follow What a failover looks like. - If the status is
NXDOMAIN, or the answer holds no A record, search fornason the Search screen of dns-control.
The usual causes are these:
- Both resolver nodes are in the keepalived state FAULT, so no node holds the VIPs.
- The main recursor process
pdns-recursor@mainstopped on the node that holds the VIPs. - A change deleted or renamed the entry
nas.
DNS resolver VIP (Lab)
Section titled “DNS resolver VIP (Lab)”This monitor is the same as DNS resolver VIP (Core), with the Lab VIP
192.168.200.5. The Lab VIP is the resolver for the clients on the Lab VLAN. The same node holds
all the VIPs, so a fault on the holder usually fails both monitors.
- On a workstation, run
dig @192.168.200.5 nas.fzymgc.house A. - If the Core monitor reads UP and this monitor reads DOWN, the Lab leg of the node that holds the
VIPs is the fault. On that node, run
ip -br addrand make sure that192.168.200.5is on the Lab interface. - If both monitors read DOWN, follow the steps of DNS resolver VIP (Core).
The usual causes are the causes of the Core monitor, and a fault on the Lab leg of the holder.
ClickStack alerts
Section titled “ClickStack alerts”argocd/app-configs/clickstack-alerts/bootstrap-script.yaml holds the HyperDX dashboard
DNS control plane and its 13 tile alerts. Each alert pages through the webhook
Pushover (cluster-infra), and its page links to its tile. The tile is the first reading. The
interval of each alert is in
Per-alert interval map.
Do not re-sync the Application clickstack-alerts while you read an alert that fires. A sync sets
each alert back to OK. The counter tiles add delta points, as Metrics describes.
DNS control: MCP write burst
Section titled “DNS control: MCP write burst”The alert reads the sum of dns_control_change_applies_total with surface mcp. It fires when
more than 20 changes with a final outcome were applied through MCP in 15 minutes. An apply through
the UI, REST, the CLI or the system surface does not count.
- Run the query of What a call logs with
AND LogAttributes['tool'] = 'dns_apply_change'. The columncallernames the agent or the key. - If the calls are not expected, stop the caller. Revoke a static key with
Add or revoke a static key. For a Keycloak caller, take the role
dns-writerfrom its user or its service account.
The usual causes are these:
- An agent works through a large task that a person asked for.
- An agent repeats one change in a loop.
DNS control: delete through MCP
Section titled “DNS control: delete through MCP”The alert reads the sum of dns_control_change_applies_total with surface mcp and op
delete. It fires on each delete of an entry, a zone or a record set through MCP, in a window of
5 minutes. A delete that a person asked an agent for also pages.
- Read the newest changes on the Changes screen, or with the tool
dns_list_changes. The change row names the subject that the change deleted. - Run the query of What a call logs with
AND LogAttributes['tool'] = 'dns_apply_change'to find the caller. - If nobody asked for the delete, read the subject before the change in
before,zone_beforeorrrset_beforeof the change row. Plan a create with those values and apply it.
The usual causes are these:
- A person asked an agent to delete a name. The page is expected.
- An agent chose a delete that nobody asked for.
DNS control: MCP refusal burst
Section titled “DNS control: MCP refusal burst”The alert reads the sum of dns_control_change_refusals_total with surface mcp. It fires when
dns-control refused more than 10 plans or applies through MCP in 15 minutes.
-
Read the refusals by step and reason:
SELECT Attributes['step'] AS step, Attributes['reason'] AS reason, sum(Value) AS refusalsFROM default.otel_metrics_sumWHERE MetricName = 'dns_control_change_refusals_total' AND Attributes['surface'] = 'mcp'AND TimeUnix > now() - INTERVAL 1 HOURGROUP BY step, reasonORDER BY refusals DESC -
Run the query of What a call logs to find the caller.
The usual causes are these:
role_required: a caller withoutdns-writertries to plan or apply.- A
policy_*reason: an agent tries names that the write policy refuses. change_in_progress: an agent applies again while a change is pending.plan_expired: an agent applies a plan after its 30 minutes.
DNS control: change failed
Section titled “DNS control: change failed”The alert reads the sum of dns_control_change_applies_total with outcome failed, from every
surface. It fires on each change that ended with the outcome failed, in a window of 15 minutes.
- Read the newest failed change on the Changes screen, or with the tool
dns_get_change. The change row lists each step of The workflow and its result. - If
pdns_writefailed, follow Node unreadable for the writer node. - If
reservation_writefailed, follow A node worker does not answer.
The usual causes are these:
- The writer pdns node refused the write or did not answer.
- A node worker did not answer during the reservation write.
DNS control: export pending
Section titled “DNS control: export pending”The alert reads the maximum of dns_control_export_pending_age_seconds for each sink. It fires when
a sink has not held a succeeded change for more than one hour. The tile shows the sink, nas or
b2. Each sink retries for up to seven days, so an alert means a sink that is down, not a retry.
- Read the sink on the tile.
- Read the dns-control log lines of the export in ClickStack. Filter on
Body ILIKE '%export%'in the query of When a screen shows a data alert. - For the sink
nas, make sure that RustFS on the NAS answers. See the RustFS runbook. - For the sink
b2, make sure that the application keydns-control-exportoftf/backblaze/dns_control_export.tfcan still write the bucketfzymgc-dns-control-export.
The usual causes are these:
- RustFS on the NAS is down.
- The B2 key expired, or a change removed its access to the bucket.
DNS control: lineage drift
Section titled “DNS control: lineage drift”The alert reads the sum of dns_control_lineage_rrsets in the states orphan, conflict,
content_drift and dangling with the finding settled. A finding settles after two readings and
five minutes. The alert fires on any settled finding, in a window of 30 minutes, so an adopt inside
that time pages nobody.
- Run
dns-control drift --state <state>for each state, as Drift from the command line describes. The tooldns_list_driftand the Health screen show the same findings. - If an RRset reads
content_driftafter a direct write to pdns, follow Adopt a break-glass edit. - If an RRset reads
orphan, no row of the store projects it. If dns-control must hold it, adopt it with the sameadoptcommand.
The usual causes are these:
- An operator wrote pdns directly and did not adopt the edit.
- A writer outside dns-control wrote pdns.
- A pdns write of a change failed, so a row of the store reads
dangling.
DNS control: pdns nodes disagree
Section titled “DNS control: pdns nodes disagree”The alert reads the sum of dns_control_lineage_node_disagreements with the finding settled. It
fires when the pdns nodes sandiego and portland hold a settled disagreement on an RRset, in a
window of 15 minutes. LightningStream copies a write between the nodes in seconds, so a copy in
flight never settles.
- Read the node agreement on the Health screen, or with the tool
dns_get_health. - If the result is
unreadable, follow Node unreadable. - If the result is
differormissing, follow Convergence lag in the DNS runbook.
The usual causes are these:
- LightningStream stopped on one node.
- RustFS on the NAS is down, so LightningStream cannot copy the change.
- A person wrote the pdns node that is not the writer.
DNS control: VIP holder not single
Section titled “DNS control: VIP holder not single”The alert reads the maximum of dns_control_vip_holder_status for the status none or split.
The operations poller reads the holder from the node workers. The alert fires when the VIPs have
no holder or two holders, in a window of 5 minutes. The status not_read does not page.
- Read the VIP holder and the keepalived state of each node on the Health screen.
- If the status is
none, follow What a failover looks like. The Uptime Kuma VIP monitors usually fire too. - If the status is
split, both nodes claim the VIPs. On each node, runjournalctl -u keepalived -n 50and read the state changes.
The usual causes are these:
- Both nodes are in the keepalived state FAULT.
- The two nodes do not receive the VRRP adverts of each other, so each node holds the VIPs.
DNS control: lineage index stale
Section titled “DNS control: lineage index stale”The alert reads the maximum increase of dns_control_lineage_refresh_timestamp_seconds in the
window. The refresh loop publishes an index every 30 seconds. The alert fires when the timestamp
did not move in 15 minutes, and also when dns-control sends no metric.
- Look at the monitor dns-control readiness. If it reads DOWN, follow its entry.
- Read the spans
lineage.refreshof dns-control in ClickStack. A span with the statusErrornames the read that failed. Then read the log lines with the query of When a screen shows a data alert.
The usual causes are these:
- The dns-control pod is down, or it does not reach the collector.
- The refresh loop stopped in a running pod.
- dns-control started after an outage near the end of a window. The pod sent too few points to show a change, and the alert clears at its next evaluation.
DNS control: observed AAAA sweep stale
Section titled “DNS control: observed AAAA sweep stale”The alert reads the maximum increase of dns_control_observed_sweep_timestamp_seconds, the time of
the last good read of the UniFi client table. The sweep runs every 5 minutes. The alert fires when
no sweep read the table in one hour.
A paused sweep fires this alert. If you paused the sweep with Pause and purge, the alert fires after one hour and stays in ALERT until the sweep runs again.
- Read the observed AAAA sweep on the Health screen, or with the tool
dns_get_health. It shows the last good read, the last error and the lane size. - If
observed_aaaa.enabledisfalseinargocd/app-configs/dns-control/config.yaml, the alert is expected. - Otherwise, follow When a sweep fails.
The usual causes are these:
- An operator paused the sweep.
- The read of the UniFi client table fails, for example on HTTP 429 or a timeout.
DNS control: node worker silent
Section titled “DNS control: node worker silent”The alert reads the count of the series of dns_worker_drained. It fires when fewer than two node
workers sent their gauges in 15 minutes.
The threshold counts one series for each worker. HyperDX counts a series, not a data point, and the resource of the worker holds no process id. Both workers therefore read 2, and a restart of a worker keeps its series. A process id or another value that changes at each start, added to the resource of the worker, would break this threshold.
-
Find the node that is missing:
SELECT ResourceAttributes['host.name'] AS node, max(TimeUnix) AS lastFROM default.otel_metrics_gaugeWHERE MetricName = 'dns_worker_drained' AND TimeUnix > now() - INTERVAL 1 HOURGROUP BY node -
Follow A node worker does not answer for that node.
The usual causes are these:
- The worker stopped on the node.
- The collector on the node stopped, so the gauges of a running worker do not arrive.
- The node is down.
DNS control: drain overdue
Section titled “DNS control: drain overdue”The alert reads the maximum of dns_worker_drain_overdue. The worker sets it to 1 when a drain
outlives its expected duration, 30 minutes by default. The alert fires in a window of 5 minutes and
stays in ALERT until the undrain.
- Read the card of the node on the Health screen. It shows Drain overdue.
- If the work on the node is complete, follow The undrain.
The usual causes are these:
- An operator did not undrain the node after the work.
- The expected duration of the drain was shorter than the work.
DNS control: server errors
Section titled “DNS control: server errors”The alert reads the count of the server spans of dns-control and dns-control-webhook with the
status Error. A server span is one request to the REST, MCP or webhook listener, and it has the
status Error on a 5xx answer. The alert fires on more than 10 such spans in 15 minutes.
-
Count the error spans by service and route:
SELECT ServiceName, SpanName, count() AS errors, any(TraceId) AS traceFROM default.otel_tracesWHERE ServiceName IN ('dns-control', 'dns-control-webhook') AND SpanKind = 'Server'AND StatusCode = 'Error' AND Timestamp > now() - INTERVAL 1 HOURGROUP BY ServiceName, SpanNameORDER BY errors DESC -
Open the trace of the busiest route in HyperDX. The client span with an error names the dependency that failed.
The usual causes are these:
- The CNPG cluster
mainis down, so each route that reads the database fails. - A pdns node does not answer.
- The webhook cannot reach dns-control, because the dns-control pod is down.
When a screen shows a data alert
Section titled “When a screen shows a data alert”Start each investigation in ClickStack. The dns-control log lines are in default.otel_logs:
SELECT Timestamp, SeverityText, BodyFROM default.otel_logsWHERE ServiceName = 'dns-control' AND Timestamp > now() - INTERVAL 1 HOURORDER BY Timestamp DESCLIMIT 100Use the Kubernetes API only for the current state of the pod and its events. The binary reads its Secrets once at start. The Deployment carries the Reloader annotation, so Reloader restarts the pod when a Secret or ConfigMap that it uses changes.
Node unreadable
Section titled “Node unreadable”The alert reads “dns-control failed to read sandiego.” or the same for portland. The data for
that node on each screen is from the last good read.
- Read the dns-control log lines in ClickStack for the node name and the error.
- Make sure that the pdns API on the node answers on port 8081. From a k8s node, run
curl -s -o /dev/null -w '%{http_code}\n' http://192.168.20.6:8081/api/v1/serversforsandiegoor192.168.20.7forportland. Any HTTP status means that the port answers. A timeout means that a firewall or the pdns service stops the call. - Make sure that the network policy
dns-control-defaultallows egress to the node address on TCP 8081. - The resolver firewall admits the pdns API from
pdns_api_source_setonly. Ansible derives that set from the k8s node addresses inansible/inventory/group_vars/dns_hosts.yml. If a k8s node address changed, run thedns-commonrole on both resolvers withscripts/dns-playbook.sh. - If the log shows
401, the pdns API key is wrong. dns-control readspdns_api_keyfromfzymgc-house/infrastructure/dns/pdns-api. After a rotation of that key, the ExternalSecret refreshes within 5 minutes, and Reloader then restarts the pod.
Query log unavailable
Section titled “Query log unavailable”The alert reads “dns-control failed to read the query log from ClickStack.” Search and Name detail still work.
- Read the dns-control log lines in ClickStack for the ClickHouse error.
- If the error is an authentication error, the password of
dns_control_readerdiffers between dns-control and ClickHouse. Both readdns_control_reader_passwordfromfzymgc-house/cluster/clickstack. See the ClickStack runbook for the seed of that key. - Make sure that the network policy
clickstack-defaultadmits thedns-controlnamespace on TCP 8123.
A pull request that adds a key to the ExternalSecret clickstack-secret can fail to reach the
cluster. ArgoCD reports clickstack as Synced, but the live ExternalSecret keeps the old key
list, and a ClickHouse replica does not start. Issue #2412 tracks the cause. Until it closes,
compare the key list of the live clickstack-secret with git after each such merge. If they
differ, follow the unstick procedure in the ArgoCD runbook. The documented command removes the
argocd.argoproj.io/tracking-id annotation, so add that annotation to the manifest before the
apply.
The record index does not finish
Section titled “The record index does not finish”The screen “dns-control is building the record index.” is normal for about 30 seconds after a
start. The screen tries again every 5 seconds. The pod is not Ready until the first index exists,
because /readyz waits for it. Until then, the Service sends no request to the pod, and each
read of the index answers index_not_ready.
The first index waits for two inputs. First, the entry ledger reads the entries and the recent changes from the database. The ledger is the record of what dns-control wrote. Then the builder reads the pdns nodes. An index built without the ledger reads every curated RRset as an orphan, so dns-control publishes none.
If the screen stays, or the pod does not become Ready:
- Read the dns-control log lines in ClickStack.
- If the log shows
waiting for the entry ledger before the first index build, and thenentry ledger not reloadedevery 10 seconds, the database read fails. The error of the second line names the cause. Make sure that the CNPG clustermainis up and that the ExternalSecretdns-control-dbrenderedDATABASE_URL. - If the log shows a pdns node error, follow Node unreadable for the node that the log names.
- If the log has no lines, the pod did not start. Look at the pod events. A missing leg file or an empty Secret value stops the binary at start.
Related
Section titled “Related”- ClickStack: the
dns_control_readeruser and its password. - ArgoCD: the unstick procedure for an ExternalSecret array.
- DNS: the resolver pair, pdns, the Ansible roles and the manual undrain.
- Temporal: the namespace
dns-control, the node gateway and the node workers. - NATS: the stream
DNS_EVENTSand thedns-controluser. - MCP gateway clients: the LiteLLM routes, the two headers and the Claude Code entries.
- LiteLLM: the virtual keys and the key of
dns-agent-triage. - Services Catalog and Secrets.