dns-control telemetry over OTLP: design
This is a copy of the design of the OpenSpec change dns-control-otlp-observability. The change
holds the proposal, the spec and the tasks.
Context
Section titled “Context”See proposal.md for the motivation and specs/dns-control-telemetry/spec.md for the requirements. The state that shapes the approach:
internal/observeinapps/dns-controlsets up one OTLP log provider withotlploghttp. dns-control uses it for one event,dns_control.ops_audit. Both binaries write JSON logs withslogto stdout. No code makes a span or an OTLP metric. The module pinsgo.opentelemetry.io/otelv1.46.0 andotlploghttpv0.22.0, and it holds nogo.opentelemetry.io/contribmodule as a direct dependency.- The
dns-controlbinary runs the server when its first argument names no command. The server runs the REST and MCP listeners, the lineage refresh loop, the observed AAAA sweep, the ops poller, the session sweeper, the reload of the store ledger, and the Temporal worker of the change, export and ops workflows. The ops poller starts oneops.statusworkflow for each node every 30 seconds. - The
webhookcommand runs as the containerwebhookin the external-dns pod, from the same image. It sends each external-dns sync to the lane API of dns-control (/api/lanes/external-dns/...) with a client-credentials token from Keycloak. It reads only the variables with the prefixDNS_CONTROL_WEBHOOK_. - The operator commands run as a second process in the dns-control pod through
kubectl exec, and they inherit the environment of the server. They areentries,zones(import,export,delete,grant),rrsets,adopt,restore,driftandlane. Each one opens a pgx pool.zones import,zones grant,adopt,restore,driftandlane purgeread pdns, andrestorereads S3.zones grant,adopt --apply,restoreandlane purgestart a change workflow through a Temporal client of their own. No operator command writes pdns. The workflow in the server does. The operator commands log text to stderr at the level Warn. zones importandentries importwrite their audit rows straight to the store, with no workflow (recordDirectincmd/dns-control/zones.go,importEntriesinentries.go).- The node worker runs on sandiego and portland as the systemd unit
dns-node-worker. It reads/etc/dns-node-worker/config.jsonand refuses unknown fields. It runs activities only, with no workflow worker. It calls Kea and Blocky over HTTP, and it builds the Kea client on each call. It runs commands throughnodeworker.ExecRunner, which gives a child onlyPATH. It serves three Prometheus gauges on127.0.0.1:9477and stops when that listener fails. The tasks inverify.ymlof the roledns-node-workerread the certificate gauge from that page. - The unit
dns-node-worker-pkiruns Vault Agent, which renews the client certificate of the worker. It is a separate process and sends no OTLP. - The observed AAAA sweep is a Temporal workflow that the Schedule
observed-aaaa-sweepstarts every 5 minutes. It reads the client table of the UniFi controller throughinternal/unifi. It plans the record sets of theobserved-aaaalane as one batch on thesystemsurface. The Postgres tableobserved_sweepholdslast_good_read_at, the time of the last run that read the controller and did not fail. - The change service stores each plan in
change_plansand each applied change inchange_audit. A row ofchange_auditholdssurface,subject,op,outcome,caller_kindandcaller_azp. The tablechange_exportsholds one row for each sink (nas,b2) that holds a change. A batch plan always records the opchange. No MCP tool plans a batch. The batch callers are the lane API, the sweep andlane purge. - The lineage index gives each live RRset one of the states
attributed,orphan,conflictandcontent_drift, and each claim with no live RRset the statedangling. The node comparison gives each RRset one ofagree,differ,missingandunreadable. Each finding that is notattributedoragreecarries a settle status,pendingorsettled. A finding settles after two readings and five minutes (internal/lineage/debounce.go), so a copy that LightningStream finishes in seconds never settles. The ops poller gives the VIP holder one ofone_holder,none,splitandnot_read. - The binary has three Temporal clients: the server (
cmd/dns-control/main.go), the operator commands (cmd/dns-control/zones.go) and the worker (cmd/dns-node-worker/main.go, with its dial options inconfig.go). Each one chainswire.Interceptor(), which validates every payload. The worker dials the Temporal node gateway atdns_node_worker_gateway_addr, today192.168.200.146:7233, with mTLS and no gRPC compression. An Envoy gateway with an OPAext_authzsidecar checks each call. The policy reads the body and thetemporal-namespaceheader only. - The Postgres pools come from
pgxpool.Newincmd/dns-control/main.go,entries.goandzones.go. events.Publisher.Publishsends each event as anats.Msgthat already carries headers. The subjects aredns.change.<op>.<outcome>anddns.ops.<action>.<outcome>.- A release follows
docs/operations/releases.md. A pull request that changesapps/dns-control/adds a changie fragment. GoReleaser builds and signs the image when the release pull request merges, and it keeps the debug information in the binary. A re-pin sets the version in four places: theimage:line of the Deployment of dns-control (version and index digest),provider.webhook.image.tagof external-dns,dns_node_worker_versioninansible/roles/dns-node-worker/defaults/main.yml, and the copy of the pin indocs/reference/services.md. - The Deployment of dns-control sets
OTEL_EXPORTER_OTLP_ENDPOINTtohttp://cs-otel-collector.clickstack.svc.cluster.local:4318,OTEL_SERVICE_NAMEtodns-control, andOTEL_EXPORTER_OTLP_HEADERSfrom the Secretdns-control-otlp-headers. The network policy allows egress to theclickstacknamespace on TCP 4318. The pod runs as user 65532 withrunAsNonRoot: true, and its container drops every capability. - Vector collects the stdout of every container in the cluster through one
kubernetes_logssource with no namespace filter (argocd/app-configs/monitoring-vector/vector.yaml). Vector leaves out each container that the pod annotationvector.dev/exclude-containersnames. The values of external-dns set no pod annotation today. - The collector
otelcol-dnson each Pi reads the journal of seven units through the receiverjournald. It scrapes Blocky, LightningStream and the worker through the receiverprometheus/blocky. It sends logs and metrics over OTLP gRPC tootel-gateway.fzymgc.house:443with mTLS. Thelogspipeline runs the processorresource, which setsservice.nametootelcol-dns. Themetricspipeline runsresource/metrics, which keeps the service name of each point. The collector has no OTLP receiver and no traces pipeline, and nothing on the Pis listens on 4317 or 4318. The role renders the collector configuration with the ingest token in it, so its tasks inverify.ymlnever read the file. - The in-cluster collector
cs-otel-collectoraccepts OTLP traces, metrics and logs and writesdefault.otel_traces,default.otel_metrics_*anddefault.otel_logs. Traces and logs keep 14 days. - HyperDX treats each distinct set of scope, resource and point attributes as one series. Its alert evaluator reduces each series to its last value in a bucket, then aggregates across the series.
- This change allocates no address or prefix, so
docs/reference/network.mddoes not change. No Terraform resource changes.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- One trace per request or change, across dns-control, its webhook, its operator commands and both workers, as the spec describes.
- One instrumentation route for every process, with no privileged container.
- Every signal leaves each binary over OTLP, and each log line reaches ClickStack once.
- The application metrics that the alerts of
dns-control-alerts-and-monitorsread.
Non-Goals:
- The telemetry of Blocky, pdns, Kea, keepalived, LightningStream and Vault Agent. The collector on each Pi keeps its journal and Prometheus inputs for those units.
apps/hl-assets-uploader, which holds its own copy ofinternal/observe. That copy stays as it is.- Tail sampling. The ops poller starts about 5,800 traces a day, and the lineage refresh about 2,900. The volume stays small for 14 days in ClickHouse, so every span is kept.
- Dashboards, alerts and monitors. The change
dns-control-alerts-and-monitorsholds them. - A rate limit or a breaker on writes. Sean decided on 2026-10-03 that alerts on this telemetry replace them.
- A count of the rows that
zones importandentries importwrite. Their rows stay inchange_audit.
Decisions
Section titled “Decisions”D-1: Every process runs the OpenTelemetry SDK in process
Section titled “D-1: Every process runs the OpenTelemetry SDK in process”Sean decided on 2026-10-04 that every process uses the OpenTelemetry Go SDK in process: the
server, the webhook command, the operator commands and the node worker. Each process builds a
tracer provider, a meter provider and a logger provider with the OTLP HTTP exporters, and it sets
the W3C propagator.
This decision replaces the decision of 2026-09-29, which ran the OpenTelemetry Go eBPF agent
(autoinstrumentation-go) as a privileged sidecar in the dns-control pod. On 2026-10-04 a reading
of the upstream project at its latest release, v0.24.0 of 2026-04-27, found these facts:
| Fact | Effect on dns-control |
|---|---|
v0.24.0 holds offsets for Go up to 1.26.1, otel up to v1.43.0 and grpc up to v1.80.0 |
dns-control uses Go 1.26.6, otel v1.46.0 and grpc v1.83.2. The agent reads each offset from the debug information of the binary instead, and the project tests no such build |
| The agent attaches to the first process whose executable matches, and it never looks again | After a restart of the dns-control container, the sidecar stays on the old process. The spans stop, and nothing reports an error |
| The agent keeps its server span in an eBPF map, not in the Go request context | A log record that a REST handler writes carries no trace id |
The project documents only privileged: true with runAsUser: 0, and its write helper needs CAP_SYS_ADMIN |
A root container with every capability shares the process namespace of the process that holds the pdns API key, the OIDC client secret and the S3 keys |
| The agent has no probe for pgx or NATS | Those seams need code on either route |
The same reading found that the agent does not need kernel BTF. The decision of 2026-09-29 gave the missing BTF on the Pis as the reason for one route on the server and another on the worker. That reason was wrong, and with one route for every process the split goes away.
Alternative: the eBPF agent as a sidecar, as decided on 2026-09-29. Rejected for the facts above.
Alternative: the OpenTelemetry Operator, which injects the same agent from a pod annotation. The operator is not installed, and the injected agent has the same limits. Rejected.
D-2: One setup in internal/observe serves every process
Section titled “D-2: One setup in internal/observe serves every process”internal/observe gains one function that builds the three providers and the propagator from the
standard variables of D-7 and a service name. Each process calls it once when it starts. Each one
calls Shutdown before it exits, with a timeout of 5 seconds, so a short operator command still
exports its spans and a collector that does not answer never holds a process. An unset endpoint
gives no-op providers, as internal/observe does for logs today. The exporters batch in the
background, so no request waits on an export. The audit event keeps its logger on the same logger
provider.
Each process passes its service name in code, and the name in code wins over
OTEL_SERVICE_NAME. The operator commands inherit the environment of the server, so a name from
the environment labels their spans as the server.
| Process | Service name |
|---|---|
| The server | dns-control |
webhook |
dns-control-webhook |
| An operator command | dns-control-cli |
| The node worker | dns-node-worker |
A test in internal/observe fails when a collector that refuses every connection stops the
caller, or when a global provider after the setup is not the provider that the setup built.
D-3: The worker sends to the collector on its own node
Section titled “D-3: The worker sends to the collector on its own node”The worker sends every signal to http://127.0.0.1:4318 on its own node. The collector there
forwards it through the gateway that it uses today. The unit already allows localhost in
IPAddressAllow=.
The collector gains an otlp receiver on 127.0.0.1:4318 (HTTP only) and two pipelines for it:
traces, and logs/otlp. Each runs resourcedetection/system, resource/metrics and batch,
and each exports to otlp/gateway. resource/metrics sets host.name and
deployment.environment and keeps the service name. Neither pipeline runs the processor
resource, so the spans and the log records keep the service name dns-node-worker. The otlp receiver also joins the metrics pipeline, whose
processor resource/metrics keeps the service name. The template task that renders the
configuration gains validate: /usr/local/bin/otelcol-contrib validate --config %s, so a broken
file never replaces a working one, and no task in verify.yml reads the token.
Alternative: the worker sends straight to the gateway. The worker then needs its own mTLS client certificate for the gateway and the ingest token. The local collector already holds both. Rejected.
Alternative: the otlp receiver joins the existing logs pipeline. The processor resource then
renames the service of every worker log record to otelcol-dns. Rejected.
D-4: The seams of each process
Section titled “D-4: The seams of each process”| Seam | Instrument |
|---|---|
| The REST and MCP listeners of the server, and the record API of the webhook | otelhttp.NewHandler |
| Every outbound HTTP client: pdns, Keycloak, the UniFi controller, the S3 sinks and ClickHouse in the server, the lane API and Keycloak in the webhook, pdns and S3 in the operator commands, Kea and Blocky in the worker | otelhttp.NewTransport. The worker builds its Kea client on each call, so the transport goes where the client is built |
| The Temporal client, workflows and activities | the tracing interceptor of go.temporal.io/sdk/contrib/opentelemetry on all three clients, placed before wire.Interceptor(), so a refused payload shows as a failed span |
| The gRPC calls to Temporal | otelgrpc.NewClientHandler in the dial options of all three clients |
| The Temporal SDK metrics | NewMetricsHandler of the same contrib module, in the server and the worker |
| Postgres | the tracer of github.com/exaring/otelpgx on the connection configuration. The three pools move from pgxpool.New to pgxpool.ParseConfig and pgxpool.NewWithConfig |
| The NATS publish | a span in events.Publisher.Publish, and the global propagator writes traceparent into nats.Msg.Header, for the dns.change and the dns.ops subjects |
| An MCP tool | a span for each tool in the logged() wrapper of internal/mcpserver, named after the tool, as a child of the server span |
| A worker command | a span for each command in nodeworker.ExecRunner.Run. The child gets only PATH, and systemctl does not pass its environment to the unit that it starts, so no trace context goes to the child |
| A run of a background loop | one root span for each run of the lineage refresh, the ops poller, the session sweeper and the reload of the store ledger, named after the loop, so the client spans of one run share one trace |
| An applied change | the span of the apply carries dns_control.surface, dns_control.subject, dns_control.caller_kind and dns_control.caller_azp, so a trace names the client that started the change |
Temporal carries the trace context in the headers of each workflow, child workflow and activity.
The trace therefore crosses from the server to ExportWorkflow, to the worker queues and back,
with no change to the protobuf payloads. The worker runs no workflows, so its spans are activity
spans and their children.
D-5: Logs go through the otelslog bridge and the existing output
Section titled “D-5: Logs go through the otelslog bridge and the existing output”Each process builds its slog logger from two handlers: the otelslog bridge to the logger
provider, and its existing output. The server, the webhook and the worker keep the JSON handler on
stdout, which adds trace_id and span_id from the context of each record. The operator
commands keep their text handler on stderr at the level Warn.
The bridge makes OTLP the only path to ClickStack for these logs. The Pod of dns-control gains the
annotation vector.dev/exclude-containers: dns-control, and the external-dns pod gains
vector.dev/exclude-containers: webhook through podAnnotations in its values. The receiver
journald of the Pi collector drops the unit dns-node-worker.service. It keeps
dns-node-worker-pki.service, because Vault Agent sends no OTLP.
D-6: Metrics use one meter provider per process
Section titled “D-6: Metrics use one meter provider per process”Each process sets one meter provider with the OTLP HTTP metric exporter and a periodic reader of
60 seconds. The server and the worker add the Go runtime metrics from
go.opentelemetry.io/contrib/instrumentation/runtime and the Temporal SDK metrics. The worker
moves its three gauges to observable gauges under their current names. The Prometheus endpoint of
the worker, the variable dns_node_worker_metrics_addr and the scrape job dns-node-worker of
the Pi collector go away in the same pull request. The task in verify.yml of the role dns-node-worker
that read the certificate gauge from the endpoint changes to an openssl x509 -checkend 0 on the
bundle, run as the worker user.
D-7: The OTLP settings come from the standard environment variables
Section titled “D-7: The OTLP settings come from the standard environment variables”Each process reads OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS and
OTEL_RESOURCE_ATTRIBUTES as the SDK defines them. The Deployment of dns-control adds
k8s.pod.name and k8s.namespace.name through the downward API. Each process sets
service.version from the version that its build stamps, and that value wins over the
environment. A version label on the pod is a fifth copy of the pin, so the Deployment sets none.
The worker unit sets its variables with Environment= lines, so config.json keeps its schema.
The resource of a process holds no process id. A restart of the worker therefore keeps its metric
series, and HyperDX counts one series for each worker.
D-8: The webhook reads its OTLP settings from the values of external-dns
Section titled “D-8: The webhook reads its OTLP settings from the values of external-dns”The webhook reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS from new entries
in provider.webhook.env of argocd/app-configs/external-dns/values.yaml. The headers come from a
new ExternalSecret external-dns-otlp-headers in the external-dns namespace. It reads the same
Vault key as dns-control-otlp-headers: fzymgc-house/cluster/clickstack, property
otel_ingest_api_key. The policy external-secrets-operator already reads the paths under
fzymgc-house/cluster/, so no Vault policy changes. No network policy selects the external-dns
namespace, so no egress rule changes.
The webhook sends traceparent on each request to dns-control, so the server span of dns-control
continues the trace of the external-dns sync.
D-9: The server sends eight application metrics
Section titled “D-9: The server sends eight application metrics”The metrics carry the state that a span does not show. Each name has the prefix dns_control_,
in the same form as the worker gauges. The server sends all eight.
| Metric | Kind | Attributes | Source |
|---|---|---|---|
dns_control_change_applies_total |
counter | surface, subject, op, outcome, caller_kind |
the Finalize activity, once for each change when its outcome is final (succeeded or failed) |
dns_control_change_refusals_total |
counter | surface, step (plan or apply), reason |
the change service, once for each refused plan or apply, with the reason code of the refusal |
dns_control_export_pending_age_seconds |
gauge | sink (nas or b2) |
change_audit and change_exports, read at each collection: the age of the oldest succeeded change that the sink does not hold yet, or 0 |
dns_control_lineage_rrsets |
gauge | state, finding |
the published lineage index: the number of RRsets in each state, for each settle status |
dns_control_lineage_refresh_timestamp_seconds |
gauge | none | the refresh loop: the Unix time of the last published index |
dns_control_observed_sweep_timestamp_seconds |
gauge | none | last_good_read_at of the table observed_sweep, read at each collection |
dns_control_lineage_node_disagreements |
gauge | result, finding |
the node comparison of the published index: the number of RRsets for each result that is not agree, for each settle status |
dns_control_vip_holder_status |
gauge | status (one_holder, none, split or not_read) |
the ops poller (internal/ops/poller.go): 1 for the current status and 0 for the other three |
The attribute values come from the code and form fixed sets:
| Attribute | Values |
|---|---|
surface |
ui, rest, mcp, cli, system |
subject |
entry, zone, rrset, batch |
op |
create, change, delete. A batch always has change |
outcome |
succeeded, failed |
caller_kind |
person, client, static, system |
reason |
the refusal codes of internal/change: role_required, index_not_ready, invalid_entry, plan_not_found, plan_expired, entry_changed, subject_changed, change_in_progress and the policy_* codes of the write policy |
state |
attributed, orphan, conflict, content_drift, dangling |
result |
differ, missing, unreadable |
finding |
pending, settled, and none for attributed |
Each gauge sends every value of its attributes on each collection, with 0 where nothing matches. A series then always exists, and an alert reads 0 instead of an empty bucket.
The Finalize activity runs in the server for every change that a workflow applies, the changes
that an operator command starts included. The counter therefore needs no meter provider in an
operator command. The refusals of an operator command reach the counter through the meter
provider of D-2, which Shutdown flushes before the command exits.
The three gauges that read the database or the index use observable callbacks. A restart of dns-control then reports the stored value, not a zero that looks like an outage.
The attribute caller_azp stays off the metrics. Each dynamic client registration gives a new
client id, so its set of values grows without a bound. The apply span and the audit row carry it.
The request rate, the errors and the duration of each surface come from the server spans and the
MCP tool spans. Every span is kept, so a query of default.otel_traces gives the full numbers, and
a second HTTP metric only repeats them. The failures of the change workflows and of the sweep come
from the Temporal SDK metrics, which carry the workflow type.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [A library that the code adds sets its own global provider and replaces the one of D-2.] → The test of D-2 reads the global providers after the setup and fails when one is not the provider that the setup built.
- [The OPA policy on the Temporal node gateway refuses a gRPC call that carries
traceparent.] → The policy reads the body and thetemporal-namespaceheader only, so the risk is low. Pull request 5 converges the passive node first and reads the gateway log before the active node. The rollback removes theotelgrpchandler, and the Temporal headers still carry the trace. - [During pull request 4 the dns-control logs reach ClickStack twice, from Vector and from OTLP, or not at all, for the minutes between the two changes.] → The annotation and the new pin land in one sync.
- [An attribute with many values makes many series in
otel_metrics_*.] → D-9 keeps each attribute to a fixed set of values and keepscaller_azpoff the metrics. - [A secret reaches a span or a log record. The
logged()wrapper writes the arguments of each MCP tool.] →otelhttprecords no header, and a test of pull request 2 sends each credential header through each instrumented client and reads the exported spans. - [
lb-vips-over-bgppull request 2 changesdns_node_worker_gateway_addrin the same defaults file, and it converges the same role.] → The pull request that merges second rebases on the first and runs its converge after the converge of the first.
Migration Plan
Section titled “Migration Plan”The proposal lists the five pull requests. Each live change has its rollback:
| Pull request | Live change | Rollback |
|---|---|---|
| 1 | Converge of dns-otel-collector on each node |
git revert <merge>, then scripts/dns-playbook.sh --passive --tags dns-otel-collector and scripts/dns-playbook.sh --active --tags dns-otel-collector |
| 2 | None | None |
| 3 | None. The release publishes an image that no manifest runs yet | None |
| 4 | ArgoCD sync of dns-control (new pin, resource attributes, Vector annotation) and of external-dns (new webhook pin, OTLP variables, ExternalSecret, Vector annotation) |
git revert <merge>. ArgoCD syncs the previous manifests of each app |
| 5 | Converge of dns-node-worker and dns-otel-collector on each node |
git revert <merge>, then scripts/dns-playbook.sh --passive --tags dns-node-worker,dns-otel-collector and the same command with --active |
Open Questions
Section titled “Open Questions”None. The question of 2026-09-29 about the server span of the eBPF agent closed with D-1.