Skip to content

dns-control telemetry over OTLP: design

This is a copy of the design of the OpenSpec change dns-control-otlp-observability. The change holds the proposal, the spec and the tasks.

See proposal.md for the motivation and specs/dns-control-telemetry/spec.md for the requirements. The state that shapes the approach:

  • internal/observe in apps/dns-control sets up one OTLP log provider with otlploghttp. dns-control uses it for one event, dns_control.ops_audit. Both binaries write JSON logs with slog to stdout. No code makes a span or an OTLP metric. The module pins go.opentelemetry.io/otel v1.46.0 and otlploghttp v0.22.0, and it holds no go.opentelemetry.io/contrib module as a direct dependency.
  • The dns-control binary runs the server when its first argument names no command. The server runs the REST and MCP listeners, the lineage refresh loop, the observed AAAA sweep, the ops poller, the session sweeper, the reload of the store ledger, and the Temporal worker of the change, export and ops workflows. The ops poller starts one ops.status workflow for each node every 30 seconds.
  • The webhook command runs as the container webhook in the external-dns pod, from the same image. It sends each external-dns sync to the lane API of dns-control (/api/lanes/external-dns/...) with a client-credentials token from Keycloak. It reads only the variables with the prefix DNS_CONTROL_WEBHOOK_.
  • The operator commands run as a second process in the dns-control pod through kubectl exec, and they inherit the environment of the server. They are entries, zones (import, export, delete, grant), rrsets, adopt, restore, drift and lane. Each one opens a pgx pool. zones import, zones grant, adopt, restore, drift and lane purge read pdns, and restore reads S3. zones grant, adopt --apply, restore and lane purge start a change workflow through a Temporal client of their own. No operator command writes pdns. The workflow in the server does. The operator commands log text to stderr at the level Warn.
  • zones import and entries import write their audit rows straight to the store, with no workflow (recordDirect in cmd/dns-control/zones.go, importEntries in entries.go).
  • The node worker runs on sandiego and portland as the systemd unit dns-node-worker. It reads /etc/dns-node-worker/config.json and refuses unknown fields. It runs activities only, with no workflow worker. It calls Kea and Blocky over HTTP, and it builds the Kea client on each call. It runs commands through nodeworker.ExecRunner, which gives a child only PATH. It serves three Prometheus gauges on 127.0.0.1:9477 and stops when that listener fails. The tasks in verify.yml of the role dns-node-worker read the certificate gauge from that page.
  • The unit dns-node-worker-pki runs Vault Agent, which renews the client certificate of the worker. It is a separate process and sends no OTLP.
  • The observed AAAA sweep is a Temporal workflow that the Schedule observed-aaaa-sweep starts every 5 minutes. It reads the client table of the UniFi controller through internal/unifi. It plans the record sets of the observed-aaaa lane as one batch on the system surface. The Postgres table observed_sweep holds last_good_read_at, the time of the last run that read the controller and did not fail.
  • The change service stores each plan in change_plans and each applied change in change_audit. A row of change_audit holds surface, subject, op, outcome, caller_kind and caller_azp. The table change_exports holds one row for each sink (nas, b2) that holds a change. A batch plan always records the op change. No MCP tool plans a batch. The batch callers are the lane API, the sweep and lane purge.
  • The lineage index gives each live RRset one of the states attributed, orphan, conflict and content_drift, and each claim with no live RRset the state dangling. The node comparison gives each RRset one of agree, differ, missing and unreadable. Each finding that is not attributed or agree carries a settle status, pending or settled. A finding settles after two readings and five minutes (internal/lineage/debounce.go), so a copy that LightningStream finishes in seconds never settles. The ops poller gives the VIP holder one of one_holder, none, split and not_read.
  • The binary has three Temporal clients: the server (cmd/dns-control/main.go), the operator commands (cmd/dns-control/zones.go) and the worker (cmd/dns-node-worker/main.go, with its dial options in config.go). Each one chains wire.Interceptor(), which validates every payload. The worker dials the Temporal node gateway at dns_node_worker_gateway_addr, today 192.168.200.146:7233, with mTLS and no gRPC compression. An Envoy gateway with an OPA ext_authz sidecar checks each call. The policy reads the body and the temporal-namespace header only.
  • The Postgres pools come from pgxpool.New in cmd/dns-control/main.go, entries.go and zones.go.
  • events.Publisher.Publish sends each event as a nats.Msg that already carries headers. The subjects are dns.change.<op>.<outcome> and dns.ops.<action>.<outcome>.
  • A release follows docs/operations/releases.md. A pull request that changes apps/dns-control/ adds a changie fragment. GoReleaser builds and signs the image when the release pull request merges, and it keeps the debug information in the binary. A re-pin sets the version in four places: the image: line of the Deployment of dns-control (version and index digest), provider.webhook.image.tag of external-dns, dns_node_worker_version in ansible/roles/dns-node-worker/defaults/main.yml, and the copy of the pin in docs/reference/services.md.
  • The Deployment of dns-control sets OTEL_EXPORTER_OTLP_ENDPOINT to http://cs-otel-collector.clickstack.svc.cluster.local:4318, OTEL_SERVICE_NAME to dns-control, and OTEL_EXPORTER_OTLP_HEADERS from the Secret dns-control-otlp-headers. The network policy allows egress to the clickstack namespace on TCP 4318. The pod runs as user 65532 with runAsNonRoot: true, and its container drops every capability.
  • Vector collects the stdout of every container in the cluster through one kubernetes_logs source with no namespace filter (argocd/app-configs/monitoring-vector/vector.yaml). Vector leaves out each container that the pod annotation vector.dev/exclude-containers names. The values of external-dns set no pod annotation today.
  • The collector otelcol-dns on each Pi reads the journal of seven units through the receiver journald. It scrapes Blocky, LightningStream and the worker through the receiver prometheus/blocky. It sends logs and metrics over OTLP gRPC to otel-gateway.fzymgc.house:443 with mTLS. The logs pipeline runs the processor resource, which sets service.name to otelcol-dns. The metrics pipeline runs resource/metrics, which keeps the service name of each point. The collector has no OTLP receiver and no traces pipeline, and nothing on the Pis listens on 4317 or 4318. The role renders the collector configuration with the ingest token in it, so its tasks in verify.yml never read the file.
  • The in-cluster collector cs-otel-collector accepts OTLP traces, metrics and logs and writes default.otel_traces, default.otel_metrics_* and default.otel_logs. Traces and logs keep 14 days.
  • HyperDX treats each distinct set of scope, resource and point attributes as one series. Its alert evaluator reduces each series to its last value in a bucket, then aggregates across the series.
  • This change allocates no address or prefix, so docs/reference/network.md does not change. No Terraform resource changes.

Goals:

  • One trace per request or change, across dns-control, its webhook, its operator commands and both workers, as the spec describes.
  • One instrumentation route for every process, with no privileged container.
  • Every signal leaves each binary over OTLP, and each log line reaches ClickStack once.
  • The application metrics that the alerts of dns-control-alerts-and-monitors read.

Non-Goals:

  • The telemetry of Blocky, pdns, Kea, keepalived, LightningStream and Vault Agent. The collector on each Pi keeps its journal and Prometheus inputs for those units.
  • apps/hl-assets-uploader, which holds its own copy of internal/observe. That copy stays as it is.
  • Tail sampling. The ops poller starts about 5,800 traces a day, and the lineage refresh about 2,900. The volume stays small for 14 days in ClickHouse, so every span is kept.
  • Dashboards, alerts and monitors. The change dns-control-alerts-and-monitors holds them.
  • A rate limit or a breaker on writes. Sean decided on 2026-10-03 that alerts on this telemetry replace them.
  • A count of the rows that zones import and entries import write. Their rows stay in change_audit.

D-1: Every process runs the OpenTelemetry SDK in process

Section titled “D-1: Every process runs the OpenTelemetry SDK in process”

Sean decided on 2026-10-04 that every process uses the OpenTelemetry Go SDK in process: the server, the webhook command, the operator commands and the node worker. Each process builds a tracer provider, a meter provider and a logger provider with the OTLP HTTP exporters, and it sets the W3C propagator.

This decision replaces the decision of 2026-09-29, which ran the OpenTelemetry Go eBPF agent (autoinstrumentation-go) as a privileged sidecar in the dns-control pod. On 2026-10-04 a reading of the upstream project at its latest release, v0.24.0 of 2026-04-27, found these facts:

Fact Effect on dns-control
v0.24.0 holds offsets for Go up to 1.26.1, otel up to v1.43.0 and grpc up to v1.80.0 dns-control uses Go 1.26.6, otel v1.46.0 and grpc v1.83.2. The agent reads each offset from the debug information of the binary instead, and the project tests no such build
The agent attaches to the first process whose executable matches, and it never looks again After a restart of the dns-control container, the sidecar stays on the old process. The spans stop, and nothing reports an error
The agent keeps its server span in an eBPF map, not in the Go request context A log record that a REST handler writes carries no trace id
The project documents only privileged: true with runAsUser: 0, and its write helper needs CAP_SYS_ADMIN A root container with every capability shares the process namespace of the process that holds the pdns API key, the OIDC client secret and the S3 keys
The agent has no probe for pgx or NATS Those seams need code on either route

The same reading found that the agent does not need kernel BTF. The decision of 2026-09-29 gave the missing BTF on the Pis as the reason for one route on the server and another on the worker. That reason was wrong, and with one route for every process the split goes away.

Alternative: the eBPF agent as a sidecar, as decided on 2026-09-29. Rejected for the facts above.

Alternative: the OpenTelemetry Operator, which injects the same agent from a pod annotation. The operator is not installed, and the injected agent has the same limits. Rejected.

D-2: One setup in internal/observe serves every process

Section titled “D-2: One setup in internal/observe serves every process”

internal/observe gains one function that builds the three providers and the propagator from the standard variables of D-7 and a service name. Each process calls it once when it starts. Each one calls Shutdown before it exits, with a timeout of 5 seconds, so a short operator command still exports its spans and a collector that does not answer never holds a process. An unset endpoint gives no-op providers, as internal/observe does for logs today. The exporters batch in the background, so no request waits on an export. The audit event keeps its logger on the same logger provider.

Each process passes its service name in code, and the name in code wins over OTEL_SERVICE_NAME. The operator commands inherit the environment of the server, so a name from the environment labels their spans as the server.

Process Service name
The server dns-control
webhook dns-control-webhook
An operator command dns-control-cli
The node worker dns-node-worker

A test in internal/observe fails when a collector that refuses every connection stops the caller, or when a global provider after the setup is not the provider that the setup built.

D-3: The worker sends to the collector on its own node

Section titled “D-3: The worker sends to the collector on its own node”

The worker sends every signal to http://127.0.0.1:4318 on its own node. The collector there forwards it through the gateway that it uses today. The unit already allows localhost in IPAddressAllow=.

The collector gains an otlp receiver on 127.0.0.1:4318 (HTTP only) and two pipelines for it: traces, and logs/otlp. Each runs resourcedetection/system, resource/metrics and batch, and each exports to otlp/gateway. resource/metrics sets host.name and deployment.environment and keeps the service name. Neither pipeline runs the processor resource, so the spans and the log records keep the service name dns-node-worker. The otlp receiver also joins the metrics pipeline, whose processor resource/metrics keeps the service name. The template task that renders the configuration gains validate: /usr/local/bin/otelcol-contrib validate --config %s, so a broken file never replaces a working one, and no task in verify.yml reads the token.

Alternative: the worker sends straight to the gateway. The worker then needs its own mTLS client certificate for the gateway and the ingest token. The local collector already holds both. Rejected.

Alternative: the otlp receiver joins the existing logs pipeline. The processor resource then renames the service of every worker log record to otelcol-dns. Rejected.

Seam Instrument
The REST and MCP listeners of the server, and the record API of the webhook otelhttp.NewHandler
Every outbound HTTP client: pdns, Keycloak, the UniFi controller, the S3 sinks and ClickHouse in the server, the lane API and Keycloak in the webhook, pdns and S3 in the operator commands, Kea and Blocky in the worker otelhttp.NewTransport. The worker builds its Kea client on each call, so the transport goes where the client is built
The Temporal client, workflows and activities the tracing interceptor of go.temporal.io/sdk/contrib/opentelemetry on all three clients, placed before wire.Interceptor(), so a refused payload shows as a failed span
The gRPC calls to Temporal otelgrpc.NewClientHandler in the dial options of all three clients
The Temporal SDK metrics NewMetricsHandler of the same contrib module, in the server and the worker
Postgres the tracer of github.com/exaring/otelpgx on the connection configuration. The three pools move from pgxpool.New to pgxpool.ParseConfig and pgxpool.NewWithConfig
The NATS publish a span in events.Publisher.Publish, and the global propagator writes traceparent into nats.Msg.Header, for the dns.change and the dns.ops subjects
An MCP tool a span for each tool in the logged() wrapper of internal/mcpserver, named after the tool, as a child of the server span
A worker command a span for each command in nodeworker.ExecRunner.Run. The child gets only PATH, and systemctl does not pass its environment to the unit that it starts, so no trace context goes to the child
A run of a background loop one root span for each run of the lineage refresh, the ops poller, the session sweeper and the reload of the store ledger, named after the loop, so the client spans of one run share one trace
An applied change the span of the apply carries dns_control.surface, dns_control.subject, dns_control.caller_kind and dns_control.caller_azp, so a trace names the client that started the change

Temporal carries the trace context in the headers of each workflow, child workflow and activity. The trace therefore crosses from the server to ExportWorkflow, to the worker queues and back, with no change to the protobuf payloads. The worker runs no workflows, so its spans are activity spans and their children.

D-5: Logs go through the otelslog bridge and the existing output

Section titled “D-5: Logs go through the otelslog bridge and the existing output”

Each process builds its slog logger from two handlers: the otelslog bridge to the logger provider, and its existing output. The server, the webhook and the worker keep the JSON handler on stdout, which adds trace_id and span_id from the context of each record. The operator commands keep their text handler on stderr at the level Warn.

The bridge makes OTLP the only path to ClickStack for these logs. The Pod of dns-control gains the annotation vector.dev/exclude-containers: dns-control, and the external-dns pod gains vector.dev/exclude-containers: webhook through podAnnotations in its values. The receiver journald of the Pi collector drops the unit dns-node-worker.service. It keeps dns-node-worker-pki.service, because Vault Agent sends no OTLP.

D-6: Metrics use one meter provider per process

Section titled “D-6: Metrics use one meter provider per process”

Each process sets one meter provider with the OTLP HTTP metric exporter and a periodic reader of 60 seconds. The server and the worker add the Go runtime metrics from go.opentelemetry.io/contrib/instrumentation/runtime and the Temporal SDK metrics. The worker moves its three gauges to observable gauges under their current names. The Prometheus endpoint of the worker, the variable dns_node_worker_metrics_addr and the scrape job dns-node-worker of the Pi collector go away in the same pull request. The task in verify.yml of the role dns-node-worker that read the certificate gauge from the endpoint changes to an openssl x509 -checkend 0 on the bundle, run as the worker user.

D-7: The OTLP settings come from the standard environment variables

Section titled “D-7: The OTLP settings come from the standard environment variables”

Each process reads OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS and OTEL_RESOURCE_ATTRIBUTES as the SDK defines them. The Deployment of dns-control adds k8s.pod.name and k8s.namespace.name through the downward API. Each process sets service.version from the version that its build stamps, and that value wins over the environment. A version label on the pod is a fifth copy of the pin, so the Deployment sets none. The worker unit sets its variables with Environment= lines, so config.json keeps its schema. The resource of a process holds no process id. A restart of the worker therefore keeps its metric series, and HyperDX counts one series for each worker.

D-8: The webhook reads its OTLP settings from the values of external-dns

Section titled “D-8: The webhook reads its OTLP settings from the values of external-dns”

The webhook reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS from new entries in provider.webhook.env of argocd/app-configs/external-dns/values.yaml. The headers come from a new ExternalSecret external-dns-otlp-headers in the external-dns namespace. It reads the same Vault key as dns-control-otlp-headers: fzymgc-house/cluster/clickstack, property otel_ingest_api_key. The policy external-secrets-operator already reads the paths under fzymgc-house/cluster/, so no Vault policy changes. No network policy selects the external-dns namespace, so no egress rule changes.

The webhook sends traceparent on each request to dns-control, so the server span of dns-control continues the trace of the external-dns sync.

D-9: The server sends eight application metrics

Section titled “D-9: The server sends eight application metrics”

The metrics carry the state that a span does not show. Each name has the prefix dns_control_, in the same form as the worker gauges. The server sends all eight.

Metric Kind Attributes Source
dns_control_change_applies_total counter surface, subject, op, outcome, caller_kind the Finalize activity, once for each change when its outcome is final (succeeded or failed)
dns_control_change_refusals_total counter surface, step (plan or apply), reason the change service, once for each refused plan or apply, with the reason code of the refusal
dns_control_export_pending_age_seconds gauge sink (nas or b2) change_audit and change_exports, read at each collection: the age of the oldest succeeded change that the sink does not hold yet, or 0
dns_control_lineage_rrsets gauge state, finding the published lineage index: the number of RRsets in each state, for each settle status
dns_control_lineage_refresh_timestamp_seconds gauge none the refresh loop: the Unix time of the last published index
dns_control_observed_sweep_timestamp_seconds gauge none last_good_read_at of the table observed_sweep, read at each collection
dns_control_lineage_node_disagreements gauge result, finding the node comparison of the published index: the number of RRsets for each result that is not agree, for each settle status
dns_control_vip_holder_status gauge status (one_holder, none, split or not_read) the ops poller (internal/ops/poller.go): 1 for the current status and 0 for the other three

The attribute values come from the code and form fixed sets:

Attribute Values
surface ui, rest, mcp, cli, system
subject entry, zone, rrset, batch
op create, change, delete. A batch always has change
outcome succeeded, failed
caller_kind person, client, static, system
reason the refusal codes of internal/change: role_required, index_not_ready, invalid_entry, plan_not_found, plan_expired, entry_changed, subject_changed, change_in_progress and the policy_* codes of the write policy
state attributed, orphan, conflict, content_drift, dangling
result differ, missing, unreadable
finding pending, settled, and none for attributed

Each gauge sends every value of its attributes on each collection, with 0 where nothing matches. A series then always exists, and an alert reads 0 instead of an empty bucket.

The Finalize activity runs in the server for every change that a workflow applies, the changes that an operator command starts included. The counter therefore needs no meter provider in an operator command. The refusals of an operator command reach the counter through the meter provider of D-2, which Shutdown flushes before the command exits.

The three gauges that read the database or the index use observable callbacks. A restart of dns-control then reports the stored value, not a zero that looks like an outage.

The attribute caller_azp stays off the metrics. Each dynamic client registration gives a new client id, so its set of values grows without a bound. The apply span and the audit row carry it.

The request rate, the errors and the duration of each surface come from the server spans and the MCP tool spans. Every span is kept, so a query of default.otel_traces gives the full numbers, and a second HTTP metric only repeats them. The failures of the change workflows and of the sweep come from the Temporal SDK metrics, which carry the workflow type.

  • [A library that the code adds sets its own global provider and replaces the one of D-2.] → The test of D-2 reads the global providers after the setup and fails when one is not the provider that the setup built.
  • [The OPA policy on the Temporal node gateway refuses a gRPC call that carries traceparent.] → The policy reads the body and the temporal-namespace header only, so the risk is low. Pull request 5 converges the passive node first and reads the gateway log before the active node. The rollback removes the otelgrpc handler, and the Temporal headers still carry the trace.
  • [During pull request 4 the dns-control logs reach ClickStack twice, from Vector and from OTLP, or not at all, for the minutes between the two changes.] → The annotation and the new pin land in one sync.
  • [An attribute with many values makes many series in otel_metrics_*.] → D-9 keeps each attribute to a fixed set of values and keeps caller_azp off the metrics.
  • [A secret reaches a span or a log record. The logged() wrapper writes the arguments of each MCP tool.] → otelhttp records no header, and a test of pull request 2 sends each credential header through each instrumented client and reads the exported spans.
  • [lb-vips-over-bgp pull request 2 changes dns_node_worker_gateway_addr in the same defaults file, and it converges the same role.] → The pull request that merges second rebases on the first and runs its converge after the converge of the first.

The proposal lists the five pull requests. Each live change has its rollback:

Pull request Live change Rollback
1 Converge of dns-otel-collector on each node git revert <merge>, then scripts/dns-playbook.sh --passive --tags dns-otel-collector and scripts/dns-playbook.sh --active --tags dns-otel-collector
2 None None
3 None. The release publishes an image that no manifest runs yet None
4 ArgoCD sync of dns-control (new pin, resource attributes, Vector annotation) and of external-dns (new webhook pin, OTLP variables, ExternalSecret, Vector annotation) git revert <merge>. ArgoCD syncs the previous manifests of each app
5 Converge of dns-node-worker and dns-otel-collector on each node git revert <merge>, then scripts/dns-playbook.sh --passive --tags dns-node-worker,dns-otel-collector and the same command with --active

None. The question of 2026-09-29 about the server span of the eBPF agent closed with D-1.