ClickStack Runbook

ClickStack (ClickHouse + HyperDX + MongoDB + OTel Collector) is the cluster’s logs, metrics, and traces backend.

Endpoints

Service	Endpoint
HyperDX UI	https://hyperdx.fzymgc.house (Keycloak OIDC)
OTLP, in-cluster	cs-otel-collector.clickstack.svc:4317 (gRPC) / :4318 (HTTP), bearer token
OTLP, external	otel-gateway.fzymgc.house:443 (Traefik, mTLS client certificate, gRPC) → cs-otel-collector:4317, bearer token
ClickHouse HTTP	https://ch.fzymgc.house (Traefik) → cs-clickstack-clickhouse-clickhouse-headless:8123
ClickHouse native	ch-native.fzymgc.house:443 (Traefik TCP, TLS SNI) → cs-clickstack-clickhouse-clickhouse-headless:9000
ClickHouse, in-cluster	cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc, :8123 HTTP / :9000 native, no TLS

The otel-collector-lb LoadBalancer (192.168.20.149:4317/:4318) also exposes the collector. It is plaintext with bearer auth: the collector’s receivers have no TLS, and the otel-collector-tls Certificate issued for that address is mounted by nothing. Send external OTLP through otel-gateway.fzymgc.house instead.

Secrets

Vault path fzymgc-house/cluster/clickstack:

Key	Purpose
admin_password	ClickHouse admin password
mongo_admin_password	MongoDB admin password
mongo_replicaset_keyfile	MongoDB internal replica-set auth
hyperdx_bootstrap_secret	HyperDX session-signing key, delivered to the app as EXPRESS_SESSION_SECRET. Rotating it signs every user out once; Reloader restarts the app when the Secret changes.
hyperdx_oidc_client_secret	Written by tf/keycloak/clickstack.tf from the Keycloak client secret
otel_ingest_api_key	OTLP ingest bearer token (HyperDX team key) — the key the otlp/hyperdx receiver actually validates; used by the otel-scraper collectors (INGEST_TOKEN) and any trace producer. See Traces Ingestion.
otel_ingest_token	Legacy/vestigial — does not authenticate OTLP (returns 401). No longer mapped into clickstack-secret; the only remaining reference is the vector/otel-ingest-token ExternalSecret, where it survives solely to keep that ExternalSecret valid (ESO rejects a data-less ExternalSecret). Retire fully once Vector’s CLICKHOUSE_PASSWORD is Vault-sourced. See the token-gotcha note below.

Bootstrap (one-time)

Run from a workstation with vault token:

vault kv put fzymgc-house/cluster/clickstack \
  admin_password="$(openssl rand -base64 32)" \
  mongo_admin_password="$(openssl rand -base64 32)" \
  mongo_replicaset_keyfile="$(openssl rand -base64 756)" \
  hyperdx_bootstrap_secret="$(openssl rand -base64 32)" \
  hyperdx_oidc_client_secret="placeholder" \
  otel_ingest_token="$(openssl rand -base64 48)"

The hyperdx_oidc_client_secret placeholder is overwritten by tf/keycloak/clickstack.tf on first apply.

otel_ingest_token does not authenticate OTLP; it is seeded only because the vector/otel-ingest-token ExternalSecret still references it (ESO requires a non-empty data block). The real OTLP key — otel_ingest_api_key — cannot be pre-seeded: it is the HyperDX team key, generated on first boot. Once HyperDX is up, capture it into Vault:

key=$(kubectl exec -n clickstack deploy/cs-otel-collector -- \
  sh -c "grep -A2 'bearertokenauth/hyperdx' /etc/otel/supervisor-data/effective.yaml \
         | grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | head -1")
[ -n "$key" ] || { echo "ERROR: no HyperDX key found in effective.yaml"; exit 1; }
# Pipe via stdin (key=-) so the token isn't exposed in shell history / ps:
printf '%s' "$key" | vault kv patch fzymgc-house/cluster/clickstack otel_ingest_api_key=-

Verify: vault kv get fzymgc-house/cluster/clickstack | rg created_time

OTel Ingest Bearer-Token Rotation

The token that authenticates OTLP ingestion is the HyperDX team API key, mirrored into Vault as otel_ingest_api_key. The collector validates the HyperDX-issued key (delivered via OpAMP), so HyperDX is the source of truth and Vault must be updated to match — do not rotate otel_ingest_token (it does not authenticate OTLP; see the token-gotcha note below).

1. In HyperDX UI → Team Settings → API Keys, regenerate the team ingestion key. The collector picks up the new key automatically over OpAMP.

2. Update Vault to match so producers stay in sync:

   vault kv patch fzymgc-house/cluster/clickstack \
     otel_ingest_api_key="<new-hyperdx-team-key>"

3. ESO refreshes the otel-scraper INGEST_TOKEN within 15 minutes; force it and restart the scraper collectors if you need it sooner:

   kubectl annotate externalsecret otel-ingest-token -n otel-scraper \
     force-sync=$(date +%s) --overwrite
   kubectl rollout restart deployment/otel-scrape-opentelemetry-collector -n otel-scraper
   kubectl rollout restart daemonset/otel-node-opentelemetry-collector-agent -n otel-scraper

4. Update any external producers with the new key (e.g. the Firewalla edge collector — ansible/roles/otel-collector, property otel_ingest_api_key).

Collector Config Is OpAMP-Managed (not the static relay.yaml)

cs-otel-collector runs the clickstack-otel-collector image, which is an OpAMP supervisor, not a bare collector. On startup the supervisor connects to the HyperDX OpAMP server (OPAMP_SERVER_URL=http://cs-clickstack-app:4320, from clickstack-config) and is handed an effective config that replaces the bootstrap file. The two configs differ completely:

Source	Where	Exporters
Bootstrap (chart default)	ConfigMap cs-otel-collector key relay, mounted --config=/conf/relay.yaml	debug only — all pipelines
Effective (live)	/etc/otel/supervisor-data/effective.yaml in the pod	clickhouse — logs, metrics, and traces

Consequence: you cannot change ingestion by editing the Helm otelCollector values or the bootstrap relay.yaml — OpAMP overwrites it at runtime. The debug-only relay.yaml is a red herring; inspect the effective config instead:

kubectl exec -n clickstack deploy/cs-otel-collector -- \
  cat /etc/otel/supervisor-data/effective.yaml

Traces Ingestion (why otel_traces can be empty)

The effective config’s traces pipeline is fully wired: otlp/hyperdx receiver (:4317 gRPC / :4318 HTTP, bearertokenauth/hyperdx) → clickhouse exporter → default.otel_traces. The pipeline is not the bottleneck — an empty otel_traces means no application is sending OTLP traces, not that ingestion is broken. (Logs come from Vector writing directly to ClickHouse, bypassing the collector entirely; metrics come from the otel-scraper collectors pushing OTLP + the collector’s own prometheus self-scrape — neither path carries traces.)

Producer contract — to land traces, an app MUST send OTLP with the HyperDX ingestion API key as the authorization header:

From	Endpoint	Auth header
In-cluster	cs-otel-collector.clickstack.svc.cluster.local:4317 (gRPC) / :4318 (HTTP POST /v1/traces)	authorization: <hyperdx-api-key>
External	otel-gateway.fzymgc.house:443 (gRPC, mTLS client certificate)	authorization: <hyperdx-api-key>

Without the header the receiver returns 401 missing or empty authorization header; all spans are dropped.

Configured producers:

• litellm — traces and GenAI metrics via native OTLP push from the proxy (ServiceName='litellm'). Four metric instruments arrive, all histograms: gen_ai.client.operation.duration, gen_ai.client.response.duration, gen_ai.client.token.usage, gen_ai.usage.cost (the two gen_ai.server.time_* instruments are streaming-only and have never been observed). On the metrics gen_ai.request.model is the upstream model, not the lane alias; per-key attribution is metadata.user_api_key_alias. The retired gateway’s ServiceName='agentgateway' rows stay queryable until TTL; nothing backfills them into the LiteLLM vocabulary.

Token gotcha — two similarly-named Vault keys, only one authenticates OTLP. The otlp/hyperdx receiver’s bearertokenauth validates HyperDX’s internally-generated team ingestion API key (a UUID, created on first boot, stored in Mongo and pushed to the collector via OpAMP). Vault path fzymgc-house/cluster/clickstack holds it under key otel_ingest_api_key — this is the one the otel-scraper collectors use (mapped to their INGEST_TOKEN), and it is what a trace producer must send. The sibling key otel_ingest_token (a base64 openssl rand string, now referenced only by the vector/otel-ingest-token ExternalSecret’s OTEL_INGEST_TOKEN) does not authenticate OTLP — posting with it returns 401. It is effectively vestigial: Vector writes straight to ClickHouse (no OTLP), and the collector gets its real key via OpAMP. To rotate the real key, follow the OTel Ingest Bearer-Token Rotation runbook above (it operates on otel_ingest_api_key, not this key). To read the live key, use HyperDX UI → Team Settings → API Keys or bearertokenauth/hyperdx.tokens[0] in /etc/otel/supervisor-data/effective.yaml.

Smoke test (confirms the pipeline end-to-end without instrumenting an app):

# Extract the live HyperDX team key (full UUID pattern, not a loose hex match).
TOKEN=$(kubectl exec -n clickstack deploy/cs-otel-collector -- \
  sh -c "grep -A2 'bearertokenauth/hyperdx' /etc/otel/supervisor-data/effective.yaml \
         | grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | head -1")
# Fail loudly if extraction came up empty — otherwise the POST below sends an
# empty authorization header and the 401 looks like a broken pipeline.
[ -n "$TOKEN" ] || { echo "ERROR: no HyperDX key found in effective.yaml (layout changed?)"; exit 1; }

kubectl port-forward -n clickstack svc/cs-otel-collector 14318:4318 &
NOW=$(date +%s)000000000
# Pass the token via a 0600 header file (curl -H @file) so it does not appear in
# the process args / `ps` output on shared or audited hosts.
HDR=$(mktemp); chmod 600 "$HDR"; printf 'authorization: %s\n' "$TOKEN" > "$HDR"
curl -s -w '%{http_code}\n' -X POST http://127.0.0.1:14318/v1/traces \
  -H 'Content-Type: application/json' -H @"$HDR" \
  --data "{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"trace-probe\"}}]},\"scopeSpans\":[{\"spans\":[{\"traceId\":\"5b8efff798038103d269b633813fc60c\",\"spanId\":\"eee19b7ec3c1b174\",\"name\":\"probe\",\"kind\":1,\"startTimeUnixNano\":\"$NOW\",\"endTimeUnixNano\":\"$NOW\"}]}]}]}"
rm -f "$HDR"
# Expect HTTP 200 {"partialSuccess":{}}, then:
#   SELECT count() FROM default.otel_traces WHERE ServiceName='trace-probe';  -- > 0

Mongo-Down Failure Mode

HyperDX UI requires MongoDB for session state. Mongo unavailable causes HyperDX to return 5xx.

MongoDB runs as a 3-member replica set with a PDB of minAvailable: 2, so it tolerates the loss of one member. Investigate:

kubectl get pods -n clickstack -l app=cs-clickstack-mongodb-svc
kubectl logs -n clickstack -l app=cs-clickstack-mongodb-svc -c mongod --tail=50

Watchdog signal contract

clickstack-watchdog (CronJob, every 5 min, argocd/app-configs/clickstack/watchdog-cronjob.yaml) runs two health checks and pushes an uptime-kuma heartbeat only if both pass. Since ALERT-02 its two failure modes are split, so the Job’s exit code and the heartbeat mean different things. Read this table before drawing any conclusion from kubectl get jobs -n clickstack.

Case	Exit code	Heartbeat sent?	Which alert speaks	Where the reason is readable
ClickStack unhealthy — ClickHouse unreachable, ingest stale (>300s), engine drift, or a malformed query result	0 (Job SUCCEEDS)	no	clickstack-watchdog uptime-kuma monitor goes DOWN after ~2 missed beats → Pushover (cluster-edge)	pod stdout, HEALTH CHECK FAILED: …, 14 days in default.otel_logs
Push delivery broken — endpoint error, Service unreachable, token rejected	1 (Job FAILS)	no	ClickStack watchdog push delivery FAILING, a HyperDX tile alert on kube_job_failed → Pushover (cluster-infra)	pod stdout, PUSH DELIVERY FAILED: … (wget exit status N), 14 days in default.otel_logs
Both fine	0	yes	none	pod stdout, stale=…s drift=… then {"ok":true}
Job died before or during the checks — activeDeadlineSeconds: 120 exceeded (Job condition reason DeadlineExceeded), OOMKilled against the 256Mi limit, an image-pull failure, or node eviction	Job FAILS — and backoffLimit: 0 means there is no retry	no	the same ClickStack watchdog push delivery FAILING alert as row 2, which is the whole problem: the two cases are indistinguishable from the alert alone	kubectl get job -n clickstack <name> -o jsonpath='{.status.conditions[*].reason}' — pod stdout may be truncated or empty, because the container was killed rather than exiting

Row 4 is the one that bites mid-incident. If ClickHouse is alive but saturated and the freshness query blocks past the 120s deadline, the Job is killed with DeadlineExceeded, no heartbeat is sent, this alert fires AND the paired clickstack-watchdog uptime-kuma monitor goes DOWN — and reading row 2’s meaning into that page points you away from the actual fault. Read the Job condition reason first, before either branch.

A green Job is not a health verdict. Rows 1 and 3 both exit 0. The ClickStack-health signal is heartbeat absence, which is ADR hl-hiiu’s dead-man’s switch by design — an alert living inside the observability stack cannot fire when the stack is dark. A non-zero exit means the notification path is broken or the Job died before the script finished — row 4’s jsonpath is what tells the two apart.

Note that ADR hl-hiiu describes the heartbeat message as carrying the failing check for triage. The implementation does not do that and cannot: the push happens only on success, so there is no failing check to carry. Triage reads the pod’s stdout in default.otel_logs instead (the column is Timestamp, never TimestampTime):

kubectl exec -n clickstack "$(kubectl get pod -n clickstack \
  -l clickhouse.com/role=clickhouse-server -o name | head -1 | cut -d/ -f2)" \
  -- clickhouse-client --query "SELECT Timestamp, Body FROM default.otel_logs \
  WHERE ResourceAttributes['k8s.pod.name'] LIKE 'clickstack-watchdog-%' \
  ORDER BY Timestamp DESC LIMIT 20"

Triage: ClickStack watchdog push delivery FAILING

The long form of the body ClickStack watchdog push delivery FAILING pages. The page carries the observation, the first triage command and a pointer here; the triage below is the rest of it.

Rules in. The uptime-kuma push endpoint returning an error, the internal Service uptime-kuma.uptime-kuma.svc.cluster.local being unreachable, or the push token being rejected — and equally every Job-level failure the script never sees: the Job exceeding activeDeadlineSeconds: 120 and being killed with reason DeadlineExceeded, an OOMKill against the 256Mi memory limit, an image-pull failure, or node eviction. Those bounds are set in argocd/app-configs/clickstack/watchdog-cronjob.yaml.

Rules out — nothing unconditionally, and in particular NOT ClickStack ingest health. The freshness and engine-drift checks ran and came back clean only if the script ran to completion: a health failure withholds the heartbeat and exits 0, so a script that finished and still failed did fail on the push. A Job killed before or during those checks says nothing about ingest health either way. If ClickHouse is alive but saturated and the freshness query blocks past the 120s deadline, the Job is killed, no heartbeat is sent, this alert fires AND the paired uptime-kuma monitor goes DOWN — and reading the push-delivery meaning into that page points you away from the actual fault. That is row 4 of the table above.

A failed Job stays visible after it is gone. The alert reads STORED kube_job_failed samples over a 1h lookback rather than live object state, so deleting a failed Job clears the alert only after a full bucket has passed — re-reading the tile immediately reproduces the red and looks like a failed cleanup. The two eviction mechanisms also do not cover for each other: ttlSecondsAfterFinished (300s here) binds only Jobs created after it was set, and failedJobsHistoryLimit evicts an old failed Job only when a NEWER failed Job arrives — which the TTL now deletes first. A failed Job predating the TTL is reaped by neither and must be deleted by hand.

Query. The pod prints the stale=…s drift=… line and then the failure reason with wget’s exit status. The column is Timestamp, never TimestampTime:

SELECT Timestamp, Body FROM default.otel_logs WHERE ResourceAttributes['k8s.pod.name'] LIKE 'clickstack-watchdog-%' ORDER BY Timestamp DESC

Recover. Do not go looking for a watchdog pod to exec into — ttlSecondsAfterFinished on the Job spec reaps finished watchdog Jobs, so there is usually none left. Use the throwaway pod in Recover: re-run the push by hand below, and take Pod B specifically: it is the body-capable client, and the body is what turns a bare non-2xx into a named cause — uptime-kuma maps every internal exception to 404 and names the reason only in msg, which the watchdog’s own BusyBox wget discards. Then check uptime-kuma and the ExternalSecret refresh.

Recover: re-run the push by hand

The failure line deliberately carries only wget’s exit status — never the URL or the token, because pod stdout is retained 14 days in a queryable store. Recovering more than that means re-running the push by hand, and which client you re-run it with decides what you get back.

What each client recovers. The watchdog image is clickhouse-server:*-alpine, so its only HTTP client is BusyBox wget. -S buys the status line and the response headers, and nothing beyond them: on a non-2xx status BusyBox wget writes wget: server returned error: HTTP/1.1 404 Not Found to stderr and exits without reading the body. That is measured rather than inferred, and it is precisely why some 404s seen in the retained window are unfalsifiable. It matters because uptime-kuma answers {"ok":false,"msg":"<the real reason>"} and its push route maps every internal exception to HTTP 404: a 404 from that endpoint does not mean “bad token”, and msg is the only field that names the cause. BusyBox wget discards that field at the instant it is produced.

uptime-kuma’s own log is not a substitute: its INFO-level output does not distinguish these failures after the fact; neither does default.otel_logs, nor uptime-kuma’s heartbeat table. msg has to be caught live, by a client that keeps it.

The token is never typed. Both pods below read PUSH_TOKEN from a secretKeyRef on Secret clickstack-watchdog, key token, exactly the way the CronJob does, and reference it only as ${PUSH_TOKEN} from inside the pod. Do not paste the token into a kubectl run argument or into a URL on your own shell: a pasted token survives in your shell history and in the Pod object, where anyone with read on the namespace can recover it.

ndots: "1" is mandatory in both pods, not defensive. musl’s getaddrinfo short-circuits on the cluster-appended fzymgc.house search domain, which answers empty-NOERROR/NODATA for unknown subdomains, so under the cluster default ndots:5 the push host never resolves and you get wget: bad address — a failure that is not the incident you were paged for. c-ares (clickhouse-client) tolerates NODATA, which is why the watchdog’s queries work while its push does not. The full root cause is in the dnsConfig comment in argocd/app-configs/clickstack/watchdog-cronjob.yaml. A bare kubectl run inherits the cluster default and reproduces that failure.

Both pods are throwaway (--rm --restart=Never) and carry no service account token, mirroring the CronJob’s automountServiceAccountToken: false. They exist because ttlSecondsAfterFinished: 300 deletes finished watchdog Jobs — there is usually no watchdog pod left to kubectl exec into, which is what makes these the only parity path.

Pod A — parity. Reproduces the watchdog’s exact client behaviour, including the status line. Use it to confirm what the watchdog itself saw. It does not recover msg.

Pod A’s image must equal the image in argocd/app-configs/clickstack/watchdog-cronjob.yaml. That equality is its claim to parity — with a different client this stops being a reproduction of what the watchdog saw and becomes a different experiment that looks like one. So when the CronJob’s client is bumped, this recipe moves with it in the same change; nothing asserts the equality, so the tag here is part of that change. Pod B below is deliberately a different client and is not held to it.

kubectl run -n clickstack watchdog-probe --rm -it --restart=Never \
  --image=clickhouse/clickhouse-server:26.7-alpine \
  --overrides='{
    "spec": {
      "automountServiceAccountToken": false,
      "dnsConfig": {"options": [{"name": "ndots", "value": "1"}]},
      "containers": [{
        "name": "watchdog-probe",
        "image": "clickhouse/clickhouse-server:26.7-alpine",
        "stdin": true,
        "tty": true,
        "command": ["sh"],
        "env": [{"name": "PUSH_TOKEN", "valueFrom": {"secretKeyRef":
          {"name": "clickstack-watchdog", "key": "token"}}}]
      }]
    }
  }'

Then, from inside that pod:

wget -S -O- "http://uptime-kuma.uptime-kuma.svc.cluster.local:3001/api/push/${PUSH_TOKEN}?status=up&msg=manual&ping=1"

Pod B — body capture. Same namespace, same dnsConfig, same secretKeyRef, but a client that keeps the body of a non-2xx response. This is the path that explains an otherwise bare 404: run it on the next occurrence and capture msg.

kubectl run -n clickstack push-body-probe --rm -it --restart=Never \
  --image=curlimages/curl:8.21.0 \
  --overrides='{
    "spec": {
      "automountServiceAccountToken": false,
      "dnsConfig": {"options": [{"name": "ndots", "value": "1"}]},
      "containers": [{
        "name": "push-body-probe",
        "image": "curlimages/curl:8.21.0",
        "stdin": true,
        "tty": true,
        "command": ["sh"],
        "env": [{"name": "PUSH_TOKEN", "valueFrom": {"secretKeyRef":
          {"name": "clickstack-watchdog", "key": "token"}}}]
      }]
    }
  }'

Then, from inside that pod — -i prints the status line and headers, and curl does not suppress the body on a non-2xx status, so msg arrives with them:

curl -sS -i "http://uptime-kuma.uptime-kuma.svc.cluster.local:3001/api/push/${PUSH_TOKEN}?status=up&msg=manual&ping=1"

Then check, in order: the clickstack-watchdog monitor is present and active in uptime-kuma; the clickstack-watchdog ExternalSecret is Ready/SecretSynced; and the value of ping being sent is a non-negative integer.

TLS

Traefik terminates every TLS endpoint: hyperdx.fzymgc.house (Secret hyperdx-tls), ch.fzymgc.house and ch-native.fzymgc.house (Secret ch-tls), and otel-gateway.fzymgc.house. Traefik picks up a renewed Secret by itself, so certificate rotation needs no restart. ClickHouse and the collector serve no TLS.

Backups

Data	Protection
ClickHouse data volumes (longhorn-single)	Two ReplicatedMergeTree replicas, plus Longhorn’s daily-backup recurring job (the volumes are in the default group). These are crash-consistent volume backups. Velero skips longhorn-single volumes. No native BACKUP ... TO S3 job exists.
MongoDB volumes (longhorn)	The default Velero schedule (the clickstack namespace is not excluded) and Longhorn’s daily-backup. No fsync hooks are configured, so both are crash-consistent.
Keeper volumes (longhorn)	The same as MongoDB.

MongoDB holds the HyperDX state that ClickHouse does not: users, teams, sources, dashboards, saved searches, and alerts. Check a ClickHouse volume’s newest Longhorn backup:

kubectl get backups.longhorn.io -n longhorn-system -l backup-volume=<pv-name> \
  -o jsonpath='{range .items[*]}{.status.backupCreatedAt} {.status.state}{"\n"}{end}' | sort | tail -1

Useful Queries

HyperDX UI search uses Lucene syntax. Direct ClickHouse queries:

kubectl exec -it -n clickstack \
  $(kubectl get pod -n clickstack -l clickhouse.com/role=clickhouse-server -o name | head -1) \
  -- clickhouse-client

-- Log count last hour
SELECT count(*) FROM otel_logs WHERE Timestamp > now() - INTERVAL 1 HOUR;

-- Metrics count last hour
SELECT count(*) FROM otel_metrics_gauge WHERE TimeUnix > now() - INTERVAL 1 HOUR;

-- Trace span count last hour, by service (0 rows total => no producer is
-- emitting OTLP traces; the pipeline itself is verified working)
SELECT ServiceName, count(*) AS spans
FROM otel_traces WHERE Timestamp > now() - INTERVAL 1 HOUR
GROUP BY ServiceName ORDER BY spans DESC;

-- Top namespaces by log volume
SELECT ResourceAttributes['k8s.namespace.name'] AS namespace,
       count(*) AS cnt
FROM otel_logs
WHERE Timestamp > now() - INTERVAL 1 HOUR
GROUP BY namespace
ORDER BY cnt DESC
LIMIT 20;

-- MCP calls by server (LiteLLM relay). LiteLLM relays MCP over one route per
-- registered server and emits ONE server span per call, named for the TEMPLATED
-- route, so counting those spans counts MCP calls. The group-by extracts only
-- the server segment from http.target — never group on the raw target, which is
-- the concrete request path and can carry key material.
SELECT extract(SpanAttributes['http.target'], '^/([^/]+)/mcp') AS mcp_server,
       count(*) AS calls
FROM otel_traces
WHERE ServiceName = 'litellm'
  AND SpanName = 'POST /{mcp_server_name}/mcp'
  AND Timestamp > now() - INTERVAL 24 HOUR
GROUP BY mcp_server
ORDER BY calls DESC;

The same query is the LiteLLM — MCP calls by server (relay spans) tile on the LiteLLM — LLM & key usage dashboard. There is deliberately no by-method equivalent: LiteLLM relays the JSON-RPC payload rather than parsing it, so no MCP method name reaches a span attribute — see agentgateway is decommissioned → Consequences for the measurement and the reopen condition.

nas-support (.202) collector — verification queries

Series-arrival runbook for the native otelcol-nas-support collector on the nas-support LXC (192.168.20.202, --tags nas-support-otel-collector). These are the repeatable proofs for BKP-07 (journald logs + host metrics), BKP-08 (backup-freshness file.mtime), and BKP-09 (backup duration + per-source failure gauges). All rows are keyed on ResourceAttributes['host.name']='nas-support' (set by the collector’s resource processor).

Open a ClickHouse client against the ClickStack CHI pod:

kubectl exec -it -n clickstack \
  $(kubectl get pod -n clickstack -l clickhouse.com/role=clickhouse-server -o name | head -1) \
  -- clickhouse-client

-- [BKP-07] journald logs arriving from .202 (expect > 0)
SELECT count() AS logs_last_hour
FROM otel_logs
WHERE ResourceAttributes['host.name'] = 'nas-support'
  AND Timestamp > now() - INTERVAL 1 HOUR;

-- [BKP-07] host metrics arriving from .202 (expect > 0)
SELECT count() AS hostmetric_points
FROM otel_metrics_gauge
WHERE ResourceAttributes['host.name'] = 'nas-support';

-- [BKP-08] backup-freshness file.mtime — a recent epoch value, small age.
-- NOTE: file.mtime is a non-monotonic *Sum* (epoch-seconds), NOT a gauge —
-- it lands in otel_metrics_sum, so query that table, not otel_metrics_gauge.
SELECT MetricName,
       Value,
       now() - Value AS age_s
FROM otel_metrics_sum
WHERE MetricName = 'file.mtime'
  AND ResourceAttributes['host.name'] = 'nas-support'
ORDER BY TimeUnix DESC
LIMIT 1;

-- [BKP-09] backup duration + per-source failure gauges from the last run
-- (expect nas.kopia.backup.duration.seconds present; snapshot.failed rows only
-- if a source failed — carrying a 'source' attribute).
SELECT MetricName,
       Value,
       Attributes['source'] AS source
FROM otel_metrics_gauge
WHERE MetricName LIKE 'nas.kopia%'
  AND TimeUnix > now() - INTERVAL 10 MINUTE
ORDER BY TimeUnix DESC;

Smoke checks (on .202, over the nas-automation SSH connection):

# Collector process is up
systemctl is-active otelcol-nas-support        # -> active

# health_check extension answers on the loopback bind
curl -fsS http://127.0.0.1:13133/              # -> HTTP 200

Column names follow the OTel ClickHouse-exporter schema — MetricName, Value, Attributes, ResourceAttributes, TimeUnix for metrics; Body, ServiceName, ResourceAttributes, Timestamp for logs. file.mtime is the one series that is a Sum rather than a gauge — querying otel_metrics_gauge for it returns zero rows and looks like a broken pipeline.

NAS backup freshness — GitOps alerts (BKP-10)

Two ClickStack tile alerts watch NAS→B2 Kopia backup freshness. They live in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml on the NAS backup — freshness dashboard and are delivered purely via GitOps — no kubectl apply. The path is: PR → merge → ArgoCD auto-sync → PostSync mongo:8.0 Job re-runs bootstrap.js → idempotent MongoDB upsert-by-name. Both deliver to the shared cluster-infra Pushover webhook.

The staleness signal (backups stopped while telemetry still flows) is not a ClickStack alert — it stays on-host as the kopia-freshness.timer dead-man’s-switch (see below). This was de-scoped from ClickStack after live verification (17-03): HyperDX 2.29.0 cannot compute a last-success age from file.mtime inside an alert (see “Why staleness is on-host” below).

Defense-in-depth ladder

Three independent signals, each covering the others’ blind spot. The first two are ClickStack tile alerts; the third is the on-host dead-man’s-switch (the staleness primary) retained from Phase 15.

Signal	Threshold / interval	Catches	Path
Snapshot failure (NAS Kopia snapshot FAILED)	above_exclusive 0, 1d	a specific source broke — earliest “a backup failed” signal	ClickStack
Presence (NAS backup telemetry MISSING)	below 1, 1h	telemetry stopped — the .202 collector or box is dark	ClickStack
Staleness (kopia-freshness.timer)	>36h, hourly	backups stopped (needs no telemetry) — the primary staleness signal	.202 on-host

The two ClickStack alert titles (distinct so the operator can tell which fired):

• NAS Kopia snapshot FAILED
• NAS backup telemetry MISSING (collector dark)

The on-host push keeps its own NAS Kopia backup STALE title at priority=1.

Why staleness is on-host, not a ClickStack alert

file.mtime is a non-monotonic Sum (epoch-seconds). Computing “hours since last success” needs (now - max(file.mtime)) / 3600, but HyperDX 2.29.0 offers no way to get that in an alert: a metric-builder tile rate-computes the Sum (HDX-1543, near-zero delta), a builder tile as gauge reads the wrong table (otel_metrics_gauge, empty), and a raw-SQL tile alert is rejected (“multi select or string select on metrics not supported”) because no generic ClickHouse source exists over otel_metrics_sum. So the last-success age threshold stays on the on-host kopia-freshness.timer (which needs no telemetry at all). The BKP-08 verification query above still reads the age by hand from ClickHouse:

SELECT (toUnixTimestamp(now()) - toUInt64(max(Value))) / 3600 AS hours_since_success
FROM otel_metrics_sum
WHERE MetricName = 'file.mtime' AND ResourceAttributes['host.name'] = 'nas-support';

Snapshot-failure emit semantics (important for the runbook)

The producer (ansible/roles/nas-kopia-backup/templates/kopia-b2-backup.sh.j2) emits nas.kopia.snapshot.failed = 1 only when a source’s snapshot fails, tagging each emit with a source attribute. A healthy run emits nothing for this metric — there is no 0 sample. The alert (above_exclusive 0, i.e. strict > 0) therefore fires when a 1 appears in the window; absence in-window means “no known failure”, not a healthy zero.

The alert tile itself is intentionally un-grouped, so it reliably fires on any failed source. To identify which source failed, use the per-source nas-support — backup freshness & health dashboard (groups nas.kopia.snapshot.failed by Attributes['source']) or a ClickHouse query grouping that metric by Attributes['source'] — see the BKP-09 verification block above.

The failure alert uses the 1d interval — the widest enum window HyperDX offers. The gauge is emitted once per failed source at end-of-run and the backup cadence is ~25h+, so contiguous daily windows catch each failure emit once without the sparse point falling between windows. The on-host kopia-freshness.timer (>36h) covers the sustained-outage tail after a failed point ages out of the 24h window.

On-host backstop retained

The ClickStack alerts are purely additive: they do not replace the on-host switch. The kopia-freshness.{service,timer,sh} units and nas_kopia_freshness_max_age_hours = 36 stand on their own. Confirm the backstop is still armed on .202:

systemctl is-enabled kopia-freshness.timer     # -> enabled