ClickStack Runbook

ClickStack (ClickHouse + HyperDX + MongoDB + OTel Collector) provides a parallel logs, metrics, and traces backend for evaluation against the existing Loki + Grafana stack.

Endpoints

Service	Endpoint
HyperDX UI	https://hyperdx.fzymgc.house (Keycloak OIDC)
OTel Collector OTLP gRPC	192.168.20.149:4317 (MetalLB, TLS + bearer token)
OTel Collector OTLP HTTP	192.168.20.149:4318 (MetalLB, TLS + bearer token)
ClickHouse native	clickhouse.clickstack.svc:9000 (cluster-internal)
ClickHouse HTTPS	clickhouse.clickstack.svc:8443 (Grafana datasource)

Secrets

Vault path fzymgc-house/cluster/clickstack:

Key	Purpose
admin_password	ClickHouse admin password
mongo_admin_password	MongoDB admin password
mongo_replicaset_keyfile	MongoDB internal replica-set auth
hyperdx_bootstrap_secret	HyperDX session signing key
hyperdx_oidc_client_secret	Written by tf/keycloak/clickstack.tf from the Keycloak client secret
otel_ingest_api_key	OTLP ingest bearer token (HyperDX team key) — the key the otlp/hyperdx receiver actually validates; used by the otel-scraper collectors (INGEST_TOKEN) and any trace producer. See Traces Ingestion.
otel_ingest_token	Legacy/vestigial — does not authenticate OTLP (returns 401). No longer mapped into clickstack-secret; the only remaining reference is the vector/otel-ingest-token ExternalSecret, where it survives solely to keep that ExternalSecret valid (ESO rejects a data-less ExternalSecret). Retire fully once Vector’s CLICKHOUSE_PASSWORD is Vault-sourced. See the token-gotcha note below.

Bootstrap (one-time, before PR1 merge)

Run from a workstation with vault token:

vault kv put fzymgc-house/cluster/clickstack \
  admin_password="$(openssl rand -base64 32)" \
  mongo_admin_password="$(openssl rand -base64 32)" \
  mongo_replicaset_keyfile="$(openssl rand -base64 756)" \
  hyperdx_bootstrap_secret="$(openssl rand -base64 32)" \
  hyperdx_oidc_client_secret="placeholder" \
  otel_ingest_token="$(openssl rand -base64 48)"

The hyperdx_oidc_client_secret placeholder is overwritten by tf/keycloak/clickstack.tf on first apply.

otel_ingest_token does not authenticate OTLP; it is seeded only because the vector/otel-ingest-token ExternalSecret still references it (ESO requires a non-empty data block). The real OTLP key — otel_ingest_api_key — cannot be pre-seeded: it is the HyperDX team key, generated on first boot. Once HyperDX is up, capture it into Vault:

key=$(kubectl exec -n clickstack deploy/cs-otel-collector -- \
  sh -c "grep -A2 'bearertokenauth/hyperdx' /etc/otel/supervisor-data/effective.yaml \
         | grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | head -1")
[ -n "$key" ] || { echo "ERROR: no HyperDX key found in effective.yaml"; exit 1; }
# Pipe via stdin (key=-) so the token isn't exposed in shell history / ps:
printf '%s' "$key" | vault kv patch fzymgc-house/cluster/clickstack otel_ingest_api_key=-

Verify: vault kv get fzymgc-house/cluster/clickstack | rg created_time

OTel Ingest Bearer-Token Rotation

The token that authenticates OTLP ingestion is the HyperDX team API key, mirrored into Vault as otel_ingest_api_key. The collector validates the HyperDX-issued key (delivered via OpAMP), so HyperDX is the source of truth and Vault must be updated to match — do not rotate otel_ingest_token (it does not authenticate OTLP; see the token-gotcha note below).

1. In HyperDX UI → Team Settings → API Keys, regenerate the team ingestion key. The collector picks up the new key automatically over OpAMP.

2. Update Vault to match so producers stay in sync:

   vault kv patch fzymgc-house/cluster/clickstack \
     otel_ingest_api_key="<new-hyperdx-team-key>"

3. ESO refreshes the otel-scraper INGEST_TOKEN within 15 minutes; force it and restart the scraper collectors if you need it sooner:

   kubectl annotate externalsecret otel-ingest-token -n otel-scraper \
     force-sync=$(date +%s) --overwrite
   kubectl rollout restart deployment/otel-scrape-opentelemetry-collector -n otel-scraper
   kubectl rollout restart daemonset/otel-node-opentelemetry-collector-agent -n otel-scraper

4. Update any external producers with the new key (e.g. the Firewalla edge collector — ansible/roles/otel-collector, property otel_ingest_api_key).

Collector Config Is OpAMP-Managed (not the static relay.yaml)

cs-otel-collector runs the clickstack-otel-collector image, which is an OpAMP supervisor, not a bare collector. On startup the supervisor connects to the HyperDX OpAMP server (OPAMP_SERVER_URL=http://cs-clickstack-app:4320, from clickstack-config) and is handed an effective config that replaces the bootstrap file. The two configs differ completely:

Source	Where	Exporters
Bootstrap (chart default)	ConfigMap cs-otel-collector key relay, mounted --config=/conf/relay.yaml	debug only — all pipelines
Effective (live)	/etc/otel/supervisor-data/effective.yaml in the pod	clickhouse — logs, metrics, and traces

Consequence: you cannot change ingestion by editing the Helm otelCollector values or the bootstrap relay.yaml — OpAMP overwrites it at runtime. The debug-only relay.yaml is a red herring; inspect the effective config instead:

kubectl exec -n clickstack deploy/cs-otel-collector -- \
  cat /etc/otel/supervisor-data/effective.yaml

Traces Ingestion (why otel_traces can be empty)

The effective config’s traces pipeline is fully wired: otlp/hyperdx receiver (:4317 gRPC / :4318 HTTP, bearertokenauth/hyperdx) → clickhouse exporter → default.otel_traces. The pipeline is not the bottleneck — an empty otel_traces means no application is sending OTLP traces, not that ingestion is broken. (Logs come from Vector writing directly to ClickHouse, bypassing the collector entirely; metrics come from the otel-scraper collectors pushing OTLP + the collector’s own prometheus self-scrape — neither path carries traces.)

Producer contract — to land traces, an app MUST send OTLP with the HyperDX ingestion API key as the authorization header:

From	Endpoint	Auth header
In-cluster	cs-otel-collector.clickstack.svc.cluster.local:4317 (gRPC) / :4318 (HTTP POST /v1/traces)	authorization: <hyperdx-api-key>
External	192.168.20.149:4317 / :4318 (MetalLB, TLS)	authorization: <hyperdx-api-key>

Without the header the receiver returns 401 missing or empty authorization header; all spans are dropped.

Configured producers:

• agentgateway — traces via native OTLP/gRPC push (AgentgatewayParameters rawConfig.config.tracing); metrics via monitoring-otel-scraper scrape of :15000/metrics (no OTLP metrics exporter upstream). ServiceName='agentgateway'.

Token gotcha — two similarly-named Vault keys, only one authenticates OTLP. The otlp/hyperdx receiver’s bearertokenauth validates HyperDX’s internally-generated team ingestion API key (a UUID, created on first boot, stored in Mongo and pushed to the collector via OpAMP). Vault path fzymgc-house/cluster/clickstack holds it under key otel_ingest_api_key — this is the one the otel-scraper collectors use (mapped to their INGEST_TOKEN), and it is what a trace producer must send. The sibling key otel_ingest_token (a base64 openssl rand string, now referenced only by the vector/otel-ingest-token ExternalSecret’s OTEL_INGEST_TOKEN) does not authenticate OTLP — posting with it returns 401. It is effectively vestigial: Vector writes straight to ClickHouse (no OTLP), and the collector gets its real key via OpAMP. To rotate the real key, follow the OTel Ingest Bearer-Token Rotation runbook above (it operates on otel_ingest_api_key, not this key). To read the live key, use HyperDX UI → Team Settings → API Keys or bearertokenauth/hyperdx.tokens[0] in /etc/otel/supervisor-data/effective.yaml.

Smoke test (confirms the pipeline end-to-end without instrumenting an app):

# Extract the live HyperDX team key (full UUID pattern, not a loose hex match).
TOKEN=$(kubectl exec -n clickstack deploy/cs-otel-collector -- \
  sh -c "grep -A2 'bearertokenauth/hyperdx' /etc/otel/supervisor-data/effective.yaml \
         | grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | head -1")
# Fail loudly if extraction came up empty — otherwise the POST below sends an
# empty authorization header and the 401 looks like a broken pipeline.
[ -n "$TOKEN" ] || { echo "ERROR: no HyperDX key found in effective.yaml (layout changed?)"; exit 1; }

kubectl port-forward -n clickstack svc/cs-otel-collector 14318:4318 &
NOW=$(date +%s)000000000
# Pass the token via a 0600 header file (curl -H @file) so it does not appear in
# the process args / `ps` output on shared or audited hosts.
HDR=$(mktemp); chmod 600 "$HDR"; printf 'authorization: %s\n' "$TOKEN" > "$HDR"
curl -s -w '%{http_code}\n' -X POST http://127.0.0.1:14318/v1/traces \
  -H 'Content-Type: application/json' -H @"$HDR" \
  --data "{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"trace-probe\"}}]},\"scopeSpans\":[{\"spans\":[{\"traceId\":\"5b8efff798038103d269b633813fc60c\",\"spanId\":\"eee19b7ec3c1b174\",\"name\":\"probe\",\"kind\":1,\"startTimeUnixNano\":\"$NOW\",\"endTimeUnixNano\":\"$NOW\"}]}]}]}"
rm -f "$HDR"
# Expect HTTP 200 {"partialSuccess":{}}, then:
#   SELECT count() FROM default.otel_traces WHERE ServiceName='trace-probe';  -- > 0

Mongo-Down Failure Mode

HyperDX UI requires MongoDB for session state. Mongo unavailable causes HyperDX to return 5xx.

HA path (3 replicas + PDB minAvailable 2): Tolerates 1 replica loss. Investigate:

kubectl get pods -n clickstack -l app=mongodb
kubectl logs -n clickstack -l app=mongodb --tail=50

Single-replica path: Mongo restart means UI is down. Restore from latest Velero snapshot or wait for pod to recover.

Cert Rotation Pod Restart

ClickHouse reads its TLS cert at startup only. After cert-manager rotates the cert, restart ClickHouse:

kubectl rollout restart statefulset/chi-cs-clickhouse-0-0 -n clickstack

OTel Collector similarly:

kubectl rollout restart deployment/cs-otel-collector -n clickstack

Velero Backups

The clickstack namespace is included in the default Velero schedule. No per-resource backup hooks are required for single-replica ClickHouse (MergeTree is durable on disk; PVC snapshot is consistent).

For MongoDB, add pre/post fsync hooks to the MongoDB pods:

pre.hook.backup.velero.io/command: '["mongosh","--eval","db.fsyncLock()"]'
post.hook.backup.velero.io/command: '["mongosh","--eval","db.fsyncUnlock()"]'

For 3-replica Mongo, hooks run on primary only. Identify primary:

kubectl exec -n clickstack cs-mongodb-0 -- mongosh --eval 'rs.status()' \
  | rg 'PRIMARY'

Useful Queries

HyperDX UI search uses Lucene syntax. Direct ClickHouse queries:

kubectl exec -it -n clickstack \
  $(kubectl get pod -n clickstack -l clickhouse.altinity.com/chi=cs -o name | head -1) \
  -- clickhouse-client

-- Log count last hour
SELECT count(*) FROM otel_logs WHERE Timestamp > now() - INTERVAL 1 HOUR;

-- Metrics count last hour
SELECT count(*) FROM otel_metrics_gauge WHERE TimeUnix > now() - INTERVAL 1 HOUR;

-- Trace span count last hour, by service (0 rows total => no producer is
-- emitting OTLP traces; the pipeline itself is verified working)
SELECT ServiceName, count(*) AS spans
FROM otel_traces WHERE Timestamp > now() - INTERVAL 1 HOUR
GROUP BY ServiceName ORDER BY spans DESC;

-- Top namespaces by log volume
SELECT ResourceAttributes['k8s.namespace.name'] AS namespace,
       count(*) AS cnt
FROM otel_logs
WHERE Timestamp > now() - INTERVAL 1 HOUR
GROUP BY namespace
ORDER BY cnt DESC
LIMIT 20;

nas-support (.202) collector — verification queries

Series-arrival runbook for the native otelcol-nas-support collector on the nas-support LXC (192.168.20.202, --tags nas-support-otel-collector). These are the repeatable proofs for BKP-07 (journald logs + host metrics), BKP-08 (backup-freshness file.mtime), and BKP-09 (backup duration + per-source failure gauges). All rows are keyed on ResourceAttributes['host.name']='nas-support' (set by the collector’s resource processor).

Open a ClickHouse client against the ClickStack CHI pod:

kubectl exec -it -n clickstack \
  $(kubectl get pod -n clickstack -l clickhouse.altinity.com/chi=cs -o name | head -1) \
  -- clickhouse-client

-- [BKP-07] journald logs arriving from .202 (expect > 0)
SELECT count() AS logs_last_hour
FROM otel_logs
WHERE ResourceAttributes['host.name'] = 'nas-support'
  AND Timestamp > now() - INTERVAL 1 HOUR;

-- [BKP-07] host metrics arriving from .202 (expect > 0)
SELECT count() AS hostmetric_points
FROM otel_metrics_gauge
WHERE ResourceAttributes['host.name'] = 'nas-support';

-- [BKP-08] backup-freshness file.mtime — a recent epoch value, small age.
-- NOTE: file.mtime is a non-monotonic *Sum* (epoch-seconds), NOT a gauge —
-- it lands in otel_metrics_sum, so query that table, not otel_metrics_gauge.
SELECT MetricName,
       Value,
       now() - Value AS age_s
FROM otel_metrics_sum
WHERE MetricName = 'file.mtime'
  AND ResourceAttributes['host.name'] = 'nas-support'
ORDER BY TimeUnix DESC
LIMIT 1;

-- [BKP-09] backup duration + per-source failure gauges from the last run
-- (expect nas.kopia.backup.duration.seconds present; snapshot.failed rows only
-- if a source failed — carrying a 'source' attribute).
SELECT MetricName,
       Value,
       Attributes['source'] AS source
FROM otel_metrics_gauge
WHERE MetricName LIKE 'nas.kopia%'
  AND TimeUnix > now() - INTERVAL 10 MINUTE
ORDER BY TimeUnix DESC;

Smoke checks (on .202, over the nas-automation SSH connection):

# Collector process is up
systemctl is-active otelcol-nas-support        # -> active

# health_check extension answers on the loopback bind
curl -fsS http://127.0.0.1:13133/              # -> HTTP 200

Column names follow the OTel ClickHouse-exporter schema — MetricName, Value, Attributes, ResourceAttributes, TimeUnix for metrics; Body, ServiceName, ResourceAttributes, Timestamp for logs. file.mtime is the one series that is a Sum rather than a gauge — querying otel_metrics_gauge for it returns zero rows and looks like a broken pipeline.

NAS backup freshness — GitOps alerts (BKP-10)

Two ClickStack tile alerts watch NAS→B2 Kopia backup freshness. They live in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml on the NAS backup — freshness dashboard and are delivered purely via GitOps — no kubectl apply. The path is: PR → merge → ArgoCD auto-sync → PostSync mongo:8.0 Job re-runs bootstrap.js → idempotent MongoDB upsert-by-name. Both deliver to the shared cluster-infra Pushover webhook.

The staleness signal (backups stopped while telemetry still flows) is not a ClickStack alert — it stays on-host as the kopia-freshness.timer dead-man’s-switch (see below). This was de-scoped from ClickStack after live verification (17-03): HyperDX 2.29.0 cannot compute a last-success age from file.mtime inside an alert (see “Why staleness is on-host” below).

Defense-in-depth ladder

Three independent signals, each covering the others’ blind spot. The first two are ClickStack tile alerts; the third is the on-host dead-man’s-switch (the staleness primary) retained from Phase 15.

Signal	Threshold / interval	Catches	Path
Snapshot failure (NAS Kopia snapshot FAILED)	above_exclusive 0, 1d	a specific source broke — earliest “a backup failed” signal	ClickStack
Presence (NAS backup telemetry MISSING)	below 1, 1h	telemetry stopped — the .202 collector or box is dark	ClickStack
Staleness (kopia-freshness.timer)	>36h, hourly	backups stopped (needs no telemetry) — the primary staleness signal	.202 on-host

The two ClickStack alert titles (distinct so the operator can tell which fired):

• NAS Kopia snapshot FAILED
• NAS backup telemetry MISSING (collector dark)

The on-host push keeps its own NAS Kopia backup STALE title at priority=1.

Why staleness is on-host, not a ClickStack alert

file.mtime is a non-monotonic Sum (epoch-seconds). Computing “hours since last success” needs (now - max(file.mtime)) / 3600, but HyperDX 2.29.0 offers no way to get that in an alert: a metric-builder tile rate-computes the Sum (HDX-1543, near-zero delta), a builder tile as gauge reads the wrong table (otel_metrics_gauge, empty), and a raw-SQL tile alert is rejected (“multi select or string select on metrics not supported”) because no generic ClickHouse source exists over otel_metrics_sum. So the last-success age threshold stays on the on-host kopia-freshness.timer (which needs no telemetry at all). The BKP-08 verification query above still reads the age by hand from ClickHouse:

SELECT (toUnixTimestamp(now()) - toUInt64(max(Value))) / 3600 AS hours_since_success
FROM otel_metrics_sum
WHERE MetricName = 'file.mtime' AND ResourceAttributes['host.name'] = 'nas-support';

Snapshot-failure emit semantics (important for the runbook)

The producer (ansible/roles/nas-kopia-backup/templates/kopia-b2-backup.sh.j2) emits nas.kopia.snapshot.failed = 1 only when a source’s snapshot fails, tagging each emit with a source attribute. A healthy run emits nothing for this metric — there is no 0 sample. The alert (above_exclusive 0, i.e. strict > 0) therefore fires when a 1 appears in the window; absence in-window means “no known failure”, not a healthy zero.

The alert tile itself is intentionally un-grouped, so it reliably fires on any failed source. To identify which source failed, use the per-source nas-support — backup freshness & health dashboard (groups nas.kopia.snapshot.failed by Attributes['source']) or a ClickHouse query grouping that metric by Attributes['source'] — see the BKP-09 verification block above.

The failure alert uses the 1d interval — the widest enum window HyperDX offers. The gauge is emitted once per failed source at end-of-run and the backup cadence is ~25h+, so contiguous daily windows catch each failure emit once without the sparse point falling between windows. The on-host kopia-freshness.timer (>36h) covers the sustained-outage tail after a failed point ages out of the 24h window.

On-host backstop retained (SC#3)

This phase does not touch the on-host switch (D-A2). The kopia-freshness.{service,timer,sh} units and nas_kopia_freshness_max_age_hours = 36 are unchanged — the ClickStack alerts are purely additive. Confirm the backstop is still armed on .202:

systemctl is-enabled kopia-freshness.timer     # -> enabled