ClickStack Runbook
ClickStack (ClickHouse, HyperDX, MongoDB and an OTel Collector) is the cluster’s logs, metrics
and traces backend. This page states what exists, where each part is defined, the query forms,
and the procedures that touch it. The chart values are in argocd/app-configs/clickstack-chart/.
The certificates, the ExternalSecret, the IngressRoutes, the network policy, the collector’s custom
configuration, the LoadBalancer Service, the schema bootstrap and the watchdog are in
argocd/app-configs/clickstack/. The dashboards and alerts bootstrap is in
argocd/app-configs/clickstack-alerts/. The MCP servers over ClickHouse are in
argocd/app-configs/clickhouse-mcp/.
Endpoints
Section titled “Endpoints”| Service | Endpoint |
|---|---|
| HyperDX UI | https://hyperdx.fzymgc.house (Keycloak OIDC, client clickstack-hyperdx from tf/keycloak/clickstack.tf) |
| OTLP, in-cluster | cs-otel-collector.clickstack.svc:4317 (gRPC) / :4318 (HTTP), bearer token |
| OTLP, external | otel-gateway.fzymgc.house:443 (Traefik, mTLS client certificate, gRPC) → cs-otel-collector:4317, bearer token. Defined in argocd/app-configs/monitoring-otel-scraper/otlp-gateway.yaml with clientAuthType: RequireAndVerifyClientCert |
| ClickHouse HTTP | https://ch.fzymgc.house (Traefik) → cs-clickstack-clickhouse-clickhouse-headless:8123 |
| ClickHouse native | ch-native.fzymgc.house:443 (Traefik TCP, TLS SNI) → cs-clickstack-clickhouse-clickhouse-headless:9000 |
| ClickHouse, in-cluster | cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc, :8123 HTTP / :9000 native, no TLS |
The otel-collector-vip LoadBalancer (192.168.250.149 and fddb:f665:73f7:250::149, ports
4317 and 4318) also exposes the collector on the routed VIP prefix. It is plaintext with
bearer auth. The receivers of the collector have no TLS, and nothing
mounts the otel-collector-tls Certificate. Send external OTLP through
otel-gateway.fzymgc.house instead.
The tables
Section titled “The tables”The schema comes from argocd/app-configs/clickstack/bootstrap-schema-configmap.yaml, which
scripts/clickstack/generate-bootstrap-ddl.sh generates from the HyperDX collector’s seed schema.
Do not hand-edit it. The bootstrap Job applies it. The DNS query rollup is the one exception, as
section DNS query rollups states.
| Table | Time column | Retention |
|---|---|---|
default.otel_logs |
Timestamp |
14 days |
default.otel_metrics_gauge, otel_metrics_sum, otel_metrics_histogram, otel_metrics_exponential_histogram, otel_metrics_summary |
TimeUnix |
30 days |
default.otel_traces |
Timestamp |
14 days |
default.hyperdx_sessions |
TimestampTime |
14 days |
default.otel_logs_kv_rollup_15m, otel_traces_kv_rollup_15m |
Timestamp |
14 days |
default.dns_query_daily |
day |
No expiry |
default.otel_logs has a Timestamp column and no TimestampTime. Use Timestamp in every
query and in every HyperDX timestampValueExpression on a logs source. Column names follow the
OTel ClickHouse exporter schema: MetricName, Value, Attributes, ResourceAttributes and
TimeUnix for metrics, and Body, ServiceName, ResourceAttributes and Timestamp for logs.
The ClickHouse system log tables carry a 7-day TTL set in the chart values.
DNS query rollups
Section titled “DNS query rollups”default.dns_query_daily holds the number of DNS queries of each UTC day, with no expiry. Two
materialized views fill it from default.otel_logs as rows arrive. dns_query_daily_mv counts the
rows of Blocky, the client tier until 2026-10-14, and it counts no row after the cut-over.
dns_query_daily_resolver_mv counts the rows of the recursor, with the attribute names of The
resolver query rows below. argocd/app-configs/clickstack/dns-query-schema.yaml holds the table
and both views, and its PostSync Job creates them as schema_admin. The OpenSpec change
dns-query-log-retention states the requirements.
| Column | dns_query_daily_mv |
dns_query_daily_resolver_mv |
|---|---|---|
day |
toDate(Timestamp), a UTC day |
toDate(Timestamp), a UTC day |
service |
ServiceName: blocky or blocky-public |
ServiceName: dns-resolver or dns-resolver-public |
client |
LogAttributes['client_ip'] |
LogAttributes['client.address'] |
name |
LogAttributes['question_name'] |
LogAttributes['dns.question.name'] |
type |
LogAttributes['question_type'] |
LogAttributes['dns.question.type'] |
result |
LogAttributes['response_type'], such as RESOLVED or BLOCKED |
LogAttributes['dns.policy.action']: nxdomain, nodata, passthru, or empty when no rule decided |
code |
LogAttributes['response_code'] |
LogAttributes['dns.response.code'] |
queries |
The number of query rows with these values | The number of query rows with these values |
Each view counts query rows only. A query row is a row with a question name. The daemon lines of a
resolver carry the same ServiceName and no question name, so no view counts them.
The table is a ReplicatedSummingMergeTree. ClickHouse adds up the rows with the same key in the
background, at no fixed time. Read the table with sum(queries) and GROUP BY, because one row is
not a count:
-- The names that a rule blocked most often in the last 7 days, before and after the cut-over.SELECT name, sum(queries) AS nFROM default.dns_query_dailyWHERE day >= today() - 7 AND result IN ('BLOCKED', 'nxdomain', 'nodata')GROUP BY name ORDER BY n DESC LIMIT 20;The MCP users read no row of the table or of either view, because of the row filter "0" on each
in argocd/app-configs/clickstack-chart/values.yaml. A SELECT on a view reads the table, and
ClickHouse looks up a row policy by the name that the query names, so each view needs its own
filter. The HyperDX user app reads it, as it reads the
DNS rows of otel_logs.
The boundaries and the fills
Section titled “The boundaries and the fills”Each view counts the rows with a Timestamp at or after the boundary in its WHERE clause. A
one-time fill counts the rows before the boundary, as schema_admin, at least 6 hours after the
boundary.
dns_query_daily_mv has the boundary 2026-10-13 00:00:00 UTC. Its fill ran once, from the oldest
day that otel_logs held in full:
INSERT INTO default.dns_query_dailySELECT toDate(Timestamp) AS day, ServiceName AS service, LogAttributes['client_ip'] AS client, LogAttributes['question_name'] AS name, LogAttributes['question_type'] AS type, LogAttributes['response_type'] AS result, LogAttributes['response_code'] AS code, count() AS queriesFROM default.otel_logsWHERE ServiceName IN ('blocky', 'blocky-public') AND LogAttributes['question_name'] != '' AND Timestamp < toDateTime64('2026-10-13 00:00:00', 9) AND toDate(Timestamp) >= today() - 13GROUP BY day, service, client, name, type, result, code;dns_query_daily_resolver_mv has the boundary 2026-10-23 00:00:00 UTC. Its fill counts every row
of the recursor before the boundary. The first rows are from 2026-10-14, and otel_logs drops them
on 2026-10-28, so run the fill between 2026-10-23 06:00 and 2026-10-28 00:00 UTC:
INSERT INTO default.dns_query_dailySELECT toDate(Timestamp) AS day, ServiceName AS service, LogAttributes['client.address'] AS client, LogAttributes['dns.question.name'] AS name, LogAttributes['dns.question.type'] AS type, LogAttributes['dns.policy.action'] AS result, LogAttributes['dns.response.code'] AS code, count() AS queriesFROM default.otel_logsWHERE ServiceName IN ('dns-resolver', 'dns-resolver-public') AND LogAttributes['dns.question.name'] != '' AND Timestamp < toDateTime64('2026-10-23 00:00:00', 9)GROUP BY day, service, client, name, type, result, code;Run each fill once. A second run counts each day before the boundary twice. The boundary of a view
must be later than the first sync that creates the view, or the rows between the boundary and the
creation count nowhere. If the sync comes later than the boundary, move the boundary to a later day
on the branch before the merge, and use the same day in the fill. After the first sync, do not
change a boundary in dns-query-schema.yaml. The Job runs CREATE ... IF NOT EXISTS, so a new
boundary never reaches the live view.
Compare the rollup with the raw rows
Section titled “Compare the rollup with the raw rows”This query returns one row for each day of the last 13 days on which the rollup and the query rows
differ. An empty result is the pass. A day that has no rollup row shows rolled_up 0, also in a
session that sets join_use_nulls = 1. Until the fill of dns_query_daily_resolver_mv runs, each
day from 2026-10-14 to its boundary differs:
SELECT l.day, l.c AS queried, ifNull(r.q, 0) AS rolled_upFROM( SELECT toDate(Timestamp) AS day, count() AS c FROM default.otel_logs WHERE ((ServiceName IN ('blocky', 'blocky-public') AND LogAttributes['question_name'] != '') OR (ServiceName IN ('dns-resolver', 'dns-resolver-public') AND LogAttributes['dns.question.name'] != '')) AND toDate(Timestamp) BETWEEN today() - 13 AND yesterday() GROUP BY day) AS lLEFT JOIN( SELECT day, sum(queries) AS q FROM default.dns_query_daily WHERE day BETWEEN today() - 13 AND yesterday() GROUP BY day) AS r USING dayWHERE ifNull(r.q, 0) != l.cORDER BY l.day;Each replica reads its own copy of both tables, so run the query on each replica. schema_admin
has no READ ON REMOTE, so clusterAllReplicas fails for it. Save the query as compare.sql and
run it from a workstation with kubectl access to the cluster:
for r in 0 1; do echo "replica $r" kubectl -n clickstack exec -i "cs-clickstack-clickhouse-clickhouse-0-$r-0" -c clickhouse-server -- \ sh -c 'clickhouse-client --user schema_admin --password "$SCHEMA_ADMIN_PASSWORD" --multiquery' < compare.sqldoneStop the count
Section titled “Stop the count”Each view runs inside each insert into otel_logs. If a view fails, every insert fails, and all
log ingest stops. If the collector logs insert errors that name a view, run this as schema_admin
first, with the view that the errors name:
DROP VIEW default.dns_query_daily_resolver_mv SYNC;The table and its rows stay. Each view runs as its definer, schema_admin. If schema_admin loses
INSERT on default.dns_query_daily, every insert fails the same way. DROP TABLE default.dns_query_daily SYNC deletes the history for good, so run it only on Sean’s decision.
DNS query archive
Section titled “DNS query archive”The CronJob dns-query-archive (argocd/app-configs/clickstack/dns-query-archive.yaml) writes the
DNS rows of each UTC day to Backblaze B2 as one Parquet file. A file holds every stored column of
otel_logs and the raw Body. The job runs at 06:00 UTC (timeZone: Etc/UTC) as the user
dns_query_archiver. The OpenSpec change dns-query-log-retention states the requirements.
| Item | Value |
|---|---|
| File | fzymgc-k8s-backups/dns-query-log/<yyyy>/<mm>/<dd>.parquet, compressed with zstd |
| Rows | Every row of blocky, blocky-public, dns-resolver and dns-resolver-public of the day, also the lines that are not queries |
| Window | Each run checks each day from 13 days back to the previous day. It writes a missing file and rewrites a file that holds fewer rows than otel_logs. It keeps a full file. The TTL of otel_logs can drop part of the day 14 days back, so the window stops at 13 |
| Retention | The bucket’s rule on dns-query-log/ hides each file 365 days after upload and deletes it a day later. The earlier version of a rewritten file goes a day after the rewrite |
| Credentials | The s3 endpoint dns_query_archive of the chart values, from Secret dns-query-archive-b2 (Vault cluster/clickstack/dns-archive-b2). No SQL statement names a key |
| User | dns_query_archiver reads only the DNS rows of otel_logs and writes only below dns-query-log/. The scope of its S3 grant needs access_control_improvements.enable_read_write_grants in the chart values. Without the setting, ClickHouse turns the grant into S3 ON *.* |
| Alert | DNS query archive FAILED, when a Job of the CronJob failed in the last day |
The log has one line for each day:
dns-query-archive <day> logged=<rows in otel_logs> archived=<rows in the file> action=keep|write|none.
A day that failed has a line with FAILED and the reason. A run fails when a count or a write
fails, or when a written file holds fewer rows than otel_logs. The run goes on with the other
days, and then exits 1.
A run repairs every day in the window, so the next night writes the day that a failed night missed.
Keep a failed Job until the alert has fired. The alert reads kube_job_failed of the Job at the end
of each hour, so an earlier delete clears the signal. ttlSecondsAfterFinished removes the Job after
a day.
To run the job now, give the Job a new name:
kubectl create job -n clickstack --from=cronjob/dns-query-archive dns-query-archive-manualkubectl -n clickstack logs -f job/dns-query-archive-manualQuery the archive
Section titled “Query the archive”Run these as schema_admin. ClickHouse takes the key from the endpoint section, so no query names
a key:
-- The rows of one day.SELECT count() FROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/2026/10/12.parquet', 'Parquet');
-- The names that one client looked up in a month.SELECT LogAttributes['question_name'] AS name, count() AS nFROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/2026/10/*.parquet', 'Parquet')WHERE LogAttributes['client_ip'] = '192.168.20.220' AND LogAttributes['question_name'] != ''GROUP BY name ORDER BY n DESC LIMIT 50;
-- The answers of the blocklists in a month, by day.SELECT toDate(Timestamp) AS day, count() AS blockedFROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/2026/10/*.parquet', 'Parquet')WHERE LogAttributes['response_type'] = 'BLOCKED'GROUP BY day ORDER BY day;If ClickHouse is down, clickhouse local on a workstation reads the same files with the key from
fzymgc-house/cluster/clickstack/dns-archive-b2. Use it as the fallback only, because that key can
also write the prefix.
Compare the archive with the raw rows
Section titled “Compare the archive with the raw rows”This query returns one row for each day of the window on which the file and otel_logs differ. An
empty result is the pass. A day that has no file shows archived 0, also in a session that sets
join_use_nulls = 1:
SELECT l.day, l.c AS logged, ifNull(a.c, 0) AS archivedFROM( SELECT toDate(Timestamp) AS day, count() AS c FROM default.otel_logs WHERE ServiceName IN ('blocky', 'blocky-public', 'dns-resolver', 'dns-resolver-public') AND toDate(Timestamp) BETWEEN today() - 13 AND yesterday() GROUP BY day) AS lLEFT JOIN( SELECT toDate(replaceRegexpOne(_path, '^.*/dns-query-log/(\\d{4})/(\\d{2})/(\\d{2})[.]parquet$', '\\1-\\2-\\3')) AS day, count() AS c FROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/**/*.parquet', 'Parquet', 'Timestamp DateTime64(9)') WHERE day BETWEEN today() - 13 AND yesterday() GROUP BY day) AS a USING dayWHERE ifNull(a.c, 0) != l.cORDER BY l.day;The acceptance of the change compares four counts for one day. archived equals logged, and
rolled_up equals queried. queried counts the query rows of both name sets, as the two views
do, see DNS query rollups:
SELECT (SELECT count() FROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/<yyyy>/<mm>/<dd>.parquet', 'Parquet')) AS archived, (SELECT count() FROM default.otel_logs WHERE ServiceName IN ('blocky', 'blocky-public', 'dns-resolver', 'dns-resolver-public') AND toDate(Timestamp) = '<day>') AS logged, (SELECT count() FROM default.otel_logs WHERE ((ServiceName IN ('blocky', 'blocky-public') AND LogAttributes['question_name'] != '') OR (ServiceName IN ('dns-resolver', 'dns-resolver-public') AND LogAttributes['dns.question.name'] != '')) AND toDate(Timestamp) = '<day>') AS queried, (SELECT sum(queries) FROM default.dns_query_daily WHERE day = '<day>') AS rolled_up;Upgrade ClickStack
Section titled “Upgrade ClickStack”Renovate proposes each ClickStack release in one attended pull request, update clickstack observability plane. It moves the HyperDX image tag and the collector image tag in
argocd/app-configs/clickstack-chart/values.yaml, and the chart version in kustomization.yaml.
The bootstrap Job runs only CREATE ... IF NOT EXISTS, so it never changes a table that exists.
Before you merge the pull request, do these steps on its branch:
-
Make sure that the HyperDX tag and the collector tag in
values.yamlshow the same version. -
From the repository root, run
scripts/clickstack/generate-bootstrap-ddl.sh. It reads the seed version from the collector tag. Commit the changedbootstrap-schema-configmap.yaml. -
List the seed files that changed between the two collector versions:
Terminal window gh api 'repos/hyperdxio/hyperdx/compare/@hyperdx/otel-collector@<old>...@hyperdx/otel-collector@<new>' \--jq '.files[] | select(.filename | test("docker/otel-collector/schema/seed/")) | .filename' -
If a seed file of an existing table changed, the live table needs a migration script on the model of
scripts/clickstack/migrate-schema-2.37.0.sh. Write it, rehearse it, and run it as part of the upgrade. -
If the chart version changed, compare the rendered manifests of the old and the new chart (
argocd/CLAUDE.md, Change safety).
What feeds it
Section titled “What feeds it”| Source | Signal | Path |
|---|---|---|
Vector (argocd/app-configs/monitoring-vector/vector.yaml) |
logs from every node | the clickhouse_logs sink writes straight to ClickHouse. No OTLP, no collector |
otel-scrape and otel-node (argocd/app-configs/monitoring-otel-scraper/) |
cluster metrics | OTLP to the collector with the ingest key from the otel-ingest-token ExternalSecret in otel-scraper |
otel-network (argocd/app-configs/monitoring-otel-scraper/network-collector-values.yaml) |
the syslog of the UniFi gateway, switches and APs, ServiceName='unifi' |
UDP from each device to 192.168.250.150, then OTLP to the collector with the same ingest key (The UniFi rows) |
The NAS, nas-support and resolver collectors (ansible/roles/{nas-otel-collector,nas-support-otel-collector,dns-otel-collector}) |
host metrics and journals | OTLP through otel-gateway.fzymgc.house with a client certificate from the ICA1 PKI and the ingest key from Vault |
dnscrypt-proxy on the resolver nodes (ansible/roles/dns-otel-collector) |
its journal, ServiceName='dnscrypt-proxy', with no client address |
the resolver collector, as the resolver query rows |
LiteLLM (argocd/app-configs/litellm-chart/values.yaml) |
traces and GenAI metrics, ServiceName='litellm' |
OTLP HTTP to the collector, OTEL_HEADERS from the litellm-otel-ingest Secret |
engram (argocd/cluster-app/templates/agent-memory.yaml), headroom and hl-assets |
traces and metrics | OTLP with an otlp-headers Secret each |
dns-control, its webhook and its operator commands (argocd/app-configs/dns-control/, argocd/app-configs/external-dns/values.yaml) |
traces, metrics and logs, ServiceName dns-control, dns-control-webhook or dns-control-cli |
OTLP HTTP to the collector, OTEL_EXPORTER_OTLP_HEADERS from the dns-control-otlp-headers and external-dns-otlp-headers Secrets. Vector leaves the two containers out (Telemetry) |
The resolver query rows
Section titled “The resolver query rows”Each resolver node runs two PowerDNS Recursor processes, pdns-recursor@main for the trusted
legs and pdns-recursor@public for Guest and Telework. Each process sends one protobuf message
for each answer to DNS-collector on the node, the unit dns-resolver-reader.service, which writes
one JSON row for each answer to its journal. The resolver collector parses the row and gives it
one set of engine-neutral names, so dns-control reads one shape whatever engine answers. The
OpenSpec change dns-policy-as-rpz defines them. Where an OpenTelemetry semantic convention
exists, the name follows it.
| Name | Holds |
|---|---|
ServiceName |
the process that answered: dns-resolver for the main process or dns-resolver-public for the public process. The name is dnstap.identity of the row, the server_id of the process. Until 2026-10-14 the names were blocky and blocky-public |
LogAttributes['client.address'] |
the client address |
LogAttributes['dns.question.name'], LogAttributes['dns.question.type'] |
the question. The name keeps the trailing dot |
LogAttributes['dns.response.code'], LogAttributes['dns.answer'] |
the answer. The answer holds up to three record values, separated by a comma |
LogAttributes['dns.policy.zone'], LogAttributes['dns.policy.trigger'], LogAttributes['dns.policy.hit'] |
the policy zone that decided the answer, such as hagezi.feed.rpz.internal., the rule that matched and the name that matched it. A row that no rule decided carries none of the policy attributes |
LogAttributes['dns.policy.action'] |
nxdomain for a block, nodata for the rebinding zone of the public process, and passthru for an allow |
The raw names of the reader stay on the row beside the neutral names, such as dns.qname,
dns.rcode and powerdns.applied-policy.
ansible/roles/dns-otel-collector/templates/otel-config.yaml.j2 holds the mapping. No row of the
two health names, dns-health.fzymgc.house and health.dns.invalid, reaches ClickStack: the Lua
script of each process keeps them out of the protobuf, and the processor filter/health of the
collector drops a row that arrives anyway. The daemon lines of the two units
pdns-recursor@main.service and pdns-recursor@public.service carry the same two ServiceNames
as the query rows, and the other lines of the reader carry dns-resolver-reader.
The collector scrapes /metrics of each process, 127.0.0.1:8082 for main and
127.0.0.1:8083 for public, every 60 seconds. The scrape job name is the ServiceName of the
metric, dns-resolver or dns-resolver-public. Two counters pass: pdns_recursor_questions,
every query with the RD bit set, and pdns_recursor_servfail_answers, every SERVFAIL the process
gave. The collector copies them to dns_resolver_responses_total and dns_resolver_errors_total
under the same ServiceName. dns-control, the tile DNS public resolver failed resolutions per answer and the alert DNS public resolver failing (Guest/Telework) of
argocd/app-configs/clickstack-alerts/bootstrap-script.yaml read the copies. The tile and the
alert filter on ServiceName = 'dns-resolver-public'.
The UniFi rows
Section titled “The UniFi rows”The Deployment otel-network in the namespace otel-scraper receives the syslog of the UniFi
gateway, switches and APs on UDP 514 of the VIP 192.168.250.150. The controller sends one
syslog target to every device. The collector runs the contrib distribution, because the
otelcol-k8s image of otel-scrape has no syslog receiver. Three replicas run on three nodes,
and a PodDisruptionBudget lets one of them go at a time, so the VIP stays announced while a
replica restarts. The network policy network-collector admits the Lab address of the gateway,
192.168.200.1, and Core, 192.168.40.0/22, where the switches and APs have their addresses.
Each syslog line becomes one row with ServiceName='unifi'. The receiver parses RFC 3164 and
reads the timestamp as America/New_York time. Each device puts a token between its hostname and
the program: the gateway writes its hostname a second time, and a switch or an AP writes its MAC
address and firmware. A regex in the receiver skips the token. The parts of the line go to these
attributes:
| Attribute | Holds |
|---|---|
LogAttributes['hostname'] |
the name of the device, such as router, ap-basement or switch-library |
LogAttributes['appname'], LogAttributes['proc_id'] |
the program and its process id, such as hostapd and 9561 |
LogAttributes['message'] |
the text after the program |
A line that does not match the regex keeps the attributes of the RFC 3164 parser. A line that
does not parse keeps its raw body and gets no timestamp from the line. Two kinds of line do not
match, so search them in Body:
- A zone policy with logging writes one line for each match, with no program:
router [LAN_WAN-A-10000] DESCR="internal-log-external-dns" IN=br3000 OUT=eth4 ... SRC=... DST=... PROTO=UDP ... DPT=53.appnameis empty. - The Network application sends its events in CEF, a line format for SIEM tools, with no
priority:
router CEF:0|Ubiquiti|UniFi Network|11.0.89|545|Config Created|5|UNIFIcategory=Audit .... The RFC 3164 parser does not take a line without a priority, sohostnameis empty.UNIFIcategorynames the content, such asAudit,Client DevicesorSecurity.
The collector takes no IPFIX. The gateway meters IPFIX flows in the kernel module ipt_NETFLOW,
which sees only the packets that pass through the CPU. While IPFIX export is on, the gateway turns
off its hardware offload, and every forwarded packet goes through the CPU.
To count the lines of the last hour by device and program:
SELECT LogAttributes['hostname'] AS device, LogAttributes['appname'] AS program, count() AS linesFROM default.otel_logsWHERE ServiceName = 'unifi' AND Timestamp > now() - INTERVAL 1 HOURGROUP BY device, programORDER BY lines DESCTo list the DNS connections from Internal to the internet of the last day, by client and
server (Zone firewall):
SELECT extract(Body, 'SRC=(\\S+)') AS client, extract(Body, 'DST=(\\S+)') AS server, extract(Body, 'DPT=(\\d+)') AS port, count() AS lines, max(Timestamp) AS lastFROM default.otel_logsWHERE ServiceName = 'unifi' AND position(Body, 'DESCR="internal-log-external-dns"') > 0 AND Timestamp > now() - INTERVAL 1 DAYGROUP BY client, server, portORDER BY lines DESCMCP servers over ClickHouse
Section titled “MCP servers over ClickHouse”argocd/app-configs/clickhouse-mcp/ runs two mcp-clickhouse instances. clickhouse-mcp-ro
connects as mcp_readonly and clickhouse-mcp-rw as mcp_readwrite. Both passwords come from
the clickstack Vault path. LiteLLM exposes them as the clickhouse_ro and clickhouse_rw MCP
servers. Read the argocd/app-configs/litellm-chart/values.yaml entries for which keys reach the
read-write instance.
A row filter hides every resolver log row from mcp_readonly and mcp_readwrite. The filter is
a databases key under extraUsersConfig in argocd/app-configs/clickstack-chart/values.yaml.
It removes each row of default.otel_logs with a ServiceName of dns-resolver or
dns-resolver-public. The two values mark every line of the two recursor processes on the two DNS
resolver hosts: the query rows of the reader, and also the daemon lines of
pdns-recursor@main.service and pdns-recursor@public.service.
dns_otel_recursor_log_service_names in ansible/roles/dns-otel-collector/defaults/main.yml
sets them. The filter must name every value of the Ansible map, because nothing compares the two.
The filter does not name blocky and blocky-public, the rows of the client tier until
2026-10-14. The MCP users can read those rows until the 14-day TTL of otel_logs removes the last
of them, about 2026-10-30.
An agent on clickhouse_ro or clickhouse_rw therefore reads no resolver journal line. It reads
resolver health from the metrics, for example dns_resolver_errors_total under ServiceName
dns-resolver or dns-resolver-public.
Every other ClickHouse user reads every row. The HyperDX Logs source shows the resolver rows
next to the other services. The HyperDX MCP (the clickstack tools) reads ClickHouse as app, so
an agent with that MCP reads every resolver row, client addresses included. Issue #2378 tracks the
access paths that bypass the filter. The filter covers default.otel_logs only. The metrics carry
no data for each client, and default.otel_logs_kv_rollup_15m holds no attribute values.
dns-control connects to ClickHouse as dns_control_reader. The user has SELECT on
default.otel_logs, default.otel_metrics_sum and default.otel_metrics_gauge only. The row
filter does not apply to it, because the dns-control Query log screen shows the resolver rows. The
hyperdx_reader profile limits each of its queries to 2 GiB of memory. The clickstack-default
network policy admits the dns-control namespace on port 8123.
Secrets
Section titled “Secrets”Vault path fzymgc-house/cluster/clickstack, read by the clickstack ExternalSecret:
| Key | Purpose |
|---|---|
admin_password |
ClickHouse admin password |
mongo_admin_password |
MongoDB admin password |
mongo_replicaset_keyfile |
MongoDB internal replica-set auth |
hyperdx_bootstrap_secret |
HyperDX session-signing key, delivered to the app as EXPRESS_SESSION_SECRET. Rotating it signs every user out once. Reloader restarts the app when the Secret changes |
otel_ingest_api_key |
The OTLP ingest bearer token. It is the HyperDX team key, and it is the only key the otlp/hyperdx receiver validates. Every producer and every external collector reads it |
mcp_readonly_password, mcp_readwrite_password |
The two ClickHouse users the MCP servers connect as |
schema_admin_password |
The user the schema bootstrap Job connects as |
dns_control_reader_password |
The ClickHouse user dns-control connects as |
app_password |
The password of the ClickHouse user app. HyperDX and the watchdog CronJob connect as app. HyperDX keeps its copy in MongoDB, so a rotation also sets the HyperDX connection (step 5 of the rotation) |
otelcollector_password |
The password of the ClickHouse user otelcollector. The ClickStack collector and Vector write rows as otelcollector. The otel-ingest-token ExternalSecret in the vector namespace also reads this key |
dns_query_archiver_password |
The ClickHouse user the DNS query archive connects as |
Two more paths belong to the stack. fzymgc-house/cluster/clickstack/oidc holds
hyperdx_oidc_client_secret, which tf/keycloak/clickstack.tf writes. Do not put it by hand.
fzymgc-house/cluster/clickstack-watchdog holds token, the uptime-kuma push token, which
tf/uptime-kuma/clickstack-watchdog.tf writes.
The three external collector roles and the six in-cluster producers read otel_ingest_api_key
from the same path. The Vault policies clickstack-hyperdx, clickstack-otel-collector,
clickstack-clickhouse, otel-collector-nas, otel-collector-nas-support and
otel-collector-dns grant that path.
The ClickHouse user default has no password and holds GRANT ALL. It accepts connections only
from the loopback addresses of its own pod. clickhouse-client through kubectl exec connects as
default, and a request through Traefik as default gets Authentication failed. If you need an
administrative session from outside the pod, connect as schema_admin.
Seed the Vault path
Section titled “Seed the Vault path”Run from a workstation with a Vault token. vault kv put replaces every key at the path, so run
it only on a path that does not exist yet:
vault kv put secret/fzymgc-house/cluster/clickstack \ admin_password="$(openssl rand -base64 32)" \ mongo_admin_password="$(openssl rand -base64 32)" \ mongo_replicaset_keyfile="$(openssl rand -base64 756)" \ hyperdx_bootstrap_secret="$(openssl rand -base64 32)" \ mcp_readonly_password="$(openssl rand -base64 32)" \ mcp_readwrite_password="$(openssl rand -base64 32)" \ schema_admin_password="$(openssl rand -base64 32)" \ dns_control_reader_password="$(openssl rand -base64 32)" \ app_password="$(openssl rand -base64 32)" \ otelcollector_password="$(openssl rand -base64 32)" \ dns_query_archiver_password="$(openssl rand -base64 32)"If the path exists and has no dns_control_reader_password, seed only that key. vault kv patch
keeps the other keys. Seed it before the change that adds the key to the ExternalSecret merges,
because a ClickHouse pod does not start when a secretKeyRef key is missing:
printf '%s' "$(openssl rand -base64 32)" | vault kv patch secret/fzymgc-house/cluster/clickstack dns_control_reader_password=-If ClickStack already runs and the path has no app_password or otelcollector_password, seed
the value that each user has now. That is hyperdx for app and otelcollectorpass for
otelcollector, the chart defaults. The next sync then changes no password. To change a value
later, follow Rotate a ClickHouse password.
If the path has no dns_query_archiver_password, seed a new value before the change that adds the
key merges. The ClickHouse pods and the CronJob dns-query-archive read it:
printf '%s' "$(openssl rand -base64 32)" | vault kv patch secret/fzymgc-house/cluster/clickstack dns_query_archiver_password=-The ingest key cannot be seeded. HyperDX generates its team key on first boot. When HyperDX is up, capture it into Vault:
key=$(kubectl exec -n clickstack deploy/cs-otel-collector -- \ sh -c "grep -A2 'bearertokenauth/hyperdx' /etc/otel/supervisor-data/effective.yaml \ | grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | head -1")[ -n "$key" ] || { echo "ERROR: no HyperDX key found in effective.yaml"; exit 1; }# Pipe through stdin (key=-) so that the token stays out of the shell history and ps.printf '%s' "$key" | vault kv patch secret/fzymgc-house/cluster/clickstack otel_ingest_api_key=-Make sure that the write landed: vault kv get secret/fzymgc-house/cluster/clickstack | rg created_time.
Rotate the OTLP ingest key
Section titled “Rotate the OTLP ingest key”HyperDX is the source of truth for the ingest key. The collector validates the key HyperDX hands it, so Vault must follow HyperDX.
-
In the HyperDX UI, open Team Settings, then API Keys, and regenerate the team ingestion key. The collector picks up the new key by itself.
-
Update Vault to match:
Terminal window vault kv patch secret/fzymgc-house/cluster/clickstack otel_ingest_api_key="<new-hyperdx-team-key>" -
ESO refreshes the in-cluster consumers within their refresh interval. To force the scraper collectors now:
Terminal window kubectl annotate externalsecret otel-ingest-token -n otel-scraper force-sync=$(date +%s) --overwritekubectl rollout restart deployment/otel-scrape-opentelemetry-collector -n otel-scraperkubectl rollout restart daemonset/otel-node-opentelemetry-collector-agent -n otel-scraperThe
litellm,agent-memory,headroom-agentsandhl-assetsExternalSecrets refresh the same way. -
Converge the three external collector roles, because each one reads the key at deploy time:
nas-otel-collector,nas-support-otel-collectoranddns-otel-collector. -
Make sure that rows still arrive:
SELECT count() FROM default.otel_metrics_gauge WHERE TimeUnix > now() - INTERVAL 10 MINUTE;
Rotate a ClickHouse password
Section titled “Rotate a ClickHouse password”This procedure changes app_password or otelcollector_password. Run it from a LAN host with a
Vault token and the fzymgc-house kube context. Set the Vault key and the ClickHouse user first:
key=otelcollector_password user=otelcollector field=CLICKHOUSE_PASSWORD# or: key=app_password user=app field=CLICKHOUSE_APP_PASSWORDLogins fail from step 2 until the end of step 3, because the consumers and ClickHouse hold different values. The collector holds rows in a 2Gi disk queue, and Vector holds rows in a 5Gi disk buffer. Do all six steps in one session.
-
Write a new value to Vault:
Terminal window printf '%s' "$(openssl rand -base64 32)" | vault kv patch secret/fzymgc-house/cluster/clickstack "$key=-" -
Sync the two ExternalSecrets now, and wait until
clickstack-secretholds the new value. Forotelcollector, also wait forotel-ingest-token. The annotate command returns before ESO reads Vault, so aSecretSyncedstatus alone does not prove the new value. The loops compare SHA-256 hashes, so no value reaches the terminal. If a loop runs for more than a minute, read the status of that ExternalSecret:Terminal window kubectl --context fzymgc-house -n clickstack annotate externalsecret clickstack-secret force-sync="$(date +%s)" --overwritekubectl --context fzymgc-house -n vector annotate externalsecret otel-ingest-token force-sync="$(date +%s)" --overwritewant=$(printf '%s' "$(vault kv get -field="$key" secret/fzymgc-house/cluster/clickstack)" | shasum -a 256)have() {printf '%s' "$(kubectl --context fzymgc-house -n "$1" get secret "$2" -o jsonpath="{.data.$3}" | base64 -d)" | shasum -a 256}until [ "$(have clickstack clickstack-secret "$field")" = "$want" ]; do sleep 5; done# otelcollector only:until [ "$(have vector otel-ingest-token CLICKHOUSE_PASSWORD)" = "$want" ]; do sleep 5; done -
Delete the ClickHouse pods one at a time,
0-1-0first. The operator recreates each pod, and the new process reads the new value fromclickstack-secret:Terminal window for pod in cs-clickstack-clickhouse-clickhouse-0-1-0 cs-clickstack-clickhouse-clickhouse-0-0-0; dokubectl --context fzymgc-house -n clickstack delete pod "$pod"kubectl --context fzymgc-house -n clickstack wait --for=create "pod/$pod" --timeout=5mkubectl --context fzymgc-house -n clickstack wait --for=condition=Ready "pod/$pod" --timeout=15mdone -
Make sure that Reloader restarted the consumers after step 2. Reloader restarts
cs-clickstack-appandcs-otel-collectorwhenclickstack-secretchanges. It restarts thevectorDaemonSet whenotel-ingest-tokenchanges, which happens forotelcollectoronly. The annotation names the Secret, and itsobservedAtis a Unix time after step 2:Terminal window for w in clickstack/deployment/cs-clickstack-app clickstack/deployment/cs-otel-collector vector/daemonset/vector; doIFS=/ read -r ns kind name <<<"$w"kubectl --context fzymgc-house -n "$ns" get "$kind" "$name" \-o jsonpath='{.spec.template.metadata.annotations.reloader\.stakater\.com/last-reloaded-from}{"\n"}'kubectl --context fzymgc-house -n "$ns" rollout status "$kind/$name"doneThe
clickstack-watchdogCronJob reads the Secret at each run, so it needs no restart. -
For
apponly: set the new value on the HyperDX connectionLocal ClickHouse. HyperDX keeps the connection in MongoDB, and it readsconnections.jsononly while its team has no connection. Until this step runs, HyperDX dashboards, alerts and the HyperDX MCP tools read nothing. The commands use the HyperDX API with your personal API access key, which HyperDX shows in Team Settings, API Keys. The frontend proxy removes one/api, so the URL has two:Terminal window read -rs -p 'HyperDX personal API access key: ' hdx_key; echoapi=https://hyperdx.fzymgc.house/api/api/v2/connectionshdr() { printf 'header = "Authorization: Bearer %s"\nheader = "Content-Type: application/json"\n' "$hdx_key"; }conn=$(curl -sSf --config <(hdr) "$api" | jq -c '.data[] | select(.name == "Local ClickHouse")')jq -n --argjson c "$conn" --rawfile pw <(vault kv get -field=app_password secret/fzymgc-house/cluster/clickstack) \'{name: $c.name, host: $c.host, username: $c.username, password: ($pw | rtrimstr("\n"))}' \| curl -sSf --config <(hdr) -X PUT --data-binary @- "$api/$(jq -r .id <<<"$conn")" \| jq -c '.data | {name, updatedAt}'The HyperDX UI makes the same change: Team Settings, then Connections, then
Local ClickHouse. -
Make sure that the old value fails and the new value works. Run this before any other write to the Vault path, because the old value comes from the previous version of the secret:
Terminal window v=$(vault kv metadata get -format=json secret/fzymgc-house/cluster/clickstack | jq '.data.current_version')old=$(vault kv get -version=$((v - 1)) -field="$key" secret/fzymgc-house/cluster/clickstack)new=$(vault kv get -field="$key" secret/fzymgc-house/cluster/clickstack)printf 'user = "%s:%s"\n' "$user" "$old" | curl -sS --config - 'https://ch.fzymgc.house/?query=SELECT%201'# Code: 516 ... Authentication failedprintf 'user = "%s:%s"\n' "$user" "$new" | curl -sS --config - 'https://ch.fzymgc.house/?query=SELECT%201'# 1For
otelcollector, wait fifteen minutes after the last pod restart. Then make sure that new rows arrive from both writers. Vector rows carrylog.file.path, and rows through the collector do not. The output must have two rows,1for Vector and0for the collector, and each count must be above zero. A writer with no rows has no output row:Terminal window kubectl --context fzymgc-house -n clickstack exec -i cs-clickstack-clickhouse-clickhouse-0-0-0 -c clickhouse-server -- \sh -c 'clickhouse-client --user schema_admin --password "$SCHEMA_ADMIN_PASSWORD"' <<'SQL'SELECT LogAttributes['log.file.path'] != '' AS from_vector, count()FROM default.otel_logsWHERE Timestamp > now() - INTERVAL 5 MINUTEGROUP BY from_vectorSQLFor
app, make sure that the HyperDX Logs view shows rows from the last five minutes. Then run the watchdog by hand. The Job must complete, and the uptime-kuma push monitor must read up:Terminal window job="watchdog-rotation-$(date +%s)"kubectl --context fzymgc-house -n clickstack create job --from=cronjob/clickstack-watchdog "$job"kubectl --context fzymgc-house -n clickstack wait --for=condition=complete "job/$job" --timeout=5m
The collector configuration
Section titled “The collector configuration”cs-otel-collector runs the ClickStack collector image, which is an OpAMP supervisor. HyperDX
hands it an effective configuration at runtime, and that configuration replaces the chart’s
bootstrap file. The chart’s otelCollector values and the bootstrap relay.yaml therefore do not
decide what the collector does. Read the live configuration instead:
kubectl exec -n clickstack deploy/cs-otel-collector -- cat /etc/otel/supervisor-data/effective.yamlThe repository’s own additions are in the ConfigMap
argocd/app-configs/clickstack/otel-collector-custom-config.yaml. It gives the ClickHouse
exporters a persistent sending queue on disk, sets the batch size, and redacts sk- key material
from span attributes, log bodies and log attributes before export.
Traces
Section titled “Traces”The effective configuration wires the traces pipeline: the otlp/hyperdx receiver on :4317
and :4318 with bearertokenauth/hyperdx, then the clickhouse exporter into
default.otel_traces. An empty otel_traces means that no producer is sending traces. Most logs
come from Vector straight into ClickHouse, and that path carries no trace. A producer that sends
its logs over OTLP, such as dns-control, writes the trace id and the span id of each record into
TraceId and SpanId of default.otel_logs, so a log row joins its trace.
A producer must send OTLP with the ingest key as the authorization header:
| From | Endpoint | Auth header |
|---|---|---|
| In-cluster | cs-otel-collector.clickstack.svc.cluster.local:4317 (gRPC) / :4318 (HTTP POST /v1/traces) |
authorization: <ingest-key> |
| External | otel-gateway.fzymgc.house:443 (gRPC, mTLS client certificate) |
authorization: <ingest-key> |
Without the header the receiver returns 401 missing or empty authorization header and drops
every span.
LiteLLM sends traces and four GenAI metric histograms: gen_ai.client.operation.duration,
gen_ai.client.response.duration, gen_ai.client.token.usage and gen_ai.usage.cost. On the
metrics, gen_ai.request.model is the upstream model, not the lane alias, and per-key attribution
is metadata.user_api_key_alias.
Smoke test
Section titled “Smoke test”This proves the pipeline end to end without an instrumented application:
# Read the live team key (the full UUID pattern, not a loose hex match).TOKEN=$(kubectl exec -n clickstack deploy/cs-otel-collector -- \ sh -c "grep -A2 'bearertokenauth/hyperdx' /etc/otel/supervisor-data/effective.yaml \ | grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | head -1")# Stop here if the read is empty. An empty header makes the 401 below look like a broken pipeline.[ -n "$TOKEN" ] || { echo "ERROR: no HyperDX key found in effective.yaml (layout changed?)"; exit 1; }
kubectl port-forward -n clickstack svc/cs-otel-collector 14318:4318 &NOW=$(date +%s)000000000# Pass the token through a 0600 header file so that it stays out of the process arguments.HDR=$(mktemp); chmod 600 "$HDR"; printf 'authorization: %s\n' "$TOKEN" > "$HDR"curl -s -w '%{http_code}\n' -X POST http://127.0.0.1:14318/v1/traces \ -H 'Content-Type: application/json' -H @"$HDR" \ --data "{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"trace-probe\"}}]},\"scopeSpans\":[{\"spans\":[{\"traceId\":\"5b8efff798038103d269b633813fc60c\",\"spanId\":\"eee19b7ec3c1b174\",\"name\":\"probe\",\"kind\":1,\"startTimeUnixNano\":\"$NOW\",\"endTimeUnixNano\":\"$NOW\"}]}]}]}"rm -f "$HDR"# Expect HTTP 200 {"partialSuccess":{}}, then:# SELECT count() FROM default.otel_traces WHERE ServiceName='trace-probe'; -- > 0Mongo-down failure mode
Section titled “Mongo-down failure mode”HyperDX needs MongoDB for session state. When Mongo is unavailable, HyperDX returns 5xx. MongoDB
runs as a 3-member replica set with a PodDisruptionBudget of minAvailable: 2, so it tolerates the
loss of one member. Investigate:
kubectl -n clickstack get pods -o wide | rg mongodbkubectl -n clickstack logs <mongodb-pod> --all-containers --tail=50Watchdog signal contract
Section titled “Watchdog signal contract”clickstack-watchdog (CronJob, every 5 min, argocd/app-configs/clickstack/watchdog-cronjob.yaml)
runs two health checks and pushes an uptime-kuma heartbeat only if both pass. Its two failure
modes are split, so the Job’s exit code and the heartbeat mean different things. Read this table
before drawing any conclusion from kubectl get jobs -n clickstack.
| Case | Exit code | Heartbeat sent? | Which alert speaks | Where the reason is readable |
|---|---|---|---|---|
| ClickStack unhealthy: ClickHouse unreachable, ingest stale (newest metric over 300 s old), engine drift, or a malformed query result | 0 (the Job succeeds) | no | the clickstack-watchdog uptime-kuma monitor goes DOWN after about 2 missed beats |
pod stdout, HEALTH CHECK FAILED: <reason> |
| Push delivery broken: endpoint error, Service unreachable, token rejected | 1 (the Job fails) | no | ClickStack watchdog Job failures (push delivery), a HyperDX tile alert on kube_job_failed, to Pushover (cluster-infra) |
pod stdout, PUSH DELIVERY FAILED with the wget exit status |
| Both fine | 0 | yes | none | pod stdout, stale=…s drift=… then the push |
The Job died before or during the checks: activeDeadlineSeconds: 120 exceeded (Job condition reason DeadlineExceeded), OOMKilled, an image-pull failure, or node eviction |
the Job fails | no | both: the tile alert fires and the uptime-kuma monitor goes DOWN | the Job condition reason, kubectl get job <name> -n clickstack -o jsonpath='{.status.conditions[*].reason}' |
Row 4 is the one that bites mid-incident. If ClickHouse is alive but saturated and the freshness
query blocks past the 120 s deadline, the Job is killed with DeadlineExceeded, no heartbeat is
sent, and both alerts speak. Reading row 2’s meaning into that page points you away from the
fault. Read the Job condition reason first.
A green Job is not a health verdict. Rows 1 and 3 both exit 0. The ClickStack health signal is
heartbeat absence, a dead-man’s switch by design, because an alert that lives inside the
observability stack cannot fire when the stack is dark. The heartbeat carries no failing check,
because the push happens only on success. Triage reads the pod’s stdout in default.otel_logs:
kubectl exec -n clickstack "$(kubectl get pod -n clickstack \ -l clickhouse.com/role=clickhouse-server -o name | head -1 | cut -d/ -f2)" \ -- clickhouse-client --query "SELECT Timestamp, Body FROM default.otel_logs \ WHERE ResourceAttributes['k8s.pod.name'] LIKE 'clickstack-watchdog-%' \ ORDER BY Timestamp DESC LIMIT 20"Triage: the push-delivery alert
Section titled “Triage: the push-delivery alert”The alert fires on stored kube_job_failed samples over a 1 h lookback, not on live object state.
Deleting a failed Job clears the alert only after a full bucket passes, so a re-read of the tile
right after the cleanup still shows red. The two eviction mechanisms do not cover for each other.
ttlSecondsAfterFinished (300 s) binds only Jobs created after it was set, and
failedJobsHistoryLimit (3) evicts an old failed Job only when a newer failed Job arrives. A failed
Job that predates the TTL is reaped by neither, so delete it by hand.
Rules in: the uptime-kuma push endpoint returning an error, the internal Service
uptime-kuma.uptime-kuma.svc.cluster.local unreachable, the push token rejected, and every
Job-level failure the script never sees (row 4 above).
Rules out: nothing unconditionally. A health failure withholds the heartbeat and exits 0, so a script that ran to completion and still failed did fail on the push. A Job killed before or during the checks says nothing about ingest health either way.
Recover with the throwaway pods below. Take Pod B: it keeps the response body, and the body is
what turns a bare non-2xx into a named cause. uptime-kuma maps every internal exception to 404
and names the reason only in msg, which the watchdog’s own BusyBox wget discards. Then make
sure that the clickstack-watchdog monitor is present and active in uptime-kuma, that the
clickstack-watchdog ExternalSecret is Ready and SecretSynced, and that the ping value the
script sends is a non-negative integer.
Recover: re-run the push by hand
Section titled “Recover: re-run the push by hand”The failure line carries only wget’s exit status, never the URL or the token, because pod stdout
is retained 14 days in a queryable store. Recovering more than that means re-running the push by
hand, and the client you re-run it with decides what you get back.
The watchdog image is clickhouse-server:*-alpine, so its only HTTP client is BusyBox wget.
On a non-2xx status BusyBox wget writes the status line to stderr and exits without reading the
body. uptime-kuma answers {"ok":false,"msg":"<the real reason>"}, and its push route maps every
internal exception to HTTP 404, so a 404 from that endpoint does not mean “bad token”. msg is
the only field that names the cause, and it has to be caught live by a client that keeps it.
uptime-kuma’s own log and default.otel_logs do not carry it.
The token is never typed. Both pods read PUSH_TOKEN from a secretKeyRef on Secret
clickstack-watchdog, key token, the way the CronJob does, and reference it only as
${PUSH_TOKEN} from inside the pod. Do not paste the token into a kubectl run argument or into a
URL in your shell. A pasted token survives in the shell history and in the Pod object.
ndots: "1" is mandatory in both pods. musl’s getaddrinfo short-circuits on the cluster-appended
fzymgc.house search domain, which answers NODATA for unknown subdomains, so under the cluster
default ndots:5 the push host never resolves and you get wget: bad address. The dnsConfig
comment in argocd/app-configs/clickstack/watchdog-cronjob.yaml carries the full reason. A bare
kubectl run inherits the cluster default and reproduces that failure.
Both pods are throwaway (--rm --restart=Never) and carry no service account token, mirroring the
CronJob’s automountServiceAccountToken: false. They exist because ttlSecondsAfterFinished
deletes finished watchdog Jobs, so there is usually no watchdog pod left to kubectl exec into.
Pod A reproduces the watchdog’s exact client behaviour, including the status line. Use it to
confirm what the watchdog itself saw. It does not recover msg. Its image must equal the image in
argocd/app-configs/clickstack/kustomization.yaml (images:), because that equality is its claim to
parity. Nothing asserts the equality, so when that image moves, this recipe moves with it
in the same change.
kubectl run -n clickstack watchdog-probe --rm -it --restart=Never \ --image=clickhouse/clickhouse-server:26.7.5.10-alpine \ --overrides='{ "spec": { "automountServiceAccountToken": false, "dnsConfig": {"options": [{"name": "ndots", "value": "1"}]}, "containers": [{ "name": "watchdog-probe", "image": "clickhouse/clickhouse-server:26.7.5.10-alpine", "stdin": true, "tty": true, "command": ["sh"], "env": [{"name": "PUSH_TOKEN", "valueFrom": {"secretKeyRef": {"name": "clickstack-watchdog", "key": "token"}}}] }] } }'Then, from inside that pod:
wget -S -O- "http://uptime-kuma.uptime-kuma.svc.cluster.local:3001/api/push/${PUSH_TOKEN}?status=up&msg=manual&ping=1"Pod B keeps the body of a non-2xx response. Run it on the next occurrence and capture msg.
kubectl run -n clickstack push-body-probe --rm -it --restart=Never \ --image=curlimages/curl:8.21.0 \ --overrides='{ "spec": { "automountServiceAccountToken": false, "dnsConfig": {"options": [{"name": "ndots", "value": "1"}]}, "containers": [{ "name": "push-body-probe", "image": "curlimages/curl:8.21.0", "stdin": true, "tty": true, "command": ["sh"], "env": [{"name": "PUSH_TOKEN", "valueFrom": {"secretKeyRef": {"name": "clickstack-watchdog", "key": "token"}}}] }] } }'Then, from inside that pod. -i prints the status line and the headers, and curl keeps the body
on a non-2xx status, so msg arrives with them:
curl -sS -i "http://uptime-kuma.uptime-kuma.svc.cluster.local:3001/api/push/${PUSH_TOKEN}?status=up&msg=manual&ping=1"Traefik terminates every TLS endpoint: hyperdx.fzymgc.house (Secret hyperdx-tls),
ch.fzymgc.house and ch-native.fzymgc.house (Secret ch-tls), and otel-gateway.fzymgc.house
(Secret otel-gateway-tls, client CA in otel-gateway-mtls). Traefik picks up a renewed Secret by
itself, so certificate rotation needs no restart. ClickHouse and the collector serve no TLS.
Backups
Section titled “Backups”| Data | Protection |
|---|---|
| ClickHouse tables | Two ReplicatedMergeTree replicas. The native backups below hold only the tables with no TTL. The data volumes (longhorn-single) carry only the Longhorn fstrim job, so Longhorn takes no backup of them, and Velero skips them by resource policy |
MongoDB volumes (longhorn) |
The default Velero schedule (the clickstack namespace is not excluded) and Longhorn’s daily-backup. No fsync hooks are configured, so both are crash-consistent |
Keeper volumes (longhorn) |
The same as MongoDB |
MongoDB holds the HyperDX state that ClickHouse does not: users, teams, sources, dashboards, saved searches and alerts. Read a Longhorn volume’s newest backup:
kubectl get backups.longhorn.io -n longhorn-system -l backup-volume=<pv-name> \ -o jsonpath='{range .items[*]}{.status.backupCreatedAt} {.status.state}{"\n"}{end}' | sort | tail -1Native ClickHouse backups
Section titled “Native ClickHouse backups”The CronJob clickhouse-backup (argocd/app-configs/clickstack/clickhouse-backup.yaml) runs
BACKUP at 05:00 America/New_York, the node zone that CronJobs without timeZone use, as
schema_admin. It backs up every Replicated* table in default that has no TTL to the B2 bucket
fzymgc-k8s-backups under clickhouse/. Today that is only dns_query_daily. The tables with a
TTL hold telemetry that expires after 14 or 30 days, and only the two replicas hold it:
| Item | Value |
|---|---|
| Full backup | clickhouse/full-<UTC timestamp>, on the first run of each ISO week (Monday start) |
| Incremental backup | clickhouse/incr-<UTC timestamp>, on every other run, against the newest full backup |
| Retention | The bucket’s lifecycle rule on clickhouse/ hides each file 15 days after upload and deletes it a day later. Nothing in the cluster deletes a backup |
| Credentials | The s3 endpoint section of the chart values, from Secret clickhouse-backup-b2 (Vault cluster/clickstack/b2). No SQL statement names a key |
| Upload | Each S3 request has 20 retries, so a stuck upload fails the backup instead of retrying for a day. B2 stores a file on one vault, a group of B2 storage servers. A multipart upload stays on its vault, so a vault that refuses writes fails every retry. B2 picks a new vault for each PutObject |
| Not in the backup | Every table with a TTL: the OpenTelemetry tables, their 15-minute rollups and hyperdx_sessions. Also the materialized views and the metrics_ts TimeSeries table. bootstrap-schema-configmap.yaml recreates them, and dns-query-schema.yaml recreates dns_query_daily_mv and dns_query_daily_resolver_mv. The backup holds dns_query_daily, because its engine is Replicated* and it has no TTL |
| Alert | ClickHouse backup FAILED, when a Job of the CronJob failed in the last day |
The 15-day expiry always leaves the newest full backup and its incremental backups, because a new full backup starts every week.
List the backups:
b2 ls b2://fzymgc-k8s-backups/clickhouse/Run a backup now:
kubectl create job -n clickstack --from=cronjob/clickhouse-backup clickhouse-backup-manualkubectl -n clickstack logs -f job/clickhouse-backup-manualTriage: ClickHouse backup FAILED
Section titled “Triage: ClickHouse backup FAILED”The alert fires when a Job of clickhouse-backup failed. The Job and its log stay for one day.
kubectl get jobs -n clickstack gives the name of the Job.
-
Read the log of the Job:
kubectl -n clickstack logs job/<name>. -
Read the reason of the failure:
kubectl get job -n clickstack <name> -o jsonpath='{.status.conditions[*].reason}'. -
If the reason is
DeadlineExceeded, find the BACKUP that still runs. The server does not stop a BACKUP when its client disconnects:Terminal window for pod in $(kubectl get pod -n clickstack -l clickhouse.com/role=clickhouse-server -o name); dokubectl exec -n clickstack "$pod" -- clickhouse-client --query \"SELECT hostName(), query_id, round(elapsed) FROM system.processes WHERE query LIKE 'BACKUP%'"done -
Stop that BACKUP on the pod that runs it. Do not leave it to run. At shutdown, the server waits for a running BACKUP, and Kubernetes kills the server after 30 seconds:
Terminal window kubectl exec -n clickstack <pod> -- clickhouse-client --query "KILL QUERY WHERE query_id = '<query_id>'" -
List the uploads that B2 did not finish:
b2 file large unfinished list b2://fzymgc-k8s-backups. The_v<number>_field of a file ID names the vault of the upload. -
If all unfinished uploads under
clickhouse/name one vault, send their file IDs to Backblaze support. In October 2026, vaultv0001059refused writes and stalled every multipart upload on it (Backblaze request 1298650). -
When the cause is gone, run a backup now with the commands above.
Restore from a native backup
Section titled “Restore from a native backup”Restore into a new database, then compare, then move the data you need. A restore of an incremental
backup reads its full backup too. The tables use the Keeper path /clickhouse/tables/{uuid}/{shard},
and a restore under a new name gets new UUIDs, so it does not collide with the live tables.
-- Run as schema_admin on one replica. <name> is a full-... or incr-... directory.CREATE DATABASE restore_check;RESTORE TABLE default.dns_query_daily AS restore_check.dns_query_dailyFROM S3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/clickhouse/<name>');
SELECT count(), min(day), max(day) FROM restore_check.dns_query_daily;Drop the scratch database when you finish: DROP DATABASE restore_check SYNC.
Dashboards and alerts
Section titled “Dashboards and alerts”Dashboards and alerts are GitOps state. argocd/app-configs/clickstack-alerts/bootstrap-script.yaml
holds bootstrap.js, and a PostSync Job (mongo:8.0) runs it against HyperDX’s MongoDB on every
sync. It upserts each dashboard, tile and alert by name, so a merge is the whole deployment. There
is no kubectl apply. Alerts deliver to the Pushover (cluster-infra) webhook, whose token comes
from argocd/app-configs/clickstack-alerts/external-secret.yaml. tools/hyperdx/dashboards/ holds
exported dashboard JSON, and tools/hyperdx/README.md states how it is used.
To add a dashboard or an alert, edit bootstrap.js, open a PR, and after the merge make sure that
the object exists in the HyperDX UI. To read the alert intervals in force, read the script.
Useful queries
Section titled “Useful queries”HyperDX UI search uses Lucene syntax. For direct ClickHouse queries:
kubectl exec -it -n clickstack \ $(kubectl get pod -n clickstack -l clickhouse.com/role=clickhouse-server -o name | head -1) \ -- clickhouse-client-- Log count last hourSELECT count(*) FROM otel_logs WHERE Timestamp > now() - INTERVAL 1 HOUR;
-- Metrics count last hourSELECT count(*) FROM otel_metrics_gauge WHERE TimeUnix > now() - INTERVAL 1 HOUR;
-- Trace span count last hour, by service (0 rows total means that no producer-- is emitting OTLP traces; the pipeline itself is wired)SELECT ServiceName, count(*) AS spansFROM otel_traces WHERE Timestamp > now() - INTERVAL 1 HOURGROUP BY ServiceName ORDER BY spans DESC;
-- Top namespaces by log volumeSELECT ResourceAttributes['k8s.namespace.name'] AS namespace, count(*) AS cntFROM otel_logsWHERE Timestamp > now() - INTERVAL 1 HOURGROUP BY namespaceORDER BY cnt DESCLIMIT 20;
-- MCP calls by server (LiteLLM relay). LiteLLM relays MCP over one route per-- registered server and emits ONE server span per call, named for the TEMPLATED-- route, so counting those spans counts MCP calls. The group-by extracts only-- the server segment from http.target. Never group on the raw target, which is-- the concrete request path and can carry key material.SELECT extract(SpanAttributes['http.target'], '^/([^/]+)/mcp') AS mcp_server, count(*) AS callsFROM otel_tracesWHERE ServiceName = 'litellm' AND SpanName = 'POST /{mcp_server_name}/mcp' AND Timestamp > now() - INTERVAL 24 HOURGROUP BY mcp_serverORDER BY calls DESC;The MCP query is also a tile on the LiteLLM — LLM & key usage dashboard. There is no by-method
equivalent, because LiteLLM relays the JSON-RPC payload without parsing it, so no MCP method name
reaches a span attribute.
nas-support collector verification queries
Section titled “nas-support collector verification queries”The native otelcol-nas-support collector on the nas-support LXC (192.168.40.32, role
nas-support-otel-collector) sends journald logs, host metrics, the backup-freshness file.mtime
series and the backup duration and per-source failure gauges. Every row is keyed on
ResourceAttributes['host.name']='nas-support', set by the collector’s resource processor.
Open a ClickHouse client against the ClickStack pod:
kubectl exec -it -n clickstack \ $(kubectl get pod -n clickstack -l clickhouse.com/role=clickhouse-server -o name | head -1) \ -- clickhouse-client-- journald logs arriving from .202 (expect > 0)SELECT count() AS logs_last_hourFROM otel_logsWHERE ResourceAttributes['host.name'] = 'nas-support' AND Timestamp > now() - INTERVAL 1 HOUR;
-- host metrics arriving from .202 (expect > 0)SELECT count() AS hostmetric_pointsFROM otel_metrics_gaugeWHERE ResourceAttributes['host.name'] = 'nas-support';
-- backup-freshness file.mtime: a recent epoch value, small age.-- file.mtime is a non-monotonic Sum (epoch seconds), NOT a gauge. It lands in-- otel_metrics_sum, so query that table, not otel_metrics_gauge.SELECT MetricName, Value, now() - Value AS age_sFROM otel_metrics_sumWHERE MetricName = 'file.mtime' AND ResourceAttributes['host.name'] = 'nas-support'ORDER BY TimeUnix DESCLIMIT 1;
-- backup duration and per-source failure gauges from the last run-- (expect nas.kopia.backup.duration.seconds present; snapshot.failed rows only-- if a source failed, carrying a 'source' attribute).SELECT MetricName, Value, Attributes['source'] AS sourceFROM otel_metrics_gaugeWHERE MetricName LIKE 'nas.kopia%' AND TimeUnix > now() - INTERVAL 10 MINUTEORDER BY TimeUnix DESC;Smoke checks on .202, over the nas-automation SSH connection:
# Collector process is upsystemctl is-active otelcol-nas-support # -> active
# health_check extension answers on the loopback bindcurl -fsS http://127.0.0.1:13133/ # -> HTTP 200file.mtime is the one series that is a Sum rather than a gauge. A query against
otel_metrics_gauge for it returns zero rows and looks like a broken pipeline.
NAS backup freshness alerts
Section titled “NAS backup freshness alerts”Two ClickStack tile alerts on the NAS backup — freshness dashboard watch the NAS to B2 Kopia
backups. They live in bootstrap-script.yaml and deliver to the cluster-infra Pushover webhook.
The staleness signal is not a ClickStack alert. It stays on the host as the
kopia-freshness.timer dead-man’s switch, because HyperDX cannot compute a last-success age from a
Sum inside an alert. Three signals cover each other’s blind spot:
| Signal | Threshold and interval | Catches | Path |
|---|---|---|---|
NAS Kopia snapshot FAILED |
above_exclusive 0, 1d |
a specific source broke, the earliest “a backup failed” signal | ClickStack |
NAS backup telemetry MISSING (collector dark) |
below 1, 1h |
telemetry stopped, the .202 collector or the box is dark |
ClickStack |
NAS Kopia backup STALE (kopia-freshness.timer) |
over nas_kopia_freshness_max_age_hours (36), hourly |
backups stopped, needs no telemetry, the primary staleness signal | .202 on host |
The producer (ansible/roles/nas-kopia-backup/templates/kopia-b2-backup.sh.j2) emits
nas.kopia.snapshot.failed = 1 only when a source’s snapshot fails, with a source attribute. A
healthy run emits nothing for this metric. There is no 0 sample. The alert therefore fires when a
1 appears in the window, and absence in the window means “no known failure”, not a healthy zero.
The alert tile is un-grouped so that it fires on any failed source. To identify which source
failed, use the per-source nas-support — backup freshness & health dashboard or the last query
above. The failure alert uses the 1d interval, the widest window HyperDX offers, so the once-per-run
emit cannot fall between windows. The on-host timer covers the sustained-outage tail after a failed
point ages out of the window.
Read the last-success age by hand:
SELECT (toUnixTimestamp(now()) - toUInt64(max(Value))) / 3600 AS hours_since_successFROM otel_metrics_sumWHERE MetricName = 'file.mtime' AND ResourceAttributes['host.name'] = 'nas-support';Make sure that the on-host backstop is armed on .202:
systemctl is-enabled kopia-freshness.timer # -> enabled