Skip to content

ClickStack Runbook

ClickStack (ClickHouse, HyperDX, MongoDB and an OTel Collector) is the cluster’s logs, metrics and traces backend. This page states what exists, where each part is defined, the query forms, and the procedures that touch it. The chart values are in argocd/app-configs/clickstack-chart/. The certificates, the ExternalSecret, the IngressRoutes, the network policy, the collector’s custom configuration, the LoadBalancer Service, the schema bootstrap and the watchdog are in argocd/app-configs/clickstack/. The dashboards and alerts bootstrap is in argocd/app-configs/clickstack-alerts/. The MCP servers over ClickHouse are in argocd/app-configs/clickhouse-mcp/.

Service Endpoint
HyperDX UI https://hyperdx.fzymgc.house (Keycloak OIDC, client clickstack-hyperdx from tf/keycloak/clickstack.tf)
OTLP, in-cluster cs-otel-collector.clickstack.svc:4317 (gRPC) / :4318 (HTTP), bearer token
OTLP, external otel-gateway.fzymgc.house:443 (Traefik, mTLS client certificate, gRPC) → cs-otel-collector:4317, bearer token. Defined in argocd/app-configs/monitoring-otel-scraper/otlp-gateway.yaml with clientAuthType: RequireAndVerifyClientCert
ClickHouse HTTP https://ch.fzymgc.house (Traefik) → cs-clickstack-clickhouse-clickhouse-headless:8123
ClickHouse native ch-native.fzymgc.house:443 (Traefik TCP, TLS SNI) → cs-clickstack-clickhouse-clickhouse-headless:9000
ClickHouse, in-cluster cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc, :8123 HTTP / :9000 native, no TLS

The otel-collector-vip LoadBalancer (192.168.250.149 and fddb:f665:73f7:250::149, ports 4317 and 4318) also exposes the collector on the routed VIP prefix. It is plaintext with bearer auth. The receivers of the collector have no TLS, and nothing mounts the otel-collector-tls Certificate. Send external OTLP through otel-gateway.fzymgc.house instead.

The schema comes from argocd/app-configs/clickstack/bootstrap-schema-configmap.yaml, which scripts/clickstack/generate-bootstrap-ddl.sh generates from the HyperDX collector’s seed schema. Do not hand-edit it. The bootstrap Job applies it. The DNS query rollup is the one exception, as section DNS query rollups states.

Table Time column Retention
default.otel_logs Timestamp 14 days
default.otel_metrics_gauge, otel_metrics_sum, otel_metrics_histogram, otel_metrics_exponential_histogram, otel_metrics_summary TimeUnix 30 days
default.otel_traces Timestamp 14 days
default.hyperdx_sessions TimestampTime 14 days
default.otel_logs_kv_rollup_15m, otel_traces_kv_rollup_15m Timestamp 14 days
default.dns_query_daily day No expiry

default.otel_logs has a Timestamp column and no TimestampTime. Use Timestamp in every query and in every HyperDX timestampValueExpression on a logs source. Column names follow the OTel ClickHouse exporter schema: MetricName, Value, Attributes, ResourceAttributes and TimeUnix for metrics, and Body, ServiceName, ResourceAttributes and Timestamp for logs. The ClickHouse system log tables carry a 7-day TTL set in the chart values.

default.dns_query_daily holds the number of DNS queries of each UTC day, with no expiry. Two materialized views fill it from default.otel_logs as rows arrive. dns_query_daily_mv counts the rows of Blocky, the client tier until 2026-10-14, and it counts no row after the cut-over. dns_query_daily_resolver_mv counts the rows of the recursor, with the attribute names of The resolver query rows below. argocd/app-configs/clickstack/dns-query-schema.yaml holds the table and both views, and its PostSync Job creates them as schema_admin. The OpenSpec change dns-query-log-retention states the requirements.

Column dns_query_daily_mv dns_query_daily_resolver_mv
day toDate(Timestamp), a UTC day toDate(Timestamp), a UTC day
service ServiceName: blocky or blocky-public ServiceName: dns-resolver or dns-resolver-public
client LogAttributes['client_ip'] LogAttributes['client.address']
name LogAttributes['question_name'] LogAttributes['dns.question.name']
type LogAttributes['question_type'] LogAttributes['dns.question.type']
result LogAttributes['response_type'], such as RESOLVED or BLOCKED LogAttributes['dns.policy.action']: nxdomain, nodata, passthru, or empty when no rule decided
code LogAttributes['response_code'] LogAttributes['dns.response.code']
queries The number of query rows with these values The number of query rows with these values

Each view counts query rows only. A query row is a row with a question name. The daemon lines of a resolver carry the same ServiceName and no question name, so no view counts them.

The table is a ReplicatedSummingMergeTree. ClickHouse adds up the rows with the same key in the background, at no fixed time. Read the table with sum(queries) and GROUP BY, because one row is not a count:

-- The names that a rule blocked most often in the last 7 days, before and after the cut-over.
SELECT name, sum(queries) AS n
FROM default.dns_query_daily
WHERE day >= today() - 7 AND result IN ('BLOCKED', 'nxdomain', 'nodata')
GROUP BY name ORDER BY n DESC LIMIT 20;

The MCP users read no row of the table or of either view, because of the row filter "0" on each in argocd/app-configs/clickstack-chart/values.yaml. A SELECT on a view reads the table, and ClickHouse looks up a row policy by the name that the query names, so each view needs its own filter. The HyperDX user app reads it, as it reads the DNS rows of otel_logs.

Each view counts the rows with a Timestamp at or after the boundary in its WHERE clause. A one-time fill counts the rows before the boundary, as schema_admin, at least 6 hours after the boundary.

dns_query_daily_mv has the boundary 2026-10-13 00:00:00 UTC. Its fill ran once, from the oldest day that otel_logs held in full:

INSERT INTO default.dns_query_daily
SELECT
toDate(Timestamp) AS day, ServiceName AS service, LogAttributes['client_ip'] AS client,
LogAttributes['question_name'] AS name, LogAttributes['question_type'] AS type,
LogAttributes['response_type'] AS result, LogAttributes['response_code'] AS code,
count() AS queries
FROM default.otel_logs
WHERE ServiceName IN ('blocky', 'blocky-public')
AND LogAttributes['question_name'] != ''
AND Timestamp < toDateTime64('2026-10-13 00:00:00', 9)
AND toDate(Timestamp) >= today() - 13
GROUP BY day, service, client, name, type, result, code;

dns_query_daily_resolver_mv has the boundary 2026-10-23 00:00:00 UTC. Its fill counts every row of the recursor before the boundary. The first rows are from 2026-10-14, and otel_logs drops them on 2026-10-28, so run the fill between 2026-10-23 06:00 and 2026-10-28 00:00 UTC:

INSERT INTO default.dns_query_daily
SELECT
toDate(Timestamp) AS day, ServiceName AS service, LogAttributes['client.address'] AS client,
LogAttributes['dns.question.name'] AS name, LogAttributes['dns.question.type'] AS type,
LogAttributes['dns.policy.action'] AS result, LogAttributes['dns.response.code'] AS code,
count() AS queries
FROM default.otel_logs
WHERE ServiceName IN ('dns-resolver', 'dns-resolver-public')
AND LogAttributes['dns.question.name'] != ''
AND Timestamp < toDateTime64('2026-10-23 00:00:00', 9)
GROUP BY day, service, client, name, type, result, code;

Run each fill once. A second run counts each day before the boundary twice. The boundary of a view must be later than the first sync that creates the view, or the rows between the boundary and the creation count nowhere. If the sync comes later than the boundary, move the boundary to a later day on the branch before the merge, and use the same day in the fill. After the first sync, do not change a boundary in dns-query-schema.yaml. The Job runs CREATE ... IF NOT EXISTS, so a new boundary never reaches the live view.

This query returns one row for each day of the last 13 days on which the rollup and the query rows differ. An empty result is the pass. A day that has no rollup row shows rolled_up 0, also in a session that sets join_use_nulls = 1. Until the fill of dns_query_daily_resolver_mv runs, each day from 2026-10-14 to its boundary differs:

SELECT l.day, l.c AS queried, ifNull(r.q, 0) AS rolled_up
FROM
(
SELECT toDate(Timestamp) AS day, count() AS c
FROM default.otel_logs
WHERE ((ServiceName IN ('blocky', 'blocky-public') AND LogAttributes['question_name'] != '')
OR (ServiceName IN ('dns-resolver', 'dns-resolver-public') AND LogAttributes['dns.question.name'] != ''))
AND toDate(Timestamp) BETWEEN today() - 13 AND yesterday()
GROUP BY day
) AS l
LEFT JOIN
(
SELECT day, sum(queries) AS q
FROM default.dns_query_daily
WHERE day BETWEEN today() - 13 AND yesterday()
GROUP BY day
) AS r USING day
WHERE ifNull(r.q, 0) != l.c
ORDER BY l.day;

Each replica reads its own copy of both tables, so run the query on each replica. schema_admin has no READ ON REMOTE, so clusterAllReplicas fails for it. Save the query as compare.sql and run it from a workstation with kubectl access to the cluster:

Terminal window
for r in 0 1; do
echo "replica $r"
kubectl -n clickstack exec -i "cs-clickstack-clickhouse-clickhouse-0-$r-0" -c clickhouse-server -- \
sh -c 'clickhouse-client --user schema_admin --password "$SCHEMA_ADMIN_PASSWORD" --multiquery' < compare.sql
done

Each view runs inside each insert into otel_logs. If a view fails, every insert fails, and all log ingest stops. If the collector logs insert errors that name a view, run this as schema_admin first, with the view that the errors name:

DROP VIEW default.dns_query_daily_resolver_mv SYNC;

The table and its rows stay. Each view runs as its definer, schema_admin. If schema_admin loses INSERT on default.dns_query_daily, every insert fails the same way. DROP TABLE default.dns_query_daily SYNC deletes the history for good, so run it only on Sean’s decision.

The CronJob dns-query-archive (argocd/app-configs/clickstack/dns-query-archive.yaml) writes the DNS rows of each UTC day to Backblaze B2 as one Parquet file. A file holds every stored column of otel_logs and the raw Body. The job runs at 06:00 UTC (timeZone: Etc/UTC) as the user dns_query_archiver. The OpenSpec change dns-query-log-retention states the requirements.

Item Value
File fzymgc-k8s-backups/dns-query-log/<yyyy>/<mm>/<dd>.parquet, compressed with zstd
Rows Every row of blocky, blocky-public, dns-resolver and dns-resolver-public of the day, also the lines that are not queries
Window Each run checks each day from 13 days back to the previous day. It writes a missing file and rewrites a file that holds fewer rows than otel_logs. It keeps a full file. The TTL of otel_logs can drop part of the day 14 days back, so the window stops at 13
Retention The bucket’s rule on dns-query-log/ hides each file 365 days after upload and deletes it a day later. The earlier version of a rewritten file goes a day after the rewrite
Credentials The s3 endpoint dns_query_archive of the chart values, from Secret dns-query-archive-b2 (Vault cluster/clickstack/dns-archive-b2). No SQL statement names a key
User dns_query_archiver reads only the DNS rows of otel_logs and writes only below dns-query-log/. The scope of its S3 grant needs access_control_improvements.enable_read_write_grants in the chart values. Without the setting, ClickHouse turns the grant into S3 ON *.*
Alert DNS query archive FAILED, when a Job of the CronJob failed in the last day

The log has one line for each day: dns-query-archive <day> logged=<rows in otel_logs> archived=<rows in the file> action=keep|write|none. A day that failed has a line with FAILED and the reason. A run fails when a count or a write fails, or when a written file holds fewer rows than otel_logs. The run goes on with the other days, and then exits 1.

A run repairs every day in the window, so the next night writes the day that a failed night missed. Keep a failed Job until the alert has fired. The alert reads kube_job_failed of the Job at the end of each hour, so an earlier delete clears the signal. ttlSecondsAfterFinished removes the Job after a day. To run the job now, give the Job a new name:

Terminal window
kubectl create job -n clickstack --from=cronjob/dns-query-archive dns-query-archive-manual
kubectl -n clickstack logs -f job/dns-query-archive-manual

Run these as schema_admin. ClickHouse takes the key from the endpoint section, so no query names a key:

-- The rows of one day.
SELECT count() FROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/2026/10/12.parquet', 'Parquet');
-- The names that one client looked up in a month.
SELECT LogAttributes['question_name'] AS name, count() AS n
FROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/2026/10/*.parquet', 'Parquet')
WHERE LogAttributes['client_ip'] = '192.168.20.220' AND LogAttributes['question_name'] != ''
GROUP BY name ORDER BY n DESC LIMIT 50;
-- The answers of the blocklists in a month, by day.
SELECT toDate(Timestamp) AS day, count() AS blocked
FROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/2026/10/*.parquet', 'Parquet')
WHERE LogAttributes['response_type'] = 'BLOCKED'
GROUP BY day ORDER BY day;

If ClickHouse is down, clickhouse local on a workstation reads the same files with the key from fzymgc-house/cluster/clickstack/dns-archive-b2. Use it as the fallback only, because that key can also write the prefix.

This query returns one row for each day of the window on which the file and otel_logs differ. An empty result is the pass. A day that has no file shows archived 0, also in a session that sets join_use_nulls = 1:

SELECT l.day, l.c AS logged, ifNull(a.c, 0) AS archived
FROM
(
SELECT toDate(Timestamp) AS day, count() AS c
FROM default.otel_logs
WHERE ServiceName IN ('blocky', 'blocky-public', 'dns-resolver', 'dns-resolver-public')
AND toDate(Timestamp) BETWEEN today() - 13 AND yesterday()
GROUP BY day
) AS l
LEFT JOIN
(
SELECT toDate(replaceRegexpOne(_path, '^.*/dns-query-log/(\\d{4})/(\\d{2})/(\\d{2})[.]parquet$', '\\1-\\2-\\3')) AS day,
count() AS c
FROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/**/*.parquet', 'Parquet',
'Timestamp DateTime64(9)')
WHERE day BETWEEN today() - 13 AND yesterday()
GROUP BY day
) AS a USING day
WHERE ifNull(a.c, 0) != l.c
ORDER BY l.day;

The acceptance of the change compares four counts for one day. archived equals logged, and rolled_up equals queried. queried counts the query rows of both name sets, as the two views do, see DNS query rollups:

SELECT
(SELECT count() FROM s3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/dns-query-log/<yyyy>/<mm>/<dd>.parquet', 'Parquet')) AS archived,
(SELECT count() FROM default.otel_logs
WHERE ServiceName IN ('blocky', 'blocky-public', 'dns-resolver', 'dns-resolver-public') AND toDate(Timestamp) = '<day>') AS logged,
(SELECT count() FROM default.otel_logs
WHERE ((ServiceName IN ('blocky', 'blocky-public') AND LogAttributes['question_name'] != '')
OR (ServiceName IN ('dns-resolver', 'dns-resolver-public') AND LogAttributes['dns.question.name'] != ''))
AND toDate(Timestamp) = '<day>') AS queried,
(SELECT sum(queries) FROM default.dns_query_daily WHERE day = '<day>') AS rolled_up;

Renovate proposes each ClickStack release in one attended pull request, update clickstack observability plane. It moves the HyperDX image tag and the collector image tag in argocd/app-configs/clickstack-chart/values.yaml, and the chart version in kustomization.yaml. The bootstrap Job runs only CREATE ... IF NOT EXISTS, so it never changes a table that exists. Before you merge the pull request, do these steps on its branch:

  1. Make sure that the HyperDX tag and the collector tag in values.yaml show the same version.

  2. From the repository root, run scripts/clickstack/generate-bootstrap-ddl.sh. It reads the seed version from the collector tag. Commit the changed bootstrap-schema-configmap.yaml.

  3. List the seed files that changed between the two collector versions:

    Terminal window
    gh api 'repos/hyperdxio/hyperdx/compare/@hyperdx/otel-collector@<old>...@hyperdx/otel-collector@<new>' \
    --jq '.files[] | select(.filename | test("docker/otel-collector/schema/seed/")) | .filename'
  4. If a seed file of an existing table changed, the live table needs a migration script on the model of scripts/clickstack/migrate-schema-2.37.0.sh. Write it, rehearse it, and run it as part of the upgrade.

  5. If the chart version changed, compare the rendered manifests of the old and the new chart (argocd/CLAUDE.md, Change safety).

Source Signal Path
Vector (argocd/app-configs/monitoring-vector/vector.yaml) logs from every node the clickhouse_logs sink writes straight to ClickHouse. No OTLP, no collector
otel-scrape and otel-node (argocd/app-configs/monitoring-otel-scraper/) cluster metrics OTLP to the collector with the ingest key from the otel-ingest-token ExternalSecret in otel-scraper
otel-network (argocd/app-configs/monitoring-otel-scraper/network-collector-values.yaml) the syslog of the UniFi gateway, switches and APs, ServiceName='unifi' UDP from each device to 192.168.250.150, then OTLP to the collector with the same ingest key (The UniFi rows)
The NAS, nas-support and resolver collectors (ansible/roles/{nas-otel-collector,nas-support-otel-collector,dns-otel-collector}) host metrics and journals OTLP through otel-gateway.fzymgc.house with a client certificate from the ICA1 PKI and the ingest key from Vault
dnscrypt-proxy on the resolver nodes (ansible/roles/dns-otel-collector) its journal, ServiceName='dnscrypt-proxy', with no client address the resolver collector, as the resolver query rows
LiteLLM (argocd/app-configs/litellm-chart/values.yaml) traces and GenAI metrics, ServiceName='litellm' OTLP HTTP to the collector, OTEL_HEADERS from the litellm-otel-ingest Secret
engram (argocd/cluster-app/templates/agent-memory.yaml), headroom and hl-assets traces and metrics OTLP with an otlp-headers Secret each
dns-control, its webhook and its operator commands (argocd/app-configs/dns-control/, argocd/app-configs/external-dns/values.yaml) traces, metrics and logs, ServiceName dns-control, dns-control-webhook or dns-control-cli OTLP HTTP to the collector, OTEL_EXPORTER_OTLP_HEADERS from the dns-control-otlp-headers and external-dns-otlp-headers Secrets. Vector leaves the two containers out (Telemetry)

Each resolver node runs two PowerDNS Recursor processes, pdns-recursor@main for the trusted legs and pdns-recursor@public for Guest and Telework. Each process sends one protobuf message for each answer to DNS-collector on the node, the unit dns-resolver-reader.service, which writes one JSON row for each answer to its journal. The resolver collector parses the row and gives it one set of engine-neutral names, so dns-control reads one shape whatever engine answers. The OpenSpec change dns-policy-as-rpz defines them. Where an OpenTelemetry semantic convention exists, the name follows it.

Name Holds
ServiceName the process that answered: dns-resolver for the main process or dns-resolver-public for the public process. The name is dnstap.identity of the row, the server_id of the process. Until 2026-10-14 the names were blocky and blocky-public
LogAttributes['client.address'] the client address
LogAttributes['dns.question.name'], LogAttributes['dns.question.type'] the question. The name keeps the trailing dot
LogAttributes['dns.response.code'], LogAttributes['dns.answer'] the answer. The answer holds up to three record values, separated by a comma
LogAttributes['dns.policy.zone'], LogAttributes['dns.policy.trigger'], LogAttributes['dns.policy.hit'] the policy zone that decided the answer, such as hagezi.feed.rpz.internal., the rule that matched and the name that matched it. A row that no rule decided carries none of the policy attributes
LogAttributes['dns.policy.action'] nxdomain for a block, nodata for the rebinding zone of the public process, and passthru for an allow

The raw names of the reader stay on the row beside the neutral names, such as dns.qname, dns.rcode and powerdns.applied-policy. ansible/roles/dns-otel-collector/templates/otel-config.yaml.j2 holds the mapping. No row of the two health names, dns-health.fzymgc.house and health.dns.invalid, reaches ClickStack: the Lua script of each process keeps them out of the protobuf, and the processor filter/health of the collector drops a row that arrives anyway. The daemon lines of the two units pdns-recursor@main.service and pdns-recursor@public.service carry the same two ServiceNames as the query rows, and the other lines of the reader carry dns-resolver-reader.

The collector scrapes /metrics of each process, 127.0.0.1:8082 for main and 127.0.0.1:8083 for public, every 60 seconds. The scrape job name is the ServiceName of the metric, dns-resolver or dns-resolver-public. Two counters pass: pdns_recursor_questions, every query with the RD bit set, and pdns_recursor_servfail_answers, every SERVFAIL the process gave. The collector copies them to dns_resolver_responses_total and dns_resolver_errors_total under the same ServiceName. dns-control, the tile DNS public resolver failed resolutions per answer and the alert DNS public resolver failing (Guest/Telework) of argocd/app-configs/clickstack-alerts/bootstrap-script.yaml read the copies. The tile and the alert filter on ServiceName = 'dns-resolver-public'.

The Deployment otel-network in the namespace otel-scraper receives the syslog of the UniFi gateway, switches and APs on UDP 514 of the VIP 192.168.250.150. The controller sends one syslog target to every device. The collector runs the contrib distribution, because the otelcol-k8s image of otel-scrape has no syslog receiver. Three replicas run on three nodes, and a PodDisruptionBudget lets one of them go at a time, so the VIP stays announced while a replica restarts. The network policy network-collector admits the Lab address of the gateway, 192.168.200.1, and Core, 192.168.40.0/22, where the switches and APs have their addresses.

Each syslog line becomes one row with ServiceName='unifi'. The receiver parses RFC 3164 and reads the timestamp as America/New_York time. Each device puts a token between its hostname and the program: the gateway writes its hostname a second time, and a switch or an AP writes its MAC address and firmware. A regex in the receiver skips the token. The parts of the line go to these attributes:

Attribute Holds
LogAttributes['hostname'] the name of the device, such as router, ap-basement or switch-library
LogAttributes['appname'], LogAttributes['proc_id'] the program and its process id, such as hostapd and 9561
LogAttributes['message'] the text after the program

A line that does not match the regex keeps the attributes of the RFC 3164 parser. A line that does not parse keeps its raw body and gets no timestamp from the line. Two kinds of line do not match, so search them in Body:

  • A zone policy with logging writes one line for each match, with no program: router [LAN_WAN-A-10000] DESCR="internal-log-external-dns" IN=br3000 OUT=eth4 ... SRC=... DST=... PROTO=UDP ... DPT=53. appname is empty.
  • The Network application sends its events in CEF, a line format for SIEM tools, with no priority: router CEF:0|Ubiquiti|UniFi Network|11.0.89|545|Config Created|5|UNIFIcategory=Audit .... The RFC 3164 parser does not take a line without a priority, so hostname is empty. UNIFIcategory names the content, such as Audit, Client Devices or Security.

The collector takes no IPFIX. The gateway meters IPFIX flows in the kernel module ipt_NETFLOW, which sees only the packets that pass through the CPU. While IPFIX export is on, the gateway turns off its hardware offload, and every forwarded packet goes through the CPU.

To count the lines of the last hour by device and program:

SELECT LogAttributes['hostname'] AS device, LogAttributes['appname'] AS program, count() AS lines
FROM default.otel_logs
WHERE ServiceName = 'unifi' AND Timestamp > now() - INTERVAL 1 HOUR
GROUP BY device, program
ORDER BY lines DESC

To list the DNS connections from Internal to the internet of the last day, by client and server (Zone firewall):

SELECT extract(Body, 'SRC=(\\S+)') AS client, extract(Body, 'DST=(\\S+)') AS server,
extract(Body, 'DPT=(\\d+)') AS port, count() AS lines, max(Timestamp) AS last
FROM default.otel_logs
WHERE ServiceName = 'unifi' AND position(Body, 'DESCR="internal-log-external-dns"') > 0
AND Timestamp > now() - INTERVAL 1 DAY
GROUP BY client, server, port
ORDER BY lines DESC

argocd/app-configs/clickhouse-mcp/ runs two mcp-clickhouse instances. clickhouse-mcp-ro connects as mcp_readonly and clickhouse-mcp-rw as mcp_readwrite. Both passwords come from the clickstack Vault path. LiteLLM exposes them as the clickhouse_ro and clickhouse_rw MCP servers. Read the argocd/app-configs/litellm-chart/values.yaml entries for which keys reach the read-write instance.

A row filter hides every resolver log row from mcp_readonly and mcp_readwrite. The filter is a databases key under extraUsersConfig in argocd/app-configs/clickstack-chart/values.yaml. It removes each row of default.otel_logs with a ServiceName of dns-resolver or dns-resolver-public. The two values mark every line of the two recursor processes on the two DNS resolver hosts: the query rows of the reader, and also the daemon lines of pdns-recursor@main.service and pdns-recursor@public.service. dns_otel_recursor_log_service_names in ansible/roles/dns-otel-collector/defaults/main.yml sets them. The filter must name every value of the Ansible map, because nothing compares the two. The filter does not name blocky and blocky-public, the rows of the client tier until 2026-10-14. The MCP users can read those rows until the 14-day TTL of otel_logs removes the last of them, about 2026-10-30.

An agent on clickhouse_ro or clickhouse_rw therefore reads no resolver journal line. It reads resolver health from the metrics, for example dns_resolver_errors_total under ServiceName dns-resolver or dns-resolver-public.

Every other ClickHouse user reads every row. The HyperDX Logs source shows the resolver rows next to the other services. The HyperDX MCP (the clickstack tools) reads ClickHouse as app, so an agent with that MCP reads every resolver row, client addresses included. Issue #2378 tracks the access paths that bypass the filter. The filter covers default.otel_logs only. The metrics carry no data for each client, and default.otel_logs_kv_rollup_15m holds no attribute values.

dns-control connects to ClickHouse as dns_control_reader. The user has SELECT on default.otel_logs, default.otel_metrics_sum and default.otel_metrics_gauge only. The row filter does not apply to it, because the dns-control Query log screen shows the resolver rows. The hyperdx_reader profile limits each of its queries to 2 GiB of memory. The clickstack-default network policy admits the dns-control namespace on port 8123.

Vault path fzymgc-house/cluster/clickstack, read by the clickstack ExternalSecret:

Key Purpose
admin_password ClickHouse admin password
mongo_admin_password MongoDB admin password
mongo_replicaset_keyfile MongoDB internal replica-set auth
hyperdx_bootstrap_secret HyperDX session-signing key, delivered to the app as EXPRESS_SESSION_SECRET. Rotating it signs every user out once. Reloader restarts the app when the Secret changes
otel_ingest_api_key The OTLP ingest bearer token. It is the HyperDX team key, and it is the only key the otlp/hyperdx receiver validates. Every producer and every external collector reads it
mcp_readonly_password, mcp_readwrite_password The two ClickHouse users the MCP servers connect as
schema_admin_password The user the schema bootstrap Job connects as
dns_control_reader_password The ClickHouse user dns-control connects as
app_password The password of the ClickHouse user app. HyperDX and the watchdog CronJob connect as app. HyperDX keeps its copy in MongoDB, so a rotation also sets the HyperDX connection (step 5 of the rotation)
otelcollector_password The password of the ClickHouse user otelcollector. The ClickStack collector and Vector write rows as otelcollector. The otel-ingest-token ExternalSecret in the vector namespace also reads this key
dns_query_archiver_password The ClickHouse user the DNS query archive connects as

Two more paths belong to the stack. fzymgc-house/cluster/clickstack/oidc holds hyperdx_oidc_client_secret, which tf/keycloak/clickstack.tf writes. Do not put it by hand. fzymgc-house/cluster/clickstack-watchdog holds token, the uptime-kuma push token, which tf/uptime-kuma/clickstack-watchdog.tf writes.

The three external collector roles and the six in-cluster producers read otel_ingest_api_key from the same path. The Vault policies clickstack-hyperdx, clickstack-otel-collector, clickstack-clickhouse, otel-collector-nas, otel-collector-nas-support and otel-collector-dns grant that path.

The ClickHouse user default has no password and holds GRANT ALL. It accepts connections only from the loopback addresses of its own pod. clickhouse-client through kubectl exec connects as default, and a request through Traefik as default gets Authentication failed. If you need an administrative session from outside the pod, connect as schema_admin.

Run from a workstation with a Vault token. vault kv put replaces every key at the path, so run it only on a path that does not exist yet:

Terminal window
vault kv put secret/fzymgc-house/cluster/clickstack \
admin_password="$(openssl rand -base64 32)" \
mongo_admin_password="$(openssl rand -base64 32)" \
mongo_replicaset_keyfile="$(openssl rand -base64 756)" \
hyperdx_bootstrap_secret="$(openssl rand -base64 32)" \
mcp_readonly_password="$(openssl rand -base64 32)" \
mcp_readwrite_password="$(openssl rand -base64 32)" \
schema_admin_password="$(openssl rand -base64 32)" \
dns_control_reader_password="$(openssl rand -base64 32)" \
app_password="$(openssl rand -base64 32)" \
otelcollector_password="$(openssl rand -base64 32)" \
dns_query_archiver_password="$(openssl rand -base64 32)"

If the path exists and has no dns_control_reader_password, seed only that key. vault kv patch keeps the other keys. Seed it before the change that adds the key to the ExternalSecret merges, because a ClickHouse pod does not start when a secretKeyRef key is missing:

Terminal window
printf '%s' "$(openssl rand -base64 32)" | vault kv patch secret/fzymgc-house/cluster/clickstack dns_control_reader_password=-

If ClickStack already runs and the path has no app_password or otelcollector_password, seed the value that each user has now. That is hyperdx for app and otelcollectorpass for otelcollector, the chart defaults. The next sync then changes no password. To change a value later, follow Rotate a ClickHouse password.

If the path has no dns_query_archiver_password, seed a new value before the change that adds the key merges. The ClickHouse pods and the CronJob dns-query-archive read it:

Terminal window
printf '%s' "$(openssl rand -base64 32)" | vault kv patch secret/fzymgc-house/cluster/clickstack dns_query_archiver_password=-

The ingest key cannot be seeded. HyperDX generates its team key on first boot. When HyperDX is up, capture it into Vault:

Terminal window
key=$(kubectl exec -n clickstack deploy/cs-otel-collector -- \
sh -c "grep -A2 'bearertokenauth/hyperdx' /etc/otel/supervisor-data/effective.yaml \
| grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | head -1")
[ -n "$key" ] || { echo "ERROR: no HyperDX key found in effective.yaml"; exit 1; }
# Pipe through stdin (key=-) so that the token stays out of the shell history and ps.
printf '%s' "$key" | vault kv patch secret/fzymgc-house/cluster/clickstack otel_ingest_api_key=-

Make sure that the write landed: vault kv get secret/fzymgc-house/cluster/clickstack | rg created_time.

HyperDX is the source of truth for the ingest key. The collector validates the key HyperDX hands it, so Vault must follow HyperDX.

  1. In the HyperDX UI, open Team Settings, then API Keys, and regenerate the team ingestion key. The collector picks up the new key by itself.

  2. Update Vault to match:

    Terminal window
    vault kv patch secret/fzymgc-house/cluster/clickstack otel_ingest_api_key="<new-hyperdx-team-key>"
  3. ESO refreshes the in-cluster consumers within their refresh interval. To force the scraper collectors now:

    Terminal window
    kubectl annotate externalsecret otel-ingest-token -n otel-scraper force-sync=$(date +%s) --overwrite
    kubectl rollout restart deployment/otel-scrape-opentelemetry-collector -n otel-scraper
    kubectl rollout restart daemonset/otel-node-opentelemetry-collector-agent -n otel-scraper

    The litellm, agent-memory, headroom-agents and hl-assets ExternalSecrets refresh the same way.

  4. Converge the three external collector roles, because each one reads the key at deploy time: nas-otel-collector, nas-support-otel-collector and dns-otel-collector.

  5. Make sure that rows still arrive:

    SELECT count() FROM default.otel_metrics_gauge WHERE TimeUnix > now() - INTERVAL 10 MINUTE;

This procedure changes app_password or otelcollector_password. Run it from a LAN host with a Vault token and the fzymgc-house kube context. Set the Vault key and the ClickHouse user first:

Terminal window
key=otelcollector_password user=otelcollector field=CLICKHOUSE_PASSWORD
# or: key=app_password user=app field=CLICKHOUSE_APP_PASSWORD

Logins fail from step 2 until the end of step 3, because the consumers and ClickHouse hold different values. The collector holds rows in a 2Gi disk queue, and Vector holds rows in a 5Gi disk buffer. Do all six steps in one session.

  1. Write a new value to Vault:

    Terminal window
    printf '%s' "$(openssl rand -base64 32)" | vault kv patch secret/fzymgc-house/cluster/clickstack "$key=-"
  2. Sync the two ExternalSecrets now, and wait until clickstack-secret holds the new value. For otelcollector, also wait for otel-ingest-token. The annotate command returns before ESO reads Vault, so a SecretSynced status alone does not prove the new value. The loops compare SHA-256 hashes, so no value reaches the terminal. If a loop runs for more than a minute, read the status of that ExternalSecret:

    Terminal window
    kubectl --context fzymgc-house -n clickstack annotate externalsecret clickstack-secret force-sync="$(date +%s)" --overwrite
    kubectl --context fzymgc-house -n vector annotate externalsecret otel-ingest-token force-sync="$(date +%s)" --overwrite
    want=$(printf '%s' "$(vault kv get -field="$key" secret/fzymgc-house/cluster/clickstack)" | shasum -a 256)
    have() {
    printf '%s' "$(kubectl --context fzymgc-house -n "$1" get secret "$2" -o jsonpath="{.data.$3}" | base64 -d)" | shasum -a 256
    }
    until [ "$(have clickstack clickstack-secret "$field")" = "$want" ]; do sleep 5; done
    # otelcollector only:
    until [ "$(have vector otel-ingest-token CLICKHOUSE_PASSWORD)" = "$want" ]; do sleep 5; done
  3. Delete the ClickHouse pods one at a time, 0-1-0 first. The operator recreates each pod, and the new process reads the new value from clickstack-secret:

    Terminal window
    for pod in cs-clickstack-clickhouse-clickhouse-0-1-0 cs-clickstack-clickhouse-clickhouse-0-0-0; do
    kubectl --context fzymgc-house -n clickstack delete pod "$pod"
    kubectl --context fzymgc-house -n clickstack wait --for=create "pod/$pod" --timeout=5m
    kubectl --context fzymgc-house -n clickstack wait --for=condition=Ready "pod/$pod" --timeout=15m
    done
  4. Make sure that Reloader restarted the consumers after step 2. Reloader restarts cs-clickstack-app and cs-otel-collector when clickstack-secret changes. It restarts the vector DaemonSet when otel-ingest-token changes, which happens for otelcollector only. The annotation names the Secret, and its observedAt is a Unix time after step 2:

    Terminal window
    for w in clickstack/deployment/cs-clickstack-app clickstack/deployment/cs-otel-collector vector/daemonset/vector; do
    IFS=/ read -r ns kind name <<<"$w"
    kubectl --context fzymgc-house -n "$ns" get "$kind" "$name" \
    -o jsonpath='{.spec.template.metadata.annotations.reloader\.stakater\.com/last-reloaded-from}{"\n"}'
    kubectl --context fzymgc-house -n "$ns" rollout status "$kind/$name"
    done

    The clickstack-watchdog CronJob reads the Secret at each run, so it needs no restart.

  5. For app only: set the new value on the HyperDX connection Local ClickHouse. HyperDX keeps the connection in MongoDB, and it reads connections.json only while its team has no connection. Until this step runs, HyperDX dashboards, alerts and the HyperDX MCP tools read nothing. The commands use the HyperDX API with your personal API access key, which HyperDX shows in Team Settings, API Keys. The frontend proxy removes one /api, so the URL has two:

    Terminal window
    read -rs -p 'HyperDX personal API access key: ' hdx_key; echo
    api=https://hyperdx.fzymgc.house/api/api/v2/connections
    hdr() { printf 'header = "Authorization: Bearer %s"\nheader = "Content-Type: application/json"\n' "$hdx_key"; }
    conn=$(curl -sSf --config <(hdr) "$api" | jq -c '.data[] | select(.name == "Local ClickHouse")')
    jq -n --argjson c "$conn" --rawfile pw <(vault kv get -field=app_password secret/fzymgc-house/cluster/clickstack) \
    '{name: $c.name, host: $c.host, username: $c.username, password: ($pw | rtrimstr("\n"))}' \
    | curl -sSf --config <(hdr) -X PUT --data-binary @- "$api/$(jq -r .id <<<"$conn")" \
    | jq -c '.data | {name, updatedAt}'

    The HyperDX UI makes the same change: Team Settings, then Connections, then Local ClickHouse.

  6. Make sure that the old value fails and the new value works. Run this before any other write to the Vault path, because the old value comes from the previous version of the secret:

    Terminal window
    v=$(vault kv metadata get -format=json secret/fzymgc-house/cluster/clickstack | jq '.data.current_version')
    old=$(vault kv get -version=$((v - 1)) -field="$key" secret/fzymgc-house/cluster/clickstack)
    new=$(vault kv get -field="$key" secret/fzymgc-house/cluster/clickstack)
    printf 'user = "%s:%s"\n' "$user" "$old" | curl -sS --config - 'https://ch.fzymgc.house/?query=SELECT%201'
    # Code: 516 ... Authentication failed
    printf 'user = "%s:%s"\n' "$user" "$new" | curl -sS --config - 'https://ch.fzymgc.house/?query=SELECT%201'
    # 1

    For otelcollector, wait fifteen minutes after the last pod restart. Then make sure that new rows arrive from both writers. Vector rows carry log.file.path, and rows through the collector do not. The output must have two rows, 1 for Vector and 0 for the collector, and each count must be above zero. A writer with no rows has no output row:

    Terminal window
    kubectl --context fzymgc-house -n clickstack exec -i cs-clickstack-clickhouse-clickhouse-0-0-0 -c clickhouse-server -- \
    sh -c 'clickhouse-client --user schema_admin --password "$SCHEMA_ADMIN_PASSWORD"' <<'SQL'
    SELECT LogAttributes['log.file.path'] != '' AS from_vector, count()
    FROM default.otel_logs
    WHERE Timestamp > now() - INTERVAL 5 MINUTE
    GROUP BY from_vector
    SQL

    For app, make sure that the HyperDX Logs view shows rows from the last five minutes. Then run the watchdog by hand. The Job must complete, and the uptime-kuma push monitor must read up:

    Terminal window
    job="watchdog-rotation-$(date +%s)"
    kubectl --context fzymgc-house -n clickstack create job --from=cronjob/clickstack-watchdog "$job"
    kubectl --context fzymgc-house -n clickstack wait --for=condition=complete "job/$job" --timeout=5m

cs-otel-collector runs the ClickStack collector image, which is an OpAMP supervisor. HyperDX hands it an effective configuration at runtime, and that configuration replaces the chart’s bootstrap file. The chart’s otelCollector values and the bootstrap relay.yaml therefore do not decide what the collector does. Read the live configuration instead:

Terminal window
kubectl exec -n clickstack deploy/cs-otel-collector -- cat /etc/otel/supervisor-data/effective.yaml

The repository’s own additions are in the ConfigMap argocd/app-configs/clickstack/otel-collector-custom-config.yaml. It gives the ClickHouse exporters a persistent sending queue on disk, sets the batch size, and redacts sk- key material from span attributes, log bodies and log attributes before export.

The effective configuration wires the traces pipeline: the otlp/hyperdx receiver on :4317 and :4318 with bearertokenauth/hyperdx, then the clickhouse exporter into default.otel_traces. An empty otel_traces means that no producer is sending traces. Most logs come from Vector straight into ClickHouse, and that path carries no trace. A producer that sends its logs over OTLP, such as dns-control, writes the trace id and the span id of each record into TraceId and SpanId of default.otel_logs, so a log row joins its trace.

A producer must send OTLP with the ingest key as the authorization header:

From Endpoint Auth header
In-cluster cs-otel-collector.clickstack.svc.cluster.local:4317 (gRPC) / :4318 (HTTP POST /v1/traces) authorization: <ingest-key>
External otel-gateway.fzymgc.house:443 (gRPC, mTLS client certificate) authorization: <ingest-key>

Without the header the receiver returns 401 missing or empty authorization header and drops every span.

LiteLLM sends traces and four GenAI metric histograms: gen_ai.client.operation.duration, gen_ai.client.response.duration, gen_ai.client.token.usage and gen_ai.usage.cost. On the metrics, gen_ai.request.model is the upstream model, not the lane alias, and per-key attribution is metadata.user_api_key_alias.

This proves the pipeline end to end without an instrumented application:

Terminal window
# Read the live team key (the full UUID pattern, not a loose hex match).
TOKEN=$(kubectl exec -n clickstack deploy/cs-otel-collector -- \
sh -c "grep -A2 'bearertokenauth/hyperdx' /etc/otel/supervisor-data/effective.yaml \
| grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | head -1")
# Stop here if the read is empty. An empty header makes the 401 below look like a broken pipeline.
[ -n "$TOKEN" ] || { echo "ERROR: no HyperDX key found in effective.yaml (layout changed?)"; exit 1; }
kubectl port-forward -n clickstack svc/cs-otel-collector 14318:4318 &
NOW=$(date +%s)000000000
# Pass the token through a 0600 header file so that it stays out of the process arguments.
HDR=$(mktemp); chmod 600 "$HDR"; printf 'authorization: %s\n' "$TOKEN" > "$HDR"
curl -s -w '%{http_code}\n' -X POST http://127.0.0.1:14318/v1/traces \
-H 'Content-Type: application/json' -H @"$HDR" \
--data "{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"trace-probe\"}}]},\"scopeSpans\":[{\"spans\":[{\"traceId\":\"5b8efff798038103d269b633813fc60c\",\"spanId\":\"eee19b7ec3c1b174\",\"name\":\"probe\",\"kind\":1,\"startTimeUnixNano\":\"$NOW\",\"endTimeUnixNano\":\"$NOW\"}]}]}]}"
rm -f "$HDR"
# Expect HTTP 200 {"partialSuccess":{}}, then:
# SELECT count() FROM default.otel_traces WHERE ServiceName='trace-probe'; -- > 0

HyperDX needs MongoDB for session state. When Mongo is unavailable, HyperDX returns 5xx. MongoDB runs as a 3-member replica set with a PodDisruptionBudget of minAvailable: 2, so it tolerates the loss of one member. Investigate:

Terminal window
kubectl -n clickstack get pods -o wide | rg mongodb
kubectl -n clickstack logs <mongodb-pod> --all-containers --tail=50

clickstack-watchdog (CronJob, every 5 min, argocd/app-configs/clickstack/watchdog-cronjob.yaml) runs two health checks and pushes an uptime-kuma heartbeat only if both pass. Its two failure modes are split, so the Job’s exit code and the heartbeat mean different things. Read this table before drawing any conclusion from kubectl get jobs -n clickstack.

Case Exit code Heartbeat sent? Which alert speaks Where the reason is readable
ClickStack unhealthy: ClickHouse unreachable, ingest stale (newest metric over 300 s old), engine drift, or a malformed query result 0 (the Job succeeds) no the clickstack-watchdog uptime-kuma monitor goes DOWN after about 2 missed beats pod stdout, HEALTH CHECK FAILED: <reason>
Push delivery broken: endpoint error, Service unreachable, token rejected 1 (the Job fails) no ClickStack watchdog Job failures (push delivery), a HyperDX tile alert on kube_job_failed, to Pushover (cluster-infra) pod stdout, PUSH DELIVERY FAILED with the wget exit status
Both fine 0 yes none pod stdout, stale=…s drift=… then the push
The Job died before or during the checks: activeDeadlineSeconds: 120 exceeded (Job condition reason DeadlineExceeded), OOMKilled, an image-pull failure, or node eviction the Job fails no both: the tile alert fires and the uptime-kuma monitor goes DOWN the Job condition reason, kubectl get job <name> -n clickstack -o jsonpath='{.status.conditions[*].reason}'

Row 4 is the one that bites mid-incident. If ClickHouse is alive but saturated and the freshness query blocks past the 120 s deadline, the Job is killed with DeadlineExceeded, no heartbeat is sent, and both alerts speak. Reading row 2’s meaning into that page points you away from the fault. Read the Job condition reason first.

A green Job is not a health verdict. Rows 1 and 3 both exit 0. The ClickStack health signal is heartbeat absence, a dead-man’s switch by design, because an alert that lives inside the observability stack cannot fire when the stack is dark. The heartbeat carries no failing check, because the push happens only on success. Triage reads the pod’s stdout in default.otel_logs:

Terminal window
kubectl exec -n clickstack "$(kubectl get pod -n clickstack \
-l clickhouse.com/role=clickhouse-server -o name | head -1 | cut -d/ -f2)" \
-- clickhouse-client --query "SELECT Timestamp, Body FROM default.otel_logs \
WHERE ResourceAttributes['k8s.pod.name'] LIKE 'clickstack-watchdog-%' \
ORDER BY Timestamp DESC LIMIT 20"

The alert fires on stored kube_job_failed samples over a 1 h lookback, not on live object state. Deleting a failed Job clears the alert only after a full bucket passes, so a re-read of the tile right after the cleanup still shows red. The two eviction mechanisms do not cover for each other. ttlSecondsAfterFinished (300 s) binds only Jobs created after it was set, and failedJobsHistoryLimit (3) evicts an old failed Job only when a newer failed Job arrives. A failed Job that predates the TTL is reaped by neither, so delete it by hand.

Rules in: the uptime-kuma push endpoint returning an error, the internal Service uptime-kuma.uptime-kuma.svc.cluster.local unreachable, the push token rejected, and every Job-level failure the script never sees (row 4 above).

Rules out: nothing unconditionally. A health failure withholds the heartbeat and exits 0, so a script that ran to completion and still failed did fail on the push. A Job killed before or during the checks says nothing about ingest health either way.

Recover with the throwaway pods below. Take Pod B: it keeps the response body, and the body is what turns a bare non-2xx into a named cause. uptime-kuma maps every internal exception to 404 and names the reason only in msg, which the watchdog’s own BusyBox wget discards. Then make sure that the clickstack-watchdog monitor is present and active in uptime-kuma, that the clickstack-watchdog ExternalSecret is Ready and SecretSynced, and that the ping value the script sends is a non-negative integer.

The failure line carries only wget’s exit status, never the URL or the token, because pod stdout is retained 14 days in a queryable store. Recovering more than that means re-running the push by hand, and the client you re-run it with decides what you get back.

The watchdog image is clickhouse-server:*-alpine, so its only HTTP client is BusyBox wget. On a non-2xx status BusyBox wget writes the status line to stderr and exits without reading the body. uptime-kuma answers {"ok":false,"msg":"<the real reason>"}, and its push route maps every internal exception to HTTP 404, so a 404 from that endpoint does not mean “bad token”. msg is the only field that names the cause, and it has to be caught live by a client that keeps it. uptime-kuma’s own log and default.otel_logs do not carry it.

The token is never typed. Both pods read PUSH_TOKEN from a secretKeyRef on Secret clickstack-watchdog, key token, the way the CronJob does, and reference it only as ${PUSH_TOKEN} from inside the pod. Do not paste the token into a kubectl run argument or into a URL in your shell. A pasted token survives in the shell history and in the Pod object.

ndots: "1" is mandatory in both pods. musl’s getaddrinfo short-circuits on the cluster-appended fzymgc.house search domain, which answers NODATA for unknown subdomains, so under the cluster default ndots:5 the push host never resolves and you get wget: bad address. The dnsConfig comment in argocd/app-configs/clickstack/watchdog-cronjob.yaml carries the full reason. A bare kubectl run inherits the cluster default and reproduces that failure.

Both pods are throwaway (--rm --restart=Never) and carry no service account token, mirroring the CronJob’s automountServiceAccountToken: false. They exist because ttlSecondsAfterFinished deletes finished watchdog Jobs, so there is usually no watchdog pod left to kubectl exec into.

Pod A reproduces the watchdog’s exact client behaviour, including the status line. Use it to confirm what the watchdog itself saw. It does not recover msg. Its image must equal the image in argocd/app-configs/clickstack/kustomization.yaml (images:), because that equality is its claim to parity. Nothing asserts the equality, so when that image moves, this recipe moves with it in the same change.

Terminal window
kubectl run -n clickstack watchdog-probe --rm -it --restart=Never \
--image=clickhouse/clickhouse-server:26.7.5.10-alpine \
--overrides='{
"spec": {
"automountServiceAccountToken": false,
"dnsConfig": {"options": [{"name": "ndots", "value": "1"}]},
"containers": [{
"name": "watchdog-probe",
"image": "clickhouse/clickhouse-server:26.7.5.10-alpine",
"stdin": true,
"tty": true,
"command": ["sh"],
"env": [{"name": "PUSH_TOKEN", "valueFrom": {"secretKeyRef":
{"name": "clickstack-watchdog", "key": "token"}}}]
}]
}
}'

Then, from inside that pod:

Terminal window
wget -S -O- "http://uptime-kuma.uptime-kuma.svc.cluster.local:3001/api/push/${PUSH_TOKEN}?status=up&msg=manual&ping=1"

Pod B keeps the body of a non-2xx response. Run it on the next occurrence and capture msg.

Terminal window
kubectl run -n clickstack push-body-probe --rm -it --restart=Never \
--image=curlimages/curl:8.21.0 \
--overrides='{
"spec": {
"automountServiceAccountToken": false,
"dnsConfig": {"options": [{"name": "ndots", "value": "1"}]},
"containers": [{
"name": "push-body-probe",
"image": "curlimages/curl:8.21.0",
"stdin": true,
"tty": true,
"command": ["sh"],
"env": [{"name": "PUSH_TOKEN", "valueFrom": {"secretKeyRef":
{"name": "clickstack-watchdog", "key": "token"}}}]
}]
}
}'

Then, from inside that pod. -i prints the status line and the headers, and curl keeps the body on a non-2xx status, so msg arrives with them:

Terminal window
curl -sS -i "http://uptime-kuma.uptime-kuma.svc.cluster.local:3001/api/push/${PUSH_TOKEN}?status=up&msg=manual&ping=1"

Traefik terminates every TLS endpoint: hyperdx.fzymgc.house (Secret hyperdx-tls), ch.fzymgc.house and ch-native.fzymgc.house (Secret ch-tls), and otel-gateway.fzymgc.house (Secret otel-gateway-tls, client CA in otel-gateway-mtls). Traefik picks up a renewed Secret by itself, so certificate rotation needs no restart. ClickHouse and the collector serve no TLS.

Data Protection
ClickHouse tables Two ReplicatedMergeTree replicas. The native backups below hold only the tables with no TTL. The data volumes (longhorn-single) carry only the Longhorn fstrim job, so Longhorn takes no backup of them, and Velero skips them by resource policy
MongoDB volumes (longhorn) The default Velero schedule (the clickstack namespace is not excluded) and Longhorn’s daily-backup. No fsync hooks are configured, so both are crash-consistent
Keeper volumes (longhorn) The same as MongoDB

MongoDB holds the HyperDX state that ClickHouse does not: users, teams, sources, dashboards, saved searches and alerts. Read a Longhorn volume’s newest backup:

Terminal window
kubectl get backups.longhorn.io -n longhorn-system -l backup-volume=<pv-name> \
-o jsonpath='{range .items[*]}{.status.backupCreatedAt} {.status.state}{"\n"}{end}' | sort | tail -1

The CronJob clickhouse-backup (argocd/app-configs/clickstack/clickhouse-backup.yaml) runs BACKUP at 05:00 America/New_York, the node zone that CronJobs without timeZone use, as schema_admin. It backs up every Replicated* table in default that has no TTL to the B2 bucket fzymgc-k8s-backups under clickhouse/. Today that is only dns_query_daily. The tables with a TTL hold telemetry that expires after 14 or 30 days, and only the two replicas hold it:

Item Value
Full backup clickhouse/full-<UTC timestamp>, on the first run of each ISO week (Monday start)
Incremental backup clickhouse/incr-<UTC timestamp>, on every other run, against the newest full backup
Retention The bucket’s lifecycle rule on clickhouse/ hides each file 15 days after upload and deletes it a day later. Nothing in the cluster deletes a backup
Credentials The s3 endpoint section of the chart values, from Secret clickhouse-backup-b2 (Vault cluster/clickstack/b2). No SQL statement names a key
Upload Each S3 request has 20 retries, so a stuck upload fails the backup instead of retrying for a day. B2 stores a file on one vault, a group of B2 storage servers. A multipart upload stays on its vault, so a vault that refuses writes fails every retry. B2 picks a new vault for each PutObject
Not in the backup Every table with a TTL: the OpenTelemetry tables, their 15-minute rollups and hyperdx_sessions. Also the materialized views and the metrics_ts TimeSeries table. bootstrap-schema-configmap.yaml recreates them, and dns-query-schema.yaml recreates dns_query_daily_mv and dns_query_daily_resolver_mv. The backup holds dns_query_daily, because its engine is Replicated* and it has no TTL
Alert ClickHouse backup FAILED, when a Job of the CronJob failed in the last day

The 15-day expiry always leaves the newest full backup and its incremental backups, because a new full backup starts every week.

List the backups:

Terminal window
b2 ls b2://fzymgc-k8s-backups/clickhouse/

Run a backup now:

Terminal window
kubectl create job -n clickstack --from=cronjob/clickhouse-backup clickhouse-backup-manual
kubectl -n clickstack logs -f job/clickhouse-backup-manual

The alert fires when a Job of clickhouse-backup failed. The Job and its log stay for one day. kubectl get jobs -n clickstack gives the name of the Job.

  1. Read the log of the Job: kubectl -n clickstack logs job/<name>.

  2. Read the reason of the failure: kubectl get job -n clickstack <name> -o jsonpath='{.status.conditions[*].reason}'.

  3. If the reason is DeadlineExceeded, find the BACKUP that still runs. The server does not stop a BACKUP when its client disconnects:

    Terminal window
    for pod in $(kubectl get pod -n clickstack -l clickhouse.com/role=clickhouse-server -o name); do
    kubectl exec -n clickstack "$pod" -- clickhouse-client --query \
    "SELECT hostName(), query_id, round(elapsed) FROM system.processes WHERE query LIKE 'BACKUP%'"
    done
  4. Stop that BACKUP on the pod that runs it. Do not leave it to run. At shutdown, the server waits for a running BACKUP, and Kubernetes kills the server after 30 seconds:

    Terminal window
    kubectl exec -n clickstack <pod> -- clickhouse-client --query "KILL QUERY WHERE query_id = '<query_id>'"
  5. List the uploads that B2 did not finish: b2 file large unfinished list b2://fzymgc-k8s-backups. The _v<number>_ field of a file ID names the vault of the upload.

  6. If all unfinished uploads under clickhouse/ name one vault, send their file IDs to Backblaze support. In October 2026, vault v0001059 refused writes and stalled every multipart upload on it (Backblaze request 1298650).

  7. When the cause is gone, run a backup now with the commands above.

Restore into a new database, then compare, then move the data you need. A restore of an incremental backup reads its full backup too. The tables use the Keeper path /clickhouse/tables/{uuid}/{shard}, and a restore under a new name gets new UUIDs, so it does not collide with the live tables.

-- Run as schema_admin on one replica. <name> is a full-... or incr-... directory.
CREATE DATABASE restore_check;
RESTORE TABLE default.dns_query_daily AS restore_check.dns_query_daily
FROM S3('https://s3.us-west-000.backblazeb2.com/fzymgc-k8s-backups/clickhouse/<name>');
SELECT count(), min(day), max(day) FROM restore_check.dns_query_daily;

Drop the scratch database when you finish: DROP DATABASE restore_check SYNC.

Dashboards and alerts are GitOps state. argocd/app-configs/clickstack-alerts/bootstrap-script.yaml holds bootstrap.js, and a PostSync Job (mongo:8.0) runs it against HyperDX’s MongoDB on every sync. It upserts each dashboard, tile and alert by name, so a merge is the whole deployment. There is no kubectl apply. Alerts deliver to the Pushover (cluster-infra) webhook, whose token comes from argocd/app-configs/clickstack-alerts/external-secret.yaml. tools/hyperdx/dashboards/ holds exported dashboard JSON, and tools/hyperdx/README.md states how it is used.

To add a dashboard or an alert, edit bootstrap.js, open a PR, and after the merge make sure that the object exists in the HyperDX UI. To read the alert intervals in force, read the script.

HyperDX UI search uses Lucene syntax. For direct ClickHouse queries:

Terminal window
kubectl exec -it -n clickstack \
$(kubectl get pod -n clickstack -l clickhouse.com/role=clickhouse-server -o name | head -1) \
-- clickhouse-client
-- Log count last hour
SELECT count(*) FROM otel_logs WHERE Timestamp > now() - INTERVAL 1 HOUR;
-- Metrics count last hour
SELECT count(*) FROM otel_metrics_gauge WHERE TimeUnix > now() - INTERVAL 1 HOUR;
-- Trace span count last hour, by service (0 rows total means that no producer
-- is emitting OTLP traces; the pipeline itself is wired)
SELECT ServiceName, count(*) AS spans
FROM otel_traces WHERE Timestamp > now() - INTERVAL 1 HOUR
GROUP BY ServiceName ORDER BY spans DESC;
-- Top namespaces by log volume
SELECT ResourceAttributes['k8s.namespace.name'] AS namespace,
count(*) AS cnt
FROM otel_logs
WHERE Timestamp > now() - INTERVAL 1 HOUR
GROUP BY namespace
ORDER BY cnt DESC
LIMIT 20;
-- MCP calls by server (LiteLLM relay). LiteLLM relays MCP over one route per
-- registered server and emits ONE server span per call, named for the TEMPLATED
-- route, so counting those spans counts MCP calls. The group-by extracts only
-- the server segment from http.target. Never group on the raw target, which is
-- the concrete request path and can carry key material.
SELECT extract(SpanAttributes['http.target'], '^/([^/]+)/mcp') AS mcp_server,
count(*) AS calls
FROM otel_traces
WHERE ServiceName = 'litellm'
AND SpanName = 'POST /{mcp_server_name}/mcp'
AND Timestamp > now() - INTERVAL 24 HOUR
GROUP BY mcp_server
ORDER BY calls DESC;

The MCP query is also a tile on the LiteLLM — LLM & key usage dashboard. There is no by-method equivalent, because LiteLLM relays the JSON-RPC payload without parsing it, so no MCP method name reaches a span attribute.

nas-support collector verification queries

Section titled “nas-support collector verification queries”

The native otelcol-nas-support collector on the nas-support LXC (192.168.40.32, role nas-support-otel-collector) sends journald logs, host metrics, the backup-freshness file.mtime series and the backup duration and per-source failure gauges. Every row is keyed on ResourceAttributes['host.name']='nas-support', set by the collector’s resource processor.

Open a ClickHouse client against the ClickStack pod:

Terminal window
kubectl exec -it -n clickstack \
$(kubectl get pod -n clickstack -l clickhouse.com/role=clickhouse-server -o name | head -1) \
-- clickhouse-client
-- journald logs arriving from .202 (expect > 0)
SELECT count() AS logs_last_hour
FROM otel_logs
WHERE ResourceAttributes['host.name'] = 'nas-support'
AND Timestamp > now() - INTERVAL 1 HOUR;
-- host metrics arriving from .202 (expect > 0)
SELECT count() AS hostmetric_points
FROM otel_metrics_gauge
WHERE ResourceAttributes['host.name'] = 'nas-support';
-- backup-freshness file.mtime: a recent epoch value, small age.
-- file.mtime is a non-monotonic Sum (epoch seconds), NOT a gauge. It lands in
-- otel_metrics_sum, so query that table, not otel_metrics_gauge.
SELECT MetricName,
Value,
now() - Value AS age_s
FROM otel_metrics_sum
WHERE MetricName = 'file.mtime'
AND ResourceAttributes['host.name'] = 'nas-support'
ORDER BY TimeUnix DESC
LIMIT 1;
-- backup duration and per-source failure gauges from the last run
-- (expect nas.kopia.backup.duration.seconds present; snapshot.failed rows only
-- if a source failed, carrying a 'source' attribute).
SELECT MetricName,
Value,
Attributes['source'] AS source
FROM otel_metrics_gauge
WHERE MetricName LIKE 'nas.kopia%'
AND TimeUnix > now() - INTERVAL 10 MINUTE
ORDER BY TimeUnix DESC;

Smoke checks on .202, over the nas-automation SSH connection:

Terminal window
# Collector process is up
systemctl is-active otelcol-nas-support # -> active
# health_check extension answers on the loopback bind
curl -fsS http://127.0.0.1:13133/ # -> HTTP 200

file.mtime is the one series that is a Sum rather than a gauge. A query against otel_metrics_gauge for it returns zero rows and looks like a broken pipeline.

Two ClickStack tile alerts on the NAS backup — freshness dashboard watch the NAS to B2 Kopia backups. They live in bootstrap-script.yaml and deliver to the cluster-infra Pushover webhook. The staleness signal is not a ClickStack alert. It stays on the host as the kopia-freshness.timer dead-man’s switch, because HyperDX cannot compute a last-success age from a Sum inside an alert. Three signals cover each other’s blind spot:

Signal Threshold and interval Catches Path
NAS Kopia snapshot FAILED above_exclusive 0, 1d a specific source broke, the earliest “a backup failed” signal ClickStack
NAS backup telemetry MISSING (collector dark) below 1, 1h telemetry stopped, the .202 collector or the box is dark ClickStack
NAS Kopia backup STALE (kopia-freshness.timer) over nas_kopia_freshness_max_age_hours (36), hourly backups stopped, needs no telemetry, the primary staleness signal .202 on host

The producer (ansible/roles/nas-kopia-backup/templates/kopia-b2-backup.sh.j2) emits nas.kopia.snapshot.failed = 1 only when a source’s snapshot fails, with a source attribute. A healthy run emits nothing for this metric. There is no 0 sample. The alert therefore fires when a 1 appears in the window, and absence in the window means “no known failure”, not a healthy zero. The alert tile is un-grouped so that it fires on any failed source. To identify which source failed, use the per-source nas-support — backup freshness & health dashboard or the last query above. The failure alert uses the 1d interval, the widest window HyperDX offers, so the once-per-run emit cannot fall between windows. The on-host timer covers the sustained-outage tail after a failed point ages out of the window.

Read the last-success age by hand:

SELECT (toUnixTimestamp(now()) - toUInt64(max(Value))) / 3600 AS hours_since_success
FROM otel_metrics_sum
WHERE MetricName = 'file.mtime' AND ResourceAttributes['host.name'] = 'nas-support';

Make sure that the on-host backstop is armed on .202:

Terminal window
systemctl is-enabled kopia-freshness.timer # -> enabled