Skip to content

Alerting Operations (HyperDX)

Alerting runs on HyperDX tile alerts, metric thresholds against ClickHouse otel_metrics_*, delivered to Pushover through a generic webhook.

Alerts and dashboards must be metric-based. Counting rows in otel_logs to drive a threshold (HyperDX saved_search alerts) is permitted only as a documented last-resort exception when no metric or telemetry signal exists. The criteria and the register of exceptions taken are in The last-resort log exception.

Alert, saved-search, dashboard and webhook definitions are provisioned idempotently by the clickstack-alerts bootstrap script, which runs as an ArgoCD PostSync hook Job on every sync of that Application:

Component Location
Bootstrap script (source of truth) argocd/app-configs/clickstack-alerts/bootstrap-script.yaml
Bootstrap Job (PostSync hook) argocd/app-configs/clickstack-alerts/bootstrap-job.yaml
Pushover token ExternalSecret argocd/app-configs/clickstack-alerts/external-secret.yaml

The script upserts by name (db.webhooks, db.savedsearches, db.alerts), so re-runs converge state rather than duplicating records. Upsert alone never removes or renames a record. A removed or renamed alert stays live in MongoDB until the script’s retirement list deletes it by name, so keep each tombstone in that list.

The script queues every alert document, validates the whole set (the interval enum and the message length gates), and only then writes. A throw during validation leaves db.alerts unchanged.

The three Keycloak saved searches (login failures, password changes, admin configuration changes) exist without an alert on them. They depend on spi-events-listener--jboss-logging--success-level=info in argocd/app-configs/keycloak/keycloak-cr.yaml, which raises Keycloak’s success and admin events from DEBUG so that they reach otel_logs. Keep that option if the searches stay.

All alerts route through a single generic webhook, Pushover (cluster-infra), which posts to https://api.pushover.net/1/messages.json with the cluster-infra Pushover app token. There is no per-domain routing tree. The webhook sends title as {{title}} ({{state}}) and message as {{body}}.

  1. Add a tile-alert block to bootstrap-script.yaml, a metric threshold against otel_metrics_*, following the existing closures in that file. Every alert goes through the queue, so a new helper cannot bypass the gates.
  2. Give the body a Runbook: pointer to the triage section that owns the symptom.
  3. Merge. The PostSync hook re-runs the script. If the gate of the script rejects a document, the Job fails, names the cure it wants, and writes nothing.

Do not add a saved_search alert. If a signal has no metric or telemetry equivalent, it must clear The last-resort log exception below, both halves of the qualifying test, a named owner, a migration trigger and a row in the register, before it is written.

An alert body communicates what is wrong, in as few words as that takes. Say what is observed wrong, add a Runbook: pointer to the triage section, and stop.

Two numbers bound that, and they do different jobs:

  • 1024 characters is Pushover’s refusal threshold, and the only one enforced. Nothing between the alert body and the phone truncates. The body becomes HyperDX’s rendered body, which maps into Pushover’s message. An over-cap body is a page refused, or one that arrives without the instructions it exists to carry.
  • 350 characters is a ceiling to stay well under, never a target to fill. A body approaching it has drifted into being a runbook. It is held by review, not by CI.

Two facts about how a page arrives are why a short body is enough. The webhook already sends the alert’s name and state in the title, ahead of the body. And a Pushover lock-screen preview shows only the leading portion of message. Text past the preview is read only after the phone is unlocked, and by then the operator is better served by opening the runbook.

Mechanism, taxonomy, instructions addressed to whoever next edits the query, and triage beyond the first discriminating command belong in a triage section of this runbook, not on the page.

No body is over the cap today, so KNOWN_OVER_CAP is empty. That map records an over-cap body as debt, not as an exemption. It lives in bootstrap-script.yaml, where the runtime gate reads it. Each number is a ceiling. If a body in the map changes:

  • It stays over the cap: record its new measured length in the same change.
  • It drops to at most 1024 characters: delete its entry in the same change. The gate throws on a stale entry.

A length problem fails the PostSync Job, and that run writes no alert. Measure a new body before you merge it.

The provisioner upserts by name. A re-run converges state, and a rename is a create plus an orphan. Changes reach the cluster by merge and ArgoCD sync only. Do not edit the MongoDB of HyperDX by hand, and do not use kubectl apply. The next sync reverts an out-of-band write.

Retiring an alert needs a tombstone, not a deletion

Section titled “Retiring an alert needs a tombstone, not a deletion”

If you delete the upsert call of an alert, the alert stays. upsert() never reconciles deletions, so the document stays in MongoDB, stays bound to its tile and keeps paging.

Retirement is two edits, and both are required:

  1. Remove the upsert or *TileAlert call, so that the script does not create the alert again.
  2. Add a retireAlert("…", "…") call, so that the script deletes the document already in MongoDB. The second argument is only the wording of the log line.

The retireAlert call is a tombstone. It stays in the script after the alert is gone. On a fresh cluster it does nothing, and on this one it is the only thing that stops a retired alert from coming back. A saved search works the same way: add its name to retiredSavedSearches.

retireAlert() queues the deletion. The script writes nothing to db.alerts until the two-pass gate validated every document (see Alert intervals). After the sync, query db.alerts for the name and expect zero documents.

Re-syncing the alerts Application resets every alert it touches to OK

Section titled “Re-syncing the alerts Application resets every alert it touches to OK”

Every alert closure in bootstrap-script.yaml puts state: "OK" in its document, and upsert() sets the whole document on every run. Each run of the PostSync Job therefore writes state: "OK" back to every alert that the script defines. An alert that the script does not define keeps its state.

The alert fires again at its next evaluation, but the page and the state history across that window are lost. Two rules follow:

  • Do not re-sync the clickstack-alerts Application while you investigate an alert that fires. The sync clears the state you are reading.
  • An ALERT → OK transition after a sync is not a recovery. The provisioner wrote that OK. Read the data, as in Host-collector liveness, to know whether the condition cleared.

Every alert document carries an interval, the evaluation window. HyperDX validates it against a fixed enum on its own write path (AlertIntervalSchema). These are the only values that can appear in bootstrap-script.yaml:

1m 5m 15m 30m 1h 6h 12h 1d

The PostSync Job writes straight to MongoDB, not through the API of the product, so nothing in HyperDX rejects an out-of-enum value. Such an alert still evaluates, because the evaluator parses the string (ms(alert.interval)). The value is still illegal by the schema of the product, and an upstream change can break it with no error. Do not call an out-of-enum alert dormant or failed.

The two-pass gate in bootstrap-script.yaml holds the ALERT_INTERVALS constant and runs the validation pass at sync time, in the PostSync Job. The gate throws. bootstrap-job.yaml sets restartPolicy: OnFailure with backoffLimit: 3, so a throw is a visible Job failure, not a partial apply.

The script validates every document before it writes one. Alert closures call queueAlert() and tombstones call retireAlert(). Both only append to a list in memory. Nothing touches db.alerts until the validation pass ran over the whole list.

upsert() sets the whole document, with state: "OK", as it reaches each one. A gate that validated and wrote in one pass would reset the state of every alert before the one it threw on. With two passes, a rejected run leaves db.alerts as it found it, updatedAt included. Do not merge the two passes.

Write each interval as a quoted literal in its *TileAlert call. Do not use a shared constant, a variable or a concatenation. A reader of the script then sees the interval in force at each call site.

An interval other than 15m is not a defect. The window must suit the emit cadence of the carrier.

Alert Interval Why
nas host collector MISSING (host scraper dark) 15m presence detector; the carrier emits every 30 s
nas-support host collector MISSING (host scraper dark) 15m same
ClickStack gateway log intake MISSING 15m presence detector on the receiver counters of the gateway
ClickStack OTel collector ClickHouse exporter missing 15m presence detector
NAS backup telemetry MISSING (collector dark) 1h see below
NAS Kopia snapshot FAILED 1d the gauge is emitted only for a failed source at the end of a run
DNS control: MCP write burst 15m more than 20 agent applies in 15 minutes is a burst, not a task
DNS control: delete through MCP 5m each delete through MCP pages within one short window
DNS control: MCP refusal burst 15m the same window as the write burst
DNS control: change failed 15m each failed change pages
DNS control: export pending 15m the gauge carries the age, so the window only sets how soon the page arrives
DNS control: lineage drift 30m a finding settles after 5 minutes, so an adopt inside the window pages nobody
DNS control: pdns nodes disagree 15m a copy in flight never settles
DNS control: VIP holder not single 5m a VIP with no holder or two holders breaks resolution at once
DNS control: lineage index stale 15m the index refreshes every 30 s, so 15 minutes holds 30 missed runs
DNS control: observed AAAA sweep stale 1h the sweep runs every 5 minutes, so one hour holds 12 missed runs
DNS control: node worker silent 15m presence detector; each worker sends its gauges every 60 s
DNS control: drain overdue 5m the worker sets the gauge only when a drain outlives its duration
DNS control: server errors 15m a single 5xx is noise, and more than 10 in 15 minutes is a failing surface
Certificate expires in under 7 days: {{{group}}} 1h the threshold is 604800 seconds. The collectors report every 5 minutes, and a certificate loses 7 days in 7 days. The alert has one state for each host and target, and {{{group}}} puts both in the title. See docs/operations/certificates.md

15m gives a 30 s carrier about 30 points per window. A short gap does not page, and a real outage does.

NAS backup telemetry MISSING (collector dark) keeps 1h because its carrier is the file.mtime heartbeat of the backup pipeline, which is written about once a day. A 15m window fires on a healthy box, and the first answer to a noisy alert is a mute.

Each host that runs its own OTel collector has a presence tile and an alert on the Host collectors - liveness dashboard. If the collector or the box goes quiet, everything it ships goes quiet with it.

The host identity of these metrics is ResourceAttributes['host.name'], and only that:

SELECT count() FROM default.otel_metrics_gauge
WHERE ResourceAttributes['host.name'] = 'nas'
AND MetricName = 'system.memory.utilization'
AND Attributes['state'] = 'used'
AND TimeUnix >= now() - INTERVAL 1 HOUR

The scrapers run every 30 s, so a healthy host returns about 120 rows per hour. host.name is case-sensitive.

Do not filter on ServiceName. A resource processor inserts service.name only when a record has none, and some pipelines set their own. A ServiceName filter can under-match on one carrier and match on another. An under-match on a live host looks like a dark host.

Host Tile Alert Carrier Recovery
nas nas-host-present nas host collector MISSING (host scraper dark) system.memory.utilization reprovision the otelcol-nas TrueNAS app: scripts/nas-playbook.sh --tags nas-otel
nas-support nas-support-host-present nas-support host collector MISSING (host scraper dark) system.memory.utilization systemctl reset-failed and restart otelcol-nas-support

nas-support also carries NAS backup telemetry MISSING (collector dark) on a second carrier, the file.mtime heartbeat of the backup pipeline. If both fire, the collector or the box is gone. If only the backup alert fires, the collector runs and the fault is in the backup pipeline or its mount.

Two rules govern this map. If you break either one, a host loses its coverage with no error:

  • Every tile carries its own host.name literal. The Kubernetes node collectors also emit system.memory.utilization, so an unfiltered clause never reaches zero.
  • Never put several hosts in one alert with a groupBy. The empty-bucket branch fires only when the whole bucket is empty, so a grouped alert stays quiet when one host goes dark. Add one alert per filtered tile.

ClickStack gateway log intake MISSING is on the same dashboard but is not a per-host detector. It watches the OTLP receiver of the cluster gateway. It proves that the gateway accepts logs from some source, and it cannot name a host that went dark.

kube-state-metrics series: labels land in Attributes

Section titled “kube-state-metrics series: labels land in Attributes”

A gotcha with one failure mode and no error message. kube-state-metrics series arrive through the prometheus receiver with their labels in Attributes. For kube_node_status_condition that is {condition, node, status}. ResourceAttributes on those same series carries scrape-target metadata only (host.name, server.address, service.name, url.scheme) and contains neither condition nor status.

The nearest-looking analog in bootstrap-script.yaml is rh-operator-avail, and it filters ResourceAttributes, correctly, because it reads an OTel k8s-cluster-receiver gauge (k8s.deployment.available), where the labels do live there. Do not copy that filter onto a kube_* series. A tile whose where names ResourceAttributes['condition'] matches nothing, and a tile that matches nothing looks like a healthy cluster. Nothing goes red, and the alert is dead for as long as nobody checks.

Check attribute placement before writing the where for any new series:

SELECT mapKeys(Attributes) AS a, mapKeys(ResourceAttributes) AS ra, count() FROM default.otel_metrics_gauge WHERE MetricName = '<metric>' AND TimeUnix > now() - INTERVAL 1 HOUR GROUP BY a, ra

The long form of the body Kubernetes node NOT READY pages. The page carries the observation and a pointer here. The triage below is the rest of it.

A NotReady that starts and clears inside one window does not page. The HyperDX chart-query builder reduces each series to last_value per granularity bucket before applying the tile’s aggregation across series. AttributesHash hashes the whole attribute map and is what defines a series, so each (node, condition, status) triple is its own. The tile’s max therefore runs across those per-series last values rather than across the raw samples in the window. It reads end-of-bucket state, not “any sample inside the bucket”. That is a consequence of the 1h interval chosen to tolerate k3s upgrade reboots, not a defect.

The tile reads both statuses, false and unknown, and must not be simplified to status = 'false'. A dead kubelet stops renewing its Node lease, which the control plane records as unknown. false is a live kubelet self-reporting sick. An alert on false alone stays silent through a dead node.

This alert is un-grouped by design and names no node. Use the drill-down to find it:

SELECT DISTINCT Attributes['node'], Attributes['status'] FROM default.otel_metrics_gauge WHERE MetricName = 'kube_node_status_condition' AND Attributes['condition'] = 'Ready' AND Attributes['status'] IN ('false','unknown') AND Value = 1

Recover: kubectl describe node <node>, then the kubelet, then the node’s root filesystem. A disk error on the volume holding /var/lib/{kubelet,longhorn,rancher} can kill all userspace while the kernel stays up, so the box still answers ICMP. Ping is not a health signal.

Triage: Kubernetes node readiness signal INCOMPLETE

Section titled “Triage: Kubernetes node readiness signal INCOMPLETE”

The long form of the body Kubernetes node readiness signal INCOMPLETE pages. The page carries the observation, the hardcoded cluster-size threshold and what to do about it, and a pointer here. The triage below is the rest of it.

This alert rules in two causes, and they are not the same fault. Either a Node object was deleted from the API server while the remaining nodes keep reporting normally, so the node set shrank, or kube-state-metrics is down, unscraped, or its metric was renamed upstream, which empties the bucket. The first is a cluster change. The second is a telemetry failure that says nothing about the cluster. kubectl get nodes separates them in one command.

This alert does not overlap Kubernetes node NOT READY. A dead node keeps emitting its series with status=unknown, so the distinct-node count holds and this alert stays quiet. Kubernetes node NOT READY owns that case.

Drill-down. Compare the returned set against the nodes you expect:

SELECT DISTINCT Attributes['node'] FROM default.otel_metrics_gauge WHERE MetricName = 'kube_node_status_condition' AND Attributes['condition'] = 'Ready'

Recover: kubectl get nodes first. If the node set is intact, the fault is in telemetry, not in the cluster. Check the kube-state-metrics workload, then its scrape job in argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml. If the node set changed, update the threshold in the script. It is the hardcoded cluster size.

The metric-only rule names an escape hatch. This section defines it, so that anyone who claims it can be audited. An undocumented escape hatch invoked repeatedly is the policy repealed by practice.

Both halves are required, not either.

  1. No metric exists for the property. Not “no metric is convenient”. No metric or telemetry signal carries the fact the alert is about. Check otel_metrics_gauge and otel_metrics_sum before claiming this.
  2. Creating one is disproportionate to the signal. “Disproportionate” is measured against the value of the signal, never against effort in the abstract. A cheap metric for a rare event still wins. An expensive metric for a rare event does not.

A failing example: “there is no Keycloak metric” satisfies half 1 and not half 2. A Keycloak event-listener SPI exporting a counter is proportionate work, so the retired Keycloak alerts are a migration to do, not an exception to grant.

Requirement Why
A named owner An exception with no owner is a permanent exception.
A migration trigger, stated as a condition and not a date A date on work nobody is scheduled to do expires quietly and changes nothing. The trigger is the event that makes the metric cheap.
A row in the register below The count has to be visible. That is the point of counting.
The argument, inline in bootstrap-script.yaml Every other alert decision in that file is recorded in its own voice. A granted exception whose argument was deleted is indistinguishable from an oversight.

Live saved_search alerts running under this section:

Alert Owner Migration trigger
Resolver delegated-prefix change (RA-WATCH-PREFIX-CHANGE) DNS serving plane The first time a metrics pipeline exists on the resolver pair for any other reason. The alert then becomes a counter incremented on the detector’s exit-3 path plus a tile alert, and this row leaves the register. The trigger condition holds: otelcol-dns runs a metrics pipeline for the blocky-public alert.
LiteLLM key material in logs (OBS-02) LiteLLM plane The first time a litellm.key_material_seen counter is emitted by the scrubber (a transform statement can increment none). The alert then becomes a tile alert on that counter and this row leaves the register.

Retired alerts are not entries in this register. A retired alert has no live alert and no granted exception. The retired Keycloak, Traefik and oauth2-proxy log-count alerts are the open migration backlog, not exceptions.

The bootstrap Job reads PUSHOVER_TOKEN and PUSHOVER_USER_KEY from the pushover-hyperdx-tokens Secret (an ExternalSecret refreshed from Vault every 15 minutes) and bakes the value into the webhook document in HyperDX’s MongoDB at Job-run time. It is not read live per alert. After rotating the token in Vault:

Terminal window
# Wait for ESO to refresh pushover-hyperdx-tokens (up to 15min), then
# trigger the bootstrap script to re-run by syncing the clickstack-alerts
# Application in ArgoCD (re-fires the PostSync hook Job).

Make sure that the webhook document carries the new token by sending a test page from the HyperDX alert UI after the Job completes.