Alerting Operations (HyperDX)
Alerting runs on HyperDX tile alerts, metric thresholds against ClickHouse otel_metrics_*,
delivered to Pushover through a generic webhook.
Alerts and dashboards must be metric-based. Counting rows in otel_logs to drive a threshold
(HyperDX saved_search alerts) is permitted only as a documented last-resort exception when no
metric or telemetry signal exists. The criteria and the register of exceptions taken are in
The last-resort log exception.
Where alert config lives
Section titled “Where alert config lives”Alert, saved-search, dashboard and webhook definitions are provisioned idempotently by the
clickstack-alerts bootstrap script, which runs as an ArgoCD PostSync hook Job on every sync of
that Application:
| Component | Location |
|---|---|
| Bootstrap script (source of truth) | argocd/app-configs/clickstack-alerts/bootstrap-script.yaml |
| Bootstrap Job (PostSync hook) | argocd/app-configs/clickstack-alerts/bootstrap-job.yaml |
| Pushover token ExternalSecret | argocd/app-configs/clickstack-alerts/external-secret.yaml |
The script upserts by name (db.webhooks, db.savedsearches, db.alerts), so re-runs converge
state rather than duplicating records. Upsert alone never removes or renames a record. A removed
or renamed alert stays live in MongoDB until the script’s retirement list deletes it by name, so
keep each tombstone in that list.
The script queues every alert document, validates the whole set (the interval enum and the
message length gates), and only then writes. A throw during validation leaves db.alerts
unchanged.
The three Keycloak saved searches (login failures, password changes, admin configuration changes)
exist without an alert on them. They depend on
spi-events-listener--jboss-logging--success-level=info in
argocd/app-configs/keycloak/keycloak-cr.yaml, which raises Keycloak’s success and admin events
from DEBUG so that they reach otel_logs. Keep that option if the searches stay.
Delivery
Section titled “Delivery”All alerts route through a single generic webhook, Pushover (cluster-infra), which posts to
https://api.pushover.net/1/messages.json with the cluster-infra Pushover app token. There is
no per-domain routing tree. The webhook sends title as {{title}} ({{state}}) and message as
{{body}}.
Adding a new alert
Section titled “Adding a new alert”- Add a tile-alert block to
bootstrap-script.yaml, a metric threshold againstotel_metrics_*, following the existing closures in that file. Every alert goes through the queue, so a new helper cannot bypass the gates. - Give the body a
Runbook:pointer to the triage section that owns the symptom. - Merge. The
PostSynchook re-runs the script. If the gate of the script rejects a document, the Job fails, names the cure it wants, and writes nothing.
Do not add a saved_search alert. If a signal has no metric or telemetry equivalent, it must
clear The last-resort log exception below, both halves of the
qualifying test, a named owner, a migration trigger and a row in the register, before it is
written.
Keep alert bodies short
Section titled “Keep alert bodies short”An alert body communicates what is wrong, in as few words as that takes. Say what is observed
wrong, add a Runbook: pointer to the triage section, and stop.
Two numbers bound that, and they do different jobs:
- 1024 characters is Pushover’s refusal threshold, and the only one enforced. Nothing between the
alert body and the phone truncates. The body becomes HyperDX’s rendered body, which maps into
Pushover’s
message. An over-cap body is a page refused, or one that arrives without the instructions it exists to carry. - 350 characters is a ceiling to stay well under, never a target to fill. A body approaching it has drifted into being a runbook. It is held by review, not by CI.
Two facts about how a page arrives are why a short body is enough. The webhook already sends the
alert’s name and state in the title, ahead of the body. And a Pushover lock-screen preview shows
only the leading portion of message. Text past the preview is read only after the phone is
unlocked, and by then the operator is better served by opening the runbook.
Mechanism, taxonomy, instructions addressed to whoever next edits the query, and triage beyond the first discriminating command belong in a triage section of this runbook, not on the page.
No body is over the cap today, so KNOWN_OVER_CAP is empty. That map records an over-cap body as
debt, not as an exemption. It lives in bootstrap-script.yaml, where the runtime gate reads it.
Each number is a ceiling. If a body in the map changes:
- It stays over the cap: record its new measured length in the same change.
- It drops to at most 1024 characters: delete its entry in the same change. The gate throws on a stale entry.
A length problem fails the PostSync Job, and that run writes no alert. Measure a new body before you merge it.
Alert lifecycle
Section titled “Alert lifecycle”The provisioner upserts by name. A re-run converges state, and a rename is a create plus an
orphan. Changes reach the cluster by merge and ArgoCD sync only. Do not edit the MongoDB of
HyperDX by hand, and do not use kubectl apply. The next sync reverts an out-of-band write.
Retiring an alert needs a tombstone, not a deletion
Section titled “Retiring an alert needs a tombstone, not a deletion”If you delete the upsert call of an alert, the alert stays. upsert() never reconciles
deletions, so the document stays in MongoDB, stays bound to its tile and keeps paging.
Retirement is two edits, and both are required:
- Remove the upsert or
*TileAlertcall, so that the script does not create the alert again. - Add a
retireAlert("…", "…")call, so that the script deletes the document already in MongoDB. The second argument is only the wording of the log line.
The retireAlert call is a tombstone. It stays in the script after the alert is gone. On a fresh
cluster it does nothing, and on this one it is the only thing that stops a retired alert from
coming back. A saved search works the same way: add its name to retiredSavedSearches.
retireAlert() queues the deletion. The script writes nothing to db.alerts until the two-pass
gate validated every document (see Alert intervals). After the sync, query
db.alerts for the name and expect zero documents.
Re-syncing the alerts Application resets every alert it touches to OK
Section titled “Re-syncing the alerts Application resets every alert it touches to OK”Every alert closure in bootstrap-script.yaml puts state: "OK" in its document, and
upsert() sets the whole document on every run. Each run of the PostSync Job therefore writes
state: "OK" back to every alert that the script defines. An alert that the script does not
define keeps its state.
The alert fires again at its next evaluation, but the page and the state history across that window are lost. Two rules follow:
- Do not re-sync the
clickstack-alertsApplication while you investigate an alert that fires. The sync clears the state you are reading. - An
ALERT → OKtransition after a sync is not a recovery. The provisioner wrote thatOK. Read the data, as in Host-collector liveness, to know whether the condition cleared.
Alert intervals
Section titled “Alert intervals”Every alert document carries an interval, the evaluation window. HyperDX validates it against a
fixed enum on its own write path (AlertIntervalSchema). These are the only values that can
appear in bootstrap-script.yaml:
1m 5m 15m 30m 1h 6h 12h 1dThe PostSync Job writes straight to MongoDB, not through the API of the product, so nothing in
HyperDX rejects an out-of-enum value. Such an alert still evaluates, because the evaluator parses
the string (ms(alert.interval)). The value is still illegal by the schema of the product, and an
upstream change can break it with no error. Do not call an out-of-enum alert dormant or failed.
The runtime gate enforces the enum
Section titled “The runtime gate enforces the enum”The two-pass gate in bootstrap-script.yaml holds the ALERT_INTERVALS constant and runs the
validation pass at sync time, in the PostSync Job. The gate throws. bootstrap-job.yaml sets restartPolicy: OnFailure with
backoffLimit: 3, so a throw is a visible Job failure, not a partial apply.
The gate has two passes
Section titled “The gate has two passes”The script validates every document before it writes one. Alert closures call queueAlert() and
tombstones call retireAlert(). Both only append to a list in memory. Nothing touches
db.alerts until the validation pass ran over the whole list.
upsert() sets the whole document, with state: "OK", as it reaches each one. A gate that
validated and wrote in one pass would reset the state of every alert before the one it threw on.
With two passes, a rejected run leaves db.alerts as it found it, updatedAt included. Do not
merge the two passes.
Intervals are literals at every call site
Section titled “Intervals are literals at every call site”Write each interval as a quoted literal in its *TileAlert call. Do not use a shared constant, a
variable or a concatenation. A reader of the script then sees the interval in force at each call
site.
Per-alert interval map
Section titled “Per-alert interval map”An interval other than 15m is not a defect. The window must suit the emit cadence of the
carrier.
| Alert | Interval | Why |
|---|---|---|
nas host collector MISSING (host scraper dark) |
15m |
presence detector; the carrier emits every 30 s |
nas-support host collector MISSING (host scraper dark) |
15m |
same |
ClickStack gateway log intake MISSING |
15m |
presence detector on the receiver counters of the gateway |
ClickStack OTel collector ClickHouse exporter missing |
15m |
presence detector |
NAS backup telemetry MISSING (collector dark) |
1h |
see below |
NAS Kopia snapshot FAILED |
1d |
the gauge is emitted only for a failed source at the end of a run |
DNS control: MCP write burst |
15m |
more than 20 agent applies in 15 minutes is a burst, not a task |
DNS control: delete through MCP |
5m |
each delete through MCP pages within one short window |
DNS control: MCP refusal burst |
15m |
the same window as the write burst |
DNS control: change failed |
15m |
each failed change pages |
DNS control: export pending |
15m |
the gauge carries the age, so the window only sets how soon the page arrives |
DNS control: lineage drift |
30m |
a finding settles after 5 minutes, so an adopt inside the window pages nobody |
DNS control: pdns nodes disagree |
15m |
a copy in flight never settles |
DNS control: VIP holder not single |
5m |
a VIP with no holder or two holders breaks resolution at once |
DNS control: lineage index stale |
15m |
the index refreshes every 30 s, so 15 minutes holds 30 missed runs |
DNS control: observed AAAA sweep stale |
1h |
the sweep runs every 5 minutes, so one hour holds 12 missed runs |
DNS control: node worker silent |
15m |
presence detector; each worker sends its gauges every 60 s |
DNS control: drain overdue |
5m |
the worker sets the gauge only when a drain outlives its duration |
DNS control: server errors |
15m |
a single 5xx is noise, and more than 10 in 15 minutes is a failing surface |
Certificate expires in under 7 days: {{{group}}} |
1h |
the threshold is 604800 seconds. The collectors report every 5 minutes, and a certificate loses 7 days in 7 days. The alert has one state for each host and target, and {{{group}}} puts both in the title. See docs/operations/certificates.md |
15m gives a 30 s carrier about 30 points per window. A short gap does not page, and a real
outage does.
NAS backup telemetry MISSING (collector dark) keeps 1h because its carrier is the file.mtime
heartbeat of the backup pipeline, which is written about once a day. A 15m window fires on a
healthy box, and the first answer to a noisy alert is a mute.
Host-collector liveness
Section titled “Host-collector liveness”Each host that runs its own OTel collector has a presence tile and an alert on the
Host collectors - liveness dashboard. If the collector or the box goes quiet, everything it
ships goes quiet with it.
The host identity of these metrics is ResourceAttributes['host.name'], and only that:
SELECT count() FROM default.otel_metrics_gaugeWHERE ResourceAttributes['host.name'] = 'nas' AND MetricName = 'system.memory.utilization' AND Attributes['state'] = 'used' AND TimeUnix >= now() - INTERVAL 1 HOURThe scrapers run every 30 s, so a healthy host returns about 120 rows per hour. host.name is
case-sensitive.
Do not filter on ServiceName. A resource processor inserts service.name only when a record has
none, and some pipelines set their own. A ServiceName filter can under-match on one carrier and
match on another. An under-match on a live host looks like a dark host.
| Host | Tile | Alert | Carrier | Recovery |
|---|---|---|---|---|
nas |
nas-host-present |
nas host collector MISSING (host scraper dark) |
system.memory.utilization |
reprovision the otelcol-nas TrueNAS app: scripts/nas-playbook.sh --tags nas-otel |
nas-support |
nas-support-host-present |
nas-support host collector MISSING (host scraper dark) |
system.memory.utilization |
systemctl reset-failed and restart otelcol-nas-support |
nas-support also carries NAS backup telemetry MISSING (collector dark) on a second carrier,
the file.mtime heartbeat of the backup pipeline. If both fire, the collector or the box is gone.
If only the backup alert fires, the collector runs and the fault is in the backup pipeline or its
mount.
Two rules govern this map. If you break either one, a host loses its coverage with no error:
- Every tile carries its own
host.nameliteral. The Kubernetes node collectors also emitsystem.memory.utilization, so an unfiltered clause never reaches zero. - Never put several hosts in one alert with a
groupBy. The empty-bucket branch fires only when the whole bucket is empty, so a grouped alert stays quiet when one host goes dark. Add one alert per filtered tile.
ClickStack gateway log intake MISSING is on the same dashboard but is not a per-host detector.
It watches the OTLP receiver of the cluster gateway. It proves that the gateway accepts logs from
some source, and it cannot name a host that went dark.
kube-state-metrics series: labels land in Attributes
Section titled “kube-state-metrics series: labels land in Attributes”A gotcha with one failure mode and no error message. kube-state-metrics series arrive through the
prometheus receiver with their labels in Attributes. For kube_node_status_condition that is
{condition, node, status}. ResourceAttributes on those same series carries scrape-target
metadata only (host.name, server.address, service.name, url.scheme) and contains neither
condition nor status.
The nearest-looking analog in bootstrap-script.yaml is rh-operator-avail, and it filters
ResourceAttributes, correctly, because it reads an OTel k8s-cluster-receiver gauge
(k8s.deployment.available), where the labels do live there. Do not copy that filter onto a
kube_* series. A tile whose where names ResourceAttributes['condition'] matches nothing,
and a tile that matches nothing looks like a healthy cluster. Nothing goes red, and the alert is
dead for as long as nobody checks.
Check attribute placement before writing the where for any new series:
SELECT mapKeys(Attributes) AS a, mapKeys(ResourceAttributes) AS ra, count() FROM default.otel_metrics_gauge WHERE MetricName = '<metric>' AND TimeUnix > now() - INTERVAL 1 HOUR GROUP BY a, raTriage: Kubernetes node NOT READY
Section titled “Triage: Kubernetes node NOT READY”The long form of the body Kubernetes node NOT READY pages. The page carries the observation and
a pointer here. The triage below is the rest of it.
A NotReady that starts and clears inside one window does not page. The HyperDX chart-query
builder reduces each series to last_value per granularity bucket before applying the tile’s
aggregation across series. AttributesHash hashes the whole attribute map and is what defines a
series, so each (node, condition, status) triple is its own. The tile’s max therefore runs
across those per-series last values rather than across the raw samples in the window. It reads
end-of-bucket state, not “any sample inside the bucket”. That is a consequence of the 1h interval
chosen to tolerate k3s upgrade reboots, not a defect.
The tile reads both statuses, false and unknown, and must not be simplified to
status = 'false'. A dead kubelet stops renewing its Node lease, which the control plane records
as unknown. false is a live kubelet self-reporting sick. An alert on false alone stays
silent through a dead node.
This alert is un-grouped by design and names no node. Use the drill-down to find it:
SELECT DISTINCT Attributes['node'], Attributes['status'] FROM default.otel_metrics_gauge WHERE MetricName = 'kube_node_status_condition' AND Attributes['condition'] = 'Ready' AND Attributes['status'] IN ('false','unknown') AND Value = 1Recover: kubectl describe node <node>, then the kubelet, then the node’s root filesystem. A
disk error on the volume holding /var/lib/{kubelet,longhorn,rancher} can kill all userspace
while the kernel stays up, so the box still answers ICMP. Ping is not a health signal.
Triage: Kubernetes node readiness signal INCOMPLETE
Section titled “Triage: Kubernetes node readiness signal INCOMPLETE”The long form of the body Kubernetes node readiness signal INCOMPLETE pages. The page carries
the observation, the hardcoded cluster-size threshold and what to do about it, and a pointer
here. The triage below is the rest of it.
This alert rules in two causes, and they are not the same fault. Either a Node object was deleted
from the API server while the remaining nodes keep reporting normally, so the node set shrank, or
kube-state-metrics is down, unscraped, or its metric was renamed upstream, which empties the
bucket. The first is a cluster change. The second is a telemetry failure that says nothing about
the cluster. kubectl get nodes separates them in one command.
This alert does not overlap Kubernetes node NOT READY. A dead node keeps emitting its series
with status=unknown, so the distinct-node count holds and this alert stays quiet. Kubernetes node NOT READY owns that case.
Drill-down. Compare the returned set against the nodes you expect:
SELECT DISTINCT Attributes['node'] FROM default.otel_metrics_gauge WHERE MetricName = 'kube_node_status_condition' AND Attributes['condition'] = 'Ready'Recover: kubectl get nodes first. If the node set is intact, the fault is in telemetry, not in
the cluster. Check the kube-state-metrics workload, then its scrape job in
argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml. If the node set
changed, update the threshold in the script. It is the hardcoded cluster size.
The last-resort log exception
Section titled “The last-resort log exception”The metric-only rule names an escape hatch. This section defines it, so that anyone who claims it can be audited. An undocumented escape hatch invoked repeatedly is the policy repealed by practice.
What qualifies
Section titled “What qualifies”Both halves are required, not either.
- No metric exists for the property. Not “no metric is convenient”. No metric or telemetry
signal carries the fact the alert is about. Check
otel_metrics_gaugeandotel_metrics_sumbefore claiming this. - Creating one is disproportionate to the signal. “Disproportionate” is measured against the value of the signal, never against effort in the abstract. A cheap metric for a rare event still wins. An expensive metric for a rare event does not.
A failing example: “there is no Keycloak metric” satisfies half 1 and not half 2. A Keycloak event-listener SPI exporting a counter is proportionate work, so the retired Keycloak alerts are a migration to do, not an exception to grant.
What an exception must carry
Section titled “What an exception must carry”| Requirement | Why |
|---|---|
| A named owner | An exception with no owner is a permanent exception. |
| A migration trigger, stated as a condition and not a date | A date on work nobody is scheduled to do expires quietly and changes nothing. The trigger is the event that makes the metric cheap. |
| A row in the register below | The count has to be visible. That is the point of counting. |
The argument, inline in bootstrap-script.yaml |
Every other alert decision in that file is recorded in its own voice. A granted exception whose argument was deleted is indistinguishable from an oversight. |
The register of exceptions taken
Section titled “The register of exceptions taken”Live saved_search alerts running under this section:
| Alert | Owner | Migration trigger |
|---|---|---|
Resolver delegated-prefix change (RA-WATCH-PREFIX-CHANGE) |
DNS serving plane | The first time a metrics pipeline exists on the resolver pair for any other reason. The alert then becomes a counter incremented on the detector’s exit-3 path plus a tile alert, and this row leaves the register. The trigger condition holds: otelcol-dns runs a metrics pipeline for the blocky-public alert. |
LiteLLM key material in logs (OBS-02) |
LiteLLM plane | The first time a litellm.key_material_seen counter is emitted by the scrubber (a transform statement can increment none). The alert then becomes a tile alert on that counter and this row leaves the register. |
Retired alerts are not entries in this register. A retired alert has no live alert and no granted exception. The retired Keycloak, Traefik and oauth2-proxy log-count alerts are the open migration backlog, not exceptions.
Pushover token rotation
Section titled “Pushover token rotation”The bootstrap Job reads PUSHOVER_TOKEN and PUSHOVER_USER_KEY from the
pushover-hyperdx-tokens Secret (an ExternalSecret refreshed from Vault every 15 minutes) and
bakes the value into the webhook document in HyperDX’s MongoDB at Job-run time. It is not read
live per alert. After rotating the token in Vault:
# Wait for ESO to refresh pushover-hyperdx-tokens (up to 15min), then# trigger the bootstrap script to re-run by syncing the clickstack-alerts# Application in ArgoCD (re-fires the PostSync hook Job).Make sure that the webhook document carries the new token by sending a test page from the HyperDX alert UI after the Job completes.