Alerting Operations (HyperDX)

Alertmanager and kube-prometheus-stack were decommissioned in epic hl-qjh3 (see ADR hl-toju). Alerting runs on HyperDX TILE alerts — metric thresholds against ClickHouse otel_metrics_* — delivered to Pushover via a generic webhook.

Alerts (and dashboards) MUST be metric-based. Counting rows in otel_logs to drive a threshold (HyperDX saved_search / log-count alerts) is an anti-pattern, permitted only as a documented last-resort exception when no metric or telemetry signal exists — the criteria and the register of exceptions taken are in The last-resort log exception. The log-based saved_search alerts this page previously described were retired in PR #1638 (hl-pziv) — they had silently frozen on a Logs-source schema mismatch and never fired. Their metric/telemetry replacements are tracked in hl-qjh3.17.

Where alert config lives

Alert, saved-search, and webhook definitions are provisioned idempotently by the clickstack-alerts bootstrap script, which runs as an ArgoCD PostSync hook Job on every sync of that Application:

Component	Location
Bootstrap script (source of truth)	argocd/app-configs/clickstack-alerts/bootstrap-script.yaml
Bootstrap Job (PostSync hook)	argocd/app-configs/clickstack-alerts/bootstrap-job.yaml
Pushover token ExternalSecret	argocd/app-configs/clickstack-alerts/external-secret.yaml

The script upserts by name (db.webhooks, db.savedsearches, db.alerts) so re-runs converge state rather than duplicating records. Deleting or renaming an alert requires an explicit deleteOne/updateOne — upsert alone never removes or renames a record.

Delivery

All migrated alerts route through a single generic webhook, Pushover (cluster-infra), which posts to https://api.pushover.net/1/messages.json using the cluster-infra Pushover app token. There is no per-domain routing tree like Alertmanager’s — every alert currently created by the bootstrap script uses this one webhook.

Adding a new alert

Add a new TILE-alert upsert block to bootstrap-script.yaml (a metric threshold against otel_metrics_*), following the existing patterns in that file, then let ArgoCD sync the change — the PostSync hook re-runs the script. Do not add saved_search / log-count alerts (see the metric-only rule above); if a signal genuinely has no metric or telemetry equivalent, it must clear The last-resort log exception below — both halves of the qualifying test, a named owner, a migration trigger, and a row in the register — before it is written.

Keep alert bodies short

An alert body communicates what is wrong, in as few words as that takes. Shorter is better. There is no minimum. Say what is observed wrong, add a Runbook: pointer to the triage section, and stop.

Two numbers bound that, and they do different jobs:

• 1024 characters is Pushover’s refusal threshold — a failure boundary, and the only one enforced. It is real here because nothing between the alert body and the phone truncates: the body becomes HyperDX’s rendered body, which maps straight into Pushover’s message. An over-cap body is a page refused, or one that arrives without the instructions it exists to carry.
• 350 characters is a ceiling to stay well under, never a target to fill. It is a sanity bound: a body approaching it has drifted into being a runbook. It is held by review rather than by CI, and nothing reds a body for sitting between the two numbers.

Two facts about how a page arrives are why a short body is enough:

• The webhook already sends title as "{{title}} ({{state}})", so the alert’s name and its state reach the phone ahead of the body. WHAT is broken is carried by the title; the body is additional text on top of it.
• A Pushover lock-screen preview shows only the leading portion of message — on the order of 100–200 characters. That figure has no named authority: Pushover does not document it, and the real cut-off is set by the phone’s OS and lock-screen settings. What does not vary is the ordering — text past the preview is read only after the phone is unlocked, and by then the operator is better served by opening the runbook.

Mechanism, taxonomy, instructions addressed to whoever next edits the query, and triage beyond the first discriminating command belong in a triage section of this runbook, not on the page.

Three pre-existing Firewalla bodies are still over the cap. They are recorded as debt, not exempted, and they belong to the fw-observability workstream. The record lives in KNOWN_OVER_CAP, which exists twice: in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml (the runtime gate) and in tools/clickstack-alert-lint/lint.py (the pre-merge check). The two copies must stay identical, and a rule in the check reds when they disagree.

So if you edit one of those recorded bodies:

• It stays over the cap → re-record its new measured length in both copies, in the same change. The number is pinned to the measurement, so a body that merely shrinks inside its old allowance is a finding too.
• It drops to at most 1024 characters → delete its entry from both copies, in the same change. That is the goal.

Both files carry a doc block explaining why the two instruments enforce different bounds on the same numbers; read those rather than a third copy here. What matters operationally is where you meet the failure: the alert-lint required status check on the pull request, which names the cure it wants. It is deliberately the strict half, so a length problem is caught at review and not by a bootstrap Job that would otherwise stop every alert reconciling.

kube-state-metrics series: labels land in Attributes

A gotcha with one failure mode and no error message. kube-state-metrics series arrive through the prometheus receiver with their labels in Attributes — for kube_node_status_condition that is {condition, node, status}. ResourceAttributes on those same series carries scrape-target metadata only (host.name, server.address, service.name, url.scheme, …) and contains neither condition nor status.

The nearest-looking analog in bootstrap-script.yaml is rh-operator-avail, and it filters ResourceAttributes — correctly, because it reads an OTel k8s-cluster-receiver gauge (k8s.deployment.available), where the labels really do live there. Do not copy that filter onto a kube_* series. A tile whose where names ResourceAttributes['condition'] matches nothing, and a tile that matches nothing is indistinguishable from a healthy cluster: it reads as a permanently-quiet alert rather than as an error. Nothing goes red, and the alert is dead for as long as nobody checks.

Check attribute placement before writing the where for any new series:

SELECT mapKeys(Attributes) AS a, mapKeys(ResourceAttributes) AS ra, count() FROM default.otel_metrics_gauge WHERE MetricName = '<metric>' AND TimeUnix > now() - INTERVAL 1 HOUR GROUP BY a, ra

Triage: Kubernetes node NOT READY

The long form of the body Kubernetes node NOT READY pages. The page carries the observation and a pointer here; the triage below is the rest of it.

Why a NotReady that starts and clears inside one window does NOT page. The deployed HyperDX chart-query builder reduces each series to last_value per granularity bucket before applying the tile’s aggregation across series. AttributesHash hashes the whole attribute map and is what defines a series, so each (node, condition, status) triple is its own. The tile’s max therefore runs across those per-series last values rather than across the raw samples in the window: it reads END-OF-BUCKET state, not “any sample inside the bucket”. A tile-shaped query answered by the running evaluator returns 8 where the raw-row alternative returns 960, and 8 is exactly the distinct-node count. That end-of-bucket reading is a deliberate consequence of the 1h interval chosen to tolerate k3s upgrade reboots, not a defect.

Why BOTH statuses, and why this must NOT be “simplified” to status = 'false'. Across the full 27-day tpi-beta-4 outage (2026-07-27 to 2026-08-23) this metric logged ~2880 Ready=unknown samples per day and ZERO Ready=false samples. A dead kubelet stops renewing its Node lease, which the control plane records as unknown; false is a live kubelet self-reporting sick. An alert on false alone would have been silent for all 27 days.

Rules out: nothing narrower. This alert is un-grouped by design and names NO node — use the drill-down below to find it.

Drill-down. Which node, and which status it is reporting:

SELECT DISTINCT Attributes['node'], Attributes['status'] FROM default.otel_metrics_gauge WHERE MetricName = 'kube_node_status_condition' AND Attributes['condition'] = 'Ready' AND Attributes['status'] IN ('false','unknown') AND Value = 1

Recover. kubectl describe node <node>, then the kubelet, then the node’s root filesystem. The 2026-07 event was a btrfs errno=-5 EIO on the volume holding /var/lib/{kubelet,longhorn,rancher} that killed all userspace while the kernel stayed up — so the box still answered ICMP and ping was NOT a health signal.

Triage: Kubernetes node readiness signal INCOMPLETE

The long form of the body Kubernetes node readiness signal INCOMPLETE pages. The page carries the observation, the hardcoded cluster-size threshold and what to do about it, and a pointer here; the triage below is the rest of it.

What this rules in. Two causes, and they are not the same fault. Either a Node object was deleted from the API server while the remaining nodes keep reporting normally — the node set shrank rather than a node going unhealthy — or kube-state-metrics is down, unscraped, or its metric was renamed upstream, which empties the bucket entirely. The first is a cluster change; the second is a telemetry failure that says nothing about the cluster. kubectl get nodes separates them in one command, which is why it is the first step under Recover below.

Why this does not overlap Kubernetes node NOT READY, measured rather than assumed. Across the full 27-day tpi-beta-4 outage (2026-07-27 to 2026-08-23) the dead node kept emitting its series and the distinct-node count held at 8 the whole time, so this alert would NOT have fired on that incident. Kubernetes node NOT READY owns that case; the two are non-overlapping by measurement.

Drill-down. Compare the returned set against the nodes you expect:

SELECT DISTINCT Attributes['node'] FROM default.otel_metrics_gauge WHERE MetricName = 'kube_node_status_condition' AND Attributes['condition'] = 'Ready'

Recover. kubectl get nodes first. If the node set is intact the fault is in telemetry, not in the cluster: check the kube-state-metrics workload, then its scrape job in argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml.

The last-resort log exception

The metric-only rule above names an escape hatch. Until 2026-08-10 this page referred to that hatch twice and defined it nowhere, which meant anyone could claim it and nobody could audit it. This section is the definition.

It exists because hl-pziv is a policy being actively paid down: it retired the router-hosts edge heartbeat, the Traefik 5xx spike, the oauth2-proxy session-store alert and two Keycloak audit alerts. An undocumented escape hatch invoked a third time is just the policy being repealed by practice.

What qualifies

Both halves are required — not either.

1. No metric exists for the property. Not “no metric is convenient”: no metric or telemetry signal carries the fact the alert is about. Check otel_metrics_gauge / otel_metrics_sum before claiming this.
2. Creating one is disproportionate to the signal. “Disproportionate” is measured against the value of the signal, never against effort in the abstract. A cheap metric for a rare event still wins; an expensive metric for a rare event does not.

The worked example is the entry in the register below. Emitting a prefix-change counter would have meant standing up a metrics pipeline on the resolver pair — those nodes run a logs-only collector, and the estate has no systemd unit-state metric anywhere — to watch for an event that fires when the ISP re-delegates. That is a larger change than the fix it would instrument, against a signal that fires perhaps twice a year.

Failing test: “there is no Keycloak metric today” satisfies half 1 and not half 2 — a Keycloak event-listener SPI exporting a counter is proportionate work, which is exactly why hl-qjh3.17 still tracks those two as a migration rather than granting them an exception.

What an exception must carry

Requirement	Why
A named owner	An exception with no owner is a permanent exception.
A migration trigger, stated as a CONDITION and not a date	A date on work nobody is scheduled to do expires quietly and changes nothing. The trigger is the event that would make the metric cheap.
A row in the register below	The count has to be visible. That is the whole point of counting.
The argument, inline in bootstrap-script.yaml	Every other alert decision in that file is recorded in its own voice; a granted exception whose argument was deleted is indistinguishable from an oversight.

The register of exceptions taken

Exceptions taken — live log-content alerts running under this section. Two rows today.

Alert	Owner	Migration trigger	Granted
Resolver delegated-prefix change (RA-WATCH-PREFIX-CHANGE)	DNS serving plane	The first time a metrics pipeline exists on the resolver pair for any other reason. The alert then becomes a counter incremented on the detector’s exit-3 path plus a tile alert, and this row leaves the register.	2026-08-10
LiteLLM key material in logs (OBS-02)	LiteLLM plane	The first time a litellm.key_material_seen counter is emitted by the scrubber (a transform statement can increment none today); the alert then becomes a tile alert on that counter and this row leaves the register.	2026-09-08

The retired Keycloak audit alerts are NOT entries in this register, and the distinction is not pedantry. They were retired, not excepted — they have no live alert and no granted exception. They remain the open paydown backlog under hl-qjh3.17, alongside the retired Traefik and oauth2-proxy log-count alerts. Naming them here stops a one-row register from reading as “this has only ever come up once”: the shape has come up repeatedly, and exactly two instances have been granted.

Keycloak security-event alerts (hl-0ahs.14) — RETIRED in PR #1638

These three alerts were retired in PR #1638 (hl-pziv). They were log-count saved_search alerts that had silently frozen — the Logs source referenced a TimestampTime column absent from the compat otel_logs schema, so they never actually fired. Rather than repair a log-count alert (an anti-pattern), they are being re-homed to metric/telemetry signals under hl-qjh3.17. Open blocker: no Keycloak metrics flow to otel_metrics today, and the password-change / admin-config-change signals are audit events with no native Keycloak metric — their replacement needs a Keycloak event-listener SPI (counter metric) or a documented exception. The definitions below are retained as historical reference and the starting point for that work.

Three log-based alerts on the Keycloak otel_logs stream replaced the alerts Authentik used to send via email (its tf/authentik/notifications.tf, since deleted):

Alert	Keycloak signal	Threshold
Keycloak login failure rate	LOGIN_ERROR events	>5 in 15m
Keycloak password changed	UPDATE_PASSWORD events	any occurrence in 15m
Keycloak admin config change	admin CREATE/UPDATE/DELETE events, excluding the admin-cli automation client	any occurrence in 15m

Two of the original five alert categories were dropped rather than faked: suspicious_request has no Keycloak equivalent event (partially covered by the login-failure-rate alert), and new-sign-in notified the specific signed-in user — a destination this single-shared-webhook architecture has no equivalent for (and an operator-facing alert on every login would be pure noise).

The password-changed and admin-config-change alerts required argocd/app-configs/keycloak/keycloak-cr.yaml’s spi-events-listener--jboss-logging--success-level=info option: by default Keycloak’s built-in event listener only logs error-type events (WARN); success events (LOGIN, UPDATE_PASSWORD) and all admin events have no error concept and log at DEBUG, never reaching otel_logs.

Pushover token rotation

The bootstrap Job reads PUSHOVER_TOKEN/PUSHOVER_USER_KEY from the pushover-hyperdx-tokens Secret (ExternalSecret, refreshed from Vault every 15 min) and bakes the value into the webhook document in HyperDX’s MongoDB at Job-run time — not read live per-alert. After rotating the token in Vault:

# Wait for ESO to refresh pushover-hyperdx-tokens (up to 15min), then
# trigger the bootstrap script to re-run by syncing the clickstack-alerts
# Application in ArgoCD (re-fires the PostSync hook Job).