Alerting Operations (HyperDX)

Alertmanager and kube-prometheus-stack were decommissioned in epic hl-qjh3 (see ADR hl-toju). Alerting runs on HyperDX TILE alerts — metric thresholds against ClickHouse otel_metrics_* — delivered to Pushover via a generic webhook.

Alerts (and dashboards) MUST be metric-based. Counting rows in otel_logs to drive a threshold (HyperDX saved_search / log-count alerts) is an anti-pattern, permitted only as a documented last-resort exception when no metric or telemetry signal exists. The log-based saved_search alerts this page previously described were retired in PR #1638 (hl-pziv) — they had silently frozen on a Logs-source schema mismatch and never fired. Their metric/telemetry replacements are tracked in hl-qjh3.17.

Where alert config lives

Alert, saved-search, and webhook definitions are provisioned idempotently by the clickstack-alerts bootstrap script, which runs as an ArgoCD PostSync hook Job on every sync of that Application:

Component	Location
Bootstrap script (source of truth)	argocd/app-configs/clickstack-alerts/bootstrap-script.yaml
Bootstrap Job (PostSync hook)	argocd/app-configs/clickstack-alerts/bootstrap-job.yaml
Pushover token ExternalSecret	argocd/app-configs/clickstack-alerts/external-secret.yaml

The script upserts by name (db.webhooks, db.savedsearches, db.alerts) so re-runs converge state rather than duplicating records. Deleting or renaming an alert requires an explicit deleteOne/updateOne — upsert alone never removes or renames a record.

Delivery

All migrated alerts route through a single generic webhook, Pushover (cluster-infra), which posts to https://api.pushover.net/1/messages.json using the cluster-infra Pushover app token. There is no per-domain routing tree like Alertmanager’s — every alert currently created by the bootstrap script uses this one webhook.

Adding a new alert

Add a new TILE-alert upsert block to bootstrap-script.yaml (a metric threshold against otel_metrics_*), following the existing patterns in that file, then let ArgoCD sync the change — the PostSync hook re-runs the script. Do not add saved_search / log-count alerts (see the metric-only rule above); if a signal genuinely has no metric or telemetry equivalent, raise it as an explicit exception first.

Keycloak security-event alerts (hl-0ahs.14) — RETIRED in PR #1638

These three alerts were retired in PR #1638 (hl-pziv). They were log-count saved_search alerts that had silently frozen — the Logs source referenced a TimestampTime column absent from the compat otel_logs schema, so they never actually fired. Rather than repair a log-count alert (an anti-pattern), they are being re-homed to metric/telemetry signals under hl-qjh3.17. Open blocker: no Keycloak metrics flow to otel_metrics today, and the password-change / admin-config-change signals are audit events with no native Keycloak metric — their replacement needs a Keycloak event-listener SPI (counter metric) or a documented exception. The definitions below are retained as historical reference and the starting point for that work.

Three log-based alerts on the Keycloak otel_logs stream replaced the alerts Authentik used to send via email (its tf/authentik/notifications.tf, since deleted):

Alert	Keycloak signal	Threshold
Keycloak login failure rate	LOGIN_ERROR events	>5 in 15m
Keycloak password changed	UPDATE_PASSWORD events	any occurrence in 15m
Keycloak admin config change	admin CREATE/UPDATE/DELETE events, excluding the admin-cli automation client	any occurrence in 15m

Two of the original five alert categories were dropped rather than faked: suspicious_request has no Keycloak equivalent event (partially covered by the login-failure-rate alert), and new-sign-in notified the specific signed-in user — a destination this single-shared-webhook architecture has no equivalent for (and an operator-facing alert on every login would be pure noise).

The password-changed and admin-config-change alerts required argocd/app-configs/keycloak/keycloak-cr.yaml’s spi-events-listener--jboss-logging--success-level=info option: by default Keycloak’s built-in event listener only logs error-type events (WARN); success events (LOGIN, UPDATE_PASSWORD) and all admin events have no error concept and log at DEBUG, never reaching otel_logs.

Pushover token rotation

The bootstrap Job reads PUSHOVER_TOKEN/PUSHOVER_USER_KEY from the pushover-hyperdx-tokens Secret (ExternalSecret, refreshed from Vault every 15 min) and bakes the value into the webhook document in HyperDX’s MongoDB at Job-run time — not read live per-alert. After rotating the token in Vault:

# Wait for ESO to refresh pushover-hyperdx-tokens (up to 15min), then
# trigger the bootstrap script to re-run by syncing the clickstack-alerts
# Application in ArgoCD (re-fires the PostSync hook Job).