Decommission kube-prometheus-stack; migrate alerting to HyperDX TILE alerts
Date: 2026-06-30 Status: Accepted Decision: hl-toju Deciders: Sean Brandt, Claude
Context
Section titled “Context”After epic hl-nlxl retired Grafana, ClickStack/HyperDX became the sole observability UI and a parallel OTel pipeline (monitoring-otel-scraper) already feeds all cluster metrics into ClickHouse otel_metrics_. Live grounding (2026-06-30) found zero PromQL consumers besides Alertmanager (no prometheus-adapter, zero HPAs, no KEDA), only 6 alert rules, and all 6 source metrics already present in otel_metrics_. ADR hl-5xgq adopted ClickStack but explicitly RETAINED kube-prometheus-stack as out-of-scope for hl-nlxl; this decision reverses that deferral while leaving hl-5xgq’s core ClickStack-adoption decision intact (relates-to, not supersedes).
Decision
Section titled “Decision”Remove kube-prometheus-stack entirely (Prometheus, Alertmanager, prometheus-operator, bundled node-exporter, bundled kube-state-metrics, operator CRDs, the prometheus namespace) and migrate all 6 PromQL alert rules to HyperDX TILE alerts via the extended clickstack-alerts mongosh bootstrap, routed to the existing cluster-infra Pushover token. ClickHouse otel_metrics_* becomes the sole metrics store.
Rationale
Section titled “Rationale”- No PromQL consumers remain once Alertmanager is gone: prometheus-adapter absent, zero HPAs, KEDA not installed; metrics-server (kube-system) is independent and unaffected.
- The OTel pipeline is currently parasitic on kps-bundled scrape targets (node-exporter, kube-state-metrics); decommission severs that hidden coupling.
- HyperDX TILE alerts cover all required patterns (threshold, ratio via asRatio, presence/no-data); Pushover delivery is unchanged via the cluster-infra webhook.
Alternatives Considered
Section titled “Alternatives Considered”- Decommission kps; migrate 6 alerts to HyperDX TILE alerts (chosen): eliminates the redundant scrape stack; all source metrics already in ClickHouse. Cost: TILE no-data behavior needs live validation and a mandatory 24h parallel-soak interlock before Alertmanager teardown.
- Retain kps indefinitely (rejected): no migration risk, but keeps a redundant scrape engine and leaves the OTel pipeline coupled to kps-bundled collectors.
Consequences
Section titled “Consequences”- Positive: single observability platform; operator CRDs (ServiceMonitor/PodMonitor/PrometheusRule) removed from the cluster API surface; OTel pipeline no longer depends on kps availability; two stateful PVCs (Prometheus 250Gi, Alertmanager 10Gi) reclaimed.
- Negative: TILE no-data behavior must be validated before W2; the W3 CRD/namespace teardown is the sole non-revertable wave.
- Neutral: metrics-server (kube-system) is independent and unaffected; relates-to ADR hl-5xgq (which remains Accepted).