Prometheus / kube-prometheus-stack Decommission Design

Design bead: hl-qjh3 · Date: 2026-06-30 · Author: Sean Brandt (+ Claude)

Predecessor: Grafana-Labs sunset (epic hl-nlxl, DONE) — reuse its 6-wave drop-then-remove playbook. ADR hl-5xgq deferred Prometheus as explicitly out-of-scope for that epic; this spec revisits and reverses that decision.

1. Context

The Grafana-Labs sunset (epic hl-nlxl) removed Grafana — the PromQL query UI — and a parallel OTel metrics pipeline (monitoring-otel-scraper) already collects the cluster’s metrics into ClickHouse otel_metrics_*. Prometheus has degraded from “metrics backbone” to “an alerting engine with a now-redundant collector attached.”

Grounding against the live cluster (2026-06-30) confirmed:

• No PromQL consumers besides Alertmanager. prometheus-adapter is absent, there are zero HorizontalPodAutoscalers, KEDA is not installed. metrics-server (kube-system) is independent and stays. Alertmanager is the sole consumer of Prometheus query results.
• Only 6 alert rules exist: nats group (4 alerts) + clickstack-otel-collector group (2 alerts). defaultRules.create: false — no chart-shipped rules. No recording rules.
• All 6 alerts’ source metrics are already in ClickHouse (nats_*/gnatsd_* = 114 series; otelcol_* collector self-telemetry). Alert migration is independent of the collector work.
• HyperDX supports metric alerts via the TILE alert source type (metric Source over otel_metrics_* → dashboard tile → tile alert with threshold). The existing clickstack-alerts mongosh-bootstrap pattern (which today only does log-count saved_search alerts) extends to cover this.

1.1 The reframe — kps bundles two collectors the OTel pipeline depends on

kube-prometheus-stack is not just Prometheus + Alertmanager. It bundles node-exporter (feeds node_* = 306 series) and kube-state-metrics (feeds kube_* = 192 series — the single largest metric producer in ClickHouse). The OTel scrape collector has static scrape jobs (node-exporter, kube-state-metrics) that target exactly these bundled components. Removing kps therefore deletes the scrape targets that currently feed node_*/kube_* into ClickHouse — the OTel pipeline is today partly parasitic on kps. This collector decision, not the alerting, is the design’s center of gravity.

2. Decisions (resolved during brainstorming)

#	Decision	Choice
D1	How node_* / kube_* keep flowing after kps removal	Hybrid: redeploy kube-state-metrics standalone (the k8s_cluster receiver is not a 1:1 replacement for kube_*); drop node-exporter and rely on OTel hostmetrics (system.*).
D2	Teardown depth	Full clean: remove the operator CRDs (ServiceMonitor/PodMonitor/PrometheusRule), the prometheus namespace, and every object of those CRD kinds (at its producing source) so the CRDs can be deleted. Of the 26 live ServiceMonitors, the 12 kps-internal/control-plane ones (all in ns prometheus) vanish with the kps app in W2; the ~14 app-owned ones (exact set confirmed via live kubectl get servicemonitor -A at plan time) plus the nats PodMonitor and the 2 PrometheusRules are removed at source in W3 (see §6/W3 for the producer breakdown).
D3	Non-alert parity scope	Close miniflux + keycloak; document cadvisor → kubeletstats and etcd-server as accepted deltas.
D4	keycloak metric path	Full OTel-native push (experimental opentelemetry-metrics feature → OTLP to cs-otel-collector); no scrape job. Accepted-experimental per owner.
D5	Alert migration mechanism	Extend the existing clickstack-alerts mongosh bootstrap to create a metric Source + dashboard tiles + TILE alerts.

3. Target end-state (architecture)

Removed: Prometheus, Alertmanager, prometheus-operator, bundled node-exporter; the prometheus namespace; operator CRDs; the ~14 app-owned ServiceMonitors + the nats PodMonitor + the 2 PrometheusRules (the 12 kps-internal ServiceMonitors die with the kps app); alertmanager-pushover ExternalSecret; the prometheus IngressRoute/Certificate; the uptime-kuma prometheus monitor; the Vault pushover SA binding to ns prometheus.

Retained / added: kube-state-metrics standalone (feeds kube_* unchanged); host metrics from OTel hostmetrics (system.*); all metrics in ClickHouse otel_metrics_*; the 6 alerts as HyperDX TILE alerts via the extended clickstack-alerts bootstrap, routed to the existing cluster-infra Pushover token.

Metric ingestion after cutover has zero dependency on kps:

• static OTel scrape — KSM-standalone, miniflux, + existing app jobs (argocd, vault, nats, apiserver, coredns, cert-manager, temporal, velero, dolt, agentgateway, cilium-hubble)
• OTLP push — traefik, keycloak, the collectors themselves
• hostmetrics / kubeletstats / k8s_cluster — host + pod/container + k8s object state

4. Component changes

Component	Change	File(s)
kube-state-metrics	New standalone prometheus-community/kube-state-metrics ArgoCD app; repoint OTel kube-state-metrics scrape job to its Service	new argocd/app-configs/monitoring-kube-state-metrics/ + new cluster-app template; argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml
node-exporter	Drop. Remove the node-exporter OTel scrape job; rely on hostmetrics system.*	scrape-collector-values.yaml
miniflux	Add a miniflux static scrape job (closes the orphaned-ServiceMonitor gap; miniflux_* currently never lands in ClickHouse)	scrape-collector-values.yaml
keycloak	Enable opentelemetry-metrics feature + telemetry-metrics-enabled=true + telemetry-metrics-endpoint=http://cs-otel-collector.clickstack:4317 (grpc) in the Keycloak CR. No scrape job.	argocd/app-configs/keycloak/keycloak-cr.yaml
6 alerts	Translate to HyperDX TILE alerts (metric Source + dashboard tiles + tile alerts), bootstrapped via mongosh	argocd/app-configs/clickstack-alerts/bootstrap-script.yaml
kps app	Delete the monitoring-prometheus cluster-app template + app-config dir	argocd/cluster-app/templates/monitoring-prometheus.yaml, argocd/app-configs/monitoring-prometheus/

5. Alert migration detail (the one hard blocker)

Extend clickstack-alerts/bootstrap-script.yaml to idempotently upsert:

1. db.sources — a kind:"metric" source with metricTables:{gauge,sum,histogram} pointing at otel_metrics_gauge / otel_metrics_sum / otel_metrics_histogram. (Verify whether a metric Source already exists in live HyperDX — recall shows only Traces + Logs sources today.)
2. db.dashboards — a dashboard with one tile per alert metric (chartConfig with metricType, metricName, aggFn).
3. db.alerts — source:"tile" referencing dashboardId + tileId, with threshold, thresholdType (above/below), interval, and channel:{type:"webhook", webhookId} reusing the existing cluster-infra Pushover webhook.

HyperDX tile capabilities used (deepwiki-grounded, hyperdxio/hyperdx):

• Ratio: a tile with exactly two series + asRatio:true (seriesReturnType:"ratio") computes series[0] / series[1]; getChartConfigFromAlert/checkAlerts honour seriesReturnType, so a TILE alert can threshold a ratio. Caveat: ratio is currently computed client-side in the query layer (upstream TODO to move it DB-side) — W1 must confirm the alert evaluator computes it during headless checkAlerts runs, not just in the browser.
• Label filters: OTel metric labels live in the Attributes / ResourceAttributes map columns. A tile where (whereLanguage:"sql" or "lucene", e.g. Attributes['exporter']:'clickhouse') filters on them. The exact key (e.g. Attributes['exporter'] vs a flattened column, and how the prometheusreceiver maps the PromQL job label) must be confirmed by ClickHouse query in W1.
• Counter rate: metricType:"sum" tiles render rate with counter-reset/temporality handling.

5.1 Per-alert translation

Exact source expressions from argocd/app-configs/nats/prometheus-rules.yaml and argocd/app-configs/clickstack/otel-collector-monitoring.yaml:

Alert	Source PromQL	HyperDX tile mapping
NATSJetStreamStorageHigh	sum(nats_varz_jetstream_stats_storage) / sum(nats_varz_jetstream_config_max_storage) > 0.8	ratio tile — series[0]=sum(nats_varz_jetstream_stats_storage), series[1]=sum(nats_varz_jetstream_config_max_storage), asRatio:true, threshold 0.8, above
NATSHighConnectionCount	sum(nats_varz_connections) > 1000	gauge tile, aggFn:sum, threshold 1000, above
NATSSlowConsumers	rate(nats_varz_slow_consumers[5m]) > 0	sum tile (counter), rate, threshold 0, above
ClickStackOtelCollectorExportFailing	rate(otelcol_exporter_send_failed_log_records_total{namespace="clickstack",exporter="clickhouse"}[…]) > 0	sum tile (counter), rate, where Attributes['exporter']='clickhouse' (+ namespace), threshold 0, above
NATSClusterQuorumLost	absent(nats_varz_cores{job="nats/nats"}) == 1 or count(nats_varz_cores{job="nats/nats"}) < 2	presence/count tile on nats_varz_cores (count of series over interval), threshold 2, below — fires at 0 pods (absent) and 1 pod (no quorum). See §5.2
ClickStackOtelCollectorClickHouseExporterMissing	absent(otelcol_exporter_sent_log_records_total{namespace="clickstack",exporter="clickhouse"})	presence/count tile (count of series, where exporter='clickhouse'), threshold 1, below. See §5.2

5.2 The absent() / no-data translation risk

The two presence alerts fire when a metric stops arriving — a distinct condition from “value too high/low”. Proposed: a tile that counts series presence over the interval with a below threshold (below 2 for quorum so the one-pod case fires; below 1 for the collector exporter). HyperDX’s evaluation behaviour on a genuinely-empty result must be validated against the live instance during W1 — a threshold alert that returns “no rows” may not fire at all (the no-data case is precisely when these alerts must fire). Fallback: a saved_search count-based alert (the proven existing pattern) or a synthetic “heartbeat” tile. This no-data behaviour, plus the ratio-in-headless-checkAlerts caveat above, are the pieces requiring live validation before W2 tears Alertmanager down.

6. Sequencing — drop-then-remove, per-wave verify (reuse hl-nlxl playbook)

Each wave is an independent PR, revertable by ArgoCD (except the W3 Terraform teardown).

W0 — Parity prep (additive; no removal)

• Deploy standalone kube-state-metrics; repoint the OTel kube-state-metrics scrape job.
• Add the miniflux scrape job.
• Enable keycloak OTLP-native metrics (D4).
• Verify: kube_* (now from standalone KSM), miniflux_*, and keycloak metrics all present in ClickHouse from non-kps sources; confirm hostmetrics system.* coverage is an acceptable node-exporter replacement before node-exporter is removed in W2.

W1 — Alert migration (additive; parallel-run)

• Extend the clickstack-alerts bootstrap with the metric Source + 6 TILE alerts (§5).
• Verify (ClickHouse): confirm the exact label keys each alert filters on actually exist in otel_metrics_* — Attributes['exporter']='clickhouse', the namespace key, and how the prometheusreceiver maps the NATS job="nats/nats" label — before trusting the tile where.
• Verify (fire): trip each threshold alert in a controlled test; resolve §5.2 (ratio in headless checkAlerts; no-data firing for the two presence alerts — the genuinely-empty case cannot be exercised by a threshold trip and must be validated by actually stopping the series in a safe target or confirming HyperDX’s empty-result semantics).
• Soak (minimum 24h) in parallel with Alertmanager — both alerting paths live. The 24h floor ensures the for:-guarded alerts (e.g. ClickStackOtelCollectorClickHouseExporterMissing, for:10m) and a naturally-elevated NATS window are observed, not just a synthetic trip. This is the safety interlock: HyperDX alerts must be proven firing before Alertmanager is torn down.

W2 — Remove kps (GitOps)

• Delete the monitoring-prometheus cluster-app template (removes Prometheus, Alertmanager, operator, bundled node-exporter).
• Remove all four OTel scrape jobs that target kps components (they go to dead targets and produce permanent scrape-failure log spam otherwise): node-exporter, kps-kube-prometheus-stack-alertmanager (:9093), kps-kube-prometheus-stack-prometheus (:9090), and kps-kube-prometheus-stack-operator (:443) in scrape-collector-values.yaml.
• Remove the alertmanager-pushover ExternalSecret and the prometheus IngressRoute/Certificate.
• Verify: no metric gaps (kube_* from standalone KSM; host via system.*); HyperDX alerts still firing; Pushover delivery intact via cluster-infra; OTel collector logs free of scrape-failure spam.

W3 — Scrub CRDs + infra config (mixed GitOps + operator-applied Terraform)

Order within the wave matters — every object of the operator CRD kinds (ServiceMonitor/PodMonitor/PrometheusRule) must be deleted from the cluster before the CRDs themselves are removed, or the CRD delete will be blocked / orphan finalizers. The ServiceMonitor producers split across two toolchains:

1. (GitOps) Remove the five repo-defined manifests of these kinds: dolt/servicemonitor.yaml, temporal-server/servicemonitor.yaml, miniflux/servicemonitor.yaml, clickstack/otel-collector-monitoring.yaml (carries the cs-otel-collector SM and PrometheusRule), and nats/prometheus-rules.yaml (the nats PrometheusRule — dead the moment Alertmanager is removed in W2). Targets are already covered by static OTel jobs.
2. (Terraform / chart toggles — tf/cluster-bootstrap, operator-applied via local-exec, not merge-triggered) Disable every remaining chart-created ServiceMonitor/PodMonitor — these are live objects a GitOps PR cannot remove. Known producers: cert-manager.tf prometheus.servicemonitor.enabled = false (line ~34); argocd.tf addPrometheusAnnotations = false (line ~165) plus the six serviceMonitor.enabled = true blocks. Plan task: the live cluster also carries app-owned SMs for vault, velero, keycloak, and authentik (ak-outpost-ldap), plus the nats PodMonitor — produced by their own Helm charts/operators, not the repo SM manifests above. The W3 plan MUST inventory every SM/PM producer (start from kubectl get servicemonitor,podmonitor -A cross-referenced to each app’s chart values) and disable each at source. The verification gate in this wave is the backstop, not the discovery mechanism.
3. (Terraform — same workspace, separate apply after step 1+2 confirm zero SM/PM/PrometheusRule objects remain) Remove the operator CRDs (prometheus-crds.tf), the prometheus namespace, the prometheus_operator_crds_version var, the imports.tf blocks, and the cert-manager/argocd depends_on prometheus_operator_crds edges. Splitting steps 2 and 3 into two applies avoids a same-apply race where the CRD is destroyed while SM objects still reference it.
4. (Terraform) Remove the uptime-kuma prometheus monitor (tf/uptime-kuma/monitors-services.tf) and the Vault pushover SA→ns prometheus binding (tf/vault/k8s-pushover.tf).
5. (GitOps) Remove prometheus from Velero excludedNamespaces.

• Verify: between steps 2 and 3, kubectl get servicemonitor,podmonitor,prometheusrule -A returns nothing; after step 3, no prometheus namespace and no operator CRDs remain.

W4 — Docs + ADR

• Remove the argocd/CLAUDE.md “Prometheus retained” note (also factually stale — it cites “recording rules” as a retention reason, but grounding found zero recording rules) and the prometheus row from the Velero exclusion table there. Scrub the docs set (docs/operations/alertmanager.md, services.md, technologies.md, clickstack.md, pushover.md, cluster-bootstrap.md, incidents.md).
• Capture an ADR that supersedes the hl-5xgq Prometheus-deferral decision (note: the original retention rationale was partly mistaken re: recording rules).

7. Error handling / rollback

• W0–W2, W4 land via PR + ArgoCD and are revertable.
• W1→W2 interlock: if HyperDX alerts do not fire reliably (esp. §5.2), W2 is blocked — Prometheus keeps alerting until parity is proven.
• Keycloak (D4) is an experimental upstream feature (“not recommended for production”) with Micrometer→OTel semconv name differences; W0 verifies which series land and the documented fallback is to scrape the management :9000/metrics endpoint instead.
• W3 Terraform teardown (CRD + namespace deletion) is the only non-trivially-reversible step; it is gated behind verified W2 completion, and the namespace delete must not race ArgoCD pruning.

8. Testing / verification

• Per-wave ClickHouse parity queries: each metric family present from a non-kps source.
• HyperDX alert-fire verification: trip each threshold in a controlled test; confirm Pushover delivery on cluster-infra.
• Final sweep: zero ServiceMonitor/PodMonitor/PrometheusRule objects; no prometheus namespace; no kube-prometheus-stack workloads.

9. Out of scope / accepted deltas

• cadvisor container_* — replaced by kubeletstats semconv (k8s.pod.cpu.*, container.memory.working_set). No alert consumers; Grafana (the Prometheus-name dashboard consumer) is already gone.
• etcd server-side (etcd_server_*/etcd_mvcc_*/etcd_disk_*) — never scraped (no etcd access); only client-side etcd metrics present. No alert consumers.
• Prometheus-name vs OTel-semconv duplication (node_* vs system.*; kube_* vs k8s.*) — kube_* is kept canonical via standalone KSM; system.* becomes canonical for host metrics.