Deploy kube-state-metrics standalone; replace node-exporter with OTel hostmetrics
Date: 2026-06-30 Status: Accepted Decision: hl-lil1 Deciders: Sean Brandt, Claude
Context
Section titled “Context”kube-prometheus-stack bundles kube-state-metrics (KSM, feeds kube_* = 192 series, the top ClickHouse producer) and node-exporter (feeds node_* = 306 series). The OTel scrape collector has static jobs targeting these bundled instances, so removing kps drops kube_* and node_* from ClickHouse unless replaced. After Grafana’s removal, no alert or dashboard consumer references node_* by its Prometheus name.
Decision
Section titled “Decision”Deploy kube-state-metrics standalone (prometheus-community/kube-state-metrics ArgoCD app, chart 7.4.1) and repoint the OTel KSM scrape job to its Service. Drop node-exporter entirely and rely on the existing OTel hostmetrics receiver (system.*) for host metrics. Hybrid: keep KSM (Prometheus names preserved), drop node-exporter (semconv host metrics).
Rationale
Section titled “Rationale”- Standalone KSM preserves all kube_* series names, avoiding ClickHouse query and dashboard breakage.
- The OTel k8s_cluster receiver is NOT a 1:1 kube_* substitute; switching to it would require rewriting every downstream consumer.
- hostmetrics system.* already provides full host coverage across all 8 nodes; a separate node-exporter DaemonSet is redundant.
Alternatives Considered
Section titled “Alternatives Considered”- Standalone KSM + OTel hostmetrics for host metrics (chosen): kube_* preserved; system.* already running; no new host-metrics component. Cost: node_* (Prometheus semconv) replaced by system.* (OTel semconv); cadvisor container_* gap accepted (no consumers).
- OTel k8s_cluster receiver to replace KSM (rejected): native receiver, no separate deployment, but not a 1:1 kube_* replacement; breaks consumers.
- Retain node-exporter standalone (rejected): preserves node_* names, but the DaemonSet duplicates hostmetrics coverage and node_* has no consumers post-Grafana.
Consequences
Section titled “Consequences”- Positive: kube_* flows unchanged after kps removal; the OTel pipeline acquires no new kps dependency; smaller running footprint (one fewer DaemonSet).
- Negative: node_* replaced by system.*; any future tooling expecting Prometheus-semconv host metrics must adapt.
- Neutral: kube_* (KSM) and k8s.* (k8s_cluster/kubeletstats) coexist in otel_metrics_*; no dedup required.