Prometheus / kube-prometheus-stack Decommission Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Remove kube-prometheus-stack (Prometheus, Alertmanager, operator, bundled node-exporter, bundled kube-state-metrics) from the cluster with zero metric-collection or alerting gap, leaving ClickHouse otel_metrics_* as the sole metrics store and HyperDX TILE alerts as the sole alerting path.

Architecture: Drop-then-remove, per-wave verify (reuses the hl-nlxl Grafana-sunset playbook). W0 reproduces the metric sources kps secretly provides (standalone kube-state-metrics; keycloak OTLP-native; miniflux scrape) and proves parity in ClickHouse. W1 adds the 6 alerts as HyperDX TILE alerts and soaks them in parallel with Alertmanager. W2 deletes the kps GitOps app. W3 scrubs every ServiceMonitor/PodMonitor/PrometheusRule object then removes the operator CRDs + prometheus namespace via two sequenced Terraform applies. W4 scrubs docs and captures the ADR.

Tech Stack: ArgoCD GitOps (cluster-app templates + app-configs), Helm (kube-state-metrics 7.4.1, opentelemetry-collector 0.154.0, kube-prometheus-stack 84.5.0), OpenTelemetry Collector (prometheusreceiver static scrape + OTLP export to ClickStack), HyperDX/ClickStack (MongoDB-bootstrapped alerts), Keycloak 26.6.4 operator CR, Terraform (tf/cluster-bootstrap, tf/uptime-kuma, tf/vault), ClickHouse (mcp__clickhouse-ro for verification), jj VCS.

Design spec: docs/engineering/specs/2026-06-30-prometheus-decommission-design.md · Design bead: hl-qjh3

---

Conventions used by every task

VCS (jj-first)

Per the repo’s jj-first policy (root CLAUDE.md → “Branch Protection”; the jj:jujutsu skill): work in the dedicated workspace .worktrees/hl-qjh3-prometheus-decommission. One jj change per task, each with its own bookmark and PR (branch protection blocks direct main). Use --no-pager on every jj command and --git on diffs. Per task:

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/hl-qjh3-prometheus-decommission
jj --no-pager new main@origin -m "<type>(<scope>): <summary> [hl-qjh3.<n>]"
jj --no-pager bookmark create feat/hl-qjh3-<slug> -r @
# ... make edits ...
jj --no-pager diff --git          # review
jj --no-pager bookmark set feat/hl-qjh3-<slug> -r @
jj --no-pager git push -b feat/hl-qjh3-<slug>
gh pr create --head feat/hl-qjh3-<slug> --title "..." --body "..."

After a PR squash-merges: jj git fetch && jj rebase -s "$(jj --no-pager log -r 'roots(trunk()..@)' --no-graph -T 'change_id.short(12)')" -o main@origin --skip-emptied && jj bookmark delete feat/hl-qjh3-<slug>.

Commit-tag numbering: tasks are tagged [hl-qjh3.<n>] by task number. Tasks 4 and 6 are verification-only gates that produce no commit, so the committed tag sequence intentionally skips .4 and .6 (3 → 5 → 7).

The “test” for GitOps tasks (no unit-test TDD)

Infra has no unit tests. Each task’s gate, in order:

1. Render gate — the manifest/chart renders and is valid:

   # Kustomize app-configs:
   kubectl --context fzymgc-house kustomize argocd/app-configs/<app>
   # Helm values for a chart (validate the structure against the real chart):
   helm show values prometheus-community/kube-state-metrics --version 7.4.1 | yq '<path>'

2. Sync gate — after merge, ArgoCD reports Synced + Healthy (ArgoCD owns apply; never kubectl apply — root CLAUDE.md):

   kubectl --context fzymgc-house get application <app> -n argocd -o jsonpath='{.status.sync.status}/{.status.health.status}'
   # Expected: Synced/Healthy

3. Behavioural gate — the per-task ClickHouse parity / alert-fire query below.

ClickHouse parity helper (the W0/W1 acceptance gate)

Run via mcp__clickhouse-ro__run_query (readonly; CLAUDE.md routes telemetry checks to ClickHouse first). Template for “family present from a non-kps source in the last 30 min”:

SELECT count() AS rows, uniqExact(MetricName) AS names
FROM otel_metrics_gauge
WHERE TimeUnix >= now() - INTERVAL 30 MINUTE AND MetricName LIKE '<pattern>';

(Repeat against otel_metrics_sum for counters.) Expected: rows > 0, names matching the family.

Terraform gate (W3 tasks)

cd tf/cluster-bootstrap   # or tf/uptime-kuma / tf/vault
terraform init -backend=false && terraform validate && terraform fmt -check -recursive

Expected: Success! The configuration is valid., no fmt diff. Never commit .terraform.lock.hcl. Terraform in tf/cluster-bootstrap is operator-applied via local-exec (not merge-triggered) — the actual terraform apply is a human step noted in each W3 task.

---

Wave 0 — Parity prep (additive; no removal)

Task 1: Deploy standalone kube-state-metrics, repoint the OTel scrape job

kps bundles KSM (feeds kube_* = 192 series, the top ClickHouse producer). Before kps can go, KSM must run standalone and the OTel scraper must point at it.

Files:

• Create: argocd/app-configs/monitoring-kube-state-metrics/kube-state-metrics-values.yaml
• Create: argocd/cluster-app/templates/monitoring-kube-state-metrics.yaml
• Modify: argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml:150-153
• Step 1: Confirm the standalone KSM service DNS the OTel job will target

The chart (with fullnameOverride: kube-state-metrics) creates Service kube-state-metrics on port 8080 in its namespace. Target DNS will be kube-state-metrics.kube-state-metrics.svc.cluster.local:8080.

• Step 2: Validate the chart values structure against the real chart

Run: helm show values prometheus-community/kube-state-metrics --version 7.4.1 | yq '.fullnameOverride, .service.port, .prometheus.monitor.enabled' Expected: keys exist; prometheus.monitor.enabled default false (we keep it false — OTel scrapes statically, no ServiceMonitor).

• Step 3: Write the KSM values file

argocd/app-configs/monitoring-kube-state-metrics/kube-state-metrics-values.yaml:

# Standalone kube-state-metrics — replaces the kube-state-metrics that
# kube-prometheus-stack used to bundle (hl-qjh3 W0). Scraped statically by
# the monitoring-otel-scraper deployment collector; NO ServiceMonitor
# (prometheus.monitor stays disabled — the operator CRDs are being removed).
fullnameOverride: kube-state-metrics
prometheus:
  monitor:
    enabled: false
resources:
  requests:
    cpu: 10m
    memory: 64Mi
  limits:
    memory: 128Mi

• Step 4: Write the cluster-app Application template

argocd/cluster-app/templates/monitoring-kube-state-metrics.yaml (mirrors the multi-source pattern of monitoring-otel-scraper.yaml):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: monitoring-kube-state-metrics
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: observability
  sources:
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      ref: cluster-repo
    - chart: kube-state-metrics
      repoURL: https://prometheus-community.github.io/helm-charts
      targetRevision: 7.4.1
      helm:
        releaseName: ksm
        valueFiles:
          - $cluster-repo/argocd/app-configs/monitoring-kube-state-metrics/kube-state-metrics-values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-state-metrics
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true

• Step 5: Render gate

Run: helm template ksm prometheus-community/kube-state-metrics --version 7.4.1 -f argocd/app-configs/monitoring-kube-state-metrics/kube-state-metrics-values.yaml | yq 'select(.kind=="Service") | .metadata.name, .spec.ports[0].port' Expected: kube-state-metrics and 8080.

• Step 6: Repoint the OTel scrape job (do NOT remove it — only change the target)

In scrape-collector-values.yaml, the kube-state-metrics job (line ~150-153) currently targets kps-kube-state-metrics.prometheus.svc.cluster.local:8080. Change only the target:

          - job_name: kube-state-metrics
            scrape_interval: 30s
            static_configs:
              - targets: ["kube-state-metrics.kube-state-metrics.svc.cluster.local:8080"]

• Step 7: Commit + PR

Commit per the VCS convention (feat(observability): standalone kube-state-metrics + repoint OTel job [hl-qjh3.1]). Open PR.

• Step 8: Sync gate (after merge)

Run: kubectl --context fzymgc-house get application monitoring-kube-state-metrics -n argocd -o jsonpath='{.status.sync.status}/{.status.health.status}' Expected: Synced/Healthy; kubectl get pods -n kube-state-metrics shows the KSM pod 1/1.

• Step 9: Behavioural gate — kube_* still flowing, now from standalone KSM

ClickHouse: SELECT count() rows, uniqExact(MetricName) names FROM otel_metrics_gauge WHERE TimeUnix >= now() - INTERVAL 30 MINUTE AND MetricName LIKE 'kube\_%'; Expected: rows > 0, names ≈ 192 (parity with the pre-change baseline). Capture the baseline count BEFORE merge for comparison.

---

Task 2: Add the miniflux scrape job (close the orphaned-ServiceMonitor gap)

miniflux has a repo ServiceMonitor but no OTel job, so miniflux_* never reaches ClickHouse.

Files:

• Modify: argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml (add a job near the other app jobs, e.g. after velero ~line 234)

• Step 1: Confirm miniflux metrics endpoint

Run: kubectl --context fzymgc-house get svc -n miniflux -o custom-columns=NAME:.metadata.name,PORTS:.spec.ports[*].port and kubectl get servicemonitor -n miniflux miniflux -o yaml | yq '.spec.endpoints' Expected: a Service exposing the http port; the ServiceMonitor scrapes /metrics on that port. Note the exact service name + port.

• Step 2: Add the scrape job

In scrape-collector-values.yaml add (use the service name/port confirmed in Step 1; miniflux serves /metrics on its main HTTP port, default 8080):

          - job_name: miniflux
            scrape_interval: 30s
            static_configs:
              - targets: ["miniflux.miniflux.svc.cluster.local:8080"]

• Step 3: Commit + PR (feat(observability): scrape miniflux into OTel [hl-qjh3.2]).

• Step 4: Sync gate — monitoring-otel-scraper Synced/Healthy after merge.

• Step 5: Behavioural gate

ClickHouse: SELECT count() rows, uniqExact(MetricName) names FROM otel_metrics_gauge WHERE TimeUnix >= now() - INTERVAL 30 MINUTE AND MetricName LIKE 'miniflux\_%'; Expected: rows > 0 (was 0 before). If miniflux exposes counters, also check otel_metrics_sum.

---

Task 3: Keycloak OTLP-native metrics (drop the scrape dependency)

Keycloak 26.6.4 can push metrics over OTLP via the experimental opentelemetry-metrics feature (owner-accepted experimental). No scrape job is added — Keycloak pushes directly to cs-otel-collector.

Files:

• Modify: argocd/app-configs/keycloak/keycloak-cr.yaml:9 (add spec.features) and :42-46 (extend additionalOptions)

• Step 1: Confirm the OTLP endpoint + feature/option names

Endpoint: cs-otel-collector.clickstack.svc.cluster.local:4317 (gRPC) — same endpoint the otel-scraper exports to. Feature key: opentelemetry-metrics. Options: telemetry-metrics-enabled, telemetry-metrics-endpoint, telemetry-protocol. (Parent OPENTELEMETRY feature is DEFAULT — no need to list it.)

• Step 2: Add the features.enabled block

In keycloak-cr.yaml, under spec: (after instances: 2, before db:), add:

  features:
    enabled:
      - opentelemetry-metrics   # EXPERIMENTAL (Keycloak 26.x) — Micrometer→OTel metrics bridge. hl-qjh3.

• Step 3: Extend additionalOptions

Append to the existing additionalOptions list:

    - name: telemetry-metrics-enabled
      value: "true"
    - name: telemetry-metrics-endpoint
      value: "http://cs-otel-collector.clickstack.svc.cluster.local:4317"
    - name: telemetry-protocol
      value: grpc

(Keep the existing log-console-output: json and metrics-enabled: "true" entries.)

• Step 4: Render gate

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/keycloak | yq 'select(.kind=="Keycloak") | .spec.features, .spec.additionalOptions' Expected: the feature + three new options present; YAML valid.

• Step 5: Commit + PR (feat(keycloak): OTLP-native metrics to ClickStack (experimental) [hl-qjh3.3]).

• Step 6: Sync gate — keycloak Keycloak CR reconciles; pods roll (operator restarts the StatefulSet). kubectl get pods -n keycloak → 2/2 Ready after rollout.

• Step 7: Behavioural gate — keycloak metrics arriving via OTLP

ClickHouse: SELECT DISTINCT ServiceName, uniqExact(MetricName) names FROM otel_metrics_sum WHERE TimeUnix >= now() - INTERVAL 30 MINUTE AND (ServiceName ILIKE '%keycloak%' OR MetricName LIKE 'jvm\_%' OR MetricName LIKE 'http\_server\_%') GROUP BY ServiceName; Expected: keycloak service emitting jvm_* / http_server_* (semconv names may differ — record which families actually land). If nothing lands (experimental feature unstable), apply the documented fallback: revert this task and instead add a scrape job for the management :9000/metrics endpoint.

---

Task 4: W0 exit gate — host-metrics parity check (authorize node-exporter drop)

Before W2 removes node-exporter, confirm hostmetrics system.* is acceptable coverage.

Files: none (verification only).

• Step 1: Confirm hostmetrics coverage

ClickHouse: SELECT uniqExact(MetricName) names, groupArraySample(10)(MetricName) sample FROM otel_metrics_gauge WHERE TimeUnix >= now() - INTERVAL 30 MINUTE AND MetricName LIKE 'system.%'; Expected: system.cpu.*, system.memory.*, system.disk.*, system.network.*, system.cpu.load_average.* present across all 8 nodes (cross-check uniqExact(ResourceAttributes['host.name']) ≈ 8).

• Step 2: Record the W0 baseline + decision

bd note hl-qjh3 "W0 parity verified: kube_* from standalone KSM (~192 names), miniflux_* present, keycloak OTLP metrics <families>, hostmetrics system.* across 8 nodes. node-exporter drop authorized for W2." Expected: note recorded. This is the gate that unblocks W2’s node-exporter removal.

---

Wave 1 — Alert migration (additive; parallel-run)

Task 5: Extend clickstack-alerts with a metric Source + 6 TILE alerts

The existing bootstrap only does log saved_search alerts. This adds the metric-TILE path (per spec §5).

Files:

• Modify: argocd/app-configs/clickstack-alerts/bootstrap-script.yaml (extend bootstrap.js)

• Step 1: Discover the live HyperDX metric schema (the “failing test”)

The repo bootstrap was reverse-engineered from live docs; the TILE/metric schema is not yet used here, so confirm exact field shapes before writing upserts. Exec into the bootstrap-capable context and dump existing shapes:

# Get the Mongo URI the bootstrap job uses, then inspect via a throwaway mongosh:
kubectl --context fzymgc-house get secret cs-clickstack-mongodb-hyperdx-hyperdx -n clickstack \
  -o jsonpath='{.data.connectionString\.standard}' | base64 -d   # note URI (do NOT log it elsewhere)
# In a mongosh against that URI:
#   db.sources.find({ kind: "metric" }).pretty()        // metricTables shape (gauge/sum/histogram)
#   db.dashboards.findOne()                              // tile / chartConfig shape
#   db.alerts.find({ source: "tile" }).pretty()         // tile-alert field names (tileId, dashboardId)

Expected: confirm the field names metricTables.{gauge,sum,histogram}, the tile chartConfig keys (metricType, metricName, aggFn — confirm rate is the valid aggFn name for counter rate, vs sum, avg, count, max, seriesReturnType/asRatio, where/whereLanguage), the exact ClickHouse attribute key for namespace (ResourceAttributes['k8s.namespace.name'] vs Attributes['namespace'] — query a sample otel_metrics_sum row for the otelcol_* series to confirm), and the alert keys (source:"tile", dashboardId, tileId, threshold, thresholdType, interval). If a metric Source does not exist yet, Step 2 creates one.

• Step 2: Add the metric Source upsert

After the logsSource lookup in bootstrap.js, add (adjust table/field names to Step 1 findings):

    // ---- Metric source over ClickHouse otel_metrics_* (hl-qjh3 W1) ----
    let metricSource = db.sources.findOne({ team: teamId, kind: "metric" });
    if (!metricSource) {
      const msId = upsert(
        db.sources,
        { team: teamId, kind: "metric", name: "OTel Metrics" },
        {
          team: teamId, kind: "metric", name: "OTel Metrics",
          connection: logsSource.connection,
          from: { databaseName: "default", tableName: "" },
          metricTables: {
            gauge: "otel_metrics_gauge",
            sum: "otel_metrics_sum",
            histogram: "otel_metrics_histogram"
          }
        }
      );
      metricSource = db.sources.findOne({ _id: msId });
    }
    print("metricSource: " + metricSource._id);

• Step 3: Add a dashboard with one tile per alert metric

Add a dashboard upsert holding 6 tiles. Each tile’s chartConfig carries the metric query. Example shape for the ratio tile (NATSJetStreamStorageHigh) and a simple gauge tile (adjust to Step 1 schema):

    // ---- Alerting dashboard (tiles back the metric alerts) ----
    const tile = (id, name, cfg) => Object.assign({ id, name }, { config: cfg });
    const dashId = upsert(
      db.dashboards,
      { team: teamId, name: "hl-qjh3 metric alerts" },
      {
        team: teamId, name: "hl-qjh3 metric alerts", tags: ["alerts","metrics"],
        tiles: [
          tile("nats-js-storage", "NATS JS storage ratio", {
            source: metricSource._id, displayType: "line", seriesReturnType: "ratio",
            select: [
              { aggFn: "sum", metricType: "gauge", metricName: "nats_varz_jetstream_stats_storage", valueExpression: "Value" },
              { aggFn: "sum", metricType: "gauge", metricName: "nats_varz_jetstream_config_max_storage", valueExpression: "Value" }
            ], where: "", whereLanguage: "sql"
          }),
          tile("nats-conns", "NATS connections", {
            source: metricSource._id, displayType: "line",
            select: [ { aggFn: "sum", metricType: "gauge", metricName: "nats_varz_connections", valueExpression: "Value" } ],
            where: "", whereLanguage: "sql"
          }),
          tile("nats-slow", "NATS slow consumers rate", {
            source: metricSource._id, displayType: "line",
            // rate (NOT sum) — nats_varz_slow_consumers is a monotonic counter; sum would
            // latch the alert permanently above 0. Mirrors PromQL rate(...[5m]).
            select: [ { aggFn: "rate", metricType: "sum", metricName: "nats_varz_slow_consumers", valueExpression: "Value" } ],
            where: "", whereLanguage: "sql"
          }),
          tile("nats-cores", "NATS core count (quorum presence)", {
            source: metricSource._id, displayType: "line",
            select: [ { aggFn: "count", metricType: "gauge", metricName: "nats_varz_cores", valueExpression: "Value" } ],
            where: "", whereLanguage: "sql"
          }),
          tile("cs-export-fail", "ClickStack export failures rate", {
            source: metricSource._id, displayType: "line",
            // rate (NOT sum) — send_failed_log_records_total is a monotonic counter.
            // namespace dimension confirm exact key in Step 1 (k8s.namespace.name vs namespace).
            select: [ { aggFn: "rate", metricType: "sum", metricName: "otelcol_exporter_send_failed_log_records_total", valueExpression: "Value" } ],
            where: "Attributes['exporter'] = 'clickhouse' AND ResourceAttributes['k8s.namespace.name'] = 'clickstack'", whereLanguage: "sql"
          }),
          tile("cs-export-present", "ClickStack exporter presence", {
            source: metricSource._id, displayType: "line",
            select: [ { aggFn: "count", metricType: "sum", metricName: "otelcol_exporter_sent_log_records_total", valueExpression: "Value" } ],
            where: "Attributes['exporter'] = 'clickhouse' AND ResourceAttributes['k8s.namespace.name'] = 'clickstack'", whereLanguage: "sql"
          })
        ]
      }
    );
    print("dashboard: " + dashId);

• Step 4: Add the 6 TILE alert upserts

For each tile, add a source:"tile" alert reusing the existing channel. Example (storage ratio + quorum presence shown; repeat the pattern for the other four):

    upsert(db.alerts, { team: teamId, name: "NATS JetStream storage high" }, {
      team: teamId, source: "tile", dashboardId: dashId, tileId: "nats-js-storage",
      name: "NATS JetStream storage high",
      message: "JetStream storage above {{threshold}} of configured max.",
      threshold: 0.8, thresholdType: "above", interval: "5m", channel: channel, state: "OK"
    });
    upsert(db.alerts, { team: teamId, name: "NATS cluster quorum lost" }, {
      team: teamId, source: "tile", dashboardId: dashId, tileId: "nats-cores",
      name: "NATS cluster quorum lost",
      message: "Fewer than {{threshold}} NATS pods reporting (no quorum / all down).",
      threshold: 2, thresholdType: "below", interval: "1m", channel: channel, state: "OK"
    });
    upsert(db.alerts, { team: teamId, name: "NATS high connection count" }, {
      team: teamId, source: "tile", dashboardId: dashId, tileId: "nats-conns",
      name: "NATS high connection count",
      message: "More than {{threshold}} NATS client connections in {{interval}} — possible leak or scale event.",
      threshold: 1000, thresholdType: "above", interval: "5m", channel: channel, state: "OK"
    });
    upsert(db.alerts, { team: teamId, name: "NATS slow consumers" }, {
      team: teamId, source: "tile", dashboardId: dashId, tileId: "nats-slow",
      name: "NATS slow consumers",
      message: "NATS reporting slow consumers in {{interval}} — messages may be dropping.",
      threshold: 0, thresholdType: "above", interval: "5m", channel: channel, state: "OK"
    });
    upsert(db.alerts, { team: teamId, name: "ClickStack OTel collector export failing" }, {
      team: teamId, source: "tile", dashboardId: dashId, tileId: "cs-export-fail",
      name: "ClickStack OTel collector export failing",
      message: "cs-otel-collector ClickHouse exporter is failing to send records in {{interval}}.",
      threshold: 0, thresholdType: "above", interval: "5m", channel: channel, state: "OK"
    });
    upsert(db.alerts, { team: teamId, name: "ClickStack OTel collector ClickHouse exporter missing" }, {
      team: teamId, source: "tile", dashboardId: dashId, tileId: "cs-export-present",
      name: "ClickStack OTel collector ClickHouse exporter missing",
      message: "No otelcol_exporter_sent_log_records_total{exporter=clickhouse} series for {{interval}} — the ClickHouse exporter is gone.",
      threshold: 1, thresholdType: "below", interval: "10m", channel: channel, state: "OK"
    });
    print("metric tile alerts upserted");

• Step 5: Render gate

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/clickstack-alerts | yq 'select(.kind=="ConfigMap") | .metadata.name' and node --check <(yq '.data."bootstrap.js"' argocd/app-configs/clickstack-alerts/bootstrap-script.yaml) Expected: ConfigMap renders; bootstrap.js is syntactically valid JS.

• Step 6: Commit + PR (feat(clickstack-alerts): metric TILE alerts for the 6 Prometheus rules [hl-qjh3.5]).

• Step 7: Sync gate + bootstrap job ran

After merge, the clickstack-alerts PostSync hook job runs. kubectl --context fzymgc-house get job -n clickstack -l app=clickstack-alerts-bootstrap -o jsonpath='{.items[*].status.succeeded}' → 1. Confirm in HyperDX UI the 6 alerts exist.

---

Task 6: W1 soak + fire verification (the W1→W2 interlock)

Files: none (verification only).

• Step 1: Fire-test each threshold alert

In HyperDX, trip each threshold alert (e.g. temporarily lower a threshold against a live value) and confirm a Pushover notification arrives on cluster-infra. Restore thresholds. Expected: Pushover delivery confirmed for the 4 threshold alerts.

• Step 2: Validate the two no-data / ratio alerts specifically

Confirm the ratio alert (NATS JetStream storage high) evaluates in headless checkAlerts (not just the browser) — check the HyperDX alert history/state for a computed value, not an error. For the presence alerts, confirm HyperDX fires on a genuinely-empty series (validate against a safe target or confirm empty-result semantics from alert state). If a presence alert will not fire on no-data, switch it to the saved_search count fallback (spec §5.2) before proceeding. Expected: ratio alert shows a numeric evaluation; both presence alerts demonstrably fire on absence.

• Step 3: Soak ≥ 24h in parallel with Alertmanager

Both alerting paths stay live ≥ 24h (covers for:10m-guarded alerts + a natural NATS window). Compare HyperDX vs Alertmanager firing over the window. Expected: HyperDX alerts track Alertmanager (no missed fires).

• Step 4: Authorize W2

bd note hl-qjh3 "W1 interlock cleared: 6 HyperDX TILE alerts fire (incl ratio in headless checkAlerts + both no-data presence alerts), Pushover cluster-infra delivery confirmed, 24h parallel soak matches Alertmanager. W2 teardown authorized." Expected: note recorded.

---

Wave 2 — Remove kube-prometheus-stack

Task 7: Delete the kps app + dead scrape jobs + alerting secret + ingress

Files:

• Delete: argocd/cluster-app/templates/monitoring-prometheus.yaml
• Delete: argocd/app-configs/monitoring-prometheus/ (whole dir: values, alertmanager-pushover ExternalSecret, ingressroutes, certificate, kustomization)
• Modify: argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml (remove 4 kps-targeting jobs)
• Step 1: Remove the four kps-targeting OTel scrape jobs

In scrape-collector-values.yaml delete these job blocks (they go to dead targets after kps removal): node-exporter (~263-266), kps-kube-prometheus-stack-alertmanager (~237-240), kps-kube-prometheus-stack-prometheus (~242-245), kps-kube-prometheus-stack-operator (~254-260). Leave every other job (incl. the repointed kube-state-metrics).

• Step 2: Delete the kps cluster-app template

Delete argocd/cluster-app/templates/monitoring-prometheus.yaml.

• Step 3: Delete the kps app-config directory

Delete argocd/app-configs/monitoring-prometheus/ (this removes the alertmanager-pushover ExternalSecret, the prometheus IngressRoute, and the Certificate along with the values).

• Step 4: Render gate

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/monitoring-otel-scraper >/dev/null && echo OK and confirm no remaining repo reference to the deleted app: rg -n 'monitoring-prometheus|kps-kube-prometheus-stack|kps-prometheus-node-exporter' argocd/ Expected: OK; rg returns nothing (or only this plan/spec docs).

• Step 5: Commit + PR (feat(observability): remove kube-prometheus-stack app + dead scrape jobs [hl-qjh3.7]).

• Step 6: Sync gate — kps gone, no orphan

After merge, ArgoCD prunes the monitoring-prometheus Application. Confirm: kubectl --context fzymgc-house get application monitoring-prometheus -n argocd → NotFound; kubectl get pods -n prometheus → only kps-internal pods terminating/gone (the namespace itself is removed in W3).

• Step 7: Behavioural gate — no metric/alert gap, no scrape spam

ClickHouse: re-run the Task 1 kube_* query (still ~192 from standalone KSM) and the Task 4 system.* query (host coverage intact via hostmetrics). Confirm HyperDX alerts still firing and Pushover intact. Check OTel collector logs are free of scrape-failure spam: kubectl --context fzymgc-house logs -n otel-scraper deploy/otel-scrape-opentelemetry-collector --tail=200 | rg -i 'scrape|connection refused|no such host' | head Expected: no errors referencing the removed kps targets.

---

Wave 3 — Scrub CRDs + infra config

Ordering invariant: every ServiceMonitor/PodMonitor/PrometheusRule object must be deleted from the cluster before the operator CRDs are removed. Tasks 8 + 9 clear the objects; Task 10 (a separate Terraform apply) removes the CRDs + namespace.

Task 8: (GitOps) Remove the 5 repo-defined SM/PM/Rule manifests

Files:

• Delete: argocd/app-configs/dolt/servicemonitor.yaml
• Delete: argocd/app-configs/temporal-server/servicemonitor.yaml
• Delete: argocd/app-configs/miniflux/servicemonitor.yaml
• Delete: argocd/app-configs/clickstack/otel-collector-monitoring.yaml (SM + cs-otel-collector PrometheusRule)
• Delete: argocd/app-configs/nats/prometheus-rules.yaml (nats PrometheusRule)
• Modify: the kustomization.yaml in each of dolt/, temporal-server/, miniflux/, clickstack/, nats/ to drop the deleted file from resources:
• Step 1: Find each kustomization reference

Run: rg -n 'servicemonitor.yaml|otel-collector-monitoring.yaml|prometheus-rules.yaml' argocd/app-configs/*/kustomization.yaml Expected: one resources: entry per file above.

• Step 2: Delete the 5 manifests and their kustomization entries

Delete the files; remove each from its kustomization.yaml resources: list.

• Step 3: Render gate

Run for each affected app: kubectl --context fzymgc-house kustomize argocd/app-configs/<app> >/dev/null && echo OK. Confirm: rg -n 'kind:\s*(ServiceMonitor|PodMonitor|PrometheusRule)' argocd/ returns nothing. Expected: all OK; rg empty.

• Step 4: Commit + PR (feat(observability): remove repo Prometheus SM/rule manifests [hl-qjh3.8]).

• Step 5: Sync gate — the 5 apps reconcile; ArgoCD prunes the 4 SM objects + 2 PrometheusRule objects. kubectl get servicemonitor,prometheusrule -A | rg -i 'dolt|temporal|miniflux|cs-otel|nats' → empty.

---

Task 9: (Terraform apply #1) Disable all chart-created ServiceMonitors

Files:

• Modify: tf/cluster-bootstrap/cert-manager.tf:~34 (prometheus.servicemonitor.enabled = false)
• Modify: tf/cluster-bootstrap/argocd.tf:~165 (addPrometheusAnnotations = false + the six serviceMonitor.enabled = true → false)
• Modify (plan-discovered): the chart values for vault, velero, keycloak, authentik (ak-outpost-ldap) and the nats PodMonitor — wherever each enables its SM/PM
• Step 1: Inventory every remaining SM/PM producer

Run: kubectl --context fzymgc-house get servicemonitor,podmonitor -A and for each remaining object trace its owning chart/release (-o yaml | yq '.metadata.labels'). Cross-reference to the chart values that enable it. Known: cert-manager + argocd (Terraform, below). Locate vault/velero/keycloak/authentik/nats-PodMonitor producers. Expected: a complete list mapping each live SM/PM → the file that enables it.

• Step 2: Disable cert-manager + argocd ServiceMonitors (Terraform)

In tf/cluster-bootstrap/cert-manager.tf set prometheus.servicemonitor.enabled = false (line ~34). In tf/cluster-bootstrap/argocd.tf set addPrometheusAnnotations = false (line ~165) and each of the six serviceMonitor.enabled = true → false.

• Step 3: Disable the remaining chart SMs/PM

For each producer found in Step 1 (vault, velero, keycloak, authentik, nats PodMonitor), set its serviceMonitor.enabled/podMonitor.enabled (or equivalent) to false in the owning values/Terraform.

• Step 4: Terraform gate

Run: cd tf/cluster-bootstrap && terraform init -backend=false && terraform validate && terraform fmt -check -recursive Expected: valid, no fmt diff. Confirm .terraform.lock.hcl untracked.

• Step 5: Commit + PR (feat(observability): disable all chart ServiceMonitors [hl-qjh3.9]). Merge GitOps-side values; operator runs terraform apply on main-cluster-bootstrap for the cluster-bootstrap changes.

• Step 6: Verification gate — ZERO SM/PM/rules remain (blocks Task 10)

Run: kubectl --context fzymgc-house get servicemonitor,podmonitor,prometheusrule -A Expected: No resources found. Task 10 MUST NOT start until this is empty.

---

Task 10: (Terraform apply #2) Remove operator CRDs, namespace, and remaining bindings

Files:

• Modify/Delete: tf/cluster-bootstrap/prometheus-crds.tf (the helm_release.prometheus_operator_crds + kubernetes_namespace_v1.prometheus)
• Modify: tf/cluster-bootstrap/variables.tf:~39 (remove prometheus_operator_crds_version)
• Modify: tf/cluster-bootstrap/imports.tf:~5,~36 (remove the prometheus ns + CRD import blocks)
• Modify: tf/cluster-bootstrap/cert-manager.tf:~67, tf/cluster-bootstrap/argocd.tf:~278 (remove depends_on edges to prometheus_operator_crds)
• Modify: tf/uptime-kuma/monitors-services.tf:56-59 (remove the prometheus /-/healthy monitor)
• Modify: tf/vault/k8s-pushover.tf:12 (remove the SA binding to namespace prometheus)
• Modify: argocd/app-configs/velero/backup-schedule.yaml:41,101 (remove prometheus from excludedNamespaces)
• Step 1: Precondition check

Re-run Task 9 Step 6 (kubectl get servicemonitor,podmonitor,prometheusrule -A → empty). If anything remains, STOP and finish Task 9.

• Step 2: Remove the CRD release + namespace + var + imports + depends_on

Edit prometheus-crds.tf (remove both resources), variables.tf (remove the var), imports.tf (remove both import blocks), and the two depends_on prometheus_operator_crds edges in cert-manager.tf/argocd.tf.

• Step 3: Remove the uptime-kuma monitor + Vault SA binding

tf/uptime-kuma/monitors-services.tf: delete the whole resource "uptimekuma_monitor_http" "prometheus" block (starts ~line 57, runs ~13 lines through the closing brace) plus its preceding comment. tf/vault/k8s-pushover.tf: remove prometheus from the bound SA namespaces (line ~12) — Alertmanager is gone.

• Step 4: Remove the Velero exclusion (GitOps)

argocd/app-configs/velero/backup-schedule.yaml: remove the prometheus entry from excludedNamespaces in both schedules (lines ~41, ~101).

• Step 5: Terraform gate (all three workspaces)

Run terraform init -backend=false && terraform validate && terraform fmt -check -recursive in tf/cluster-bootstrap, tf/uptime-kuma, tf/vault. Expected: all valid, no fmt diff.

• Step 6: Commit + PR (feat(observability): remove Prometheus CRDs, namespace, monitors [hl-qjh3.10]). Operator runs terraform apply on main-cluster-bootstrap, uptime-kuma, vault.

• Step 7: Verification gate — full clean

Run: kubectl --context fzymgc-house get ns prometheus → NotFound; kubectl get crd | rg 'monitoring.coreos.com' → empty; kubectl get application -n argocd | rg prometheus → empty. Expected: no prometheus namespace, no operator CRDs, no kps app. Confirm a Velero backup run still succeeds.

---

Wave 4 — Docs + ADR

Task 11: Scrub docs and capture the ADR

Files:

• Modify: argocd/CLAUDE.md (remove the “Prometheus retained” note — also factually stale re: recording rules — and the prometheus row in the Velero exclusion table)
• Modify: docs/operations/alertmanager.md, docs/reference/services.md, docs/reference/technologies.md, docs/operations/clickstack.md, docs/operations/pushover.md, docs/operations/cluster-bootstrap.md, docs/operations/incidents.md
• Create: docs/adr/<bd-id>-prometheus-decommission.md (via capture-adrs)
• Step 1: Find every prometheus doc reference

Run: rg -li 'prometheus|kube-prometheus-stack|alertmanager' docs/ argocd/CLAUDE.md | rg -v 'engineering/(specs|plans)' Expected: the file list above (specs/plans excepted — they are historical record).

• Step 2: Rewrite each reference

Update each doc to reflect: metrics in ClickHouse otel_metrics_*, alerting via HyperDX TILE alerts → Pushover cluster-infra, no Prometheus/Alertmanager. Remove the argocd/CLAUDE.md “Prometheus retained” note and the Velero prometheus exclusion-table row. Rewrite docs/operations/alertmanager.md as a HyperDX-alerting doc (or redirect to the clickstack-alerts pattern).

• Step 3: Docs build gate (Starlight)

Run: pnpm --dir website build Expected: build passes — frontmatter title: present on any new/changed doc; starlight-links-validator passes (internal links use root-relative route slugs with trailing slash, no .md).

• Step 4: Commit + PR (docs(observability): scrub Prometheus refs post-decommission [hl-qjh3.11]).

• Step 5: Capture the ADR

The ADR (superseding the hl-5xgq Prometheus-deferral decision; note the original “recording rules” retention rationale was mistaken) is captured by the capture-adrs auto-fire after this plan is READY. It writes docs/adr/<bd-id>-prometheus-decommission.md + a bd create -t decision record. Expected: ADR file + decision bead exist.

---

Plan-wide acceptance criteria (maps to spec §8)

• No kube-prometheus-stack workloads; no prometheus namespace; no monitoring.coreos.com CRDs.
• kube_* (standalone KSM), miniflux_*, keycloak metrics, and host system.* all present in ClickHouse from non-kps sources.
• All 6 former PromQL alerts fire as HyperDX TILE alerts → Pushover cluster-infra.
• No PromQL consumers remain (already confirmed: no adapter/HPA/KEDA).
• kubectl get servicemonitor,podmonitor,prometheusrule -A → empty.
• Docs + argocd/CLAUDE.md scrubbed; ADR captured; Velero backup still green.