Full Grafana-Labs Sunset Design

For Claude: REQUIRED SUB-SKILL: Use superpowers:writing-plans to turn this spec into a task-by-task implementation plan, then superpowers:executing-plans to implement it wave-by-wave.

Goal: Remove the entire Grafana-Labs observability stack — Grafana UI, grafana-operator, grafana-mcp, the CNPG db-grafana Postgres database, Loki, and the host-level Grafana Alloy agent — together with all supporting Terraform/IAM/DNS/backup wiring. The stack is fully superseded by HyperDX/ClickStack (per the 2026-06-15 operator decision; engram memory 67a2c263). Prometheus (kube-prometheus-stack) is explicitly retained as the cluster’s metrics/alerting backbone.

Architecture: A dependency-ordered, six-wave teardown spanning all three layers (ArgoCD GitOps, Terraform/HCP, Ansible), each wave landing as its own PR. The ordering is dictated by hard removal constraints — operator finalizers, ArgoCD prune cascades, Vector’s dual-write log path, and CNPG’s retain-by-default reclaim policy — not by convenience. This mirrors the repository’s established Calico-removal precedent (cilium-pr1..pr5 + tracking plan doc).

Design bead: hl-nlxl.

Background & rationale

The observability stack already migrated to HyperDX/ClickStack: dashboards are MongoDB-stored in HyperDX (memory c6c4c846), alerts run via a ClickStack bootstrap Job to Pushover (memory d6444e83), and OTLP logs/metrics/traces all land in ClickHouse default.otel_{logs,metrics_*,traces}. The 2026-06-15 operator decision (67a2c263) directed that Grafana be retired, that grafana component version bumps be dropped from dependency-upgrade waves, and that grafana-operator removal was pending cleanup work (sibling to the Calico removal, hl-onvz.18). This spec executes that decision in full.

Grafana-Labs components on the chopping block are either redundant with ClickStack (Grafana UI, Loki) or pinned to unmaintained/dead chart releases (grafana/loki 7.0.0 is the GEL-only release; memory ffb43c27). The host Alloy agent and one in-cluster OTLP target already point at removed infrastructure (see W6) — their removal is cleanup of broken config, not a behavior change.

Scope

In scope (removed)

ArgoCD: grafana, grafana-operator, grafana-mcp, monitoring-loki Applications and their app-configs/*; the CNPG db-grafana database; Velero backup entries; the grafana-OCI Helm repo; the ClickStack NetworkPolicy rule for the grafana namespace; the Vector loki sink.
Terraform: tf/grafana/ module + its HCP workspace; tf/authentik/grafana.tf + group/user references; the grafana HCP workspace definition + Vault JWT role + policy; Grafana Vault secret paths; the grafana and grafana_mcp Uptime Kuma monitors; the host-Alloy Vault resources (tf/vault/alloy.tf).
Ansible: roles/alloy/ and the Firewalla host deployment it manages.
Docs/meta: Grafana skill routing and telemetry-table references in the four CLAUDE.md files and tools/docs-audit; retirement of the fzymgc-house:grafana skill.

Out of scope (retained)

Prometheus / kube-prometheus-stack — kept. It actively scrapes via ServiceMonitors for dolt, miniflux, nats, temporal-server, and the ClickStack otel-collector, and its Grafana subchart is already enabled: false. It is a metrics/alerting backbone, not a Grafana UI component.
HyperDX / ClickStack — the replacement; unaffected.
Vector (monitoring-vector) — retained; only its loki sink is removed (the clickhouse sink stays).
Historical docs/engineering/{plans,specs} Grafana documents — retained as dated record.

Load-bearing constraints

Operator finalizers (context7-confirmed). grafana-operator places finalizers on its CRs (Grafana, GrafanaDashboard, GrafanaContactPoint, GrafanaNotificationPolicy, GrafanaAlertRuleGroup). “If the operator is not running, resources with finalizers cannot be deleted.” CRs must be deleted while the operator is alive, or the namespace hangs in Terminating. There are two Grafana instances — app-configs/grafana/grafana.yaml (ns grafana) and app-configs/grafana-operator/grafana.yaml (ns grafana-operator) — both must be removed in this step. Escape hatch: kubectl patch <cr> -p '{"metadata":{"finalizers":null}}' --type=merge.
ArgoCD resources-finalizer cascade (memory 45dfa12c). Deleting an Application with the finalizer prunes everything it owns — including PVCs/DBs. Anything to preserve must carry argocd.argoproj.io/sync-options: Prune=false before the Application is removed.
Vector dual-writes logs. monitoring-vector ships to both a loki sink (loki-gateway.loki.svc:80) and a clickhouse sink (otel_logs). The loki sink + k8s_logs_for_loki transform must be removed before Loki is deleted.
CNPG retains by default. db-grafana.yaml declares a CNPG Database CR with ensure: present. Deleting the CR or pruning the Application does not drop the Postgres database (reclaim defaults to retain). To actually drop it, set ensure: absent first (CNPG runs DROP DATABASE), then remove the manifest and the grafana database entry in postgres-cluster.yaml.
HCP Terraform is GitOps-only. Workspace destroys land through the PR’s speculative→apply run; never run terraform apply/destroy manually. The PR’s speculative plan is the verification authority (uptime-kuma precedent, memory 2bc8c657).
Vault policy hygiene. Removing a secret path requires updating the policies that grant it (repo rule: update Vault policies when secret paths change).

Wave plan

Each wave is one PR. A wave merges only after the prior wave is verified healthy in-cluster.

W1 — Pre-flight + sever Grafana-specific dependencies

Take a final CNPG logical backup of the grafana database (snapshot safety net per 45dfa12c); record the R2 loki-{chunks,ruler,admin} bucket inventory.
Remove the “Allow Grafana to query ClickHouse datasource” ingress rule (source namespace grafana) from argocd/app-configs/clickstack/networkpolicy.yaml.
Risk: low. No Grafana-serving resources removed yet.

W2 — Remove Grafana custom resources (operator stays running)

Delete the grafana Application (argocd/cluster-app/templates/grafana.yaml) and its argocd/app-configs/grafana/ tree: the Grafana instance CR, GrafanaDashboard/GrafanaDatasource CRs (dashboards/, datasources/), notification-policy.yaml, pushover-contact-points.yaml, certificate, OIDC-creds and admin ExternalSecrets.
Remove the second Grafana CR at argocd/app-configs/grafana-operator/grafana.yaml (ns grafana-operator), keeping the operator Deployment + CRDs so finalizers process.
Verify the grafana namespace fully drains (kubectl get ns grafana → NotFound, no CRs stuck on finalizers) before W3.

W3 — Remove operator + grafana-mcp + backing database

Drop the Postgres objects via CNPG’s declarative reclaim (retain is the default — deleting the manifests alone leaves the DB and role on disk). Two ensure: present declarations must be flipped to ensure: absent, applied, and confirmed dropped before the manifests are removed:
- the Database CR in argocd/app-configs/cnpg/db-grafana.yaml (drops the grafana database);
- the managed.roles[name=grafana] stanza in argocd/app-configs/cnpg/postgres-cluster.yaml (drops the grafana Postgres role, backed by main-grafana-credentials).
Then remove db-grafana.yaml, the - db-grafana.yaml line in cnpg/kustomization.yaml, and the managed.roles[name=grafana] stanza.
Delete the grafana-operator Application + app-configs/grafana-operator/ (operator, namespace) and the grafana-mcp Application + app-configs/grafana-mcp/.
Manually remove the orphaned grafana.integreatly.org CRDs (grafanas, grafanadashboards, grafanadatasources, grafanacontactpoints, grafananotificationpolicies, grafanaalertrulegroups, etc.). CRDs carrying helm.sh/resource-policy: keep survive the ArgoCD Application cascade-delete and orphan — they must be deleted with kubectl delete crd after the operator is gone (reusable lesson from the tailscale-operator retirement, engram 87d3a170). This must run after W2 confirmed no CR instances remain (a leftover CR with a finalizer + no operator would block CRD deletion).
Remove the grafana-operator and grafana-mcp namespace entries from argocd/app-configs/velero/backup-schedule.yaml.
Remove the ghcr-grafana-helm-oci repository from argocd/app-configs/shared-resources/argocd-repositories.yaml (consumed only by grafana-operator).
Verify grafana-operator/grafana-mcp namespaces NotFound; ClickStack telemetry still flowing.

W4 — Terraform / IAM teardown

Remove tf/grafana/ (entire module: contact_points.tf, dashboard_temporal.tf, datasources.tf, service-accounts.tf, vault.tf, main.tf, outputs.tf, variables.tf, terraform.tf, versions.tf); the HCP workspace destroy applies via its run.
Remove tf/authentik/grafana.tf + GRAFANA_IDS.txt, and the grafana references in tf/authentik/groups.tf and users.tf.
Remove the grafana HCP workspace from tf/hcp-terraform/workspaces.tf, the Vault JWT role in tf/vault/jwt-hcp-terraform.tf, and the grafana stanza in tf/vault/policy-terraform-workspaces.tf.
Delete the Grafana Vault secret paths (admin, OIDC creds, pushover tokens) and update granting policies.
Remove the grafana monitor (uptimekuma_monitor_http.grafana, in tf/uptime-kuma/main.tf) and the grafana_mcp monitor (uptimekuma_monitor_http.grafana_mcp, in tf/uptime-kuma/monitors-services.tf), updating tf/uptime-kuma/outputs.tf and README.md.
Verify each HCP run’s speculative plan shows only the intended destroys; Authentik app, Vault paths, and both Uptime Kuma monitors are gone.

W5 — Loki retirement

Remove the loki sink and the k8s_logs_for_loki transform from argocd/app-configs/monitoring-vector/vector.yaml; confirm the clickhouse sink still carries logs to otel_logs.
Delete the monitoring-loki Application (argocd/cluster-app/templates/monitoring-loki.yaml) and the loki-s3 ExternalSecret.
Empty and delete the R2 loki-{chunks,ruler,admin} buckets (after the short retention window from W1).
Verify Vector healthy, no writers/readers reference Loki, logs still land in ClickStack.

W6 — Host Alloy decommission + stale-endpoint cleanup + docs

Remove ansible/roles/alloy/ and tear down the Firewalla deployment it manages (Docker container, boot script, Vault Agent credentials, directories) via the role’s teardown path.
Tear down the host-Alloy Vault resources in tf/vault/alloy.tf: vault_kv_secret_v2.alloy (path fzymgc-house/cluster/alloy), vault_policy.alloy_agent (alloy-agent), vault_approle_auth_backend_role.alloy_agent, and random_password.alloy_external_token. The KV secret carries a documented terraform state rm teardown note in-file — follow that procedure so the destroy doesn’t fail on a protected resource.
Repoint the stale OTLP endpoint in argocd/cluster-app/templates/temporal-workers.yaml from alloy.alloy.svc.cluster.local:4317 (removed K8s Alloy) to cs-otel-collector.clickstack.svc.cluster.local:4317.
Update CLAUDE.md (root), argocd/CLAUDE.md, tf/CLAUDE.md, ansible/CLAUDE.md: drop the fzymgc-house:grafana skill routing and the Grafana/Loki/Alloy telemetry-table entries; update tools/docs-audit/groom-manifest.md; tidy the commented-out grafana service-URL references in tf/cloudflare/variables.tf.
Retire the fzymgc-house:grafana skill.
Add a short note documenting the Prometheus-retained rationale.
Verify Firewalla no longer runs Alloy; temporal worker telemetry reaches ClickStack.

Data disposition

Chosen approach: snapshot/export, then hard-delete. Dashboards and alerts are already in HyperDX, so the Grafana DB and Loki buckets are expected to hold nothing of unique value — but a reversible safety net is kept:

Grafana Postgres (db-grafana): final CNPG backup in W1 → ensure: absent drop in W3.
Loki R2 buckets: inventory recorded in W1 → buckets emptied/deleted in W5 after a short retention window.

Rollback

Each wave is an isolated PR → bisectable; ArgoCD auto-sync applies on merge, so reverting the PR re-creates the resources (except hard-deleted data).
Finalizer-stuck namespace: kubectl patch <cr> -p '{"metadata":{"finalizers":null}}' --type=merge, then re-sync.
Data recovery within the retention window: restore the CNPG backup / re-create buckets from the W1 inventory.
Terraform: a destroy can be reverted by re-applying the module from git history (HCP re-creates the workspace) before secrets are purged from Vault; after W4 the IAM artifacts are gone and would need re-provisioning.

Verification (per wave)

ArgoCD apps Healthy/Synced (or absent — no OutOfSync/Missing orphans).
kubectl get ns grafana grafana-operator grafana-mcp loki → NotFound (never stuck Terminating).
ClickStack default.otel_{logs,metrics_*,traces} show fresh rows for representative services after each wave.
Prometheus targets remain up for dolt, miniflux, nats, temporal-server, clickstack otel-collector.
HCP workspace destroyed (zero managed resources); Authentik Grafana app, Vault Grafana paths, and both Uptime Kuma monitors (grafana, grafana_mcp) are gone.

Open questions

Loki retention window length. The concrete number of days to retain the R2 bucket inventory and the CNPG backup before W5/W3 deletion — operator to confirm during planning.

Resolved during review

grafana-mcp Helm chart source — closed. argocd/cluster-app/templates/grafana-mcp.yaml sources its chart from https://grafana.github.io/helm-charts (HTTP), not the ghcr-grafana-helm-oci OCI repo. The OCI repo is grafana-operator-only, so its W3 removal carries no risk to grafana-mcp.