Full Grafana-Labs Sunset Implementation Plan

Full Grafana-Labs Sunset Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Remove the entire Grafana-Labs observability stack (Grafana UI, grafana-operator, grafana-mcp, the CNPG grafana database, Loki, host Alloy) and all its Terraform/IAM/DNS/backup wiring, which is fully superseded by HyperDX/ClickStack — while retaining Prometheus.

Architecture: A six-wave, dependency-ordered teardown across ArgoCD (GitOps), Terraform/HCP, and Ansible. Each wave is its own PR that must merge and reconcile in-cluster (ArgoCD auto-sync) before the next begins, because later waves’ verification depends on earlier waves’ live state. Ordering is dictated by hard removal constraints: operator finalizers, ArgoCD prune cascades, Vector’s dual-write log path, and CNPG’s retain-by-default reclaim policy.

Tech Stack: Jujutsu (jj) VCS, ArgoCD, Kubernetes/k3s, grafana-operator (grafana.integreatly.org), CloudNativePG (postgresql.cnpg.io), Vector, Helm, HCP Terraform (tfe/vault/authentik/grafana/uptimekuma providers), Ansible, Cilium NetworkPolicy.

Spec: docs/engineering/specs/2026-06-25-grafana-labs-sunset-design.md Design bead: hl-nlxl

---

Conventions for every wave

• VCS (jj): This is a colocated jj repo. Per wave, start from current main:
  • jj git fetch
  • jj new main@origin then jj bookmark create feat/grafana-sunset-wN -r @ (replace N)
  • Make edits, jj describe -m "<conventional commit msg>", jj git push -b feat/grafana-sunset-wN
  • gh pr create --head feat/grafana-sunset-wN --title "..." --body "..."
  • MUST NOT push to main directly; MUST NOT kubectl apply app manifests (ArgoCD owns sync). The one sanctioned manual kubectl mutation is the CRD deletion in Wave 3, Step 8.
• Merge gate: Each PR merges only after its predecessor’s resources are confirmed gone/healthy in-cluster. Waves are NOT parallel.
• Verification kubectl is read-only (get/describe). Telemetry checks query ClickHouse (ClickStack MCP) or the HyperDX UI.
• AI authorship byline on every commit and PR body.
• Model intent (all tasks): mechanical GitOps/file removal + read-only verification → dispatch beads with model:sonnet. The pure-deletion-plus-edit waves (W1, W2, W6) are fine on model:haiku; W3 and W4 keep model:sonnet for the CNPG drop-sequencing and HCP/Vault teardown nuance. No task needs opus.

File map (what each wave touches)

Wave / PR	Files
W1 Pre-flight + sever deps	argocd/app-configs/clickstack/networkpolicy.yaml (edit); out-of-band: CNPG backup, R2 inventory
W2 Grafana CRs (operator alive)	Delete argocd/cluster-app/templates/grafana.yaml, argocd/app-configs/grafana/**; edit argocd/app-configs/grafana-operator/kustomization.yaml + delete argocd/app-configs/grafana-operator/grafana.yaml
W3 Operator + mcp + DB + CRDs	Delete argocd/cluster-app/templates/grafana-operator.yaml, argocd/cluster-app/templates/grafana-mcp.yaml, argocd/app-configs/grafana-operator/**, argocd/app-configs/grafana-mcp/**; edit argocd/app-configs/cnpg/db-grafana.yaml, argocd/app-configs/cnpg/kustomization.yaml, argocd/app-configs/cnpg/postgres-cluster.yaml, argocd/app-configs/velero/backup-schedule.yaml, argocd/app-configs/shared-resources/argocd-repositories.yaml; manual kubectl delete crd
W4 Terraform / IAM	Delete tf/grafana/**, tf/authentik/grafana.tf, tf/authentik/GRAFANA_IDS.txt; edit tf/authentik/users.tf, tf/authentik/groups.tf, tf/hcp-terraform/workspaces.tf, tf/vault/jwt-hcp-terraform.tf, tf/vault/policy-terraform-workspaces.tf, tf/uptime-kuma/main.tf, tf/uptime-kuma/monitors-services.tf, tf/uptime-kuma/outputs.tf
W5 Loki retirement	Edit argocd/app-configs/monitoring-vector/vector.yaml; delete argocd/cluster-app/templates/monitoring-loki.yaml + argocd/app-configs/monitoring-loki/** (2nd git source: kustomization.yaml + external-secret-s3.yaml/loki-s3); out-of-band: R2 bucket deletion
W6 Host Alloy + endpoints + docs	Delete ansible/roles/alloy/**, tf/vault/alloy.tf; edit argocd/cluster-app/templates/temporal-workers.yaml, CLAUDE.md, argocd/CLAUDE.md, tf/CLAUDE.md, ansible/CLAUDE.md, tf/cloudflare/variables.tf, tools/docs-audit/groom-manifest.md

---

Task 1: Wave 1 — Pre-flight backup + sever Grafana-specific dependencies

Goal: Capture reversible data snapshots and remove the one cluster dependency that points into the Grafana namespace, so nothing else references grafana before the stack comes down. Lowest-risk wave; removes no Grafana-serving resources.

Files:

• Modify: argocd/app-configs/clickstack/networkpolicy.yaml (remove lines 17-20)

• Step 1: Take a final logical backup of the Grafana Postgres database

This is the data safety net (reversible per engram 45dfa12c). Run against the CNPG main cluster (database grafana, owner role grafana):

kubectl -n postgres exec main-1 -c postgres -- \
  pg_dump -U postgres -d grafana --no-owner --format=custom \
  > /tmp/grafana-db-$(date +%Y%m%d).dump
ls -la /tmp/grafana-db-*.dump   # confirm non-empty

Expected: a non-zero-byte dump file. Stash it off-cluster (operator’s secure storage) and note its location in the bead.

• Step 2: Record the Loki R2 bucket inventory

Capture object counts/sizes for the three buckets so W5’s deletion is auditable. Using the rclone/aws profile already configured for the cluster’s R2:

for b in chunks ruler admin; do
  echo "== loki-$b =="; rclone size "r2:$(loki bucket prefix)-$b" 2>/dev/null || echo "(record manually from R2 dashboard)"
done

Expected: a recorded size/object-count per bucket (exact bucket prefix is in the loki-s3 secret’s S3_BUCKET). Note results in the bead. No deletion yet.

• Step 3: Remove the Grafana ingress rule from the ClickStack NetworkPolicy

In argocd/app-configs/clickstack/networkpolicy.yaml, delete this block (the comment + its fromEndpoints entry):

    # Allow Grafana to query ClickHouse datasource
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: grafana

Leave the clickstack and clickstack-operators fromEndpoints entries intact.

• Step 4: Commit and open the PR

jj describe -m "feat(observability): sever Grafana NetworkPolicy dep + pre-flight backups (grafana-sunset W1) [hl-nlxl]"
jj git push -b feat/grafana-sunset-w1
gh pr create --head feat/grafana-sunset-w1 --title "Grafana sunset W1: pre-flight + sever deps" --body "Wave 1 of the Grafana-Labs sunset (spec: docs/engineering/specs/2026-06-25-grafana-labs-sunset-design.md). Removes the grafana-namespace ingress rule from the ClickStack CiliumNetworkPolicy; CNPG backup + R2 inventory captured out-of-band."

• Step 5: Verify after merge + ArgoCD sync

kubectl -n argocd get application clickstack -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'
kubectl -n clickstack get ciliumnetworkpolicy -o yaml | rg -c 'namespace: grafana'

Expected: Synced Healthy; the grep count is 0. ClickStack still ingesting (spot-check otel_logs has rows from the last 5 min via ClickStack MCP).

---

Task 2: Wave 2 — Remove Grafana custom resources (operator stays running)

Goal: Delete every operator-managed Grafana CR while the operator Deployment is still alive, so its finalizers clean up cleanly and the grafana namespace drains. The operator + CRDs are left in place for Wave 3.

Constraint: grafana-operator CRs (Grafana, GrafanaDashboard, GrafanaDatasource, GrafanaContactPoint, GrafanaNotificationPolicy, GrafanaAlertRuleGroup) carry finalizers — they are unremovable if the operator is not running. There are two Grafana instances: one in ns grafana, one in ns grafana-operator.

Files:

• Delete: argocd/cluster-app/templates/grafana.yaml (the grafana Application)
• Delete: argocd/app-configs/grafana/** (entire directory: grafana.yaml, dashboards/, datasources/, notification-policy.yaml, pushover-contact-points.yaml, *-external-secret.yaml, grafana-certificate.yaml, grafana-oidc-creds.yaml, namespace.yaml, kustomization.yaml)
• Delete: argocd/app-configs/grafana-operator/grafana.yaml (the second Grafana CR, ns grafana-operator)
• Modify: argocd/app-configs/grafana-operator/kustomization.yaml (drop the grafana.yaml resource entry)
• Step 1: Confirm the operator is healthy before deleting CRs

kubectl -n grafana-operator get deploy grafana-operator -o jsonpath='{.status.readyReplicas}{"\n"}'

Expected: 1 (operator running — required for finalizer processing).

• Step 2: Delete the grafana Application template and its app-config tree

jj new main@origin
jj bookmark create feat/grafana-sunset-w2 -r @
rm argocd/cluster-app/templates/grafana.yaml
rm -rf argocd/app-configs/grafana

• Step 3: Remove the second Grafana CR from the operator app-config

rm argocd/app-configs/grafana-operator/grafana.yaml

Then edit argocd/app-configs/grafana-operator/kustomization.yaml and delete the line referencing it:

  - grafana.yaml

Leave namespace.yaml (and any operator-chart resources) in place — the operator must survive this wave.

• Step 4: Commit and open the PR

jj describe -m "feat(observability): remove Grafana CRs, keep operator (grafana-sunset W2) [hl-nlxl]"
jj git push -b feat/grafana-sunset-w2
gh pr create --head feat/grafana-sunset-w2 --title "Grafana sunset W2: remove Grafana CRs (operator alive)" --body "Wave 2. Deletes the grafana Application + app-configs/grafana tree and the second Grafana CR in grafana-operator ns, leaving the operator running so finalizers process. Requires W1 merged."

• Step 5: Verify the grafana namespace fully drains (no stuck finalizers)

After merge + ArgoCD sync (the grafana Application disappears; prune removes its resources):

kubectl get ns grafana 2>&1            # Expected: NotFound (eventually)
kubectl get grafanas.grafana.integreatly.org -A 2>&1   # Expected: No resources found
kubectl -n grafana-operator get grafana 2>&1           # Expected: No resources found

Expected: namespace grafana is NotFound (not stuck Terminating); zero Grafana CRs remain. If a CR is stuck on a finalizer (operator already gone for any reason), apply the escape hatch: kubectl patch grafana <name> -n <ns> -p '{"metadata":{"finalizers":null}}' --type=merge.

• Step 6: Confirm the operator is still running

kubectl -n argocd get application grafana-operator -o jsonpath='{.status.health.status}{"\n"}'

Expected: Healthy (operator + CRDs untouched; Wave 3 removes them).

---

Task 3: Wave 3 — Remove operator + grafana-mcp + backing database + CRDs

Goal: Tear down the operator, grafana-mcp, the CNPG grafana database/role, and the orphaned CRDs. Drop the Postgres objects via CNPG’s declarative reclaim (retain is the default — deleting manifests alone leaves them on disk).

Files:

• Modify then delete: argocd/app-configs/cnpg/db-grafana.yaml
• Modify: argocd/app-configs/cnpg/postgres-cluster.yaml (the managed.roles[name=grafana] stanza)
• Modify: argocd/app-configs/cnpg/kustomization.yaml (drop - db-grafana.yaml)
• Delete: argocd/cluster-app/templates/grafana-operator.yaml, argocd/cluster-app/templates/grafana-mcp.yaml
• Delete: argocd/app-configs/grafana-operator/**, argocd/app-configs/grafana-mcp/**
• Modify: argocd/app-configs/velero/backup-schedule.yaml (4 lines), argocd/app-configs/shared-resources/argocd-repositories.yaml (one Secret doc)
• Step 1: Drop the Postgres database — flip ensure to absent first

In argocd/app-configs/cnpg/db-grafana.yaml, change the Database CR:

spec:
  cluster:
    name: main
  ensure: absent   # was: present
  name: grafana
  owner: grafana

Also remove the main-grafana-credentials ExternalSecret document from the same file (it lives in ns postgres).

• Step 2: Drop the Postgres role — flip ensure to absent

In argocd/app-configs/cnpg/postgres-cluster.yaml, change the managed.roles entry:

      - name: grafana
        ensure: absent   # was: present
        passwordSecret:
          name: main-grafana-credentials
        superuser: false
        login: true
        connectionLimit: -1
        inherit: true

• Step 3: Commit the drop-only change as its own PR and let it apply

Drop must reconcile before the manifests are removed (removing the manifest while ensure: absent hasn’t applied leaves the DB on disk).

jj new main@origin
jj bookmark create feat/grafana-sunset-w3a -r @
jj describe -m "feat(cnpg): drop grafana database + role (ensure=absent) (grafana-sunset W3a) [hl-nlxl]"
jj git push -b feat/grafana-sunset-w3a
gh pr create --head feat/grafana-sunset-w3a --title "Grafana sunset W3a: drop grafana DB + role" --body "Sets the grafana CNPG Database CR and managed role to ensure=absent so CNPG drops them. Verify dropped before W3b removes the manifests."

• Step 4: Verify the database and role are dropped

After merge + sync:

kubectl -n postgres exec main-1 -c postgres -- psql -U postgres -tAc "SELECT 1 FROM pg_database WHERE datname='grafana';"
kubectl -n postgres exec main-1 -c postgres -- psql -U postgres -tAc "SELECT 1 FROM pg_roles WHERE rolname='grafana';"

Expected: both return empty (no rows) — database and role gone.

• Step 5: Remove the now-empty DB manifests + Applications + backup/repo refs

jj new main@origin
jj bookmark create feat/grafana-sunset-w3b -r @
rm argocd/app-configs/cnpg/db-grafana.yaml
rm argocd/cluster-app/templates/grafana-operator.yaml
rm argocd/cluster-app/templates/grafana-mcp.yaml
rm -rf argocd/app-configs/grafana-operator argocd/app-configs/grafana-mcp

Edit argocd/app-configs/cnpg/kustomization.yaml — delete:

  - db-grafana.yaml

Then edit argocd/app-configs/cnpg/postgres-cluster.yaml and delete the now-dead managed.roles stanza (the drop already applied in W3a; the spec requires removing the stanza, not leaving a permanent ensure: absent block):

      - name: grafana
        ensure: absent
        passwordSecret:
          name: main-grafana-credentials
        superuser: false
        login: true
        connectionLimit: -1
        inherit: true

• Step 6: Remove the Velero backup entries

In argocd/app-configs/velero/backup-schedule.yaml, delete all four Grafana lines (two schedules × grafana-operator + grafana-mcp):

      - grafana-operator

      - grafana-mcp

(They appear near lines 26, 39, 85, 98. Remove each list item; leave surrounding namespaces intact.)

• Step 7: Remove the grafana-operator OCI Helm repo

In argocd/app-configs/shared-resources/argocd-repositories.yaml, delete the entire ghcr-grafana-helm-oci Secret document (consumed only by the now-deleted grafana-operator Application):

---
apiVersion: v1
kind: Secret
metadata:
  name: ghcr-grafana-helm-oci
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
  url: ghcr.io/grafana/helm-charts
  name: ghcr-grafana-charts
  type: helm
  enableOCI: "true"

• Step 8: Commit, open PR, and after merge manually delete the orphaned CRDs

jj describe -m "feat(observability): remove grafana-operator + grafana-mcp + DB manifests (grafana-sunset W3b) [hl-nlxl]"
jj git push -b feat/grafana-sunset-w3b
gh pr create --head feat/grafana-sunset-w3b --title "Grafana sunset W3b: remove operator, mcp, DB, backups" --body "Removes the grafana-operator + grafana-mcp Applications/app-configs, the dropped db-grafana manifests, Velero excludes, and the grafana OCI repo. Requires W3a verified."

After merge + sync, the grafana.integreatly.org CRDs survive the Application cascade-delete (they carry helm.sh/resource-policy: keep; reusable lesson from the tailscale-operator retirement, engram 87d3a170). Delete them manually — this is the one sanctioned manual mutation:

kubectl get crd -o name | rg 'grafana\.integreatly\.org'
kubectl delete crd \
  grafanas.grafana.integreatly.org \
  grafanadashboards.grafana.integreatly.org \
  grafanadatasources.grafana.integreatly.org \
  grafanacontactpoints.grafana.integreatly.org \
  grafananotificationpolicies.grafana.integreatly.org \
  grafanaalertrulegroups.grafana.integreatly.org \
  grafanafolders.grafana.integreatly.org \
  grafanalibrarypanels.grafana.integreatly.org 2>&1

(Delete only the CRDs that the first command lists — names may vary by operator version.)

• Step 9: Verify

kubectl get ns grafana-operator grafana-mcp 2>&1          # Expected: NotFound for both
kubectl get crd -o name | rg -c 'grafana\.integreatly\.org'   # Expected: 0
kubectl -n argocd get applications | rg -i 'grafana'      # Expected: no rows

Expected: both namespaces NotFound, zero grafana CRDs, no grafana Applications. Prometheus targets still up (Task 6 has the full check); ClickStack still ingesting.

---

Task 4: Wave 4 — Terraform / IAM teardown

Goal: Remove all Terraform-managed Grafana identity/secret/monitoring artifacts. Authentik OIDC, Vault policies, and Uptime Kuma target live services and destroy cleanly; tf/grafana targets the now-dead Grafana API and needs a state-level teardown.

Files:

• Delete: tf/grafana/**, tf/authentik/grafana.tf, tf/authentik/GRAFANA_IDS.txt
• Modify: tf/authentik/users.tf, tf/authentik/groups.tf, tf/hcp-terraform/workspaces.tf, tf/vault/jwt-hcp-terraform.tf, tf/vault/policy-terraform-workspaces.tf, tf/uptime-kuma/main.tf, tf/uptime-kuma/monitors-services.tf, tf/uptime-kuma/outputs.tf
• Step 1: Tear down the main-cluster-grafana HCP workspace (dead-provider path)

The Grafana provider objects were already destroyed with Grafana (W2/W3), so the workspace state references nonexistent objects and a normal destroy fails on provider connect. Operator action (HCP, out-of-band — GitOps cannot force-delete a workspace it is mid-removing):

1. In HCP, queue a destroy on main-cluster-grafana; if it errors on the Grafana provider, terraform state rm the stale resources (or enable force-delete), leaving empty state.
2. Confirm the workspace has zero managed resources.

Record the outcome in the bead. Only proceed once the workspace is empty.

• Step 2: Remove the tf/grafana module and the workspace definition

jj new main@origin
jj bookmark create feat/grafana-sunset-w4 -r @
rm -rf tf/grafana

In tf/hcp-terraform/workspaces.tf, delete the workspace block:

    main-cluster-grafana = {
      dir  = "tf/grafana"
      tags = ["main-cluster", "grafana"]
    }

• Step 3: Remove the Authentik Grafana OIDC + group membership

rm tf/authentik/grafana.tf tf/authentik/GRAFANA_IDS.txt

In tf/authentik/users.tf, delete the group-membership line:

    authentik_group.grafana_admin.id,

In tf/authentik/groups.tf, update the comment on line 5 to drop the grafana.tf reference (it now reads # Integration-specific groups are in: argocd.tf, vault.tf, etc.).

• Step 4: Remove the Vault JWT role and policies

In tf/vault/jwt-hcp-terraform.tf, delete the whole vault_jwt_auth_backend_role.tfc_grafana resource (the block resource "vault_jwt_auth_backend_role" "tfc_grafana" { ... }).

In tf/vault/policy-terraform-workspaces.tf:

• Delete the standalone resource "vault_policy" "terraform_grafana_admin" { ... } block (the # Grafana workspace - manages Grafana secrets... resource).
• Delete the inline grafana path blocks embedded in the broader workspace policy:

path "secret/data/fzymgc-house/cluster/grafana" {
  capabilities = ["create", "read", "update", "delete"]
}

path "secret/metadata/fzymgc-house/cluster/grafana" {
  capabilities = ["read", "list", "delete"]
}

• Step 5: Delete the Grafana Vault KV secrets (operator action — not Terraform-managed)

tf/grafana/vault.tf reads the Grafana secret via a data "vault_kv_secret_v2" "grafana" source — a data source never deletes, so destroying the workspace leaves the secret live. The spec requires deleting it. Delete the KV paths out-of-band with a write-capable Vault token (a normal OIDC login maps to read-only and will 403):

vault kv metadata delete secret/fzymgc-house/cluster/grafana
# plus any MCP service-account token sub-paths the grafana workspace wrote:
vault kv list secret/fzymgc-house/cluster/grafana/ 2>/dev/null && \
  vault kv metadata delete -mount=secret fzymgc-house/cluster/grafana/<each-subpath>

Expected: vault kv get secret/fzymgc-house/cluster/grafana → No value found. Record in the bead.

• Step 6: Remove the Uptime Kuma monitors

In tf/uptime-kuma/main.tf, delete the whole resource "uptimekuma_monitor_http" "grafana" { ... } block (name Grafana Observability, url https://grafana.fzymgc.house/api/health).

In tf/uptime-kuma/monitors-services.tf, delete the whole resource "uptimekuma_monitor_http" "grafana_mcp" { ... } block.

In tf/uptime-kuma/outputs.tf, delete the output map entries:

    grafana   = uptimekuma_monitor_http.grafana.id

(and the grafana_mcp entry if present in the outputs map).

• Step 7: Commit and open the PR

jj describe -m "feat(terraform): remove Grafana TF — module, Authentik OIDC, Vault policy, Uptime Kuma (grafana-sunset W4) [hl-nlxl]"
jj git push -b feat/grafana-sunset-w4
gh pr create --head feat/grafana-sunset-w4 --title "Grafana sunset W4: Terraform/IAM teardown" --body "Removes tf/grafana module + HCP workspace, Authentik Grafana OIDC + group, Vault grafana policies/JWT role, and both Uptime Kuma grafana monitors. Requires the main-cluster-grafana HCP workspace emptied (Step 1)."

• Step 8: Verify via HCP speculative plans + live services

Expected: each affected workspace’s speculative plan on the PR shows only destroys of the intended resources (no creates/replaces). After apply:

# Authentik: the Grafana application/provider is gone (check Authentik admin UI or provider list)
# Vault: policy gone
vault policy list | rg -c 'terraform-grafana-admin'   # Expected: 0
vault kv get secret/fzymgc-house/cluster/grafana 2>&1 # Expected: not found (or operator-deleted)
# Uptime Kuma: status.fzymgc.house no longer lists Grafana / Grafana MCP monitors

---

Task 5: Wave 5 — Loki retirement

Goal: Sever Vector’s loki write path (ClickHouse sink already carries logs), then delete the Loki Application and its object storage.

Constraint: Vector dual-writes logs to Loki and ClickHouse. The loki sink + k8s_logs_for_loki transform must be removed before Loki, or Vector errors on a missing sink target.

Files:

• Modify: argocd/app-configs/monitoring-vector/vector.yaml (remove the k8s_logs_for_loki transform + loki sink)
• Delete: argocd/cluster-app/templates/monitoring-loki.yaml and argocd/app-configs/monitoring-loki/** (the second git source — kustomization.yaml + external-secret-s3.yaml, which holds the loki-s3 secret)
• Step 1: Remove the Vector Loki sink and transform

In argocd/app-configs/monitoring-vector/vector.yaml:

• Delete the transforms.k8s_logs_for_loki block (the type: remap transform fed by k8s_logs, ~lines 36-onwards).
• Delete the sinks.loki block (type: loki, endpoint: "http://loki-gateway.loki.svc.cluster.local:80", ~lines 99-onwards through its labels/encoding).
• Confirm the sinks.clickhouse_logs block (type: clickhouse, endpoint cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc:8123) and its inputs: [k8s_logs_for_clickhouse] transform remain intact and untouched (they are a separate path from the deleted loki sink / k8s_logs_for_loki transform).
• Step 2: Commit the Vector change as its own PR and verify logs still flow

jj new main@origin
jj bookmark create feat/grafana-sunset-w5a -r @
jj describe -m "feat(observability): drop Vector Loki sink + transform (grafana-sunset W5a) [hl-nlxl]"
jj git push -b feat/grafana-sunset-w5a
gh pr create --head feat/grafana-sunset-w5a --title "Grafana sunset W5a: sever Vector Loki sink" --body "Removes Vector's loki sink + k8s_logs_for_loki transform; ClickHouse sink remains the sole log path. Verify otel_logs still ingesting before deleting Loki."

After merge + sync:

kubectl -n vector rollout status deploy/vector 2>&1 || kubectl -n vector get pods

Expected: Vector healthy (the Application deploys into namespace vector). Query ClickHouse (ClickStack MCP): SELECT count() FROM otel_logs WHERE Timestamp > now() - INTERVAL 5 MINUTE returns a growing count. No Vector errors about the loki sink.

• Step 3: Delete the Loki Application and its S3 secret

The monitoring-loki Application has two git sources — the template plus the app-config directory (monitoring-loki.yaml line 84: path: argocd/app-configs/monitoring-loki). Remove both so nothing orphans:

jj new main@origin
jj bookmark create feat/grafana-sunset-w5b -r @
rm argocd/cluster-app/templates/monitoring-loki.yaml
rm -rf argocd/app-configs/monitoring-loki   # kustomization.yaml + external-secret-s3.yaml (loki-s3)

Sanity-check no other manifest still references the loki-s3 secret:

rg -rl --color=never 'name: loki-s3' argocd   # Expected: no results

• Step 4: Commit and open the PR

jj describe -m "feat(observability): remove Loki Application + S3 secret (grafana-sunset W5b) [hl-nlxl]"
jj git push -b feat/grafana-sunset-w5b
gh pr create --head feat/grafana-sunset-w5b --title "Grafana sunset W5b: remove Loki" --body "Deletes the monitoring-loki Application and loki-s3 ExternalSecret. Requires W5a verified (logs flowing to ClickHouse only)."

• Step 5: Verify and delete R2 buckets

kubectl get ns loki 2>&1                                  # Expected: NotFound
kubectl -n argocd get applications | rg -c 'monitoring-loki'  # Expected: 0

After confirming + the retention window from W1, delete the loki-{chunks,ruler,admin} R2 buckets (operator action via R2 dashboard/rclone). Record deletion in the bead.

---

Task 6: Wave 6 — Host Alloy decommission + stale endpoints + docs

Goal: Remove the orphaned host Alloy agent and its Vault IAM, repoint the stale OTLP endpoint, and scrub all remaining Grafana/Loki/Alloy references from docs and routing.

Files:

• Delete: ansible/roles/alloy/**, tf/vault/alloy.tf
• Modify: argocd/cluster-app/templates/temporal-workers.yaml, CLAUDE.md, argocd/CLAUDE.md, tf/CLAUDE.md, ansible/CLAUDE.md, tf/cloudflare/variables.tf, tools/docs-audit/groom-manifest.md
• Step 1: Tear down the Alloy deployment on the Firewalla host

Run the role’s teardown (stop/remove the Docker container, boot script, Vault Agent credentials, directories). If the role lacks a teardown play, do it manually over SSH to the Firewalla:

# on the Firewalla host (uid 501 / launchd or docker as applicable)
docker rm -f alloy 2>/dev/null; docker rmi grafana/alloy 2>/dev/null
# remove the boot script + vault-agent units + /data/alloy dirs the role created

Confirm Alloy is no longer running before removing its config from the repo.

• Step 2: Remove the Alloy Ansible role

jj new main@origin
jj bookmark create feat/grafana-sunset-w6 -r @
rm -rf ansible/roles/alloy

Also remove any alloy role include from the Firewalla host playbook/group_vars if one exists (grep rg -rn 'alloy' ansible --glob '!ansible/roles/alloy/**').

• Step 3: Tear down the Alloy Vault resources

tf/vault/alloy.tf contains vault_kv_secret_v2.alloy (path fzymgc-house/cluster/alloy), vault_policy.alloy_agent, vault_approle_auth_backend_role.alloy_agent, and random_password.alloy_external_token. The KV secret has an in-file terraform state rm teardown note — follow it (the secret may be protected against destroy), then:

rm tf/vault/alloy.tf

The vault workspace’s apply will destroy the AppRole/policy (and the secret, once removed from state).

• Step 4: Repoint the stale temporal-workers OTLP endpoint

In argocd/cluster-app/templates/temporal-workers.yaml, change the dead Alloy endpoint:

            otel:
              endpoint: "cs-otel-collector.clickstack.svc.cluster.local:4317"

(was alloy.alloy.svc.cluster.local:4317 — the K8s Alloy was removed in the ClickStack migration; ClickStack’s collector is the live OTLP target.)

• Step 5: Scrub docs and routing

Edit the four CLAUDE.md files and tools/docs-audit/groom-manifest.md:

• CLAUDE.md (root), tf/CLAUDE.md: remove the fzymgc-house:grafana skill-routing rows.
• argocd/CLAUDE.md: remove grafana-mcp, loki, and alloy/l from the Telemetry “stateless collectors” table row; remove grafana helm-search examples.
• ansible/CLAUDE.md: remove the alloy role reference.
• tf/cloudflare/variables.tf: delete the two commented-out grafana service-URL lines (135/141).
• tools/docs-audit/groom-manifest.md: remove the grafana entry.

Add a one-line note (in argocd/CLAUDE.md or a short doc) recording that Prometheus/kube-prometheus-stack is intentionally retained (live ServiceMonitors; Grafana subchart already disabled).

• Step 6: Commit and open the PR

jj describe -m "feat(observability): decommission host Alloy + scrub Grafana/Loki/Alloy refs (grafana-sunset W6) [hl-nlxl]"
jj git push -b feat/grafana-sunset-w6
gh pr create --head feat/grafana-sunset-w6 --title "Grafana sunset W6: host Alloy + endpoints + docs" --body "Removes the host Alloy role + its Vault IAM, repoints temporal-workers OTLP to the ClickStack collector, and scrubs all remaining Grafana/Loki/Alloy references. Final wave."

• Step 7: Final verification (whole-sunset acceptance)

# No Grafana-Labs apps/namespaces anywhere
kubectl -n argocd get applications | rg -i 'grafana|loki' ; echo "exit=$?"   # Expected: no rows
kubectl get ns | rg -i 'grafana|loki' ; echo "exit=$?"                       # Expected: no rows
# Prometheus RETAINED and scraping its consumers
kubectl get servicemonitor -A | rg -i 'dolt|miniflux|nats|temporal|otel-collector'  # Expected: rows present
kubectl -n monitoring get pods | rg -i 'prometheus'                          # Expected: running
# Repo is clean of live grafana/alloy/loki wiring (docs/historical plans may still mention them)
rg -l --color=never -i 'grafana|alloy-ingest' argocd tf ansible --glob '!**/CLAUDE.md' --glob '!docs/**' ; echo "exit=$?"
# Temporal telemetry reaches ClickStack

Expected: zero grafana/loki Applications or namespaces; Prometheus running with its ServiceMonitors intact; ClickStack receiving temporal-worker telemetry. Historical docs/engineering/{plans,specs} Grafana documents are retained by design.

---

Spec-coverage check

Spec wave	Plan task	Covered
W1 pre-flight + sever deps	Task 1	✓
W2 Grafana CRs (operator alive)	Task 2	✓
W3 operator + mcp + DB + CRDs	Task 3 (split W3a drop / W3b remove)	✓
W4 Terraform / IAM	Task 4	✓
W5 Loki retirement	Task 5 (split W5a Vector / W5b Loki)	✓
W6 host Alloy + endpoints + docs	Task 6	✓
Prometheus retained	Task 6 Step 5 + Step 7 verification	✓
Data disposition (snapshot→hard-delete)	Task 1 Steps 1-2, Task 3 Steps 1-4, Task 5 Step 5	✓