Pushover + Alertmanager Migration — PR5: Cut Grafana ntfy Receivers

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Remove the three Grafana ntfy-* GrafanaContactPoint resources and their ExternalSecret. Keep the 12 pushover-*-* contact points as an escape hatch for future ad-hoc Grafana-evaluated alerts. Re-point the GrafanaNotificationPolicy root receiver from ntfy-critical to pushover-infra-warning since AM now owns all real routing. The ntfy server itself is removed in PR6.

Architecture: Three-edit cleanup of argocd/app-configs/grafana/: (1) update notification-policy.yaml to drop its dependency on the ntfy receiver before we remove it, (2) update kustomization.yaml to stop syncing the two ntfy files, (3) delete the two now-orphaned ntfy files. ArgoCD’s prune: true (already set on the grafana app) removes the cluster-side GrafanaContactPoints and the ExternalSecret (and via deletionPolicy: Delete, the rendered Kubernetes Secret).

Tech Stack: Kubernetes (grafana-operator v1beta1 CRDs, ExternalSecrets v1). No new infrastructure, no Helm changes, no Vault changes (Vault data deleted in PR6).

Spec reference: docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md (the “PR5 — Cut Grafana ntfy receivers” row).

Workspace: .worktrees/feat-pushover-alertmanager (jj workspace pushover-alertmanager, parented to current main).

Manual prerequisites

None. All four prior PRs are merged and verified end-to-end:

PR1 (#995/#996/#997) — Grafana-side Pushover plumbing.
PR2 (#998/#999/#1000) — Alertmanager + receivers.
PR3 (#1001) — NATS rules migrated to PrometheusRule.
PR4 (#1002) — router-hosts rules migrated to PrometheusRule.

Sanity check before kicking off:

# AM healthy
kubectl --context fzymgc-house -n prometheus get statefulset alertmanager-kps-kube-prometheus-stack-alertmanager -o jsonpath='{.status.readyReplicas}/{.spec.replicas}{"\n"}'

# Both PrometheusRules present
kubectl --context fzymgc-house -n prometheus get prometheusrule nats router-hosts -o jsonpath='{range .items[*]}{.metadata.name}{","}{end}{"\n"}'

# No remaining GrafanaAlertRuleGroup resources (all migrated)
kubectl --context fzymgc-house -n grafana get grafanaalertrulegroup 2>&1 | head

Expected: 1/1, nats,router-hosts,, and No resources found in grafana namespace. respectively. The third is the load-bearing prereq — no Grafana-evaluated rules means no real alert traffic depends on the ntfy contact points we’re about to delete.

Important context

Why we keep the 12 Pushover Grafana contact points

The spec calls these “an escape hatch for future ad-hoc Grafana-evaluated alerts.” If you ever want to create a one-off Grafana alert (e.g., on a Loki query that doesn’t fit the PrometheusRule model), you’ll have a working Pushover delivery path through Grafana’s contact-point system without re-doing PR1’s setup. Cost: 12 unused YAML files + 5 ESO secret keys, ~250 lines total. Worth it.

Why we re-point the root receiver instead of deleting the notification policy

For the same reason. Without a GrafanaNotificationPolicy, future ad-hoc alerts have no default route. Pointing root to pushover-infra-warning (priority 0, default sound) is a sensible “any unlabeled alert” fallback that matches what we’d want anyway.

Ordering matters for git-diff readability, not for ArgoCD

All 4 commits land together at merge time. ArgoCD sees the final state and reconciles to it — there’s no intermediate state where the notification policy references a missing receiver. The chosen commit ordering (notification policy → kustomization → file delete) is for the reviewer’s reading flow, not a hard correctness requirement.

What stays, what goes

After this PR:

Resource	Status
`argocd/app-configs/grafana/pushover-tokens-external-secret.yaml`	KEEP
`argocd/app-configs/grafana/pushover-contact-points.yaml` (12 contact points)	KEEP
`argocd/app-configs/grafana/notification-policy.yaml` (root re-pointed)	KEEP
`argocd/app-configs/grafana/ntfy-tokens-external-secret.yaml`	DELETE
`argocd/app-configs/grafana/ntfy-contact-points.yaml` (3 contact points)	DELETE

Vault paths secret/fzymgc-house/cluster/ntfy* stay untouched in PR5 — destroyed in PR6.

File structure

File	Status	Responsibility
`argocd/app-configs/grafana/notification-policy.yaml`	Modify	Change `route.receiver` from `ntfy-critical` to `pushover-infra-warning`. Update the header comment to remove the “Removed by PR5 once Alertmanager owns routing” note (since the policy now stays).
`argocd/app-configs/grafana/kustomization.yaml`	Modify	Remove `- ntfy-tokens-external-secret.yaml` and `- ntfy-contact-points.yaml` from the `resources:` list.
`argocd/app-configs/grafana/ntfy-tokens-external-secret.yaml`	Delete	ArgoCD prune removes the in-cluster `ntfy-grafana-tokens` ExternalSecret + (via `deletionPolicy: Delete`) the rendered Kubernetes Secret.
`argocd/app-configs/grafana/ntfy-contact-points.yaml`	Delete	ArgoCD prune removes the 3 in-cluster `GrafanaContactPoint`s (`ntfy-critical`/`ntfy-warning`/`ntfy-info`). The Grafana operator removes the contact points from Grafana’s database.

Task 1: Verify workspace + prior PRs are intact

Files: none (verification only)

Step 1.1: Confirm you are in the correct workspace, parented to current main

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager git fetch
jj --no-pager workspace list | rg pushover-alertmanager
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

Expected: workspace listed; main is at PR #1002’s merge commit (feat(router-hosts): migrate alert rules to PrometheusRule (PR4/6)) or later. If @ is parented to an older commit, run jj new main.

Step 1.2: Confirm Alertmanager is up + both PrometheusRules loaded

kubectl --context fzymgc-house -n prometheus get statefulset alertmanager-kps-kube-prometheus-stack-alertmanager -o jsonpath='{.status.readyReplicas}/{.spec.replicas}{"\n"}'
kubectl --context fzymgc-house -n prometheus get prometheusrule nats router-hosts 2>&1 | head

Expected: 1/1 and both rules listed. If either is missing, stop — don’t strip the ntfy fallback while the new path is broken.

Step 1.3: Confirm no remaining GrafanaAlertRuleGroup (all rules migrated to Prometheus)

kubectl --context fzymgc-house -n grafana get grafanaalertrulegroup 2>&1

Expected: No resources found in grafana namespace. This is the load-bearing prereq. If any GrafanaAlertRuleGroup still exists, it would be silently broken once we remove the ntfy contact points (and possibly the notification policy fallback). Stop and finish migrating.

Step 1.4: Confirm the 3 ntfy contact points are currently active (baseline for prune verification)

kubectl --context fzymgc-house -n grafana get grafanacontactpoint -l '!nope' 2>&1 | rg ntfy

Expected: 3 lines for ntfy-critical, ntfy-info, ntfy-warning. These are the resources ArgoCD will prune post-merge.

Step 1.5: Snapshot current alert state (typically empty)

kubectl --context fzymgc-house -n prometheus exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
  amtool --alertmanager.url=http://localhost:9093 alert query 2>&1 | rg -v '^Alertname|^$' | head

Expected: nothing (no active alerts) on a healthy cluster. Note any active ones — they’ll continue routing through Alertmanager (not Grafana) so this PR doesn’t affect them.

Task 2: Re-point the notification policy root receiver

Files:

Modify: argocd/app-configs/grafana/notification-policy.yaml
Step 2.1: Edit the root receiver and the header comment

Find the header comment block (lines 1-5) and the route.receiver line. Two edits in the same file.

Edit A — replace the header comment:

---
# Routes alerts to the per-(domain, severity) Pushover contact point. 12 leaf routes
# total: 4 domains × 3 severities. Default route falls back to ntfy-critical for any
# alert that doesn't carry both `domain` and `severity` labels.
# Removed by PR5 once Alertmanager owns routing.

with:

---
# Routes Grafana-evaluated alerts to the per-(domain, severity) Pushover contact point.
# 12 leaf routes total: 4 domains × 3 severities. Default route falls back to
# pushover-infra-warning for any alert that doesn't carry both `domain` and `severity`
# labels. Real alert traffic flows through Alertmanager (deployed in PR2) — these
# Grafana contact points + this policy exist as an escape hatch for any future ad-hoc
# Grafana-evaluated alerts (e.g., Loki queries that don't fit the PrometheusRule model).

Edit B — change the root receiver value (single line):

  route:
    receiver: ntfy-critical

to:

  route:
    receiver: pushover-infra-warning

The 12 child routes under routes: are unchanged.

Step 2.2: Lint YAML

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
yamllint argocd/app-configs/grafana/notification-policy.yaml

Expected: no output.

Step 2.3: Validate kustomize render

kubectl --context fzymgc-house apply --dry-run=client -k argocd/app-configs/grafana 2>&1 | rg notification-policy

Expected: grafananotificationpolicy.grafana.integreatly.org/notification-policy configured (dry run) (note “configured” — it’s a modify, not a create).

Step 2.4: Commit

jj --no-pager commit -m "feat(grafana): re-point notification policy root receiver from ntfy-critical to pushover-infra-warning

PR2's Alertmanager owns all real alert routing now (PR3 migrated NATS,
PR4 migrated router-hosts). Grafana has zero evaluated alert rules.
The Grafana notification policy and 12 Pushover contact points stay
as an escape hatch for any future ad-hoc Grafana-evaluated alerts
(e.g., Loki queries that don't fit the PrometheusRule model), but
the root fallback receiver no longer needs to be ntfy-critical —
ntfy is being decommissioned in PR6.

Re-point root to pushover-infra-warning: priority 0 (normal sound,
respects quiet hours), routes via the cluster-infra Pushover app.
Sensible default for any unlabeled ad-hoc alert. Updated the header
comment to reflect the policy's new role."

Verify:

jj --no-pager log -r '@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Task 3: Update kustomization to drop ntfy entries

Files:

Modify: argocd/app-configs/grafana/kustomization.yaml
Step 3.1: Edit kustomization.yaml

Read current contents:

cat argocd/app-configs/grafana/kustomization.yaml

The current resources list includes - ntfy-tokens-external-secret.yaml and - ntfy-contact-points.yaml. Remove BOTH lines. The remaining resources stay in their current order.

After edit, the file should reference (in order):

namespace.yaml
grafana.yaml
grafana-certificate.yaml
grafana-admin-external-secret.yaml
grafana-oidc-creds.yaml
pushover-tokens-external-secret.yaml
pushover-contact-points.yaml
notification-policy.yaml
datasources
dashboards

(The exact ordering of the surviving entries comes from the current file — preserve whatever order is there; only delete the two ntfy lines.)

Step 3.2: Validate kustomize render

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
kubectl --context fzymgc-house apply --dry-run=client -k argocd/app-configs/grafana 2>&1 | tee /tmp/kustomize-grafana.txt
rg 'ntfy' /tmp/kustomize-grafana.txt || echo "no_ntfy_in_render_good"

Expected: no_ntfy_in_render_good. The dry-run output should contain pushover contact points + notification policy + grafana resources, but ZERO ntfy* references.

Step 3.3: Lint YAML

yamllint argocd/app-configs/grafana/kustomization.yaml

Expected: no output.

Step 3.4: Commit

jj --no-pager commit -m "feat(grafana): drop ntfy entries from kustomization

Removes ntfy-tokens-external-secret.yaml and ntfy-contact-points.yaml
from the grafana app's kustomization. ArgoCD's prune=true on the
grafana app removes the in-cluster ntfy-grafana-tokens ExternalSecret
(and via deletionPolicy: Delete, the rendered Kubernetes Secret) plus
the three GrafanaContactPoint resources (ntfy-critical, ntfy-warning,
ntfy-info) on next sync. The Grafana operator removes the contact
points from Grafana's alerting database. The notification policy
(updated in the previous commit) no longer references ntfy-critical,
so there's no broken-receiver gap during the prune.

The next commit deletes the now-orphaned ntfy yaml files."

Task 4: Delete the orphaned ntfy yaml files

Files:

Delete: argocd/app-configs/grafana/ntfy-tokens-external-secret.yaml
Delete: argocd/app-configs/grafana/ntfy-contact-points.yaml
Step 4.1: Delete the files

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
rm argocd/app-configs/grafana/ntfy-tokens-external-secret.yaml
rm argocd/app-configs/grafana/ntfy-contact-points.yaml

Step 4.2: Verify the directory state

ls argocd/app-configs/grafana/ | rg ntfy || echo "no_ntfy_files_good"

Expected: no_ntfy_files_good.

Step 4.3: Confirm jj sees both deletions

jj --no-pager st

Expected:

D argocd/app-configs/grafana/ntfy-contact-points.yaml
D argocd/app-configs/grafana/ntfy-tokens-external-secret.yaml

Step 4.4: Commit

jj --no-pager commit -m "chore(grafana): remove legacy ntfy contact points and ExternalSecret manifests

Deletes the two ntfy yaml files from argocd/app-configs/grafana/.
Both files are no longer referenced by the kustomization (previous
commit), and the cluster-side resources have already been pruned by
ArgoCD. The ntfy server itself + Vault data are removed in PR6.

After this PR:
  - 0 GrafanaAlertRuleGroup (all rules migrated to PrometheusRule)
  - 0 ntfy-* GrafanaContactPoint
  - 12 pushover-*-* GrafanaContactPoint (escape hatch, kept)
  - 1 GrafanaNotificationPolicy (root fallback re-pointed to pushover-infra-warning)
  - 1 ntfy-grafana-tokens ExternalSecret → deleted by prune
  - 1 pushover-grafana-tokens ExternalSecret (kept for the 12 contact points)"

Verify all 4 commits landed:

jj --no-pager log -r 'main..@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"' | head

Expected (4 commits, newest first):

<id> chore(grafana): remove legacy ntfy contact points and ExternalSecret manifests
<id> feat(grafana): drop ntfy entries from kustomization
<id> feat(grafana): re-point notification policy root receiver from ntfy-critical to pushover-infra-warning
<id> docs(superpowers): add PR5 implementation plan for cutting Grafana ntfy receivers

Task 5: Pre-push verification

Files: none (verification only)

Step 5.1: Confirm all expected commits exist

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager log -r 'main..@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Expected: 4 commits.

Step 5.2: Confirm no secrets in the diff

jj --no-pager diff -r 'main..@-' | rg -i 'token|user.?key|password|secret' | rg -v 'secretKeyRef|secret/data|secretStoreRef|.token|api[_-]?token|ntfy-grafana-tokens|writer_|admin_|subscriber_' || echo "no_secrets_good"

Expected: no_secrets_good. The deletions touch references to ntfy secret keys (e.g., writer_critical_password) but the values themselves are Vault placeholders, not actual credentials.

Step 5.3: Verify cluster-app Helm template still renders cleanly

cd argocd/cluster-app
helm template . --name-template cluster-app --namespace argocd >/dev/null
echo "exit=$?"
cd ../..

Expected: exit=0. PR5 doesn’t touch argocd/cluster-app/templates/, so this should pass.

Step 5.4: Run rumdl on the plan doc explicitly

rumdl check docs/engineering/plans/2026-05-10-pushover-pr5-cut-grafana-ntfy.md

Expected: Success: No issues found. If errors, run rumdl fmt then jj squash --into <plan-change-id> to fold autofixes back into the plan commit.

Step 5.5: Run all repo linters

lefthook run pre-commit --all-files 2>&1 | tail -30

Expected: hooks on touched files pass. Pre-existing failures on untouched files are NOT blockers.

Step 5.6: Verify the working copy base is current

jj --no-pager git fetch
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

If main has moved, rebase:

jj --no-pager rebase -s 'roots(main..@-)' -d main --skip-emptied

Task 6: Push and open PR

Files: none (VCS + GitHub operations)

Step 6.1: Create the bookmark and push

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager bookmark create feat/pushover-pr5-cut-grafana-ntfy -r @-
jj --no-pager git push -b feat/pushover-pr5-cut-grafana-ntfy

Step 6.2: Open the PR

gh pr create \
  --head feat/pushover-pr5-cut-grafana-ntfy \
  --base main \
  --title "feat(grafana): cut ntfy contact points + re-point notification policy (PR5/6)" \
  --body "$(cat <<'EOF'
## Summary

PR5 of the pushover + alertmanager migration. Removes the three Grafana \`ntfy-*\` contact points and their ExternalSecret. Re-points the GrafanaNotificationPolicy root receiver from \`ntfy-critical\` to \`pushover-infra-warning\` since AM now owns all real routing. Keeps the 12 \`pushover-*-*\` Grafana contact points as an escape hatch for future ad-hoc Grafana-evaluated alerts.

The ntfy server itself + Vault data are removed in PR6.

Spec: \`docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md\`
Plan: \`docs/engineering/plans/2026-05-10-pushover-pr5-cut-grafana-ntfy.md\`

## Changes

- \`argocd/app-configs/grafana/notification-policy.yaml\` — \`route.receiver\` changed from \`ntfy-critical\` to \`pushover-infra-warning\`. Header comment updated to describe the policy's new role as escape-hatch routing for ad-hoc Grafana alerts.
- \`argocd/app-configs/grafana/kustomization.yaml\` — drops \`- ntfy-tokens-external-secret.yaml\` and \`- ntfy-contact-points.yaml\` from resources list.
- \`argocd/app-configs/grafana/ntfy-tokens-external-secret.yaml\` — deleted. ArgoCD prune removes the in-cluster \`ntfy-grafana-tokens\` ExternalSecret + (via \`deletionPolicy: Delete\`) the rendered Kubernetes Secret.
- \`argocd/app-configs/grafana/ntfy-contact-points.yaml\` — deleted. ArgoCD prune removes the three \`GrafanaContactPoint\`s (\`ntfy-critical\`, \`ntfy-warning\`, \`ntfy-info\`).

## Pre-merge prerequisites

None. PR1-PR4 are all merged and verified end-to-end. The load-bearing prereq is that no \`GrafanaAlertRuleGroup\` resources remain in the \`grafana\` namespace (verified in Task 1.3) — meaning no real alert traffic depends on the ntfy contact points being deleted.

## Post-merge verification

- [ ] HCP TFC: no Vault workspace run triggered (no TF changes).
- [ ] ArgoCD \`grafana\` app syncs cleanly; \`Synced\` + \`Healthy\`.
- [ ] \`kubectl -n grafana get grafanacontactpoint\` shows zero \`ntfy-*\` entries; the 12 \`pushover-*-*\` entries remain with \`READY=True\`.
- [ ] \`kubectl -n grafana get externalsecret ntfy-grafana-tokens\` returns \`NotFound\`.
- [ ] \`kubectl -n grafana get secret ntfy-grafana-tokens\` returns \`NotFound\` (deletionPolicy: Delete on the ExternalSecret cascades).
- [ ] \`kubectl -n grafana get grafananotificationpolicy notification-policy -o jsonpath='{.spec.route.receiver}'\` returns \`pushover-infra-warning\`.
- [ ] In Grafana UI: Alerting → Contact points lists no \`ntfy-*\` entries; lists 12 \`pushover-*-*\` entries.
- [ ] In Grafana UI: Alerting → Notification policies → root policy default receiver is \`pushover-infra-warning\`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Step 6.3: Record the PR URL

The gh pr create command prints the PR URL.

Task 7: Post-merge verification (executes after CI + merge)

Files: none (live cluster verification)

Step 7.1: Wait for ArgoCD sync via Monitor (state-change emit pattern)

Track:

grafana ArgoCD app: Synced + Healthy
3 ntfy GrafanaContactPoints: gone
ntfy-grafana-tokens ExternalSecret: gone
ntfy-grafana-tokens Kubernetes Secret: gone
notification-policy root receiver: pushover-infra-warning

Sample Monitor command shape (adapt as needed):

prev=""
while true; do
  ga_sync=$(kubectl --context fzymgc-house get application grafana -n argocd -o jsonpath='{.status.sync.status}/{.status.health.status}' 2>/dev/null || echo "?/?")
  ntfy_cps=$(kubectl --context fzymgc-house -n grafana get grafanacontactpoint -o jsonpath='{range .items[*]}{.metadata.name}{","}{end}' 2>/dev/null | tr ',' '\n' | rg -c '^ntfy-' || echo "0")
  ntfy_es=$(kubectl --context fzymgc-house -n grafana get externalsecret ntfy-grafana-tokens -o jsonpath='{.metadata.name}' 2>/dev/null || echo "gone")
  ntfy_sec=$(kubectl --context fzymgc-house -n grafana get secret ntfy-grafana-tokens -o jsonpath='{.metadata.name}' 2>/dev/null || echo "gone")
  np_root=$(kubectl --context fzymgc-house -n grafana get grafananotificationpolicy notification-policy -o jsonpath='{.spec.route.receiver}' 2>/dev/null || echo "?")
  cur="ga=${ga_sync} ntfyCPs=${ntfy_cps} ntfyES=${ntfy_es} ntfySecret=${ntfy_sec} npRoot=${np_root}"
  if [ "$cur" != "$prev" ]; then
    echo "$(date -u +%H:%M:%S) $cur"
    prev=$cur
  fi
  if [ "$ga_sync" = "Synced/Healthy" ] && [ "$ntfy_cps" = "0" ] && [ "$ntfy_es" = "gone" ] && [ "$ntfy_sec" = "gone" ] && [ "$np_root" = "pushover-infra-warning" ]; then
    echo "$(date -u +%H:%M:%S) READY"
    break
  fi
  sleep 15
done

Step 7.2: Confirm the 12 Pushover contact points are still healthy

kubectl --context fzymgc-house -n grafana get grafanacontactpoint -l '!nope' 2>&1 | rg pushover | wc -l

Expected: 12. None of the Pushover contact points should be affected by removing the ntfy ones.

Step 7.3: Confirm the ntfy server itself is still up (will be removed in PR6)

kubectl --context fzymgc-house -n ntfy get pods 2>&1

Expected: ntfy server pod(s) running. PR5 only removes the Grafana-side ntfy plumbing; the ntfy server’s own ExternalSecret (ntfy-tokens in the ntfy namespace) is unrelated and stays until PR6.

Step 7.4: Smoketest (optional) — fire a manual alert through Grafana to confirm the new root receiver path

Skip this if Grafana ad-hoc alerting isn’t a workflow you’ll actually use — the load-bearing alerts already flow through Alertmanager (verified in PR3 and PR4 smoketests). If you want to verify the Grafana escape hatch:

In Grafana UI: Alerting → Contact points → click pushover-infra-warning → “Test”. Confirm Pushover notification arrives via cluster-infra app at priority 0.

Step 7.5: Update workspace state for PR6

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager git fetch
jj --no-pager new main

Workspace ready for PR6 (decommission ntfy stack + Vault data).

Risks and rollback

Risk	Detection	Rollback
Hidden GrafanaAlertRuleGroup somewhere we missed	Grafana UI shows alert rules; some firing alert hits the now-missing ntfy receiver	Step 1.3 explicitly checks for this. If the check missed something, revert the merge commit (restores the ntfy contact points). The escape-hatch Pushover contact points stay either way.
Grafana-operator fails to delete the contact points	UI still shows ntfy contact points after sync	`kubectl -n grafana describe grafanacontactpoint ntfy-critical` shows error in conditions. Manual cleanup: `kubectl -n grafana delete grafanacontactpoint ntfy-critical ntfy-warning ntfy-info`.
Notification policy’s new root receiver fails validation	`kubectl describe grafananotificationpolicy notification-policy` shows error referencing `pushover-infra-warning`	The receiver exists (verified in PR1 #997). If somehow not present, the policy’s old root (`ntfy-critical`) wouldn’t have worked either. Investigate Grafana operator logs.
ArgoCD `grafana` app sync fails because of the ESO ignoreDifferences interaction	App stays OutOfSync after merge	The grafana app already has `ignoreDifferences` for ExternalSecret defaults (verified pre-plan). Should not happen, but if it does, check whether the ESO drift fields are different for the surviving `pushover-grafana-tokens`.

Out of scope for PR5 (handled in PR6)

ntfy server decommission — the entire argocd/app-configs/ntfy/ directory, the ntfy.yaml cluster-app template, the Vault Terraform (tf/vault/policy-ntfy.tf, tf/vault/k8s-ntfy.tf), the Cloudflare DNS, and the docs entries for ntfy.
Vault KV destruction — vault kv metadata delete secret/fzymgc-house/cluster/ntfy and secret/fzymgc-house/cluster/ntfy-grafana. Per spec: irreversible, run only after consumers are gone (which PR5 ensures for the Grafana-side consumer).
Documentation cleanup — removing ntfy from docs/reference/services.md and docs/reference/secrets.md. Done in PR6 alongside the rest of the decommission.
CI lint asserting all PrometheusRules carry domain + severity — tracked as a follow-up bead.