Pushover + Alertmanager Migration — PR6: Decommission ntfy

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Fully remove the self-hosted ntfy stack: ArgoCD application + manifests, Terraform Vault policy + role, Cloudflare DNS entry, docs entries, AND (manual, post-merge, irreversible) the Vault KV data at secret/fzymgc-house/cluster/ntfy + .../ntfy-grafana. This is the final PR of the pushover + alertmanager migration.

Architecture: Three-front cleanup happens in the PR diff (Kubernetes manifests, Terraform, docs). One manual post-merge step happens after the PR merges and ArgoCD/TFC have propagated the deletes (Vault KV metadata destruction). Each front is its own commit for reviewer readability.

Tech Stack: Kubernetes (deletes the entire ntfy ArgoCD application + its manifests), HCP Terraform (vault and cloudflare workspaces both apply destroys), Vault (manual kv metadata delete), markdown docs.

Spec reference: docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md (the “PR6 — ordered decommission steps” table). The spec’s 6.1-6.8 ordering is preserved in this plan.

Workspace: .worktrees/feat-pushover-alertmanager (jj workspace pushover-alertmanager, parented to current main).

---

Manual prerequisites at start

None. PR1-PR5 are all merged and verified end-to-end. After PR5 the only remaining ntfy footprint is:

• The ntfy ArgoCD application + its 10 manifest files (the actual server)
• The defensive Vault TF policy + auth role
• The Cloudflare DNS entry
• Docs references in services.md and secrets.md
• Vault KV data at two paths

Sanity check before kicking off:

# Grafana side already cleaned by PR5 — these should all be gone
kubectl --context fzymgc-house -n grafana get grafanacontactpoint 2>&1 | rg ntfy || echo "no_grafana_ntfy_good"
kubectl --context fzymgc-house -n grafana get externalsecret ntfy-grafana-tokens 2>&1 | head -1

# ntfy server itself still running (we're about to remove it)
kubectl --context fzymgc-house -n ntfy get pods 2>&1 | head -3

# All real alert routing already on AM (PR3 + PR4)
kubectl --context fzymgc-house -n prometheus get prometheusrule 2>&1

Expected: no_grafana_ntfy_good; ExternalSecret NotFound; ntfy pod(s) Running; both nats and router-hosts PrometheusRules listed. The third check is the load-bearing prereq — no rules depend on the ntfy notification path.

---

Important context

Two-phase delete: in-PR vs post-merge manual

In the PR diff (this commit chain):

• Delete ArgoCD manifests → ArgoCD prunes the cluster-side ntfy Deployment, Service, IngressRoute, Certificate, PVC, namespace
• Delete TF Vault files → HCP TFC destroys the defensive policy + role
• Edit Cloudflare TF → HCP TFC removes the DNS record
• Edit docs

After the PR merges, you (the user) manually run:

• vault kv metadata delete secret/fzymgc-house/cluster/ntfy
• vault kv metadata delete secret/fzymgc-house/cluster/ntfy-grafana

The Vault KV destruction is irreversible (per the spec’s “what NOT to save” trap — vault kv delete is reversible, vault kv metadata delete destroys all versions and metadata). It belongs as a manual post-merge step rather than automation because:

1. The author needs to confirm consumer-side cleanup actually completed before destroying the data.
2. There’s no PR-diff representation for “delete data from a database.”
3. If something goes wrong with the in-PR cleanup, you might need the data back from version history — which vault kv metadata delete makes impossible.

The plan’s Task 9 is the explicit post-merge manual step with confirmation.

Three TFC workspaces will run on merge

PR6 touches both tf/vault/ and tf/cloudflare/, so on merge HCP TFC kicks off:

• main-cluster-vault — destroys vault_policy.ntfy, vault_kubernetes_auth_backend_role.ntfy, vault_policy.grafana_ntfy_tokens. ~3 to destroy, 0 to add.
• main-cluster-cloudflare — destroys the ntfy DNS record from the internal_services tunnel routing. ~1 to destroy, 0 to add.
• (main-cluster-router-hosts and others should report “no run triggered” since their working dirs aren’t touched.)

Order matters at the merge boundary

Per the spec’s ordering rationale: 6.3 (Vault KV destruction) MUST happen AFTER 6.1 (ArgoCD prune) propagates AND 6.2 (no remaining ExternalSecret) is confirmed. Otherwise ESO will keep retrying reads against the dead path and produce noise for ~15min. The plan enforces this ordering by having Task 9 (manual Vault destruction) come after Task 8 (post-merge Monitor confirms ArgoCD prune + all in-cluster resources are gone).

What’s NOT in this PR

• The 12 pushover-*-* Grafana contact points and pushover-grafana-tokens ExternalSecret stay. Per PR5, they’re the escape hatch for future ad-hoc Grafana alerts.
• The notification-policy.yaml GrafanaNotificationPolicy stays (root receiver pushover-infra-warning).
• The pushover Vault policy + k8s auth role + KV data stay.
• The Alertmanager + Pushover receivers config stays.

After PR6, the only thing that will be aware of “ntfy” in the entire codebase is git history.

---

File structure

Files to delete

File	Effect
argocd/cluster-app/templates/ntfy.yaml	Removes the ArgoCD Application object → ntfy app deleted from ArgoCD → prune: true cascades and removes everything in the ntfy namespace
argocd/app-configs/ntfy/bootstrap-job.yaml	(Pruned by ArgoCD via the app deletion above; removed from git too)
argocd/app-configs/ntfy/certificate.yaml	(Pruned via app deletion)
argocd/app-configs/ntfy/configmap.yaml	(Pruned via app deletion)
argocd/app-configs/ntfy/deployment.yaml	(Pruned via app deletion)
argocd/app-configs/ntfy/external-secret.yaml	(Pruned via app deletion)
argocd/app-configs/ntfy/ingressroute-internal.yaml	(Pruned via app deletion)
argocd/app-configs/ntfy/kustomization.yaml	(Pruned via app deletion)
argocd/app-configs/ntfy/namespace.yaml	(Pruned via app deletion — the namespace itself goes)
argocd/app-configs/ntfy/pvc.yaml	(Pruned via app deletion — the Longhorn PVC + underlying volume is destroyed)
argocd/app-configs/ntfy/service.yaml	(Pruned via app deletion)
tf/vault/policy-ntfy.tf	TFC destroys vault_policy.ntfy
tf/vault/k8s-ntfy.tf	TFC destroys vault_kubernetes_auth_backend_role.ntfy and vault_policy.grafana_ntfy_tokens

Files to modify

File	Change
tf/cloudflare/variables.tf	Remove the ntfy entry from the internal_services default map (lines 136-137 area) → TFC removes the DNS record + tunnel route
docs/reference/services.md	Remove the ntfy row from the services catalog table
docs/reference/secrets.md	Remove the ntfy and ntfy-grafana rows from the Vault paths table

Manual operations after merge

Operation	Why manual
vault kv metadata delete secret/fzymgc-house/cluster/ntfy	Irreversible. Consumer-side prune must be confirmed first.
vault kv metadata delete secret/fzymgc-house/cluster/ntfy-grafana	Same

---

Task 1: Verify workspace + PR5 state + load-bearing prereqs

Files: none (verification only)

• Step 1.1: Confirm you are in the correct workspace, parented to current main

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager git fetch
jj --no-pager workspace list | rg pushover-alertmanager
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

Expected: workspace listed; main is at PR #1003’s merge commit (feat(grafana): cut ntfy contact points + re-point notification policy (PR5/6)) or later. If @ is parented to an older commit, run jj new main.

• Step 1.2: Confirm PR5 cleanup is fully landed (no Grafana-side ntfy resources)

kubectl --context fzymgc-house -n grafana get grafanacontactpoint 2>&1 | rg ntfy || echo "no_grafana_ntfy_good"
kubectl --context fzymgc-house -n grafana get externalsecret ntfy-grafana-tokens 2>&1 | head -1
kubectl --context fzymgc-house -n grafana get secret ntfy-grafana-tokens 2>&1 | head -1

Expected: no_grafana_ntfy_good; both ntfy-grafana-tokens resources NotFound. If any are present, PR5 didn’t fully land — stop and verify.

• Step 1.3: Confirm ntfy server itself is still running (we’re about to remove it)

kubectl --context fzymgc-house -n ntfy get pods 2>&1
kubectl --context fzymgc-house -n ntfy get pvc 2>&1

Expected: at least one ntfy pod Running; one PVC Bound. This is the baseline — ArgoCD will prune both post-merge.

• Step 1.4: Confirm all real alert routing is on Alertmanager (LOAD-BEARING PREREQ)

kubectl --context fzymgc-house -n prometheus get prometheusrule 2>&1
kubectl --context fzymgc-house -n grafana get grafanaalertrulegroup 2>&1

Expected: PrometheusRules nats and router-hosts present in prometheus namespace; No resources found in grafana namespace. for GrafanaAlertRuleGroup. If anything in Grafana is still evaluating rules, the ntfy server might still receive a notification when the namespace is deleted — investigate before continuing.

• Step 1.5: Snapshot current alert state (typically empty)

kubectl --context fzymgc-house -n prometheus exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
  amtool --alertmanager.url=http://localhost:9093 alert query 2>&1 | rg -v '^Alertname|^$' | head

Expected: nothing (no active alerts) on a healthy cluster.

• Step 1.6: Note the ntfy hostname for later DNS verification

grep -A 3 'ntfy' tf/cloudflare/variables.tf | head -5

Note the service_url and the implied hostname. After PR6 merges + Cloudflare TFC applies, that hostname should no longer resolve.

---

Task 2: Delete the ntfy ArgoCD app + manifests

Files:

• Delete: argocd/cluster-app/templates/ntfy.yaml
• Delete: entire directory argocd/app-configs/ntfy/
• Step 2.1: Delete the ArgoCD Application template

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
rm argocd/cluster-app/templates/ntfy.yaml

• Step 2.2: Delete the entire ntfy app-configs directory

rm -r argocd/app-configs/ntfy/

• Step 2.3: Verify the directory is gone

ls argocd/app-configs/ | rg ntfy || echo "no_ntfy_dir_good"
ls argocd/cluster-app/templates/ | rg ntfy || echo "no_ntfy_template_good"

Expected: no_ntfy_dir_good and no_ntfy_template_good.

• Step 2.4: Confirm jj sees the deletions

jj --no-pager st | head -20

Expected: 11 deletion lines (the cluster-app template + 10 files in app-configs/ntfy/).

• Step 2.5: Verify cluster-app Helm template still renders cleanly without the ntfy app

cd argocd/cluster-app
helm template . --name-template cluster-app --namespace argocd 2>&1 | rg -c 'kind: Application' || echo "0"
cd ../..

Expected: a number — count of remaining Application objects in the cluster-app template output. Should be one less than before PR6. If helm template errors, the deletion broke something — investigate.

• Step 2.6: Commit

jj --no-pager commit -m "feat(ntfy): remove ArgoCD application and manifests

Deletes argocd/cluster-app/templates/ntfy.yaml (the ArgoCD Application
object) and the entire argocd/app-configs/ntfy/ directory (10 manifest
files: bootstrap-job, certificate, configmap, deployment, external-secret,
ingressroute-internal, kustomization, namespace, pvc, service).

ArgoCD's prune=true on the cluster-app cascades the Application deletion
to remove every resource in the ntfy namespace: Deployment, Service,
IngressRoute, Certificate, PVC (Longhorn volume destroyed), namespace
itself. The ntfy-tokens ExternalSecret in the ntfy namespace is also
removed (deletionPolicy: Delete cascades to its rendered Secret).

After ArgoCD prune completes:
  - ntfy server pod terminated
  - ntfy namespace deleted
  - ntfy PVC + Longhorn volume destroyed (all message history lost — intentional)

The TF Vault policy + role are removed in the next commit. The Cloudflare
DNS record is removed in the commit after that. Vault KV data destruction
is the final manual post-merge step (Task 9)."

Verify:

jj --no-pager log -r '@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

---

Task 3: Delete the Vault Terraform files

Files:

• Delete: tf/vault/policy-ntfy.tf
• Delete: tf/vault/k8s-ntfy.tf
• Step 3.1: Delete both files

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
rm tf/vault/policy-ntfy.tf
rm tf/vault/k8s-ntfy.tf

• Step 3.2: Verify deletion

ls tf/vault/ | rg ntfy || echo "no_ntfy_tf_good"

Expected: no_ntfy_tf_good.

• Step 3.3: Format + validate Terraform

cd tf/vault
terraform fmt -check -recursive
terraform init -backend=false
terraform validate
cd ../..

Expected: terraform fmt -check no output (exit 0); terraform validate returns Success! The configuration is valid. If the fmt check fails, run terraform fmt -recursive to fix.

• Step 3.4: Commit

jj --no-pager commit -m "feat(vault): remove defensive ntfy policy and k8s auth role

Deletes tf/vault/policy-ntfy.tf and tf/vault/k8s-ntfy.tf. HCP TFC
applies the destroy on the main-cluster-vault workspace:
  - vault_policy.ntfy (defensive — never had a live consumer)
  - vault_kubernetes_auth_backend_role.ntfy (defensive — bound to the
    default SA in the ntfy namespace, which no longer exists post-Task 2)
  - vault_policy.grafana_ntfy_tokens (defensive — never bound to a
    grafana k8s auth role)

All three were defensive pre-declarations (per the file comments) — the
actual Vault read path was always ESO using the wildcard
external-secrets-operator policy. With the ntfy app gone (previous
commit), nothing references these policies anymore.

The Vault KV data at secret/fzymgc-house/cluster/ntfy and
secret/fzymgc-house/cluster/ntfy-grafana is destroyed manually
post-merge (Task 9 in the plan)."

Verify:

jj --no-pager log -r '@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

---

Task 4: Remove the Cloudflare DNS entry

Files:

• Modify: tf/cloudflare/variables.tf

• Step 4.1: Locate the ntfy entry in the internal_services map

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
grep -n -B 2 -A 3 'ntfy' tf/cloudflare/variables.tf | head -10

Note the exact line range of the ntfy = { ... } block (probably 4 lines including the closing }).

• Step 4.2: Read the file around the ntfy block

sed -n '130,142p' tf/cloudflare/variables.tf

You need the surrounding context to make a clean Edit (the Edit tool requires uniquely identifying the block). Adjust line range as needed.

• Step 4.3: Remove the ntfy block

Use the Edit tool to remove only the ntfy = { ... } block (including its trailing newline if it has one). Preserve the surrounding entries in the internal_services map. Do NOT delete any other entry.

• Step 4.4: Verify the deletion

grep -c 'ntfy' tf/cloudflare/variables.tf || echo "0"

Expected: 0 — no remaining ntfy references in tf/cloudflare/variables.tf.

• Step 4.5: Format + validate Terraform

cd tf/cloudflare
terraform fmt -check -recursive
terraform init -backend=false
terraform validate
cd ../..

Expected: clean fmt; Success! The configuration is valid.

• Step 4.6: Commit

jj --no-pager commit -m "feat(cloudflare): remove ntfy DNS entry from internal_services

Removes the ntfy entry from tf/cloudflare/variables.tf's internal_services
default map. HCP TFC applies the destroy on the main-cluster-cloudflare
workspace, removing the DNS record + tunnel route for the ntfy hostname.

After TFC apply, the ntfy hostname no longer resolves. Combined with
the ntfy namespace deletion (Task 2), there's no in-cluster service
for the hostname to route to even if it were resolved."

---

Task 5: Update docs to remove ntfy references

Files:

• Modify: docs/reference/services.md
• Modify: docs/reference/secrets.md
• Step 5.1: Find ntfy references in services.md

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
grep -n -B 2 -A 3 'ntfy' docs/reference/services.md | head -20

Likely a table row with the ntfy service name, namespace, ingress hostname, and a brief description. Note exact line range.

• Step 5.2: Remove the ntfy row from services.md

Use Edit to remove the entire ntfy row from the services catalog table. Preserve other rows and the table header/separator.

• Step 5.3: Find ntfy references in secrets.md

grep -n -B 2 -A 3 'ntfy' docs/reference/secrets.md | head -30

Likely 4 rows in the Vault paths table:

• fzymgc-house/cluster/ntfy (admin/writer/subscriber credentials for the ntfy server)
• fzymgc-house/cluster/ntfy-grafana (Grafana-side basic-auth tokens — already orphaned by PR5)

(Possibly more rows depending on how the docs structured the ntfy paths — count the actual hits.)

• Step 5.4: Remove the ntfy rows from secrets.md

Use Edit to remove every ntfy row from the Vault paths table. Preserve other rows.

• Step 5.5: Final grep — confirm no ntfy references in docs/

rg -i 'ntfy' docs/ 2>&1 | head

Expected: zero matches (or only matches in docs/engineering/ which are migration history — those stay). If any match in docs/operations/ or docs/reference/, investigate and remove unless it’s a historical reference that’s appropriate to keep.

• Step 5.6: Lint markdown

rumdl check docs/reference/services.md docs/reference/secrets.md

Expected: Success: No issues found in 2 files. If errors, run rumdl fmt then re-check.

• Step 5.7: Commit

jj --no-pager commit -m "docs(ntfy): remove from services catalog and secrets reference

Deletes the ntfy row from docs/reference/services.md and the ntfy +
ntfy-grafana rows from docs/reference/secrets.md. The ntfy stack is
fully decommissioned across all earlier commits in this PR; the docs
now reflect the post-migration state.

Migration design + per-PR plans in docs/engineering/ are historical
artifacts — they're left in place as the migration record."

---

Task 6: Pre-push verification

Files: none (verification only)

• Step 6.1: Confirm all expected commits exist

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager log -r 'main..@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Expected: 5 commits (4 implementation + 1 plan).

• Step 6.2: Confirm no secrets in the diff

jj --no-pager diff -r 'main..@-' | rg -i 'token|user.?key|password|secret' | rg -v 'secretKeyRef|secret/data|secretStoreRef|.token|api[_-]?token|writer_|admin_|subscriber_|external-secrets' || echo "no_secrets_good"

Expected: no_secrets_good. The diff touches references to ntfy secret keys but values themselves are Vault placeholders, not actual credentials.

• Step 6.3: Verify cluster-app Helm template still renders cleanly

cd argocd/cluster-app
helm template . --name-template cluster-app --namespace argocd >/dev/null
echo "exit=$?"
cd ../..

Expected: exit=0.

• Step 6.4: Run rumdl on the plan doc explicitly

rumdl check docs/engineering/plans/2026-05-10-pushover-pr6-decommission-ntfy.md

Expected: Success: No issues found. If errors, run rumdl fmt then jj squash --into <plan-change-id> to fold autofixes back into the plan commit.

• Step 6.5: Run all repo linters

lefthook run pre-commit --all-files 2>&1 | tail -30

Expected: hooks on touched files pass. Pre-existing failures on untouched files are NOT blockers.

• Step 6.6: Verify the working copy base is current

jj --no-pager git fetch
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

If main has moved, rebase:

jj --no-pager rebase -s 'roots(main..@-)' -d main --skip-emptied

• Step 6.7: Sanity-check the final ntfy footprint in the working tree

rg -l -i 'ntfy' --type yaml --type tf 2>&1 | head
rg -l -i 'ntfy' docs/ 2>&1 | head

Expected: only matches in docs/engineering/ (migration history). Zero matches in argocd/, tf/vault/, tf/cloudflare/, docs/reference/, docs/operations/. If any other matches, investigate.

---

Task 7: Push and open PR

Files: none (VCS + GitHub operations)

• Step 7.1: Create the bookmark and push

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager bookmark create feat/pushover-pr6-decommission-ntfy -r @-
jj --no-pager git push -b feat/pushover-pr6-decommission-ntfy

• Step 7.2: Open the PR

gh pr create \
  --head feat/pushover-pr6-decommission-ntfy \
  --base main \
  --title "feat(ntfy): decommission self-hosted ntfy stack (PR6/6)" \
  --body "$(cat <<'EOF'
## Summary

PR6/6 — final PR of the pushover + alertmanager migration. Fully removes the self-hosted ntfy stack: ArgoCD application + 10 manifest files, Terraform Vault policy + role (defensive — never had live consumers), Cloudflare DNS entry, and docs entries. Vault KV data destruction is a manual post-merge step (irreversible).

Spec: \`docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md\`
Plan: \`docs/engineering/plans/2026-05-10-pushover-pr6-decommission-ntfy.md\`

## Changes

- \`argocd/cluster-app/templates/ntfy.yaml\` — deleted (ArgoCD Application object).
- \`argocd/app-configs/ntfy/\` — entire directory deleted (10 manifest files: bootstrap-job, certificate, configmap, deployment, external-secret, ingressroute-internal, kustomization, namespace, pvc, service).
- \`tf/vault/policy-ntfy.tf\`, \`tf/vault/k8s-ntfy.tf\` — deleted (defensive policy + role; ESO never used them).
- \`tf/cloudflare/variables.tf\` — \`ntfy\` entry removed from \`internal_services\` map.
- \`docs/reference/services.md\`, \`docs/reference/secrets.md\` — ntfy rows removed.

## Pre-merge prerequisites

None. PR1-PR5 merged and verified end-to-end. The load-bearing prereq is that no \`GrafanaAlertRuleGroup\` or Grafana ntfy contact points remain (verified in Task 1.2 and 1.4) — meaning no real alert traffic depends on the ntfy stack.

## Post-merge sequence (controller-driven)

1. **HCP TFC applies on \`main-cluster-vault\`** — destroys 3 resources: \`vault_policy.ntfy\`, \`vault_kubernetes_auth_backend_role.ntfy\`, \`vault_policy.grafana_ntfy_tokens\`.
2. **HCP TFC applies on \`main-cluster-cloudflare\`** — destroys 1 DNS record (the ntfy hostname).
3. **ArgoCD prunes** — removes the entire ntfy namespace: Deployment, Service, IngressRoute, Certificate, PVC (Longhorn volume destroyed; message history lost), and the namespace itself.
4. **Manual Vault KV destruction (irreversible)** — after the above propagates and confirms no consumers remain:

   \`\`\`bash
   vault kv metadata delete secret/fzymgc-house/cluster/ntfy
   vault kv metadata delete secret/fzymgc-house/cluster/ntfy-grafana
   \`\`\`

   This destroys all versions and metadata permanently. Per the spec, run only after consumer-side cleanup is confirmed.

## Post-merge verification

- [ ] HCP TFC \`main-cluster-vault\`: 3 destroys, 0 adds.
- [ ] HCP TFC \`main-cluster-cloudflare\`: 1 destroy (DNS).
- [ ] \`kubectl get ns ntfy\` returns \`NotFound\`.
- [ ] \`vault kv list secret/fzymgc-house/cluster/\` returns no \`ntfy\` or \`ntfy-grafana\` entries (after manual destruction).
- [ ] DNS for the ntfy hostname does not resolve.
- [ ] \`rg -i 'ntfy' argocd/ tf/vault/ tf/cloudflare/ docs/reference/ docs/operations/\` returns zero matches.
- [ ] Migration is complete: all 6 PRs landed, no remaining ntfy footprint outside git history.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

• Step 7.3: Record the PR URL

The gh pr create command prints the PR URL.

---

Task 8: Post-merge — wait for ArgoCD prune + TFC applies

Files: none (live cluster verification)

This task runs after the PR merges. Use Monitor to track propagation.

• Step 8.1: Monitor: ntfy namespace deletion + TFC applies

prev=""
while true; do
  ns_state=$(kubectl --context fzymgc-house get ns ntfy -o jsonpath='{.status.phase}' 2>/dev/null || echo "gone")
  ntfy_pods=$(kubectl --context fzymgc-house -n ntfy get pods --no-headers 2>/dev/null | wc -l | tr -d ' ' || echo "?")
  vault_policy=$(kubectl --context fzymgc-house exec -n vault vault-0 -- vault read -field=name sys/policies/acl/ntfy 2>/dev/null || echo "gone")
  cur="ns=${ns_state} ntfyPods=${ntfy_pods} vaultPolicy=${vault_policy}"
  if [ "$cur" != "$prev" ]; then
    echo "$(date -u +%H:%M:%S) $cur"
    prev=$cur
  fi
  if [ "$ns_state" = "gone" ] && [ "$vault_policy" = "gone" ]; then
    echo "$(date -u +%H:%M:%S) READY"
    break
  fi
  sleep 15
done

The Vault vault read may need different shell access depending on cluster auth setup — adapt if the exec pattern is wrong for this cluster.

Wait for READY. The ntfy namespace deletion can be slow if there are finalizers; if it sticks Terminating for >5 min, investigate stuck finalizers.

• Step 8.2: Verify Cloudflare DNS removal

# Resolve the ntfy hostname (note: this is the hostname captured in Task 1.6)
dig +short ntfy.fzymgc.house @8.8.8.8 2>&1
dig +short ntfy.fzymgc.net @8.8.8.8 2>&1

Expected: empty result for whichever hostname was the ntfy DNS entry. (DNS propagation might take a minute.)

• Step 8.3: Verify Vault TF apply destroyed the policies

Use the terraform skill to check the latest run on main-cluster-vault. Expected: status applied, with ~3 destroyed, 0 added.

TFE_ORG=fzymgc-house /Users/sean/.claude/plugins/cache/fzymgc-house-skills/homelab/308716db5cf4/skills/terraform/scripts/terraform_mcp.py list-runs main-cluster-vault --limit 3

Pick the latest applied run and confirm via run-details.

---

Task 9: Manual Vault KV data destruction (REVERSIBLE: NO)

This task is the irreversible final step. Read carefully.

Per the spec’s PR6.3 ordering: this MUST happen AFTER Task 8 confirms ArgoCD prune + TFC applies have propagated AND no consumer is still trying to read the data. ESO will retry reads against the dead path for ~15 minutes if a consumer remains, producing log noise (not breakage, but noise).

• Step 9.1: Final consumer-clear check

# Confirm no ExternalSecret anywhere references the ntfy paths
kubectl --context fzymgc-house get externalsecret -A 2>&1 | rg -i 'ntfy' || echo "no_ntfy_es_good"

# Confirm the ntfy namespace is fully gone (no straggler resources)
kubectl --context fzymgc-house get ns ntfy 2>&1

Expected: no_ntfy_es_good; Error from server (NotFound): namespaces "ntfy" not found.

• Step 9.2: STOP — request user confirmation before running irreversible commands

The next two vault kv metadata delete commands are irreversible. They destroy ALL versions and metadata for the two ntfy paths. Once run, the data cannot be recovered from Vault (only from external backups, if any exist).

Get explicit user confirmation before proceeding. Show the user:

• The exact commands that will be run.
• The fact that this destroys all version history.
• A reminder that PR6.3 in the spec called this out as the deliberate end state.

If user does not confirm, STOP and leave the data in place — it can be destroyed in a future cleanup operation. The migration is otherwise complete.

• Step 9.3: Destroy the Vault KV paths (after user confirmation)

vault kv metadata delete secret/fzymgc-house/cluster/ntfy
vault kv metadata delete secret/fzymgc-house/cluster/ntfy-grafana

Both commands print confirmation. Capture the output for the migration completion log.

• Step 9.4: Verify destruction

vault kv list secret/fzymgc-house/cluster/ 2>&1 | rg -i 'ntfy' || echo "no_ntfy_in_vault_good"
vault kv get secret/fzymgc-house/cluster/ntfy 2>&1 | head -2
vault kv get secret/fzymgc-house/cluster/ntfy-grafana 2>&1 | head -2

Expected: no_ntfy_in_vault_good; both vault kv get commands return errors about the path not existing.

---

Task 10: Migration completion

Files: none (final summary)

• Step 10.1: Final sweep — confirm zero ntfy footprint outside git history

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster
git checkout main && git pull
rg -i 'ntfy' argocd/ tf/vault/ tf/cloudflare/ docs/reference/ docs/operations/ 2>&1 | head

Expected: zero matches. Migration history in docs/engineering/ is the only place “ntfy” still appears.

• Step 10.2: Update workspace state

The migration is complete. Workspace can be cleaned up:

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster
jj --no-pager workspace forget pushover-alertmanager
rm -rf .worktrees/feat-pushover-alertmanager

Or kept around if you want the migration plan/spec docs accessible from the worktree path.

• Step 10.3: Migration declared complete

All 6 PRs merged:

• PR1 (#995/#996/#997) — Grafana-side Pushover plumbing
• PR2 (#998/#999/#1000) — Alertmanager + receivers + ignoreDifferences
• PR3 (#1001) — NATS rules → PrometheusRule
• PR4 (#1002) — router-hosts rules → PrometheusRule
• PR5 (#1003) — Grafana ntfy contact points removed, notification policy re-pointed
• PR6 (this PR) — ntfy stack fully decommissioned

End state: Prometheus → Alertmanager → Pushover, with the four cluster-{infra,data,apps,edge} Pushover apps doing severity-priority delivery to the device. Grafana keeps dashboards + 12 Pushover contact points as escape hatch for ad-hoc alerts.

---

Risks and rollback

Risk	Detection	Rollback
ntfy namespace stuck Terminating due to finalizers	Step 8.1 monitor never sees ns=gone	kubectl get ns ntfy -o yaml shows finalizer; manually remove via kubectl patch ns ntfy --type=json -p='[{"op":"remove","path":"/spec/finalizers"}]'. Investigate which controller’s finalizer is stuck.
HCP TFC vault apply fails (e.g., policy in use)	TFC run shows error referencing one of the destroyed resources	Defensive policies have no live consumers — should not fail. If it does, the error message identifies the consumer; investigate. The PR can be reverted if needed (restores the .tf files; Vault re-creates the policies).
Cloudflare TFC apply fails	TFC run shows error referencing the DNS record	Should not fail — record removal is straightforward. If it does, manually remove the DNS record in Cloudflare UI.
Vault KV destruction (Task 9) runs before prune completes	ESO logs show “secret not found” repeating for ~15 min	Cosmetic noise, not breakage. ESO eventually gives up. Avoid by ensuring Step 8.1 READY before Task 9.
Vault data destroyed but a forgotten consumer surfaces later	Some workload (forgotten about) tries to read the path and gets denied	The data IS gone. If the workload is essential, recreate the credentials (new Pushover-style flow probably) and re-grant. Per spec: irreversibility is the deliberate end state.
Docs sweep misses an ntfy reference	Step 6.7 or Step 10.1 finds an unexpected match	Investigate the file; if it’s a legitimate reference (e.g., “previously used ntfy”), keep it; otherwise remove with a follow-up commit.

---

Out of scope for PR6

• Removing the spec/plans from docs/engineering/ — they’re migration history, kept intentionally.
• Removing the pushover Vault TF policy + role — that’s PR2’s defensive plumbing for Alertmanager, still in use.
• Removing the 12 Grafana Pushover contact points + ESO — PR5 explicitly kept these as the escape hatch.
• Pushover delivery-rate alert / quota dashboard — tracked as a follow-up bead; not part of this initiative.
• Reloader integration on the AM StatefulSet for hot-reload — manual kubectl rollout restart is the documented procedure (PR2’s docs/operations/alertmanager.md); revisit if rotations become frequent.