Pushover + Alertmanager Migration — PR3: NATS Rule Migration

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Migrate the four NATS alert rules from GrafanaAlertRuleGroup (Grafana-evaluated) to PrometheusRule (Prometheus-evaluated, routed by Alertmanager → Pushover). First real-traffic test of the AM routing tree set up in PR2.

Architecture: Replace argocd/app-configs/nats/grafana-alerts.yaml (which contains a GrafanaFolder + GrafanaAlertRuleGroup with 4 rules) with argocd/app-configs/nats/prometheus-rules.yaml (a single PrometheusRule resource with the same 4 rules, expressions hoisted from the Grafana __expr__ reducer style into native PromQL). Add the required domain: data label so Alertmanager routes them to pushover-data. Update the kustomization to swap the manifests. ArgoCD’s prune: true (already set on the nats app) removes the deleted Grafana resources from the cluster.

Tech Stack: Kubernetes (Prometheus Operator-managed PrometheusRule CRD, kube-prometheus-stack v84.5.0), PromQL, Alertmanager (deployed in PR2). No new infrastructure; this PR purely converts rule format.

Spec reference: docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md (the “PrometheusRule conversion pattern” section).

Workspace: .worktrees/feat-pushover-alertmanager (jj workspace pushover-alertmanager, parented to current main).

---

Manual prerequisites

None. PR1 (Grafana-side Pushover plumbing — #995/#996/#997) and PR2 (Alertmanager + receivers — #998/#999/#1000) are merged and verified. The infrastructure this PR depends on is fully live.

Sanity check before kicking off:

# Alertmanager pod must be 1/1
kubectl --context fzymgc-house -n prometheus get statefulset alertmanager-kps-kube-prometheus-stack-alertmanager -o jsonpath='{.status.readyReplicas}/{.spec.replicas}{"\n"}'

# Prometheus must have explicit ruleSelector (from PR2's monitoring-prometheus.yaml change)
kubectl --context fzymgc-house -n prometheus get prometheus kps-kube-prometheus-stack-prometheus -o jsonpath='{.spec.ruleSelector.matchLabels}{"\n"}'

Expected: 1/1 and {"release":"kps"} respectively. If either is wrong, stop — PR2 didn’t fully land.

---

Important context: noDataState semantics differ between Grafana and Prometheus

The Grafana rules use four noDataState values that have no direct Prometheus equivalent:

Grafana noDataState	Meaning	Prometheus equivalent
OK	If the query returns no data, treat as “not firing”	Default Prometheus behavior — no data means rule doesn’t evaluate
NoData	Treat no-data as a separate alerting state (DatasourceNoData)	No equivalent — Prometheus is silent on no-data
Alerting	If the query returns no data, FIRE the alert	Wrap the expression with absent_over_time(...) or or vector(0) to detect absence and trigger
Error	If the query errors, fire (separate from no-data)	Prometheus shows ALERTS{alertstate="firing"} only when expr matches; query errors produce a separate prometheus_rule_evaluation_failures_total metric (out of scope for this PR)

Three of the four NATS rules used noDataState: NoData or noDataState: OK — all behaviorally equivalent to Prometheus’s silent-on-no-data default. They translate cleanly with no expression change.

The fourth rule (nats-cluster-quorum-lost) uses noDataState: Alerting — alerting on metric absence is the intent (NATS pods unreachable = page). Prometheus needs absent semantics added to the expression to preserve this. The plan handles this in Task 2.

---

File structure

File	Status	Responsibility
argocd/app-configs/nats/prometheus-rules.yaml	Create	Single PrometheusRule resource in the prometheus namespace containing all four NATS alert rules. Each rule has labels: { severity, service: nats, domain: data } plus the release: kps label on the resource itself for kps Prometheus discovery.
argocd/app-configs/nats/kustomization.yaml	Modify	Replace - grafana-alerts.yaml with - prometheus-rules.yaml in the resources: list.
argocd/app-configs/nats/grafana-alerts.yaml	Delete	The GrafanaFolder and GrafanaAlertRuleGroup are removed; ArgoCD’s prune: true (on the nats app) deletes them from the cluster. The nats-alerts Grafana folder itself is also removed.

Cross-namespace placement note (mirrors PR1’s pattern): the nats ArgoCD app’s destination.namespace is nats, but argocd/app-configs/nats/prometheus-rules.yaml will carry metadata.namespace: prometheus so the kps Prometheus operator (which only watches the prometheus namespace by default) discovers it. The core-services AppProject allows namespace: *, so this is permitted. Same pattern as today’s grafana-alerts.yaml which sits in the same directory but specifies namespace: grafana.

---

Task 1: Verify workspace + PR2 infrastructure is live

Files: none (verification only)

• Step 1.1: Confirm you are in the correct workspace

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager git fetch
jj --no-pager workspace list | rg pushover-alertmanager
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

Expected: workspace listed; main is at PR #1000’s merge commit (fix(monitoring-prometheus): ignore ESO admission-webhook default fields) or later. If @ is parented to an older commit, run jj new main to reset.

• Step 1.2: Confirm Alertmanager is up

kubectl --context fzymgc-house -n prometheus get statefulset alertmanager-kps-kube-prometheus-stack-alertmanager -o jsonpath='{.status.readyReplicas}/{.spec.replicas}{"\n"}'

Expected: 1/1. If 0/1 or absent, stop — PR2 didn’t fully land and PR3 has nothing to route to.

• Step 1.3: Confirm Prometheus has the explicit ruleSelector

kubectl --context fzymgc-house -n prometheus get prometheus kps-kube-prometheus-stack-prometheus -o jsonpath='{.spec.ruleSelector.matchLabels}{"\n"}'

Expected: {"release":"kps"}. If empty or different, stop — without the explicit selector, our new PrometheusRule (which carries release: kps label) won’t be discovered.

• Step 1.4: Confirm the existing GrafanaAlertRuleGroup is currently active

kubectl --context fzymgc-house -n grafana get grafanaalertrulegroup nats-alerts -o jsonpath='{.status.conditions[?(@.type=="GroupSynchronized")].status}{"\n"}'

Expected: True. This is the rule we’re replacing; confirming it’s currently working ensures we have a clean baseline to compare post-merge against (the four NATS alerts still firing, just via a different path).

• Step 1.5: Snapshot current alert state (so post-merge diff is meaningful)

# Any active NATS alerts right now? (Probably none if cluster is healthy.)
kubectl --context fzymgc-house -n prometheus exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
  amtool --alertmanager.url=http://localhost:9093 alert query 2>&1 | rg -i 'nats|^Alertname' || echo "no active NATS alerts"

Expected: typically no active NATS alerts in a healthy cluster. Note any active ones — they should still appear in the same shape (perhaps with domain=data label added) after PR3 merges and Prometheus re-evaluates.

---

Task 2: Write the PrometheusRule manifest

Files:

• Create: argocd/app-configs/nats/prometheus-rules.yaml

• Step 2.1: Write the manifest

---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: nats
  namespace: prometheus
  labels:
    release: kps                 # required for kps Prometheus to discover this rule
spec:
  groups:
    - name: nats
      interval: 1m
      rules:
        # Critical: JetStream storage usage above 80% for 5 minutes
        - alert: NATSJetStreamStorageHigh
          expr: |
            (sum(nats_varz_jetstream_stats_storage)
             / sum(nats_varz_jetstream_config_max_storage)) > 0.8
          for: 5m
          labels:
            severity: critical
            service: nats
            domain: data
          annotations:
            summary: "NATS JetStream storage usage is high"
            description: "JetStream storage usage is above 80%."

        # Critical: NATS cluster lost quorum (fewer than 2 healthy peers OR all unreachable)
        # The Grafana version used noDataState=Alerting to fire when metrics were absent
        # entirely (NATS pods all down). The `absent(...)` clause preserves that intent in
        # native PromQL without relying on a Grafana-only "no-data" state.
        - alert: NATSClusterQuorumLost
          expr: |
            absent(up{job="nats/nats"}) == 1
            or count(up{job="nats/nats"} == 1) < 2
          for: 30s
          labels:
            severity: critical
            service: nats
            domain: data
          annotations:
            summary: "NATS cluster lost quorum"
            description: "NATS cluster has fewer than 2 healthy JetStream peers, or no NATS pods are reachable. The cluster cannot maintain consensus."

        # Warning: NATS reporting slow consumers (messages dropping)
        - alert: NATSSlowConsumers
          expr: rate(nats_varz_slow_consumers[5m]) > 0
          for: 5m
          labels:
            severity: warning
            service: nats
            domain: data
          annotations:
            summary: "NATS has slow consumers"
            description: "NATS is detecting slow consumers. Messages may be dropped."

        # Warning: high client connection count (potential leak or scale event)
        - alert: NATSHighConnectionCount
          expr: sum(nats_varz_connections) > 1000
          for: 5m
          labels:
            severity: warning
            service: nats
            domain: data
          annotations:
            summary: "NATS connection count is high"
            description: "NATS has a high number of client connections. Consider scaling or investigating connection leaks."

Note on naming: alert names use PascalCase (NATSJetStreamStorageHigh) per Prometheus convention, replacing the existing GrafanaAlertRuleGroup’s uid slugs (nats-jetstream-storage-high). The service: nats label preserves the relationship; the alertname change is acceptable because alerts are routed by labels, not by name.

Note on for: shortening: the Grafana rule uses 5m0s and 30s — Prometheus accepts both 5m and 5m0s; using 5m matches Prometheus convention.

• Step 2.2: Lint YAML

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
yamllint argocd/app-configs/nats/prometheus-rules.yaml

Expected: no output (exit 0).

• Step 2.3: Validate the PromQL with promtool (optional if not installed locally)

If promtool is available locally:

promtool check rules argocd/app-configs/nats/prometheus-rules.yaml

Expected: SUCCESS: 4 rules found.

If promtool isn’t installed (brew install prometheus to add it locally), skip — the post-merge kubectl get prometheusrule step catches schema and PromQL errors when the operator processes the resource.

• Step 2.4: Commit

jj --no-pager commit -m "feat(nats): migrate alert rules from GrafanaAlertRuleGroup to PrometheusRule

Converts the four NATS alert rules (JetStream storage, cluster quorum,
slow consumers, high connections) from Grafana-evaluated to Prometheus-
evaluated. Each rule now carries labels { severity, service: nats,
domain: data } so Alertmanager (added in PR2) routes them to the
pushover-data receiver and onward to the cluster-data Pushover app.

Expressions hoist the threshold into native PromQL (e.g. > 0.8 instead
of relying on Grafana's __expr__ reducer + threshold step). The cluster
quorum rule preserves the Grafana noDataState=Alerting intent via an
absent() clause so the alert still fires when all NATS pods are
unreachable, not just when up < 2.

The replaced GrafanaAlertRuleGroup is removed in the next commit. The
PrometheusRule lives in metadata.namespace: prometheus (operator
discovers only that namespace) while the file sits with the nats app's
manifests — same cross-namespace pattern the existing GrafanaAlertRuleGroup
used (it sits here but deploys to namespace: grafana)."

Verify:

jj --no-pager log -r '@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

---

Task 3: Update kustomization to swap manifests

Files:

• Modify: argocd/app-configs/nats/kustomization.yaml

• Step 3.1: Edit kustomization.yaml

Current contents:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - external-secret.yaml
  - certificate.yaml
  - network-policy.yaml
  - grafana-alerts.yaml
  - account-push-job.yaml

Replace with:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - external-secret.yaml
  - certificate.yaml
  - network-policy.yaml
  - prometheus-rules.yaml
  - account-push-job.yaml

Only one line changes — grafana-alerts.yaml → prometheus-rules.yaml. The position in the list (between network-policy.yaml and account-push-job.yaml) is preserved.

• Step 3.2: Validate kustomize render

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
kubectl --context fzymgc-house apply --dry-run=client -k argocd/app-configs/nats 2>&1 | tee /tmp/kustomize-nats.txt
rg 'prometheusrule|GrafanaAlertRuleGroup|GrafanaFolder' /tmp/kustomize-nats.txt

Expected:

• prometheusrule.monitoring.coreos.com/nats created (dry run) appears.
• GrafanaAlertRuleGroup and GrafanaFolder should NOT appear (they’re no longer in the rendered kustomization).
• Step 3.3: Lint YAML

yamllint argocd/app-configs/nats/kustomization.yaml

Expected: no output.

• Step 3.4: Commit

jj --no-pager commit -m "feat(nats): swap kustomization to point at prometheus-rules

Replaces grafana-alerts.yaml with prometheus-rules.yaml in the nats
app's kustomization resources list. The position is preserved
(between network-policy and account-push-job). The next commit
deletes the now-orphaned grafana-alerts.yaml file; ArgoCD's prune=true
on the nats app removes the GrafanaFolder + GrafanaAlertRuleGroup
from the cluster on next sync."

---

Task 4: Delete the old GrafanaAlertRuleGroup file

Files:

• Delete: argocd/app-configs/nats/grafana-alerts.yaml

• Step 4.1: Delete the file

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
rm argocd/app-configs/nats/grafana-alerts.yaml

• Step 4.2: Verify the directory state

ls argocd/app-configs/nats/

Expected:

account-push-job.yaml
certificate.yaml
external-secret.yaml
kustomization.yaml
network-policy.yaml
prometheus-rules.yaml
values.yaml

No grafana-alerts.yaml. The new prometheus-rules.yaml is present.

• Step 4.3: Confirm jj sees the deletion

jj --no-pager st

Expected: D argocd/app-configs/nats/grafana-alerts.yaml in the working copy changes.

• Step 4.4: Commit

jj --no-pager commit -m "chore(nats): remove legacy GrafanaAlertRuleGroup manifest

Deletes argocd/app-configs/nats/grafana-alerts.yaml. The file is no
longer referenced by the kustomization (previous commit), and the
PrometheusRule equivalent is now in place. ArgoCD's prune=true on
the nats app removes the now-unmanaged GrafanaFolder 'nats-alerts'
and GrafanaAlertRuleGroup 'nats-alerts' from the cluster on next
sync. The Grafana operator will then delete the rules from Grafana's
Alertmanager database, leaving Prometheus as the sole evaluator for
NATS alerts."

Verify:

jj --no-pager log -r 'main..@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Expected (3 commits, newest first):

<id> chore(nats): remove legacy GrafanaAlertRuleGroup manifest
<id> feat(nats): swap kustomization to point at prometheus-rules
<id> feat(nats): migrate alert rules from GrafanaAlertRuleGroup to PrometheusRule

---

Task 5: Pre-push verification

Files: none (verification only)

• Step 5.1: Confirm all expected commits exist

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager log -r 'main..@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Expected: 4 commits — the 3 implementation commits above plus the plan commit (docs(superpowers): add PR3 implementation plan...). If anything is missing, return BLOCKED.

• Step 5.2: Confirm no secrets in the diff

jj --no-pager diff -r 'main..@-' | rg -i 'token|user.?key|password|secret' | rg -v 'secretKeyRef|secret/data|secretStoreRef|.token|api[_-]?token' || echo "no_secrets_good"

Expected: no_secrets_good. The diff should be entirely PromQL + YAML metadata — no credential material.

• Step 5.3: Verify cluster-app Helm template still renders cleanly

cd argocd/cluster-app
helm template . --name-template cluster-app --namespace argocd >/dev/null
echo "exit=$?"

Expected: exit=0. PR3 doesn’t touch argocd/cluster-app/templates/, so this should pass — but verify regardless because any future Helm-template breakage would mask PR3’s apply (per the cluster-app double-rendering trap from PR2 #999).

• Step 5.4: Run all repo linters

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
lefthook run pre-commit --all-files 2>&1 | tail -30

Expected: hooks on touched files (lint-yaml, lint-markdown, fmt-yaml) pass. Pre-existing failures on files this PR doesn’t touch are NOT blockers.

• Step 5.5: Verify the working copy base is current

jj --no-pager git fetch
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

If main has moved during PR3 work, rebase:

jj --no-pager rebase -s 'roots(main..@-)' -d main --skip-emptied

Then re-run Step 5.1 to confirm chain integrity.

---

Task 6: Push and open PR

Files: none (VCS + GitHub operations)

• Step 6.1: Create the bookmark and push

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager bookmark create feat/pushover-pr3-nats-rule-migration -r @-
jj --no-pager git push -b feat/pushover-pr3-nats-rule-migration

Expected: push succeeds.

• Step 6.2: Open the PR

gh pr create \
  --head feat/pushover-pr3-nats-rule-migration \
  --base main \
  --title "feat(nats): migrate alert rules to PrometheusRule (PR3/6)" \
  --body "$(cat <<'EOF'
## Summary

PR3 of the pushover + alertmanager migration. Converts the four NATS alert rules from `GrafanaAlertRuleGroup` (Grafana-evaluated, routed via Grafana contact points) to `PrometheusRule` (Prometheus-evaluated, routed via Alertmanager → Pushover). First real-traffic test of the AM routing tree set up in PR2.

Spec: \`docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md\`
Plan: \`docs/engineering/plans/2026-05-10-pushover-pr3-nats-rule-migration.md\`

## Changes

- \`argocd/app-configs/nats/prometheus-rules.yaml\` — new PrometheusRule with all four NATS rules. Each rule has labels \`{ severity, service: nats, domain: data }\`. The cluster-quorum rule preserves the Grafana \`noDataState: Alerting\` intent via \`absent(up{...}) == 1\` so it still fires when NATS pods are entirely unreachable.
- \`argocd/app-configs/nats/grafana-alerts.yaml\` — deleted. ArgoCD prune removes the cluster-side \`GrafanaFolder\` and \`GrafanaAlertRuleGroup\` on next sync.
- \`argocd/app-configs/nats/kustomization.yaml\` — swap \`grafana-alerts.yaml\` for \`prometheus-rules.yaml\` in resources list.

## Pre-merge prerequisites

None. PR1 (Pushover plumbing) and PR2 (Alertmanager + receivers) are merged and verified end-to-end.

## Post-merge verification

- [ ] HCP TFC: no Vault workspace run triggered (no TF changes).
- [ ] ArgoCD \`nats\` app syncs cleanly; \`Synced\` + \`Healthy\`.
- [ ] \`kubectl -n prometheus get prometheusrule nats\` exists with 4 rules.
- [ ] \`kubectl -n prometheus exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager -- promtool check rules /etc/prometheus/rules/...\` (or via Prometheus's \`/api/v1/rules\` endpoint) shows the 4 rules loaded.
- [ ] \`kubectl -n grafana get grafanaalertrulegroup nats-alerts\` returns \`NotFound\`.
- [ ] \`kubectl -n grafana get grafanafolder nats-alerts\` returns \`NotFound\`.
- [ ] In Grafana UI: Alerting → Folders no longer shows "NATS Alerts".
- [ ] Smoketest: temporarily lower one threshold (e.g. \`> 0.8\` → \`> 0\` on the storage rule) in a throwaway commit OR manually create an alert via \`amtool\` with \`severity=critical domain=data alertname=NATSManualSmoketest\` to confirm Pushover delivery via \`cluster-data\` app at priority 1.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

• Step 6.3: Record the PR URL

The gh pr create command prints the PR URL. Capture it for the post-merge verification.

---

Task 7: Post-merge verification (executes after CI + merge)

Files: none (live cluster verification)

This task runs after the PR merges and ArgoCD applies the changes.

• Step 7.1: Wait for ArgoCD sync

Use the established Monitor pattern (per memory: prefer Monitor over silent polling). Watch:

• nats ArgoCD app: Synced + Healthy
• New PrometheusRule nats exists in prometheus namespace
• Old GrafanaAlertRuleGroup nats-alerts in grafana namespace is NotFound
• Old GrafanaFolder nats-alerts in grafana namespace is NotFound

Sample Monitor command shape (adapt as needed):

prev=""
while true; do
  na_sync=$(kubectl --context fzymgc-house get application nats -n argocd -o jsonpath='{.status.sync.status}/{.status.health.status}' 2>/dev/null || echo "?/?")
  pr_present=$(kubectl --context fzymgc-house -n prometheus get prometheusrule nats -o jsonpath='{.metadata.name}' 2>/dev/null || echo "absent")
  garg_gone=$(kubectl --context fzymgc-house -n grafana get grafanaalertrulegroup nats-alerts -o jsonpath='{.metadata.name}' 2>/dev/null || echo "gone")
  gf_gone=$(kubectl --context fzymgc-house -n grafana get grafanafolder nats-alerts -o jsonpath='{.metadata.name}' 2>/dev/null || echo "gone")
  cur="nats=${na_sync} pr=${pr_present} garg=${garg_gone} gf=${gf_gone}"
  if [ "$cur" != "$prev" ]; then
    echo "$(date -u +%H:%M:%S) $cur"
    prev=$cur
  fi
  if [ "$na_sync" = "Synced/Healthy" ] && [ "$pr_present" = "nats" ] && [ "$garg_gone" = "gone" ] && [ "$gf_gone" = "gone" ]; then
    echo "$(date -u +%H:%M:%S) READY"
    break
  fi
  sleep 15
done

• Step 7.2: Confirm Prometheus loaded the rules

kubectl --context fzymgc-house -n prometheus exec prometheus-kps-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  wget -qO- http://localhost:9090/api/v1/rules 2>&1 | jq '.data.groups[] | select(.name == "nats") | {name, rules: [.rules[].name]}'

Expected:

{
  "name": "nats",
  "rules": [
    "NATSJetStreamStorageHigh",
    "NATSClusterQuorumLost",
    "NATSSlowConsumers",
    "NATSHighConnectionCount"
  ]
}

If wget isn’t available in the prometheus pod, swap for curl or kubectl port-forward svc/kps-kube-prometheus-stack-prometheus 9090:9090 and curl from local.

• Step 7.3: Confirm Grafana side is cleaned up

kubectl --context fzymgc-house -n grafana get grafanaalertrulegroup nats-alerts 2>&1
kubectl --context fzymgc-house -n grafana get grafanafolder nats-alerts 2>&1

Expected: both return Error from server (NotFound).

In the Grafana UI: Alerting → Alert rules. The “NATS Alerts” folder should no longer appear; NATSJetStreamStorageHigh etc. should not appear in the rule list. (The Grafana operator removes the rules from Grafana’s database when the CRDs are deleted.)

• Step 7.4: Smoketest end-to-end alert delivery

Two options (pick one):

Option A (recommended for proving the path): Manual amtool alert injection. This bypasses Prometheus entirely but exercises the full Alertmanager → Pushover path with the new label set:

kubectl --context fzymgc-house -n prometheus exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
  amtool --alertmanager.url=http://localhost:9093 alert add \
    alertname=NATSManualSmoketest \
    severity=critical \
    service=nats \
    domain=data \
    --annotation=summary="PR3 manual smoketest of NATS alert routing"

Wait ~30s. Pushover notification arrives via the cluster-data app at priority 1 (high; bypasses quiet hours).

Option B (proves Prometheus evaluation, more invasive): Temporarily lower a threshold in a throwaway commit (e.g., set NATSHighConnectionCount from > 1000 to > 0), wait for Prometheus to re-evaluate (default 1m interval), confirm the alert fires through to Pushover, revert the throwaway. Higher confidence but requires a second PR cycle.

For PR3, Option A is sufficient — the actual Prometheus PromQL is identical to what Grafana evaluated, so trusting the smoketest of label-driven routing is enough. Option B is appropriate before PR4 if you want extra confidence.

• Step 7.5: Confirm no rule-evaluation errors in Prometheus

kubectl --context fzymgc-house -n prometheus exec prometheus-kps-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=prometheus_rule_evaluation_failures_total{rule_group=~".*nats.*"}' 2>&1 | jq '.data.result[] | {metric: .metric.rule_group, value: .value[1]}'

Expected: empty result (no failures) or all values are "0". If any rule shows a non-zero failure count, check the rule’s evaluation log:

kubectl --context fzymgc-house -n prometheus logs prometheus-kps-kube-prometheus-stack-prometheus-0 -c prometheus --tail=200 | rg -i 'nats|rule.*err'

• Step 7.6: Update workspace state for PR4

The workspace stays at .worktrees/feat-pushover-alertmanager for PR4 (router-hosts rule migration). After PR3 merges:

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager git fetch
jj --no-pager new main

Workspace ready for PR4.

---

Risks and rollback

Risk	Detection	Rollback
New PrometheusRule fails schema validation	kubectl get prometheusrule nats -o yaml shows error in events; Prometheus operator logs show rejection	Revert the merge commit; the GrafanaAlertRuleGroup file isn’t in main anymore so a clean revert restores both kustomization + the deleted file. NATS alerting reverts to the Grafana path.
PromQL expression doesn’t return expected data	prometheus_rule_evaluation_failures_total shows non-zero; AM never sees alerts that should fire	Investigate the expression; common cause: nats_varz_* metric labels changed in a NATS exporter upgrade. Compare against actual scraped metrics: kubectl exec prometheus-... -- wget -qO- 'http://localhost:9090/api/v1/query?query=nats_varz_jetstream_stats_storage'
absent() clause on cluster-quorum rule fires falsely (e.g., during normal NATS pod restart)	Pushover delivery for NATSClusterQuorumLost arrives during a planned NATS restart	Either: (a) increase the for: 30s to e.g. for: 5m to ride out short outages, or (b) drop the absent() clause and rely on the count(... < 2) form alone. The Grafana version had for: 30s; preserved here.
ArgoCD prune doesn’t actually remove the Grafana CRDs	Grafana UI still shows the “NATS Alerts” folder after merge	Manually delete: kubectl -n grafana delete grafanaalertrulegroup nats-alerts; kubectl -n grafana delete grafanafolder nats-alerts. Investigate why prune didn’t run (check ArgoCD app’s syncOptions includes Prune=true).
Alertmanager routing misses the new label	NATS alerts fire in Prometheus but no Pushover delivery (or wrong receiver)	Check amtool alert query for the alert; verify it has domain=data label; check AM config dump (amtool config show) for the pushover-data route — should match domain="data".

---

Out of scope for PR3 (handled in subsequent PRs)

• router-hosts PrometheusRule migration (PR4) — same shape as this PR but domain: edge and a different file path (argocd/app-configs/router-hosts-alerts/).
• Grafana ntfy contact-point removal (PR5) — at that point all alerts route through Alertmanager; the Grafana ntfy receivers become dead code.
• ntfy stack decommission (PR6) — full removal of the ntfy ArgoCD app, Vault data, Cloudflare DNS, and docs.
• CI lint asserting all PrometheusRules carry domain + severity — tracked as a follow-up bead.
• Promotion to emergency priority (Pushover priority 2) for NATSClusterQuorumLost if cluster-quorum-loss should bypass-and-ack-required — design decision deferred per spec.