Pushover + Alertmanager Migration — PR4: router-hosts Rule Migration

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Migrate the four router-hosts alert rules from GrafanaAlertRuleGroup (Grafana-evaluated) to PrometheusRule (Prometheus-evaluated, routed by Alertmanager → Pushover via the cluster-edge app). Same shape as PR3 but domain: edge.

Architecture: Replace argocd/app-configs/router-hosts-alerts/grafana-alerts.yaml (GrafanaFolder + GrafanaAlertRuleGroup with 4 rules) with argocd/app-configs/router-hosts-alerts/prometheus-rules.yaml (one PrometheusRule resource, expressions hoisted into native PromQL with thresholds inlined). Add domain: edge label so Alertmanager routes them to pushover-edge. Update kustomization. ArgoCD’s prune: true removes the deleted Grafana resources.

Tech Stack: Kubernetes (Prometheus Operator-managed PrometheusRule CRD, kube-prometheus-stack v84.5.0), PromQL, Alertmanager (deployed in PR2). No new infrastructure.

Spec reference: docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md (the “PrometheusRule conversion pattern” section).

Workspace: .worktrees/feat-pushover-alertmanager (jj workspace pushover-alertmanager, parented to current main).

---

Manual prerequisites

None. PR1 (#995/#996/#997), PR2 (#998/#999/#1000), and PR3 (#1001) are all merged and verified. The infrastructure this PR depends on is fully live.

Sanity check before kicking off:

# Alertmanager pod must be 1/1 (single replica StatefulSet)
kubectl --context fzymgc-house -n prometheus get statefulset alertmanager-kps-kube-prometheus-stack-alertmanager -o jsonpath='{.status.readyReplicas}/{.spec.replicas}{"\n"}'

# Prometheus must have explicit ruleSelector { release: kps } (from PR2)
kubectl --context fzymgc-house -n prometheus get prometheus kps-kube-prometheus-stack-prometheus -o jsonpath='{.spec.ruleSelector.matchLabels}{"\n"}'

# PR3's NATS PrometheusRule should already be loaded (proves the new path is live)
kubectl --context fzymgc-house -n prometheus get prometheusrule nats -o jsonpath='{.metadata.name}{"\n"}'

Expected: 1/1, {"release":"kps"}, and nats respectively.

---

Important context

noDataState carries over differently per rule

The four router-hosts rules all use noDataState: OK — equivalent to Prometheus’s silent-on-no-data default. They translate without needing absent() clauses (unlike PR3’s NATSClusterQuorumLost which used noDataState: Alerting).

ContainerDown’s or vector(0) is intentional

The original RouterHostsContainerDown expression already wraps count(...) with or vector(0) to make the count return 0 (firing) when no container_last_seen samples are present. This is the same intent as absent() but expressed as a fallback value. Preserve verbatim — it’s a deliberate idiom by whoever wrote the original rule, and it works the same in native Prometheus as it did in Grafana.

HighErrorRate has divide-by-zero protection

The original expression uses sum(rate(...)) > 0 OR vector(1) as the denominator, ensuring a non-zero divisor when there’s no traffic. Preserve verbatim — same idiom, same correctness.

for: 0s on OOMKilled is intentional

The RouterHostsOOMKilled rule fires immediately on detection (no for: window) because OOM events are point-in-time and you want to know the moment they happen. Preserve for: 0s.

---

File structure

File	Status	Responsibility
argocd/app-configs/router-hosts-alerts/prometheus-rules.yaml	Create	Single PrometheusRule resource in the prometheus namespace containing all four router-hosts alert rules. Each rule has labels: { severity, service: router-hosts, domain: edge } plus release: kps on the resource itself.
argocd/app-configs/router-hosts-alerts/kustomization.yaml	Modify	Replace - grafana-alerts.yaml with - prometheus-rules.yaml.
argocd/app-configs/router-hosts-alerts/grafana-alerts.yaml	Delete	ArgoCD prune removes the cluster-side GrafanaFolder + GrafanaAlertRuleGroup on next sync.

Cross-namespace placement note (mirrors PR3 + the existing pattern): the router-hosts-alerts ArgoCD app’s destination.namespace is grafana, but the new prometheus-rules.yaml will carry metadata.namespace: prometheus so the kps Prometheus operator discovers it. The observability AppProject (verified) allows namespace: *, so this is permitted.

---

Task 1: Verify workspace + PR2/PR3 infrastructure is live

Files: none (verification only)

• Step 1.1: Confirm you are in the correct workspace, parented to current main

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager git fetch
jj --no-pager workspace list | rg pushover-alertmanager
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

Expected: workspace listed; main is at PR #1001’s merge commit (feat(nats): migrate alert rules to PrometheusRule (PR3/6)) or later. If @ is parented to an older commit, run jj new main.

• Step 1.2: Confirm Alertmanager is up + ESO secret materialised

kubectl --context fzymgc-house -n prometheus get statefulset alertmanager-kps-kube-prometheus-stack-alertmanager -o jsonpath='{.status.readyReplicas}/{.spec.replicas}{"\n"}'

Expected: 1/1.

• Step 1.3: Confirm Prometheus has the explicit ruleSelector and PR3’s NATS rule loaded

kubectl --context fzymgc-house -n prometheus get prometheus kps-kube-prometheus-stack-prometheus -o jsonpath='{.spec.ruleSelector.matchLabels}{"\n"}'
kubectl --context fzymgc-house -n prometheus get prometheusrule nats -o jsonpath='{.metadata.name}{"\n"}'

Expected: {"release":"kps"} and nats. Both confirm the rule-discovery + PR3-path are healthy.

• Step 1.4: Confirm the existing router-hosts-alerts GrafanaAlertRuleGroup is currently active

kubectl --context fzymgc-house -n grafana get grafanaalertrulegroup router-hosts-alerts -o jsonpath='{.status.conditions[?(@.type=="AlertGroupSynchronized")].status}{"\n"}'

Expected: True. Note: condition type is AlertGroupSynchronized (per PR3 task 1 discovery).

• Step 1.5: Snapshot current alert state

kubectl --context fzymgc-house -n prometheus exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
  amtool --alertmanager.url=http://localhost:9093 alert query 2>&1 | rg -i 'router|^Alertname' || echo "no active router-hosts alerts"

Expected: typically no active router-hosts alerts. Note any active ones — they should reappear after PR4 merges and Prometheus re-evaluates.

---

Task 2: Write the PrometheusRule manifest

Files:

• Create: argocd/app-configs/router-hosts-alerts/prometheus-rules.yaml

• Step 2.1: Write the manifest

---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: router-hosts
  namespace: prometheus
  labels:
    release: kps                 # required for kps Prometheus to discover this rule
spec:
  groups:
    - name: router-hosts
      interval: 1m
      rules:
        # Critical: container is not running (no cAdvisor metrics for 5 minutes)
        # The `or vector(0)` fallback ensures count() returns 0 (firing) when
        # container_last_seen samples are absent entirely — same intent as the
        # Grafana version's noDataState semantics, but expressed inline so the
        # Prometheus rule fires even when the metric is missing.
        - alert: RouterHostsContainerDown
          expr: |
            (count(last_over_time(container_last_seen{name="router-hosts-server", host="firewalla"}[5m]))
             or vector(0)) < 1
          for: 5m
          labels:
            severity: critical
            service: router-hosts
            domain: edge
          annotations:
            summary: "router-hosts container is not running"
            description: "No container metrics received from router-hosts-server for 5 minutes. The container may be down or crashed."

        # Critical: gRPC error rate above 5% over 5-minute window
        # The `OR vector(0)` numerator and `> 0 OR vector(1)` denominator
        # protect against divide-by-zero when there's no traffic. Preserved
        # verbatim from the Grafana version.
        - alert: RouterHostsHighErrorRate
          expr: |
            100 * (
              sum(rate(router_hosts_requests_total{status="error"}[5m])) OR vector(0)
            ) / (
              sum(rate(router_hosts_requests_total[5m])) > 0 OR vector(1)
            ) > 5
          for: 5m
          labels:
            severity: critical
            service: router-hosts
            domain: edge
          annotations:
            summary: "router-hosts gRPC error rate is high"
            description: "Error rate is above 5%."

        # Warning: post-edit hook executions failing
        # NOTE: router_hosts_hook_executions_total metric is not yet emitted by
        # router-hosts itself — Prometheus's silent-on-no-data behavior keeps
        # this rule quiet until the metric appears.
        - alert: RouterHostsHookFailures
          expr: increase(router_hosts_hook_executions_total{status="error"}[5m]) > 0
          for: 5m
          labels:
            severity: warning
            service: router-hosts
            domain: edge
          annotations:
            summary: "router-hosts post-edit hooks are failing"
            description: "Hook failures detected in the last 5 minutes. DNS updates may not be taking effect."

        # Warning: container OOM killed (fires immediately on detection)
        - alert: RouterHostsOOMKilled
          expr: increase(container_oom_events_total{name="router-hosts-server", host="firewalla"}[5m]) > 0
          for: 0s
          labels:
            severity: warning
            service: router-hosts
            domain: edge
          annotations:
            summary: "router-hosts container was OOM killed"
            description: "The router-hosts-server container experienced an out-of-memory event."

Notes:

• Alert names use PascalCase (Prometheus convention), replacing the Grafana uid slugs.
• for: durations use Prometheus shorthand: 5m instead of 5m0s, 0s for immediate.
• All four rules carry domain: edge (the routing key for pushover-edge in Alertmanager) and service: router-hosts.
• Step 2.2: Lint YAML

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
yamllint argocd/app-configs/router-hosts-alerts/prometheus-rules.yaml

Expected: no output.

• Step 2.3: Validate the PromQL with promtool (optional)

If promtool is available locally:

yq '.spec' argocd/app-configs/router-hosts-alerts/prometheus-rules.yaml | promtool check rules /dev/stdin

Expected: SUCCESS: 4 rules found. Skip if not installed.

• Step 2.4: Commit

jj --no-pager commit -m "feat(router-hosts): migrate alert rules from GrafanaAlertRuleGroup to PrometheusRule

Converts the four router-hosts alert rules (container down, high error
rate, hook failures, OOM killed) from Grafana-evaluated to Prometheus-
evaluated. Each rule now carries labels { severity, service:
router-hosts, domain: edge } so Alertmanager (PR2) routes them to the
pushover-edge receiver and onward to the cluster-edge Pushover app.

Expressions hoist the threshold into native PromQL. The container-down
rule preserves the original 'or vector(0)' fallback that makes count()
return 0 when container_last_seen samples are absent — same intent as
the Grafana version's noDataState semantics. The error-rate rule
preserves its divide-by-zero protection idiom verbatim. OOM keeps
for: 0s for immediate firing.

The replaced GrafanaAlertRuleGroup is removed in the next commit. The
PrometheusRule lives in metadata.namespace: prometheus while the file
sits with the router-hosts-alerts app's manifests — same cross-namespace
pattern PR3 used."

Verify:

jj --no-pager log -r '@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

---

Task 3: Update kustomization to swap manifests

Files:

• Modify: argocd/app-configs/router-hosts-alerts/kustomization.yaml

• Step 3.1: Edit kustomization.yaml

Current contents:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - grafana-alerts.yaml

Replace with:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - prometheus-rules.yaml

Single-line swap. Position is preserved (it’s the only entry).

• Step 3.2: Validate kustomize render

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
kubectl --context fzymgc-house apply --dry-run=client -k argocd/app-configs/router-hosts-alerts 2>&1 | tee /tmp/kustomize-rh.txt
rg 'prometheusrule|grafanaalertrulegroup|grafanafolder' /tmp/kustomize-rh.txt

Expected: prometheusrule.monitoring.coreos.com/router-hosts created (dry run) appears. No grafanaalertrulegroup or grafanafolder in output.

• Step 3.3: Lint YAML

yamllint argocd/app-configs/router-hosts-alerts/kustomization.yaml

Expected: no output.

• Step 3.4: Commit

jj --no-pager commit -m "feat(router-hosts): swap kustomization to point at prometheus-rules

Replaces grafana-alerts.yaml with prometheus-rules.yaml in the
router-hosts-alerts app's kustomization. The next commit deletes the
now-orphaned grafana-alerts.yaml file; ArgoCD's prune=true on the
router-hosts-alerts app removes the GrafanaFolder + GrafanaAlertRuleGroup
from the cluster on next sync."

---

Task 4: Delete the old GrafanaAlertRuleGroup file

Files:

• Delete: argocd/app-configs/router-hosts-alerts/grafana-alerts.yaml

• Step 4.1: Delete the file

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
rm argocd/app-configs/router-hosts-alerts/grafana-alerts.yaml

• Step 4.2: Verify the directory state

ls argocd/app-configs/router-hosts-alerts/

Expected:

kustomization.yaml
prometheus-rules.yaml

No grafana-alerts.yaml.

• Step 4.3: Confirm jj sees the deletion

jj --no-pager st

Expected: D argocd/app-configs/router-hosts-alerts/grafana-alerts.yaml.

• Step 4.4: Commit

jj --no-pager commit -m "chore(router-hosts): remove legacy GrafanaAlertRuleGroup manifest

Deletes argocd/app-configs/router-hosts-alerts/grafana-alerts.yaml.
ArgoCD's prune=true on the router-hosts-alerts app removes the
GrafanaFolder 'router-hosts-alerts' and GrafanaAlertRuleGroup
'router-hosts-alerts' from the cluster on next sync. The Grafana
operator removes the rules from Grafana's database, leaving Prometheus
as the sole evaluator for router-hosts alerts."

Verify both commits landed:

jj --no-pager log -r 'main..@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"' | head -5

Expected (4 commits, newest first):

<id> chore(router-hosts): remove legacy GrafanaAlertRuleGroup manifest
<id> feat(router-hosts): swap kustomization to point at prometheus-rules
<id> feat(router-hosts): migrate alert rules from GrafanaAlertRuleGroup to PrometheusRule
<id> docs(superpowers): add PR4 implementation plan for router-hosts rule migration

---

Task 5: Pre-push verification

Files: none (verification only)

• Step 5.1: Confirm all expected commits exist

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager log -r 'main..@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Expected: 4 commits.

• Step 5.2: Confirm no secrets in the diff

jj --no-pager diff -r 'main..@-' | rg -i 'token|user.?key|password|secret' | rg -v 'secretKeyRef|secret/data|secretStoreRef|.token|api[_-]?token' || echo "no_secrets_good"

Expected: no_secrets_good.

• Step 5.3: Verify cluster-app Helm template still renders cleanly

cd argocd/cluster-app
helm template . --name-template cluster-app --namespace argocd >/dev/null
echo "exit=$?"
cd ../..

Expected: exit=0. PR4 doesn’t touch argocd/cluster-app/templates/, so this should pass — verify regardless (per cluster-app double-rendering trap memory).

• Step 5.4: Run rumdl on the plan doc explicitly (catches MD031/MD032/MD040 that lefthook autofixes silently)

rumdl check docs/engineering/plans/2026-05-10-pushover-pr4-router-hosts-rule-migration.md

Expected: Success: No issues found. If errors, run rumdl fmt on the plan, then jj squash --into <plan-change-id> to fold the autofixes back into the plan commit. CI’s Validate Changes runs rumdl check (no autofix) — must pass cleanly there.

• Step 5.5: Run all repo linters

lefthook run pre-commit --all-files 2>&1 | tail -30

Expected: hooks on touched files (lint-yaml, lint-markdown, fmt-yaml) pass. Pre-existing failures on untouched files (older docs archives, OpenClaw GHA workflow, scripts/migrate-secrets-to-vault.py) are NOT blockers — flag only.

• Step 5.6: Verify the working copy base is current

jj --no-pager git fetch
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

If main has moved, rebase:

jj --no-pager rebase -s 'roots(main..@-)' -d main --skip-emptied

Then re-run Step 5.1.

---

Task 6: Push and open PR

Files: none (VCS + GitHub operations)

• Step 6.1: Create the bookmark and push

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager bookmark create feat/pushover-pr4-router-hosts-rule-migration -r @-
jj --no-pager git push -b feat/pushover-pr4-router-hosts-rule-migration

Expected: push succeeds.

• Step 6.2: Open the PR

Use explicit --head/--base flags (jj-workspace gh-quirk per PR2/PR3 lessons):

gh pr create \
  --head feat/pushover-pr4-router-hosts-rule-migration \
  --base main \
  --title "feat(router-hosts): migrate alert rules to PrometheusRule (PR4/6)" \
  --body "$(cat <<'EOF'
## Summary

PR4 of the pushover + alertmanager migration. Converts the four router-hosts alert rules from \`GrafanaAlertRuleGroup\` (Grafana-evaluated) to \`PrometheusRule\` (Prometheus-evaluated, routed via Alertmanager → Pushover \`cluster-edge\` app). Same shape as PR3 but \`domain: edge\`.

Spec: \`docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md\`
Plan: \`docs/engineering/plans/2026-05-10-pushover-pr4-router-hosts-rule-migration.md\`

## Changes

- \`argocd/app-configs/router-hosts-alerts/prometheus-rules.yaml\` — new PrometheusRule with all four router-hosts rules. Each carries labels \`{ severity, service: router-hosts, domain: edge }\`. The container-down rule preserves the original \`or vector(0)\` fallback for absent-metric handling. The error-rate rule preserves its divide-by-zero protection idiom. OOM keeps \`for: 0s\` for immediate firing.
- \`argocd/app-configs/router-hosts-alerts/grafana-alerts.yaml\` — deleted. ArgoCD prune removes the cluster-side \`GrafanaFolder\` and \`GrafanaAlertRuleGroup\` on next sync.
- \`argocd/app-configs/router-hosts-alerts/kustomization.yaml\` — swap \`grafana-alerts.yaml\` for \`prometheus-rules.yaml\`.

## Pre-merge prerequisites

None. PR1 (Pushover plumbing), PR2 (Alertmanager + receivers), and PR3 (NATS rule migration) are all merged and verified end-to-end.

## Post-merge verification

- [ ] HCP TFC: no Vault workspace run triggered (no TF changes).
- [ ] ArgoCD \`router-hosts-alerts\` app syncs cleanly; \`Synced\` + \`Healthy\`.
- [ ] \`kubectl -n prometheus get prometheusrule router-hosts\` exists with 4 rules.
- [ ] Prometheus \`/api/v1/rules\` shows the 4 router-hosts rules loaded (RouterHostsContainerDown, RouterHostsHighErrorRate, RouterHostsHookFailures, RouterHostsOOMKilled).
- [ ] \`kubectl -n grafana get grafanaalertrulegroup router-hosts-alerts\` returns \`NotFound\`.
- [ ] \`kubectl -n grafana get grafanafolder router-hosts-alerts\` returns \`NotFound\`.
- [ ] In Grafana UI: Alerting → Folders no longer shows "Router Hosts Alerts".
- [ ] Smoketest: \`amtool alert add alertname=RouterHostsManualSmoketest severity=critical service=router-hosts domain=edge\` from inside the AM pod confirms Pushover delivery via \`cluster-edge\` app at priority 1.
- [ ] No \`prometheus_rule_evaluation_failures_total\` non-zero values for router-hosts rules.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

• Step 6.3: Record the PR URL

The gh pr create command prints the PR URL.

---

Task 7: Post-merge verification (executes after CI + merge)

Files: none (live cluster verification)

• Step 7.1: Wait for ArgoCD sync via Monitor (state-change emit pattern, not silent polling)

Use the established Monitor pattern (per memory: prefer Monitor over silent polling). Watch:

• router-hosts-alerts ArgoCD app: Synced + Healthy
• New PrometheusRule router-hosts exists in prometheus namespace
• Old GrafanaAlertRuleGroup router-hosts-alerts in grafana: NotFound
• Old GrafanaFolder router-hosts-alerts in grafana: NotFound

Sample shape (adapt as needed):

prev=""
while true; do
  rha_sync=$(kubectl --context fzymgc-house get application router-hosts-alerts -n argocd -o jsonpath='{.status.sync.status}/{.status.health.status}' 2>/dev/null || echo "?/?")
  pr_present=$(kubectl --context fzymgc-house -n prometheus get prometheusrule router-hosts -o jsonpath='{.metadata.name}' 2>/dev/null || echo "absent")
  garg_gone=$(kubectl --context fzymgc-house -n grafana get grafanaalertrulegroup router-hosts-alerts -o jsonpath='{.metadata.name}' 2>/dev/null || echo "gone")
  gf_gone=$(kubectl --context fzymgc-house -n grafana get grafanafolder router-hosts-alerts -o jsonpath='{.metadata.name}' 2>/dev/null || echo "gone")
  cur="rha=${rha_sync} pr=${pr_present} garg=${garg_gone} gf=${gf_gone}"
  if [ "$cur" != "$prev" ]; then
    echo "$(date -u +%H:%M:%S) $cur"
    prev=$cur
  fi
  if [ "$rha_sync" = "Synced/Healthy" ] && [ "$pr_present" = "router-hosts" ] && [ "$garg_gone" = "gone" ] && [ "$gf_gone" = "gone" ]; then
    echo "$(date -u +%H:%M:%S) READY"
    break
  fi
  sleep 15
done

• Step 7.2: Confirm Prometheus loaded the rules

kubectl --context fzymgc-house -n prometheus exec prometheus-kps-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  wget -qO- http://localhost:9090/api/v1/rules 2>&1 | jq '.data.groups[] | select(.name == "router-hosts") | {name, rules: [.rules[].name]}'

Expected:

{
  "name": "router-hosts",
  "rules": [
    "RouterHostsContainerDown",
    "RouterHostsHighErrorRate",
    "RouterHostsHookFailures",
    "RouterHostsOOMKilled"
  ]
}

• Step 7.3: Confirm Grafana side is cleaned up

kubectl --context fzymgc-house -n grafana get grafanaalertrulegroup router-hosts-alerts 2>&1
kubectl --context fzymgc-house -n grafana get grafanafolder router-hosts-alerts 2>&1

Expected: both return Error from server (NotFound).

• Step 7.4: Smoketest end-to-end alert delivery

Manual amtool injection (proves the AM → Pushover cluster-edge path):

kubectl --context fzymgc-house -n prometheus exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
  amtool --alertmanager.url=http://localhost:9093 alert add \
    alertname=RouterHostsManualSmoketest \
    severity=critical \
    service=router-hosts \
    domain=edge \
    --annotation=summary="PR4 manual smoketest of router-hosts alert routing"

Wait ~30s. Pushover notification arrives via the cluster-edge app at priority 1 (high; bypasses quiet hours).

• Step 7.5: Confirm no rule-evaluation errors in Prometheus

kubectl --context fzymgc-house -n prometheus exec prometheus-kps-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=prometheus_rule_evaluation_failures_total{rule_group=~".*router-hosts.*"}' 2>&1 | jq '.data.result[] | {metric: .metric.rule_group, value: .value[1]}'

Expected: empty result or all values "0".

• Step 7.6: Silence the smoketest

kubectl --context fzymgc-house -n prometheus exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
  amtool --alertmanager.url=http://localhost:9093 silence add \
    'alertname="RouterHostsManualSmoketest"' --duration=1m --comment="post-PR4 smoketest cleanup"

• Step 7.7: Update workspace state for PR5

After PR4 merges:

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager git fetch
jj --no-pager new main

Workspace ready for PR5 (cut Grafana ntfy receivers).

---

Risks and rollback

Risk	Detection	Rollback
New PrometheusRule fails schema validation	kubectl describe prometheusrule router-hosts shows error; operator logs show rejection	Revert the merge commit; the GrafanaAlertRuleGroup file is restored, Grafana takes back over
PromQL expression behavior differs from Grafana evaluation	prometheus_rule_evaluation_failures_total non-zero, OR alerts fire/don’t-fire differently than expected	Investigate: the four expressions are character-identical to the Grafana ones (just with the threshold inlined). Most likely cause if differences emerge: a Grafana-specific reducer (type: last) was implicit in the original and not preserved. The expressions here use straight comparison instead.
RouterHostsContainerDown fires incorrectly during cluster-app/operator restarts that briefly pause cAdvisor scrape	Pushover notification for container-down arrives during a known restart event	The for: 5m window protects against most short restarts. If problematic, increase to for: 10m
ArgoCD prune doesn’t remove the Grafana CRDs	Grafana UI still shows “Router Hosts Alerts” folder after merge	Manual delete: kubectl -n grafana delete grafanaalertrulegroup router-hosts-alerts; kubectl -n grafana delete grafanafolder router-hosts-alerts
Alertmanager misroutes (sends to wrong receiver)	Smoketest delivery shows wrong app icon (e.g. cluster-data instead of cluster-edge)	Check amtool alert query for the test alert; verify it has domain=edge; check AM config dump for the route. PR2’s routing tree already covers the four domains — should not be a real issue

---

Out of scope for PR4 (handled in subsequent PRs)

• Grafana ntfy contact-point removal (PR5) — at that point all alerts route through Alertmanager; Grafana ntfy receivers become dead code.
• ntfy stack decommission (PR6) — full removal of ntfy ArgoCD app, Vault data, Cloudflare DNS, and docs.
• CI lint asserting all PrometheusRules carry domain + severity — tracked as a follow-up bead.