Pushover + Alertmanager Migration — PR2: Alertmanager Up + Pushover Receivers

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Enable the Alertmanager component of kube-prometheus-stack, declare four Pushover receivers (one per coarse domain) plus a default-route + per-domain routing tree, and prove with amtool that a manually-created alert routes end-to-end to Pushover. No PrometheusRules yet, so no real alerts traverse this path until PR3.

Architecture: Flip alertmanager.enabled: true in the existing kube-prometheus-stack Helm release. Add an Alertmanager config block under the same Helm values (single source of truth — ArgoCD owns the whole Helm release). Add an ExternalSecret in the prometheus namespace mounting the existing Pushover Vault credentials into a Kubernetes Secret consumed by Alertmanager via alertmanagerSpec.secrets[]. Prometheus gets explicit ruleSelector matchLabels so PR3+ rule discovery is visible in git rather than relying on the chart’s implicit defaults.

Tech Stack: Kubernetes (Prometheus Operator-managed Alertmanager CRD via kube-prometheus-stack v84.5.0), External Secrets Operator v1, Pushover REST API (consumed natively by Alertmanager’s pushover_configs receiver). HCP Terraform — no TF changes in PR2 (defensive policy + role from PR1 already exist). ArgoCD app monitoring-prometheus (auto-sync, prune) reconciles both the Helm release and the kustomize directory.

Spec reference: docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md (the “Alertmanager configuration” section for the canonical Helm values shape).

Workspace: .worktrees/feat-pushover-alertmanager (jj workspace pushover-alertmanager, parented to current main).

---

Manual prerequisites

None. PR1 established the Pushover account, the four application tokens in https://pushover.net/apps, and the Vault KV data at secret/fzymgc-house/cluster/pushover/*. PR2 reuses all of that — no new manual steps before kicking off.

If you cannot read vault kv list secret/fzymgc-house/cluster/pushover/app and see cluster-{apps,data,edge,infra}, stop — PR1 did not actually land cleanly and you need to fix that first.

---

Important context: Alertmanager templating works (unlike Grafana)

Grafana 11.x’s contact-point provisioning API parses typed receiver fields (Pushover priority) via strconv.ParseInt at config-load and rejects template strings — that’s why PR1’s Grafana side ended up as 12 contact points with hardcoded priorities (PR #997).

Alertmanager is a different code path. Its pushover_configs.priority is a Go template string evaluated per-notification at send time, fully supported. PR2 uses four receivers (one per domain) with a templated priority that reads .CommonLabels.severity — the spec’s original design intent works here.

Don’t extrapolate the 12-contact-point workaround into PR2.

---

File structure

File	Status	Responsibility
argocd/cluster-app/templates/monitoring-prometheus.yaml	Modify	(1) Add prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues: false + ruleSelector: matchLabels: { release: kps } so PR3+ rule discovery is explicit. (2) Flip alertmanager.enabled: true and add alertmanagerSpec (replicas, Longhorn storage, secrets: [alertmanager-pushover]). (3) Add alertmanager.config with global.resolve_timeout, the four-leaf route tree, explicit empty inhibit_rules: [], and four pushover_configs receivers using *_file credential references.
argocd/app-configs/monitoring-prometheus/alertmanager-pushover-external-secret.yaml	Create	ExternalSecret in the prometheus namespace materialising the Pushover user key + four app tokens into a Kubernetes Secret named alertmanager-pushover. The Prometheus Operator mounts this at /etc/alertmanager/secrets/alertmanager-pushover/ based on the alertmanagerSpec.secrets[] list.
argocd/app-configs/monitoring-prometheus/kustomization.yaml	Modify	Add alertmanager-pushover-external-secret.yaml to the resources: list.
docs/operations/alertmanager.md	Create	Operations runbook: where Alertmanager runs, how to silence via amtool, the Pushover token rotation procedure (manual kubectl rollout restart because Reloader doesn’t fit operator-managed StatefulSets), the priority/severity mapping, and the convention that all PrometheusRules carry domain + severity labels.

No new TF: PR1 already declared vault_policy.pushover and the vault_kubernetes_auth_backend_role.pushover defensive role. No actual Vault auth happens — ESO uses the wildcard external-secrets-operator policy.

---

Task 1: Verify workspace + look up Alertmanager service account name

Files: none (verification only)

• Step 1.1: Confirm you are in the correct workspace

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager git fetch
jj --no-pager workspace list | rg pushover-alertmanager
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

Expected: workspace listed; main is at PR #997’s merge commit (or later if upstream has moved). If the working copy is stale, run jj new main to reset @ to a fresh empty change parented to current main.

• Step 1.2: Confirm Pushover prereqs are still intact

vault kv list secret/fzymgc-house/cluster/pushover/app

Expected: cluster-apps, cluster-data, cluster-edge, cluster-infra. If absent, stop — PR1 has not actually landed and PR2 has nothing to wire to.

• Step 1.3: Look up the Alertmanager service account name (from kps when AM is currently disabled)

The alertmanagerSpec.serviceAccountName defaults to <release-name>-alertmanager for the kps chart with releaseName: kps. The expected SA name is therefore kps-alertmanager. Confirm this matches what the chart produces by rendering the values locally:

helm template kps prometheus-community/kube-prometheus-stack --version 84.5.0 \
  --set alertmanager.enabled=true \
  --set alertmanager.alertmanagerSpec.replicas=1 \
  --namespace prometheus \
  2>/dev/null | rg -A 1 'kind: ServiceAccount' | rg -A 1 'kps-alertmanager|alertmanager$'

Expected: a ServiceAccount named kps-alertmanager (or similar — note exact name).

If the rendered SA name differs from kps-alertmanager, update the verification steps in Task 8 that reference statefulset/alertmanager-kps-kube-prometheus-stack-alertmanager to match. The defensive Vault auth role binding from PR1 is bound to the default SA in the prometheus namespace, not the AM SA — that’s intentional defensive pre-declaration and does not need to change here.

Confirmed during PR2 implementation (2026-05-10): the rendered ServiceAccount name is kps-kube-prometheus-stack-alertmanager, NOT kps-alertmanager. The kps chart uses different naming templates for Prometheus (short {{ .Release.Name }}-prometheus → kps-prometheus) and Alertmanager (chart-fullname-prefixed {{ template "kube-prometheus-stack.fullname" . }}-alertmanager → kps-kube-prometheus-stack-alertmanager). Tasks 5 (operations doc), 7.2 (PR body verification), and 8.x have been updated accordingly. Don’t try to “fix” the naming with cleanPrometheusOperatorObjectNames or fullnameOverride — that reshapes too many other resources for too small a benefit.

• Step 1.4: Confirm the kps version in monitoring-prometheus.yaml is current

helm search repo prometheus-community/kube-prometheus-stack --versions | head -3

Expected: a list with the latest version. The repo currently pins targetRevision: 84.5.0 in argocd/cluster-app/templates/monitoring-prometheus.yaml. If a newer version exists, do not bump it in this PR — version bumps are out of scope and have their own change-management cost. If a known-broken version is currently pinned, stop and escalate.

If helm is not installed locally, skip this step — the chart version is already pinned and the existing repo declaration is the source of truth.

---

Task 2: Add the Alertmanager Pushover ExternalSecret

Files:

• Create: argocd/app-configs/monitoring-prometheus/alertmanager-pushover-external-secret.yaml

• Step 2.1: Write the ExternalSecret manifest

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: alertmanager-pushover
  namespace: prometheus
spec:
  refreshPolicy: Periodic
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: alertmanager-pushover            # name referenced by alertmanagerSpec.secrets[]
    creationPolicy: Owner
    deletionPolicy: Delete
  data:
    - secretKey: user_key
      remoteRef:
        key: fzymgc-house/cluster/pushover
        property: user_key
    - secretKey: cluster-infra
      remoteRef:
        key: fzymgc-house/cluster/pushover/app/cluster-infra
        property: token
    - secretKey: cluster-data
      remoteRef:
        key: fzymgc-house/cluster/pushover/app/cluster-data
        property: token
    - secretKey: cluster-apps
      remoteRef:
        key: fzymgc-house/cluster/pushover/app/cluster-apps
        property: token
    - secretKey: cluster-edge
      remoteRef:
        key: fzymgc-house/cluster/pushover/app/cluster-edge
        property: token

Note: secret keys here use hyphens (cluster-infra) not underscores. The keys map to filenames inside /etc/alertmanager/secrets/alertmanager-pushover/ and the Alertmanager config in Task 3 references them via token_file: /etc/alertmanager/secrets/alertmanager-pushover/cluster-infra. The Grafana-side ExternalSecret used underscores (cluster_infra_token) for k8s Secret friendliness in valuesFrom — different consumer, different key style.

• Step 2.2: Lint YAML

yamllint argocd/app-configs/monitoring-prometheus/alertmanager-pushover-external-secret.yaml

Expected: no output (exit 0).

• Step 2.3: Commit

jj --no-pager commit -m "feat(monitoring-prometheus): add ExternalSecret for alertmanager pushover tokens

Materialises the Pushover user key + four per-app tokens from
secret/fzymgc-house/cluster/pushover/* in Vault into a Kubernetes Secret
named alertmanager-pushover in the prometheus namespace. Consumed by the
Alertmanager pod via alertmanagerSpec.secrets[] (mounted at
/etc/alertmanager/secrets/alertmanager-pushover/) once Alertmanager is
enabled in the next commit."

Verify:

jj --no-pager log -r '@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

---

Task 3: Enable Alertmanager + add config in monitoring-prometheus.yaml

Files:

• Modify: argocd/cluster-app/templates/monitoring-prometheus.yaml

This is the largest change in the PR. Three logical edits to the single file:

1. Inside prometheus.prometheusSpec, add explicit ruleSelector config (4 new lines).
2. Replace alertmanager.enabled: false with the full alertmanager: block (~70 new lines).
3. No deletions; everything else stays.

• Step 3.1: Edit prometheus.prometheusSpec to add explicit ruleSelector

Locate the prometheus.prometheusSpec block. Find the line serviceMonitorSelectorNilUsesHelmValues: false and the two sibling lines below it (podMonitorSelectorNilUsesHelmValues: false, probeSelectorNilUsesHelmValues: false). Add a fourth selector line group immediately after probeSelectorNilUsesHelmValues: false:

              ruleSelectorNilUsesHelmValues: false
              ruleSelector:
                matchLabels:
                  release: kps

The full surrounding block should now look like (the new lines are the last 4):

              serviceMonitorSelectorNilUsesHelmValues: false
              podMonitorSelectorNilUsesHelmValues: false
              probeSelectorNilUsesHelmValues: false
              ruleSelectorNilUsesHelmValues: false
              ruleSelector:
                matchLabels:
                  release: kps
              storageSpec:
                volumeClaimTemplate:
                  ...

• Step 3.2: Replace the disabled alertmanager: block with the full config

Find the line alertmanager: followed by enabled: false. Replace those two lines with the following block. Indentation matters — the alertmanager: key sits at the same indent level as the existing prometheus: and grafana: keys (8 spaces under valuesObject:).

          alertmanager:
            enabled: true
            alertmanagerSpec:
              replicas: 1
              storage:
                volumeClaimTemplate:
                  spec:
                    storageClassName: longhorn
                    accessModes: ["ReadWriteOnce"]
                    resources:
                      requests:
                        storage: 10Gi
              secrets:
                - alertmanager-pushover
            # NOTE: overriding alertmanager.config replaces the chart's default config
            # WHOLESALE (no merge). The chart default ships with a `null` receiver and a
            # set of inhibit_rules. We intentionally drop those because:
            #   - `defaultRules: create: false` means no Watchdog/etc rules exist for the
            #     default inhibit_rules to act on.
            #   - Re-introduce inhibit_rules deliberately if/when we have alerts that
            #     should suppress one another.
            # `global.resolve_timeout` is set explicitly to match Alertmanager's built-in
            # default; making it visible avoids future "where did this come from" hunts.
            config:
              global:
                resolve_timeout: 5m
              route:
                receiver: pushover-infra
                group_by: ['alertname', 'service', 'namespace']
                group_wait: 30s
                group_interval: 5m
                repeat_interval: 4h
                routes:
                  - matchers: ['domain="data"']
                    receiver: pushover-data
                  - matchers: ['domain="apps"']
                    receiver: pushover-apps
                  - matchers: ['domain="edge"']
                    receiver: pushover-edge
              inhibit_rules: []
              receivers:
                - name: pushover-infra
                  pushover_configs:
                    - user_key_file: /etc/alertmanager/secrets/alertmanager-pushover/user_key
                      token_file:    /etc/alertmanager/secrets/alertmanager-pushover/cluster-infra
                      priority: '{{ if eq .CommonLabels.severity "critical" }}1{{ else if eq .CommonLabels.severity "warning" }}0{{ else }}-1{{ end }}'
                      title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
                      message: '{{ .CommonAnnotations.summary }}{{ if .CommonAnnotations.description }}\n{{ .CommonAnnotations.description }}{{ end }}'
                      url: '{{ .ExternalURL }}/#/alerts'
                      url_title: 'Alertmanager'
                - name: pushover-data
                  pushover_configs:
                    - user_key_file: /etc/alertmanager/secrets/alertmanager-pushover/user_key
                      token_file:    /etc/alertmanager/secrets/alertmanager-pushover/cluster-data
                      priority: '{{ if eq .CommonLabels.severity "critical" }}1{{ else if eq .CommonLabels.severity "warning" }}0{{ else }}-1{{ end }}'
                      title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
                      message: '{{ .CommonAnnotations.summary }}{{ if .CommonAnnotations.description }}\n{{ .CommonAnnotations.description }}{{ end }}'
                      url: '{{ .ExternalURL }}/#/alerts'
                      url_title: 'Alertmanager'
                - name: pushover-apps
                  pushover_configs:
                    - user_key_file: /etc/alertmanager/secrets/alertmanager-pushover/user_key
                      token_file:    /etc/alertmanager/secrets/alertmanager-pushover/cluster-apps
                      priority: '{{ if eq .CommonLabels.severity "critical" }}1{{ else if eq .CommonLabels.severity "warning" }}0{{ else }}-1{{ end }}'
                      title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
                      message: '{{ .CommonAnnotations.summary }}{{ if .CommonAnnotations.description }}\n{{ .CommonAnnotations.description }}{{ end }}'
                      url: '{{ .ExternalURL }}/#/alerts'
                      url_title: 'Alertmanager'
                - name: pushover-edge
                  pushover_configs:
                    - user_key_file: /etc/alertmanager/secrets/alertmanager-pushover/user_key
                      token_file:    /etc/alertmanager/secrets/alertmanager-pushover/cluster-edge
                      priority: '{{ if eq .CommonLabels.severity "critical" }}1{{ else if eq .CommonLabels.severity "warning" }}0{{ else }}-1{{ end }}'
                      title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
                      message: '{{ .CommonAnnotations.summary }}{{ if .CommonAnnotations.description }}\n{{ .CommonAnnotations.description }}{{ end }}'
                      url: '{{ .ExternalURL }}/#/alerts'
                      url_title: 'Alertmanager'

The four receivers are intentionally near-identical — only the token_file path differs. Do not refactor with YAML anchors or any DRY transformation; keep all four blocks explicit so a reader can see each receiver’s complete config without indirection. Same convention as PR1’s contact-points file.

• Step 3.3: Lint YAML

yamllint argocd/cluster-app/templates/monitoring-prometheus.yaml

Expected: no output (exit 0). If yamllint flags trailing whitespace or other issues, fix and re-run.

• Step 3.4: Sanity-check the Helm values via helm template

If helm is available locally, render the chart with the new values to confirm the structure parses and produces an Alertmanager CRD:

yq '.spec.sources[0].helm.valuesObject' argocd/cluster-app/templates/monitoring-prometheus.yaml \
  > /tmp/kps-values.yaml
helm template kps prometheus-community/kube-prometheus-stack --version 84.5.0 \
  -f /tmp/kps-values.yaml \
  --namespace prometheus 2>&1 | rg -B 1 -A 5 'kind: Alertmanager$' | head -20

Expected: a single kind: Alertmanager block with metadata.name: kps-kube-prometheus-stack-alertmanager. If helm or yq is unavailable, skip this step — Task 6’s kubectl --dry-run=client will catch CRD-level errors after the kustomize render.

• Step 3.5: Commit

jj --no-pager commit -m "feat(monitoring-prometheus): enable alertmanager + pushover routing

Flips alertmanager.enabled to true in the kube-prometheus-stack Helm
release and adds the full Alertmanager config: single replica with a
10Gi Longhorn-backed PVC for silences, four pushover_configs receivers
(one per coarse domain: infra/data/apps/edge), a four-leaf route tree
matching on the domain label, explicit empty inhibit_rules, and an
explicit global.resolve_timeout. Severity drives Pushover priority via
Go template (critical=1, warning=0, info/other=-1). Credentials are
read from /etc/alertmanager/secrets/alertmanager-pushover/* via the
ExternalSecret added in the previous commit.

Also adds explicit prometheus.ruleSelector matchLabels (release=kps)
and ruleSelectorNilUsesHelmValues=false so PR3+ PrometheusRule
discovery is visible in git rather than relying on the chart's
implicit default. Mirrors the existing pattern for the three other
selectors (serviceMonitor, podMonitor, probe).

No PrometheusRules exist yet, so Alertmanager has no traffic until
PR3 (NATS rule migration). Smoketest path is 'amtool alert add' from a
debug pod."

---

Task 4: Wire the new ExternalSecret into the kustomization

Files:

• Modify: argocd/app-configs/monitoring-prometheus/kustomization.yaml

• Step 4.1: Edit kustomization.yaml

Current contents:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ./certificate.yaml
  - ./ingressroutes.yaml

Replace with:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ./certificate.yaml
  - ./ingressroutes.yaml
  - ./alertmanager-pushover-external-secret.yaml

• Step 4.2: Validate kustomize render

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
kubectl --context fzymgc-house apply --dry-run=client -k argocd/app-configs/monitoring-prometheus 2>&1 | tee /tmp/kustomize-am.txt
rg 'alertmanager-pushover' /tmp/kustomize-am.txt

Expected: externalsecret.external-secrets.io/alertmanager-pushover created (dry run) in the output.

• Step 4.3: Lint YAML

yamllint argocd/app-configs/monitoring-prometheus/kustomization.yaml

Expected: no output (exit 0).

• Step 4.4: Commit

jj --no-pager commit -m "feat(monitoring-prometheus): include alertmanager-pushover ExternalSecret in kustomization

Adds alertmanager-pushover-external-secret.yaml to the
argocd/app-configs/monitoring-prometheus kustomization so ArgoCD
materialises the Kubernetes Secret on next sync."

---

Task 5: Add the Alertmanager operations runbook

Files:

• Create: docs/operations/alertmanager.md

• Step 5.1: Write the operations doc

# Alertmanager Operations

Alertmanager is the routing brain for Prometheus-evaluated alerts in this
cluster. It runs as a single replica inside the `kube-prometheus-stack`
Helm release in the `prometheus` namespace and ships notifications to
Pushover (per-domain apps) via the four `pushover_configs` receivers
declared in `argocd/cluster-app/templates/monitoring-prometheus.yaml`.

See `../engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md`
for the full migration design.

## Where it runs

| Component | Location |
|---|---|
| Pod | `alertmanager-kps-kube-prometheus-stack-alertmanager-0` in namespace `prometheus` (StatefulSet, 1 replica) |
| Service | `kps-kube-prometheus-stack-alertmanager.prometheus.svc.cluster.local:9093` (web UI + API) |
| Storage | `10Gi` PVC on Longhorn (silence persistence across pod restarts) |
| Config secret | `alertmanager-pushover` (mounted at `/etc/alertmanager/secrets/alertmanager-pushover/`) |
| Helm values | `argocd/cluster-app/templates/monitoring-prometheus.yaml` (under `alertmanager.config` and `alertmanager.alertmanagerSpec`) |

## Routing model

The route tree branches on the `domain` alert label:

| `domain` label | Receiver | Pushover app |
|---|---|---|
| (missing or unknown) | `pushover-infra` (default route) | `cluster-infra` |
| `infra` | `pushover-infra` | `cluster-infra` |
| `data` | `pushover-data` | `cluster-data` |
| `apps` | `pushover-apps` | `cluster-apps` |
| `edge` | `pushover-edge` | `cluster-edge` |

Within each receiver, severity drives Pushover priority via Go template:

| `severity` label | Pushover `priority` | Effect |
|---|---|---|
| `critical` | `1` | High; bypasses Pushover quiet hours; alert sound |
| `warning` | `0` | Normal; respects quiet hours |
| (anything else, including `info`) | `-1` | Low; no sound or vibration |

Grouping: alerts are grouped by `(alertname, service, namespace)` with `group_wait: 30s`, `group_interval: 5m`, `repeat_interval: 4h`. Re-paging on a still-firing alert is therefore every 4 hours; tune the `repeat_interval` in the Helm values if real traffic shows it's too aggressive or too quiet.

## Adding a new PrometheusRule

Every `PrometheusRule` MUST carry both:

- `labels.severity`: one of `critical` / `warning` / `info` (other values silently fall through to priority `-1`).
- `labels.domain`: one of `infra` / `data` / `apps` / `edge` (missing or unknown silently routes to `pushover-infra`).

Plus the standard `service` label (used for grouping). Example shape:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-service
  namespace: prometheus
  labels:
    release: kps                  # required for kps Prometheus to discover it
spec:
  groups:
    - name: my-service
      interval: 1m
      rules:
        - alert: MyServiceUnreachable
          expr: up{job="my-service"} == 0
          for: 5m
          labels:
            severity: critical
            service: my-service
            domain: apps
          annotations:
            summary: "my-service is down"
            description: "The my-service job has been unreachable for 5 minutes."
```

The required `release: kps` label on the `PrometheusRule` resource itself comes from the explicit `prometheus.prometheusSpec.ruleSelector` in the Helm values. Without it, the operator will not pick the rule up.

## Silencing alerts

From a debug pod with `amtool` available (or any pod with `alertmanager` image):

```bash
# Add a silence for 1 hour matching alertname=NATSJetStreamStorageHigh
amtool --alertmanager.url=http://kps-kube-prometheus-stack-alertmanager.prometheus.svc.cluster.local:9093 \
  silence add alertname=NATSJetStreamStorageHigh \
  --duration=1h \
  --comment="investigating disk usage on nats-1"

# List active silences
amtool --alertmanager.url=http://kps-kube-prometheus-stack-alertmanager.prometheus.svc.cluster.local:9093 \
  silence query

# Expire a silence
amtool --alertmanager.url=http://kps-kube-prometheus-stack-alertmanager.prometheus.svc.cluster.local:9093 \
  silence expire <silence-id>
```

Silences persist across pod restarts because the Alertmanager StatefulSet has a Longhorn PVC.

## Manually firing an alert (smoketest)

To prove end-to-end routing without writing a `PrometheusRule`:

```bash
amtool --alertmanager.url=http://kps-kube-prometheus-stack-alertmanager.prometheus.svc.cluster.local:9093 \
  alert add alertname=ManualSmoketest \
  severity=warning \
  domain=infra \
  --annotation=summary="Manual smoketest of pushover-infra receiver"
```

The alert appears in `amtool alert query` output. After `group_wait: 30s`, Alertmanager sends a Pushover notification via the `cluster-infra` app at priority 0.

To resolve the alert (silence delivery), wait for `resolve_timeout: 5m` (default) or send an `EndsAt` in the past:

```bash
amtool --alertmanager.url=http://kps-kube-prometheus-stack-alertmanager.prometheus.svc.cluster.local:9093 \
  alert add alertname=ManualSmoketest --end="$(date -u -v-1H +%FT%TZ)"
```

## Pushover token rotation

Pushover tokens are read by ESO every 15 minutes and rendered into the
`alertmanager-pushover` Kubernetes Secret. Alertmanager reads the
`*_file` values at config-load (pod start), **not** on every notification.

The Prometheus Operator does **not** automatically restart Alertmanager
when a Secret listed in `alertmanagerSpec.secrets[]` changes (it watches
the Alertmanager CR and its configSecret, not arbitrary mounted secrets).
Stakater Reloader does not fit cleanly here either — the operator owns
the StatefulSet's metadata, so reloader-style annotations on the pod
template don't propagate to the StatefulSet.

After rotating a Pushover token in Vault:

```bash
# Wait for ESO to refresh (up to 15min) — verify with:
kubectl --context fzymgc-house -n prometheus get secret alertmanager-pushover \
  -o jsonpath='{.metadata.annotations.reconcile\.external-secrets\.io/data-hash}'

# Then restart Alertmanager:
kubectl --context fzymgc-house -n prometheus rollout restart \
  statefulset/alertmanager-kps-kube-prometheus-stack-alertmanager
```

If rotations become frequent enough to justify automation, add a
kustomize patch annotating the Alertmanager StatefulSet directly with
`secret.reloader.stakater.com/reload: alertmanager-pushover`. Until
then, manual restart is the documented procedure.

• Step 5.2: Lint markdown

rumdl docs/operations/alertmanager.md

Expected: Success: No issues found in 1 file. If rumdl reports MD070 about nested fences, bump the outer markdown fence to 4 backticks (````) — same fix pattern as PR1’s plan doc.

• Step 5.3: Commit

jj --no-pager commit -m "docs(alertmanager): add operations runbook

Covers where Alertmanager runs, the routing model (domain label drives
receiver, severity drives Pushover priority), the required labels on
new PrometheusRules, silencing via amtool, the manual smoketest path,
and the manual rollout-restart procedure for Pushover token rotation
(Reloader doesn't fit operator-managed StatefulSets cleanly)."

---

Task 6: Pre-push verification

Files: none (verification only)

• Step 6.1: Confirm all expected commits exist

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager log -r 'main..@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Expected (4 commits, newest first):

<id> docs(alertmanager): add operations runbook
<id> feat(monitoring-prometheus): include alertmanager-pushover ExternalSecret in kustomization
<id> feat(monitoring-prometheus): enable alertmanager + pushover routing
<id> feat(monitoring-prometheus): add ExternalSecret for alertmanager pushover tokens

If any commit is missing, return BLOCKED with details.

• Step 6.2: Confirm no secrets in the diff

jj --no-pager diff -r 'main..@-' | rg -i 'token|user.?key|password|secret' \
  | rg -v 'secretKeyRef|secret/data|secretStoreRef|target.name|key:|user_key|cluster-(infra|data|apps|edge)|token_file|user_key_file|.token|userKey|apiToken|alertmanager-pushover|api-token|secret.reloader'

Expected: no output (after filtering legitimate references). Any literal token value would be a critical bug — stop and remove it.

• Step 6.3: Run all repo linters

lefthook run pre-commit --all-files

Expected: hooks that run on the touched files (lint-yaml, lint-markdown, lint-markdown-docs, fmt-yaml) all pass. Pre-existing failures on files this PR doesn’t touch (older docs archives, OpenClaw GHA workflow, scripts/migrate-secrets-to-vault.py) are NOT blockers — flag them in the report but proceed.

• Step 6.4: Verify the working copy base is current

jj --no-pager git fetch
jj --no-pager log -r 'main' --no-graph -T 'commit_id.short(12) ++ " " ++ description.first_line() ++ "\n"' | head -2

If main has moved during PR2 work, rebase:

jj --no-pager rebase -s 'roots(main..@-)' -d main --skip-emptied

Then re-run Step 6.1 to confirm the four-commit chain is intact.

---

Task 7: Push and open PR

Files: none (VCS + GitHub operations)

• Step 7.1: Create the bookmark and push

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager bookmark create feat/pushover-pr2-alertmanager-receivers -r @-
jj --no-pager git push -b feat/pushover-pr2-alertmanager-receivers

Expected: push succeeds.

• Step 7.2: Open the PR

gh pr create \
  --title "feat(monitoring-prometheus): enable alertmanager + pushover receivers (PR2/6)" \
  --body "$(cat <<'EOF'
## Summary

PR2 of the pushover + alertmanager migration. Enables the Alertmanager component of `kube-prometheus-stack` (currently `enabled: false`), declares four Pushover receivers + a four-leaf routing tree, and adds the operations runbook. No `PrometheusRule`s yet; Alertmanager will sit idle until PR3 migrates the NATS rules.

Spec: `docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md`
Plan: `docs/engineering/plans/2026-05-10-pushover-pr2-alertmanager-receivers.md`

## Changes

- `argocd/cluster-app/templates/monitoring-prometheus.yaml` — flip `alertmanager.enabled: true`; add `alertmanagerSpec` (replicas, Longhorn storage, secrets[]); add `alertmanager.config` (global, route tree with default + 3 domain matchers, explicit empty inhibit_rules, 4 pushover_configs receivers); add explicit `prometheus.ruleSelector` matchLabels for PR3+ rule discovery visibility.
- `argocd/app-configs/monitoring-prometheus/alertmanager-pushover-external-secret.yaml` — new ExternalSecret in `prometheus` namespace materialising the existing Pushover Vault credentials into a Kubernetes Secret named `alertmanager-pushover`.
- `argocd/app-configs/monitoring-prometheus/kustomization.yaml` — wire the new ExternalSecret in.
- `docs/operations/alertmanager.md` — new runbook (where AM runs, routing model, required labels on new PrometheusRules, silencing via amtool, manual smoketest path, Pushover token rotation procedure).

## Pre-merge prerequisites

None. PR1 already established the Pushover account, four application tokens, and the Vault KV data at `secret/fzymgc-house/cluster/pushover/*`. The defensive Vault policy + auth role landed in PR1's TF.

## Post-merge verification

- [ ] HCP TFC: no vault workspace run triggered (PR2 has no TF changes).
- [ ] ArgoCD `monitoring-prometheus` app syncs cleanly; `Synced` + `Healthy`.
- [ ] `kubectl -n prometheus get statefulset alertmanager-kps-kube-prometheus-stack-alertmanager` exists with `READY=1/1`.
- [ ] `kubectl -n prometheus get pod alertmanager-kps-kube-prometheus-stack-alertmanager-0` shows `Running`; `kubectl -n prometheus logs alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager` shows `"msg":"Loading configuration file"` followed by no error and `"msg":"Completed loading of configuration file"`.
- [ ] `kubectl -n prometheus get externalsecret alertmanager-pushover` reports `Ready=True`.
- [ ] `kubectl -n prometheus get secret alertmanager-pushover -o jsonpath='{.data}' | jq 'keys'` returns `["cluster-apps","cluster-data","cluster-edge","cluster-infra","user_key"]`.
- [ ] From a debug pod or via port-forward, `amtool --alertmanager.url=http://kps-kube-prometheus-stack-alertmanager.prometheus.svc.cluster.local:9093 config show` returns the expected four receivers and four-leaf route tree.
- [ ] `amtool alert add alertname=PR2Smoketest severity=warning domain=data --annotation=summary="PR2 smoketest"` followed by `amtool alert query` shows the alert grouped to `pushover-data`. Within ~30s a Pushover notification arrives on the device via the `cluster-data` app at priority 0.
- [ ] Repeat the smoketest with `domain=edge severity=critical` to confirm the `pushover-edge` route + priority 1 mapping.
- [ ] `kubectl -n prometheus get prometheus kps-prometheus -o jsonpath='{.spec.ruleSelector.matchLabels}{"\n"}'` returns `{"release":"kps"}`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

• Step 7.3: Record the PR URL

The gh pr create command prints the PR URL. Save it for use during post-merge verification.

---

Task 8: Post-merge verification (executes after CI + merge)

Files: none (live cluster verification)

This task runs after the PR merges and ArgoCD applies the changes.

• Step 8.1: Wait for ArgoCD sync

until [ "$(kubectl --context fzymgc-house get application monitoring-prometheus -n argocd -o jsonpath='{.status.sync.status}')" = "Synced" ] && [ "$(kubectl --context fzymgc-house get application monitoring-prometheus -n argocd -o jsonpath='{.status.health.status}')" = "Healthy" ]; do sleep 5; done
echo "monitoring-prometheus app: Synced + Healthy"

The kps Helm release reconciliation can take longer than typical ArgoCD apps because the chart contains many CRDs and the upgrade-job runs first.

• Step 8.2: Verify Alertmanager pod up

kubectl --context fzymgc-house -n prometheus get statefulset alertmanager-kps-kube-prometheus-stack-alertmanager -o jsonpath='{.status.readyReplicas}/{.spec.replicas}{"\n"}'

Expected: 1/1. If it shows 0/1, check the pod’s events:

kubectl --context fzymgc-house -n prometheus describe pod alertmanager-kps-kube-prometheus-stack-alertmanager-0 | rg -A 5 'Events:'

Common first-time failure: PVC binding pending if Longhorn is slow on a new claim. Resolve by waiting; kubectl get pvc -n prometheus | rg alertmanager shows binding status.

• Step 8.3: Verify Alertmanager config loaded without errors

kubectl --context fzymgc-house -n prometheus logs alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager --tail=200 | rg -i 'loading|completed|error|invalid'

Expected: "Loading configuration file" followed by "Completed loading of configuration file" and no error or invalid lines. Any error here means the config block in monitoring-prometheus.yaml is malformed — read the error, fix in a follow-up commit, push.

• Step 8.4: Verify ExternalSecret + Kubernetes Secret

kubectl --context fzymgc-house -n prometheus get externalsecret alertmanager-pushover \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'

Expected: True.

kubectl --context fzymgc-house -n prometheus get secret alertmanager-pushover \
  -o jsonpath='{.data}' | jq 'keys'

Expected:

[
  "cluster-apps",
  "cluster-data",
  "cluster-edge",
  "cluster-infra",
  "user_key"
]

• Step 8.5: Verify Alertmanager sees the rendered config and routing tree

Port-forward to the Alertmanager service, then use amtool from your local machine (assumes amtool is installed; if not, the same commands work from kubectl exec into the AM pod):

kubectl --context fzymgc-house -n prometheus port-forward svc/kps-kube-prometheus-stack-alertmanager 9093:9093 &
PF_PID=$!
sleep 2

amtool --alertmanager.url=http://localhost:9093 config show 2>&1 | head -40

Expected: the config dump shows the four pushover-{infra,data,apps,edge} receivers and the four-leaf routing tree. If amtool is not installed, an equivalent check is curl http://localhost:9093/api/v2/status | jq '.config'.

Kill the port-forward before continuing:

kill $PF_PID 2>/dev/null

• Step 8.6: Verify the explicit ruleSelector is applied to Prometheus

kubectl --context fzymgc-house -n prometheus get prometheus kps-prometheus \
  -o jsonpath='{.spec.ruleSelector.matchLabels}{"\n"}'

Expected: {"release":"kps"}.

If this shows empty {} or a missing field, the explicit ruleSelector from Step 3.1 didn’t make it into the rendered Prometheus CR. PR3 will fail to discover its PrometheusRule if this is broken — fix in a follow-up before PR3 starts.

• Step 8.7: End-to-end smoketest — fire a fake alert and confirm Pushover delivery

This is the load-bearing post-merge check. Fire a manual alert via amtool and confirm a Pushover notification arrives on your phone.

kubectl --context fzymgc-house -n prometheus port-forward svc/kps-kube-prometheus-stack-alertmanager 9093:9093 &
PF_PID=$!
sleep 2

amtool --alertmanager.url=http://localhost:9093 alert add \
  alertname=PR2Smoketest \
  severity=warning \
  domain=data \
  --annotation=summary="PR2 smoketest of pushover-data receiver"

# Wait at least 30s (group_wait) for the notification to dispatch.
sleep 35

amtool --alertmanager.url=http://localhost:9093 alert query

Expected:

1. The alert appears in amtool alert query output.
2. Within ~30s, a Pushover notification arrives on your device via the cluster-data app.
3. Notification has title PR2Smoketest (firing) and priority 0 (normal sound).

Now repeat for domain=edge severity=critical:

amtool --alertmanager.url=http://localhost:9093 alert add \
  alertname=PR2SmoketestCritical \
  severity=critical \
  domain=edge \
  --annotation=summary="PR2 smoketest of pushover-edge receiver, priority 1"

sleep 35
amtool --alertmanager.url=http://localhost:9093 alert query

Expected: Pushover notification arrives via cluster-edge app at priority 1 (high priority sound, bypasses quiet hours).

If either smoketest fails to deliver:

• Check kubectl -n prometheus logs alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager --tail=100 for delivery errors.
• Common failures: token mismatch (re-verify the secret keys map to the right tokens), routing miss (confirm domain label matched a route), Pushover API rate limit (check the response code in AM logs).

Kill the port-forward:

kill $PF_PID 2>/dev/null

• Step 8.8: Confirm no regression in existing ntfy paths (PR1’s plumbing)

kubectl --context fzymgc-house -n grafana get grafanacontactpoint -l '!nope'

Expected: the 12 pushover-*-* contact points + 3 ntfy-* contact points all reconciled (READY=True). PR2 doesn’t touch Grafana, but verify the kps Helm-release reconcile didn’t ripple-restart anything that affected the operator.

• Step 8.9: Clean up the smoketest alerts

The smoketest alerts auto-expire after resolve_timeout: 5m of no further “alert add” activity, but you can resolve them explicitly:

kubectl --context fzymgc-house -n prometheus port-forward svc/kps-kube-prometheus-stack-alertmanager 9093:9093 &
PF_PID=$!
sleep 2

# Resolve by ending in the past
amtool --alertmanager.url=http://localhost:9093 alert add alertname=PR2Smoketest --end="$(date -u -v-1H +%FT%TZ)"
amtool --alertmanager.url=http://localhost:9093 alert add alertname=PR2SmoketestCritical --end="$(date -u -v-1H +%FT%TZ)"

kill $PF_PID 2>/dev/null

A “resolved” Pushover notification will arrive on your device for each (priority 0/-1 depending on severity).

• Step 8.10: Workspace state

The workspace pushover-alertmanager stays at .worktrees/feat-pushover-alertmanager for PR3 (NATS rule migration). After PR2 merges:

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager git fetch
jj --no-pager new main      # parent fresh empty @ to current main

This puts the workspace ready for PR3 work.

---

Risks and rollback

Risk	Detection	Rollback
Alertmanager StatefulSet stuck 0/1 due to PVC binding	kubectl describe pod alertmanager-kps-kube-prometheus-stack-alertmanager-0 shows FailedScheduling or Unschedulable	Wait for Longhorn to provision (usually <1min); if persistent, check Longhorn events for replica scheduling issues
Helm values block fails to render (indentation or unknown key)	ArgoCD app sync fails with helm template error in the app’s Conditions	Revert the monitoring-prometheus.yaml commit (git revert <merge-commit>); push a fix
Alertmanager loads config but rejects routing tree	kubectl logs alertmanager-kps-kube-prometheus-stack-alertmanager-0 -c alertmanager shows error parsing config	Same as above; iteration on the alertmanager.config block
ExternalSecret fails to populate (Vault path missing or key mismatch)	kubectl describe externalsecret alertmanager-pushover -n prometheus shows “key not found”	The Vault paths exist (verified in Task 1); recheck the data[].remoteRef.key and .property values match secret/fzymgc-house/cluster/pushover/* and user_key / token exactly
Pushover API rejects notifications (token wrong)	AM logs show "msg":"Notify failed" with HTTP 4xx for the Pushover URL	Confirm the secret key cluster-<domain> actually contains the expected token (kubectl get secret alertmanager-pushover -o jsonpath='{.data.cluster-data}' | base64 -d); re-write the Vault path if wrong
Per-domain receiver routes alerts to the wrong Pushover app	Smoketest delivery shows wrong app icon	Likely a mis-mapping in alertmanager.config.receivers[].pushover_configs[].token_file. Verify each receiver’s token_file matches its name (pushover-data → cluster-data, etc.)
ruleSelector change unexpectedly hides existing rules	kubectl -n prometheus get prometheusrule returns fewer rules than before	None exist today (defaultRules: create: false is set). If a future-added third-party chart shipped a PrometheusRule without the release: kps label, it’d silently disappear. Fix by labeling the rule or adding additional matchExpressions to the selector.

---

Out of scope for PR2 (handled in subsequent PRs)

• NATS PrometheusRule migration (PR3) — converts argocd/app-configs/nats/grafana-alerts.yaml to a PrometheusRule with domain: data labels. First real traffic for Alertmanager.
• router-hosts PrometheusRule migration (PR4) — same shape, domain: edge.
• Grafana ntfy contact-point removal (PR5) — strips ntfy-tokens-external-secret.yaml and ntfy-contact-points.yaml. By then Alertmanager owns all alert routing; the Grafana ntfy receivers are unused.
• ntfy stack decommission (PR6) — removes the entire ntfy ArgoCD app, Vault data, Cloudflare DNS, and docs.
• Alertmanager HA — single replica is acceptable; HA is a follow-up if it becomes necessary.
• Reloader integration on the AM StatefulSet — manual restart on rotation is the documented procedure (see operations doc); revisit if rotations become frequent.
• Pushover delivery-rate alert / quota dashboard — tracked as a follow-up bead, not part of this initiative.