Pushover + Alertmanager Migration

Status: Design approved, ready for implementation planning Date: 2026-05-10 Owner: Sean Workspace: .worktrees/feat-pushover-alertmanager

Summary

Replace the current notification stack (Grafana-evaluated alert rules → Grafana contact points → self-hosted ntfy) with a Prometheus-native stack (PrometheusRule CRDs → Alertmanager → Pushover). After migration, Grafana keeps dashboards and ALERTS{} viewing only. The ntfy server, its Vault secrets, its Cloudflare DNS entry, and its docs are removed.

Goals

Single source of truth for alert evaluation: Prometheus.
Single routing brain: Alertmanager.
Mobile-first delivery via Pushover with per-domain mute granularity.
Clean removal of ntfy from the cluster, Vault, DNS, and docs.

Non-goals

Alertmanager HA (single replica is acceptable; HA is a follow-up if it becomes necessary).
Replacing kube-prometheus-stack’s defaultRules: create: false with a curated rule set (separate initiative).
Adding new alerts or changing alert thresholds beyond what’s required to port the existing rules.

Decisions (locked in)

Decision	Choice
Pushover account model	One user key, four application tokens (one per coarse domain)
Domain buckets	`infra` / `data` / `apps` / `edge`
Severity → Pushover priority	`info=-1`, `warning=0`, `critical=1` (no emergency-with-ack by default)
Alertmanager deployment	Enabled inside the existing `kube-prometheus-stack` Helm release; single replica; Longhorn-backed PVC for silences
Rule evaluation migration	Migrate everything (NATS + router-hosts) to `PrometheusRule`
ntfy disposition	Fully decommissioned (manifests, secrets, policy, role, DNS, docs)
Phasing	Approach A — staged single-receiver, six PRs

Architecture

End state

┌────────────┐    ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
│ Prometheus │───▶│ Alertmanager│───▶│ Pushover API │───▶│ iOS / desktop│
│ rule eval  │    │ route+group │    │ (per-domain  │    │ Pushover app │
│ via PRule  │    │ +inhibit    │    │  app token)  │    │              │
└────────────┘    └─────────────┘    └──────────────┘    └──────────────┘
       ▲                  ▲
       │                  │
   PrometheusRule    Alertmanager
   CRDs (operator)   config (kps Helm values)

Components retained from today: Prometheus operator, kube-prometheus-stack Helm release, Grafana for dashboards, External Secrets Operator, Vault. Components added: Alertmanager (already shipped by kps, currently disabled), PrometheusRule CRDs (already supported by the Prometheus Operator). Components removed: GrafanaContactPoint resources for ntfy, GrafanaAlertRuleGroup resources, the entire ntfy stack.

Why `domain` is a routing label

Every PrometheusRule will set labels: { domain: infra|data|apps|edge, severity: info|warning|critical, service: <name> }. The Alertmanager route tree branches on domain (four leaves) regardless of how many services exist. Adding a new service is a label change in the rule; no Alertmanager change is needed.

Pushover account setup (manual, one-time)

Create four applications in https://pushover.net/apps:

App name	Domain	Suggested icon
`cluster-infra`	k8s core, longhorn, cnpg, vault, argocd, ingress, cert-manager	gear
`cluster-data`	nats, postgres, dolt, redis, jetstream	database
`cluster-apps`	tandoor, karakeep, grafana stack itself, user-facing	grid
`cluster-edge`	router-hosts, cloudflared, external probes	globe

Each application yields one API token. The single Pushover user key is shared across all four. Per-app sound/icon preferences are configured on the device (out of scope for git).

Vault layout

New KV v2 paths under the existing secret/ mount:

fzymgc-house/cluster/pushover/
  user_key                         # property: user_key
  app/cluster-infra/token          # property: token
  app/cluster-data/token
  app/cluster-apps/token
  app/cluster-edge/token

New Terraform files (mirror the karakeep / ntfy pattern):

tf/vault/policy-pushover.tf — read-only policy on secret/data/fzymgc-house/cluster/pushover/*.
tf/vault/k8s-pushover.tf — Kubernetes auth role binding the policy to the Alertmanager service account in the prometheus namespace.

Note on actual access path: following the existing ntfy / karakeep pattern, this policy + role pair is defensive pre-declaration. Pushover secrets are read via the ESO ClusterSecretStore (which uses the external-secrets-operator policy with a wildcard secret/data/* grant), not via per-app Kubernetes auth. The dedicated policy exists in case a future bootstrap mechanism authenticates to Vault directly.

Alertmanager configuration

In argocd/cluster-app/templates/monitoring-prometheus.yaml:

prometheus:
  prometheusSpec:
    # Existing fields unchanged. Add the rule selector pair below so the discovery
    # behavior is explicit in git rather than relying on the chart's
    # `ruleSelectorNilUsesHelmValues: true` default. Mirrors the explicit pattern
    # already used for serviceMonitor / podMonitor / probe selectors.
    ruleSelectorNilUsesHelmValues: false
    ruleSelector:
      matchLabels:
        release: kps

alertmanager:
  enabled: true
  alertmanagerSpec:
    replicas: 1
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: longhorn
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 10Gi
    secrets:
      - alertmanager-pushover         # ESO-rendered, mounted at /etc/alertmanager/secrets/alertmanager-pushover/
  # NOTE: overriding alertmanager.config replaces the chart's default config
  # WHOLESALE (no merge). The chart default ships with a `null` receiver and a
  # set of inhibit_rules. We intentionally drop those because:
  #   - `defaultRules: create: false` means no Watchdog/etc rules exist for the
  #     default inhibit_rules to act on.
  #   - Re-introduce inhibit_rules deliberately if/when we have alerts that
  #     should suppress one another.
  # `global.resolve_timeout` is set explicitly to match Alertmanager's built-in
  # default; making it visible avoids future "where did this come from" hunts.
  config:
    global:
      resolve_timeout: 5m
    route:
      receiver: pushover-infra        # default route + catches alerts missing domain label
      group_by: ['alertname', 'service', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
        - matchers: ['domain="data"']
          receiver: pushover-data
        - matchers: ['domain="apps"']
          receiver: pushover-apps
        - matchers: ['domain="edge"']
          receiver: pushover-edge
    inhibit_rules: []                 # explicit empty; see NOTE above
    receivers:
      - name: pushover-infra
        pushover_configs:
          - user_key_file: /etc/alertmanager/secrets/alertmanager-pushover/user_key
            token_file:    /etc/alertmanager/secrets/alertmanager-pushover/cluster-infra
            priority: '{{ if eq .CommonLabels.severity "critical" }}1{{ else if eq .CommonLabels.severity "warning" }}0{{ else }}-1{{ end }}'
            title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
            message: '{{ .CommonAnnotations.summary }}{{ if .CommonAnnotations.description }}\n{{ .CommonAnnotations.description }}{{ end }}'
            url: '{{ .ExternalURL }}/#/alerts'
            url_title: 'Alertmanager'
      # pushover-data, pushover-apps, pushover-edge follow the identical shape as
      # pushover-infra above; the only field that differs is token_file, which points to
      # cluster-data, cluster-apps, or cluster-edge respectively under
      # /etc/alertmanager/secrets/alertmanager-pushover/.

Credentials use *_file suffixes so the rendered Helm values can be committed verbatim — the actual secret material lives in a Kubernetes Secret mounted by the Prometheus Operator under /etc/alertmanager/secrets/<secret-name>/<key>.

ExternalSecret for Alertmanager

argocd/app-configs/monitoring-prometheus/alertmanager-pushover-external-secret.yaml:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: alertmanager-pushover
  namespace: prometheus
spec:
  refreshPolicy: Periodic                  # match the existing ntfy/karakeep ExternalSecret pattern
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: alertmanager-pushover            # name referenced by alertmanagerSpec.secrets[]
    creationPolicy: Owner
    deletionPolicy: Delete
  data:
    - secretKey: user_key
      remoteRef: { key: fzymgc-house/cluster/pushover, property: user_key }
    - secretKey: cluster-infra
      remoteRef: { key: fzymgc-house/cluster/pushover/app/cluster-infra, property: token }
    - secretKey: cluster-data
      remoteRef: { key: fzymgc-house/cluster/pushover/app/cluster-data, property: token }
    - secretKey: cluster-apps
      remoteRef: { key: fzymgc-house/cluster/pushover/app/cluster-apps, property: token }
    - secretKey: cluster-edge
      remoteRef: { key: fzymgc-house/cluster/pushover/app/cluster-edge, property: token }

PrometheusRule conversion pattern

A GrafanaAlertRuleGroup rule with separate data blocks (PromQL + __expr__ reducer) collapses into a single PromQL expression once the threshold is hoisted into the query.

Example — the existing nats-jetstream-storage-high rule becomes:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: nats
  namespace: prometheus
  labels:
    release: kps                     # required for kps to discover the rule
spec:
  groups:
    - name: nats
      interval: 1m
      rules:
        - alert: NATSJetStreamStorageHigh
          expr: |
            (sum(nats_varz_jetstream_stats_storage)
             / sum(nats_varz_jetstream_config_max_storage)) > 0.8
          for: 5m
          labels:
            severity: critical
            service: nats
            domain: data
          annotations:
            summary: "NATS JetStream storage usage is high"
            description: "JetStream storage usage is above 80%."

The router-hosts rules follow the same pattern with { severity: critical, service: router-hosts, domain: edge }.

Cross-namespace placement (matches existing repo pattern)

Both new PrometheusRule files carry metadata.namespace: prometheus so the Prometheus Operator (which only watches the prometheus namespace by default kps configuration) discovers them. The files themselves live with their owning ArgoCD app (argocd/app-configs/nats/, argocd/app-configs/router-hosts-alerts/) even though those apps’ default destination is nats and grafana respectively. This mirrors the existing pattern: today’s argocd/app-configs/nats/grafana-alerts.yaml already specifies metadata.namespace: grafana while the nats ArgoCD app deploys to namespace: nats. Per-app co-location of alert rules is the established convention; the spec preserves it.

Phased PR sequence

PR	Scope	Files touched
PR1 — Pushover credentials & Grafana receiver	Vault paths + TF policy/role; ExternalSecret in `grafana/`; one new `GrafanaContactPoint` per domain (4); add a `GrafanaNotificationPolicy` that routes by a new `domain` label to Pushover, default-route still ntfy. This is a net-new resource — there is no existing `notification-policy.yaml` in `grafana/` today. The CRD will replace Grafana’s built-in default policy. PR1 verification must include reading the policy back from the Grafana UI to confirm the operator-rendered tree matches the CRD.	new: `tf/vault/policy-pushover.tf`, `tf/vault/k8s-pushover.tf`, `argocd/app-configs/grafana/pushover-tokens-external-secret.yaml`, `argocd/app-configs/grafana/pushover-contact-points.yaml`, `argocd/app-configs/grafana/notification-policy.yaml`
PR2 — Alertmanager up, no traffic	Flip `alertmanager.enabled: true`; declare Alertmanager config + receivers + routing tree; ExternalSecret for Pushover tokens. No `PrometheusRule`s yet.	`argocd/cluster-app/templates/monitoring-prometheus.yaml`; new `argocd/app-configs/monitoring-prometheus/alertmanager-pushover-external-secret.yaml`
PR3 — Migrate NATS rules	Write `argocd/app-configs/nats/prometheus-rules.yaml`. Delete `argocd/app-configs/nats/grafana-alerts.yaml`. NATS alerts now flow Prometheus → Alertmanager → `pushover-data`.	`argocd/app-configs/nats/`
PR4 — Migrate router-hosts rules	Same pattern for router-hosts; `domain: edge`, `service: router-hosts`. Delete the Grafana counterpart.	`argocd/app-configs/router-hosts-alerts/`
PR5 — Cut Grafana ntfy receivers	Delete `grafana/ntfy-contact-points.yaml`, `grafana/ntfy-tokens-external-secret.yaml`, and any notification-policy entries referencing ntfy. Keep PR1’s Pushover Grafana contact points as an escape hatch for future ad-hoc Grafana-evaluated alerts.	`argocd/app-configs/grafana/`
PR6 — Decommission ntfy	See ordered table below.	many files; mostly deletions

PR6 — ordered decommission steps

Step	Action	Command/file
6.1	Delete ArgoCD app & manifests	`git rm -r argocd/app-configs/ntfy/`, `git rm argocd/cluster-app/templates/ntfy.yaml` — ArgoCD’s `prune: true` removes the in-cluster Deployment, Service, IngressRoute, Certificate, PVC, and the `ntfy` namespace
6.2	Confirm no ExternalSecret still references ntfy paths	`kubectl -n grafana get externalsecret ntfy-grafana-tokens` → NotFound (gone in PR5); `kubectl -n ntfy get externalsecret` → namespace gone
6.3	Destroy Vault KV data (manual, irreversible)	`vault kv metadata delete secret/fzymgc-house/cluster/ntfy` and `vault kv metadata delete secret/fzymgc-house/cluster/ntfy-grafana`
6.4	Verify Vault is clean	`vault kv list secret/fzymgc-house/cluster/` → no `ntfy` or `ntfy-grafana` entries
6.5	Remove Terraform policy + auth role	`git rm tf/vault/policy-ntfy.tf tf/vault/k8s-ntfy.tf`; HCP TFC plan should show 1 `vault_policy` and 1 `vault_kubernetes_auth_backend_role` destroyed
6.6	Remove Cloudflare DNS / variable entry	edit `tf/cloudflare/variables.tf` to drop the ntfy hostname; verify the DNS record is removed by the next TF apply
6.7	Docs cleanup	`docs/reference/services.md` (remove ntfy row), `docs/reference/secrets.md` (remove `fzymgc-house/cluster/ntfy*` rows), `rg -i ntfy docs/` for stragglers
6.8	Final sweep	`rg -i 'ntfy\.fzymgc' .` returns zero matches; `rg -l ntfy argocd/ tf/` returns zero matches

Ordering rationale: the protective guard is 6.1 + 6.2 (consumer gone), not 6.5. ESO reads Vault via the wildcard external-secrets-operator policy with secret/data/* access, not via the per-app ntfy policy — so removing tf/vault/policy-ntfy.tf in 6.5 does not stop ESO from attempting a read. The actual sequencing constraint: 6.3 (KV metadata delete) must come after 6.1 has propagated and 6.2 confirms no ExternalSecret will attempt to refresh against the dead path. Otherwise expect noisy secret not found events from ESO for up to 15 minutes. Step 6.5 (TF policy removal) can run in any order relative to 6.3/6.4; it’s defensive cleanup of an unused policy.

Verification per PR

The non-negotiable verification step every PR is fire a real alert and confirm it lands on the phone. Notification-path failures are silent otherwise — ArgoCD green is not evidence of delivery.

PR	Verification
PR1	(a) `vault kv get secret/fzymgc-house/cluster/pushover/app/cluster-infra` returns the token. (b) Create a throwaway `GrafanaAlertRuleGroup` with `labels: { domain: infra, severity: warning }` that fires immediately; confirm Pushover notification on phone with correct sound/priority. (c) Delete the throwaway. (d) Repeat for one other domain to verify routing.
PR2	(a) `kubectl -n prometheus get pods` shows `alertmanager-kps-alertmanager-0` Ready. (b) From a debug pod: `amtool --alertmanager.url=http://kps-alertmanager.prometheus.svc.cluster.local:9093 alert add alertname=ManualTest severity=warning domain=infra` then check `amtool alert query` shows it grouped to `pushover-infra` and Pushover delivered it. (c) `amtool config show` matches the expected routing tree. (d) `kubectl -n prometheus get prometheus kps-prometheus -o jsonpath='{.spec.ruleSelector}'` returns the explicit `release: kps` matchLabel.
PR3	(a) `kubectl -n prometheus get prometheusrule nats` exists. (b) Force the rule by temporarily lowering the threshold or use `promtool test rules`. Confirm Pushover delivery via `cluster-data`. (c) Confirm the old GrafanaAlertRuleGroup is gone (`kubectl -n grafana get grafanaalertrulegroup nats-alerts` → NotFound). (d) ArgoCD app green.
PR4	Same shape as PR3, for router-hosts.
PR5	(a) `kubectl -n grafana get grafanacontactpoint` shows no `ntfy-*`. (b) Grafana UI: Alerting → Contact points page does not list ntfy. (c) Trigger any remaining ad-hoc Grafana-evaluated alert; confirm Grafana logs do not error about a missing ntfy contact point.
PR6	(a) `kubectl get ns ntfy` → NotFound. (b) `vault kv list secret/fzymgc-house/cluster/` returns no `ntfy*` entries. (c) DNS for `ntfy.fzymgc.net` does not resolve. (d) `rg -i 'ntfy' docs/ argocd/ tf/` returns no matches.

Failure modes & risks

Pushover monthly limit (10,000 messages, paid tier). A flapping alert can torch the quota. Mitigation: repeat_interval: 4h, group_wait: 30s, and for: clauses ≥ 5m on every rule. Tracking a delivery-rate alert is a follow-up bead, not in this spec.
Single-replica Alertmanager. Pod restarts mid-incident lose in-flight notifications and silences for the gap. Mitigation: PVC on Longhorn so silences survive restart. HA is a follow-up if it becomes necessary.
Stale Pushover token after rotation. ExternalSecret refreshes every 15m, but Alertmanager reads *_file values at config-load. The Prometheus Operator does not automatically restart Alertmanager when a Secret listed in alertmanagerSpec.secrets[] changes — it only watches the Alertmanager CR and its configSecret. Stakater Reloader (already deployed in this cluster) cannot easily annotate the operator-managed StatefulSet (annotations applied via alertmanagerSpec.podMetadata land on the Pod, not the StatefulSet Reloader watches). The documented procedure in docs/operations/alertmanager.md is therefore: after rotating a Pushover token in Vault, run kubectl -n prometheus rollout restart statefulset/alertmanager-kps-alertmanager. Revisit Reloader integration if rotations become frequent enough to justify a kustomize patch on the operator-managed StatefulSet.
Watchdog / community-chart null receiver assumption. The chart default null receiver is intentionally dropped (see the alertmanager config NOTE). This is safe today because defaultRules: create: false means kps does not create the Watchdog PrometheusRule. If a future third-party Helm chart ships a rule named Watchdog (or any other always-firing heartbeat), it will route to pushover-infra and page every repeat_interval. Mitigation when adding new charts: audit their bundled PrometheusRule resources for always-firing alerts and either silence them or restore a null receiver + matching route.
Domain label drift. A new PrometheusRule that forgets domain: falls into the default route (pushover-infra) — acceptable default, never drops the alert. Add a CI check (follow-up bead) asserting all rules carry domain and severity.
Pushover priority 1 and quiet hours. Priority 1 bypasses the user’s Pushover quiet hours and uses the alert sound. If criticals should honor quiet hours, switch criticals to priority 0 in PR1 before settling.
GrafanaAlertRuleGroup deletion ordering. Delete the Grafana rule in the same PR that introduces its PrometheusRule replacement. If they overlap during deploy, expect duplicate notifications — annoying, not dangerous.
iOS notification UX surprise. Pushover’s per-app sound/icon settings live on the device. Configure them once in PR1, after notifications start arriving.
Vault soft-delete vs metadata delete. vault kv delete is reversible for ~30 days (or whatever metadata retention is configured); vault kv metadata delete is not. PR6 step 6.3 uses metadata delete deliberately — leaked credentials must not sit in version history after decommissioning.

Documentation updates

Touched across PRs:

docs/reference/services.md — remove ntfy row (PR6). No Pushover row needed (Pushover is SaaS, not a service we operate).
docs/reference/secrets.md — add fzymgc-house/cluster/pushover/* paths (PR1); remove ntfy* paths (PR6).
docs/operations/alertmanager.md (new, PR2) — covers (a) where Alertmanager runs, (b) silencing via amtool, (c) Pushover token rotation procedure, (d) priority/severity mapping reference, (e) how to add a new PrometheusRule (the domain + severity label requirement). Updated in each rule-migration PR.

Out of scope (track as follow-up beads)

Pushover delivery-rate alert / message-quota dashboard panel.
Alertmanager HA (2 replicas with gossip).
CI lint asserting all PrometheusRules carry domain + severity labels.
Replacing kube-prometheus-stack defaultRules: create: false with a curated set.
Future Grafana-only synthetic alerts (none today).