Pushover + Alertmanager Migration — PR1: Grafana Pushover Receiver

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add Pushover as a parallel notification path in Grafana — four GrafanaContactPoint resources (one per coarse domain) routed by a new domain alert label, with credentials sourced from Vault via ExternalSecrets. ntfy stays as the default route until subsequent PRs migrate rule evaluation to Alertmanager.

Architecture: Mirror the existing argocd/app-configs/grafana/ntfy-*.yaml pattern. Defensive Vault policy + Kubernetes auth role in tf/vault/ (no live consumer in PR1, since ESO uses the wildcard external-secrets-operator policy). New GrafanaContactPoint CRDs use the grafana-operator’s native pushover receiver type with priority templated from the severity label. A new GrafanaNotificationPolicy CRD installs a root policy that routes by domain label to the four Pushover receivers, with the existing ntfy-critical receiver as the default route fallback.

Tech Stack: Terraform (HashiCorp Vault provider), Kubernetes (grafana-operator v1beta1 CRDs, ExternalSecrets v1), Pushover REST API (consumed indirectly via grafana-operator). HCP Terraform Cloud workspace main-cluster-vault applies the TF on PR merge. ArgoCD app grafana (auto-sync, prune) reconciles the CRDs.

Spec reference: docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md

Workspace: .worktrees/feat-pushover-alertmanager (jj workspace pushover-alertmanager, parented to main).

Manual prerequisites (engineer must do before starting)

These produce credentials that the implementation tasks consume. Do not commit any of the resulting tokens or keys to git.

Prereq A: Create four Pushover applications

In the Pushover console (https://pushover.net/apps), create four applications. Record the API token for each.

App name	Description	Icon (suggested)
`cluster-infra`	k8s core, longhorn, cnpg, vault, argocd, ingress, cert-manager	gear
`cluster-data`	nats, postgres, dolt, redis, jetstream	database
`cluster-apps`	tandoor, karakeep, grafana stack itself, user-facing	grid
`cluster-edge`	router-hosts, cloudflared, external probes	globe

Note your Pushover user key from the main account page (https://pushover.net/).

Prereq B: Write Vault KV secrets

Authenticate to Vault as a user with write access to secret/data/fzymgc-house/cluster/pushover/* (e.g., your admin role). Run:

vault kv put secret/fzymgc-house/cluster/pushover \
  user_key='<your-pushover-user-key>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-infra \
  token='<token-for-cluster-infra-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-data \
  token='<token-for-cluster-data-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-apps \
  token='<token-for-cluster-apps-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-edge \
  token='<token-for-cluster-edge-app>'

Verify each path:

vault kv list secret/fzymgc-house/cluster/pushover/app
# expected: cluster-infra, cluster-data, cluster-apps, cluster-edge
vault kv get -field=user_key secret/fzymgc-house/cluster/pushover
# expected: your user key

The TF resources added in Task 2 do not create or read this data. They define a defensive policy whose actual access path (ESO) already has wildcard read via external-secrets-operator policy.

File structure

File	Status	Responsibility
`tf/vault/policy-pushover.tf`	Create	Defensive Vault policy granting read on the pushover KV path tree. Mirror of `policy-ntfy.tf`.
`tf/vault/k8s-pushover.tf`	Create	Defensive Kubernetes auth role binding the policy to the `default` SA in the `prometheus` namespace (for future Alertmanager direct auth in PR2+).
`argocd/app-configs/grafana/pushover-tokens-external-secret.yaml`	Create	ExternalSecret materializing all five Pushover credentials (user key + four app tokens) into one Kubernetes Secret in the `grafana` namespace.
`argocd/app-configs/grafana/pushover-contact-points.yaml`	Create	Four `GrafanaContactPoint` CRDs (one per domain). Each uses the operator’s native `pushover` receiver type. Priority is templated from `.CommonLabels.severity`.
`argocd/app-configs/grafana/notification-policy.yaml`	Create	One `GrafanaNotificationPolicy` CRD. Root receiver = `ntfy-critical` (default fallback). Child routes match on `domain` label and forward to `pushover-infra`/`pushover-data`/`pushover-apps`/`pushover-edge`.
`argocd/app-configs/grafana/kustomization.yaml`	Modify	Add the three new YAML files to the `resources:` list.
`docs/reference/secrets.md`	Modify	Add the `fzymgc-house/cluster/pushover` and `fzymgc-house/cluster/pushover/app/*` paths under the Vault secrets table.
`docs/operations/pushover.md`	Create	Short operations doc: how to create new Pushover apps, where tokens live in Vault, the manual `vault kv put` runbook from Prereq B, and how to add a new Pushover app + ExternalSecret data entry.

Task 1: Verify workspace and toolchain

Files: none (verification only)

Step 1.1: Confirm you are in the correct workspace

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager workspace list | rg pushover-alertmanager

Expected output line:

pushover-alertmanager: <change-id> <commit-id> <description>

Step 1.2: Confirm Pushover prereqs are complete

vault kv list secret/fzymgc-house/cluster/pushover/app

Expected output:

Keys
----
cluster-apps
cluster-data
cluster-edge
cluster-infra

If this fails, complete Prereqs A and B above before continuing.

Step 1.3: Confirm grafana-operator’s GrafanaContactPoint pushover schema

kubectl --context fzymgc-house get crd grafanacontactpoints.grafana.integreatly.org -o yaml \
  | rg -A 80 'pushover'

Read the resulting schema. Confirm that spec.receivers[].type: pushover is supported and that spec.receivers[].settings accepts both userKey and apiToken (or equivalent) string fields. Also confirm whether priority accepts Go-template strings or requires a static int.

If priority accepts templates: proceed with Task 5 as written (4 contact points).
If priority requires static int: stop and replan Task 5 to use 12 contact points (4 domains × 3 severities) before proceeding.
Step 1.4: Confirm Pushover schema decision

Document the outcome of Step 1.3 in a one-line note in this plan file (under this task) so subsequent reviewers know which path was taken.

Task 2: Add Vault Terraform — defensive policy + role

Files:

Create: tf/vault/policy-pushover.tf
Create: tf/vault/k8s-pushover.tf
Step 2.1: Write tf/vault/policy-pushover.tf

# SPDX-License-Identifier: MIT
# terraform: language=hcl

# Defensive pre-declaration: pushover secrets are read via ESO + the vault ClusterSecretStore
# (which uses the external-secrets-operator policy with a wildcard secret/data/* grant).
# This policy exists in case a future bootstrap mechanism authenticates to Vault directly
# (e.g. a Job using vault-agent or the Vault SDK). Mirror of the ntfy / karakeep pattern.
resource "vault_policy" "pushover" {
  name   = "pushover"
  policy = <<EOT
# Allow pushover-aware workloads to read the user key and per-app tokens
# (used if a workload authenticates to Vault directly rather than via ESO)
path "secret/data/fzymgc-house/cluster/pushover" {
  capabilities = ["read", "list"]
}

path "secret/data/fzymgc-house/cluster/pushover/*" {
  capabilities = ["read", "list"]
}
EOT
}

Step 2.2: Write tf/vault/k8s-pushover.tf

# SPDX-License-Identifier: MIT
# terraform: language=hcl

# Defensive pre-declaration: pushover secrets are read via ESO + the vault ClusterSecretStore
# today, not by Alertmanager authenticating to Vault directly. This role exists in case a
# future bootstrap mechanism needs direct Vault auth (e.g. Alertmanager with a vault-agent
# sidecar). Mirror of the ntfy pattern.
resource "vault_kubernetes_auth_backend_role" "pushover" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "pushover"
  bound_service_account_names      = ["default"]
  bound_service_account_namespaces = ["prometheus"]
  audience                         = "https://kubernetes.default.svc.cluster.local"
  token_ttl                        = 3600
  token_policies                   = ["default", vault_policy.pushover.name]
}

Step 2.3: Format and validate Terraform

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager/tf/vault
terraform fmt -check -recursive

Expected: no output (exit 0). If terraform fmt -check reports diffs, run terraform fmt -recursive to fix and re-run the check.

terraform init -backend=false
terraform validate

Expected: Success! The configuration is valid.

Step 2.4: Commit

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager commit -m "feat(vault): add defensive pushover policy and k8s auth role

Mirrors the ntfy / karakeep defensive pattern. Pushover secrets are read
via ESO using the wildcard external-secrets-operator policy; this policy
+ role pair exist for future direct-auth scenarios (e.g. Alertmanager
with vault-agent sidecar). No live consumer in PR1."

Verify the commit landed:

jj --no-pager log -r '@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Expected: a line ending in feat(vault): add defensive pushover policy and k8s auth role.

Task 3: Add Grafana ExternalSecret for Pushover tokens

Files:

Create: argocd/app-configs/grafana/pushover-tokens-external-secret.yaml
Step 3.1: Write the ExternalSecret manifest

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: pushover-grafana-tokens
  namespace: grafana
spec:
  refreshPolicy: Periodic
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: pushover-grafana-tokens
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      type: Opaque
      data:
        # Pushover user key — same value across all four apps.
        user_key: "{{ .user_key }}"
        # Per-app API tokens. Underscored keys (cluster_infra) for k8s Secret friendliness;
        # the Pushover application names use hyphens (cluster-infra) but k8s Secret keys
        # are restricted to [-._a-zA-Z0-9] so underscores read more naturally in valuesFrom.
        cluster_infra_token: "{{ .cluster_infra_token }}"
        cluster_data_token:  "{{ .cluster_data_token }}"
        cluster_apps_token:  "{{ .cluster_apps_token }}"
        cluster_edge_token:  "{{ .cluster_edge_token }}"
  data:
    - secretKey: user_key
      remoteRef:
        key: fzymgc-house/cluster/pushover
        property: user_key
    - secretKey: cluster_infra_token
      remoteRef:
        key: fzymgc-house/cluster/pushover/app/cluster-infra
        property: token
    - secretKey: cluster_data_token
      remoteRef:
        key: fzymgc-house/cluster/pushover/app/cluster-data
        property: token
    - secretKey: cluster_apps_token
      remoteRef:
        key: fzymgc-house/cluster/pushover/app/cluster-apps
        property: token
    - secretKey: cluster_edge_token
      remoteRef:
        key: fzymgc-house/cluster/pushover/app/cluster-edge
        property: token

Step 3.2: Lint YAML

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
yamllint argocd/app-configs/grafana/pushover-tokens-external-secret.yaml

Expected: no output (exit 0). If yamllint reports errors, fix them before continuing.

Step 3.3: Commit

jj --no-pager commit -m "feat(grafana): add ExternalSecret materialising pushover tokens

Pulls user key + four per-app tokens from secret/fzymgc-house/cluster/pushover/*
in Vault and renders them into a single Kubernetes Secret consumed by the
GrafanaContactPoint resources added in the next commit. Mirrors the
ntfy-tokens-external-secret pattern."

Verify:

jj --no-pager log -r '@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Expected: a line ending in feat(grafana): add ExternalSecret materialising pushover tokens.

Task 4: Add the four GrafanaContactPoint resources

Files:

Create: argocd/app-configs/grafana/pushover-contact-points.yaml
Step 4.1: Write the four contact point manifests

---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaContactPoint
metadata:
  name: pushover-infra
  namespace: grafana
spec:
  name: pushover-infra
  instanceSelector:
    matchLabels:
      dashboards: grafana
  editable: false
  receivers:
    - type: pushover
      settings:
        # Severity → Pushover priority mapping per the design spec.
        # Templating reads the Grafana alert's severity label.
        # priority values: -2 (lowest, log only), -1 (low/no sound), 0 (normal),
        # 1 (high/bypass quiet hours), 2 (emergency/requires ack).
        priority: '{{ if eq (index .CommonLabels "severity") "critical" }}1{{ else if eq (index .CommonLabels "severity") "warning" }}0{{ else }}-1{{ end }}'
        title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
        message: '{{ .CommonAnnotations.summary }}{{ if .CommonAnnotations.description }}\n{{ .CommonAnnotations.description }}{{ end }}'
      valuesFrom:
        - targetPath: settings.userKey
          valueFrom:
            secretKeyRef:
              name: pushover-grafana-tokens
              key: user_key
        - targetPath: settings.apiToken
          valueFrom:
            secretKeyRef:
              name: pushover-grafana-tokens
              key: cluster_infra_token
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaContactPoint
metadata:
  name: pushover-data
  namespace: grafana
spec:
  name: pushover-data
  instanceSelector:
    matchLabels:
      dashboards: grafana
  editable: false
  receivers:
    - type: pushover
      settings:
        priority: '{{ if eq (index .CommonLabels "severity") "critical" }}1{{ else if eq (index .CommonLabels "severity") "warning" }}0{{ else }}-1{{ end }}'
        title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
        message: '{{ .CommonAnnotations.summary }}{{ if .CommonAnnotations.description }}\n{{ .CommonAnnotations.description }}{{ end }}'
      valuesFrom:
        - targetPath: settings.userKey
          valueFrom:
            secretKeyRef:
              name: pushover-grafana-tokens
              key: user_key
        - targetPath: settings.apiToken
          valueFrom:
            secretKeyRef:
              name: pushover-grafana-tokens
              key: cluster_data_token
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaContactPoint
metadata:
  name: pushover-apps
  namespace: grafana
spec:
  name: pushover-apps
  instanceSelector:
    matchLabels:
      dashboards: grafana
  editable: false
  receivers:
    - type: pushover
      settings:
        priority: '{{ if eq (index .CommonLabels "severity") "critical" }}1{{ else if eq (index .CommonLabels "severity") "warning" }}0{{ else }}-1{{ end }}'
        title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
        message: '{{ .CommonAnnotations.summary }}{{ if .CommonAnnotations.description }}\n{{ .CommonAnnotations.description }}{{ end }}'
      valuesFrom:
        - targetPath: settings.userKey
          valueFrom:
            secretKeyRef:
              name: pushover-grafana-tokens
              key: user_key
        - targetPath: settings.apiToken
          valueFrom:
            secretKeyRef:
              name: pushover-grafana-tokens
              key: cluster_apps_token
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaContactPoint
metadata:
  name: pushover-edge
  namespace: grafana
spec:
  name: pushover-edge
  instanceSelector:
    matchLabels:
      dashboards: grafana
  editable: false
  receivers:
    - type: pushover
      settings:
        priority: '{{ if eq (index .CommonLabels "severity") "critical" }}1{{ else if eq (index .CommonLabels "severity") "warning" }}0{{ else }}-1{{ end }}'
        title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
        message: '{{ .CommonAnnotations.summary }}{{ if .CommonAnnotations.description }}\n{{ .CommonAnnotations.description }}{{ end }}'
      valuesFrom:
        - targetPath: settings.userKey
          valueFrom:
            secretKeyRef:
              name: pushover-grafana-tokens
              key: user_key
        - targetPath: settings.apiToken
          valueFrom:
            secretKeyRef:
              name: pushover-grafana-tokens
              key: cluster_edge_token

Step 4.2: Lint YAML

yamllint argocd/app-configs/grafana/pushover-contact-points.yaml

Expected: no output (exit 0).

Step 4.3: Commit

jj --no-pager commit -m "feat(grafana): add four pushover contact points (infra/data/apps/edge)

One GrafanaContactPoint per coarse domain, each consuming the user key
and its corresponding app token from pushover-grafana-tokens. Severity
label drives Pushover priority via Go template (critical=1, warning=0,
info/other=-1). No alerts route to these yet — wired up by the
notification policy in the next commit."

Task 5: Add the GrafanaNotificationPolicy

Files:

Create: argocd/app-configs/grafana/notification-policy.yaml
Step 5.1: Write the notification policy

---
# Replaces Grafana's built-in default notification policy. Root receiver is
# ntfy-critical (the existing default) so any alert that doesn't carry a
# `domain` label still goes somewhere sensible during the migration.
# Once Alertmanager owns routing (PR2-PR5), this CRD will be removed
# along with the ntfy contact points.
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaNotificationPolicy
metadata:
  name: notification-policy
  namespace: grafana
spec:
  instanceSelector:
    matchLabels:
      dashboards: grafana
  editable: false
  route:
    receiver: ntfy-critical
    group_by:
      - alertname
      - service
      - namespace
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: pushover-infra
        object_matchers:
          - [domain, =, infra]
        continue: false
      - receiver: pushover-data
        object_matchers:
          - [domain, =, data]
        continue: false
      - receiver: pushover-apps
        object_matchers:
          - [domain, =, apps]
        continue: false
      - receiver: pushover-edge
        object_matchers:
          - [domain, =, edge]
        continue: false

Step 5.2: Lint YAML

yamllint argocd/app-configs/grafana/notification-policy.yaml

Expected: no output (exit 0).

Step 5.3: Commit

jj --no-pager commit -m "feat(grafana): add notification policy routing by domain label

Net-new GrafanaNotificationPolicy resource. Root receiver is the existing
ntfy-critical (default fallback for alerts without a domain label).
Child routes match on domain={infra,data,apps,edge} and forward to the
corresponding pushover-<domain> contact point. Removed by PR5 once
Alertmanager owns routing."

Task 6: Wire new resources into the grafana kustomization

Files:

Modify: argocd/app-configs/grafana/kustomization.yaml
Step 6.1: Edit kustomization.yaml

Current contents:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - grafana.yaml
  - grafana-certificate.yaml
  - grafana-admin-external-secret.yaml
  - grafana-oidc-creds.yaml
  - ntfy-tokens-external-secret.yaml
  - ntfy-contact-points.yaml
  - datasources
  - dashboards

Replace with:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - grafana.yaml
  - grafana-certificate.yaml
  - grafana-admin-external-secret.yaml
  - grafana-oidc-creds.yaml
  - ntfy-tokens-external-secret.yaml
  - ntfy-contact-points.yaml
  - pushover-tokens-external-secret.yaml
  - pushover-contact-points.yaml
  - notification-policy.yaml
  - datasources
  - dashboards

Step 6.2: Validate kustomize output

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
kubectl --context fzymgc-house apply --dry-run=client -k argocd/app-configs/grafana 2>&1 | tee /tmp/kustomize-dryrun.txt

Expected: every resource in the kustomization output reports created (dry run) or unchanged (dry run) with no error lines. Specifically grep for the new resources:

rg 'pushover-(infra|data|apps|edge)|pushover-grafana-tokens|notification-policy' /tmp/kustomize-dryrun.txt

Expected: at least 6 hits (4 contact points + 1 ExternalSecret + 1 NotificationPolicy).

Step 6.3: Lint YAML

yamllint argocd/app-configs/grafana/kustomization.yaml

Expected: no output.

Step 6.4: Commit

jj --no-pager commit -m "feat(grafana): include pushover and notification-policy in kustomization

Adds pushover-tokens-external-secret.yaml, pushover-contact-points.yaml,
and notification-policy.yaml to the grafana kustomization so ArgoCD will
sync them once the PR merges."

Task 7: Document Vault paths and add operations runbook

Files:

Modify: docs/reference/secrets.md
Create: docs/operations/pushover.md
Step 7.1: Read the current secrets reference layout

rg -A 2 -B 1 'fzymgc-house/cluster/ntfy' docs/reference/secrets.md

Note the table style and column ordering used for ntfy entries — replicate it exactly for pushover.

Step 7.2: Add pushover entries to docs/reference/secrets.md

In the same table where the ntfy paths are listed, add rows for the pushover paths. The exact format must match the surrounding rows — read the existing ntfy rows in Step 7.1 and use that shape. Required new rows (one per Vault path):

Vault path	Properties	Consumer
`fzymgc-house/cluster/pushover`	`user_key`	grafana (PR1), Alertmanager (PR2)
`fzymgc-house/cluster/pushover/app/cluster-infra`	`token`	grafana (PR1), Alertmanager (PR2)
`fzymgc-house/cluster/pushover/app/cluster-data`	`token`	grafana (PR1), Alertmanager (PR2)
`fzymgc-house/cluster/pushover/app/cluster-apps`	`token`	grafana (PR1), Alertmanager (PR2)
`fzymgc-house/cluster/pushover/app/cluster-edge`	`token`	grafana (PR1), Alertmanager (PR2)

Match whatever column headers and Markdown table format the existing file uses; do not invent new columns.

Step 7.3: Write docs/operations/pushover.md

# Pushover Operations

Pushover is the cluster's mobile notification provider, replacing the
self-hosted ntfy stack across PRs 1–6 of the pushover + alertmanager
migration. See `../engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md`
for the full design.

## Application layout

The cluster registers four Pushover applications. Each application has
its own API token and corresponds to one coarse alert domain.

| App name | Domain | What it covers |
|---|---|---|
| `cluster-infra` | infra | k8s core, longhorn, cnpg, vault, argocd, ingress, cert-manager |
| `cluster-data`  | data  | nats, postgres, dolt, redis, jetstream |
| `cluster-apps`  | apps  | tandoor, karakeep, grafana stack itself, user-facing apps |
| `cluster-edge`  | edge  | router-hosts, cloudflared, external probes |

The Pushover **user key** is shared across all four applications. Per-app
sound, icon, and quiet-hours preferences are configured device-side in
the Pushover mobile app (not in git).

## Vault paths

All credentials live under `secret/fzymgc-house/cluster/pushover/`:

| Path | Property | Notes |
|---|---|---|
| `secret/fzymgc-house/cluster/pushover` | `user_key` | Single user key, shared |
| `secret/fzymgc-house/cluster/pushover/app/cluster-infra` | `token` | API token for cluster-infra app |
| `secret/fzymgc-house/cluster/pushover/app/cluster-data`  | `token` | API token for cluster-data app  |
| `secret/fzymgc-house/cluster/pushover/app/cluster-apps`  | `token` | API token for cluster-apps app  |
| `secret/fzymgc-house/cluster/pushover/app/cluster-edge`  | `token` | API token for cluster-edge app  |

Read access is granted via the `external-secrets-operator` wildcard
policy on `secret/data/*`. The dedicated `pushover` policy in `tf/vault/` is defensive
pre-declaration only (see `tf/vault/policy-pushover.tf` and
`tf/vault/k8s-pushover.tf`).

## Bootstrap runbook (one-time)

After provisioning the Pushover applications in the Pushover console,
write each token to Vault:

```bash
vault kv put secret/fzymgc-house/cluster/pushover \
  user_key='<your-pushover-user-key>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-infra \
  token='<token-for-cluster-infra-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-data \
  token='<token-for-cluster-data-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-apps \
  token='<token-for-cluster-apps-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-edge \
  token='<token-for-cluster-edge-app>'
```

Verify:

```bash
vault kv list secret/fzymgc-house/cluster/pushover/app
# expected: cluster-apps, cluster-data, cluster-edge, cluster-infra
```

## Adding a new Pushover application

1. Create the application in `https://pushover.net/apps`. Record the
   API token.
2. Pick a domain bucket (infra/data/apps/edge) the new app belongs to,
   OR add a new domain. Adding a new domain requires PR-level changes
   to the Grafana notification policy and (post-PR2) the Alertmanager
   route tree.
3. `vault kv put secret/fzymgc-house/cluster/pushover/app/<app-name> token='<api-token>'`.
4. Add a new entry to the `data:` block of `pushover-tokens-external-secret.yaml`
   referencing the new path.
5. Add the new key to the ExternalSecret's `target.template.data` block.
6. Reference it from the corresponding `GrafanaContactPoint` (PR1+) and
   `pushover_configs.token_file` Alertmanager receiver (PR2+).

## Token rotation

Pushover tokens are read by ESO every 15 minutes. Consumers (Grafana,
Alertmanager) read the resulting Kubernetes Secret at config-load, not
on every alert. After rotating a token in Vault:

- **Grafana**: the grafana-operator reconciles ContactPoint resources
  on a short interval; rotation should propagate within a minute.
- **Alertmanager** (post-PR2): manual restart required —
  `kubectl -n prometheus rollout restart statefulset/alertmanager-kps-alertmanager`.

## Quota and rate

Pushover's free tier allows 10,000 messages per month per application
(check current limits at `https://pushover.net/api`). The four
applications share the user account but have independent monthly
counters. Monitor via:

```bash
curl -s "https://api.pushover.net/1/apps/limits.json?token=<api-token>" | jq
```

A delivery-rate alert is tracked as a follow-up bead; not part of PR1.

NOTE: the tf/vault/k8s-pushover.tf reference above is correct — we removed the redundant vault_policy.grafana_pushover_tokens resource during code review. The file currently contains only the auth backend role.

Step 7.4: Lint markdown

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
rumdl docs/reference/secrets.md docs/operations/pushover.md

Expected: no errors.

Step 7.5: Commit

jj --no-pager commit -m "docs(pushover): document vault paths, bootstrap runbook, and operations

Adds pushover entries to docs/reference/secrets.md and a new
docs/operations/pushover.md covering app layout, vault paths, the
one-time bootstrap runbook, adding new apps, token rotation, and quota."

Task 8: Pre-push verification

Files: none (verification only)

Step 8.1: Confirm all expected commits exist

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager log -r 'main..@-' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

Expected output (six commits, in this order from most recent):

<id> docs(pushover): document vault paths, bootstrap runbook, and operations
<id> feat(grafana): include pushover and notification-policy in kustomization
<id> feat(grafana): add notification policy routing by domain label
<id> feat(grafana): add four pushover contact points (infra/data/apps/edge)
<id> feat(grafana): add ExternalSecret materialising pushover tokens
<id> feat(vault): add defensive pushover policy and k8s auth role

If any commit is missing, return to the corresponding task.

Step 8.2: Confirm no secrets in the diff

jj --no-pager diff -r 'main..@-' | rg -i 'token|user.?key|password|secret' | rg -v 'secretKeyRef|secret/data|secretStoreRef|target.name|key:|user_key|cluster-(infra|data|apps|edge)_token|api[_-]?token|.token|userKey|apiToken'

Expected: no output (after filtering legitimate references). Any literal token value in the diff is a critical bug — stop and remove it.

Step 8.3: Run all repo linters

lefthook run pre-commit --all-files

Expected: all hooks pass. If a hook fails, fix the underlying issue (do not skip with --no-verify).

Step 8.4: Confirm workspace base is current

jj --no-pager git fetch
jj --no-pager log -r 'main' --no-graph -T 'change_id.short(8) ++ " " ++ description.first_line() ++ "\n"'

If main has moved since the workspace was created, rebase:

jj --no-pager rebase -s 'roots(main..@-)' -d main --skip-emptied

Then re-run Step 8.1 to confirm the commit chain is intact.

Task 9: Push and open PR

Files: none (VCS + GitHub operations)

Step 9.1: Create the bookmark and push

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/feat-pushover-alertmanager
jj --no-pager bookmark create feat/pushover-pr1-grafana-receiver -r @-
jj --no-pager git push -b feat/pushover-pr1-grafana-receiver

Expected: push succeeds. If the remote rejects with branch-protection error, that’s expected only on main — the feature branch should push cleanly.

Step 9.2: Open the PR

gh pr create \
  --title "feat(grafana): add pushover as parallel receiver alongside ntfy (PR1/6)" \
  --body "$(cat <<'EOF'
## Summary

PR1 of the pushover + alertmanager migration. Adds Pushover as a parallel notification path in Grafana — four `GrafanaContactPoint` resources (one per coarse domain) routed by a new `domain` alert label. ntfy remains the default route fallback. No existing alert behavior changes unless a rule starts emitting a `domain` label.

Spec: `docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md`
Plan: `docs/engineering/plans/2026-05-10-pushover-pr1-grafana-receiver.md`

## Changes

- `tf/vault/policy-pushover.tf`, `tf/vault/k8s-pushover.tf` — defensive Vault policy + k8s auth role (no live consumer; mirrors ntfy/karakeep pattern)
- `argocd/app-configs/grafana/pushover-tokens-external-secret.yaml` — ExternalSecret materialising user key + 4 app tokens
- `argocd/app-configs/grafana/pushover-contact-points.yaml` — four `GrafanaContactPoint` resources (pushover-infra/data/apps/edge)
- `argocd/app-configs/grafana/notification-policy.yaml` — net-new `GrafanaNotificationPolicy` routing by `domain` label
- `argocd/app-configs/grafana/kustomization.yaml` — wire new resources in
- `docs/reference/secrets.md`, `docs/operations/pushover.md` — vault paths + operations runbook

## Pre-merge prerequisites (manual, by reviewer or PR author)

- [ ] Four Pushover applications created in https://pushover.net/apps (cluster-infra, cluster-data, cluster-apps, cluster-edge); API tokens recorded.
- [ ] User key + four app tokens written to Vault per the runbook in `docs/operations/pushover.md`.
- [ ] `vault kv list secret/fzymgc-house/cluster/pushover/app` returns all four app names.

## Post-merge verification

- [ ] HCP TFC `main-cluster-vault` workspace plans + applies cleanly; new policies and role appear in Vault.
- [ ] ArgoCD `grafana` app syncs cleanly; `kubectl -n grafana get externalsecret pushover-grafana-tokens` reports `Ready=True`.
- [ ] `kubectl -n grafana get secret pushover-grafana-tokens -o jsonpath='{.data}' | jq 'keys'` returns `["cluster_apps_token","cluster_data_token","cluster_edge_token","cluster_infra_token","user_key"]`.
- [ ] `kubectl -n grafana get grafanacontactpoint` lists four `pushover-*` resources, all with `READY=True`.
- [ ] `kubectl -n grafana get grafananotificationpolicy notification-policy` reports `READY=True`.
- [ ] In Grafana UI: Alerting → Contact points lists `pushover-infra`, `pushover-data`, `pushover-apps`, `pushover-edge`. Click "Test" on one — confirm Pushover notification arrives on phone with the expected app icon and priority sound.
- [ ] Create a throwaway `GrafanaAlertRuleGroup` with `labels: { domain: data, severity: warning }` that fires immediately. Confirm the notification arrives via `cluster-data` Pushover app at priority 0. Delete the throwaway.
- [ ] Repeat the throwaway test for `domain: edge, severity: critical` (priority 1) to verify routing + priority templating both work.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Step 9.3: Record the PR URL

The gh pr create command prints the PR URL. Save it for use during post-merge verification.

Task 10: Post-merge verification (executed after CI + merge)

Files: none (live cluster verification)

This task runs after the PR merges and HCP TFC + ArgoCD have applied the changes. The PR body’s “Post-merge verification” checklist is the source of truth; use the steps below to drive it.

Step 10.1: Wait for HCP TFC apply

gh pr view <pr-number> --json state,mergedAt

Once mergedAt is set, watch the TFC run via the HCP UI or:

# Use the fzymgc-house:terraform skill to query the latest run on workspace main-cluster-vault.

Expected: latest run on main-cluster-vault shows applied status with the two new resources (vault_policy.pushover, vault_kubernetes_auth_backend_role.pushover) created.

Step 10.2: Wait for ArgoCD sync

kubectl --context fzymgc-house get application grafana -n argocd -o jsonpath='{.status.sync.status} {.status.health.status}'

Expected: Synced Healthy.

Step 10.3: Verify ExternalSecret rendered the Kubernetes Secret

kubectl --context fzymgc-house -n grafana get externalsecret pushover-grafana-tokens
kubectl --context fzymgc-house -n grafana get secret pushover-grafana-tokens \
  -o jsonpath='{.data}' | jq 'keys'

Expected ExternalSecret status: STATUS=SecretSynced, READY=True. Expected jq output:

[
  "cluster_apps_token",
  "cluster_data_token",
  "cluster_edge_token",
  "cluster_infra_token",
  "user_key"
]

Step 10.4: Verify GrafanaContactPoint reconciliation

kubectl --context fzymgc-house -n grafana get grafanacontactpoint

Expected: four rows for pushover-infra/data/apps/edge, all with READY=True.

Step 10.5: Verify GrafanaNotificationPolicy reconciliation

kubectl --context fzymgc-house -n grafana get grafananotificationpolicy notification-policy

Expected: READY=True. If the policy controller reports a conflict with an existing policy, that means the operator was managing a default policy elsewhere — investigate via kubectl -n grafana describe grafananotificationpolicy notification-policy.

Step 10.6: UI smoke test — send a test notification

In the Grafana UI: Alerting → Contact points → pushover-infra → “Test”. Confirm:

Notification arrives on the configured Pushover device(s).
Pushover application name shown is cluster-infra (verifies token routing).
Priority is 0 (test notifications use default priority since they don’t carry a severity label).
Title and message format match expectations.

Repeat for one other contact point (pushover-data is sufficient).

Step 10.7: End-to-end alert routing test

Create a throwaway GrafanaAlertRuleGroup that fires immediately (in any namespace where you can write to grafanaalertrulegroups). Example:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: pushover-pr1-smoketest
  namespace: grafana
spec:
  folderRef: nats-alerts          # or any existing folder
  instanceSelector:
    matchLabels:
      dashboards: grafana
  interval: 30s
  rules:
    - uid: pushover-pr1-smoketest-data-warning
      title: "PushoverPR1Smoketest_DataWarning"
      condition: A
      for: 0s
      execErrState: Error
      noDataState: NoData
      annotations:
        summary: "PR1 smoketest — should arrive via cluster-data at priority 0"
      labels:
        severity: warning
        domain: data
      data:
        - refId: A
          datasourceUid: __expr__
          relativeTimeRange: { from: 0, to: 0 }
          model:
            datasource: { type: __expr__, uid: __expr__ }
            type: math
            expression: '1'             # always firing
            refId: A

Apply it (e.g., kubectl apply -f /tmp/pushover-smoketest.yaml), wait ≤2 minutes, confirm the notification arrives on cluster-data at priority 0. Then change severity: warning → severity: critical and re-apply; confirm a new notification arrives at priority 1 (high priority sound, bypasses Pushover quiet hours).

Repeat once with domain: edge to confirm routing works for non-data domains.

Delete the smoketest:

kubectl --context fzymgc-house -n grafana delete grafanaalertrulegroup pushover-pr1-smoketest

Step 10.8: Confirm no regressions in existing ntfy paths

kubectl --context fzymgc-house -n grafana get grafanacontactpoint

Expected: the original ntfy-critical, ntfy-warning, ntfy-info contact points still show READY=True. Trigger any existing alert (e.g., temporarily lower a NATS threshold) and confirm it still routes to ntfy. The default-route fallback to ntfy-critical in the new notification policy preserves behavior for any rule that lacks a domain label.

Step 10.9: Clean up workspace

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster
jj --no-pager git fetch
jj --no-pager bookmark delete feat/pushover-pr1-grafana-receiver
jj --no-pager workspace forget pushover-alertmanager
rm -rf .worktrees/feat-pushover-alertmanager

Workspace is reusable for PR2 by re-running jj workspace add — or keep it around if PR2 work starts immediately.

Risks and rollback

Risk	Detection	Rollback
Pushover priority template not supported by grafana-operator’s pushover ContactPoint type	`kubectl describe grafanacontactpoint pushover-infra` shows a reconcile error referencing `priority`	Revert PR1 (`git revert <merge-commit>`) and replan Task 4 with 12 contact points (one per domain × severity, fixed priority each)
ExternalSecret fails to populate (Vault path missing or wrong property name)	`kubectl describe externalsecret pushover-grafana-tokens` shows “secret not found” or “key not found”	Re-run Prereq B with the correct `vault kv put` commands; ESO refresh is automatic within 15m
New notification policy conflicts with an existing operator-managed policy	`kubectl describe grafananotificationpolicy notification-policy` shows conflict	`kubectl -n grafana get grafananotificationpolicy` to find the conflicting resource; reconcile by deleting one or merging routes
Default-route change to `ntfy-critical` causes some warning/info alerts that previously went to `ntfy-warning`/`ntfy-info` to instead go to critical	iOS notifications start arriving with the wrong sound	Edit `notification-policy.yaml` to add `routes` for `severity={warning,info}` falling back to the appropriate ntfy receiver; commit + push
Pushover token leaked into git	`gitleaks` or a manual `rg` finds a real token in the diff	Immediately rotate the affected token in Pushover console; force-push a corrected branch; if already merged, also revoke the token Pushover-side

Out of scope for PR1 (handled in subsequent PRs)

Alertmanager deployment (PR2)
PrometheusRule conversion for NATS rules (PR3)
PrometheusRule conversion for router-hosts rules (PR4)
Removing the ntfy GrafanaContactPoint and ExternalSecret (PR5)
Decommissioning the ntfy stack + Vault data (PR6)

A new plan document will be written for PR2 once PR1 has merged and the post-merge verification checklist is complete.