Pushover Operations

Pushover is the cluster’s mobile notification provider, replacing the self-hosted ntfy stack across PRs 1–6 of the pushover + alertmanager migration. See ../engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md for the full design.

Application layout

The cluster registers four Pushover applications, each with its own API token and corresponding to one coarse alert domain. Only cluster-infra is currently consumed — by the HyperDX clickstack-alerts webhook (see Alerting Operations); there is no per-domain routing tree anymore. The other three apps remain provisioned in Vault (tokens still exist) but nothing in the cluster currently sends to them.

App name	Domain	What it covers
cluster-infra	infra	k8s core, longhorn, cnpg, vault, argocd, ingress, cert-manager (in use)
cluster-data	data	nats, postgres, dolt, redis, jetstream (unused)
cluster-apps	apps	karakeep, user-facing apps (unused)
cluster-edge	edge	router-hosts, cloudflared, external probes (unused)

The Pushover user key is shared across all four applications. Per-app sound, icon, and quiet-hours preferences are configured device-side in the Pushover mobile app (not in git).

Vault paths

All credentials live under secret/fzymgc-house/cluster/pushover/:

Path	Property	Notes
secret/fzymgc-house/cluster/pushover	user_key	Single user key, shared
secret/fzymgc-house/cluster/pushover/app/cluster-infra	token	API token for cluster-infra app
secret/fzymgc-house/cluster/pushover/app/cluster-data	token	API token for cluster-data app
secret/fzymgc-house/cluster/pushover/app/cluster-apps	token	API token for cluster-apps app
secret/fzymgc-house/cluster/pushover/app/cluster-edge	token	API token for cluster-edge app

Read access is granted via the external-secrets-operator wildcard policy on secret/data/*. The dedicated pushover policy in tf/vault/ is defensive pre-declaration only (see tf/vault/policy-pushover.tf and tf/vault/k8s-pushover.tf).

Bootstrap runbook (one-time)

After provisioning the Pushover applications in the Pushover console, write each token to Vault:

vault kv put secret/fzymgc-house/cluster/pushover \
  user_key='<your-pushover-user-key>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-infra \
  token='<token-for-cluster-infra-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-data \
  token='<token-for-cluster-data-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-apps \
  token='<token-for-cluster-apps-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-edge \
  token='<token-for-cluster-edge-app>'

Verify:

vault kv list secret/fzymgc-house/cluster/pushover/app
# expected: cluster-apps, cluster-data, cluster-edge, cluster-infra

Adding a new Pushover application

1. Create the application in https://pushover.net/apps. Record the API token.
2. vault kv put secret/fzymgc-house/cluster/pushover/app/<app-name> token='<api-token>'.
3. Add a new secretKey/remoteRef entry to the data: block of argocd/app-configs/clickstack-alerts/external-secret.yaml referencing the new path.
4. Reference the new token in a new db.webhooks upsert in the clickstack-alerts bootstrap script (see Alerting Operations) if the app is meant to receive HyperDX alerts.

Token rotation

Pushover tokens are read by ESO every 15 minutes into a Kubernetes Secret. Consumers read that Secret at config-load, not on every alert:

• HyperDX (clickstack-alerts): the bootstrap script bakes the token into the webhook document in HyperDX’s MongoDB at Job-run time. After rotating a token in Vault, wait for ESO to refresh the Secret (up to 15 min), then re-sync the clickstack-alerts Application in ArgoCD to re-fire its PostSync bootstrap Job. See Alerting Operations.

Quota and rate

Pushover’s free tier allows 10,000 messages per month per application (check current limits at https://pushover.net/api). The four applications share the user account but have independent monthly counters. Monitor via:

curl -s "https://api.pushover.net/1/apps/limits.json?token=<api-token>" | jq

A delivery-rate alert is tracked as a follow-up bead; not part of PR1.