Pushover Operations

Pushover is the cluster’s mobile notification provider, replacing the self-hosted ntfy stack across PRs 1–6 of the pushover + alertmanager migration. See ../engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md for the full design.

Application layout

The cluster registers four Pushover applications, each with its own API token. Two of the four are consumed. The partition in practice is by the alert speaker — which system is doing the talking — rather than by the subject it happens to be talking about; there is no per-domain routing tree anymore.

App name	Speaker	Consumed by
`cluster-infra`	in-cluster alerting	the HyperDX `clickstack-alerts` webhook (see Alerting Operations), and the `hl-assets operational items` scheduled workflow
`cluster-edge`	external probers	every Uptime Kuma monitor that sets a notification — the ClickStack watchdog, and the two `hl-assets` monitors
`cluster-data`	data	unused — provisioned in Vault, nothing sends to it
`cluster-apps`	apps	unused — provisioned in Vault, nothing sends to it

This table previously read that only cluster-infra was consumed and that cluster-edge was unused. That was already wrong when written — tf/uptime-kuma/clickstack-watchdog.tf routes to cluster-edge — and Phase 6 added two more monitors on it. Corrected 2026-08-15.

The Pushover user key is shared across all four applications. Per-app sound, icon, and quiet-hours preferences are configured device-side in the Pushover mobile app (not in git).

Vault paths

All credentials live under secret/fzymgc-house/cluster/pushover/:

Path	Property	Notes
`secret/fzymgc-house/cluster/pushover`	`user_key`	Single user key, shared
`secret/fzymgc-house/cluster/pushover/app/cluster-infra`	`token`	API token for cluster-infra app
`secret/fzymgc-house/cluster/pushover/app/cluster-data`	`token`	API token for cluster-data app
`secret/fzymgc-house/cluster/pushover/app/cluster-apps`	`token`	API token for cluster-apps app
`secret/fzymgc-house/cluster/pushover/app/cluster-edge`	`token`	API token for cluster-edge app

Read access is granted via the external-secrets-operator wildcard policy on secret/data/*. The dedicated pushover policy in tf/vault/ is defensive pre-declaration only (see tf/vault/policy-pushover.tf and tf/vault/k8s-pushover.tf).

Bootstrap runbook (one-time)

After provisioning the Pushover applications in the Pushover console, write each token to Vault:

vault kv put secret/fzymgc-house/cluster/pushover \
  user_key='<your-pushover-user-key>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-infra \
  token='<token-for-cluster-infra-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-data \
  token='<token-for-cluster-data-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-apps \
  token='<token-for-cluster-apps-app>'

vault kv put secret/fzymgc-house/cluster/pushover/app/cluster-edge \
  token='<token-for-cluster-edge-app>'

Verify:

vault kv list secret/fzymgc-house/cluster/pushover/app
# expected: cluster-apps, cluster-data, cluster-edge, cluster-infra

Adding a new Pushover application

Create the application in https://pushover.net/apps. Record the API token.
vault kv put secret/fzymgc-house/cluster/pushover/app/<app-name> token='<api-token>'.
Add a new secretKey/remoteRef entry to the data: block of argocd/app-configs/clickstack-alerts/external-secret.yaml referencing the new path.
Reference the new token in a new db.webhooks upsert in the clickstack-alerts bootstrap script (see Alerting Operations) if the app is meant to receive HyperDX alerts.

Token rotation

Pushover tokens are read by ESO every 15 minutes into a Kubernetes Secret. Consumers read that Secret at config-load, not on every alert:

HyperDX (clickstack-alerts): the bootstrap script bakes the token into the webhook document in HyperDX’s MongoDB at Job-run time. After rotating a token in Vault, wait for ESO to refresh the Secret (up to 15 min), then re-sync the clickstack-alerts Application in ArgoCD to re-fire its PostSync bootstrap Job. See Alerting Operations.

Quota and rate

Pushover’s free tier allows 10,000 messages per month per application (check current limits at https://pushover.net/api). The four applications share the user account but have independent monthly counters. Monitor via:

curl -s "https://api.pushover.net/1/apps/limits.json?token=<api-token>" | jq

A delivery-rate alert is tracked as a follow-up bead; not part of PR1.