Pushover Operations
Pushover delivers the estate’s mobile notifications. Four Pushover applications exist, each with its own API token, and one user key is shared by all of them. Each application is a speaker, the system that sends, not a subject.
Applications
Section titled “Applications”| Application | Speaker | Readers of its token |
|---|---|---|
cluster-infra |
in-cluster and repository automation | the HyperDX clickstack-alerts webhook through its ExternalSecret, the hl-assets-operational-items workflow through the arc-runner-jobs policy, and the NAS backup and sandbox units through the nas-kopia-backup and nas-sandbox-tools roles |
cluster-edge |
external probers | every Uptime Kuma monitor with a notification, through uptimekuma_notification_pushover.cluster_edge in tf/uptime-kuma |
cluster-data |
none | nothing reads it |
cluster-apps |
none | nothing reads it |
Per-application sound, icon and quiet hours are set in the Pushover mobile app, not in this repository.
Vault paths
Section titled “Vault paths”Every credential sits under secret/fzymgc-house/cluster/pushover/:
| Path | Property |
|---|---|
cluster/pushover |
user_key, the one shared user key |
cluster/pushover/app/<application> |
token, the API token of that application |
Three policies in tf/vault reach these paths:
external-secrets-operatorreads every path undercluster/, which covers theclickstack-alertsExternalSecret.arc-runner-jobsreadscluster/pushoverandcluster/pushover/app/cluster-infra.- The
main-cluster-uptime-kumaworkspace policy inpolicy-terraform-workspaces.tfreadscluster/pushoverandcluster/pushover/app/cluster-edge.
Make sure that the layout is complete:
vault kv list secret/fzymgc-house/cluster/pushover/app# cluster-apps, cluster-data, cluster-edge, cluster-infraAdd an application
Section titled “Add an application”-
Create the application at
https://pushover.net/appsand record its API token. -
Write the token with the value on stdin, not in argv:
Terminal window vault kv put secret/fzymgc-house/cluster/pushover/app/<application> -# type one line of JSON, then Ctrl-D:# {"token": "<api-token>"} -
Give the reader a grant and a mapping. For a HyperDX alert, add a
secretKeyandremoteRefentry to thedata:block ofargocd/app-configs/clickstack-alerts/external-secret.yaml, then adb.webhooksupsert in the bootstrap script (Alerting). For any other reader, add the path to its policy intf/vault.
Rotate a token
Section titled “Rotate a token”- Make a new API token for the application in the Pushover console.
- Write it into the field
tokenwith thevault kv putcommand above. The path holds only that field, so aputerases nothing else. - Make each reader pick up the new value:
- HyperDX bakes the token into its webhook document when the
clickstack-alertsbootstrap Job runs. ExternalSecrets refresh the Kubernetes Secret every 15 minutes (refreshIntervalin the ExternalSecret). Wait for the refresh, then sync theclickstack-alertsApplication in ArgoCD so that itsPostSyncJob runs again (Alerting). - Uptime Kuma reads
cluster-edgeat apply time. Run themain-cluster-uptime-kumaworkspace. - The NAS units read
cluster-infraat play time. Run thenas-kopia-backupandnas-sandbox-toolstags of the NAS playbook.
- HyperDX bakes the token into its webhook document when the
Pushover limits the messages each application can send per month, and each application has its own counter. Read the limit and the current usage:
curl -s "https://api.pushover.net/1/apps/limits.json?token=<api-token>" | jq