Skip to content

Pushover Operations

Pushover delivers the estate’s mobile notifications. Four Pushover applications exist, each with its own API token, and one user key is shared by all of them. Each application is a speaker, the system that sends, not a subject.

Application Speaker Readers of its token
cluster-infra in-cluster and repository automation the HyperDX clickstack-alerts webhook through its ExternalSecret, the hl-assets-operational-items workflow through the arc-runner-jobs policy, and the NAS backup and sandbox units through the nas-kopia-backup and nas-sandbox-tools roles
cluster-edge external probers every Uptime Kuma monitor with a notification, through uptimekuma_notification_pushover.cluster_edge in tf/uptime-kuma
cluster-data none nothing reads it
cluster-apps none nothing reads it

Per-application sound, icon and quiet hours are set in the Pushover mobile app, not in this repository.

Every credential sits under secret/fzymgc-house/cluster/pushover/:

Path Property
cluster/pushover user_key, the one shared user key
cluster/pushover/app/<application> token, the API token of that application

Three policies in tf/vault reach these paths:

  • external-secrets-operator reads every path under cluster/, which covers the clickstack-alerts ExternalSecret.
  • arc-runner-jobs reads cluster/pushover and cluster/pushover/app/cluster-infra.
  • The main-cluster-uptime-kuma workspace policy in policy-terraform-workspaces.tf reads cluster/pushover and cluster/pushover/app/cluster-edge.

Make sure that the layout is complete:

Terminal window
vault kv list secret/fzymgc-house/cluster/pushover/app
# cluster-apps, cluster-data, cluster-edge, cluster-infra
  1. Create the application at https://pushover.net/apps and record its API token.

  2. Write the token with the value on stdin, not in argv:

    Terminal window
    vault kv put secret/fzymgc-house/cluster/pushover/app/<application> -
    # type one line of JSON, then Ctrl-D:
    # {"token": "<api-token>"}
  3. Give the reader a grant and a mapping. For a HyperDX alert, add a secretKey and remoteRef entry to the data: block of argocd/app-configs/clickstack-alerts/external-secret.yaml, then a db.webhooks upsert in the bootstrap script (Alerting). For any other reader, add the path to its policy in tf/vault.

  1. Make a new API token for the application in the Pushover console.
  2. Write it into the field token with the vault kv put command above. The path holds only that field, so a put erases nothing else.
  3. Make each reader pick up the new value:
    • HyperDX bakes the token into its webhook document when the clickstack-alerts bootstrap Job runs. ExternalSecrets refresh the Kubernetes Secret every 15 minutes (refreshInterval in the ExternalSecret). Wait for the refresh, then sync the clickstack-alerts Application in ArgoCD so that its PostSync Job runs again (Alerting).
    • Uptime Kuma reads cluster-edge at apply time. Run the main-cluster-uptime-kuma workspace.
    • The NAS units read cluster-infra at play time. Run the nas-kopia-backup and nas-sandbox-tools tags of the NAS playbook.

Pushover limits the messages each application can send per month, and each application has its own counter. Read the limit and the current usage:

Terminal window
curl -s "https://api.pushover.net/1/apps/limits.json?token=<api-token>" | jq