Skip to content
fzymgc-house Cluster Docs

External CronJob watchdog, not an in-stack tile alert

Date: 2026-07-05 Status: Accepted Decision: hl-hiiu Deciders: Sean Brandt

Context

Section titled “Context”

The replicated rebuild (hl-uj69) needs two detections: engine drift (a future upstream seed table created as plain MergeTree would silently split data across replicas) and total ingest silence. The 2026-07-05 incident demonstrated the core problem: ClickStack is the system that goes dark in exactly the failure mode that most needs alerting, and its tile alerts fired confusing false positives (NATS quorum) off stale data rather than a clear stack-down signal.

Decision

Section titled “Decision”

Detect both schema drift and ingest silence with one external CronJob (every 5 minutes): it checks metric freshness and counts plain-MergeTree tables in the default database, and pushes an uptime-kuma heartbeat only when both pass. uptime-kuma (already pushover-wired) alerts on missed heartbeats. No in-stack tile alert for drift.

Rationale

Section titled “Rationale”
  • An alert living inside the monitored system is structurally blind to that system going dark.
  • One mechanism covers both the drift detector and the dead-man’s switch; the heartbeat message carries the failing check for triage.
  • Supersedes-in-spirit the incident pattern where three alerts fired on stale data and the real cause (disk full) had no alert at all.

Alternatives Considered

Section titled “Alternatives Considered”
  • External watchdog CronJob + uptime-kuma push monitor (chosen): alert path independent of the stack; covers total outage.
  • ClickStack tile alert on system.tables engine drift (rejected): cannot fire when ClickHouse or the alerting pipeline is down — the exact case that matters.
  • Both (rejected): redundant; the tile alert adds a second thing to maintain without covering any additional failure mode.

Consequences

Section titled “Consequences”
  • Positive: alerting survives total ClickStack outage; drift caught within 5 minutes.
  • Negative: adds a CronJob, an ExternalSecret, and a TF-managed uptime-kuma push monitor.
  • Neutral: watchdog uses the read-only app user; no new privileges.