Native CH BACKUP to RustFS replaces Velero CSI snapshots
Date: 2026-07-05 Status: Accepted Decision: hl-olui Deciders: Sean Brandt
Context
Section titled “Context”Velero backs up the ClickHouse data volume via Longhorn CSI snapshots + kopia data-mover to R2: crash-consistent at best, never restore-tested, requiring manual Keeper reconciliation (SYSTEM RESTORE REPLICA) to recover Replicated tables, and churning 200Gi of telemetry daily — data the rebuild (hl-uj69, ADR on disposable telemetry) explicitly declares disposable and now replicates at the application layer.
Decision
Section titled “Decision”Exclude ClickHouse data volumes from Velero via a resource-policies ConfigMap keyed on the longhorn-single StorageClass (action: skip), and back up the database natively instead: weekly BACKUP DATABASE to RustFS S3 plus daily incrementals (base_backup), with one rehearsed RESTORE as the acceptance gate that also qualifies RustFS’s S3 implementation. MongoDB stays on the existing Velero path.
Rationale
Section titled “Rationale”- CSI snapshots of Replicated tables are crash-consistent and require unverified manual Keeper recovery; native BACKUP is table-consistent and restore is symmetric.
- Keying the Velero skip on the StorageClass (not per-PVC labels) survives routine PVC recreation in the replicated topology.
- The bounded, testable backup workload is the right first qualification for RustFS (known S3-semantics gaps; zero prior ClickHouse track record) — unlike a live S3 data disk under merge churn.
Alternatives Considered
Section titled “Alternatives Considered”- Native BACKUP to RustFS + StorageClass-keyed Velero skip (chosen): consistent, rehearsed, qualifies RustFS cheaply.
- Keep Velero CSI snapshots of CH volumes (rejected): false safety — likely unrestorable without manual surgery, expensive for disposable data.
- No CH backups at all (documented fallback): acceptable if RustFS fails qualification — replication is the availability story; MongoDB holds the state that matters.
Consequences
Section titled “Consequences”- Positive: rehearsed, ClickHouse-consistent backups; the skip-policy pattern is reusable for future app-replicated services; RustFS gains a qualification verdict either way.
- Negative: new CronJobs, bucket lifecycle, and Vault-sourced credentials to operate; RustFS failure reverts to no CH backups pending MinIO/Garage evaluation.
- Neutral: R2 backup spend drops (200Gi/day of CH churn removed from Velero).