Use dual backup paths for Qdrant memory store
Date: 2026-06-02 Status: Accepted Decision: hl-cy3 Deciders: sean
Context
Section titled “Context”The curated Qdrant memory collection has no durable off-cluster backup today: Velero’s daily schedule omits snapshotMoveData and excludes snapshot-reference resources, so only local Longhorn snapshots exist (lost with the volume/cluster). The engram cutover (hl-3rr.10) is about to perform a kustomize→Helm field-manager handover on the same qdrant-data PVC, so a proven recovery path is required before that risk is taken. Two structurally different mechanisms address different failure modes: app-consistent logical export vs. block-level PVC disaster recovery.
Decision
Section titled “Decision”Implement BOTH the Qdrant-native S3 snapshot mechanism (primary, app-consistent) AND a dedicated Velero Schedule with snapshotMoveData: true (DR, block-level), treating them as orthogonal layers with independent, separately-rehearsed restore paths. The dedicated Velero Schedule is distinct from the broken global daily/weekly schedules.
Rationale
Section titled “Rationale”- The existing global Velero schedule is demonstrably broken for this namespace (zero DataUploads, runs land PartiallyFailed), so a Velero-only approach is unreliable without a dedicated Schedule.
- Qdrant-native snapshots provide app-consistency that block-level PVC copies cannot, plus checksum-verified restore via the REST API.
- Orthogonal failure domains: the DR path stays valid if the Qdrant API/CronJob fails, and the logical path stays valid if Velero/CSI fails.
- The acceptance gate (demonstrated restore) is only meaningful if both paths are exercised independently.
Alternatives Considered
Section titled “Alternatives Considered”Qdrant-native snapshot only — app-consistent, collection-level, no Velero/CSI dependency; BUT cannot recover from PVC corruption or cluster storage loss, and a single mechanism is itself a single point of failure. Rejected.
Velero PVC backup only — block-level DR, reuses existing infra; BUT not app-consistent (mid-write capture), and the existing global schedule is broken (snapshotMoveData:false, PartiallyFailed). Rejected.
Both (chosen) — orthogonal failure domains; a dedicated Velero Schedule (snapshotMoveData:true) closes the DataUpload gap; rehearsing both restores proves each independently. Cost: two configs to maintain + slightly more R2 storage.
Consequences
Section titled “Consequences”Positive:
- A future operator asking “why two backup mechanisms” has a documented answer with the specific failure modes each covers.
- The dedicated Velero Schedule pattern is portable to other stateful namespaces with the same global-schedule gap.
- Both restore paths are rehearsed and evidence-captured before the engram cutover, satisfying the gate for hl-3rr.10.
Negative:
- Two backup configurations to keep synchronized with namespace/workload changes.
- Additional R2 storage for both snapshot types.
Neutral:
- The snapshot-trigger CronJob (app-wiring) and the Velero Schedule (velero app-config) live in different sources and evolve independently.