Qdrant Memory Backup + Verified Restore — Design

• Bead: hl-73s.25 (P1, phase:design, area:memory)
• Date: 2026-06-02
• Status: Draft (pending design-reviewer)
• Gates: hl-3rr.10 (engram multi-source cutover) — the cutover MUST NOT proceed until the curated memory store has a demonstrated restore.

Problem

The curated memory store (Qdrant collection memory on PVC qdrant-data, namespace agent-memory) is a concentrated, sensitive distillation. The engram cutover (hl-3rr.10) is about to perform a ServerSideApply field-manager handover (kustomize → Helm) on that exact PVC. Before taking that risk we need a recovery path that is proven, not assumed.

Grounding the live cluster showed the situation is worse than the bead framed it. There is no durable, restorable backup of the memory data today:

• Velero (chart 12.0.1, features: EnableCSI, deployNodeAgent: true) sweeps agent-memory in its daily/weekly schedules (exclude-only strategy; the namespace is not excluded).
• But the schedule omits snapshotMoveData (defaults false — and the materialized daily-backup-20260602020054 confirms snapshotMoveData: false) and its includedResources whitelist omits volumesnapshots/volumesnapshotcontents (they are explicitly excluded). The most recent run (daily-backup-20260602020054) reports csiVolumeSnapshotsCompleted: 32 yet moves no data to object storage and discards the snapshot references. There were zero DataUploads, PodVolumeBackups, or VolumeSnapshots for agent-memory.
• Net: the R2 backup tarball holds only the empty PVC/PV object specs. The only copies of the data are local Longhorn type: snap snapshots that die with the volume/cluster. That run also finished PartiallyFailed (5 errors / 202 warnings).

So this work establishes — and demonstrably proves — a recoverable memory store.

Goals

1. A durable, off-cluster, app-consistent backup of the memory collection.
2. An independent disaster-recovery backup of the underlying PVC.
3. A demonstrated restore of both paths (the acceptance gate): the seeded dogfood memories (9cc04339…, 8b00b04a…) and the collection point-count are queryable after restoring into a scratch namespace.

Non-goals

• The cutover itself (hl-3rr.10) and cleanup (hl-3rr.11).
• Fixing the global Velero schedule’s exclusion/health (tracked separately).
• The in-cluster bge-m3 SPOF (hl-73s.23).
• engram-repo changes — except the cutover-continuity requirement is documented here and filed as a follow-up bead (see “Cutover continuity”).

Decisions (from brainstorming)

Decision	Choice
Mechanism	Both: Qdrant-native snapshot (primary, app-consistent) + dedicated Velero PVC backup (DR)
Snapshot delivery	Qdrant native S3 snapshot storage — Qdrant writes .snapshot files straight to R2; a CronJob only triggers + prunes
Acceptance gate	One-time demonstrated restore, documented — but rehearse both restore paths
Storage / retention	Same R2 bucket fzymgc-cluster-storage, dedicated prefix memory/qdrant-snapshots/, keep ~14 daily

Architecture

Four small, independently-understandable units.

1. Qdrant native S3 snapshot storage (config on the Qdrant Deployment)

Configure Qdrant to store snapshots directly in R2 via env vars on the qdrant container (argocd/app-configs/agent-memory/qdrant.yaml, pre-cutover):

• QDRANT__STORAGE__SNAPSHOTS_CONFIG__SNAPSHOTS_STORAGE=s3
• QDRANT__STORAGE__SNAPSHOTS_PATH=memory/qdrant-snapshots — acts as the S3 key prefix (Qdrant has no s3_config.prefix field; snapshots_path fills that role only when snapshots_storage = s3), keeping memory snapshots cleanly separate from velero/backups in the shared bucket. (Plan should confirm the exact env-var mapping for the prefix against the running v1.18.1 image before relying on it.)
• QDRANT__STORAGE__SNAPSHOTS_CONFIG__S3_CONFIG__BUCKET=fzymgc-cluster-storage
• QDRANT__STORAGE__SNAPSHOTS_CONFIG__S3_CONFIG__REGION=auto (R2)
• QDRANT__STORAGE__SNAPSHOTS_CONFIG__S3_CONFIG__ENDPOINT_URL=https://<account>.r2.cloudflarestorage.com
• ..._S3_CONFIG__ACCESS_KEY / ..._S3_CONFIG__SECRET_KEY — from a Secret (Component 2), via secretKeyRef.

Applying this restarts the Qdrant pod (strategy: Recreate); the env change does not touch the PVC, so no data risk. memory-mcp reconnects.

2. R2 credentials ExternalSecret (namespace agent-memory)

A new ExternalSecret reusing the existing R2 credentials in Vault (fzymgc-house/cluster/velero/cloudflare-credentials: aws_access_key_id, aws_secret_access_key) into a Secret consumed by the Qdrant Deployment. Verify the vault ClusterSecretStore’s Vault policy permits reading that path from the agent-memory context; extend the policy if not (per repo security constraint: update Vault policies when adding secret access).

3. Snapshot trigger CronJob (namespace agent-memory)

A daily CronJob (curl-only image) that:

1. POST /collections/memory/snapshots?wait=true — Qdrant creates the snapshot and uploads it to R2 under the prefix.
2. GET /collections/memory/snapshots — list.
3. DELETE /collections/memory/snapshots/{name} for all but the newest ~14 — retention/prune (operates on S3-stored snapshots).

A minimal curl-capable image suffices (e.g. curlimages/curl; plan to pin a concrete tag). The CronJob talks only to the Qdrant Service REST API (:6333, unauthenticated ClusterIP), so it is workload-agnostic and survives the cutover unchanged. It is part of Source B (the trimmed agent-memory kustomize that persists post-cutover), not the workload chart.

4. Dedicated Velero PVC backup (namespace velero)

A Schedule (separate from the broken global one) targeting includedNamespaces: [agent-memory] with snapshotMoveData: true and the snapshot-reference resources included, so the CSI datamover offloads the actual PVC bytes to R2 (node-agent already deployed). This is the block-level DR path, independent of the global schedule’s health.

Encryption (no extra key needed): the qdrant-data PVC is on longhorn-encrypted, but Longhorn does LUKS encryption at the CSI layer via the longhorn-crypto-config secret, wired as the node-stage/publish secret on the StorageClass (tf/cluster-bootstrap/longhorn.tf). The volume is decrypted transparently at mount, so Velero’s datamover reads plaintext and needs no crypto key. (Velero’s cloud-credentials does carry an unrelated customer-key field, but the BSL only consumes the cloud S3 key — the customer-key is not the Longhorn key.) The dedicated Schedule therefore needs no special credential wiring: storageLocation: default reuses the same, already-working BSL every other backup uses.

Data flow

Backup (daily): CronJob → Qdrant REST create snapshot → Qdrant flushes a consistent .snapshot → uploads to s3://fzymgc-cluster-storage/memory/qdrant-snapshots/ → CronJob prunes to 14. In parallel, the dedicated Velero Schedule snapshots the PVC via CSI and datamoves it to R2.

Restore (rehearsed, both paths) into a scratch namespace:

• Qdrant path (trusted/primary): deploy a throwaway Qdrant in a scratch ns, configured with the same S3 snapshot storage env (so it can read the s3:// location with the R2 credentials), then POST /collections/memory/snapshots/recover with location = the S3 URL of the latest snapshot, priority: Snapshot, and the checksum for integrity → Qdrant downloads from S3 and recreates the collection → query. (Alternative if withholding creds from the scratch pod is preferred: pre-download the snapshot and recover via a file:// or HTTP location instead.)
• Velero path (DR): velero restore create --from-backup <agent-memory backup> with a namespace mapping to a second scratch ns → PVC + Qdrant restored → query.

Both rehearsals assert: collection memory exists, point-count matches, and the two seeded memory IDs are retrievable. Evidence (command transcripts) is captured in a runbook; scratch namespaces are torn down afterward.

Cutover continuity (the native-S3 ripple)

Native S3 puts the snapshot config on the Qdrant Deployment, which the cutover (Task 7) deletes when it switches to the engram OCI chart. Components 2–4 live in cluster-wiring (Source B / the velero app-config) and survive; only the Component 1 env config is cutover-fragile.

Requirement: the engram Helm chart (or hl-3rr.10’s valuesObject) must carry the Qdrant snapshot S3 configuration so recurring backups continue post-cutover. This is engram-repo work, out of scope here, and will be filed as a follow-up bead that gates hl-3rr.10 (the cutover must not silently drop backup capability). The one-time proof in this bead is performed pre-cutover, so the acceptance gate is satisfiable independently of that follow-up.

Error handling & failure modes

• Snapshot create fails / Qdrant unreachable: CronJob exits non-zero; surfaces as a failed Job (alertable). No partial object is trusted — wait=true makes creation synchronous, and the checksum is recorded.
• R2 upload fails: Qdrant returns an error on create; CronJob fails loudly.
• Prune must never delete the newest: retention sorts by creation time and always keeps ≥1; prune failure is non-fatal to that day’s backup.
• Restore integrity: recover is always called with the SHA256 checksum so a corrupt/truncated snapshot is rejected before overwriting anything.
• Scratch isolation: restore rehearsals run in dedicated scratch namespaces and never target agent-memory, so a botched rehearsal cannot harm live data.

Verification / acceptance gate

The bead closes only when, with evidence captured in the runbook (docs/operations/memory-backup-restore.md):

1. A real snapshot exists in R2 under the prefix (Qdrant path).
2. The dedicated Velero backup completes clean and produces a DataUpload/object in R2 (DR path data actually offloads).
3. Both restore paths are rehearsed into scratch namespaces and the seeded memories + point-count are confirmed queryable.
4. Scratch namespaces are torn down.

GitOps note: the restore rehearsals create ephemeral scratch namespaces/Jobs and a Velero Restore out-of-band (not via ArgoCD), and remove them after. Nothing persists in Git/ArgoCD. This is the documented exception to the “ArgoCD manages deployments” rule, justified because it is one-time verification, not managed app state.

Security

• Reuses existing R2 credentials from Vault; no new secret material in Git.
• Snapshots inherit R2 server-side properties; the source PVC is longhorn-encrypted. (Snapshot-at-rest in R2 relies on R2/account controls — noted, not changed here.)
• Verify/extend the Vault policy so the agent-memory ExternalSecret may read the velero R2 credentials path.
• Longhorn volume encryption is handled at the CSI layer (longhorn-crypto-config, decrypted transparently at mount); the Velero datamover reads plaintext and needs no crypto key (see Component 4). No extra credential wiring required.

Out of scope

Cutover (hl-3rr.10), cleanup (hl-3rr.11), global Velero schedule remediation, bge-m3 SPOF (hl-73s.23), engram-repo changes (except the documented continuity follow-up), and any recurring/automated restore-testing (the gate is a documented one-time demonstration).

Follow-ups to file

• engram chart: carry Qdrant snapshot S3 config (gates hl-3rr.10).
• Global Velero schedule remediation — the includedResources whitelist drops volume-snapshot data cluster-wide; the daily run is PartiallyFailed. Separate from the memory store, but the same class of latent-unrestorable-backup risk applies to every other stateful namespace.

Grounding references (recorded as bd notes on hl-73s.25)

• Repo: agent-memory qdrant/velero manifests (PVC, ports, image, collection).
• Live: daily-backup-20260602020054 status (snapshotMoveData:false, excluded snapshot resources, PartiallyFailed); no DataUpload/VolumeSnapshot present.
• context7 /llmstxt/qdrant_tech_llms-full_txt: snapshot create/recover/list/delete API.
• deepwiki qdrant/qdrant: s3_config fields, snapshots_path-as-prefix, direct S3 recover, S3 list/delete.