Qdrant Memory Backup + Verified Restore — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Give the curated Qdrant memory store a durable, app-consistent backup to R2 plus an independent Velero PVC backup, and demonstrate a restore of both — the gate that unblocks the engram cutover (hl-3rr.10).

Architecture: Qdrant is configured for native S3 snapshot storage (writes .snapshot files straight to Cloudflare R2 under memory/qdrant-snapshots/); a daily CronJob triggers and prunes snapshots via the Qdrant REST API. A dedicated Velero Schedule for the agent-memory namespace uses snapshotMoveData: true to offload the PVC bytes to R2 as a second, block-level recovery path. Both paths are then rehearsed into scratch namespaces against the two seeded dogfood memories.

Tech Stack: Kubernetes · ArgoCD (GitOps) · Qdrant v1.18.1 snapshot API (native S3) · External Secrets + Vault · Velero 12.0.1 (CSI datamover, node-agent) · Cloudflare R2 · kustomize.

Spec: docs/engineering/specs/2026-06-02-qdrant-memory-backup-design.md Design bead: hl-73s.25

Conventions for this plan

GitOps: Tasks 1–4 add/modify manifests under argocd/. They are NOT applied with kubectl; they land via a PR to main that ArgoCD syncs (Task 5). The repo forbids direct kubectl apply of managed manifests.
Validation gate per manifest task (replacing unit-test TDD for declarative YAML): kubectl kustomize <dir> must build, kubectl apply --dry-run=server -k <dir> must pass, and yamllint must be clean.
Out-of-band exception (Tasks 6): The restore rehearsals create ephemeral scratch namespaces and a Velero Restore via kubectl, then delete them. This is the documented exception to “ArgoCD manages deployments” — it is one-time verification, never touches agent-memory, and persists nothing in Git.
Cutover continuity: The R2 ExternalSecret (Task 1) and the snapshot CronJob (Task 3) live in the agent-memory kustomize path = Source B and MUST survive the cutover. The Qdrant Deployment env config (Task 2) is on a manifest the cutover deletes → it is the documented engram-chart follow-up (filed in Task 7), NOT carried by this plan past the pre-cutover proof.
Data safety: Task 2 restarts the Qdrant pod (strategy: Recreate) but only changes env — it never touches the qdrant-data PVC.
Commits: jj repo. Commit per task with a conventional message; co-author byline required. Push the branch and open one PR for Phase A (Tasks 1–5).

Files touched

File	Task	Action
`argocd/app-configs/agent-memory/qdrant-snapshot-secret.yaml`	1	Create
`argocd/app-configs/agent-memory/kustomization.yaml`	1, 3	Modify
`argocd/app-configs/agent-memory/qdrant.yaml`	2	Modify
`argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml`	3	Create
`argocd/app-configs/velero/agent-memory-backup-schedule.yaml`	4	Create
`argocd/app-configs/velero/kustomization.yaml`	4	Modify
`docs/operations/memory-backup-restore.md`	7	Create

Phase A — Backup infrastructure (one PR, GitOps)

Task 1: R2 credentials ExternalSecret

Qdrant needs R2 access/secret keys to write snapshots. Reuse the existing Velero R2 credentials in Vault (fzymgc-house/cluster/velero/cloudflare-credentials); the external-secrets-operator Vault policy already grants secret/data/* read, so no policy change is needed.

Files:

Create: argocd/app-configs/agent-memory/qdrant-snapshot-secret.yaml
Modify: argocd/app-configs/agent-memory/kustomization.yaml
Step 1: Create the ExternalSecret

argocd/app-configs/agent-memory/qdrant-snapshot-secret.yaml:

---
# R2 (Cloudflare) credentials for Qdrant native S3 snapshot storage.
# Reuses the Velero R2 keys (account-scoped) from Vault; the external-secrets
# ClusterSecretStore policy already grants secret/data/* read. The Secret keys
# back the QDRANT__STORAGE__SNAPSHOTS_CONFIG__S3_CONFIG__{ACCESS,SECRET}_KEY env.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: qdrant-r2-snapshot
  namespace: agent-memory
spec:
  refreshPolicy: Periodic
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: qdrant-r2-snapshot
    creationPolicy: Owner
    deletionPolicy: Delete
  data:
    - secretKey: access_key
      remoteRef:
        key: fzymgc-house/cluster/velero/cloudflare-credentials
        property: aws_access_key_id
    - secretKey: secret_key
      remoteRef:
        key: fzymgc-house/cluster/velero/cloudflare-credentials
        property: aws_secret_access_key

Step 2: Add it to the kustomization

Edit argocd/app-configs/agent-memory/kustomization.yaml. Insert one new line into resources: directly under the existing external-secret.yaml entry — do not reorder the other entries:

  - external-secret.yaml    # ollama-only LiteLLM key (embedder, local bge-m3)
  - qdrant-snapshot-secret.yaml  # R2 creds for Qdrant native S3 snapshots  ← NEW
  - ghcr-pull-secret.yaml   # pull creds for the private memory-mcp image
  - qdrant.yaml             # persistent encrypted vector store
  - deployment.yaml         # memory-mcp MCP server
  - service.yaml

(The ← NEW marker is for the reader; do not include it in the file.)

Step 3: Validate

Run: kubectl kustomize argocd/app-configs/agent-memory | yq 'select(.kind=="ExternalSecret" and .metadata.name=="qdrant-r2-snapshot")' Expected: the rendered ExternalSecret prints (proves it’s wired into the build). Run: yamllint argocd/app-configs/agent-memory/qdrant-snapshot-secret.yaml Expected: no errors.

Step 4: Commit

jj describe/commit with: feat(agent-memory): R2 credentials ExternalSecret for Qdrant snapshots (hl-73s.25)

Task 2: Qdrant Deployment — native S3 snapshot storage

Add the Qdrant snapshot env config. Env vars use the QDRANT__ prefix with __ nesting and override file config. snapshots_path doubles as the S3 key prefix when snapshots_storage=s3 (Qdrant has no s3_config.prefix), keeping memory snapshots under memory/qdrant-snapshots/ and clear of Velero’s velero/backups in the shared bucket.

Files:

Modify: argocd/app-configs/agent-memory/qdrant.yaml (the qdrant container in the Deployment)
Step 1: Add the env: block to the qdrant container

In argocd/app-configs/agent-memory/qdrant.yaml, add an env: list to the qdrant container, immediately after the ports: block and before volumeMounts::

          env:
            - name: QDRANT__STORAGE__SNAPSHOTS_PATH
              value: "memory/qdrant-snapshots"
            - name: QDRANT__STORAGE__SNAPSHOTS_CONFIG__SNAPSHOTS_STORAGE
              value: "s3"
            - name: QDRANT__STORAGE__SNAPSHOTS_CONFIG__S3_CONFIG__BUCKET
              value: "fzymgc-cluster-storage"
            - name: QDRANT__STORAGE__SNAPSHOTS_CONFIG__S3_CONFIG__REGION
              value: "auto"   # Cloudflare R2 ignores region; "auto" satisfies the S3 client
            - name: QDRANT__STORAGE__SNAPSHOTS_CONFIG__S3_CONFIG__ENDPOINT_URL
              value: "https://40753dbbbbd1540f02bd0707935ddb3f.r2.cloudflarestorage.com"
            - name: QDRANT__STORAGE__SNAPSHOTS_CONFIG__S3_CONFIG__ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: qdrant-r2-snapshot
                  key: access_key
            - name: QDRANT__STORAGE__SNAPSHOTS_CONFIG__S3_CONFIG__SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: qdrant-r2-snapshot
                  key: secret_key

Step 2: Validate

Run: kubectl kustomize argocd/app-configs/agent-memory | yq 'select(.kind=="Deployment" and .metadata.name=="qdrant") | .spec.template.spec.containers[0].env' Expected: the seven env vars print, with ACCESS_KEY/SECRET_KEY showing valueFrom.secretKeyRef.name: qdrant-r2-snapshot. Run: kubectl apply --dry-run=server -k argocd/app-configs/agent-memory Expected: all resources (server dry run) with no errors.

Step 3: Commit

feat(agent-memory): Qdrant native S3 snapshot storage to R2 (hl-73s.25)

Task 3: Daily snapshot create + prune CronJob

Triggers an app-consistent snapshot of the memory collection (Qdrant uploads it to R2) and prunes to the newest 14. Talks only to the Qdrant Service REST API, so it is workload-agnostic and survives the cutover in Source B.

Files:

Create: argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml
Modify: argocd/app-configs/agent-memory/kustomization.yaml
Step 1: Create the CronJob

argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml:

---
# Daily app-consistent snapshot of the `memory` collection. Qdrant (configured
# with native S3 snapshot storage) uploads the .snapshot to R2; this job only
# triggers creation and prunes to the newest 14. Workload-agnostic (REST only)
# so it is part of Source B and survives the engram cutover.
apiVersion: batch/v1
kind: CronJob
metadata:
  name: qdrant-memory-snapshot
  namespace: agent-memory
spec:
  schedule: "30 2 * * *"   # daily 02:30; offset from Velero's 02:00/03:00/04:00
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 1800
      template:
        spec:
          restartPolicy: Never
          automountServiceAccountToken: false
          securityContext:
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: snapshot
              image: alpine:3.21
              securityContext:
                # Runs as root (uid 0) because `apk add` needs write access to
                # install curl+jq; the job is short-lived with all caps dropped
                # and no privilege escalation. Root rootfs must stay writable for
                # apk, so readOnlyRootFilesystem is intentionally not set.
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"]
              resources:
                requests: { cpu: 25m, memory: 32Mi }
                limits: { cpu: 100m, memory: 64Mi }
              command: ["/bin/sh", "-c"]
              args:
                - |
                  set -eu
                  apk add --no-cache --quiet curl jq
                  QDRANT="http://qdrant.agent-memory.svc.cluster.local:6333"
                  COLL="memory"
                  KEEP=14
                  echo "Creating snapshot of collection ${COLL} ..."
                  curl -fsS -X POST "${QDRANT}/collections/${COLL}/snapshots?wait=true" | jq -e '.result.name'
                  echo "Pruning to newest ${KEEP} ..."
                  old=$(curl -fsS "${QDRANT}/collections/${COLL}/snapshots" \
                    | jq -r ".result | sort_by(.creation_time) | reverse | .[${KEEP}:] | .[].name")
                  for n in ${old}; do
                    echo "Deleting old snapshot ${n}"
                    curl -fsS -X DELETE "${QDRANT}/collections/${COLL}/snapshots/${n}?wait=true" | jq -e '.result'
                  done
                  echo "Snapshot cycle complete."

Image note: alpine:3.21 installs curl+jq at runtime as root (apk requires it), which needs egress to the Alpine CDN — this cluster’s pods have general egress (memory-mcp/qdrant pull images, ESO reaches Vault). If a future NetworkPolicy restricts that egress, the drop-in replacement is to make the job create-only (curlimages/curl pinned, nonroot, readOnlyRootFilesystem) and move retention to an R2 object-lifecycle rule on the memory/qdrant-snapshots/ prefix. For now retention stays in-job (no extra tf/R2 surface).

Step 2: Add it to the kustomization

Append to resources: in argocd/app-configs/agent-memory/kustomization.yaml:

  - qdrant-snapshot-cronjob.yaml  # daily app-consistent snapshot → R2 + prune

Step 3: Validate

Run: kubectl kustomize argocd/app-configs/agent-memory | yq 'select(.kind=="CronJob")' Expected: the CronJob renders with schedule: "30 2 * * *". Run: kubectl apply --dry-run=server -k argocd/app-configs/agent-memory Expected: no errors. Run: yamllint argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml Expected: no errors.

Step 4: Commit

feat(agent-memory): daily Qdrant snapshot+prune CronJob (hl-73s.25)

Task 4: Dedicated Velero PVC backup (DR path)

A Schedule separate from the broken global one: it does NOT use a restrictive includedResources whitelist (so the CSI snapshot machinery is captured) and sets snapshotMoveData: true so the datamover offloads the PVC bytes to R2 via the node-agent. It uses storageLocation: default, reusing the existing, already-working BackupStorageLocation (R2) credentials (cloud-credentials, key cloud).

Encryption note (corrected): the qdrant-data PVC is on longhorn-encrypted, but Longhorn does LUKS encryption at the CSI layer via the longhorn-crypto-config secret (wired as the node-stage/publish secret on the StorageClass — see tf/cluster-bootstrap/longhorn.tf:80-106). The volume is decrypted transparently at mount, so Velero’s datamover reads plaintext and needs no crypto key. Velero’s customer-key cred field is unrelated to Longhorn encryption (the BSL only consumes the cloud key). No extra credential wiring is required for the encrypted volume — storageLocation: default suffices because it is the same BSL every working backup already uses.

Files:

Create: argocd/app-configs/velero/agent-memory-backup-schedule.yaml
Modify: argocd/app-configs/velero/kustomization.yaml
Step 1: Create the Schedule

argocd/app-configs/velero/agent-memory-backup-schedule.yaml:

---
# Dedicated DR backup for the curated memory store. Unlike the global
# daily/weekly schedules (which take CSI snapshots but set snapshotMoveData:false
# and exclude the snapshot-reference resources, so PVC data never reaches R2),
# this one moves the data: snapshotMoveData:true → CSI datamover (node-agent)
# uploads the qdrant-data bytes to the default BSL (R2). No includedResources
# whitelist, so the volume-snapshot machinery is included.
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: agent-memory-pvc-backup
  namespace: velero
spec:
  schedule: "0 4 * * *"   # daily 04:00, clear of the global 02:00/03:00 runs
  template:
    includedNamespaces:
      - agent-memory
    snapshotMoveData: true
    defaultVolumesToFsBackup: false
    storageLocation: default
    ttl: 720h   # 30 days

Step 2: Add it to the kustomization

Edit argocd/app-configs/velero/kustomization.yaml, appending to resources::

resources:
  - namespace.yaml
  - rbac.yaml
  - secrets.yaml
  - backup-schedule.yaml
  - agent-memory-backup-schedule.yaml   # dedicated DR backup (snapshotMoveData)

Step 3: Validate

Run: kubectl kustomize argocd/app-configs/velero | yq 'select(.kind=="Schedule" and .metadata.name=="agent-memory-pvc-backup")' Expected: renders with snapshotMoveData: true and includedNamespaces: [agent-memory]. Run: kubectl apply --dry-run=server -k argocd/app-configs/velero Expected: no errors. Run: yamllint argocd/app-configs/velero/agent-memory-backup-schedule.yaml Expected: no errors.

Step 4: Commit

feat(velero): dedicated agent-memory PVC backup with snapshotMoveData (hl-73s.25)

Task 5: PR, merge, and confirm ArgoCD sync

Files: none (VCS + cluster reconcile)

Step 1: Push branch and open the PR

Push the qdrant-backup bookmark and open a PR to main. Title: feat(memory): backup the curated Qdrant store to R2 (snapshot + Velero DR) (hl-73s.25). Body summarizes the two backup paths and links the spec.

Step 2: Wait for CI (Validate Changes) green, then merge

Expected: rumdl/yamllint pass (the docs use ```text fences and blank-line-around-lists per the v0.1.15 ruleset). Merge.

Step 3: Watch ArgoCD reconcile agent-memory and velero

Run (read-only): confirm via the Kubernetes MCP / kubectl:

kubectl -n agent-memory get externalsecret qdrant-r2-snapshot     # SecretSynced=True
kubectl -n agent-memory get secret qdrant-r2-snapshot             # present, 2 keys
kubectl -n agent-memory get cronjob qdrant-memory-snapshot        # scheduled
kubectl -n agent-memory get pod -l app.kubernetes.io/name=qdrant  # Running, restarted with new env
kubectl -n velero get schedule agent-memory-pvc-backup            # present

Expected: ExternalSecret SecretSynced=True; Qdrant pod Running with the new env (kubectl -n agent-memory get pod <qdrant> -o yaml | rg SNAPSHOTS_CONFIG); memory-mcp reconnected (kubectl -n agent-memory logs deploy/memory-mcp | tail).

Step 4: Sanity-check Qdrant accepted the S3 config (no startup error)

Run: kubectl -n agent-memory logs deploy/qdrant | rg -i 'snapshot|s3|error' | tail -20 Expected: no S3/config error; Qdrant healthy on /readyz. If it logs an S3 auth/endpoint error, fix the env/secret before proceeding (do not continue to the gate on a misconfigured store).

Phase B — Prove it (the acceptance gate)

Task 6: Trigger both backups and rehearse both restores

This is the bead’s whole point: a backup is not a backup until a restore is demonstrated. All scratch resources are applied out-of-band and torn down.

Files: none persisted (ephemeral scratch manifests + a runbook in Task 7)

Step 1: Trigger the first Qdrant snapshot manually

Run: kubectl -n agent-memory create job --from=cronjob/qdrant-memory-snapshot qdrant-snapshot-manual-1 Then: kubectl -n agent-memory logs job/qdrant-snapshot-manual-1 -f Expected: logs print a snapshot name and “Snapshot cycle complete.”

Step 2: Confirm the snapshot object exists in R2

From a debug pod or the Qdrant API: kubectl -n agent-memory exec deploy/qdrant -- sh -c 'wget -qO- http://localhost:6333/collections/memory/snapshots' Expected: a JSON result array containing ≥1 snapshot with a name, creation_time, size, and checksum. Record the newest snapshot’s name and checksum. (Qdrant’s list reflects its view of S3; the R2 console/API is the authoritative store — the runbook in Task 7 should cross-check the object exists at s3://fzymgc-cluster-storage/memory/qdrant-snapshots/.)

Step 3: Trigger the dedicated Velero backup and confirm data offload

Run: kubectl -n velero create -f - with an on-demand Backup from the schedule:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: agent-memory-manual-1
  namespace: velero
spec:
  includedNamespaces: [agent-memory]
  snapshotMoveData: true
  storageLocation: default
  ttl: 168h0m0s

Wait, then verify the data actually moved (the thing the global schedule fails to do):

kubectl -n velero get backup agent-memory-manual-1 -o jsonpath='{.status.phase}'      # Completed
kubectl -n velero get datauploads -l velero.io/backup-name=agent-memory-manual-1      # ≥1, phase Completed

Expected: Backup Completed (not PartiallyFailed) and a DataUpload in phase Completed — proving the qdrant-data bytes reached R2.

Step 4: Rehearse the Qdrant-snapshot restore into a scratch namespace

Apply out-of-band (ephemeral): a scratch namespace + a throwaway Qdrant with its own empty PVC (no S3 env needed — recover pulls the snapshot from production Qdrant over cluster DNS, with checksum verification):

apiVersion: v1
kind: Namespace
metadata: { name: agent-memory-restore-test }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: qdrant-restore, namespace: agent-memory-restore-test }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-encrypted
  resources: { requests: { storage: 5Gi } }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: qdrant-restore, namespace: agent-memory-restore-test }
spec:
  replicas: 1
  strategy: { type: Recreate }
  selector: { matchLabels: { app: qdrant-restore } }
  template:
    metadata: { labels: { app: qdrant-restore } }
    spec:
      containers:
        - name: qdrant
          image: qdrant/qdrant:v1.18.1
          ports: [{ containerPort: 6333, name: rest }]
          volumeMounts: [{ name: data, mountPath: /qdrant/storage }]
      volumes:
        - name: data
          persistentVolumeClaim: { claimName: qdrant-restore }

Then recover into the scratch instance via the recover API, pointing location at production Qdrant’s snapshot download URL (resolvable over cluster DNS) and passing the checksum recorded in Step 2 for integrity verification. recover auto-creates the memory collection:

# SNAP and SUM = the snapshot name + checksum recorded in Step 2.
kubectl -n agent-memory-restore-test exec deploy/qdrant-restore -- \
  sh -c 'curl -fsS -X PUT http://localhost:6333/collections/memory/snapshots/recover \
    -H "Content-Type: application/json" \
    -d "{\"location\":\"http://qdrant.agent-memory.svc.cluster.local:6333/collections/memory/snapshots/'"${SNAP}"'\",\"priority\":\"snapshot\",\"checksum\":\"'"${SUM}"'\"}"'

Expected: {"result":true,"status":"ok",...}. A wrong/truncated snapshot is rejected by the checksum before any data is written — satisfying the spec’s integrity requirement. (scratch Qdrant reaches production Qdrant by Service DNS; no S3 credentials are placed in the scratch namespace.)

URL-recovery flag: recovering from an HTTP location relies on Qdrant’s service.enable_snapshot_url_recovery, which is enabled by default in v1.18.1 (neither Deployment sets a config file disabling it). If a future hardened config turns it off, fall back to the download-then-/upload path (multipart snapshot=@file, ?priority=snapshot) — note /upload carries no checksum, so verify the SHA256 manually before uploading.

Step 5: Assert the restored data is correct (Qdrant path)

# Live point count for comparison:
kubectl -n agent-memory exec deploy/qdrant -- \
  sh -c 'wget -qO- http://localhost:6333/collections/memory' | jq '.result.points_count'
# Restored point count:
kubectl -n agent-memory-restore-test exec deploy/qdrant-restore -- \
  sh -c 'curl -fsS http://localhost:6333/collections/memory' | jq '.result.points_count'
# Confirm the two seeded dogfood memories are present (UUID prefixes 9cc04339, 8b00b04a):
kubectl -n agent-memory-restore-test exec deploy/qdrant-restore -- \
  sh -c 'curl -fsS -X POST http://localhost:6333/collections/memory/points/scroll \
    -H "Content-Type: application/json" \
    -d "{\"limit\":1000,\"with_payload\":true}"' \
  | jq -r '.result.points[].id' | rg '^(9cc04339|8b00b04a)'

Expected: restored points_count == live points_count; the rg prints both seeded IDs. (If the full UUIDs are known, retrieve them directly via POST /collections/memory/points {"ids":[...]} instead of scrolling.)

Step 6: Rehearse the Velero restore into a second scratch namespace

First pre-create the scratch ns and a placeholder qdrant-r2-snapshot Secret. Task 2 added secretKeyRef: qdrant-r2-snapshot env to the Qdrant Deployment, so the restored Deployment’s pod would hit CreateContainerConfigError if that Secret is absent in the target namespace. The values are dummies — this rehearsal validates the restored PVC data, not S3 snapshotting, and Qdrant validates S3 config lazily (only on snapshot ops), so a placeholder lets the pod start:

kubectl create ns agent-memory-velero-test
kubectl -n agent-memory-velero-test create secret generic qdrant-r2-snapshot \
  --from-literal=access_key=placeholder --from-literal=secret_key=placeholder

Then apply out-of-band a Velero Restore mapping agent-memory into that ns (the restore adds resources to the existing namespace):

apiVersion: velero.io/v1
kind: Restore
metadata: { name: agent-memory-velero-test, namespace: velero }
spec:
  backupName: agent-memory-manual-1
  includedNamespaces: [agent-memory]
  namespaceMapping: { agent-memory: agent-memory-velero-test }
  # Restore only what we need to validate the volume; skip the app secrets/SA churn:
  includedResources: [persistentvolumeclaims, persistentvolumes, deployments, services]
  existingResourcePolicy: none

Wait for completion, then validate Qdrant comes up on the restored PVC:

kubectl -n velero get restore agent-memory-velero-test -o jsonpath='{.status.phase}'   # Completed
kubectl -n agent-memory-velero-test get pvc qdrant-data                                # Bound
kubectl -n agent-memory-velero-test rollout status deploy/qdrant --timeout=300s
kubectl -n agent-memory-velero-test exec deploy/qdrant -- \
  sh -c 'wget -qO- http://localhost:6333/collections/memory' | jq '.result.points_count'

Expected: Restore Completed, PVC Bound, Qdrant Running, and points_count matches live — proving the block-level DR path restores real data. (The restored Qdrant may carry the live S3 snapshot env; that is harmless in the scratch ns. If memory-mcp is pulled in and crashloops on OIDC/LiteLLM, ignore it — only Qdrant is under test.)

Step 7: Tear down all scratch resources

kubectl delete ns agent-memory-restore-test agent-memory-velero-test --wait
kubectl -n velero delete restore agent-memory-velero-test
kubectl -n agent-memory delete job qdrant-snapshot-manual-1

Expected: namespaces gone; nothing left in agent-memory beyond the managed workload. (Leave agent-memory-manual-1 Velero backup to expire by TTL, or delete it — it does not affect live data.)

Task 7: Runbook, follow-ups, and close the gate

Files:

Create: docs/operations/memory-backup-restore.md
Step 1: Write the runbook

Create docs/operations/memory-backup-restore.md documenting: the two backup mechanisms, where snapshots live (s3://fzymgc-cluster-storage/memory/qdrant-snapshots/), the schedule/retention, and the two restore procedures with the captured evidence from Task 6 (the snapshot name+checksum, the live-vs-restored points_count, the DataUpload Completed status, and confirmation the seeded memories were retrievable). A backup whose restore is documented + dated is the deliverable.

Step 2: Lint and commit the runbook

Run: rumdl check docs/operations/memory-backup-restore.md (use the CI-pinned v0.1.15 ruleset — ```text fences, blank lines around lists). Expected: no issues. Commit: docs(operations): Qdrant memory backup + restore runbook (hl-73s.25). Open/extend the PR and merge.

Step 3: File the cutover-continuity follow-up (gates hl-3rr.10)

The Qdrant S3 snapshot env config (Task 2) rides on the Deployment the cutover deletes. File a bead so the engram chart carries it:

bd create --type=task \
  --title="engram chart: carry Qdrant native-S3 snapshot config (post-cutover backup continuity)" \
  --description="The engram Helm chart's qdrant template must render the QDRANT__STORAGE__SNAPSHOTS_PATH + SNAPSHOTS_CONFIG__S3_CONFIG__* env (from values + the qdrant-r2-snapshot secret) so the daily snapshot backup keeps working after hl-3rr.10 replaces the kustomize Qdrant with the OCI chart. Without it the cutover silently drops the memory backup." \
  --priority=1 --labels="area:cluster,area:engram"
# then make it block the cutover:
bd dep add hl-3rr.10 --depends-on <new-bead-id>

Also note on hl-3rr.10 that its Source-B trim MUST keep qdrant-snapshot-secret.yaml and qdrant-snapshot-cronjob.yaml:

bd note hl-3rr.10 "Source-B trim: KEEP qdrant-snapshot-secret.yaml + qdrant-snapshot-cronjob.yaml (backup wiring, workload-agnostic) alongside the two ExternalSecrets. Only delete deployment.yaml/qdrant.yaml/service.yaml."

Step 4: File the global-Velero-schedule remediation follow-up

bd create --type=task \
  --title="Velero global schedule: stop discarding PVC snapshot data (snapshotMoveData + drop the includedResources whitelist)" \
  --description="The global daily/weekly Schedules in argocd/app-configs/velero/backup-schedule.yaml take CSI snapshots (csiVolumeSnapshotsCompleted>0) but set snapshotMoveData:false and exclude volumesnapshots/volumesnapshotcontents via the includedResources whitelist, so NO stateful PVC data reaches R2 — every stateful namespace is currently unrestorable from object storage, and runs land PartiallyFailed. Fix: enable snapshotMoveData (or per-PVC fs-backup) and stop whitelisting away the snapshot resources. Grounded in hl-73s.25." \
  --priority=1 --labels="area:cluster,area:velero,area:backup"

Step 5: Close the bead

Confirm the acceptance criteria are all met with captured evidence, then: bd close hl-73s.25 --reason="Both backup paths live; both restores demonstrated into scratch namespaces (seeded memories + point-count verified); runbook docs/operations/memory-backup-restore.md; cutover-continuity + global-velero follow-ups filed." This unblocks hl-3rr.10.

Self-review checklist (run before plan-reviewer)

Spec coverage: Component 1 → Task 2; Component 2 → Task 1; Component 3 → Task 3; Component 4 → Task 4; verification (both paths) → Task 6; runbook + follow-ups → Task 7.
No placeholders: all manifests are complete YAML; all commands have expected output.
Type/name consistency: Secret qdrant-r2-snapshot (keys access_key/secret_key) is created in Task 1 and consumed by the same name/keys in Task 2; collection memory, prefix memory/qdrant-snapshots, bucket fzymgc-cluster-storage consistent across Tasks 2–6.