Qdrant Memory Backup & Restore Runbook

Operational runbook for backing up and recovering the curated memory store (Qdrant collection memory, PVC qdrant-data, namespace agent-memory).

• Bead: hl-73s.25 · ADR: hl-cy3
• Design: docs/engineering/specs/2026-06-02-qdrant-memory-backup-design.md
• First verified: 2026-06-02 (both restore paths demonstrated)

Two independent backup paths

	Primary — Qdrant snapshot	DR — Velero datamover
Mechanism	Qdrant native S3 snapshot storage	Velero Schedule snapshotMoveData: true
What it captures	App-consistent logical export of the memory collection	Block-level copy of the whole qdrant-data PVC
Where	s3://fzymgc-cluster-storage/memory/qdrant-snapshots/ (R2)	Velero BSL default (R2, velero/backups)
Schedule	CronJob qdrant-memory-snapshot @ 30 2 * * *, keep 14	agent-memory-pvc-backup @ 0 4 * * *, ttl 30d
Size (typical)	~225 KB	~196 MB (full storage dir)
Config	argocd/app-configs/agent-memory/{qdrant.yaml,qdrant-snapshot-secret.yaml,qdrant-snapshot-cronjob.yaml}	argocd/app-configs/velero/agent-memory-backup-schedule.yaml

Both reuse the existing Cloudflare R2 credentials in Vault (fzymgc-house/cluster/velero/cloudflare-credentials).

Known limitations

• S3-stored snapshots report checksum: null in the Qdrant snapshot list, so the recover API’s optional checksum integrity field is not available for these snapshots. Integrity is instead verified by comparing the restored collection’s points_count and seeded-memory IDs against live (below).
• The Qdrant S3 snapshot config lives on the Qdrant Deployment env, which the engram cutover (hl-3rr.10) will replace. The engram chart MUST carry this env or recurring snapshots stop post-cutover (tracked as a blocking follow-up).
• Pod egress to telemetry.qdrant.io is blocked (harmless); egress to the R2 endpoint works (snapshots upload successfully).

Snapshot CronJob image & rollout (k8s-utils)

The qdrant-memory-snapshot CronJob runs a small curl+jq script against the in-cluster Qdrant REST API. It uses the self-owned image ghcr.io/fzymgc-house/k8s-utils (Alpine + curl/jq/ca-certificates/bind-tools/ netcat/ldns, non-root UID 65532). It does not install packages at runtime: agent-memory pods have no egress to the Alpine CDN, so a bare-alpine + apk add curl jq job fails every run (curl (no such package)). The image is built multi-arch (amd64 + arm64, for the ARM tpi-* nodes) by .github/workflows/build-k8s-utils-image.yaml and pinned by short-SHA. See ADR hl-pvn for the tag-pinning policy.

Rolling out a new image tag (the CronJob is GitOps-managed under an ArgoCD auto-sync + self-heal app, so a broken tag would deploy immediately):

1. Land a change under images/k8s-utils/** on main; the build workflow publishes ghcr.io/fzymgc-house/k8s-utils:<short-sha> (+ :latest).
2. Ensure the GHCR package is public (new packages default to private). The CronJob also references the namespace ghcr-pull-secret as a fallback, so a private package still pulls — but public is the intended state.
3. Pin the real <short-sha> in argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml and commit. The CronJob has no spec.suspend field, so it is active by default and ArgoCD reconciles the new tag on the next sync.
4. After ArgoCD syncs, trigger a one-off run to verify: kubectl create job -n agent-memory --from=cronjob/qdrant-memory-snapshot <name>.

First-time bootstrap only: the initial PR shipped the CronJob with spec.suspend: true because the image didn’t exist yet (and agent-memory auto-syncs, so an unbuildable tag would have deployed immediately). Once the image was published+public, a follow-up commit pinned the real SHA and removed the suspend field (CronJob defaults to active). Steady-state tag bumps (step 3 above) do not touch suspend — it is no longer present.

Restore — Path A: Qdrant snapshot (primary)

Recovers the memory collection from an R2 snapshot into an isolated scratch namespace. Never target live agent-memory.

# 1. Pick the snapshot to restore (lists from S3):
kubectl -n agent-memory exec deploy/qdrant -- \
  sh -c 'curl -s http://localhost:6333/collections/memory/snapshots' | jq '.result[].name'
SNAP=<chosen-snapshot-name>

# 2. Scratch Qdrant (fresh PVC, no S3 env needed — recover pulls over cluster DNS):
kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Namespace
metadata: { name: agent-memory-restore-test }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: qdrant-restore, namespace: agent-memory-restore-test }
spec: { accessModes: [ReadWriteOnce], storageClassName: longhorn-encrypted, resources: { requests: { storage: 5Gi } } }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: qdrant-restore, namespace: agent-memory-restore-test }
spec:
  replicas: 1
  strategy: { type: Recreate }
  selector: { matchLabels: { app: qdrant-restore } }
  template:
    metadata: { labels: { app: qdrant-restore } }
    spec:
      containers:
        - name: qdrant
          image: qdrant/qdrant:v1.18.1
          ports: [{ containerPort: 6333, name: rest }]
          volumeMounts: [{ name: data, mountPath: /qdrant/storage }]
      volumes: [{ name: data, persistentVolumeClaim: { claimName: qdrant-restore } }]
YAML
kubectl -n agent-memory-restore-test wait --for=condition=Ready pod -l app=qdrant-restore --timeout=120s

# 3. Recover (scratch Qdrant server-side-fetches the snapshot from production over cluster DNS):
POD=$(kubectl -n agent-memory-restore-test get pod -l app=qdrant-restore -o jsonpath='{.items[0].metadata.name}')
kubectl -n agent-memory-restore-test port-forward "pod/$POD" 16334:6333 &
LOC="http://qdrant.agent-memory.svc.cluster.local:6333/collections/memory/snapshots/${SNAP}"
curl -X PUT http://localhost:16334/collections/memory/snapshots/recover \
  -H 'Content-Type: application/json' -d "{\"location\":\"${LOC}\",\"priority\":\"snapshot\"}"
# → {"result":true,"status":"ok"}

# 4. Verify, then tear down:
curl -s http://localhost:16334/collections/memory | jq '.result.points_count'   # == live
kubectl delete ns agent-memory-restore-test

Restore — Path B: Velero datamover (DR)

# 1. Pick a completed backup (scheduled or on-demand):
kubectl -n velero get backups.velero.io -l velero.io/schedule-name=agent-memory-pvc-backup
BK=<backup-name>

# 2. Scratch ns + placeholder R2 Secret (the restored Qdrant Deployment references
#    secretKeyRef qdrant-r2-snapshot; dummy values let the pod start):
kubectl create ns agent-memory-velero-test
kubectl -n agent-memory-velero-test create secret generic qdrant-r2-snapshot \
  --from-literal=access_key=placeholder --from-literal=secret_key=placeholder

# 3. Restore (map the namespace; restore only what validates the volume):
kubectl apply -f - <<YAML
apiVersion: velero.io/v1
kind: Restore
metadata: { name: agent-memory-velero-test, namespace: velero }
spec:
  backupName: ${BK}
  includedNamespaces: [agent-memory]
  namespaceMapping: { agent-memory: agent-memory-velero-test }
  includedResources: [persistentvolumeclaims, persistentvolumes, deployments, services]
YAML

# 4. Wait, verify, tear down:
kubectl -n velero get restores.velero.io agent-memory-velero-test -o jsonpath='{.status.phase}'  # Completed
POD=$(kubectl -n agent-memory-velero-test get pod -l app.kubernetes.io/name=qdrant -o jsonpath='{.items[0].metadata.name}')
kubectl -n agent-memory-velero-test port-forward "pod/$POD" 16335:6333 &
curl -s http://localhost:16335/collections/memory | jq '.result.points_count'   # == live
kubectl delete ns agent-memory-velero-test
kubectl -n velero delete restore.velero.io agent-memory-velero-test

Verification evidence (2026-06-02)

First proof that the backups are restorable. Live collection at the time: 12 points, status green (includes seeded dogfood memories 9cc04339… the bake-off decision and 8b00b04a…).

Check	Result
Qdrant snapshot created to R2	memory-8071602537599694-2026-06-02-20-16-13.snapshot, 229,888 bytes
Snapshot listed back from S3	✓ (S3 round-trip confirmed)
Path A restore (scratch ns)	recover → result:true; 12 points, green; both seeded memories present
Velero backup (datamover)	agent-memory-manual-1 Completed (no errors/warnings); DataUpload 205,589,374 bytes to R2
Path B restore (scratch ns)	Restore Completed; PVC qdrant-data Bound; Qdrant Running; 12 points, green; both seeded memories present
Live agent-memory impact	none (all rehearsals in scratch namespaces)

Re-verification

Re-run either restore procedure above into a scratch namespace and confirm points_count and the two seeded memory IDs match live. Recommended after any change to the Qdrant chart/Deployment, the snapshot config, or the Velero schedule — and as a periodic backup-confidence check.