Qdrant Memory Backup & Restore Runbook
The curated memory store is a Qdrant server in the agent-memory namespace, on the PVC
qdrant-data. The engram chart renders the server and reads the collection memory_v4. Two
independent backup paths cover it, and this page holds the procedures to restore from either one
into a scratch namespace. The design is in
docs/engineering/specs/2026-06-02-qdrant-memory-backup-design.md and the decision in the ADR
Use dual backup paths for the Qdrant memory store.
Two independent backup paths
Section titled “Two independent backup paths”| Primary: Qdrant snapshot | DR: Velero datamover | |
|---|---|---|
| Mechanism | Qdrant native S3 snapshot storage | Velero Schedule with snapshotMoveData: true |
| What it captures | An app-consistent logical export of each listed collection | A block-level copy of the whole qdrant-data PVC |
| Where | The B2 bucket fzymgc-k8s-backups under memory/qdrant-snapshots, through the snapshots.s3 values of the engram chart |
The Velero backup location b2, the same bucket under velero/backups |
| Schedule | CronJob qdrant-memory-snapshot at 30 2 * * *, newest 14 kept per collection |
Schedule agent-memory-pvc-backup at 0 4 * * *, ttl: 720h |
| Collections | The COLLS list in the CronJob: memory_v3 and memory_v4. A listed collection absent at runtime is skipped |
Every collection on the PVC |
| Defined in | argocd/cluster-app/templates/agent-memory.yaml (snapshots.s3), argocd/app-configs/agent-memory/qdrant-snapshot-secret.yaml and qdrant-snapshot-cronjob.yaml |
argocd/app-configs/velero/agent-memory-backup-schedule.yaml |
The Qdrant snapshots use the B2 key in Vault at fzymgc-house/cluster/agent-memory/b2. That key
is limited to the memory/ prefix. The Velero path uses the B2 key of its backup location, at
fzymgc-house/cluster/velero/b2.
Known limitations
Section titled “Known limitations”- A snapshot stored on S3 reports
checksum: nullin the Qdrant snapshot list, so therecoverAPI cannot verify integrity for it. Verify a restore by comparing the restored collection’spoints_countand a known memory id against the live collection. - The
COLLSlist is a literal in the CronJob. When a new collection becomes the live one, add it to the list, or the live corpus has no snapshot.
Snapshot CronJob image and rollout
Section titled “Snapshot CronJob image and rollout”The qdrant-memory-snapshot CronJob runs a curl and jq script against the in-cluster Qdrant
REST API. It uses the self-owned image ghcr.io/fzymgc-house/k8s-utils, built for amd64 and arm64
by .github/workflows/build-k8s-utils-image.yaml from images/k8s-utils/ and pinned by short SHA
(ADR Pin self-owned images by short SHA).
The image installs no packages at runtime, because agent-memory pods have no egress to the
Alpine package CDN. A bare alpine image with apk add fails every run.
To roll out a new image tag:
-
Land a change under
images/k8s-utils/onmain. The workflow publishesghcr.io/fzymgc-house/k8s-utils:<short-sha>. -
Make sure that the GHCR package is public. A new package defaults to private. The CronJob also names the namespace
ghcr-pull-secret, so a private package still pulls. -
Pin the new
<short-sha>inargocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yamland commit. The CronJob has nospec.suspendfield, so ArgoCD reconciles the new tag on the next sync and the job is active at once. -
After the sync, run a one-off job and read its log:
Terminal window kubectl create job -n agent-memory --from=cronjob/qdrant-memory-snapshot snapshot-checkkubectl -n agent-memory logs job/snapshot-check
Restore, path A: Qdrant snapshot (primary)
Section titled “Restore, path A: Qdrant snapshot (primary)”Recovers one collection from a stored snapshot into a scratch namespace. Never target the live
agent-memory namespace.
# 1. Pick the collection and the snapshot (the list comes from S3):COLL=memory_v4kubectl -n agent-memory exec deploy/qdrant -- \ sh -c "curl -s http://localhost:6333/collections/${COLL}/snapshots" | jq '.result[].name'SNAP=<chosen-snapshot-name>
# 2. Scratch Qdrant on a fresh PVC, with the same image as the live server:IMG=$(kubectl -n agent-memory get deploy qdrant -o jsonpath='{.spec.template.spec.containers[0].image}')kubectl apply -f - <<YAMLapiVersion: v1kind: Namespacemetadata: { name: agent-memory-restore-test }---apiVersion: v1kind: PersistentVolumeClaimmetadata: { name: qdrant-restore, namespace: agent-memory-restore-test }spec: { accessModes: [ReadWriteOnce], storageClassName: longhorn-encrypted, resources: { requests: { storage: 5Gi } } }---apiVersion: apps/v1kind: Deploymentmetadata: { name: qdrant-restore, namespace: agent-memory-restore-test }spec: replicas: 1 strategy: { type: Recreate } selector: { matchLabels: { app: qdrant-restore } } template: metadata: { labels: { app: qdrant-restore } } spec: containers: - name: qdrant image: ${IMG} ports: [{ containerPort: 6333, name: rest }] volumeMounts: [{ name: data, mountPath: /qdrant/storage }] volumes: [{ name: data, persistentVolumeClaim: { claimName: qdrant-restore } }]YAMLkubectl -n agent-memory-restore-test wait --for=condition=Ready pod -l app=qdrant-restore --timeout=120s
# 3. Recover. The scratch server fetches the snapshot from the live server over cluster DNS:POD=$(kubectl -n agent-memory-restore-test get pod -l app=qdrant-restore -o jsonpath='{.items[0].metadata.name}')kubectl -n agent-memory-restore-test port-forward "pod/$POD" 16334:6333 &LOC="http://qdrant.agent-memory.svc.cluster.local:6333/collections/${COLL}/snapshots/${SNAP}"curl -X PUT "http://localhost:16334/collections/${COLL}/snapshots/recover" \ -H 'Content-Type: application/json' -d "{\"location\":\"${LOC}\",\"priority\":\"snapshot\"}"# {"result":true,"status":"ok"}
# 4. Verify against live, then tear down:curl -s "http://localhost:16334/collections/${COLL}" | jq '.result.points_count'kubectl -n agent-memory exec deploy/qdrant -- sh -c "curl -s http://localhost:6333/collections/${COLL}" | jq '.result.points_count'kubectl delete ns agent-memory-restore-testRestore, path B: Velero datamover (DR)
Section titled “Restore, path B: Velero datamover (DR)”# 1. Pick a completed backup, scheduled or on-demand:kubectl -n velero get backups.velero.io -l velero.io/schedule-name=agent-memory-pvc-backupBK=<backup-name>
# 2. Scratch namespace plus placeholder snapshot Secrets. The restored Qdrant Deployment# references qdrant-snapshot-s3, or qdrant-r2-snapshot in a backup taken before the move# to B2. Placeholder values let the pod start:kubectl create ns agent-memory-velero-testfor s in qdrant-snapshot-s3 qdrant-r2-snapshot; do kubectl -n agent-memory-velero-test create secret generic "$s" \ --from-literal=access_key=placeholder --from-literal=secret_key=placeholderdone
# 3. Restore into the mapped namespace, only what validates the volume:kubectl apply -f - <<YAMLapiVersion: velero.io/v1kind: Restoremetadata: { name: agent-memory-velero-test, namespace: velero }spec: backupName: ${BK} includedNamespaces: [agent-memory] namespaceMapping: { agent-memory: agent-memory-velero-test } includedResources: [persistentvolumeclaims, persistentvolumes, deployments, services]YAML
# 4. Wait, verify against live, tear down:kubectl -n velero get restores.velero.io agent-memory-velero-test -o jsonpath='{.status.phase}' # CompletedPOD=$(kubectl -n agent-memory-velero-test get pod -o name | grep qdrant | head -1)kubectl -n agent-memory-velero-test port-forward "$POD" 16335:6333 &curl -s http://localhost:16335/collections/memory_v4 | jq '.result.points_count'kubectl delete ns agent-memory-velero-testkubectl -n velero delete restore.velero.io agent-memory-velero-testVerify a restore
Section titled “Verify a restore”After any change to the engram chart values, the snapshot configuration or the Velero schedule,
run either restore procedure into its scratch namespace. The restore passes when points_count
equals the live value and a known memory id resolves:
ID=<a memory id from a recent engram store>curl -s "http://localhost:16334/collections/memory_v4/points/${ID}" | jq '.status' # "ok"Run the same check on a schedule as a backup-confidence check. Every rehearsal runs in a scratch namespace, so the live namespace is never touched.