Skip to content

Qdrant Memory Backup & Restore Runbook

The curated memory store is a Qdrant server in the agent-memory namespace, on the PVC qdrant-data. The engram chart renders the server and reads the collection memory_v4. Two independent backup paths cover it, and this page holds the procedures to restore from either one into a scratch namespace. The design is in docs/engineering/specs/2026-06-02-qdrant-memory-backup-design.md and the decision in the ADR Use dual backup paths for the Qdrant memory store.

Primary: Qdrant snapshot DR: Velero datamover
Mechanism Qdrant native S3 snapshot storage Velero Schedule with snapshotMoveData: true
What it captures An app-consistent logical export of each listed collection A block-level copy of the whole qdrant-data PVC
Where The B2 bucket fzymgc-k8s-backups under memory/qdrant-snapshots, through the snapshots.s3 values of the engram chart The Velero backup location b2, the same bucket under velero/backups
Schedule CronJob qdrant-memory-snapshot at 30 2 * * *, newest 14 kept per collection Schedule agent-memory-pvc-backup at 0 4 * * *, ttl: 720h
Collections The COLLS list in the CronJob: memory_v3 and memory_v4. A listed collection absent at runtime is skipped Every collection on the PVC
Defined in argocd/cluster-app/templates/agent-memory.yaml (snapshots.s3), argocd/app-configs/agent-memory/qdrant-snapshot-secret.yaml and qdrant-snapshot-cronjob.yaml argocd/app-configs/velero/agent-memory-backup-schedule.yaml

The Qdrant snapshots use the B2 key in Vault at fzymgc-house/cluster/agent-memory/b2. That key is limited to the memory/ prefix. The Velero path uses the B2 key of its backup location, at fzymgc-house/cluster/velero/b2.

  • A snapshot stored on S3 reports checksum: null in the Qdrant snapshot list, so the recover API cannot verify integrity for it. Verify a restore by comparing the restored collection’s points_count and a known memory id against the live collection.
  • The COLLS list is a literal in the CronJob. When a new collection becomes the live one, add it to the list, or the live corpus has no snapshot.

The qdrant-memory-snapshot CronJob runs a curl and jq script against the in-cluster Qdrant REST API. It uses the self-owned image ghcr.io/fzymgc-house/k8s-utils, built for amd64 and arm64 by .github/workflows/build-k8s-utils-image.yaml from images/k8s-utils/ and pinned by short SHA (ADR Pin self-owned images by short SHA). The image installs no packages at runtime, because agent-memory pods have no egress to the Alpine package CDN. A bare alpine image with apk add fails every run.

To roll out a new image tag:

  1. Land a change under images/k8s-utils/ on main. The workflow publishes ghcr.io/fzymgc-house/k8s-utils:<short-sha>.

  2. Make sure that the GHCR package is public. A new package defaults to private. The CronJob also names the namespace ghcr-pull-secret, so a private package still pulls.

  3. Pin the new <short-sha> in argocd/app-configs/agent-memory/qdrant-snapshot-cronjob.yaml and commit. The CronJob has no spec.suspend field, so ArgoCD reconciles the new tag on the next sync and the job is active at once.

  4. After the sync, run a one-off job and read its log:

    Terminal window
    kubectl create job -n agent-memory --from=cronjob/qdrant-memory-snapshot snapshot-check
    kubectl -n agent-memory logs job/snapshot-check

Restore, path A: Qdrant snapshot (primary)

Section titled “Restore, path A: Qdrant snapshot (primary)”

Recovers one collection from a stored snapshot into a scratch namespace. Never target the live agent-memory namespace.

Terminal window
# 1. Pick the collection and the snapshot (the list comes from S3):
COLL=memory_v4
kubectl -n agent-memory exec deploy/qdrant -- \
sh -c "curl -s http://localhost:6333/collections/${COLL}/snapshots" | jq '.result[].name'
SNAP=<chosen-snapshot-name>
# 2. Scratch Qdrant on a fresh PVC, with the same image as the live server:
IMG=$(kubectl -n agent-memory get deploy qdrant -o jsonpath='{.spec.template.spec.containers[0].image}')
kubectl apply -f - <<YAML
apiVersion: v1
kind: Namespace
metadata: { name: agent-memory-restore-test }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: qdrant-restore, namespace: agent-memory-restore-test }
spec: { accessModes: [ReadWriteOnce], storageClassName: longhorn-encrypted, resources: { requests: { storage: 5Gi } } }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: qdrant-restore, namespace: agent-memory-restore-test }
spec:
replicas: 1
strategy: { type: Recreate }
selector: { matchLabels: { app: qdrant-restore } }
template:
metadata: { labels: { app: qdrant-restore } }
spec:
containers:
- name: qdrant
image: ${IMG}
ports: [{ containerPort: 6333, name: rest }]
volumeMounts: [{ name: data, mountPath: /qdrant/storage }]
volumes: [{ name: data, persistentVolumeClaim: { claimName: qdrant-restore } }]
YAML
kubectl -n agent-memory-restore-test wait --for=condition=Ready pod -l app=qdrant-restore --timeout=120s
# 3. Recover. The scratch server fetches the snapshot from the live server over cluster DNS:
POD=$(kubectl -n agent-memory-restore-test get pod -l app=qdrant-restore -o jsonpath='{.items[0].metadata.name}')
kubectl -n agent-memory-restore-test port-forward "pod/$POD" 16334:6333 &
LOC="http://qdrant.agent-memory.svc.cluster.local:6333/collections/${COLL}/snapshots/${SNAP}"
curl -X PUT "http://localhost:16334/collections/${COLL}/snapshots/recover" \
-H 'Content-Type: application/json' -d "{\"location\":\"${LOC}\",\"priority\":\"snapshot\"}"
# {"result":true,"status":"ok"}
# 4. Verify against live, then tear down:
curl -s "http://localhost:16334/collections/${COLL}" | jq '.result.points_count'
kubectl -n agent-memory exec deploy/qdrant -- sh -c "curl -s http://localhost:6333/collections/${COLL}" | jq '.result.points_count'
kubectl delete ns agent-memory-restore-test
Terminal window
# 1. Pick a completed backup, scheduled or on-demand:
kubectl -n velero get backups.velero.io -l velero.io/schedule-name=agent-memory-pvc-backup
BK=<backup-name>
# 2. Scratch namespace plus placeholder snapshot Secrets. The restored Qdrant Deployment
# references qdrant-snapshot-s3, or qdrant-r2-snapshot in a backup taken before the move
# to B2. Placeholder values let the pod start:
kubectl create ns agent-memory-velero-test
for s in qdrant-snapshot-s3 qdrant-r2-snapshot; do
kubectl -n agent-memory-velero-test create secret generic "$s" \
--from-literal=access_key=placeholder --from-literal=secret_key=placeholder
done
# 3. Restore into the mapped namespace, only what validates the volume:
kubectl apply -f - <<YAML
apiVersion: velero.io/v1
kind: Restore
metadata: { name: agent-memory-velero-test, namespace: velero }
spec:
backupName: ${BK}
includedNamespaces: [agent-memory]
namespaceMapping: { agent-memory: agent-memory-velero-test }
includedResources: [persistentvolumeclaims, persistentvolumes, deployments, services]
YAML
# 4. Wait, verify against live, tear down:
kubectl -n velero get restores.velero.io agent-memory-velero-test -o jsonpath='{.status.phase}' # Completed
POD=$(kubectl -n agent-memory-velero-test get pod -o name | grep qdrant | head -1)
kubectl -n agent-memory-velero-test port-forward "$POD" 16335:6333 &
curl -s http://localhost:16335/collections/memory_v4 | jq '.result.points_count'
kubectl delete ns agent-memory-velero-test
kubectl -n velero delete restore.velero.io agent-memory-velero-test

After any change to the engram chart values, the snapshot configuration or the Velero schedule, run either restore procedure into its scratch namespace. The restore passes when points_count equals the live value and a known memory id resolves:

Terminal window
ID=<a memory id from a recent engram store>
curl -s "http://localhost:16334/collections/memory_v4/points/${ID}" | jq '.status' # "ok"

Run the same check on a schedule as a backup-confidence check. Every rehearsal runs in a scratch namespace, so the live namespace is never touched.