Engram Cloud-Embedding Migration Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Move engram’s embedder off the contended seattle ollama lane (bge-m3, :11435) onto OpenRouter qwen/qwen3-embedding-8b (4096-dim) with ZDR, re-embedding the memory store via engram reindex into a fresh collection with a verified, zero-write-downtime cutover.

Architecture: Two GitOps PRs bracket one manual cluster operation. PR-A stands up a parallel cloud embed lane (new ZDR openrouter-embed backend + transitional route) without touching the live bge-m3 lane. A manual reindex runbook then populates a new 4096-dim memory_v2 Qdrant collection from the untouched memory source and verifies it. PR-B flips engram’s chart values to the new model/dim/collection, repoints the engram route at the cloud backend, and removes the ollama lane. The ordering is load-bearing: engram writes fail on dimension mismatch the instant dim=4096 meets the 1024-dim memory collection, so memory_v2 must be fully built and verified before any cutover.

Tech Stack: Kubernetes (ArgoCD GitOps), agentgateway CRDs (AgentgatewayBackend, AgentgatewayPolicy, Gateway API HTTPRoute), OpenRouter embeddings API (ZDR), engram v0.7.3 (engram reindex CLI), Qdrant.

Spec: docs/engineering/specs/2026-06-13-engram-cloud-embedding-migration-design.md Bead: hl-zs6l

Validation note (config, not code): These are GitOps manifests with no unit-test harness and no CI kustomize-lint gate. Per TDD’s documented config exception, each task’s verification is kubectl kustomize (renders without error) + yamllint (structural validity) + a targeted rg confirming the intended resource/field is present. True CRD-schema validation happens only at ArgoCD sync against the live cluster — called out as the post-merge gate where relevant.

PR boundaries: Tasks 1–2 = PR-A. Task 3 = the manual runbook (lands with PR-A; executed against the cluster only after PR-A merges + syncs). Tasks 4–6 = PR-B (authored after, and gated on, a verified memory_v2).

File structure

File	Responsibility	PR
`argocd/app-configs/agentgateway/llm-policies.yaml`	Add `openrouter-embed` ZDR backend (co-located with `openrouter-zdr`); edit `llm-apikey` + `llm-embeddings-retry` targetRefs	A, B
`argocd/app-configs/agentgateway/llm-routes.yaml`	Add transitional `llm-embeddings-engram-cloud` route (PR-A); repoint `llm-embeddings-engram` + delete transitional route (PR-B)	A, B
`docs/engineering/runbooks/2026-06-13-engram-reindex-qwen3.md`	Manual reindex runbook + one-shot Job manifest (NOT in any kustomization — must never auto-sync)	A
`argocd/app-configs/agentgateway/llm-backends.yaml`	Remove `ollama-engram` backend (PR-B)	B
`argocd/cluster-app/templates/agent-memory.yaml`	Flip engram `memory.embed.{model,dim}` + `memory.qdrant.collection` (PR-B)	B

Task 1: Add the `openrouter-embed` ZDR backend (PR-A)

Files:

Modify: argocd/app-configs/agentgateway/llm-policies.yaml (append a new backend block after the openrouter-zdr backend; note llm-policies.yaml intentionally co-locates the ZDR backend + its llm-chat-zdr HTTPRoute here rather than in llm-routes.yaml — follow that local convention)
Step 1: Add the backend manifest

Append this document (mirrors the live openrouter-zdr backend; routes + overrides under spec.policies.ai per the CRD — NOT under spec.ai):

---
# Dedicated ZDR embeddings backend for engram's interactive memory embeds.
# ZDR is backend-global (backend.ai.overrides has no per-model selector), so it
# is isolated to its own backend — same rationale as openrouter-zdr (hl-0sr.25.10).
# Single-model: pinning provider.openai.model collapses aliases to it (intentional).
# [hl-zs6l]
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openrouter-embed
  namespace: agentgateway
spec:
  ai:
    provider:
      openai:
        model: qwen/qwen3-embedding-8b
      host: openrouter.ai
      port: 443
      pathPrefix: /api/v1
  policies:
    auth:
      secretRef:
        name: agentgateway-openrouter
    tls: {}
    ai:
      modelAliases:
        qwen3-embedding-8b: qwen/qwen3-embedding-8b
      # Mark /v1/embeddings so the OpenAI-type backend does not reject it as a
      # chat request ("missing field messages"). Mirrors ollama-engram.
      routes:
        /v1/embeddings: Embeddings
      overrides:
        - field: provider
          value:
            zdr: true
            require_parameters: true

Step 2: Verify the kustomization renders

Run: kubectl kustomize argocd/app-configs/agentgateway > /dev/null && echo OK Expected: OK (no render error)

Step 3: Lint

Run: yamllint argocd/app-configs/agentgateway/llm-policies.yaml Expected: clean (exit 0)

Step 4: Confirm the resource is present and correctly nested

Run: kubectl kustomize argocd/app-configs/agentgateway | rg -A2 "name: openrouter-embed" and rg -n "routes:" argocd/app-configs/agentgateway/llm-policies.yaml Expected: backend openrouter-embed present; the new routes: sits under policies.ai (same indentation column as modelAliases/overrides), NOT under spec.ai.

Step 5: Commit

jj commit -m "feat(agentgateway): add openrouter-embed ZDR embeddings backend (qwen3-embedding-8b) [hl-zs6l]"

Task 2: Add the transitional cloud route + policy targetRefs (PR-A)

Files:

Modify: argocd/app-configs/agentgateway/llm-routes.yaml (append after the llm-embeddings-engram route)
Modify: argocd/app-configs/agentgateway/llm-policies.yaml (add the new route to llm-apikey and llm-embeddings-retry targetRefs)
Step 1: Add the transitional route

Append to llm-routes.yaml:

---
# Transitional cloud embed lane for the qwen3-embedding-8b migration [hl-zs6l].
# Runs PARALLEL to /engram-embed (ollama bge-m3) so the live server keeps writing
# to the 'memory' collection while `engram reindex` builds 'memory_v2' via this
# cloud lane. Removed in PR-B once /engram-embed itself is repointed at
# openrouter-embed. engram posts <url>/v1/embeddings; longest-prefix matches here.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-embeddings-engram-cloud
  namespace: agentgateway
spec:
  parentRefs:
    - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway, sectionName: llm-gw }
  hostnames: ["llm-gw.fzymgc.house"]
  rules:
    - matches:
        - { path: { type: PathPrefix, value: /engram-embed-cloud/v1/embeddings } }
      timeouts: { request: 60s, backendRequest: 45s }
      backendRefs:
        - { group: agentgateway.dev, kind: AgentgatewayBackend, name: openrouter-embed, weight: 1 }

Step 2: Add the route to the VK-auth gate (llm-apikey)

In llm-policies.yaml, in the llm-apikey policy’s targetRefs, add after the llm-chat-zdr line:

    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: llm-embeddings-engram-cloud }

Security-critical: without this, the cloud embed lane is unauthenticated for the entire PR-A window.

Step 3: Add the route to llm-embeddings-retry

In llm-policies.yaml, in the llm-embeddings-retry policy’s targetRefs, add:

    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: llm-embeddings-engram-cloud }

Step 4: Render + lint

Run: kubectl kustomize argocd/app-configs/agentgateway > /dev/null && echo OK Run: yamllint argocd/app-configs/agentgateway/llm-routes.yaml argocd/app-configs/agentgateway/llm-policies.yaml Expected: OK; lint clean.

Step 5: Confirm wiring

Run: rg -n "llm-embeddings-engram-cloud" argocd/app-configs/agentgateway/llm-policies.yaml Expected: TWO hits (one under llm-apikey, one under llm-embeddings-retry).

Step 6: Commit

jj commit -m "feat(agentgateway): transitional cloud embed route + VK-auth/retry wiring [hl-zs6l]"

PR-A boundary: open the PR with Tasks 1–2 (and Task 3’s runbook doc). After it merges and ArgoCD syncs, live-verify the cloud lane before proceeding: a probe POST through https://llm-gw.fzymgc.house/engram-embed-cloud/v1/embeddings (with engram’s VK) returns HTTP 200 and a 4096-length embedding vector. Do NOT author PR-B until this passes.

Task 3: Author the manual reindex runbook (PR-A, executed post-sync)

Files:

Create: docs/engineering/runbooks/2026-06-13-engram-reindex-qwen3.md

This artifact is operator documentation, not a synced manifest. It MUST NOT be added to any kustomization.yaml — an auto-synced Job would re-run reindex on every sync. It is applied once, by hand, with kubectl create -f, then deleted.

Step 1: Write the runbook

Create the file with this content:

# Runbook: engram reindex → qwen3-embedding-8b (memory_v2)

**Bead:** hl-zs6l · **Spec:** docs/engineering/specs/2026-06-13-engram-cloud-embedding-migration-design.md

**Preconditions:** PR-A merged + ArgoCD synced; the cloud probe (PR-A boundary)
returned a 4096-length vector. The live engram server is STILL on bge-m3/`memory`.

## 1. Capture the live Deployment env

The Job must share engram's exact connection config (Qdrant URL, OIDC, OTLP, VK
secret). Capture it so the Job matches the server:

```bash
kubectl get deploy engram -n agent-memory \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | jq .
kubectl get sa -n agent-memory          # confirm the engram ServiceAccount name
```

## 2. Apply the one-shot Job (dry-run first)

`reindex-job.yaml` — copy the Deployment env verbatim, override only the four
migration vars. `backoffLimit: 0` keeps it fail-closed.

```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: engram-reindex-memory-v2
  namespace: agent-memory
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: engram            # confirm via step 1
      imagePullSecrets: [{ name: ghcr-pull-secret }]
      containers:
        - name: reindex
          image: ghcr.io/seanb4t/engram:0.7.3
          command: ["engram", "reindex", "--target", "memory_v2", "--dry-run"]
          env:
            - { name: MEM_EMBED_MODEL,       value: "qwen3-embedding-8b" }
            - { name: MEM_EMBED_DIM,         value: "4096" }
            - { name: MEM_LITELLM_URL,       value: "https://llm-gw.fzymgc.house/engram-embed-cloud" }
            - { name: MEM_QDRANT_COLLECTION, value: "memory" }   # SOURCE (engram#117)
            # ...all other env copied from the Deployment (step 1): MEM_QDRANT_URL,
            #    the VK secret ref, OIDC, OTLP, etc.
```

```bash
kubectl create -f reindex-job.yaml
kubectl logs -n agent-memory job/engram-reindex-memory-v2 -f
# Expect: a point count matching the live 'memory' collection; NO writes.
kubectl delete -f reindex-job.yaml
```

## 3. Real run

Remove `--dry-run` from `command`, re-apply:

```bash
kubectl create -f reindex-job.yaml
kubectl logs -n agent-memory job/engram-reindex-memory-v2 -f
# Expect: memory_v2 created at dim 4096 / Cosine; all points re-embedded; exit 0.
kubectl delete -f reindex-job.yaml
```

## 4. Verify before cutover

- `memory_v2` point count == source `memory` count.
- Spot-check semantic search over `memory_v2` returns sensible neighbours for a
  known memory (via the engram UI/MCP, or a direct Qdrant query).
- `memory` is untouched (reindex never mutates the source).

Only after this passes: author PR-B (Tasks 4–6).

## Rollback

Nothing to undo — `memory_v2` is additive; `memory` is intact. If verification
fails, fix the cloud lane and re-run; do not proceed to PR-B.

Step 2: Lint

Run: rumdl check docs/engineering/runbooks/2026-06-13-engram-reindex-qwen3.md (or yamllint the embedded blocks if extracted) Expected: clean, or only style-level findings you then fix.

Step 3: Confirm it is NOT wired into any kustomization

Run: rg -n "reindex" argocd/ || echo "not referenced in argocd/ — correct" Expected: not referenced in argocd/ — correct

Step 4: Commit

jj commit -m "docs(engram): manual reindex runbook for qwen3 cutover (memory_v2) [hl-zs6l]"

Task 4: Flip engram chart values to qwen3 / 4096 / memory_v2 (PR-B)

GATE: Author this task only after the runbook’s §4 verification passed against the live cluster. PR-B merging is what triggers the dimension cutover.

Files:

Modify: argocd/cluster-app/templates/agent-memory.yaml (the memory.embed and memory.qdrant valuesObject keys; memory.litellm.url stays unchanged)
Step 1: Add the embed model + dim keys

The valuesObject does NOT currently set memory.embed — engram runs on the chart defaults (ollama/bge-m3 / 1024). Add a memory.embed block (a sibling of the existing memory.mcpPath / memory.litellm keys, same indentation):

            embed:
              model: "qwen3-embedding-8b"
              dim: 4096

Step 2: Add the collection override

memory.qdrant.collection is also unset (defaults to memory); the line-124 qdrant: block is the top-level Qdrant statefulset config, NOT memory.qdrant. Add a memory.qdrant block as a sibling of memory.embed/memory.litellm:

            qdrant:
              collection: "memory_v2"

Step 3: Update the explanatory comment

Replace the # Sends MEM_EMBED_MODEL=ollama/bge-m3 ... comment near litellm: to describe the cloud lane: model qwen3-embedding-8b (4096-dim) via the openrouter-embed ZDR backend; litellm.url unchanged (the /engram-embed route’s backend is repointed in Task 5); collection cut over to memory_v2.

Step 4: Render the template + confirm fields

The embed/qdrant keys are literal YAML inside the Application’s valuesObject (not Helm values from cluster-app/values.yaml, which is empty), so they appear verbatim in the render. Run both:

Run: helm template cluster-app argocd/cluster-app --show-only templates/agent-memory.yaml > /dev/null && echo OK (renders without Helm error) Run: rg -n "qwen3-embedding-8b|dim: 4096|collection: \"memory_v2\"" argocd/cluster-app/templates/agent-memory.yaml Expected: OK; all three fields present in the source.

Step 5: Lint

Run: yamllint argocd/cluster-app/templates/agent-memory.yaml Expected: clean.

Step 6: Commit

jj commit -m "feat(engram): cut embedder over to qwen3-embedding-8b/4096 on memory_v2 [hl-zs6l]"

Task 5: Repoint the engram route + remove the transitional route (PR-B)

Files:

Modify: argocd/app-configs/agentgateway/llm-routes.yaml (repoint llm-embeddings-engram backend; delete llm-embeddings-engram-cloud)
Modify: argocd/app-configs/agentgateway/llm-policies.yaml (remove llm-embeddings-engram-cloud from llm-apikey + llm-embeddings-retry)
Step 1: Repoint the live engram route to the cloud backend

In llm-routes.yaml, in the llm-embeddings-engram route’s backendRefs, change:

        - { group: agentgateway.dev, kind: AgentgatewayBackend, name: ollama-engram, weight: 1 }

to:

        - { group: agentgateway.dev, kind: AgentgatewayBackend, name: openrouter-embed, weight: 1 }

engram’s litellm.url (…/engram-embed) is unchanged — only what that path forwards to.

Step 2: Delete the transitional route

In llm-routes.yaml, remove the entire llm-embeddings-engram-cloud HTTPRoute document added in Task 2.

Step 3: Drop the transitional route from the policies

In llm-policies.yaml, remove the two llm-embeddings-engram-cloud targetRef lines (from llm-apikey and llm-embeddings-retry). Retain llm-embeddings-engram in llm-embeddings-retry — cloud OpenRouter can also emit transient 503s.

Step 4: Render + lint

Step 5: Confirm the transitional route is fully gone

Run: rg -n "llm-embeddings-engram-cloud|engram-embed-cloud" argocd/ || echo "clean" Expected: clean (no references anywhere).

Step 6: Commit

jj commit -m "feat(agentgateway): repoint engram embed route to openrouter-embed; drop transitional route [hl-zs6l]"

Task 6: Remove the `ollama-engram` backend (PR-B)

Files:

Modify: argocd/app-configs/agentgateway/llm-backends.yaml (remove the ollama-engram AgentgatewayBackend and its dedicated-lane comment block)
Step 1: Delete the backend

In llm-backends.yaml, remove the entire ollama-engram AgentgatewayBackend document (and its preceding # Dedicated embedding lane … comment). Leave the plain ollama backend (octopus’s bge-m3 lane) intact.

Step 2: Confirm nothing still references it

Run: rg -n "ollama-engram" argocd/ Expected: NO hits (Task 5 already repointed the route; this removes the last definition).

Step 3: Render + lint

Run: kubectl kustomize argocd/app-configs/agentgateway > /dev/null && echo OK Run: yamllint argocd/app-configs/agentgateway/llm-backends.yaml Expected: OK; clean.

Step 4: Commit

jj commit -m "chore(agentgateway): remove decommissioned ollama-engram backend [hl-zs6l]"

PR-B boundary: open the PR with Tasks 4–6. After merge + sync, verify: a new memory write succeeds (no dim-mismatch error), historical search returns results from memory_v2, engram logs show the cloud embedder, and no traffic hits seattle :11435.

Out of scope (follow-up beads)

Seattle :11435 launchd teardown (house.fzymgc.ollama-embed) — host-side; do NOT remove until after a cloud bake-in (it is the post-PR-B rollback target). File a follow-up bead.
Embedding-SPOF / in-cluster fallback — making OpenRouter the sole embed path is a deliberate accepted SPOF; tracked in hl-73s.23.

Post-merge verification (cluster, both PRs)

After PR-A sync: cloud route probe returns 4096-length vector; ZDR effective.
After reindex: memory_v2 count == memory count; search spot-check sane.
After PR-B sync: new write succeeds; historical search hits memory_v2; logs show cloud embedder; zero traffic to :11435. <!— adr-capture: sha256=fcbceea169319291; session=cli; ts=2026-06-13T19:15:36Z; adrs=hl-57x5 —>

Engram Cloud-Embedding Migration Implementation Plan