Engram cloud-embedding migration (ollama bge-m3 → OpenRouter Qwen3-Embedding-8B, ZDR)

Engram cloud-embedding migration

Problem (authoritative)

engram’s embedder runs against the contended seattle ollama embed lane (:11435, bge-m3), inherited from the embedding-SPOF mitigation in hl-92xp. That lane still carries the local-embedder pain: GPU-share contention with octopus batch indexing on the sibling :11434, transient 30s timeouts, and an out-of-band launchd unit that lives outside the cluster’s lifecycle.

The goal: move engram’s embedder off local ollama onto OpenRouter with zero-data-retention (ZDR), then decommission the seattle embed lane for engram. A model decision (keep bge-m3 vs upgrade) was required and is resolved below.

Decision

Path B — upgrade to qwen/qwen3-embedding-8b (4096-dim) via OpenRouter ZDR.

Grounded rationale (live OpenRouter API + MTEB/MMTEB, 2026-06-13):

Model	MMTEB-R (retrieval)	$/1M tok (ZDR provider)	Native dim
qwen3-embedding-8b (chosen)	70.88	0.010 (Nebius/DeepInfra)	4096
gemini-embedding-001	67.71	0.150	3072
openai/text-embedding-3-large	59.27	0.130	3072
baai/bge-m3 (current)	54.60	0.010	1024

Retrieval-quality leader and cheapest simultaneously: +16.3 MMTEB-R points over today’s bge-m3; ahead of Gemini (+3.2) and far ahead of text-embedding-3-large (which is strictly dominated — more expensive than Qwen and lower quality).
Cost is a non-factor at engram’s scale. ~141 stored points; interactive, low-QPS writes. Even a wildly generous 1.5M tokens/month is <$0.02/mo on Qwen; the one-time re-embed is fractions of a cent.
The only Qwen-8B downside (heavier/slower serving) does not bite an interactive single-embed workload, and OpenRouter fans across three ZDR providers (Nebius, DeepInfra, SiliconFlow), so a single provider under load doesn’t tank the lane. Rejected gemini-embedding-001 (first-party stability but −3 retrieval points at 15× the still-negligible cost).

The 1024→4096 dimension change requires a re-embed, which is now a first-class operation: engram reindex shipped in v0.7.3 (engram#117/#123, deployed cluster-side via PR #1289). Qdrant vector size is immutable, so the re-embed populates a fresh memory_v2 collection from the untouched memory source, enabling a verified cutover.

Core hazard: dimension-mismatch write window

engram’s running server writes interactive embeds into its Qdrant collection at a fixed vector size. The instant its env says dim=4096 while it is still pointed at the 1024-dim memory collection, every memory write fails on dimension mismatch. Therefore the cutover order is load-bearing: memory_v2 must be fully populated and verified before the live server is ever flipped to it. The design below keeps the live bge-m3 lane serving writes throughout the reindex via a temporary parallel cloud route.

Architecture

Indirection already in place: engram posts to https://llm-gw.fzymgc.house/engram-embed/v1/embeddings; the agentgateway HTTPRoute llm-embeddings-engram longest-prefix-matches and forwards to the ollama-engram backend (seattle :11435). Swapping the embedder provider is a gateway backend/route change plus engram chart-value changes — engram’s litellm.url need not move.

ZDR pattern (proven for chat as openrouter-zdr): ZDR is backend-global (backend.ai.overrides has no per-model selector), so a dedicated openrouter-embed backend isolates ZDR + the embeddings path marker.

New gateway resources

openrouter-embed AgentgatewayBackend (mirrors openrouter-zdr, adapted for embeddings):

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openrouter-embed
  namespace: agentgateway
spec:
  ai:
    provider:
      openai:
        # Single-model embed backend: pinning `model` here collapses all aliases
        # to it (documented openrouter-backend behavior — see the comment on the
        # main `openrouter` backend). Intentional here: this backend only ever
        # serves qwen3-embedding-8b, so alias-bypass is harmless.
        model: qwen/qwen3-embedding-8b
      host: openrouter.ai
      port: 443
      pathPrefix: /api/v1
  policies:
    auth:
      secretRef:
        name: agentgateway-openrouter
    tls: {}
    ai:
      # NOTE: `modelAliases`, `routes`, and `overrides` all live under
      # `spec.policies.ai.*` per the AgentgatewayBackend CRD — mirrors the live
      # `ollama-engram` (routes) and `openrouter-zdr` (overrides) backends. Do
      # NOT place `routes` under `spec.ai` (it is silently ignored there).
      modelAliases:
        qwen3-embedding-8b: qwen/qwen3-embedding-8b
      # Mark the embeddings path so the OpenAI-type backend does not reject
      # /v1/embeddings as a chat request ("missing field messages").
      routes:
        /v1/embeddings: Embeddings
      overrides:
        - field: provider
          value:
            zdr: true
            require_parameters: true   # mirror openrouter-zdr ZDR contract

Transitional route llm-embeddings-engram-cloud (/engram-embed-cloud/v1/embeddings → openrouter-embed). The route MUST be added to three policies’ targetRefs (none auto-apply to new routes — they enumerate routes explicitly):

llm-apikey — security-critical: this is the Strict VK-auth gate (agentgateway-vkeys). Omitting it leaves the cloud embed lane open to any caller with gateway network access for the entire PR-A window.
llm-embeddings-retry — ride out transient upstream 503s.

It is already covered by the gateway-scoped llm-max-buffer (targets the Gateway, not per-route).

engram chart values (cutover, PR-B)

memory.embed.model: ollama/bge-m3 → qwen3-embedding-8b
memory.embed.dim: 1024 → 4096
memory.qdrant.collection: memory → memory_v2
memory.litellm.url: unchanged (…/engram-embed); the route’s backend is repointed instead
memory.litellm.keySecret: unchanged (same client virtual key)

Migration sequence

PR-A — stand up the cloud embed lane (non-disruptive)

Add openrouter-embed backend (above) + ZDR override.
Add transitional route llm-embeddings-engram-cloud (/engram-embed-cloud/v1/embeddings → openrouter-embed); add it to BOTH llm-apikey (VK auth — security-critical) AND llm-embeddings-retry targetRefs.
Leave the live /engram-embed → ollama-engram lane untouched. The running server keeps writing to memory (bge-m3, 1024) throughout.
Kustomization wiring; live-verify the cloud route returns a 4096-dim vector for a probe input (curl through the gateway with engram’s VK).

Migration step — `engram reindex` (manual runbook, verify-before-cutover)

Run a one-shot Job from the engram image (reindex-job.yaml, committed as a runbook template, applied deliberately with kubectl create -f, deleted after).

Manifest contract — the Job MUST share the running server’s exact connection config (Qdrant URL, OIDC, OTLP, VK secret), so derive its env from the engram Deployment rather than re-deriving it. Concretely:

apiVersion: batch/v1
kind: Job
metadata:
  name: engram-reindex-memory-v2
  namespace: agent-memory
spec:
  backoffLimit: 0          # fail-closed; do not silently retry a partial migration
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: engram          # same SA as the Deployment — VERIFY
                                          # the live name (kubectl get sa -n
                                          # agent-memory); it comes from the
                                          # external chart, not this repo
      imagePullSecrets: [{ name: ghcr-pull-secret }]
      containers:
        - name: reindex
          image: ghcr.io/seanb4t/engram:0.7.3   # lockstep with deployed chart
          command: ["engram", "reindex", "--target", "memory_v2"]  # add --dry-run for step 1
          # envFrom/env: copy the engram Deployment's container env verbatim,
          # then override ONLY these four for the migration:
          env:
            - { name: MEM_EMBED_MODEL,      value: "qwen3-embedding-8b" }
            - { name: MEM_EMBED_DIM,        value: "4096" }
            - { name: MEM_LITELLM_URL,      value: "https://llm-gw.fzymgc.house/engram-embed-cloud" }
            - { name: MEM_QDRANT_COLLECTION, value: "memory" }   # SOURCE (see contract above)
            # ...all other env (MEM_QDRANT_URL, VK secret ref, OIDC, OTLP) copied
            #    from the Deployment so the Job matches the server exactly.

The implementation plan should extract the Deployment’s full env via the rendered chart (or kubectl get deploy engram -n agent-memory -o jsonpath at runbook time) to avoid drift. Steps:

# 1. dry-run: count + sanity (writes nothing)
engram reindex --target memory_v2 --dry-run
# 2. populate memory_v2 (4096-dim, Cosine; source 'memory' untouched; fail-closed)
engram reindex --target memory_v2
# 3. verify search quality against memory_v2 before any cutover

Source/target contract (engram#117 / PR#123): engram reindex reads its source from the env-configured MEM_QDRANT_COLLECTION and writes to the --target collection — this is exactly why the documented operator flow sets MEM_QDRANT_COLLECTION=memory (source) before reindex and flips it to memory_v2 only at cutover. So in the Job env, MEM_QDRANT_COLLECTION=memory is load-bearing, not cosmetic. engram reindex re-embeds payload["content"] via the env-configured (cloud) embedder into the target collection (EnsureCollection provisions it at MEM_EMBED_DIM/Cosine), carrying raw Qdrant payloads verbatim (owner/metadata preserved so pre-isolation records aren’t dropped into the anonymous bucket), and aborts before any write on an embed error. Source memory is never mutated — intact for rollback.

PR-B — cut over + decommission

engram chart values: model/dim/collection as above (url unchanged).
Repoint the llm-embeddings-engram route backend ollama-engram → openrouter-embed; delete the transitional llm-embeddings-engram-cloud route. End state: a single /engram-embed route on the cloud backend.
Remove the ollama-engram AgentgatewayBackend. Retain the llm-embeddings-engram route in llm-embeddings-retry — cloud OpenRouter upstreams can also emit transient 503s, so the retry still earns its keep; only the backend (not the route’s retry membership) is removed.
Restart engram (chart change triggers it) → serves memory_v2 (4096).

Verification

PR-A: cloud route probe returns HTTP 200 + a 4096-length embedding; ZDR effective (request carries zdr: true; provider resolves to a ZDR endpoint).
Reindex: dry-run point count matches live memory count; post-run memory_v2 point count equals source; spot-check a known memory’s semantic search returns sensible neighbors.
PR-B: post-restart, a new memory write succeeds (no dim-mismatch error); search over historical memories returns results from memory_v2; engram logs show the cloud embedder in use; no traffic to seattle :11435.

Rollback

Before PR-B: nothing to roll back — live lane untouched; memory intact; memory_v2 is additive.
After PR-B (window where Sean accepted no instant fallback): revert PR-B (chart back to memory/bge-m3/1024, route back to ollama-engram, restore ollama-engram backend). memory is still present (reindex never deletes it). Seattle :11435 must still be running for this to serve — see decommission.

Out of scope (follow-ups)

Seattle :11435 launchd teardown (house.fzymgc.ollama-embed) — host-side, outside this repo. File as a follow-up bead; do not tear down until after a cloud bake-in (it is the post-PR-B rollback target).
Embedding SPOF / in-cluster fallback — making OpenRouter the sole embed path is a deliberate, accepted SPOF here; the fallback story is tracked in hl-73s.23.
Per-key allowedModels governance (hl-y3v8) — unrelated.

Risks

Risk	Mitigation
Dimension-mismatch write failures during migration	Temp parallel cloud route keeps live bge-m3 lane serving until verified cutover (core design)
ZDR not actually applied upstream	Dedicated ZDR backend with `zdr: true` override; verify provider resolves to a ZDR endpoint before reindex
OpenRouter becomes embed SPOF	Accepted; OpenRouter fans across 3 ZDR providers; fallback tracked in hl-73s.23
Reindex re-fires / partial write	Manual one-shot Job (not auto-sync hook); fail-closed CLI; source collection never mutated
Qwen3-8B serving latency spikes	Interactive low-QPS workload tolerant; multi-provider routing; gateway retry policy covers transient 503

<!— adr-capture: sha256=3ac012d294d2b415; session=cli; ts=2026-06-13T19:15:36Z; adrs=hl-57x5 —>

Engram cloud-embedding migration (ollama bge-m3 → OpenRouter Qwen3-Embedding-8B, ZDR)