engram 0.8.0 owner-claim upgrade + memory restoration

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Upgrade engram from 0.7.16 → 0.8.0 so memory ownership keys on a stable email claim instead of the rotating OIDC sub, then re-stamp the operator’s pre-cutover (Authentik-era) and post-cutover (Keycloak UUID) records onto the new email owner so the full memory history is visible again under the Keycloak identity.

Architecture: engram 0.8.0 (PR #248, epic engram-8bsz, ADR engram-g37x supersedes engram-hvg) derives a record’s owner authz key from ENGRAM_OWNER_CLAIM (default email) and ships a migrate-remap-owner CLI sweep. The deployed chart is a multi-source ArgoCD Application (argocd/cluster-app/templates/agent-memory.yaml): Source A renders the engram OCI chart (chart+image lockstep), Source B holds cluster wiring. The upgrade is a targetRevision/image.tag bump plus an explicit ownerClaim value, landed in one PR. Restoration runs the migrate-remap-owner subcommand twice as one-shot Kubernetes Jobs (the runtime pod is distroless — no exec) against the live memory_v2 Qdrant collection.

Tech Stack: engram 0.8.0, Helm (OCI chart ghcr.io/seanb4t/charts/engram), ArgoCD multi-source Application, Qdrant v1.18 (points/scroll for owner discovery — owner is unindexed so the Facet API does not work), Keycloak realm fzymgc (mcp-public bearer), jj VCS.

---

Critical context & invariants

• The lockout gate (do NOT skip Task 1). With ownerClaim=email, engram fails closed: a token lacking email, or lacking email_verified: true as a JSON boolean, is rejected (401) on every request. engram already logs actor=sean@fuzzymagic.com (actor = email-preferred), so the email claim is confirmed present in the mcp-public access token; the only unverified gate is email_verified on the manually-created Keycloak user sean.
• Two remaps, not one. Post-upgrade, even the post-cutover records become invisible — they carry owner=ab096608-3991-4465-9bb8-d1fe640650ac (Keycloak UUID sub, 7 records as of discovery), but auth now reads email. Restoration re-stamps both the old Authentik hash and the UUID sub → sean@fuzzymagic.com.
• The old owner hash is now known (Task 3, done): 442692a783c6351dd1a2a06cb3d6e9f818e7b2c1a4c5bc3c8ba6bb17e7ef65c2 — an Authentik hashed_user_id (SHA-256), 441 records (the back-catalog). Total collection = 448 points, two owners, both the operator’s.
• RemapOwner is auth-independent. It re-stamps the owner payload string collection-wide via StoreFromEnv() with no token; it can run before or after the server flips to email. Order chosen below deploys first (proves no lockout on new writes), then backfills.
• StoreFromEnv() calls loadAndValidate() — the migration Job must carry the storage + embedder config the validator checks (ENGRAM_QDRANT_ADDR, ENGRAM_QDRANT_COLLECTION, ENGRAM_OPENAI_BASE_URL, ENGRAM_OPENAI_API_KEY, ENGRAM_EMBED_MODEL, ENGRAM_EMBED_DIM). OIDC env is not needed for the sweep.
• Lockstep PR. The chart bump and ownerClaim value ride one PR / one ArgoCD revision (same discipline as the mcp-engram-oauth issuer cutover).
• Live cluster ops require operator kubectl. The agent’s Kubernetes MCP is read-only (no exec/port-forward); Tasks 1, 3, 5, 6 are operator-run kubectl steps. The agent owns the Git/chart change (Task 4) and the Job manifests.
• Artifacts are published (verified 2026-06-29): crane manifest ghcr.io/seanb4t/engram:0.8.0 and helm show chart oci://ghcr.io/seanb4t/charts/engram --version 0.8.0 both resolve.

Files touched

• Modify: argocd/cluster-app/templates/agent-memory.yaml — Source A targetRevision 0.7.16 → 0.8.0, image.tag 0.7.16 → 0.8.0, add memory.oidc.ownerClaim: email, refresh the chart-changelog comment block.
• Create: scratchpad/engram-remap-job.yaml (operator-run, NOT committed) — one-shot migration Job template.
• Reference (no change): argocd/app-configs/agentgateway/mcp-engram.yaml (auth lanes), tf/keycloak/mcp_public.tf, tf/keycloak/users.tf.

Grounding traces (Rule 7)

engram is not indexed by deepwiki and is not a context7 library, so the API surface was grounded directly against the v0.8.0 source (gh api repos/seanb4t/engram/contents/<path>?ref=v0.8.0) and the live OCI registry. Verified:

• migrate-remap-owner is a single top-level command — cmd/engram/migrate.go: migrateRemapOwnerCmd has Use: "migrate-remap-owner" and is attached via rootCmd.AddCommand(migrateRemapOwnerCmd) (NOT a migrate parent + remap-owner child). Flags: --from (string) | --from-missing (bool) | --from-anon (bool), --to (string, required), --dry-run (bool), --timeout. The subcommand is migrate-remap-owner, but the image ENTRYPOINT is ["/engram","serve"], so the Job MUST set command: ["/engram","migrate-remap-owner"] and put only the flags in args (else they append to serve → unknown flag: --from; verified live 2026-06-29).
• StoreFromEnv() env set — internal/server/tools.go: StoreFromEnv() → loadAndValidate() → ensureStoreFromConfig(). The validated config covers the storage + embedder data plane; the six ENGRAM_* vars in the Job (Task 5) mirror the live memory-mcp deployment’s storage/embedder env. OIDC vars are not consulted by the sweep.
• Chart value + env wiring — charts/engram/values.yaml adds memory.oidc.ownerClaim (default "" ⇒ engram defaults to email); charts/engram/templates/memory-mcp.yaml gates {{- with .Values.memory.oidc.ownerClaim }} → ENGRAM_OWNER_CLAIM.
• Fail-closed contract — docs-site/src/content/docs/reference/auth.md (v0.8.0): owner claim default email; absent claim ⇒ 401; when claim is email, email_verified must be JSON boolean true.
• Live identity — engram pod logs (memory-mcp, 2026-06-29): owner=ab096608-3991-4465-9bb8-d1fe640650ac actor=sean@fuzzymagic.com (confirms email is in the token; sole active owner is the Keycloak UUID).
• Artifacts — crane manifest ghcr.io/seanb4t/engram:0.8.0 resolves; helm show chart … --version 0.8.0 → appVersion: 0.8.0.

(Recorded as grounding/ notes on bead hl-w5kl.)

Bead materialization labels (Rule 5)

Task	model	Rationale
1 Lockout gate	model:sonnet	Auth reasoning + conditional remediation; lockout blast radius
2 Artifact check	model:haiku	Mechanical verification commands
3 Discovery	model:haiku	Single read-only scroll query (already done)
4 Chart bump PR	model:sonnet	Targeted YAML edit with a load-bearing additive invariant
5 Remaps	model:sonnet	Data migration; dry-run/count reconciliation before writes
6 Verify + close	model:haiku	Observational verification

---

Task 1: Verify the email_verified lockout gate (BLOCKING)

Files: none (operator-run live verification).

This task is a hard gate. If it fails and is not remediated, deploying Task 4 locks the operator out of engram entirely (401 on every request).

• Step 1: Read the Keycloak user’s emailVerified attribute

Run (operator workstation, authenticated to Keycloak admin):

kcadm.sh get users -r fzymgc -q username=sean --fields id,username,email,emailVerified

Expected: a single user object. Inspect emailVerified.

• "emailVerified" : true → gate PASSES, continue to Step 3.
• "emailVerified" : false (or absent) → gate FAILS, go to Step 2.
• Step 2: Set emailVerified=true (only if Step 1 failed)

Run:

kcadm.sh update users/<user-id> -r fzymgc -s emailVerified=true

Expected: no error. Re-run Step 1 and confirm "emailVerified" : true.

Note: user sean is manually created / data-sourced (tf/keycloak/users.tf); full provisioning is Phase 4 of the Authentik→Keycloak migration (hl-jb6q / hl-12i8). Codifying this attribute is out of scope here — record it as a Phase 4 follow-up so it is not lost on a realm rebuild.

• Step 3: Confirm email + email_verified land in the mcp-public ACCESS token

Decode a live mcp-public access token (e.g. from a Claude Code engram session, or mint one via the device/PKCE flow) and inspect the payload:

echo "<access-token>" | cut -d. -f2 | base64 -d 2>/dev/null | jq '{email, email_verified, aud, iss, sub}'

Expected:

{
  "email": "sean@fuzzymagic.com",
  "email_verified": true,
  "aud": "mcp-public",
  "iss": "https://id.fzymgc.house/realms/fzymgc",
  "sub": "ab096608-3991-4465-9bb8-d1fe640650ac"
}

email_verified MUST be the JSON boolean true (not the string "true"). If email or email_verified is missing from the access token (it may be in the ID token only), add the built-in email client scope as a default scope on mcp-public and confirm its email/email_verified mappers have add.to.access.token = true. Do not proceed to Task 4 until this output is exactly as above.

• Step 4: Record the gate result

Run:

bd note hl-w5kl "Task1 gate: emailVerified=true confirmed; mcp-public access token carries email + email_verified:true (boolean). Safe to flip ownerClaim=email."

---

Task 2: Confirm the 0.8.0 artifacts and review the chart delta

Files: none (verification).

• Step 1: Confirm image + chart resolve

Run:

crane manifest ghcr.io/seanb4t/engram:0.8.0 >/dev/null && echo "image OK"
helm show chart oci://ghcr.io/seanb4t/charts/engram --version 0.8.0 | grep -E '^(version|appVersion):'

Expected:

image OK
appVersion: 0.8.0
version: 0.8.0

• Step 2: Diff the chart values surface 0.7.16 → 0.8.0

Run:

diff <(helm show values oci://ghcr.io/seanb4t/charts/engram --version 0.7.16) \
     <(helm show values oci://ghcr.io/seanb4t/charts/engram --version 0.8.0)

Expected: the only functional addition is memory.oidc.ownerClaim: "" (plus its comment). Confirm no other breaking value-path renames in range (0.7.17 and 0.8.0 only). If anything else changed, stop and reconcile before Task 4.

---

Task 3: Discover the old owner hash + record counts (read-only) — ✅ DONE 2026-06-29

Files: none (read-only against Qdrant).

Result (already executed this session): the memory_v2 collection holds 448 points across exactly two owners, both the operator’s — no anonymous bucket:

owner	identity	records	remap role
442692a783c6351dd1a2a06cb3d6e9f818e7b2c1a4c5bc3c8ba6bb17e7ef65c2	old Authentik hashed_user_id (SHA-256)	441	Task 5 remap #1 --from
ab096608-3991-4465-9bb8-d1fe640650ac	Keycloak UUID (current sub)	7	Task 5 remap #2 --from

These are the two --from values for Task 5. Counts are the Task 5 dry-run reconciliation targets (the UUID count may tick up if more memories are written before the remap — reconcile against a fresh count, not the literal 7).

Method (reproducible — scroll, NOT facet). Qdrant’s Facet API requires a payload index on the key; owner is unindexed, so /collections/memory_v2/facet returns No appropriate index for faceting: owner. Use points/scroll (reads payloads directly, works on any field):

kubectl -n agent-memory port-forward svc/qdrant 6333:6333 &   # read-only; kill when done
curl -s localhost:6333/collections/memory_v2/points/scroll \
  -H 'content-type: application/json' \
  -d '{"limit":1000,"with_payload":["owner"],"with_vector":false}' \
  | jq -r '.result.points[].payload.owner // "<<ownerless>>"' \
  | sort | uniq -c | sort -rn

Expected: 441 442692a7…, 7 ab096608-…. (The collection has 448 points; a single 1000-limit scroll covers it with no pagination. If a future run exceeds 1000 points, follow .result.next_page_offset.)

Recorded on the bead: bd note hl-w5kl "DISCOVERY (live qdrant scroll …): old hash=442692a7… =441; UUID ab096608… =7 …".

---

Task 4: Bump the chart to 0.8.0 and set ownerClaim=email (PR)

Files:

• Modify: argocd/cluster-app/templates/agent-memory.yaml

• Step 1: Create an isolated workspace off latest main

This repo uses jj (colocated). Mutating git commands are blocked by a guard hook — use jj:

jj git fetch
jj new main@origin

• Step 2: Edit the Application source

In argocd/cluster-app/templates/agent-memory.yaml:

1. Source A targetRevision: 0.7.16 → targetRevision: 0.8.0.
2. image.tag: "0.7.16" → image.tag: "0.8.0".
3. Under memory.oidc, add the single new key ownerClaim: email — a comment explains the hl-w5kl fix.
4. Replace the stale # 0.7.3 → 0.7.16 changelog comment with a 0.7.16 → 0.8.0 summary noting: 0.7.17 (additive) and the 0.8.0 BREAKING owner-claim change + required migrate-remap-owner runs (cross-ref hl-w5kl, ADR engram-g37x).

This is an ADDITIVE one-key insert, NOT a block replacement. The live oidc: block (argocd/cluster-app/templates/agent-memory.yaml:171-186) has four existing keys — issuer, audience, clientID: engram-ui, and clientSecret.{name,key} — that drive the engram-ui console OAuth flow. They MUST be preserved. Add only the ownerClaim line. The full block AFTER the edit (all keys retained, ownerClaim added, existing comments abbreviated here for brevity — keep the originals in place):

            oidc:
              # ENGRAM_OWNER_CLAIM (engram 0.8.0, PR #248, ADR engram-g37x): key
              # per-record ownership on the stable `email` claim instead of the
              # OIDC `sub`. Fixes hl-w5kl — the Authentik→Keycloak cutover rotated
              # `sub`, stranding the back-catalog. FAIL-CLOSED: every mcp-public
              # token MUST carry email + email_verified:true (boolean) or engram
              # 401s every request (verified in plan Task 1). Pre-0.8.0 records
              # are re-stamped onto email by migrate-remap-owner (plan Task 5).
              ownerClaim: email                                   # <-- ONLY new line
              issuer: "https://id.fzymgc.house/realms/fzymgc"     # existing — keep
              audience: mcp-public                                # existing — keep
              clientID: engram-ui                                 # existing — keep (UI console)
              clientSecret:                                       # existing — keep (UI console)
                name: engram-ui-oidc
                key: oidc_client_secret

(Set ownerClaim explicitly rather than relying on the chart default "" → email, so the cluster’s intent is self-documenting and a future chart default change can’t silently move it.)

• Step 2a: Verify the engram-ui credentials survived the edit

A dropped clientID/clientSecret renders cleanly and lints cleanly (UI mode is optional in the chart), so the Step 3/4 checks below will NOT catch it. Guard explicitly with two offset-independent whole-file scans:

rg -c '^\s*clientID: engram-ui$' argocd/cluster-app/templates/agent-memory.yaml   # expect 1
rg -c '^\s*name: engram-ui-oidc$' argocd/cluster-app/templates/agent-memory.yaml  # expect 2

Expected: 1 then 2. The first asserts the oidc.clientID line survived; the second asserts the clientSecret.name reference (the oidc block’s engram-ui-oidc Secret ref appears here and in the ui.cookieKeySecret block, so 2 is correct). Anchored to line start (^\s*…$) so they match the YAML keys, not the surrounding comment prose. If either count is low, you dropped a UI credential — restore the full block before continuing.

• Step 3: Render the chart locally to confirm the env var appears

Run:

helm template engram oci://ghcr.io/seanb4t/charts/engram --version 0.8.0 \
  --set memory.oidc.ownerClaim=email \
  | rg -A1 'ENGRAM_OWNER_CLAIM'

Expected:

- { name: ENGRAM_OWNER_CLAIM, value: "email" }

• Step 4: Lint the changed manifest

Run:

yamllint argocd/cluster-app/templates/agent-memory.yaml

Expected: no errors (Helm-templated {{ }} lines are pre-existing; confirm no new findings).

• Step 5: Commit and open the PR

jj describe -m "feat(engram): upgrade 0.8.0 — owner claim=email + restoration [hl-w5kl]"
jj bookmark create feat/engram-080-owner-claim -r @
jj git push -b feat/engram-080-owner-claim
gh pr create --head feat/engram-080-owner-claim \
  --title "feat(engram): upgrade 0.8.0 — configurable owner claim (email) [hl-w5kl]" \
  --body "Bumps engram chart+image 0.7.16→0.8.0 and sets ownerClaim=email (PR seanb4t/engram#248, ADR engram-g37x). Fixes hl-w5kl owner-continuity. Restoration (migrate-remap-owner) runs post-merge per docs/engineering/plans/2026-06-29-engram-080-owner-claim-restoration.md. Lockout gate verified (Task 1)."

• Step 6: Merge, then watch ArgoCD reconcile

After merge to main, confirm ArgoCD syncs the new revision and the memory-mcp pod restarts on 0.8.0:

kubectl -n agent-memory get deploy memory-mcp -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n agent-memory get pods -l app.kubernetes.io/name=memory-mcp

Expected: image ghcr.io/seanb4t/engram:0.8.0, pod Running and Ready.

• Step 7: Smoke-test the new-write path (no lockout)

From a Claude Code engram session, call store_memory then list_memory. Expected: the call succeeds (no 401) and the new record is owned by sean@fuzzymagic.com. Confirm in logs:

kubectl -n agent-memory logs deploy/memory-mcp --tail=20 | rg 'owner=sean@fuzzymagic.com'

If any request 401s here, the lockout gate (Task 1) regressed — roll back the PR and re-verify the token before retrying.

---

Task 5: Re-stamp old records onto the email owner (migrate-remap-owner)

Files:

• Create: scratchpad/engram-remap-job.yaml (operator-run, not committed)

Run the sweep as a one-shot Job (distroless runtime image, same 0.8.0 tag). The Job mirrors the storage + embedder env so StoreFromEnv()/loadAndValidate() succeed. Run dry-run first, confirm counts match Task 3, then run for real. Repeat for both source owners.

• Step 1: Write the Job template

Create scratchpad/engram-remap-job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: engram-remap          # rename per invocation (see steps below)
  namespace: agent-memory
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 600
  template:
    spec:
      restartPolicy: Never
      imagePullSecrets:
        - name: ghcr-pull-secret
      containers:
        - name: remap
          image: ghcr.io/seanb4t/engram:0.8.0
          # The image ENTRYPOINT is ["/engram","serve"]. A Job that sets only
          # `args` appends them to `serve` → `unknown flag: --from`. MUST override
          # `command` so the subcommand replaces `serve`. (Verified live 2026-06-29.)
          command: ["/engram", "migrate-remap-owner"]
          args: []            # flags only, set per invocation (see steps below)
          env:
            - { name: ENGRAM_QDRANT_ADDR, value: "qdrant.agent-memory.svc.cluster.local:6334" }
            - { name: ENGRAM_QDRANT_COLLECTION, value: "memory_v2" }
            - { name: ENGRAM_OPENAI_BASE_URL, value: "https://llm-gw.fzymgc.house/engram" }
            - { name: ENGRAM_EMBED_MODEL, value: "qwen3-embedding-8b" }
            - { name: ENGRAM_EMBED_DIM, value: "4096" }
            - name: ENGRAM_OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: memory-mcp-litellm
                  key: litellm_ollama_key

• Step 2: Dry-run remap #1 — back-catalog (old Authentik hash → email)

Set metadata.name: engram-remap-old-dryrun and (flags only — the subcommand is in command; old Authentik hash from Task 3): args: ["--from","442692a783c6351dd1a2a06cb3d6e9f818e7b2c1a4c5bc3c8ba6bb17e7ef65c2","--to","sean@fuzzymagic.com","--dry-run"] then:

kubectl apply -f scratchpad/engram-remap-job.yaml
kubectl -n agent-memory logs job/engram-remap-old-dryrun -f

Expected: [dry-run] would remap 441 record(s) to owner=sean@fuzzymagic.com (the Task 3 back-catalog count). If the count disagrees, stop and investigate before any real write.

• Step 3: Real remap #1 — back-catalog

Edit the Job: metadata.name: engram-remap-old, drop --dry-run from args. Delete the dry-run Job, apply, and watch:

kubectl -n agent-memory delete job engram-remap-old-dryrun
kubectl apply -f scratchpad/engram-remap-job.yaml
kubectl -n agent-memory logs job/engram-remap-old -f

Expected: remapped 441 record(s) to owner=sean@fuzzymagic.com.

• Step 4: Dry-run + real remap #2 — post-cutover (Keycloak UUID → email)

Repeat Steps 2–3 with metadata.name: engram-remap-uuid[-dryrun] and --from ab096608-3991-4465-9bb8-d1fe640650ac --to sean@fuzzymagic.com.

Expected dry-run: would remap 7 record(s); real: remapped 7 record(s). (7 as of discovery — if agents wrote more memories between discovery and the remap, the count is higher; reconcile the dry-run against a fresh Task 3 scroll, don’t hard-expect 7.)

• Step 5: Confirm the collection now has a single owner for the operator

Re-run the Task 3 scroll aggregate (NOT facet — owner is unindexed). Expected: the old hash and the UUID are gone; sean@fuzzymagic.com carries all 448 records (any genuinely anonymous <<ownerless>> bucket, if present, is left untouched — that is correct).

curl -s localhost:6333/collections/memory_v2/points/scroll \
  -H 'content-type: application/json' \
  -d '{"limit":1000,"with_payload":["owner"],"with_vector":false}' \
  | jq -r '.result.points[].payload.owner // "<<ownerless>>"' | sort | uniq -c | sort -rn

Expected: a single line 448 sean@fuzzymagic.com.

• Step 6: Clean up the Job + scratch file

kubectl -n agent-memory delete job engram-remap-old engram-remap-uuid --ignore-not-found

(The scratchpad/ file is session-local and uncommitted; no Git action needed.)

---

Task 6: Verify restoration end-to-end and close out

Files: none.

• Step 1: Recall the back-catalog under the new identity

From a fresh Claude Code engram session, run list_memory for the repo spine (scope repo:github.com/fzymgc-house/selfhosted-cluster).

Expected: the full pre-cutover history is visible (~441 back-catalog + 7 post-cutover ≈ 448 records), not just the handful of post-cutover records. Spot-check a known pre-migration memory by content.

• Step 2: Web-console re-login + owner-parity check

Open https://engram.fzymgc.house/ui. Expected: a one-time forced re-login (old sub-keyed cookies are invalidated by 0.8.0). After login, confirm the console shows the same restored records as the agent lane.

Risk to watch: the console lane authenticates via the Authentik engram-ui issuer (ENGRAM_UI_ISSUER), which must emit email + email_verified:true AND assert the same email (sean@fuzzymagic.com) as the Keycloak bearer lane. If Authentik asserts a different address (e.g. sean@fzymgc.email), the console keys to a different owner and shows a different (or empty) set. If observed, file a follow-up to align the engram-ui email (or migrate the UI lane to Keycloak in Phase 4/5) — do not attempt it inline here.

• Step 3: Close the bug

bd note hl-w5kl "Restored: migrate-remap-owner re-stamped 441 back-catalog + 7 post-cutover records → sean@fuzzymagic.com on engram 0.8.0 (ownerClaim=email). Recall + console verified."
bd close hl-w5kl --reason="Fixed by engram 0.8.0 owner-claim upgrade + migrate-remap-owner restoration"

• Step 4: File the Phase 4 follow-up (if Task 1 Step 2 set emailVerified)

If emailVerified was set live in Task 1, create a bead to codify it in Keycloak user provisioning (relates hl-jb6q / hl-12i8) so a realm rebuild cannot silently re-break the gate.

---

Open risks (carry into review)

1. Lockout (Task 1). Mitigated by the blocking pre-verification; the Task 4 Step 7 smoke test is the second backstop with PR rollback as the escape hatch.
2. UI/agent owner divergence (Task 6 Step 2). The Authentik UI lane email may not match the Keycloak bearer email; flagged for a Phase 4/5 follow-up, not blocking the agent-memory restoration that hl-w5kl is about.
3. Other scopes/collections. The chart configures a single collection (memory_v2), and the Task 3 scroll confirmed all 448 points (memories + any discoveries, which share the collection) carry one of the two known owners — there is no second collection or stray owner to chase. The remap is collection-wide, so both scopes and discoveries are covered in one sweep.
4. Non-transactional sweep. RemapOwner is idempotent and the count is a best-effort snapshot before an exact filtered SetPayload; re-running a remap is safe (the --from == --to guard fast-fails a no-op).