Extract the memory layer to `seanb4t/engram` — Design

Design bead: hl-3rr Date: 2026-06-01 Status: READY (design-reviewer round 3) — next: writing-plans

Goal

Relocate the self-hosted memory layer — the Go MCP server, its deployment chart, and CI — out of fzymgc-house/selfhosted-cluster into a new standalone repository seanb4t/engram, so the product has a single coherent home, while the cluster-specific wiring stays in selfhosted-cluster. The running deployment’s curated memories (the Qdrant PVC) must survive the cutover, and the relocation must not introduce skill/server contract drift.

Scope

In scope (this effort):

Stand up seanb4t/engram (private repo, engineered to Apache-2.0 / open-source standards) containing the Go server, a generic Helm chart, CI, and docs.
Convert the current kustomize manifests to a parameterized Helm chart.
CI that builds/publishes the container image and the OCI Helm chart on tag.
Cut the running cluster over to consume the OCI chart, adopting the existing Qdrant PVC (no data loss).
Adopt the applicable conventions from the holomush project (github.com/holomush/holomush; local checkout ~/code/github.com/holomush/holomush).

Out of scope (separate future efforts):

The curator skill — its implementation, its home in engram, and the fzymgc-house-skills#131 redirect. A skill/ slot may be reserved in the repo layout, but no skill work happens here.
The Tier-0/Tier-1 productionization roadmap beads (hl-73s.23–.29). They re-target engram but are tracked and executed separately.
Renaming the Kubernetes namespace (agent-memory stays — see Cutover).
Git history migration (fresh repo; history stays in selfhosted-cluster).
holomush tooling that does not apply to engram: buf/protobuf/api, database migrate, generate:schema.

Boundary — moves vs. stays

Concern	Destination
Go server (`main`, tools, `internal/{auth,store,embed}`)	→ engram
Helm chart (Qdrant + memory-mcp)	→ engram (new, from kustomize)
CI (image + OCI chart release)	→ engram
Contract docs (tool schemas, scope convention, auth model, config reference)	→ engram/docs
Authentik OAuth app (`tf/authentik/memory_mcp_oauth.tf`)	stays in selfhosted-cluster
LiteLLM `memory_oauth` route (litellm configmap)	stays
ExternalSecrets (`memory-mcp-litellm`, `ghcr-pull-secret`) + Vault paths	stays — kept as the Application’s Source B (trimmed `argocd/app-configs/agent-memory`, ExternalSecrets only)
ArgoCD `Application` (`cluster-app/templates/agent-memory.yaml`) + cluster values	stays — re-pointed to multi-source (OCI chart + Source B)
New: ArgoCD OCI repo credential `ghcr-seanb4t-charts` (mirrors `ghcr-fzymgc-house-charts`)	stays (Vault → ExternalSecret, `shared-resources/`)

Rationale: engram is the portable product; the Authentik app, Vault secrets, LiteLLM route, and ArgoCD wiring are environment specifics that belong with the cluster. This is the separation that makes engram reusable and keeps homelab details out of an open-source-bound repo.

Repo layout (`seanb4t/engram`)

Module github.com/seanb4t/engram at the repo root (flat — not nested under a server/ subdir); binary under cmd/ per holomush convention.

engram/
  go.mod  go.sum                       module github.com/seanb4t/engram
  cmd/engram/main.go                   thin entrypoint (wire server + auth + serve)
  internal/
    auth/      OIDC bearer-token verifier (go-oidc) + go-sdk middleware glue
    store/     Qdrant-backed store (Memory type incl. actor)
    embed/     LiteLLM-routed embedder
    server/    tool registration (moved from the current flat tools.go)
  Dockerfile
  charts/engram/
    Chart.yaml  values.yaml          # version == appVersion == image tag (lockstep)
    templates/  qdrant {Deployment,Service,PVC}, memory-mcp {Deployment,Service}
    # memory-mcp Deployment MUST template reloader.stakater.com/auto podAnnotation
    # (the memory-mcp-litellm secret refreshes every 5m; Reloader picks it up)
  docs/                                contract: tools, scope convention, auth, config reference
  .github/workflows/  ci.yaml  release.yaml
  Taskfile.yaml
  .goreleaser.yaml  .golangci.yaml  lefthook.yaml  cog.toml
  dprint.json  .yamlfmt  .markdownlint.yaml (or .rumdl.toml)  .editorconfig
  .licenserc.yaml  LICENSE (Apache-2.0)  LICENSE_HEADER
  CLAUDE.md  AGENTS.md  CONTRIBUTING.md  README.md
  # skill/  — reserved slot; OUT of scope this effort

The current server is a near drop-in: internal/{auth,store,embed} move verbatim (only the module prefix in import paths changes); the flat main.go/tools.go split into cmd/engram/main.go + internal/server.

Conventions (adopted from holomush)

Task (Taskfile.yaml) is the runner. Mirror the applicable namespaced targets: build / build:all, test / test:coverage / test:short / test:integration, lint / lint:{go,markdown,yaml,actions}, fmt / fmt:{go,dprint,yaml,check}, license:{check,add}, deps, release:{check,snapshot}, hooks:install, tools, clean. Add engram-specific: chart:lint, chart:push.
goreleaser (.goreleaser.yaml) builds the multi-arch (incl. arm64 — the cluster is all arm64) image and the GitHub release.
golangci-lint (.golangci.yaml), lefthook (git hooks), cocogitto (cog.toml, conventional commits — matches selfhosted-cluster), formatting via dprint / yamlfmt / markdown lint, .editorconfig.
License headers via .licenserc.yaml + LICENSE_HEADER (Apache-2.0), enforced by task license:check in CI.
Repo meta: CLAUDE.md / AGENTS.md / CONTRIBUTING.md / README.md.
Config files are copied/adapted from holomush during implementation, trimmed of holomush-specific concerns (no proto/buf, migrate, or schema).

Release flow

On git tag vX.Y.Z (.github/workflows/release.yaml):

goreleaser release → build/push multi-arch image ghcr.io/seanb4t/engram:X.Y.Z (+ :latest) and the GitHub release.
task chart:push → helm package charts/engram --version X.Y.Z --app-version X.Y.Z then helm push → oci://ghcr.io/seanb4t/charts/engram:X.Y.Z. The chart’s image.tag defaults to its appVersion, so image and chart move in lockstep on one tag.

On PR/push (ci.yaml): task lint, task test, task fmt:check, task license:check, task chart:lint. ghcr auth uses the workflow GITHUB_TOKEN with packages: write. The package is private initially.

This closes the current gap where the image is built manually.

Deployment cutover (data-safe)

Two invariants:

PVC identity. The chart MUST render the PVC as qdrant-data in namespace agent-memory (and keep the qdrant/memory-mcp resource names). ArgoCD reconciles by kind + name + namespace, so flipping the agent-memory Application’s workload source from the kustomize path to the OCI chart adopts the live PVC rather than delete-and-recreate; with prune: true, a resource still rendered under the same name is never pruned (verified against argoproj/argo-cd). Namespace stays agent-memory — renaming = a real data migration.
ExternalSecrets must keep being rendered. The live agent-memory Application is multi-source (sources:), and its kustomize path renders the cluster ExternalSecrets (memory-mcp-litellm, ghcr-pull-secret) alongside the workloads. Those hold Vault refs and do NOT belong in the generic chart. So the cutover stays multi-source, mirroring the existing cluster-app/templates/temporal-workers.yaml (OCI chart + inline valuesObject + reloader podAnnotation):
- Source A — OCI chart: repoURL: ghcr.io/seanb4t/charts, chart: engram, targetRevision: X.Y.Z, helm.valuesObject carrying the cluster config (OIDC issuer, LiteLLM URL, secret refs, storage class, resources, reloader.stakater.com/auto podAnnotation, imagePullSecrets: [ghcr-pull-secret]). Renders the workloads (qdrant + memory-mcp + PVC).
- Source B — trimmed kustomize: argocd/app-configs/agent-memory, reduced to only the ExternalSecrets (external-secret.yaml, ghcr-pull-secret.yaml, kustomization.yaml); the now-chart-owned deployment.yaml/qdrant.yaml/service.yaml are deleted.
Because the ExternalSecrets stay rendered by Source B, prune: true does not delete them. The Application’s existing ignoreDifferences block (ExternalSecret conversionStrategy / decodingStrategy / metadataPolicy) carries forward unchanged — without it, ArgoCD reports perpetual drift on Source B’s secrets and self-heal churns.

Credentials — pre-flight, grounded against existing cluster patterns:

OCI chart pull (ArgoCD): add a repository ExternalSecret ghcr-seanb4t-charts in argocd/app-configs/shared-resources/, mirroring the existing ghcr-fzymgc-house-charts (type: helm, enableOCI: "true", url: ghcr.io/seanb4t/charts), reusing the Vault PAT at secret/fzymgc-house/cluster/ghcr/pull-secret.
Image pull (kubelet): the existing ghcr-pull-secret (Source B) already authenticates ghcr.io with that PAT. Verify the PAT is account-scoped (sean’s, read:packages) so it covers seanb4t/engram; if it is a fine-grained token limited to fzymgc-house, extend it or add a token. No new image-pull secret otherwise. Concrete pre-flight test: with the PAT, docker pull ghcr.io/seanb4t/engram:0.2.0 and helm pull oci://ghcr.io/seanb4t/charts/engram --version 0.2.0 must both succeed before wiring the secrets.

Sequence:

hl-73s.25 backup + verified restore first — the net before any GitOps change.
Stand up engram (no cluster impact): copy server (incl. internal/auth, present since #1137), write chart + CI + conventions, tag v0.2.0 → image ghcr.io/seanb4t/engram:0.2.0 + OCI chart …/charts/engram:0.2.0 published.
Pre-flight credentials: add ghcr-seanb4t-charts; confirm ArgoCD resolves the chart and the PAT covers seanb4t packages — before any source change (bad credential blocks sync, does not lose data).
Trim Source B + add Source A: delete the workload manifests from argocd/app-configs/agent-memory (keep only ExternalSecrets) and edit cluster-app/templates/agent-memory.yaml to the multi-source form above. (The Application already uses sources: with one entry — this adds the second; no singular→plural migration is needed.) Gate the sync on a server-side dry-run (kubectl diff --server-side / ArgoCD --dry-run=server) showing: PVC and ExternalSecrets adopted/kept, nothing pruned, and no SSA field-manager conflict on the PVC. Sync, then verify list_memory returns the pre-existing memories.
Cleanup (only after step 3 verified healthy): remove services/memory-mcp/ from selfhosted-cluster. The agent-memory app-config dir is kept (trimmed) as Source B.

Rollback: until step 4, restore the workload manifests and revert the template to the single kustomize source — re-adopts the same resources. Phase-0 backup is the ultimate net.

ServerSideApply note: the Application sets ServerSideApply=true. Adoption transfers field-manager ownership from ArgoCD’s kustomize apply to the Helm-rendered apply; the PVC spec is identical so no field-level conflict is expected — but the server-side dry-run in step 3 is what confirms this rather than assuming it.

Validation / testing

task test / task lint / task fmt:check / task license:check green in CI.
helm template charts/engram (with cluster-like values) renders a PVC named qdrant-data and qdrant/memory-mcp resources with the current names.
Pre-cutover (server-side dry-run, the gate): kubectl diff --server-side / ArgoCD --dry-run=server shows the PVC and both ExternalSecrets adopted/kept, nothing pruned, and no SSA field-manager conflict on the PVC. The OCI repo credential resolves and the chart pulls.
Post-cutover: the prior memories are retrievable via list_memory/ search_memory; the memory-mcp-litellm secret still exists (pod not crash-looping); OAuth enforcement still returns 401 for an unauthenticated call (no regression in the auth posture).
Restore rehearsal from Phase 0 as the documented safety net.

Risks

PVC pruned/deleted during cutover → mitigated by the name-match invariant, the server-side dry-run gate, the rollback target, and the Phase-0 backup.
ExternalSecrets pruned on cutover (the memory-mcp-litellm secret deleted → pod crash-loop) → mitigated by the multi-source topology (ExternalSecrets stay rendered by Source B) + the dry-run gate confirming nothing is pruned.
SSA field-manager conflict adopting kustomize-applied resources under ServerSideApply=true → surfaced (not assumed away) by the server-side dry-run; PVC spec is unchanged so none is expected.
ghcr PAT scope doesn’t cover seanb4t packages → pre-flight verification in step 2 (image pull + OCI chart pull both use it).
ArgoCD cannot pull the private OCI chart → app stuck (not data loss); pre-flight the ghcr-seanb4t-charts credential in step 2.
Image arch mismatch → goreleaser must build linux/arm64 (the cluster is all arm64).
Contract drift → mitigated by co-locating the contract docs with the server in engram (the original motivation); the skill, when later built in engram, reads that single source.

Decisions made (for the record)

Repo private now, engineered to Apache-2.0 standards; flip to public later is a visibility change, not a cleanup.
No git history migration (fresh repo; avoids leaking homelab history into an open-source-bound repo).
OCI Helm chart delivery (vs. git-source) — versioned chart artifacts, product-grade release.
Image ghcr.io/seanb4t/engram; chart oci://ghcr.io/seanb4t/charts/engram.
Multi-source Application (mirrors temporal-workers): OCI chart for workloads + trimmed kustomize Source B for the cluster ExternalSecrets — they do not move into the generic chart, and stay rendered so prune can’t delete them.
First release v0.2.0 (continues from memory-mcp:0.1.1, since the OIDC/actor work was the 0.2.0 increment); Chart version == appVersion == image tag, all advancing on one git tag.
Namespace stays agent-memory; only the repo/product is named engram.