LiteLLM → agentgateway Migration Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace LiteLLM with a single standalone agentgateway that serves both the LLM proxy (llm-gw.fzymgc.house) and the MCP gateway (mcp-gw.fzymgc.house), preserving server-side ZDR (via a CEL transformation) and fixing MCP OAuth (engram stays a Resource Server; agentgateway is the AS-shim + JWT enforcement point with a generic DCR short-circuit).

Architecture: One agentgateway Deployment (standalone config.yaml, image ghcr.io/agentgateway/agentgateway:v1.2.1) in a new agentgateway namespace, deployed via a single-source kustomize ArgoCD Application mirroring argocd/app-configs/litellm/. Two Traefik IngressRoutes (llm-gw, mcp-gw) front two listener ports. LiteLLM stays live until P5; every phase is independently revertible.

Tech Stack: agentgateway (Rust), Kubernetes, kustomize, ArgoCD (GitOps), Traefik IngressRoute, ExternalSecrets + Vault, Terraform (Authentik + Vault KV), Ollama, OpenRouter, jj VCS.

Design bead: hl-0sr Spec: docs/engineering/specs/2026-06-06-llm-gw-mcp-gw-migration-design.md

---

File Structure

File	Status	Responsibility
Vault secret/fzymgc-house/cluster/agentgateway	Bootstrap (manual)	OpenRouter key + embedder VK value — placed out-of-band via vault kv put (mirrors how the LiteLLM OpenRouter key was set; not Terraform-managed, since litellm’s vault_kv_secret_v2 does not own the OpenRouter key either)
tf/authentik/mcp_public_oauth.tf	Create	Dedicated public PKCE Authentik client mcp-public (loopback redirects) for the MCP flow
argocd/app-configs/agentgateway/namespace.yaml	Create	agentgateway namespace
argocd/app-configs/agentgateway/secrets.yaml	Create	ExternalSecrets: OpenRouter key + virtual-key values
argocd/app-configs/agentgateway/ollama-endpoint.yaml	Create	Headless Service + EndpointSlice for external Ollama (192.168.218.96:11434)
argocd/app-configs/agentgateway/configmap.yaml	Create	The single config.yaml (LLM providers/models/aliases, ZDR + drop_params transforms, virtual keys, MCP route + auth)
argocd/app-configs/agentgateway/deployment.yaml	Create	agentgateway Deployment (v1.2.1, config mounted, reloader)
argocd/app-configs/agentgateway/service.yaml	Create	Service exposing the LLM (3000) + MCP (3001) ports
argocd/app-configs/agentgateway/certificate.yaml	Create	TLS cert for llm-gw + mcp-gw
argocd/app-configs/agentgateway/modern-auth.yaml	Create	Authentik forward-auth Middleware (for any exposed admin UI)
argocd/app-configs/agentgateway/ingress.yaml	Create	Two IngressRoutes: llm-gw (LLM) and mcp-gw (MCP, added in P3)
argocd/app-configs/agentgateway/kustomization.yaml	Create	Kustomize wiring
argocd/cluster-app/templates/agentgateway.yaml	Create	ArgoCD Application (project core-services, single source)
argocd/app-configs/velero/backup-schedule.yaml	Modify	Add agentgateway to excludedNamespaces in both schedules
argocd/app-configs/agent-memory/external-secret.yaml	Modify	Repoint embedder key to the agentgateway virtual key (P2)
argocd/cluster-app/templates/agent-memory.yaml	Modify	Repoint engram memory.litellm.url → agentgateway LLM service + add mcp-public audience (P2/P3)
argocd/app-configs/litellm/ingress.yaml	Modify (P2)	Drop the litellm.fzymgc.house Host match so agentgateway can claim it (no dual-route conflict)
tf/authentik/memory_mcp_oauth.tf	Modify/Delete (P5)	Confidential memory-mcp client stays for LiteLLM until P5; orphaned afterward (engram uses mcp-public) → remove or strip its litellm redirect/description at P5
argocd/app-configs/litellm/ (all)	Delete (P5)	Decommission
argocd/cluster-app/templates/litellm.yaml	Delete (P5)	Decommission Application
tf/authentik/litellm.tf	Delete (P5)	Decommission Authentik OIDC + Vault KV

There are no automated unit tests for Kubernetes manifests or Terraform in this repo; verification is local lint (rumdl, yamllint), kubectl --dry-run / kubectl kustomize, terraform validate/plan, the agentgateway JSON-schema validation, ArgoCD sync, and live curl/MCP smoke tests. “Verify” steps below are these commands. Each task ends with a jj commit.

VCS: this is a colocated jj repo. Commit with jj commit -m "<msg>". Work on the existing docs/llm-gw-mcp-gw-spec line or a fresh change off main per the jj skill; do not use mutating git commands.

---

Phase 0 — Schema pin (de-risk vendor config)

Task 1: Pin the agentgateway v1.2.1 config schema + image digest

Files:

• Create (scratch, not committed): /tmp/agentgateway-schema.json, /tmp/agw-config.yaml

• Step 1: Fetch the v1.2.1 JSON schema and a known-good standalone example

Run:

curl -fsSL https://agentgateway.dev/schema/config -o /tmp/agentgateway-schema.json
# Inspect the standalone shape: top-level llm: and mcp: blocks, model entry fields
# (name, provider, params{apiKey,hostOverride,pathOverride}, backendTLS, transformation),
# model aliasing, apiKeys, and mcp policies.mcpAuthentication (mode,issuer,jwks.url,
# audiences,resourceMetadata,clientId).
jq '.properties | keys' /tmp/agentgateway-schema.json

Expected: top-level keys include binds, llm, mcp (and/or the documented standalone equivalents). Record the exact field spellings for transformation, model aliasing, apiKeys, and mcpAuthentication — later tasks use them verbatim.

• Step 2: Resolve and pin the image digest

Run:

docker buildx imagetools inspect ghcr.io/agentgateway/agentgateway:v1.2.1 \
  --format '{{json .Manifest.Digest}}'

Expected: a sha256:... digest. Record it; the Deployment pins ghcr.io/agentgateway/agentgateway:v1.2.1@sha256:....

• Step 3: Validate a minimal combined config against the schema locally

Create /tmp/agw-config.yaml with a minimal llm: + mcp: skeleton (from the spec’s config.yaml skeleton) and validate it:

docker run --rm -v /tmp/agw-config.yaml:/c.yaml \
  ghcr.io/agentgateway/agentgateway:v1.2.1 --validate -f /c.yaml

Expected: validation passes (or prints the exact field corrections needed). Use the corrected field names in Task 5/Task 15. Do not proceed to Task 5 until a minimal combined config validates.

• Step 4: Commit a short grounding note (no repo files changed)

bd note hl-0sr "Task1: pinned agentgateway v1.2.1 schema + digest sha256:<...>; combined llm+mcp config validates"

---

Phase 1 — Stand up agentgateway (LLM), parallel to LiteLLM

Task 2: Bootstrap the agentgateway Vault secret (manual)

Files: none (out-of-band Vault write — no Terraform)

Why manual, not Terraform: the existing vault_kv_secret_v2.litellm (tf/authentik/litellm.tf) manages only master_key + oidc_client_id + oidc_client_secret; the LiteLLM OpenRouter key was placed in Vault out-of-band (no var.openrouter_api_key exists anywhere in tf/). vault_kv_secret_v2 owns the entire secret payload, so it cannot coexist with a manually-added OpenRouter key in the same path. We mirror the litellm precedent: populate the path by hand. The KV mount is secret (the ClusterSecretStore’s path: "secret"), key prefix fzymgc-house/cluster/<app> — confirmed against vault_kv_secret_v2.litellm (mount = "secret", name = "fzymgc-house/cluster/litellm").

• Step 1: Generate the embedder virtual-key value

Run: printf 'sk-%s\n' "$(openssl rand -hex 24)" Record the value (used as vk_engram_embedder).

• Step 2: Write both values to Vault (reuse the existing OpenRouter key)

Run (authenticated to Vault):

vault kv put secret/fzymgc-house/cluster/agentgateway \
  openrouter_api_key="$(vault kv get -field=openrouter_api_key secret/fzymgc-house/cluster/litellm)" \
  vk_engram_embedder="sk-...generated-in-step-1..."

Expected: Success! Data written to: secret/data/fzymgc-house/cluster/agentgateway.

• Step 3: Verify the path is readable

Run: vault kv get -mount=secret fzymgc-house/cluster/agentgateway Expected: both openrouter_api_key and vk_engram_embedder present.

• Step 4: Note bd note hl-0sr "Task2: bootstrapped secret/fzymgc-house/cluster/agentgateway (manual, mirrors litellm OpenRouter out-of-band pattern)"

Task 3: Namespace + ExternalSecrets + kustomization

Files:

• Create: argocd/app-configs/agentgateway/namespace.yaml
• Create: argocd/app-configs/agentgateway/secrets.yaml
• Create: argocd/app-configs/agentgateway/kustomization.yaml
• Step 1: namespace.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: agentgateway
  labels:
    app.kubernetes.io/name: agentgateway

• Step 2: secrets.yaml (mirrors argocd/app-configs/litellm/secrets.yaml)

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: agentgateway-app-secrets
  namespace: agentgateway
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: agentgateway-app-secrets
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      type: Opaque
      data:
        openrouter-api-key: "{{ .openrouter_api_key }}"
        vk-engram-embedder: "{{ .vk_engram_embedder }}"
  data:
    - secretKey: openrouter_api_key
      remoteRef:
        key: fzymgc-house/cluster/agentgateway
        property: openrouter_api_key
    - secretKey: vk_engram_embedder
      remoteRef:
        key: fzymgc-house/cluster/agentgateway
        property: vk_engram_embedder

• Step 3: kustomization.yaml (P1 set; ingress.yaml’s mcp route added P3)

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: agentgateway
resources:
  - namespace.yaml
  - secrets.yaml
  - ollama-endpoint.yaml
  - configmap.yaml
  - deployment.yaml
  - service.yaml
  - certificate.yaml
  - modern-auth.yaml
  - ingress.yaml

• Step 4: Verify

Run: yamllint argocd/app-configs/agentgateway/ && kubectl kustomize argocd/app-configs/agentgateway >/dev/null && echo OK Expected: OK (after later tasks create the referenced files; for now expect a missing-resource error naming only not-yet-created files).

• Step 5: Commit

jj commit -m "feat(agentgateway): namespace + ESO secrets + kustomization (hl-0sr)"

Task 4: Ollama headless Service + EndpointSlice

Files:

• Create: argocd/app-configs/agentgateway/ollama-endpoint.yaml

• Step 1: Write the headless Service + EndpointSlice (gives the external Ollama a stable in-cluster DNS name, per agentgateway’s Ollama guide)

---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: agentgateway
spec:
  clusterIP: None
  ports:
    - name: http
      port: 11434
      targetPort: 11434
      protocol: TCP
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: ollama-1
  namespace: agentgateway
  labels:
    kubernetes.io/service-name: ollama
addressType: IPv4
ports:
  - name: http
    port: 11434
    protocol: TCP
endpoints:
  - addresses:
      - "192.168.218.96"
    conditions:
      ready: true

• Step 2: Verify

Run: kubectl apply --dry-run=client -f argocd/app-configs/agentgateway/ollama-endpoint.yaml 2>&1 | head (or kubectl kustomize once present). Expected: no schema errors.

• Step 3: Commit

jj commit -m "feat(agentgateway): headless Service+EndpointSlice for external Ollama (hl-0sr)"

Task 5: agentgateway config.yaml ConfigMap (LLM half)

Files:

• Create: argocd/app-configs/agentgateway/configmap.yaml

Use the exact field spellings confirmed in Task 1. The content below is the grounded shape; reconcile any field-name corrections from the schema validation.

• Step 1: Write the ConfigMap with the LLM providers, model aliases (the full or-* set migrated 1:1 from argocd/app-configs/litellm/configmap.yaml), the ZDR transformation on or-deepseek-v4-flash-zdr, the drop_params transformation on the Ollama embeddings path, and the embedder virtual key. The mcp: block is added in Task 15 (P3).

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: agentgateway-config
  namespace: agentgateway
data:
  config.yaml: |
    # yaml-language-server: $schema=https://agentgateway.dev/schema/config
    llm:
      port: 3000
      models:
        # --- OpenRouter (OpenAI-compatible); alias -> upstream model id ---
        - name: or-claude-opus-4-7
          provider: openAI
          modelAlias: anthropic/claude-opus-4.7
          params:
            apiKey: "$OPENROUTER_API_KEY"
            hostOverride: "openrouter.ai:443"
            pathOverride: "/api/v1/chat/completions"
          backendTLS: {}
        # (repeat one entry per or-* model from the litellm configmap:
        #  or-claude-sonnet-4-6, or-claude-haiku-4-5, or-gpt-5-5-pro, or-gpt-5-5,
        #  or-gpt-5-4-mini, or-gpt-5-3-codex, or-gemini-2-5-pro, or-gemini-2-5-flash,
        #  or-deepseek-v4-pro, or-deepseek-v4-flash, or-glm-5-1, or-glm-4-6,
        #  or-kimi-k2-6, or-kimi-k2-thinking, or-minimax-m2-7, or-minimax-m2-her,
        #  or-llama-4-maverick, or-qwen3-max, or-qwen3-coder-plus — each with its
        #  modelAlias set to the openrouter slug from the litellm configmap)
        # --- ZDR-enforced variant: same upstream model, ZDR injected server-side ---
        - name: or-deepseek-v4-flash-zdr
          provider: openAI
          modelAlias: deepseek/deepseek-v4-flash
          params:
            apiKey: "$OPENROUTER_API_KEY"
            hostOverride: "openrouter.ai:443"
            pathOverride: "/api/v1/chat/completions"
          backendTLS: {}
          transformation:
            provider: 'json({"zdr": true, "require_parameters": true})'
        # --- Local Ollama: chat (wildcard) + bge-m3 embeddings ---
        - name: "ollama/*"
          provider: openAI
          params:
            hostOverride: "ollama.agentgateway.svc.cluster.local:11434"
        - name: ollama/bge-m3
          provider: openAI
          params:
            hostOverride: "ollama.agentgateway.svc.cluster.local:11434"
          # drop the base64 encoding_format Ollama rejects (LiteLLM drop_params equiv)
          transformation:
            encoding_format: 'null'
      apiKeys:
        - value: "$VK_ENGRAM_EMBEDDER"
          allowedModels: ["ollama/bge-m3"]

• Step 2: Validate the config against the schema

Run:

kubectl kustomize argocd/app-configs/agentgateway | \
  yq -r 'select(.kind=="ConfigMap" and .metadata.name=="agentgateway-config") | .data["config.yaml"]' \
  > /tmp/rendered-config.yaml
docker run --rm -v /tmp/rendered-config.yaml:/c.yaml \
  ghcr.io/agentgateway/agentgateway:v1.2.1 --validate -f /c.yaml

Expected: validation passes. Fix any field-name drift flagged.

• Step 3: Commit

jj commit -m "feat(agentgateway): LLM config (OpenRouter+Ollama, ZDR+drop_params transforms, embedder VK) (hl-0sr)"

Task 6: Deployment + Service + Certificate

Files:

• Create: argocd/app-configs/agentgateway/deployment.yaml
• Create: argocd/app-configs/agentgateway/service.yaml
• Create: argocd/app-configs/agentgateway/certificate.yaml
• Step 1: deployment.yaml (mirrors litellm’s Deployment shape; pin the digest from Task 1)

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agentgateway
  namespace: agentgateway
  labels:
    app.kubernetes.io/name: agentgateway
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: agentgateway
  template:
    metadata:
      labels:
        app.kubernetes.io/name: agentgateway
    spec:
      containers:
        - name: agentgateway
          image: ghcr.io/agentgateway/agentgateway:v1.2.1@sha256:REPLACE_WITH_TASK1_DIGEST
          imagePullPolicy: IfNotPresent
          args: ["-f", "/etc/agentgateway/config.yaml"]
          ports:
            - { name: llm, containerPort: 3000, protocol: TCP }
            - { name: mcp, containerPort: 3001, protocol: TCP }
          env:
            - name: OPENROUTER_API_KEY
              valueFrom: { secretKeyRef: { name: agentgateway-app-secrets, key: openrouter-api-key } }
            - name: VK_ENGRAM_EMBEDDER
              valueFrom: { secretKeyRef: { name: agentgateway-app-secrets, key: vk-engram-embedder } }
            - name: TZ
              value: "America/Los_Angeles"
          volumeMounts:
            - { name: config, mountPath: /etc/agentgateway }
          resources:
            requests: { memory: "128Mi", cpu: "100m" }
            limits: { memory: "512Mi", cpu: "1000m" }
          securityContext:
            runAsNonRoot: true
            runAsUser: 65532
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
      volumes:
        - name: config
          configMap: { name: agentgateway-config }
      restartPolicy: Always

Note: agentgateway is a fast Rust binary — the requests/limits are far below LiteLLM’s (512Mi–2Gi). Adjust after observing real usage.

• Step 2: service.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: agentgateway
  namespace: agentgateway
  labels:
    app.kubernetes.io/name: agentgateway
spec:
  selector:
    app.kubernetes.io/name: agentgateway
  ports:
    - { name: llm, port: 3000, targetPort: llm, protocol: TCP }
    - { name: mcp, port: 3001, targetPort: mcp, protocol: TCP }

• Step 3: certificate.yaml (mirror argocd/app-configs/litellm/certificate.yaml; SANs for both hosts)

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: agentgateway-tls
  namespace: agentgateway
spec:
  secretName: agentgateway-tls
  dnsNames:
    - llm-gw.fzymgc.house
    - mcp-gw.fzymgc.house
  issuerRef:
    name: vault-issuer
    kind: ClusterIssuer

(issuerRef: vault-issuer / ClusterIssuer — confirmed against argocd/app-configs/litellm/certificate.yaml and the cluster-wide cert pattern.)

• Step 4: Verify

Run: kubectl kustomize argocd/app-configs/agentgateway >/dev/null && yamllint argocd/app-configs/agentgateway/ && echo OK Expected: OK.

• Step 5: Commit

jj commit -m "feat(agentgateway): Deployment + Service + Certificate (hl-0sr)"

Task 7: modern-auth Middleware + llm-gw IngressRoute

Files:

• Create: argocd/app-configs/agentgateway/modern-auth.yaml
• Create: argocd/app-configs/agentgateway/ingress.yaml
• Step 1: modern-auth.yaml (copy the proven pattern from argocd/app-configs/hubble-ui/traefik-middleware.yaml / argocd/app-configs/authentik/traefik-middleware.yaml, namespaced to agentgateway — only needed if the admin UI is exposed)

---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: modern-auth
  namespace: agentgateway
spec:
  forwardAuth:
    address: http://authentik-server.authentik.svc/api/authz/forward-auth?authentik_url=https%3A%2F%2Fauth.fzymgc.house%2F
    trustForwardHeader: true
    authResponseHeaders: [Remote-User, Remote-Groups, Remote-Email, Remote-Name]

• Step 2: ingress.yaml — the llm-gw route now; the mcp-gw route is appended in Task 16 (P3)

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: agentgateway-llm
  namespace: agentgateway
  annotations:
    router-hosts.fzymgc.house/enabled: "true"
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`llm-gw.fzymgc.house`)
      kind: Rule
      services:
        - name: agentgateway
          port: 3000
  tls:
    secretName: agentgateway-tls

• Step 3: Verify

Run: kubectl kustomize argocd/app-configs/agentgateway >/dev/null && echo OK Expected: OK.

• Step 4: Commit

jj commit -m "feat(agentgateway): modern-auth middleware + llm-gw IngressRoute (hl-0sr)"

Task 8: Velero exclusion

Files:

• Modify: argocd/app-configs/velero/backup-schedule.yaml

• Step 1: Add agentgateway to excludedNamespaces under the Telemetry/stateless category in both the daily-backup and weekly-full-backup Schedules (it is config-in-Git, no PVC state — D9/D11).

• Step 2: Verify

Run: yamllint argocd/app-configs/velero/backup-schedule.yaml && rg -n 'agentgateway' argocd/app-configs/velero/backup-schedule.yaml Expected: two matches (one per schedule).

• Step 3: Commit

jj commit -m "chore(velero): exclude stateless agentgateway namespace (hl-0sr)"

Task 9: ArgoCD Application

Files:

• Create: argocd/cluster-app/templates/agentgateway.yaml

• Step 1: Write the Application (single-source kustomize; mirror argocd/cluster-app/templates/litellm.yaml for project, syncPolicy, syncOptions)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: agentgateway
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: core-services
  source:
    repoURL: https://github.com/fzymgc-house/selfhosted-cluster
    targetRevision: HEAD
    path: argocd/app-configs/agentgateway
  destination:
    server: https://kubernetes.default.svc
    namespace: agentgateway
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [ServerSideApply=true, CreateNamespace=true]

(Confirm project + syncOptions against argocd/cluster-app/templates/litellm.yaml.)

• Step 2: Verify

Run: yamllint argocd/cluster-app/templates/agentgateway.yaml Expected: clean.

• Step 3: Commit

jj commit -m "feat(agentgateway): ArgoCD Application (hl-0sr)"

Task 10: Deploy P1 and verify the LLM gateway live

Files: none (operational verification after the P1 PR merges and ArgoCD syncs)

• Step 1: Confirm rollout

Run: kubectl -n agentgateway rollout status deploy/agentgateway && kubectl -n agentgateway get pods Expected: Deployment available, pod Ready, no CrashLoop.

• Step 2: Smoke-test chat (OpenRouter) with the embedder/admin key

curl -fsS https://llm-gw.fzymgc.house/v1/chat/completions \
  -H "Authorization: Bearer $VK" -H 'Content-Type: application/json' \
  -d '{"model":"or-claude-haiku-4-5","messages":[{"role":"user","content":"ping"}],"max_tokens":5}' | jq .choices[0].message

Expected: a completion. (Use a virtual key authorized for that model.)

• Step 3: Smoke-test Ollama chat + bge-m3 embeddings

curl -fsS https://llm-gw.fzymgc.house/v1/embeddings \
  -H "Authorization: Bearer $VK_EMBEDDER" -H 'Content-Type: application/json' \
  -d '{"model":"ollama/bge-m3","input":"hello","encoding_format":"base64"}' | jq '.data[0].embedding | length'

Expected: a vector length (the encoding_format transformation strips the Ollama-incompatible param, so this succeeds instead of erroring).

• Step 4: Verify ZDR injection

Send a request to or-deepseek-v4-flash-zdr without any provider field; confirm via agentgateway request logging / OpenRouter that provider.zdr=true was forwarded. Expected: ZDR present despite the client omitting it.

• Step 5: Verify a virtual key denies a disallowed model

A request with $VK_EMBEDDER to or-claude-haiku-4-5 (not in allowedModels) → denied.

• Step 6: Note results bd note hl-0sr "P1 verified: chat+embeddings+ZDR+VK enforcement live on llm-gw"

---

Phase 2 — Cut LLM traffic + engram embedder

Task 11: Transitional litellm → llm-gw alias

Files:

• Modify: argocd/app-configs/agentgateway/ingress.yaml

• Modify: argocd/app-configs/litellm/ingress.yaml (hand off the host — same PR, no dual-route)

• Step 1: Remove the litellm.fzymgc.house route from litellm’s IngressRoute. Edit argocd/app-configs/litellm/ingress.yaml and delete the Host(litellm.fzymgc.house) route, leaving litellm reachable only at litellm.k8s.fzymgc.house (its other host). Two IngressRoutes claiming the same Host on the same entryPoint is non-deterministic in Traefik — the host must be released here in the same PR before agentgateway claims it (Step 2).

• Step 2: Add litellm.fzymgc.house to the agentgateway-llm IngressRoute (D12), so unenumerated consumers follow to agentgateway.

    - match: Host(`llm-gw.fzymgc.house`) || Host(`litellm.fzymgc.house`)
      kind: Rule
      services:
        - name: agentgateway
          port: 3000

• Step 3: Verify only one route serves the host: kubectl kustomize argocd/app-configs/agentgateway >/dev/null && rg -L 'litellm.fzymgc.house' argocd/app-configs/litellm/ingress.yaml || echo "host released from litellm OK"

• Step 4: Commit jj commit -m "feat(agentgateway): hand off litellm.fzymgc.house host to agentgateway (hl-0sr)"

Task 12: Repoint the engram embedder

Files:

• Modify: argocd/app-configs/agent-memory/external-secret.yaml

• Modify: argocd/cluster-app/templates/agent-memory.yaml

• Step 1: Point the embedder secret at the agentgateway virtual key. Change memory-mcp-litellm’s remoteRef to read the agentgateway embedder VK (fzymgc-house/cluster/agentgateway → vk_engram_embedder), keeping the existing secret key name (litellm_ollama_key) so the engram chart values need no rename, or rename consistently in Step 2.

• Step 2: Repoint the engram chart values. In agent-memory.yaml, set valuesObject.memory.litellm.url: "http://agentgateway.agentgateway.svc.cluster.local:3000" (the engram chart’s memory.litellm.url/keySecret become the agentgateway LLM endpoint + embedder VK).

• Step 3: Verify yamllint argocd/app-configs/agent-memory/external-secret.yaml argocd/cluster-app/templates/agent-memory.yaml

• Step 4: Commit jj commit -m "feat(agent-memory): repoint engram embedder to agentgateway (hl-0sr)"

Task 13: Verify LLM cutover

• Step 1: After merge+sync, repeat Task 10 smoke tests against llm-gw and the litellm.fzymgc.house alias. Expected: both serve via agentgateway.
• Step 2: Trigger an engram memory write; confirm a bge-m3 embedding flows through agentgateway (check agentgateway logs / engram health). Expected: success.
• Step 3: Rollback check: confirm repointing the embedder secret back to LiteLLM restores writes (then re-apply agentgateway). bd note hl-0sr "P2 verified: LLM + embedder cut to agentgateway; rollback confirmed"

---

Phase 3 — MCP route + auth

Task 14: Dedicated public Authentik client + engram audience

Files:

• Create: tf/authentik/mcp_public_oauth.tf
• Modify: argocd/cluster-app/templates/agent-memory.yaml (engram accepted-audiences)
• Step 1: Write the public PKCE client (model on tf/authentik/memory_mcp_oauth.tf, but public client type, PKCE, additive — leaves memory-mcp untouched)

resource "authentik_provider_oauth2" "mcp_public" {
  name               = "mcp-public"
  client_id          = "mcp-public"
  client_type        = "public"
  authorization_flow = data.authentik_flow.default_provider_authorization_explicit_consent.id
  invalidation_flow  = data.authentik_flow.default_provider_invalidation_flow.id
  # PKCE-only public client for MCP clients (Claude Code, etc.)
  allowed_redirect_uris = [
    { matching_mode = "regex", url = "http://(127\\.0\\.0\\.1|localhost):[0-9]+/.*" },
  ]
  # Same scope mappings as memory_mcp_oauth.tf (lines 40-46), verbatim:
  property_mappings = [
    data.authentik_property_mapping_provider_scope.openid.id,
    authentik_property_mapping_provider_scope.email_verified.id,
    data.authentik_property_mapping_provider_scope.profile.id,
    data.authentik_property_mapping_provider_scope.offline_access.id,
  ]
  signing_key = data.authentik_certificate_key_pair.tls.id
}

resource "authentik_application" "mcp_public" {
  name              = "mcp-public"
  slug              = "mcp-public"
  protocol_provider = authentik_provider_oauth2.mcp_public.id
}

(The data.authentik_property_mapping_provider_scope.*, authentik_property_mapping_provider_scope.email_verified, and data.authentik_certificate_key_pair.tls data sources/resources are already declared for memory_mcp_oauth.tf in the same workspace — reuse them, do not redeclare.)

(Match the exact scope/property-mapping data sources to those in memory_mcp_oauth.tf.)

• Step 2: Set engram’s accepted OIDC audience to mcp-public in the engram chart values (memory.oidc.audience in agent-memory.yaml, Source A). Note this value is a scalar string in the engram chart, currently unset — so set it to mcp-public (the audience agentgateway validates and forwards). If a future flow needs multiple audiences, confirm whether the chart accepts a comma-separated value before widening. This is a values change, not engram code.

• Step 3: Validate cd tf/authentik && terraform validate → valid; yamllint argocd/cluster-app/templates/agent-memory.yaml

• Step 4: Commit jj commit -m "feat(authentik): public mcp-public PKCE client + engram audience (hl-0sr)"

Task 15: Add the MCP block to agentgateway config

Files:

• Modify: argocd/app-configs/agentgateway/configmap.yaml

• Step 1: Append the mcp: block (fields confirmed in Task 1; generic issuer, no provider:, clientId for the DCR short-circuit)

    mcp:
      port: 3001
      policies:
        mcpAuthentication:
          mode: strict
          issuer: "https://auth.fzymgc.house/application/o/mcp-public/"
          jwks:
            url: "https://auth.fzymgc.house/application/o/mcp-public/jwks/"
          audiences: ["mcp-public"]
          clientId: "mcp-public"
          resourceMetadata:
            resource: "https://mcp-gw.fzymgc.house/mcp"
            bearerMethodsSupported: [header]
      targets:
        - name: engram
          mcp:
            host: "http://memory-mcp.agent-memory.svc.cluster.local:8080/mcp"

• Step 2: Validate against the schema (as Task 5 Step 2). Expected: passes.

• Step 3: Commit jj commit -m "feat(agentgateway): MCP route + auth (engram, generic DCR short-circuit) (hl-0sr)"

Task 16: mcp-gw IngressRoute

Files:

• Modify: argocd/app-configs/agentgateway/ingress.yaml

• Step 1: Add the mcp-gw IngressRoute → service port 3001

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: agentgateway-mcp
  namespace: agentgateway
  annotations:
    router-hosts.fzymgc.house/enabled: "true"
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`mcp-gw.fzymgc.house`)
      kind: Rule
      services:
        - name: agentgateway
          port: 3001
  tls:
    secretName: agentgateway-tls

• Step 2: Verify kubectl kustomize argocd/app-configs/agentgateway >/dev/null && echo OK

• Step 3: Commit jj commit -m "feat(agentgateway): mcp-gw IngressRoute (hl-0sr)"

Task 17: Verify MCP auth end-to-end

• Step 1: PRM + DCR short-circuit

curl -fsS https://mcp-gw.fzymgc.house/.well-known/oauth-protected-resource/mcp | jq .
curl -fsS -X POST https://mcp-gw.fzymgc.house/.well-known/oauth-authorization-server/mcp/client-registration \
  -H 'content-type: application/json' -d '{"redirect_uris":["http://127.0.0.1:8976/callback"]}' | jq .client_id

Expected: PRM lists Authentik as authorization_servers; the registration returns client_id: "mcp-public" (mock DCR).

• Step 2: Unauthenticated → 401

curl -s -i https://mcp-gw.fzymgc.house/mcp -d '{}' | head -5

Expected: 401 with a WWW-Authenticate: Bearer resource_metadata=... header.

• Step 3: Authenticated round-trip — connect a real MCP client (Claude Code, configured with mcp-public) → completes PKCE with Authentik → tool list + a store/search on engram succeeds, attributed to the verified caller. Expected: success.

• Step 4: Note bd note hl-0sr "P3 verified: DCR short-circuit + PKCE + JWT validation + engram round-trip on mcp-gw"

---

Phase 4 — Cut MCP traffic

Task 18: Repoint MCP clients to mcp-gw

Files: client-side .mcp.json / harness config (outside this repo) + docs

• Step 1: Update the MCP client config from litellm.fzymgc.house/mcp/engram to https://mcp-gw.fzymgc.house/mcp with the mcp-public client. Update docs/operations/litellm-mcp-clients.md (rename/repoint to agentgateway).
• Step 2: Verify the headless agent client authenticates and performs a memory round-trip via mcp-gw. Expected: success.
• Step 3: Rollback check — confirm repointing back to litellm/mcp/engram still works (LiteLLM live until P5). bd note hl-0sr "P4 verified: MCP cut to mcp-gw; rollback confirmed"

---

Phase 5 — Decommission LiteLLM

Task 19: Remove the LiteLLM ArgoCD app + manifests

Files:

• Delete: argocd/cluster-app/templates/litellm.yaml
• Delete: argocd/app-configs/litellm/ (all)
• Step 1: Confirm no remaining consumer of litellm.fzymgc.house (the alias on agentgateway now serves it; remove litellm’s own route first).
• Step 2: Delete the Application template and the app-configs directory.
• Step 3: Verify rg -n 'litellm' argocd/ || echo "no argocd refs" (expect only the agentgateway transitional alias, which is removed in Task 21).
• Step 4: Commit jj commit -m "chore(litellm): remove ArgoCD app + manifests (hl-0sr)"

Task 20: Remove the LiteLLM + orphaned memory-mcp Authentik/Vault Terraform

Files:

• Delete: tf/authentik/litellm.tf
• Delete: tf/authentik/memory_mcp_oauth.tf
• Step 1: Delete tf/authentik/litellm.tf (OIDC provider, application, litellm-users/litellm-admin groups, role-scope + policy bindings, random_password.litellm_master_key, vault_kv_secret_v2.litellm).
• Step 2: Delete tf/authentik/memory_mcp_oauth.tf — the memory-mcp client + application are orphaned once the LiteLLM broker is gone (engram now accepts the mcp-public audience via agentgateway). Its allowed_redirect_uris and meta_description both reference the retired litellm.fzymgc.house broker, so the resource is dead. (If any non-broker consumer of memory-mcp is found in Step 4 verification, strip the litellm redirect/description instead of deleting.)
• Step 3: Verify cd tf/authentik && terraform validate → valid; the GitOps plan shows only deletes of litellm + memory_mcp resources, no collateral.
• Step 4: Confirm engram MCP still authenticates end-to-end via mcp-public (Task 17 round-trip) AFTER the memory_mcp_oauth.tf removal — proves nothing still depends on the memory-mcp client.
• Step 5: Commit jj commit -m "chore(litellm): remove LiteLLM + orphaned memory-mcp Authentik/Vault TF (hl-0sr)"

Task 21: Remove the transitional alias + final verification

Files:

• Modify: argocd/app-configs/agentgateway/ingress.yaml
• Step 1: Remove Host(\litellm.fzymgc.house`)from theagentgateway-llm` route once clients are confirmed off it.
• Step 2: Final verification: LLM chat/embeddings on llm-gw, MCP round-trip on mcp-gw, engram memory writes, ZDR injection, virtual-key enforcement — all green; kubectl -n litellm get all empty; agentgateway in Velero exclusions.
• Step 3: Commit jj commit -m "chore(agentgateway): drop transitional litellm alias; migration complete (hl-0sr)"
• Step 4: bd note hl-0sr "P5 complete: LiteLLM decommissioned; agentgateway serves llm-gw + mcp-gw" and close the epic when merged.