Design: Replace LiteLLM with agentgateway (unified LLM + MCP gateway)

Bead: hl-0sr
Date: 2026-06-06
Status: Draft (pending design-reviewer gate)
Author: Sean (with Claude)

Problem

LiteLLM in this cluster wears four hats: (1) LLM proxy, (2) Authentik OIDC admin SSO, (3) MCP gateway fronting engram via delegate_auth_to_upstream PKCE OAuth, and (4) virtual-key governance. Hat #3 — the MCP delegate OAuth — is unreliable; recent commits (#1146 refresh-token stabilization, #1148 route rename) are repeated attempts to stabilize a fundamentally fragile pattern.

The root cause is architectural, not a bug: LiteLLM’s delegate_auth_to_upstream is a token-passthrough broker. The June 2025 MCP spec reclassified MCP servers as OAuth Resource Servers, and as of 2026-03-15 RFC 8707 resource indicators are mandatory — token passthrough is now discouraged by the spec. The ecosystem consensus is “the gateway handles auth, the MCP server stays dumb,” or, equivalently, the MCP server is the Resource Server and validates tokens itself.

engram already validates Authentik JWTs directly (JWKS signature, issuer, expiry). That is the spec-correct Resource Server role. The fragility lives entirely in the LiteLLM broker in front of it. The fix is to remove the broker, not replace it with another broker.

Goals

Remove the fragile MCP OAuth broker from the engram request path.
Centralize LLM + MCP visibility, routing, and policy in one data plane.
Keep (and improve) the LLM proxy + key governance, GitOps-native.
Preserve the server-side ZDR (zero-data-retention) guarantee without a custom binary or plugin.
Keep Authentik SSO for any admin surface without an Enterprise license.

Non-goals

Building a bespoke gateway. Use one maintained OSS tool.
Changing engram’s auth model (it stays the Resource Server).
Adopting the Kubernetes Gateway-API control-plane model (overkill for this scope — see D2).
Migrating any other service that talks to litellm.fzymgc.house beyond a base-URL/hostname repoint.

Why one tool (agentgateway)

agentgateway (Linux Foundation / Solo.io, Rust) is a single data plane that fronts “regular APIs, LLM inference, MCP tool servers, and A2A agent traffic” — explicitly designed so you don’t run separate regular and AI gateways. It covers every LiteLLM hat this cluster uses:

LLM proxy: OpenAI-compatible providers (OpenRouter and Ollama configured as openai-type backends by host/port/path), exposing /v1/chat/completions and /v1/embeddings.
Governance: virtual keys with per-key token budgets + cost tracking, token/request rate limits, model allow-lists, model aliasing.
MCP gateway: fronts engram; agentgateway Strict-validates the Authentik JWT (and provides the AS-shim + DCR short-circuit, see D3), then forwards to engram, which re-validates as defense-in-depth.
Admin SSO: any exposed UI/route is gated by the existing Authentik modern-auth forward-auth middleware.

Crucially, agentgateway’s CEL request transformations can compute and set fields in the LLM request body server-side, bound to a route. This replaces both LiteLLM features that have no native equivalent elsewhere: the ZDR guarantee (inject provider.zdr=true) and drop_params (strip provider-unsupported fields) — in pure config, with no custom image, no Go plugin, and no Postgres.

(The two-tool alternative — Bifrost for LLM + agentgateway for MCP — was rejected: see Alternatives. Its ZDR story required a custom dynamically-linked Bifrost image plus a Go .so plugin with strict Go/dependency/libc ABI matching rebuilt on every upgrade, plus a Postgres DB. One tool removes all of that.)

Decisions

#	Decision	Rationale
D1	One tool — agentgateway — for both LLM and MCP	Single data plane, single config, single auth/observability model; directly serves “centralize control in one place”
D2	Standalone mode: a raw kustomize Deployment behind a Traefik IngressRoute, mirroring the existing `litellm` app-config	The K8s control-plane + Gateway-API CRD model (`AgentgatewayPolicy`/`Backend`) is built for fleets of routes; for one LLM proxy + one MCP upstream, a standalone `config.yaml` is far simpler and matches cluster convention
D3	agentgateway is the MCP auth enforcement point + AS-shim: `Strict` JWT validation against Authentik (issuer/JWKS/audiences), serves `/.well-known/oauth-protected-resource`, proxies Authentik’s AS metadata, and provides a DCR short-circuit (`LocalMcpAuthentication.client_id` → mock RFC 7591 `/register` returning the pre-configured client_id; requires agentgateway ≥1.3.0-alpha — the `clientId` field is absent in v1.2.1, see D10). engram keeps validating the forwarded token as defense-in-depth (engram change limited to adding the `mcp-public` audience — a values change, not code)	Replaces LiteLLM’s flaky broker with a maintained, spec-correct one in the same tool; fills Authentik’s DCR gap (see D13); centralizes MCP auth — the stated goal
D13	Add a dedicated public PKCE client (e.g. `mcp-public`) in Authentik with the MCP clients’ loopback `allowed_redirect_uris`; agentgateway hands this client_id to clients via the DCR short-circuit. The existing `memory-mcp` client (itself already a public PKCE client, but bound to LiteLLM’s `litellm.fzymgc.house/callback` redirect) is left untouched until P5 so the live broker keeps working; `mcp-public` is a separate client carrying the MCP clients’ loopback redirects. engram’s accepted-audience gains the new client_id (a values change, not code).	Authentik does not support RFC 7591 DCR yet (goauthentik/authentik#8751, ~2026-08); a static public client + agentgateway’s mock DCR is the working path (the `clientId` mock-DCR field requires agentgateway ≥1.3.0-alpha; D10). Additive at P3 so the live LiteLLM broker keeps working until decommission — no dual-state conflict
D4	ZDR via a CEL request transformation bound to the `zdr` model/route (inject `provider.zdr=true`, `require_parameters=true`)	Config-only server-side guarantee a client cannot bypass; no plugin/custom image
D5	`drop_params` via a CEL transformation for the one live case (Ollama rejecting `encoding_format` on `/v1/embeddings`)	Same mechanism; no separate plugin
D6	Governance in `config.yaml`: virtual keys (`api_key`) + token/request rate limits + model allow-lists + model aliases; per-key token budgets where standalone supports them	GitOps source-of-truth; covers LiteLLM hat #4. Standalone `api_key` + `localRateLimit` are confirmed; per-key budget linkage is verified at plan time and falls back to rate limits if standalone-unsupported
D7	Authentik SSO via the existing `authentik/modern-auth` forward-auth for any exposed admin UI	No Enterprise license; proven cluster pattern
D8	Tool-neutral hostnames `llm-gw.fzymgc.house` and `mcp-gw.fzymgc.house`, both routed to the one agentgateway (separate IngressRoutes/listeners)	Decouples the DNS contract from the implementation; future swaps need no client reconfiguration
D9	`agentgateway` namespace is Velero-excluded (stateless; config is Git source-of-truth), added at P1	No PVC state. Per-key token-budget counters are in-memory; acceptable at single-replica homelab scale and documented
D10	Pin agentgateway to `v1.3.0-alpha.1` (by digest) — consciously overriding `argocd/CLAUDE.md`’s latest-GA policy	The mock-DCR `clientId` (D3/D13) was added to standalone `LocalMcpAuthentication` only in `1.3.0-alpha`; v1.2.1’s `RawConfig` is `serde(deny_unknown_fields)` and hard-rejects the field (verified empirically during execution). No GA release has mock-DCR, so DCR-client compatibility requires the alpha — accepted alpha-on-prod trade-off. Revisit when 1.3.0 reaches GA or Authentik ships DCR (~2026-08)
D11	engram embedder repointed to a `bge-m3`-scoped virtual key on `llm-gw`, via the repointed `memory-mcp-litellm` ExternalSecret	engram calls the gateway for local bge-m3 embeddings; least-privilege key mirrors today’s ollama-only key
D12	Transitional `litellm.fzymgc.house` alias → `llm-gw` through cutover, removed at decommission	De-risks any consumer not enumerated here at near-zero cost

Correction (2026-06-28): D3 / D10 / D13 are superseded. The mock-DCR clientId short-circuit was found structurally unusable against Authentik at execution time (no provider adapter — McpIDP is Auth0|Keycloak|Okta only — and Authentik exposes no registration_endpoint); the cluster standardizes on the static mcp-public PKCE client (“model B”). The v1.3.0-alpha.1 pin (D10) is obsolete — v1.3.1 GA is current (clientId shipped in GA at v1.3.0, 2026-06-18). The jwtAuthentication Strict JWT validation + PRM (resourceMetadata) enforcement surface stands unchanged. See docs/engineering/specs/2026-06-28-mcp-dcr-honesty-design.md and the ADR hl-23m 2026-06-28 addendum. Real DCR-proxy remains contingent on Authentik 2026.8.0 (hl-0sr.29).

Architecture

                          ┌───────────────────────────────────────────────┐
  LLM clients ──/v1/*───▶ │ agentgateway (Rust, standalone)                │──▶ OpenRouter
  host: llm-gw            │  ns: agentgateway                              │    (openai-compat,
  (Claude Code, Codex,    │  one config.yaml (ConfigMap):                  │     api/v1/...)
   engram embedder)       │   • LLM providers: openrouter, ollama          │──▶ Ollama @ seattle
                          │   • virtual keys + token budgets + rate limits │    (chat + bge-m3,
                          │   • model aliases (or-*)                       │     via headless Svc)
  MCP clients ──/mcp───▶  │   • CEL transforms: ZDR inject + drop_params   │
  host: mcp-gw            │   • MCP route → engram (pass-through)          │──▶ engram
  (Claude Code)           │  UI (if exposed) ◀── Traefik modern-auth       │    (validates
                          │  OTel metrics/logs/traces                      │     Authentik JWT,
                          └───────────────────────────────────────────────┘     stays the RS)

Deployment shape (mirrors `argocd/app-configs/litellm/`)

argocd/app-configs/agentgateway/:

namespace.yaml, kustomization.yaml
configmap.yaml — the single config.yaml (LLM providers, virtual keys, CEL transformations, MCP route)
deployment.yaml — ghcr.io/agentgateway/agentgateway pinned to v1.3.0-alpha.1 (by digest; see D10), config mounted, reloader.stakater.com/auto: "true"
service.yaml
secrets.yaml — ExternalSecrets for the OpenRouter key + virtual-key values (Vault fzymgc-house/cluster/agentgateway)
certificate.yaml
ingress.yaml — two Traefik IngressRoutes: llm-gw.fzymgc.house and mcp-gw.fzymgc.house → the agentgateway Service
ollama-endpoint.yaml — headless Service + EndpointSlice giving the external Ollama (192.168.218.96:11434) a stable in-cluster DNS name
modern-auth.yaml — a modern-auth Middleware in the agentgateway ns (same pattern as hubble-ui) for any exposed admin UI

ArgoCD Application: argocd/cluster-app/templates/agentgateway.yaml (single source, kustomize path — no Helm chart needed in standalone mode).

LLM provider configuration

OpenRouter as an openai-compatible backend (host: openrouter.ai, path: /api/v1/chat/completions, auth via the OpenRouter key secret). Model aliases map the or-* names to OpenRouter provider-prefixed model ids.
Ollama as an openai-compatible backend pointing at the headless Service for 192.168.218.96:11434; /v1/chat/completions (Completions route) and /v1/embeddings (Passthrough route — embeddings need no LLM-policy processing).

ZDR + drop_params (CEL transformations)

ZDR: a transformation bound to the zdr model alias injects provider = {zdr: true, require_parameters: true} into the request body before forwarding to OpenRouter. Because it is bound server-side to the route, a client selecting that model cannot bypass it — preserving today’s guarantee.
drop_params: a transformation on the Ollama embeddings route strips encoding_format (the one live incompatibility — Ollama rejects the base64 value OpenAI-SDK embedders send).

MCP / engram auth

The catch the design must handle: Authentik does not support OAuth Dynamic Client Registration (RFC 7591) yet (goauthentik/authentik#8751, planned ~2026-08), and MCP clients commonly require DCR against the authorization server. This is why LiteLLM brokers today — it presents the OAuth 2.1 AS surface (including registration) to clients while using a single static memory-mcp client against Authentik behind the scenes. Removing the broker without replacing that surface would break DCR-dependent clients.

agentgateway fills the gap natively. Its LocalMcpAuthentication config:

DCR short-circuit: set client_id to the Authentik memory-mcp client → agentgateway serves its own /register and returns a mock RFC 7591 response carrying that client_id. MCP clients that demand DCR get a valid registration without Authentik implementing DCR.
Resource + AS metadata: agentgateway serves /.well-known/oauth-protected-resource (the resourceMetadata for the mcp-gw resource) and proxies Authentik’s /.well-known/oauth-authorization-server.
Strict JWT validation: agentgateway validates the bearer token against Authentik’s issuer / jwks / audiences before forwarding to engram, then forwards the request (with the token) to engram, which re-validates it as defense-in-depth — no change to engram’s internal/auth code, and no engram PRM work (agentgateway owns PRM at mcp-gw).

The client login flow: MCP client → agentgateway /register (gets the static client_id) → PKCE authorize/token directly with Authentik (client’s loopback redirect) → agentgateway validates the JWT → engram re-validates → tool call attributed to the verified caller. Because the client now authenticates directly to Authentik with PKCE and no secret, the flow uses a dedicated public client (mcp-public, D13) added at P3 — separate from the live memory-mcp client that LiteLLM keeps using until P5, so there is no dual-state conflict. engram’s accepted-audiences config gains the mcp-public client_id (a values change, no code).

config.yaml skeleton (illustrative)

One LocalConfig carries llm: and mcp: together. Shape grounded against the agentgateway LocalConfig/LocalMcpAuthentication types and the standalone schema (https://agentgateway.dev/schema/config); exact field spellings are pinned against the v1.3.0-alpha.1 schema at plan time.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
  port: 3000
  models:
    # OpenRouter (OpenAI-compatible) — one entry per or-* alias
    - name: or-deepseek-v4-flash-zdr
      provider: openAI
      params:
        apiKey: "$OPENROUTER_API_KEY"
        hostOverride: "openrouter.ai:443"
        pathOverride: "/api/v1/chat/completions"
      backendTLS: {}
      transformation:                      # CEL — server-side ZDR injection (D4)
        provider: 'json({"zdr": true, "require_parameters": true})'
    # Ollama (local, OpenAI-compatible) — chat + bge-m3 embeddings
    - name: "ollama/*"
      provider: openAI
      params:
        hostOverride: "ollama.agentgateway.svc.cluster.local:11434"
  apiKeys:                                 # virtual keys (D6)
    - value: "$VK_ENGRAM_EMBEDDER"
      allowedModels: ["ollama/bge-m3"]     # least-privilege embedder key (D11)
mcp:
  port: 3001
  policies:
    mcpAuthentication:                     # route-level policy (D3)
      mode: strict
      issuer: "https://auth.fzymgc.house/application/o/mcp-public/"
      jwks:
        url: "https://auth.fzymgc.house/application/o/mcp-public/jwks/"
      audiences: ["mcp-public"]
      clientId: "mcp-public"               # generic DCR short-circuit; NO provider block (Authentik is generic, D3/D13)
      resourceMetadata:
        resource: "https://mcp-gw.fzymgc.house/mcp"
        bearerMethodsSupported: [header]
  targets:
    - name: engram
      mcp:
        host: "http://memory-mcp.agent-memory.svc.cluster.local:8080/mcp"

Migration sequencing (phased, reversible)

LiteLLM stays live until the final phase; every phase is independently revertible.

P1 — Stand up agentgateway (agentgateway ns, llm-gw host): bootstrap the Vault path secret/fzymgc-house/cluster/agentgateway manually (vault kv put) with the OpenRouter key (reuse the existing one) + a generated embedder virtual-key value — out-of-band, mirroring how the LiteLLM OpenRouter key was set (litellm’s vault_kv_secret_v2 does not own the OpenRouter key, so Terraform can’t own the full payload); config.yaml with OpenRouter + Ollama providers, the or-*/ollama model aliases, virtual keys + rate limits (+ token budgets where standalone supports them), and the ZDR + drop_params CEL transformations; Deployment (v1.3.0-alpha.1, pin by digest; D10); ESO secrets; Ollama headless Service/EndpointSlice; add agentgateway to Velero excludedNamespaces in both the daily and weekly schedules (argocd/app-configs/velero/backup-schedule.yaml, D9); UI (if exposed) behind modern-auth. Verify /v1/* chat + embeddings, a virtual key, and that the zdr alias injects ZDR even when the client omits it.
P2 — Cut LLM traffic + engram embedder to llm-gw. Point the transitional litellm.fzymgc.house alias at llm-gw (D12). Repoint the engram embedder (memory-mcp-litellm ExternalSecret → bge-m3 virtual key; the engram Application’s valuesObject.memory.litellm.url → the agentgateway service). Verify chat, embeddings, and engram memory writes. (Revert = repoint to LiteLLM.)
P3 — Add the MCP route + auth to engram and the mcp-gw IngressRoute. Add the dedicated public PKCE client mcp-public in Authentik (new tf/authentik/mcp_public_oauth.tf) with the MCP clients’ loopback allowed_redirect_uris — additive, leaving the memory-mcp client untouched (D13). Add mcp-public to engram’s accepted-audiences (values). Configure agentgateway LocalMcpAuthentication: issuer/jwks/audiences/resourceMetadata for mcp-public, and client_id: mcp-public (DCR short-circuit). Apply the Terraform before routing any client at mcp-gw. Verify: the DCR short-circuit returns mcp-public; a client completes PKCE with Authentik (loopback redirect); agentgateway Strict-validates the JWT and forwards to engram; engram re-validates and attributes the call; an unauthenticated call → 401 with WWW-Authenticate pointing at the mcp-gw PRM.
P4 — Cut MCP traffic to mcp-gw. Verify. (Revert = repoint to litellm/mcp.)
P5 — Decommission LiteLLM: remove the transitional litellm.fzymgc.house alias, argocd/app-configs/litellm/, its Vault paths (after confirming no other consumer), and the ArgoCD Application. Remove the LiteLLM Authentik Terraform (tf/authentik/litellm.tf — OIDC provider, application, litellm-users/ litellm-admin groups, role-scope + policy bindings, random_password master key). Also delete tf/authentik/memory_mcp_oauth.tf — the memory-mcp client is orphaned once the broker is gone (engram now accepts the mcp-public audience); strip its litellm redirect/description instead if any non-broker consumer is found.

Alternatives considered

Two tools — Bifrost (LLM) + agentgateway (MCP) — rejected in favor of one. It required a custom dynamically-linked Bifrost image plus a Go .so plugin (PreLLMHook) with strict Go-version/dependency/libc ABI matching, rebuilt on every Bifrost upgrade, and a Postgres DB, solely to preserve the server-side ZDR guarantee. agentgateway’s CEL transformation delivers the same guarantee in config. Bifrost’s advantages (polished LLM admin dashboard, turnkey USD budgets, dynamic model-add UI, semantic cache) are mostly UI conveniences a GitOps cluster doesn’t lean on; budgets become token/cost-based instead of USD caps.
Bifrost full swap (incl. MCP) — rejected: Bifrost per_user_oauth is Enterprise-gated and Web-UI-only (breaks GitOps) and re-implements the broker pattern this design removes.
agentgateway K8s control-plane (Gateway-API CRDs) — rejected for this scope: AgentgatewayPolicy/Backend + Gateway-API CRDs are built for many routes; standalone config.yaml is simpler for one LLM proxy + one MCP upstream (D2).
No MCP gateway (expose engram via its own Authentik-protected ingress) — simplest, but loses centralized MCP visibility/control, a stated goal.
Keep LiteLLM for the LLM proxy — rejected: keeps a Python proxy and a second tool, and forfeits consolidation onto one tool-neutral data plane.

Testing strategy

LLM: per-provider smoke (OpenRouter chat, Ollama chat, bge-m3 embeddings); virtual-key allow/deny + token-budget + rate-limit enforcement; OTel metrics present.
ZDR transformation: assert the zdr alias forwards provider.zdr=true even when the client sends nothing; assert a non-ZDR alias does not.
drop_params transformation: an encoding_format=base64 embeddings call to Ollama succeeds (param stripped) rather than erroring.
MCP / engram: agentgateway serves /.well-known/oauth-protected-resource for mcp-gw; the DCR short-circuit (POST /register) returns the mcp-public client_id; a client completes PKCE with Authentik and gets a token; agentgateway Strict-validates it and forwards to engram, which re-validates; a store/search round-trip is attributed to the verified caller; unauthenticated → 401 with WWW-Authenticate referencing the PRM; expired token → rejected.
engram embedder: a memory write triggers a bge-m3 embedding through llm-gw successfully.
Rollback drill: confirm repointing a client back to LiteLLM restores service at each cut phase.