Use agentgateway as the unified LLM + MCP data plane (replace LiteLLM)
Date: 2026-06-06 Status: Accepted Decision: hl-hv3 Deciders: sean
Context
Section titled “Context”LiteLLM served as LLM proxy, MCP OAuth broker, virtual-key governance, and admin SSO. Its MCP broker pattern (delegate_auth_to_upstream token passthrough) became incompatible with the June 2025 MCP spec reclassifying MCP servers as OAuth Resource Servers (RFC 8707 resource indicators mandatory since 2026-03-15), and was chronically unreliable. engram already validates Authentik JWTs (the spec-correct Resource Server role). Two multi-tool alternatives were evaluated (Bifrost+agentgateway; Bifrost full-swap).
Decision
Section titled “Decision”Replace LiteLLM with agentgateway (Rust, Linux Foundation/Solo.io) as the single data plane for BOTH the LLM proxy and the MCP gateway, configured via a single standalone config.yaml.
Rationale
Section titled “Rationale”- agentgateway’s CEL request transformations deliver the ZDR guarantee and drop_params in pure config, eliminating the custom binary/plugin/Postgres the Bifrost path required.
- One data plane = one auth model, one observability pipeline, one GitOps artifact — directly serving the goal of centralizing LLM+MCP control.
- The spec-mandated RS pattern (engram validates JWTs itself) means the gateway only enforces JWT validity and forwards — agentgateway does this natively without a broker.
Alternatives Considered
Section titled “Alternatives Considered”- agentgateway standalone (CHOSEN) — single data plane/config/auth model; CEL transforms cover ZDR+drop_params in config; native MCP AS-shim + DCR short-circuit. Cost: lighter admin dashboard than Bifrost; budgets become token/rate-based not USD.
- Bifrost (LLM) + agentgateway (MCP) — rejected: ZDR required a custom dynamically-linked Go .so plugin (strict ABI matching, rebuild every upgrade) + a Postgres DB; two tools/configs/observability models.
- Bifrost full swap (LLM+MCP) — rejected: Bifrost per_user_oauth is Enterprise-gated AND Web-UI-only (breaks GitOps); re-implements the broker anti-pattern being removed.
- Keep LiteLLM for LLM + agentgateway for MCP — rejected: keeps a Python proxy and a second tool; forfeits consolidation.
Consequences
Section titled “Consequences”Positive: no custom image builds or Go-plugin ABI management on upgrades; one config.yaml is the GitOps source of truth for all LLM+MCP routing/policy/auth; unified OTel. Negative: per-key USD budget caps replaced by token/request rate limits (standalone limitation); no Bifrost admin UI / semantic cache. Neutral: phased migration (P1–P5), LiteLLM live until P5, every phase independently revertible.