Design: agentgateway k8s-native (Gateway-API / CRD control plane)

• Bead: hl-0sr.25 (under epic hl-0sr)
• Date: 2026-06-06
• Status: Draft (pending design-reviewer gate)
• Author: Sean (with Claude)
• Supersedes: ADR hl-jd4 (D2 — “deploy agentgateway in standalone mode”)

Problem

agentgateway is currently deployed in standalone mode: a raw kustomize Deployment reading a single config.yaml (the llm:/mcp: shorthand), chosen in ADR hl-jd4 when the scope was one LLM proxy + one MCP upstream (engram).

That scope has grown. The MCP gateway must now front six servers with mixed, per-server client auth — engram via per-user OAuth/JWT, and exa/firecrawl/context7/fal via a static virtual key, deepwiki with none — and the operator wants future servers to pick OAuth or a pre-defined bearer per upstream. Standalone expresses client auth at the listener level (one mode per listener) and has no native path-based routing, so per-route/per-server auth requires an awkward verbose-config rewrite. The control-plane (CRD) model expresses exactly this natively: one HTTPRoute per server, each with its own AgentgatewayPolicy choosing the auth mode.

agentgateway’s own architecture doc states the Local (config.yaml) and XDS (control-plane) forms map to the same shared internal representation — so the CRD model is feature-complete with standalone, by design. This re-architecture moves us to the model that fits the grown requirement, and supersedes hl-jd4.

Goals

• Deploy agentgateway via its self-contained controller (Helm) driven by Gateway-API + agentgateway.dev CRDs (XDS), replacing the standalone Deployment/ConfigMap.
• Feature parity with the standalone config (no capability regressions): LLM proxy (OpenRouter + Ollama, ~20 model aliases, per-model TLS, bge-m3 embeddings, ZDR + drop_params CEL transforms, virtual-key auth), and the MCP gateway.
• Per-server MCP routing + per-server auth choice: one route per server under mcp-gw.fzymgc.house/mcp/<server>, each independently OAuth (engram) or VK/bearer (SaaS).
• Keep the existing tool-neutral hosts (llm-gw, mcp-gw, agentgateway UI) and Authentik integration.
• GitOps-native: the gateway is now Kubernetes resources (CRDs), not a monolithic config blob.

Non-goals

• Zero-downtime / parallel-build migration. The operator accepts the standalone gateway being left broken/behind during the cutover (direct replacement, not blue/green).
• Installing kgateway. agentgateway’s own controller is self-contained.
• Changing engram’s auth model (stays an OAuth Resource Server) or the SaaS upstreams’ own APIs.
• Re-litigating which MCP servers exist (the six are fixed inputs from LiteLLM’s DB).

Architecture

The agentgateway controller (Helm chart agentgateway, CRDs chart agentgateway-crds, self-contained, registers GatewayClass agentgateway) watches:

• Standard Gateway API (gateway.networking.k8s.io/v1): Gateway (listeners), HTTPRoute (path routing).
• agentgateway CRDs (agentgateway.dev/v1alpha1): AgentgatewayBackend (LLM + MCP backends), AgentgatewayPolicy (auth, transformations, backend auth), AgentgatewayParameters (data-plane pod shape).

The controller provisions + XDS-configures the data-plane pod(s). All resources are delivered via ArgoCD. The cluster already has base Gateway-API CRDs (a cilium GatewayClass exists for regular traffic and is untouched); we add the agentgateway GatewayClass alongside it.

Key local→XDS difference: the standalone local config can fetch JWKS from a URL inline; the XDS/CRD path deliberately does not fetch URLs — references are resolved via the control plane. So JWT/JWKS and any external upstream are expressed as backendRefs to a Backend/Service (e.g. the Authentik JWKS host, K14), with the path supplied separately (jwksPath). This is the main structural delta from the standalone config and is reflected throughout the decisions below.

Decisions

#	Decision	Rationale
K1	Deploy via agentgateway’s self-contained controller (Helm agentgateway-crds + agentgateway), GatewayClass agentgateway. Supersedes hl-jd4 (standalone).	The CRD/XDS model maps to the same IR as local config (parity) and natively expresses per-route auth + path routing — the grown MCP requirement. No kgateway dependency.
K2	Two ArgoCD apps: (a) agentgateway-controller (CRDs + controller charts), (b) agentgateway (the Gateway + CRs). CRDs/controller sync first (sync-wave).	Separates the platform (controller) from the workload (our gateway config); CRDs must exist before the CRs that use them.
K3	Pin the controller + data-plane images by digest (alpha; carries D10’s conscious alpha-on-prod stance). Chart agentgateway is 0.0.2.	Fast-moving alpha; reproducibility. Revisit at GA.
K4	One Gateway (gatewayClassName: agentgateway) with listeners for hosts llm-gw, mcp-gw, agentgateway, plus the transitional litellm.fzymgc.house alias on the LLM listener (carried from the standalone cutover so existing clients on the old host still reach the LLM proxy); TLS via the existing cert-manager agentgateway-tls cert (SANs already include all four hosts); pod shape via AgentgatewayParameters (image pin, resources, non-root securityContext, ADMIN_ADDR=0.0.0.0:15000 for the UI). The litellm.fzymgc.house alias is removed at final LiteLLM decommission (hl-0sr.21).	One gateway, four hostnames, mirroring the current ingress contract incl. the live alias so the migration doesn’t silently break old-host clients.
K5	LLM backends: AgentgatewayBackend openrouter (spec.ai.provider.openai, host openrouter.ai:443, path /api/v1/chat/completions, backendTLS) + ollama (192.168.218.96:11434, /v1/...). The ~20 OpenRouter aliases + embeddings via policies.ai.modelAliases / route types. The standalone ollama/* chat wildcard is NOT carried — the CRD OpenAIConfig.model is a fixed string with no wildcard routing, and the wildcard had a prefix-forwarding bug (hl-0sr.23) with no confirmed chat consumer. Only ollama/bge-m3 (embeddings) is migrated; any required Ollama chat models are enumerated as explicit aliases (none currently in use — add per-model if needed).	Direct CRD analog of standalone llm.models; the wildcard has no CRD equivalent, so chat models become explicit (parity gap resolved by enumeration, not wildcard).
K6	LLM routes + auth: HTTPRoutes for /v1/chat/completions, /v1/embeddings, /v1/models on llm-gw; one AgentgatewayPolicy with traffic.apiKeyAuthentication (mode Strict, secretRef → VKs vk_claude_code + vk_engram_embedder).	Preserves the virtual-key gate from the standalone cutover.
K7	ZDR via AgentgatewayBackend.spec.policies.ai.overrides ([]FieldDefault — the direct CRD analog of the standalone overrides.provider.zdr=true, a structured field-merge, not CEL) on the ZDR backend. drop_params (encoding_format) via AgentgatewayPolicy.traffic.transformation.request.body CEL on the embeddings route (CEL is the removal mechanism; field-default can’t delete). The plan validates both with kubectl --dry-run=server against the live CRDs before authoring.	ai.overrides is the spec-correct home for a static body field (matches standalone); CEL is reserved for the param removal which overrides can’t express.
K8	MCP: one AgentgatewayBackend + HTTPRoute + AgentgatewayPolicy per server, path mcp-gw.fzymgc.house/mcp/<server>.	Operator’s requirement: maximal per-upstream independence (path, auth mode, upstream key).
K9	engram route = OAuth: AgentgatewayPolicy.traffic.jwtAuthentication with one provider (issuer/audiences = mcp-public, Authentik, provider enum omitted), mcp.clientId mock-DCR short-circuit + mcp.resourceMetadata. JWKS is jwks.remote.{jwksPath, backendRef}, NOT a flat url — XDS (unlike local config) cannot fetch URLs, so the Authentik JWKS host needs a Backend/Service the backendRef points to (see K14); jwksPath is the path-only component (/application/o/mcp-public/jwks/). engram backend target → memory-mcp.agent-memory.svc:8080/mcp (StreamableHTTP).	Keeps engram the spec-correct OAuth Resource Server; clientId fills Authentik’s DCR gap (standalone D3/D13). The jwks split is the key local→XDS difference.
K14	Backend for the Authentik JWKS host: a static AgentgatewayBackend (or Service) for auth.fzymgc.house so K9’s jwks.remote.backendRef can resolve it (XDS resolves refs, not URLs).	Required because XDS does not fetch JWKS over arbitrary URLs the way the standalone local config did.
K10	SaaS routes = VK + per-target upstream auth: exa/firecrawl/context7/fal each traffic.apiKeyAuthentication (client = vk_claude_code) + backend.auth.secretRef injecting that server’s key (Authorization: Bearer <key>) to the HTTPS upstream; backendTLS. deepwiki = VK client auth, no upstream key.	Static-bearer client path the operator wants for non-engram; SaaS keys stay server-side (from Vault).
K11	Secrets via ESO (all values already in Vault secret/fzymgc-house/cluster/agentgateway), in two distinct Secret shapes the CRDs require: (a) client VK accept-list for traffic.apiKeyAuthentication.secretRef — one entry per key in agentgateway’s per-key JSON format ({"key":"…","metadata":{…}}) covering vk_claude_code + vk_engram_embedder; (b) per-SaaS upstream-auth Secrets for each backend.auth.secretRef — a Secret whose Authorization key holds the raw bearer (Bearer <exa_api_key> etc.), one per SaaS backend (exa/firecrawl/context7/fal). OpenRouter’s key stays the LLM backend’s policies.auth.key.	The two auth surfaces consume differently-shaped Secrets; conflating them mis-authors the keys. ESO renders each shape from the same Vault path.
K12	Decommission the standalone agentgateway kustomize app (Deployment/ConfigMap/Service/IngressRoute) as part of this migration; keep the namespace + cert. Broken interim accepted.	Direct replacement; no parallel run (per operator).
K13	Authentik unchanged for the LLM/MCP OAuth (existing mcp-public PKCE client + agentgateway proxy app for the UI). The UI route moves under the new Gateway with the same modern-auth forward-auth.	Reuse the working Authentik integration; only the data-plane delivery changes.

Component map (CRDs)

Gateway/agentgateway (gatewayClassName: agentgateway)
├─ AgentgatewayParameters/agentgateway-params      # data-plane pod shape
├─ listeners: llm-gw, mcp-gw, agentgateway (+ litellm.fzymgc.house alias on LLM) (TLS: agentgateway-tls)
├─ AgentgatewayBackend/authentik-jwks (auth.fzymgc.house)   # K14: backendRef target for engram jwks
│
├─ LLM
│  ├─ AgentgatewayBackend/openrouter (ai.provider.openai, backendTLS)        # ~20 aliases
│  ├─ AgentgatewayBackend/ollama     (ai.provider.openai → 192.168.218.96:11434)  # bge-m3 embeddings; NO chat wildcard
│  ├─ HTTPRoute/llm  (/v1/chat/completions, /v1/embeddings, /v1/models → backends)
│  └─ AgentgatewayPolicy/llm  (apiKeyAuthentication VKs + ai.overrides ZDR + transformation drop_params + modelAliases)
│
└─ MCP (per server: Backend + HTTPRoute /mcp/<server> + Policy)
   ├─ engram     → memory-mcp.agent-memory.svc:8080/mcp   | jwtAuthentication.mcp (issuer/audiences mcp-public, jwks.remote.backendRef→authentik-jwks, clientId)
   ├─ exa        → mcp.exa.ai/mcp?tools=…   (backendTLS)  | apiKeyAuthentication(VK) + backend.auth(exa_api_key)
   ├─ firecrawl  → mcp.firecrawl.dev/v2/mcp (backendTLS)  | apiKeyAuthentication(VK) + backend.auth(firecrawl_api_key)
   ├─ context7   → mcp.context7.com/mcp     (backendTLS)  | apiKeyAuthentication(VK) + backend.auth(context7_api_key)
   ├─ fal        → mcp.fal.ai/mcp           (backendTLS)  | apiKeyAuthentication(VK) + backend.auth(fal_api_key)
   └─ deepwiki   → mcp.deepwiki.com/mcp     (backendTLS)  | apiKeyAuthentication(VK), no backend auth

Migration approach

Direct replacement (operator accepts a broken interim):

1. Install the controller (CRDs chart, then controller chart) — new infra, no effect on the running standalone yet.
2. Author the Gateway + AgentgatewayParameters + LLM/MCP backends/routes/policies + ESO.
3. Point the hosts (llm-gw, mcp-gw, agentgateway) at the new Gateway.
4. Delete the standalone kustomize app.
5. Repoint claude-code: LLM → llm-gw (vk_claude_code); MCP → mcp-gw.fzymgc.house/mcp/<server> per server (engram OAuth; exa/firecrawl/context7/fal/deepwiki with vk_claude_code).

Testing / verification

• Controller healthy; Gateway Programmed=True; data-plane pod Ready.
• LLM: no-auth → 401; vk_claude_code chat (OpenRouter + Ollama) → completion; bge-m3 embeddings (base64 dropped) → vector; ZDR model injects provider.zdr=true (verify via pod log / OpenRouter).
• MCP per server: /mcp/engram → 401 then OAuth round-trip works; each SaaS /mcp/<server> → tools list/call with vk_claude_code + the upstream key applied.
• UI: agentgateway.fzymgc.house/ui → Authentik → loads.

Risks

• Alpha controller (chart 0.0.2) — immaturity; mitigated by digest pinning + the existing alpha-on-prod acceptance (D10). Revisit at GA.
• CRD field drift — the spec’s field paths are grounded against the cloned controller (agentgateway.dev/v1alpha1) but the alpha API may shift; the plan re-verifies field names before authoring and validates with kubectl --dry-run=server against the live CRDs.
• UI dashboard — the standalone UI had a configMapper bug on shorthand configs (hl-0sr.24); the CRD/XDS path may render differently — verify, not assumed.
• drop_params CEL — removing a body key via CEL must be confirmed against the alpha’s CEL surface during the plan.