Skip to content
fzymgc-house Cluster Docs

agentgateway is decommissioned — LiteLLM is the only LLM and MCP data plane; llm-gw and mcp-gw retired without alias; rollback is a revert plus a Vault undelete

Date: 2026-09-08 Status: Accepted — supersedes hl-m3c, hl-1pwl, hl-2jrh, hl-v4vo; dispositions hl-23m, hl-jd4 Deciders: Sean Brandt

Context

Section titled “Context”

The litellm-return milestone put LiteLLM back as the data plane in four phases. Phase 1 stood up the rig at llm.fzymgc.house and measured the five VAL gates that made it a go; Phase 2 landed the governing ADR (2026-08-15, reversing hl-hv3) and the supply-chain posture; Phase 3 minted the virtual keys and cut every LLM consumer over, leaving the agentgateway LLM lanes as a retained rollback plane. Phase 4 moved the ten MCP routes onto LiteLLM in three batches with mcp-gw.fzymgc.house kept up as each route’s rollback path (D-62, D-63), and closed on the finding that the relay had no client-facing OAuth contract of its own.

Phase 3 explicitly deferred the removal of the retained plane to this phase: D-43 (the LLM lanes stay until the MCP work is done), D-48 and D-49 (the accept-list overlap entries and the OVERLAP-BAKE-ENDS marker come out with agentgateway itself), and D-63 (mcp-gw is the rollback path for the duration of the bake). Those four decisions land here.

What closes the bake is operator attestation, not a traffic window (D-76). Each of the ten Claude Code routes is demonstrated on llm.fzymgc.house and signed off in the deletion PR’s checklist. The kubernetes identity route was attested on 2026-09-09 (Claude Code 2.1.266: discovery → Keycloak dynamic client registration → PKCE → tools/list → a real namespace read through the relay, with x-litellm-api-key beside the OAuth bearer and no static Authorization anywhere); the engram row is filled by the merge plan from the engram route’s own demonstration, which depends on an engram release that serves its own RFC 9728 document (seanb4t/engram#526) and runs after this record was written. This ADR therefore cites the checklist rather than a date it cannot yet know. The 2026-09-08 ClickStack baseline — the agentgateway LLM lanes at 0 requests over 7 days, the MCP lanes at ~25k requests/week — is recorded in the PR as the before-picture, not as a gate.

Decision

Section titled “Decision”

agentgateway is decommissioned in one deletion PR. LiteLLM at llm.fzymgc.house is the only LLM and MCP data plane. (D-71)

The PR removes: the two ArgoCD Applications (agentgateway, agentgateway-controller, both 1.3.1 charts); the argocd/app-configs/agentgateway/ and agentgateway-controller/ trees, including the OVERLAP-BAKE-ENDS marker and the three -litellm bypass entries; the agentgateway and agentgateway-system namespaces; the agentgateway scrape job in monitoring-otel-scraper; the agentgateway HyperDX dashboard, its panels and the alert tile that keyed on its metrics; the Terraform references (the two mcp-gw audience mappers and the oauth2-proxy dashboard entry in tf/keycloak, the uptime-kuma monitor); the mcp-gw entry in k3s_oidc_audiences; the two CoreDNS split-horizon rewrites; the Velero exclusions and the keycloak app’s AgentgatewayPolicy/ReferenceGrant; and the live docs. It removes nothing from the headroom-apps NetworkPolicy — an earlier revision of this list said it removed an llm-gw allowance there, and the diff (b9cc9ad3d) removes no rule from argocd/app-configs/headroom-apps/networkpolicy.yaml: it rewrites the egress comment that had justified the retired path, and leaves the broad toEntities: [world, cluster] port-443 rule standing. Narrowing that rule is a separate decision, not part of this decommission. The four ExternalSecrets that read secret/fzymgc-house/cluster/agentgateway are repointed at cluster/litellm (D-74) before the merge, and the agentgateway Vault path is soft-deleted only after every repointed reader reports SecretSynced. The Gateway API CRDs are not touched — the Cilium role vendors them, and they are not agentgateway’s.

llm-gw.fzymgc.house and mcp-gw.fzymgc.house are retired without alias. (D-72) This amends the litellm-return workstream’s D-09, which planned both names as compatibility aliases of llm.fzymgc.house at cutover. Every in-repo consumer is already on the canonical host, and the MCP path shape inverts (mcp-gw/mcp/<server> → llm.fzymgc.house/<server>/mcp), so an alias would preserve no client configuration. A straggler fails loudly with NXDOMAIN rather than quietly reaching a host that no longer validates anything.

The rollback is a git revert of the deletion PR plus vault kv undelete of the agentgateway path. (D-77) The path is removed with a KV v2 soft delete — never destroy, never metadata delete — so the deleted version, with the vk_* accept-list keys and llm_admin_key on it, persists indefinitely (the mount’s max_versions 0, delete_version_after 0s). The runbook section Rolling back the agentgateway decommission carries both commands.

The agentgateway ADRs get a disposition each — what happens to the decision, not a rewrite of its content. (D-73) No historical ADR file is modified; only the index’s Status cells flip.

hl-m3c — Deploy agentgateway via k8s-native Gateway-API/CRD control plane

Section titled “hl-m3c — Deploy agentgateway via k8s-native Gateway-API/CRD control plane”

Superseded by this record. The K8s-native control plane (agentgateway-crds + agentgateway charts, GatewayClass agentgateway, the agentgateway.dev/v1alpha1 CRDs) is removed with the controller. Nothing on the estate drives Gateway API resources for LLM or MCP traffic any more; LiteLLM is fronted by a Traefik IngressRoute.

hl-jd4 — Deploy agentgateway in standalone mode, not the K8s Gateway-API control plane

Section titled “hl-jd4 — Deploy agentgateway in standalone mode, not the K8s Gateway-API control plane”

Already superseded by hl-m3c; no change. Its header already reads Superseded by hl-m3c. The standalone deployment it described was replaced before this milestone began, and the replacement is now removed in turn.

hl-23m — agentgateway is the MCP auth enforcement point + DCR short-circuit

Section titled “hl-23m — agentgateway is the MCP auth enforcement point + DCR short-circuit”

Disposition already recorded in the 2026-08-15 ADR, cited here unchanged: already superseded by hl-12i8, its DCR short-circuit reverted by its own addendum, and not reproducible under LiteLLM. What replaced the short-circuit is now decided by Phase 5 D-66 and D-68: the resource server publishes its own RFC 9728 document, LiteLLM relays it, and Keycloak is the sole issuer.

hl-1pwl — Scrape agentgateway metrics via otel-scraper; push traces directly

Section titled “hl-1pwl — Scrape agentgateway metrics via otel-scraper; push traces directly”

Superseded by this record. The static agentgateway scrape job and the data plane’s OTLP trace push are gone with the data plane. LiteLLM emits its GenAI metrics over OTLP itself (gen_ai.client.operation.duration, gen_ai.client.token.usage, gen_ai.usage.cost — the instruments measured arriving), so the asymmetry hl-1pwl tracked no longer exists. Historical ServiceName='agentgateway' rows stay queryable in ClickHouse until their TTL.

hl-2jrh — Multi-route MCP OAuth discovery uses resource-specific well-known paths

Section titled “hl-2jrh — Multi-route MCP OAuth discovery uses resource-specific well-known paths”

Superseded by this record. The resource-specific path shape survives, but the mechanism it described — agentgateway synthesising a protected-resource document per route and proxying DCR to Keycloak — is retired. Discovery is now the resource server’s own document (/.well-known/oauth-protected-resource/<server>/mcp), relayed by LiteLLM with only resource rewritten (D-66, D-67, D-68). The DCR half is the 2026-09-08 DCR ADR: clients register directly against Keycloak under its loopback Trusted-Hosts policy (D-69).

hl-v4vo — Add dedicated transparent OpenRouter passthrough host on agentgateway

Section titled “hl-v4vo — Add dedicated transparent OpenRouter passthrough host on agentgateway”

Superseded by this record. openrouter-gw.fzymgc.house is retired with the other two hosts. The transparent passthrough is a LiteLLM lane (openrouter/*, with openrouter-zdr/* for ZDR) behind the openrouter-passthrough virtual key, scoped to those two wildcards; the dedicated accept-list that isolated the lane on agentgateway is replaced by that key’s models bound.

Rationale

Section titled “Rationale”
  • One PR is safe because D-77 makes it reversible by construction. Every removal in the PR is a tracked-file deletion that git revert -m 1 restores, and the one thing outside git — the Vault path — is removed with the one verb that is undoable. Splitting the removal into stages would buy nothing the revert does not already give, and would leave a controller running with nothing to serve between stages.
  • The measured inventory exceeded the decision’s list, and the PR enumerates it. D-71 named the apps, trees, scrape job, dashboards, Terraform references and CoreDNS rewrites; execution found the Velero exclusions, the keycloak app’s AgentgatewayPolicy and ReferenceGrant (which would have broken the keycloak app when the CRDs vanished), the three MCP NetworkPolicies admitting the agentgateway namespace, an orphan Traefik Middleware, and two more Vault readers (clickhouse-mcp, agent-memory). Each is in the PR with its reason.
  • Attestation over a traffic gate (D-76): the LLM lanes already read zero for a week before the decision, and the MCP lanes’ ~25k/week were the client cutover this phase performs — a window would have measured our own migration, not a regression.
  • No alias (D-72): the path inversion means a DNS alias cannot preserve a single MCP client entry, and the LLM consumers were all moved in Phase 3.

Alternatives Considered

Section titled “Alternatives Considered”
  • Staged removal in three PRs (LLM lanes, MCP routes, controller) — rejected: three merge boundaries, three reverts, and an interval with a controller and no data plane, for no gain over one revertible PR.
  • Keep llm-gw and mcp-gw as DNS aliases of llm.fzymgc.house (D-09’s plan; also the llm-gw-only variant) — rejected: no in-repo consumer needs either name, the MCP path shape inverts so the alias preserves nothing, and a silent alias hides stragglers that NXDOMAIN surfaces.
  • Destroy the agentgateway Vault path (vault kv destroy / metadata delete) — rejected: it throws away the vk_* accept-list and llm_admin_key that the revert would need, and the path costs nothing to keep as a soft-deleted version.
  • Declare no rollback — rejected: SC#1 requires a documented rollback, and the revert-plus- undelete shape is cheap.
  • A 7-day or 3-day zero-request window as the gate (D-76) — rejected: the LLM lanes were already at zero and the MCP lanes’ traffic was ours to move; a window would have delayed the merge without measuring anything the attestation does not.

Consequences

Section titled “Consequences”
  • LLM-02 parity cannot be re-measured two-sided. Harness lane B (the agentgateway comparator) is deleted with the plane; the LLM-02 evidence is frozen at the 03-18 session and the residuals that depended on lane B (R-6, R-21, R-22) close with it.
  • The kubernetes identity route validates tokens locally and depends on Keycloak. kubernetes-mcp now parses and verifies the relayed JWT against the realm’s JWKS (D-68 as measured), so a Keycloak outage answers 401 temporarily_unavailable on that route where the passthrough era validated nothing. One estate defect surfaced on the way: kubernetes-mcp v0.0.63 publishes server_url as authorization_servers and emits no resource, so server_url carries the Keycloak issuer (a recorded misuse of the key; PR #2046). That one is still open, and reopens on the upstream release that publishes a resource field. The config-hash gap it surfaced is closed. Both ConfigMap consumers now carry a digest of the content they mount on their pod template — checksum/config on Deployment/kubernetes-mcp (the sha256 prefix of config.yaml’s config.toml) and checksum/custom-config on Deployment/cs-otel-collector (the sha256 prefix of otel-collector-custom-config.yaml’s custom.config.yaml, which is the slot the ClickStack chart declares for that file; the chart’s own _helpers.tpl owns checksum/config on the collector and stamping ours there would overwrite it). A ConfigMap-only change therefore alters the pod template and ArgoCD rolls the pod, where #2046 needed a hand-run rollout restart. A gate (test_configmap_consumers_carry_a_config_checksum_annotation) recomputes both digests from the ConfigMaps, so an edit that forgets to bump one fails CI rather than shipping a ConfigMap the running pod never reads.
  • Historical agentgateway telemetry stays until TTL. ServiceName='agentgateway' rows in the ClickHouse metrics, log and trace tables are not deleted; the dashboards and tiles that read them are. The engram-embed brownout alert is rewritten on LiteLLM’s gen_ai.client.operation.duration, filtered by key alias and operation (D-80 as measured — the metric D-80 named literally, gen_ai.server.request.duration, is not emitted).
  • MCP method-level telemetry ended with the proxy. agentgateway’s MCP — calls by method panel read agentgateway_mcp_requests_total grouped by an method attribute, and nothing on the LiteLLM plane carries an equivalent. Measured on 24h of LiteLLM MCP relay spans (2026-09-10): the attribute keys present are http.*, net.*, litellm.api_key.hash and litellm.metadata.user_api_key_* — there is no MCP-method key, because LiteLLM RELAYS the JSON-RPC payload rather than parsing it, so the method name never leaves the request body and never reaches a span attribute. The panel therefore cannot be reproduced without new instrumentation on either LiteLLM or the upstream MCP servers. This is an accepted loss, not deferred work — nothing is scheduled to restore it. The other half of that coverage IS restored: litellm-mcp-calls-by-server on the LiteLLM dashboard counts POST /{mcp_server_name}/mcp relay spans grouped by the route segment, giving the same per-server call counts agentgateway’s MCP — calls by server gave and adding nothing it had. Reopen condition: an upstream LiteLLM release that emits an MCP-method span attribute, at which point the by-method panel becomes a one-tile addition rather than an instrumentation project.
  • The Keycloak audience roll has an order. The Terraform apply that drops the mcp-gw audience mappers must land before the k3s-config roll that drops the mcp-gw entry from k3s_oidc_audiences, so no live token carries an audience the API server rejects.
  • The tunnel-path DCR hardening stays deferred. Anonymous DCR from the internet reaches Keycloak through the Cloudflare tunnel (measured 2026-09-08); closing it with an Access or WAF rule is a later hardening pass, not part of this decommission.
  • Rollback is documented, not rehearsed. The revert and the undelete are the commands the runbook gives; the Terraform and Ansible halves (the Keycloak mappers, the k3s audience) are re-applied by hand after the revert and are not restored by it.

References

Section titled “References”