LiteLLM, not agentgateway, is the unified LLM and MCP data plane — reversing hl-hv3 on measured per-key USD budget enforcement, with the MCP auth model left undecided because it was never observable

Date: 2026-08-15 Status: Accepted — supersedes hl-hv3 Deciders: Sean Brandt

Context

ADR hl-hv3 (2026-06-06) decided to replace LiteLLM with agentgateway as the single data plane for both the LLM proxy and the MCP gateway. It recorded, as a Negative consequence, that “per-key USD budget caps replaced by token/request rate limits (standalone limitation)”. That negative turned out to be the thing the estate actually needed, and the litellm-return milestone exists to reverse it.

Phase 1 of that milestone ran a measurement spike against LiteLLM v1.96.2, digest-pinned, on llm.fzymgc.house. Its verdict, verbatim from SPIKE-REPORT.md:

GO for Phase 2 and Phase 3, with five conditions. NOT GO for planning Phase 4 as if the MCP auth matrix were validated.

Both halves are load-bearing and this ADR states both. The LLM lane was measured and works. The MCP auth lane was not measured at all, and the reason is a single blocking fact:

GET https://llm.fzymgc.house/v1/mcp/server  ->  200  []
proxy_config in argocd/app-configs/litellm-chart/values.yaml has no `mcp_servers` key

The proxy under test registered zero MCP servers. No route’s behaviour was ever observed. VAL-02’s per-route auth-mode matrix is therefore BLOCKED, not tested — every row of it is a design intent with an evidence pointer, and the spike says so in those words. Phase 4 must budget for measuring that matrix from scratch, on a proxy that has MCP servers registered.

This ADR is written to a deliberately narrow claim boundary for that reason: it reverses hl-hv3 on the evidence that exists, dispositions two adjacent ADRs, and records the measured constraints that later phases must plan against. It decides nothing about the MCP auth model.

Decision

LiteLLM is the unified LLM and MCP data plane for this cluster. ADR hl-hv3 is reversed and marked superseded by this record.

Two adjacent ADRs get a disposition only — what happens to each decision, not a replacement for its content. Neither file is modified by this change.

hl-23m — agentgateway is the MCP auth enforcement point + DCR short-circuit

Its header reads **Status:** Accepted today, but that is already stale for a reason that predates this milestone: hl-12i8 (Migrate cluster IdP from Authentik to Keycloak, 2026-06-28) records in its own Consequences that it “supersedes ADR hl-23m”. The supersession was recorded on the superseding side only and was never written back into hl-23m’s header. Reconciling that stale header is deferred repo-infrastructure work and is not done here; it is recorded so the next reader does not rediscover it as new.

Separately, hl-23m’s own Addendum of 2026-06-28 already reverted the mock-DCR short-circuit as structurally unusable against Authentik, standardising on the static mcp-public PKCE client.

Disposition, then, in three parts: already superseded by hl-12i8; its DCR short-circuit already reverted by its own addendum; and independently not reproducible under LiteLLM, whose inbound authorization-server advertisement is hardcoded to the gateway itself (see Consequences item 2). What replaces the short-circuit is undecided until Phase 4 measures it.

hl-nmix — VK + static upstream bearer for Firewalla MCP

Its header reads **Status:** Accepted, with no supersession claim recorded anywhere. The choice survives a change of gateway. Authenticating gateway clients with a virtual key and injecting a static bearer to the upstream is expressible on LiteLLM as the bearer_token auth mode, which is a real MCPAuth enum member in 1.96.2. hl-nmix’s content stands unchanged; only the gateway underneath it changes.

Rationale

The reversal’s strongest ground is the negative consequence hl-hv3 recorded against itself. hl-hv3 accepted losing per-key USD budget caps in exchange for consolidation. VAL-04 measured LiteLLM 1.96.2 hard-blocking per-key USD budgets on all four key paths, with real OpenRouter spend:

Key type	Mint shape	Blocked at	Spend at block
personal	user_id	429	$0.01200675 against a $0.01 ceiling
team	team_id	429	$0.01200585 against a $0.01 ceiling
custom model	models: ["gpt-4o-mini"]	429	$0.012006 against a $0.01 ceiling
standard model	models: ["openrouter/*"]	429	$0.0120066 against a $0.01 ceiling

The empty case is clean — a fresh key reports spend 0.0 and its first call returns 200. Spend accuracy measured /key/info $0.012006 against provider cost $0.012006, 0.0 % drift against a 10 % tolerance. All four documented bypass bugs (#26672, #27394, #25874, #12789) held.

Two supporting measurements, neither of which is a budget claim:

• OTel GenAI telemetry lands in ClickStack — 21642 gen_ai.client.* histogram rows and 792 trace rows, from results/otel-arrival.json.
• The block is HTTP 429, not the 403 the spike plan specified. Recorded as a criteria departure, not waived: a loop written to the plan’s letter never terminates, because it spends to its iteration cap against a budget check that is working correctly.

Alternatives Considered

• Reverse hl-hv3 and disposition hl-23m and hl-nmix only (CHOSEN). Claims exactly what was measured. Leaves Phase 4 free to decide the MCP auth model against real measurements. Cost: the MCP auth model stays undecided in writing, so Phase 4 opens with an open question rather than a decision.
• Also decide the MCP auth model now, from VAL-02’s route table — rejected. Every row of that table is a design intent with an evidence pointer, not a measurement; the proxy registered zero MCP servers, so no route behaviour was observable. An ADR asserting those rows would read as decided-and-verified downstream, and MCP-02 would be scheduled on the belief the matrix was tested. An ADR merged to main is a published record; correcting an over-claiming one costs a second ADR amending the first, after downstream phases have already planned against the wrong statement.
• Stay on agentgateway and accept token/request rate limits instead of USD ceilings — rejected. This is hl-hv3’s own recorded Negative, and it is the specific thing this milestone exists to undo.

Consequences

1. The OAuth discovery paths stay unpublished at the ingress — as intent, not as an accident

The conclusion stands; two of the three things this section said in support of it do not. Corrected 2026-08-30 by plan 04-02 under decision D-59. The superseded wording is quoted below rather than repaired, because Phases 2 and 3 planned against it and a later reader is owed the claim that was acted on.

Recorded here on 2026-08-15	Status	Why
“ingress.yaml publishes exactly five matchers: /v1/, /health, /key/, /spend/, and PathRegexp(`^(/[^/]+)?/mcp/?$`)”, and “those five matchers are genuinely all that is published”	SUPERSEDED	Commit 9193ee883 (#1975), “publish the root and state the exclusions instead”, replaced the allowlist with a denylist. There is no list of published matchers left to name.
“publishing /register on an internet-facing host creates an unauthenticated write path”	premise falsified	llm.fzymgc.house is discoverable from the internet but not internet-reachable, measured by Phase 3 (03-23-SUMMARY.md:113-123, threat flag :267). The conclusion survives on other grounds; the perimeter it invoked does not exist.

The original measurement stands as taken and is not withdrawn — at the time it was made, those two paths did return Traefik’s plain-text 404 rather than LiteLLM’s JSON, so the ingress was refusing them and not the application:

GET https://llm.fzymgc.house/.well-known/oauth-protected-resource   ->  404 "404 page not found"
GET https://llm.fzymgc.house/.well-known/oauth-authorization-server ->  404 "404 page not found"

The mechanism, as it actually stands

Since 9193ee883 the IngressRoute carries two rules, and the discovery paths are unpublished by exclusion rather than by omission from an allowlist:

• Host(`llm.fzymgc.house`) && PathPrefix(`/guardrails/`) at priority 100; and
• Host(`llm.fzymgc.house`) && PathPrefix(`/`) at priority 1, minus five exclusion clauses:

&& !PathPrefix(`/login`)
&& !PathPrefix(`/fallback/login`)
&& !PathPrefix(`/.well-known/oauth-`)
&& !PathPrefix(`/.well-known/openid-configuration`)
&& !PathRegexp(`^(/[^/]+)?/(register|authorize|token)(/complete)?$`)

Those five clauses group under two recorded decisions: the master-key password form (/login, /fallback/login), and anonymous dynamic client registration together with the OAuth endpoints it advertises. The practical consequence for any later phase: exposing discovery means editing an exclusion clause, not adding a matcher. Adding an ordinary surface needs no ingress change at all; adding an exclusion is what needs a recorded decision.

The /guardrails/ rule at priority 100 must keep outranking the root rule. /guardrails/register is the single legitimate /<segment>/register in the product and it matches the one-segment DCR regexp. Traefik’s regexp engine is RE2 and has no negative lookahead, so “any segment except guardrails” cannot be written as one expression; and Traefik’s default ordering is by rule length, under which the longer root rule would swallow it. Any future widening of an exclusion must preserve that ordering, and must preserve both /login exclusions.

The network position, measured

All three parts of it are load-bearing, and the conclusion is stated rather than left to be inferred:

1. llm.fzymgc.house has a public, grey-clouded DNS record — created by external-dns from the IngressRoute’s external-dns.alpha.kubernetes.io/target annotation, not declared in Terraform. 1.1.1.1 and 8.8.8.8 both answer it.
2. It resolves to RFC1918 192.168.20.145, the Traefik MetalLB VIP.
3. It is absent from the Cloudflare tunnel — tf/cloudflare/tunnel.tf publishes id.fzymgc.house and *.fzymgc.net behind a catch-all http_status:404.

So the host is discoverable from the internet but not internet-reachable. The hostname is additionally in public CT logs, irreversibly. This does not weaken any control here; it means the control’s justification is the LAN and tailnet attacker set, not a perimeter.

Which of the three arguments survive

Argument as recorded	Disposition
(a) an anonymous write path on an internet-facing host	Retired — this is the premise falsified above. The perimeter it assumed does not exist.
(b) publishing /register makes the gateway a second token issuer beside Keycloak	Survives, and it is the only one that does. It is architectural rather than perimeter-dependent: a second issuer is a second issuer on a LAN. Confirmed still standing after the D-58 brokering question was answered — see the Phase 4 consequence below.
(c) the master-key password form at /login	Unaffected and still correct. A separate exclusion under a separate decision; it was never a discovery-path argument.

The write path is real, not theoretical. Measured in-cluster on 2026-08-29, an unauthenticated POST /register returned 201 with a real client_id; token_endpoint_auth_methods_supported includes none. What contains it is the exclusion clause, not the network position.

Today that is a recorded decision rather than accidental containment — which is what this section set out to establish, and the only part of it that needed no correction. The minimum bar before publishing is an access policy in front of it plus a decision on what bounds the client table. Not in v1.

2. Correction: inbound DCR is NOT absent from LiteLLM OSS

This is a correction to a belief this milestone carried in writing, not a fact that was always known. Every planning artifact recorded inbound DCR as unsupported — research/FEATURES.md:180 (“NOT SUPPORTED — open feature request #31296”), research/SUMMARY.md:49, and STATE.md:51. That statement is superseded by measurement. Against 1.96.2 over a port-forward:

Element	Status in 1.96.2	Evidence
RFC 9728 protected-resource metadata	present	mcp_server/discoverable_endpoints.py:2309, 2340, 2360-2361
RFC 7591 registration endpoint routes	present	discoverable_endpoints.py:2530-2531
Spec-shaped 401 with WWW-Authenticate: Bearer resource_metadata="…"	present	mcp_server/auth/user_api_key_auth_mcp.py:223-253
RFC 8414 authorization-server metadata	present	served at /.well-known/oauth-authorization-server/mcp
authorization_servers configurable to an external IdP	ABSENT	hardcoded at discoverable_endpoints.py:2273; no config knob exists

RFC 9728, RFC 8414 and RFC 7591 all serve. The single remaining gap is that the advertised authorization server is hardcoded to the gateway itself, so LiteLLM cannot point a client at Keycloak — a spec-compliant client that auto-discovers LiteLLM is steered into LiteLLM’s own sign-in and token flow. grep -rn "mcp_protected_resource" over the installed package returns nothing: the config block issue #31296 proposes does not exist.

Why this matters enough to write down: a v2 decision costed against the old statement would size the work wrongly. It would budget for building a discovery stack that already exists, and would miss that the actual blocker is one hardcoded list plus a second-IdP architecture question.

3. Virtual-keys-only is a REGRESSION in MCP client onboarding, not parity

For v1, MCP clients (Claude Code, Cursor, VS Code) get a virtual key pasted into their config instead of an auto-discovered sign-in. Against agentgateway as it stands today — which short-circuits Keycloak DCR per hl-23m — that is a regression in onboarding ergonomics. It is recorded here as a regression rather than presented as parity. What it does not cost: nothing in the v1 requirement set needs inbound DCR, and AUTH-03/AUTH-04 are already deferred to v2.

Amended 2026-08-30 (plan 04-02, Phase 4 research against the pinned 1.96.2 source). The mechanism that would retire this regression — JWT-to-virtual-key mapping via general_settings.enable_jwt_auth plus litellm_jwtauth.virtual_key_claim_field — is not absent from the product. It is present and Enterprise-gated:

# litellm/proxy/auth/user_api_key_auth.py:1199-1210
if general_settings.get("enable_jwt_auth", False) is True:
    is_jwt = jwt_handler.is_jwt(token=api_key)
    if is_jwt:
        from litellm.proxy.proxy_server import premium_user
        if premium_user is not True:
            raise ValueError(
                f"JWT Auth is an enterprise only feature. {CommonProxyErrors.not_premium_user.value}"
            )

premium_user is _license_check.is_premium() (proxy_server.py:682), and this deployment sets no LITELLM_LICENSE, so it is False. The regression is therefore a licensing position, not a product gap — which changes how it must be read, not whether it stands. MCP admission runs through the same proxy-wide user_api_key_auth (user_api_key_auth_mcp.py:418, 482, 495), so nothing about MCP routes is special here.

This is the third instance of the same shape in this milestone, and it should be recognisable by now rather than re-derived: POST /key/regenerate (Phase 3, residual R-9 — an Enterprise gate at key_management_endpoints.py:4771-4778 found only after research had concluded YES from four correct source paths sixty lines below it), then model_max_budget, now enable_jwt_auth. Read “not supported” in this milestone’s artifacts as “not measured against the premium gate” until someone has looked.

The single reopen condition is a LITELLM_LICENSE, and nothing else. No config change, no version bump within 1.96.x, and no MCP-specific setting reopens it.

The regression is narrowed but not retired, and 04-05 settles by how much. The working identity path measured in Phase 4 is the dual-header contract — x-litellm-api-key carrying the admission credential (whose value must itself be prefixed Bearer, space included — measured in 04-01) alongside Authorization carrying an unrelated upstream token via extra_headers. That contract still pastes a virtual key into a client config, so the ergonomics half of this regression survives even on the best available path. What 04-05 measures is the identity half.

4. A budget is a bound, not a cap (spike condition C2)

Enforcement is eventual, not per-request. LiteLLM batches spend writes; flush latency measured 4.2–10.5 s across runs (6.2–8.4 s in the committed snapshot), and that latency is the width of the window in which an over-budget key still serves traffic. Observed directly during calibration: a call took spend to $0.0109 against a $0.01 ceiling, the next call was admitted, and only the one after that was refused.

A max_budget therefore bounds spend to roughly ceiling + (burn rate × ~10 s). At homelab request rates that is cents. It is not zero, and it is not a hard cap.

The trap, recorded so it is not misread: calls_after_ceiling_crossed reads 0 in the committed run because the loop waits for the flush rather than racing the writer. That zero is a property of the test’s pacing, not a property of the system.

5. MCP session ids are NOT validated in 1.96.2 (spike condition C3)

Probe	Expected	Measured
Session id never issued	400 session ID not found	200 with a valid JSON-RPC result
Session id from replica A used on replica B	per-process, refuse	200, accepted

There is no session validation at all — there is no 400 edge case to capture because there is nothing to fail. No authorization boundary may be built on MCP session state on this release. This is a hard constraint on Phase 4’s design. Bearer auth is still enforced independently: an unauthenticated POST /mcp/ returns 401 authentication_required, verified through a port-forward.

6. The shared virtual-key auth cache was OFF in production, for a second reason beyond D-07

argocd/app-configs/litellm-chart/values.yaml:138 set enable_redis_auth_cache: true under general_settings, and LiteLLM 1.96.2 reads that flag only from litellm_settings (proxy/proxy_server.py:4092, 4475, 7826). Both replicas logged the negative branch verbatim:

proxy_server.py:3757 - enable_redis_auth_cache is not set: user_api_key_cache remains
  in-memory only (per-worker). Set litellm_settings.enable_redis_auth_cache: true ...

So the virtual-key auth cache was per-worker in-memory on both replicas — while spend counters and cross-pod coordination, which attach unconditionally, ran on a cache that was itself only about one-third working (D-07: a plain Redis client against a cluster-mode Valkey, measured 3/9 successful round-trips against 9/9 for a cluster-aware client). Both defects were compounding on the same traffic. Plan 02-01 corrected both — enable_redis_auth_cache relocated to litellm_settings, and cache_params.redis_startup_nodes seeded so a cluster-aware client is constructed.

This is the second half of why FND-02’s “shared auth cache from day one” was not true. Recording D-07 alone tells half the story.

7. The failure class this milestone keeps hitting

Three stated criteria in this milestone would have been satisfied by a gate asserting the wrong thing — the PathSuffix matcher that Traefik v3 dropped silently while a green static gate asserted the broken rule as correct, and the two success-criterion rg assertions that are red on a correct tree. Gates in this milestone assert on parsed properties, not on file text, for that reason.

8. Recorded, not acted on: three sibling apps interpolate DB passwords unescaped

octopus, miniflux and mealie each interpolate their database password into a connection string inside their ExternalSecret with no percent-encoding (argocd/app-configs/octopus/secrets.yaml:20, miniflux/secrets.yaml:24, mealie/secrets.yaml:20). This works only while those passwords happen to be encoding-safe. Out of scope for this phase by operator ruling — each carries its own rotation risk. Recorded here so it is not rediscovered as new.

9. Accepted risk: the single-source DB credential path is untested end to end (D-24)

Phase 2 moved the LiteLLM database password to exactly one Vault property and composed DATABASE_URL in the cluster (ARCH-02), and it deliberately did not rotate a live password to demonstrate it — D-24: the structural change is the evidence, and single-sourcing is provable by reading the configuration. The accepted risk that buys is that nothing here has been exercised by a value actually changing. The ExternalSecret template, its urlquery escaping and the CNPG reconnect path are untested end to end as of this phase.

The escaping in particular is held only statically: urlquery renders a space as +, which inside a URL userinfo component is a literal plus and would authenticate wrongly with no parse error anywhere, so the seed script’s alphabet is pinned by a gate rather than proven at runtime. The UAT items that would exercise the path (Phase 2’s 02-UAT.md tests 3 then 4, in that order) are pending, and Phase 3’s KEY-02 rotation runbook work is the first place a value changes for real. Treat that as the proof, not as a repeat of a proven procedure.

Mode-vocabulary correction a Phase 4 author will hit

The MCPAuth enum members in 1.96.2 are true_passthrough, oauth_delegate, bearer_token and api_key. Earlier planning vocabulary in this milestone named two strings that are not members: bearer (the member is bearer_token) and delegate_auth_to_upstream (not an enum member at all — it is a separate boolean field; the nearest member is oauth_delegate). A Phase 4 route written in the planning vocabulary is rejected at config load.

Phase 4 consequence (2026-08-30): D-58 resolves NEGATIVE — the discovery exclusions stay

Phase 4’s decision D-58 made lifting the §1 exclusions conditional on one question: does LiteLLM’s MCP OAuth broker to Keycloak? Research answered it from the pinned 1.96.2 source. The verdict is the same as before the question was asked, so the evidence is recorded rather than only the conclusion — a reader who sees an unchanged outcome should be able to tell that somebody looked.

The answer splits, and which /authorize you mean decides it.

Endpoint	Behaviour	Source at 1.96.2
Per-server /{mcp_server_name}/authorize	Resolves the named server, calls _raise_if_not_oauth2(mcp_server), and redirects to that server’s own upstream OAuth provider. It does not touch the generic SSO at all.	discoverable_endpoints.py:1646-1709
Aggregate /authorize with a gateway-DCR client_id	aggregate_authorize does chain user authentication to LiteLLM’s SSO — RedirectResponse to /sso/key/generate when there is no session user — and /sso/… is D-41’s Keycloak generic-OIDC wiring.	gateway_dcr_flow.py:357-359

Why the aggregate flow is still not brokering. LiteLLM chains authentication to Keycloak; it does not delegate issuance. It mints its own DCR clients and its own session bearers, and its protected-resource metadata names the gateway itself as the authorization server — "authorization_servers": [f"{request_base_url}/{mcp_server_name}"] (discoverable_endpoints.py:2148) and [f"{request_base_url}/mcp"] (:2273). That is §2’s finding still standing; only the line numbers moved. A spec-compliant client that auto-discovers this host is steered into LiteLLM’s own token flow, which is precisely the second-issuer condition §1’s surviving argument names.

A further cost the aggregate flow carries: it lands the browser on /ui/connect — the Admin UI — so taking that branch would put an admin surface on the ordinary client path.

Decision. D-58 takes its negative branch:

• The ingress exclusion clauses in argocd/app-configs/litellm/ingress.yaml are unchanged. Plan 04-02 touches no manifest, so there is no diff there to go looking for.
• MCP clients get static Keycloak client configuration against the existing mcp-public PKCE client (tf/keycloak/mcp_public.tf — public client, PKCE S256, loopback redirect URIs, audience mapper forcing aud=mcp-public). No new Keycloak object is created.
• §1’s threat restatement follows from this: T-02-19 in Phase 2’s security register is re-justified on the second-token-issuer argument, which this measurement confirms rather than retires.

What would reopen it: a configurable authorization_servers (upstream BerriAI/litellm#31296), or a deliberate decision to accept a second IdP in the estate. Neither is v1.

Supersession

• Supersedes: hl-hv3

References

• hl-hv3 — Use agentgateway as the unified LLM + MCP data plane (replace LiteLLM) — reversed by this record
• hl-23m — agentgateway is the MCP auth enforcement point + DCR short-circuit — disposition only
• hl-nmix — Use VK + static upstream bearer auth for Firewalla MCP (not per-user OIDC) — disposition only
• hl-12i8 — Migrate cluster IdP from Authentik to Keycloak — records that it supersedes hl-23m
• hl-bc94 — Hosted Kubernetes MCP uses Keycloak-OIDC identity passthrough, not a shared ServiceAccount — the Phase 4 boundary this record does not cross
• .planning/workstreams/litellm-return/phases/01-spike-foundation/SPIKE-REPORT.md — the Phase 1 measurement spike; the decision input for this ADR, including the GO verdict, its five conditions, and the VAL-02 blocking measurement
• Upstream issue BerriAI/litellm#31296 — configurable authorization_servers for the inbound MCP discovery path