MCP Gateway — Client Setup

Operational guide for connecting Claude Code and Claude Desktop to the cluster’s MCP plane: LiteLLM at llm.fzymgc.house, which fronts ten upstream MCP servers behind one host and exposes each at its own path. One header admits you on every route; on the two identity routes (engram, kubernetes) a second header carries you, obtained by the client itself through standard OAuth discovery → Keycloak dynamic client registration → PKCE.

llm.fzymgc.house is the only host. The previous MCP gateway (agentgateway, mcp-gw.fzymgc.house) was decommissioned in Phase 5 of the litellm-return milestone; its hostnames no longer resolve and there is no alias (D-72). The historical litellm.fzymgc.house host is likewise gone (hl-0sr). New work MUST NOT target either.

Quick reference

Tool names are prefixed with the server name. LiteLLM namespaces every tool, so resolve-library-id on context7 is context7-resolve-library-id here. Anything that pins a tool name by string — an allowlist, a disallowedTools entry, an agent prompt, a hook — needs the prefix. This is a client-visible property and it is stated here rather than left to be discovered.

Property	Value
Gateway host	https://llm.fzymgc.house
MCP endpoint (per server)	https://llm.fzymgc.house/<server>/mcp — no trailing slash (see below)
Servers	Ten routes, three dispositions — see the table below
Admission	Per-user LiteLLM virtual key in the x-litellm-api-key header, value Bearer <key>, on every route
Identity (engram, kubernetes)	Authorization: Bearer <Keycloak access token> beside the admission header — obtained by the client through discovery → DCR → PKCE, never handed over out of band
Tool names	Prefixed with the server name — context7-resolve-library-id, exa-, firecrawl-, fal-, deepwiki-, firewalla_ro-, clickhouse_ro-/clickhouse_rw-, engram-, kubernetes-
Key scoping	The key’s object_permission.mcp_servers names the servers it may reach; a key with no object_permission reaches none
Write surface	clickhouse_rw is the only route on this host that can MUTATE state. Reaching it requires a key whose object_permission.mcp_servers names clickhouse_rw specifically. clickhouse_ro is the read-only ClickHouse instance and is what almost every client wants
Upstream credential	Held by LiteLLM, injected outbound — never reaches the client
Claude Code support	Native — handles OAuth and custom headers directly; all ten routes
Claude Desktop support	The eight key-scoped routes via the mcp-remote bridge; the two identity routes are not reachable from Claude Desktop (its connector cannot send the second header — deferred, D-81)

The ten routes and what each does with your Authorization header

Route	Upstream	Disposition	What the client sends
context7	mcp.context7.com (SaaS)	overwrite — LiteLLM composes its own Authorization from a secret it holds	x-litellm-api-key only
exa	mcp.exa.ai (SaaS, three-tool allow-list in the URL)	overwrite	x-litellm-api-key only
firecrawl	mcp.firecrawl.dev (SaaS)	overwrite	x-litellm-api-key only
fal	mcp.fal.ai (SaaS)	overwrite	x-litellm-api-key only
firewalla_ro	firewalla-mcp (in-cluster, read-only)	overwrite	x-litellm-api-key only
clickhouse_ro	clickhouse-mcp-ro (in-cluster, read-only)	overwrite	x-litellm-api-key only
clickhouse_rw	clickhouse-mcp-rw (in-cluster, read-write)	overwrite	x-litellm-api-key only — and a key scoped to clickhouse_rw
engram	memory-mcp (in-cluster)	relay — oauth_passthrough: true; your Keycloak token goes to the upstream verbatim	x-litellm-api-key and Authorization: Bearer <Keycloak token> (obtained by the client)
kubernetes	kubernetes-mcp (in-cluster)	relay — and the upstream validates the token itself	x-litellm-api-key and Authorization: Bearer <Keycloak token> (obtained by the client)
deepwiki	mcp.deepwiki.com (SaaS)	credential-free — no upstream secret to overwrite with, no relay flag	x-litellm-api-key only

Server names use the underscore spelling (firewalla_ro, clickhouse_ro, clickhouse_rw). LiteLLM rejects a hyphen in an MCP server name — - is the separator it splices between server and tool to namespace tool names, so a hyphen in the server half makes that boundary ambiguous. The rejection happens at config load and takes the whole proxy down, not one route. When you scope a virtual key, use the underscore form: that is the name LiteLLM matches object_permission.mcp_servers against.

Why x-litellm-api-key and not Authorization. LiteLLM accepts either header for admission, x-litellm-api-key first. Using it uniformly on every route is the safer contract: it disambiguates admission from any credential the client also needs to relay upstream, and on the identity-bearing routes (engram, kubernetes) it is what keeps the virtual key from being forwarded to the upstream. Using it on a route where it makes no difference means no client config has to change when a route gains an identity relay.

The x-litellm-api-key value MUST carry the Bearer scheme prefix. Measured 2026-08-30 against the deployed 1.96.2: x-litellm-api-key: <key> is rejected 401 Malformed API Key passed in, while x-litellm-api-key: Bearer <key> is admitted. The header name changes; the value shape does not.

It fails closed, and that is the sharp edge. x-litellm-api-key takes precedence over Authorization, so a request carrying a bare key in x-litellm-api-key is rejected even when a perfectly valid Authorization: Bearer <key> is also present — there is no fallback to the second header. Get the prefix wrong on the dual-header routes and the symptom is a 401 that looks like a bad key.

The two identity routes send TWO headers

engram and kubernetes are the only routes on this host that relay your own credential to the upstream. On seven of the remaining eight, LiteLLM injects a secret it owns and your identity stops at the gateway; the eighth, deepwiki, holds no upstream credential either but carries no relay flag — see How authentication works for what that means. On these two the credential is your own Keycloak token, and LiteLLM relays it to the upstream untouched — which is what lets engram attribute a memory to you and lets the Kubernetes MCP reach the k3s API as you rather than as a shared service account.

Header	Carries	Purpose
x-litellm-api-key: Bearer <virtual key>	your LiteLLM virtual key	admission, route scoping, rate limits, budget, spend attribution
Authorization: Bearer <Keycloak access token>	you	relayed verbatim to the upstream, which validates it

Putting your virtual key in Authorization on these two routes forwards it to the upstream. Not a redirect, not a 401 — LiteLLM copies that header through to engram or the Kubernetes MCP as-is, and the route keeps working, so nothing tells you it happened. This is the reason x-litellm-api-key is used uniformly on all ten routes rather than only where it is strictly required: a client that never puts a virtual key in Authorization cannot leak one, and no client config has to change when a route gains an identity relay.

You need a different Keycloak token per route, and this is not new. The two upstreams require different token audiences, and Keycloak stamps each as a fixed string rather than deriving it from the URL you call:

Route	Required token audience (aud)
engram	mcp-public
kubernetes	https://llm.fzymgc.house/kubernetes/mcp

The audience identifies the protected resource, not the host you happen to reach it through (hl-94n1): Keycloak stamps the kubernetes value through two mappers (tf/keycloak/mcp_kubernetes.tf — one on the mcp-kubernetes optional scope, one on the static client), the k3s API server accepts it (k3s_oidc_audiences), and kubernetes-mcp validates against it before the API server ever sees the token. You do not obtain these tokens yourself: Claude Code does, through discovery → registration → PKCE — see the Claude Code section below.

What to expect when a token is wrong. LiteLLM does not validate the token it relays; the upstream does, and both fail closed. So a bad, expired or wrong-audience token produces an error from the upstream, arriving as a successful HTTP response carrying a JSON-RPC error — not a LiteLLM 401. If a call is refused while tools/list works, look at the token’s aud and exp before suspecting the route.

Endpoint spelling: no trailing slash. https://llm.fzymgc.house/context7/mcp serves. The trailing-slash form /context7/mcp/ answers 307 — and its Location is an http:// URL, so a client that follows redirects gets a scheme downgrade, and one that does not follow redirects on a POST silently loses its body. Write the path without the slash.

This inverts the aggregate /mcp/ endpoint’s behaviour, where the slash is required and /mcp is the form that redirects. Do not carry the habit across.

Tool names are prefixed per server. The prefix is LiteLLM’s own namespacing, applied uniformly (measured name-set to name-set on context7 against the previous gateway, 2026-08-30). If a client needs an exact list for a route, read it from that route’s tools/list rather than deriving it — the prefix is predictable, the underlying names are the vendor’s.

How authentication works

The dual-header contract

One header admits you on every route. A second header carries you, on two of them. That is the whole model, and it is uniform on purpose.

Header	Sent on	Carries	Who reads it
x-litellm-api-key: Bearer <virtual key>	all ten routes, always	your LiteLLM virtual key	LiteLLM — admission, route scoping, rate limits, budget, spend attribution
Authorization: Bearer <Keycloak access token>	engram and kubernetes only	you	the upstream, which validates it. LiteLLM relays it verbatim and validates nothing

Three authorization dispositions, not a two-way safety split

Ten entries render under mcp_servers, and each falls into exactly one disposition. The arithmetic below is held before merge by test_every_mcp_route_declares_one_authorization_disposition in tools/litellm-verify/tests/test_phase03_config_gates.py, so a route that joins the estate without a disposition turns that gate red rather than landing here unnoticed.

• Seven routes — context7, exa, firecrawl, fal, firewalla_ro, clickhouse_ro and clickhouse_rw — carry auth_type: bearer_token with an authentication_token. LiteLLM composes its own Authorization from that secret, so whatever a caller put in that header is overwritten before the request leaves the gateway. The upstream sees the gateway, not you, and the upstream secret never reaches the client.

• Two routes — engram and kubernetes — carry extra_headers: ["Authorization"] together with oauth_passthrough: true and relay the caller’s header to the upstream by design. LiteLLM holds no upstream credential for them at all: your own Keycloak token is the credential, which is what lets engram attribute a memory to you and lets the Kubernetes MCP reach the k3s API as oidc:<your-email> under your normal RBAC. This is also what makes a virtual key placed here an exfiltration.

  kubernetes validates the relayed token itself (litellm-return 05-01, D-68 as measured): issuer https://id.fzymgc.house/realms/fzymgc, signature against the realm’s JWKS, audience https://llm.fzymgc.house/kubernetes/mcp — and the k3s API server then validates it a second time. Two consequences. A token minted for another audience is refused by kubernetes-mcp (401 invalid_token) before the API server ever sees it. And a Keycloak outage answers 401 temporarily_unavailable on this route, where before this change nothing was validated locally and the route degraded only when the k3s API server itself could not reach the issuer.

• One route — deepwiki — carries neither an upstream credential to overwrite with nor oauth_passthrough, and its upstream is the third party mcp.deepwiki.com. It is in neither of the other two groups, and the estate has no observation of what LiteLLM does with a caller’s header on it.

What was measured, and what was not. The two-arm capture of 2026-09-02 (phase 04 plan 08) sent a non-credential canary in Authorization to deepwiki and recorded the disposition INCONCLUSIVE: mcp.deepwiki.com is a third party with no log sink this estate can read, so nothing there distinguishes a header LiteLLM stripped from an upstream that ignored one it did not recognise. The same run probed engram and recorded INCONCLUSIVE for it as well — the probe’s requests demonstrably reached memory-mcp (seven lines in the window carry LiteLLM’s own outbound user-agent) but that access log records method, path, status and user-agent and no request headers at all, so the canary’s absence is a property of the sink rather than evidence of a strip. The relay group is declared by configuration — LiteLLM 1.96.2’s own GET /v1/mcp/server reads back oauth_passthrough: true for engram and kubernetes and false for the remaining eight — and what crosses the wire on a tools/call is not observed from outside the gateway.

The client half of the contract is demonstrated. That Claude Code sends the static x-litellm-api-key alongside the OAuth bearer it obtains for itself was demonstrated on 2026-09-09 with Claude Code 2.1.266 on the kubernetes route (litellm-return 05-03): a ~/.claude.json entry carrying one header (x-litellm-api-key) and no Authorization header of any kind connected through discovery → Keycloak dynamic client registration → PKCE, listed 13 tools, and a namespace read returned real cluster data — which is only possible if the relayed token reached kubernetes-mcp with the LiteLLM audience and the API server accepted it. The token’s aud claim itself was not inspected on the client (the D-70 gates prove both audiences are stamped, and the API server’s acceptance is the functional consequence). The contract is one contract, and one demonstration proves it; the engram route’s own demonstration (scopes advertised, a >30-minute pause survived) is recorded in the decommission PR’s attestation checklist, not here.

Why the admission credential goes in x-litellm-api-key on all ten routes and not only where it is required. LiteLLM accepts either header for admission and prefers x-litellm-api-key. On a relay route, a virtual key placed in Authorization is forwarded to the upstream — LiteLLM copies that header through to engram or the Kubernetes MCP as-is. There is no redirect, no 401, no log line, and the route keeps working, so nothing tells you it happened; you find out when the key turns up somewhere it was never meant to be. Using x-litellm-api-key uniformly means a client that never puts a virtual key in Authorization cannot construct that shape, and no client config has to change on the day a route gains an identity relay.

Concretely: never collapse the two headers into one, and never move the virtual key into Authorization “because it also works”. On the seven overwriting routes it does work and costs nothing, because the upstream credential replaces whatever you sent. On the two relay routes it silently exfiltrates the key. And on the credential-free route there is nothing to overwrite it with and no relay flag to make the behaviour deliberate, so where the key ends up rests on LiteLLM’s own strip logic — which the capture above could not observe from outside the gateway. The rule is unchanged and the justification is the rendered configuration: the credential you send for admission goes in x-litellm-api-key, always, on every route.

LiteLLM validates the relayed token not at all. The upstream does, and it fails closed. A bad, expired or wrong-audience token therefore produces an error from the upstream, arriving as a successful HTTP response carrying a JSON-RPC error — not a LiteLLM 401. See Troubleshooting.

How the identity routes discover Keycloak

An unauthenticated request to engram or kubernetes answers 401 with WWW-Authenticate: Bearer resource_metadata="https://llm.fzymgc.house/.well-known/oauth-protected-resource/<server>/mcp". That document is the upstream’s own RFC 9728 protected-resource document, relayed by LiteLLM with only resource rewritten to the LiteLLM URL (D-66, D-67, D-68) — LiteLLM never synthesises one, and returns 502 when the upstream has none. It names Keycloak (https://id.fzymgc.house/realms/fzymgc) as the authorization server and the scopes the route wants (mcp-kubernetes, groups, offline_access on the kubernetes route). The client then registers itself dynamically against Keycloak (RFC 7591 — allowed for loopback redirect URIs by the realm’s Trusted-Hosts policy, D-69), runs PKCE in your browser, and presents the resulting bearer. Nothing is handed over out of band, and no static Keycloak client id is required.

# What a conforming client reads first. resource == the LiteLLM URL; authorization_servers[0] == Keycloak.
curl -sS https://llm.fzymgc.house/.well-known/oauth-protected-resource/kubernetes/mcp | jq .

Measured 2026-09-09 after PR #2046: resource, authorization_servers[0] and the three scopes as above, stable across eight consecutive reads. (kubernetes-mcp v0.0.63 publishes its server_url as authorization_servers, so that key holds the Keycloak issuer — a recorded misuse, argocd/app-configs/kubernetes-mcp/config.yaml.)

SSE is served at the aggregate path only

There is no per-server SSE endpoint on LiteLLM. The legacy transport is mounted once, at the aggregate path /mcp/sse, and there is no path segment that selects a server on it. Measured 2026-08-31 against the deployed 1.96.2: /{server}/mcp/sse and /{server}/sse both return 404, and so does a bare /sse. Do not construct a per-server SSE URL — there is no spelling of one that works.

Server selection by the x-mcp-servers header was measured on the streamable-HTTP aggregate surface /mcp/, and that is the surface the property is attributed to. It has not been demonstrated on the SSE lane: the SSE lane was observed listing zero tools both with the header and without it, so that run distinguishes nothing about scoping there.

Streamable HTTP at /{server}/mcp is the transport every client on this page uses, and it is what you want. The aggregate lane exists for clients that speak only the older transport; the zero-tool inventory above is recorded as an observation of 1.96.2, not asserted as a fixed property, and it is why the SSE lane is not a working substitute today.

Getting a client virtual key

One key per (user, server set)

A LiteLLM virtual key is minted at runtime against /key/generate with the master key, and it carries three things that matter: who the key is for, the servers it may reach, and its limits.

curl -sS -X POST https://llm.fzymgc.house/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' \
  -d '{"key_alias":"<name>-mcp",
       "user_id":"<name>",
       "object_permission":{"mcp_servers":["context7","exa"]},
       "rpm_limit":120,
       "max_budget":10}' \
  | jq -r '.key'

Field	What it does
object_permission.mcp_servers	The route names this key may reach. This list is the entire boundary. Use the underscore spellings (clickhouse_ro, not clickhouse-ro) — that is the name LiteLLM matches against
user_id	Whose key it is. Spend and the identity attribute on every MCP span are attributed to it, which is what makes SC#3’s per-caller accounting work
key_alias	The human-readable label that appears on the span and in spend reports
rpm_limit / max_budget	Per-key limits. Optional, and worth setting on anything shared

A key with no object_permission reaches nothing, and that is the intended default. It is not an error state to fix by adding a broad permission — it is the deny-by-default the model rests on. Mint per consumer, scoped to the set that consumer actually needs; one broad key shared around is both a larger blast radius and harder to attribute.

Scope is a real boundary, not a label. A key scoped to a subset sees only that subset’s tools in tools/list and is refused tools/call on the others by name. This is the only thing separating clickhouse_ro from clickhouse_rw: both instances validate the same upstream token, so nothing about the secret distinguishes them. Adding clickhouse_rw “to be safe” grants the write surface and nothing downstream will object.

The master key lives in Vault at secret/fzymgc-house/cluster/litellm, property master_key. The workstation key for Claude Code is claude_code_api_key on the same path — see LiteLLM → Minting and scoping virtual keys for the mint bodies and the budget doctrine.

object_permission.mcp_servers accepts the server name on the way in, and GET /key/info reads it back as the server’s internal id — the same scope, recorded canonically. That is not drift; do not “correct” it.

A key with no object_permission is not refused at connect time. It completes initialize, receives a session, and gets an empty tools/list. Only tools/call refuses it, as a JSON-RPC error inside an HTTP 200 (isError: true, "User not allowed to call this tool."; the underlying status is 403). So “the client connected” proves nothing about scoping, and a health check that only asserts a 200 would pass on a key entitled to nothing.

Claude Code

All ten servers are served by LiteLLM — the five SaaS routes, the three in-cluster ones, and the two identity routes. Add only the ones you need; all flags precede the server name in claude mcp add, and --scope user makes the server available across all projects (--scope local limits it to the current repo).

The eight key-scoped routes (virtual key in a header):

claude mcp add --transport http context7 https://llm.fzymgc.house/context7/mcp \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http exa https://llm.fzymgc.house/exa/mcp \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http firecrawl https://llm.fzymgc.house/firecrawl/mcp \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http fal https://llm.fzymgc.house/fal/mcp \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http deepwiki https://llm.fzymgc.house/deepwiki/mcp \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http firewalla_ro https://llm.fzymgc.house/firewalla_ro/mcp \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http clickhouse_ro https://llm.fzymgc.house/clickhouse_ro/mcp \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user

clickhouse_rw is deliberately not in that list. It is the write instance of the telemetry store, and adding it should be a decision rather than a paste:

# Only if this client genuinely needs to WRITE to ClickHouse. The key must also
# name clickhouse_rw in object_permission.mcp_servers, or every tools/call is
# refused (see the scoping note below).
claude mcp add --transport http clickhouse_rw https://llm.fzymgc.house/clickhouse_rw/mcp \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user

The identity routes

engram and kubernetes authenticate you through OAuth — there is no second static header. The virtual key still admits you to LiteLLM, in x-litellm-api-key; the Keycloak token that tells the upstream who you are is obtained by Claude Code itself, through standard discovery (the RFC 9728 protected-resource document on llm.fzymgc.house, which LiteLLM relays from the upstream’s own — D-66, D-67, D-68) → Keycloak dynamic client registration → PKCE in your browser. Nothing is handed over out of band. The two entries use different fixed callback ports so their OAuth callbacks never collide:

# kubernetes: kubernetes-mcp publishes its own discovery document and validates
# the token itself; Keycloak is the issuer. The token's aud carries
# https://llm.fzymgc.house/kubernetes/mcp and the k3s API authorizes it as
# `oidc:<your-email>`. Demonstrated 2026-09-09, Claude Code 2.1.266.
claude mcp add --transport http --scope user --callback-port 51337 \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" \
  kubernetes https://llm.fzymgc.house/kubernetes/mcp
# then in Claude Code: /mcp → kubernetes → Authenticate (browser: Keycloak) → tools appear

# engram: the same shape. This entry works only once the engram release that
# serves its own discovery document (seanb4t/engram#526) is pinned in the
# cluster; until then the route answers 502 on discovery (see Troubleshooting).
claude mcp add --transport http --scope user --callback-port 51338 \
  --header "x-litellm-api-key: Bearer <your-virtual-key>" \
  engram https://llm.fzymgc.house/engram/mcp
# then: /mcp → engram → Authenticate (browser: Keycloak) → tools appear

NO static Authorization header on either entry. A configured headers.Authorization is used by Claude Code as the server’s credential and disables its OAuth flow for that server — there is no fallback — and on these two routes LiteLLM relays whatever that header carries to the upstream. The virtual key goes in x-litellm-api-key and nowhere else.

Fallback if registration is refused. If Keycloak refuses the dynamic registration (the symptom reads like “does not support dynamic client registration”), add --client-id mcp-public to the command — the realm’s static public PKCE client — and, for kubernetes, set that entry’s oauth.scopes in ~/.claude.json to mcp-kubernetes groups offline_access so the token still carries the resource audience. The 2026-09-09 demonstration needed neither: the connect succeeded first try with no --client-id, so registration against Keycloak under the loopback Trusted-Hosts policy (D-69) is the measured path.

Do not collapse these to a single Authorization header holding the virtual key. That is the leak described above: LiteLLM forwards it to the upstream and the route keeps working, so nothing surfaces the mistake.

Scoping notes

The two ClickHouse routes present the same upstream credential — both MCP instances validate one token — so nothing about the secret separates them. The only thing that does is which server your key is scoped to. A key scoped to clickhouse_ro sees none of the write instance’s tools — that half is measured (read_only_key_on_write_route: [], 2026-09-07).

The by-name refusal on this pair has never been observed — do not rely on it. Scoping demonstrably hides the write instance’s tools from a read-only key, and the general refusal-at-the-call mechanism is measured elsewhere (04-01: an unentitled key gets a session and is then refused at the call, HTTP 200 carrying isError: true). What has not been measured is a clickhouse_ro-scoped key being refused a clickhouse_rw tool by name. The probe declines to invoke any tool on the estate’s only MCP write surface while no served tool declares annotations.readOnlyHint, because that assertion’s failure mode is the call succeeding — probing with a possibly-mutating tool would perform the very write it exists to prove impossible. Accepted unmet for this milestone by operator ruling (04-09-DECISION.md, Ruling 2, option C, 2026-09-07). It reopens when the clickhouse-mcp upstream declares readOnlyHint on any served tool, which is a data change upstream needing no change here. Treat per-key scoping as the boundary you have evidence for.

The three in-cluster routes reach Kubernetes Services rather than internet vendors. That is invisible from the client side — same host, same header, same path shape — and is noted only so a slower first response from firewalla_ro reads as the appliance being busy rather than as a gateway fault.

The client header is identical on all eight key-scoped routes, including deepwiki. deepwiki needs no upstream credential — that is LiteLLM’s side of the connection, not yours — but every route on llm.fzymgc.house still requires an admitted virtual key, and a key not scoped to deepwiki is refused it like any other.

Mint the virtual key against LiteLLM, scoped to the servers this client actually needs — see Getting a client virtual key for the /key/generate body and what each field does. A key with no object_permission reaches no MCP server at all, and clickhouse_rw belongs only on a key that must write to the telemetry store.

Claude Desktop

Claude Desktop has no field for a static auth header on remote connectors, so the mcp-remote stdio bridge injects it. The value carries the Bearer scheme prefix and therefore contains a space, so the quoting gotcha below applies in full: keep the value in an env var and write --header with no space after the colon. Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "context7": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://llm.fzymgc.house/context7/mcp",
        "--header", "x-litellm-api-key:${LITELLM_MCP_KEY}"
      ],
      "env": {
        "LITELLM_MCP_KEY": "Bearer <your-virtual-key>"
      }
    },
    "exa": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://llm.fzymgc.house/exa/mcp",
        "--header", "x-litellm-api-key:${LITELLM_MCP_KEY}"
      ],
      "env": {
        "LITELLM_MCP_KEY": "Bearer <your-virtual-key>"
      }
    }
  }
}

Repeat the exa block for firecrawl, fal, deepwiki, firewalla_ro and clickhouse_ro, swapping the server name in both the key and the /<server>/mcp path. The same virtual key works for every server it is scoped to; env is per-entry in this file, so the value is repeated rather than shared.

Add a clickhouse_rw block only if this client must write to the telemetry store — and only alongside a key scoped to clickhouse_rw. Without that scope the entry connects and lists no tools, which reads like a broken server rather than like the refusal it is.

The identity routes (engram, kubernetes) are not configured here. A native Claude Desktop connector runs the OAuth flow but cannot send the x-litellm-api-key admission header beside it, and the bridge sends a static header but has no OAuth flow — neither produces the two-header shape these routes require. Claude Desktop and other non-CLI clients are out of scope for the cutover (D-81); Claude Code is the client for those two routes today.

Header-quoting gotcha. Claude Desktop’s argument parser mangles header values that contain spaces (like Bearer <key>). Put the value in an env var and write the --header flag with no space after the colon.

Fully quit Claude Desktop (⌘Q) and relaunch — it reads the config only at startup.

Verify

A key-scoped route — smoke-test the virtual key directly (no client needed). Note the path shape, the missing trailing slash and the Bearer scheme prefix inside x-litellm-api-key, all three of which are easy to get wrong and all three of which surface as a 401 or a 307 rather than as a useful message:

curl -sS -X POST https://llm.fzymgc.house/exa/mcp \
  -H "x-litellm-api-key: Bearer <your-virtual-key>" \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","method":"initialize","id":1,
       "params":{"protocolVersion":"2025-06-18","capabilities":{},
                 "clientInfo":{"name":"curl","version":"0"}}}' -D-

initialize first, because LiteLLM issues an mcp-session-id response header that tools/list needs. Which servers a key may reach is settled at tools/call, not at connect — see the note under Getting a client virtual key about what an empty tools/list does and does not prove.

An identity route, unauthenticated, must answer the discovery challenge:

curl -s -D - -o /dev/null -X POST https://llm.fzymgc.house/kubernetes/mcp \
  -H 'content-type: application/json' -d '{}' | rg -i '^HTTP/|^www-authenticate'
# HTTP/2 401
# www-authenticate: Bearer resource_metadata="https://llm.fzymgc.house/.well-known/oauth-protected-resource/kubernetes/mcp"

The connected state from the client side:

claude mcp get kubernetes     # shows connected state + tool count after OAuth

Which routes the proxy actually serves, independent of any client:

curl -sS https://llm.fzymgc.house/v1/mcp/server \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -r '.[].server_name'

Rollback

A client-side problem is fixed on the client. There is no second host to repoint at: llm.fzymgc.house is the only MCP plane, and the previous gateway’s hostnames do not resolve (D-72). A wrong header, a wrong path spelling, a key not scoped to the route — each is a client-config fix, diagnosed under Troubleshooting.

A LiteLLM-side regression on one route is a GitOps revert of that route’s commit. A route lands as one change — its mcp_servers entry in argocd/app-configs/litellm-chart/values.yaml, its extraEnvVars entry and its litellm-secrets key — precisely so that reverting the commit removes all three together. Nothing on the client changes.

The decommission of the previous gateway is itself reversible, and that is a different procedure. It is a git revert of the deletion PR plus a Vault undelete, documented once, in the operator runbook: LiteLLM → Rolling back the agentgateway decommission. It restores the retired plane in full; it is not a per-route lever, and it is not something a client-side fault ever calls for.

Troubleshooting

LiteLLM: the upstream returns 401 and it looks like a bad secret

Symptom. initialize succeeds, tools/list returns tools, and tools/call comes back as an HTTP 200 carrying a JSON-RPC error whose text is the upstream’s own “unauthorized” / “invalid token” wording. Nothing in LiteLLM’s own response is a 401, and rotating the upstream secret does not help.

Cause — a wrong header name, not a wrong secret. Two shapes produce this, and both survive a credential rotation because the credential was never the problem:

1. Client side, on engram or kubernetes. The virtual key was placed in Authorization instead of x-litellm-api-key. Those two routes relay Authorization to the upstream verbatim, so the upstream is handed a LiteLLM virtual key and rejects it — correctly. Fix: move the virtual key into x-litellm-api-key: Bearer <key> and reserve Authorization for your Keycloak token. (This is also the leak path: the key reached the upstream.)
2. Config side, on any credentialed route. The Vault property was templated with an Authorization: Bearer … prefix already baked in. LiteLLM’s auth_type: bearer_token composes the header itself, so a pre-prefixed value goes out as Bearer Bearer <token>. The upstream sees a malformed credential and returns 401. Fix: store the bare token — a value that starts with the scheme word and a space is wrong by exactly 7 characters (len("Bearer ")).

Distinguish them in one step: if you can reproduce it with curl sending only x-litellm-api-key, it is shape 2 (a config fault, file it). If sending the headers correctly makes it go away, it was shape 1 — and the key that leaked should be rotated.

LiteLLM: the client connects but sees no tools

Symptom. The client reports the server as connected. initialize returns 200 and issues an mcp-session-id. tools/list returns {"tools":[]}. Every tools/call comes back as an HTTP 200 with isError: true and "User not allowed to call this tool."

Cause — the key is not scoped to that server. This is a refusal, not an outage: object_permission.mcp_servers on your virtual key does not name the route you are calling (or the key has no object_permission at all, which reaches nothing by design). LiteLLM admits the key — it is a valid key — and then entitles it to nothing on this route.

Why it reads as a broken server. Admission and entitlement are separate steps, and only the first one produces a status code you can see. A health check that asserts a 200 from initialize passes on a key entitled to nothing.

Fix. Read the key’s scope back and compare it to the route name, using the underscore spelling:

curl -sS "https://llm.fzymgc.house/key/info" \
  -H "Authorization: Bearer <your-virtual-key>" | jq '.info.object_permission'

Authenticate AS the key; never put it in the query string. With no key parameter, /key/info reports the key in the Authorization header — key = key or user_api_key_dict.api_key in litellm/proxy/management_endpoints/key_management_endpoints.py at the v1.96.2 tag this estate runs. ?key=<value> instead lands the credential in LiteLLM’s access log, in Traefik’s on the way through the public ingress, in the url.full attribute of the OTel span (retained 30 days in ClickStack), and in /proc/<pid>/cmdline for the life of the curl. See “Verifying a minted key” for the same rule stated with its upstream reference. If a key is restricted to llm_api_routes it cannot call /key/info at all and returns 401 — use the /key/list + /budget/list pair described there, and rotate any key you have already exposed in a URL.

Then mint or update a key naming that server. Note that GET /key/info reads the scope back as internal ids rather than names — that is canonical storage, not drift.

LiteLLM: something that used to arrive stopped arriving, with no error

Symptom. tools/list and tools/call still work. What stopped is anything needing session continuity — a server-initiated notification, a sampling or elicitation round-trip. There is no error to look for; the symptom of a lost session is the missing notification.

Cause. MCP session state is process-local, so a proxy restart or a request landing on the other replica drops it, and LiteLLM downgrades to stateless service rather than erroring.

Do not debug this from the client. The behaviour, what does and does not survive, and the two conditions that do produce errors are documented once, in the operator runbook: LiteLLM → MCP session semantics across restarts and replicas.

LiteLLM: an identity route’s upstream is down and nothing says so

Symptom. engram or kubernetes fails at tools/call with no proxy-side warning beforehand. No failing health check, no route marked unhealthy, no metric. The first signal is a client’s failed call.

Cause. Both identity routes carry an auth-bearing extra_headers with no credential mode of their own, which is the configuration that makes the relay work. LiteLLM reads that shape as requires_per_user_auth and skips health checks for the server entirely — it has no credential with which to probe the upstream on its own behalf, so it does not try. This is correct and is not to be “fixed” by giving those routes a credential; doing so would end the per-user identity relay they exist to provide.

Fix. Check the upstream Deployment directly — memory-mcp in agent-memory, or kubernetes-mcp in kubernetes-mcp. The proxy will not tell you, and the eight credentialed routes are unaffected because they are health-checked normally.

engram / kubernetes: client never connects, or no OAuth prompt

The client only starts its OAuth flow if discovery works. Check the relayed document first:

curl -sS -o /dev/null -w '%{http_code}\n' \
  https://llm.fzymgc.house/.well-known/oauth-protected-resource/engram/mcp

• 502 — the upstream is not serving its own protected-resource document, or LiteLLM’s negative cache is holding an earlier failure. For engram this is the expected state until the release that serves the document (seanb4t/engram#526) is pinned; for kubernetes it means the authorization_url/server_url block in the kubernetes-mcp ConfigMap is not what the pod is running (a ConfigMap-only change does not roll the pod — kubectl -n kubernetes-mcp rollout restart deploy/kubernetes-mcp). After the upstream starts serving, restart LiteLLM to clear the negative cache (kubectl rollout restart deployment/litellm -n litellm).
• 404 — the URL is wrong. The document lives at /.well-known/oauth-protected-resource/<server>/mcp on the host root; the server root and the /mcp/<server> order both miss.
• 200 but the client still shows no prompt — the server URL in the client entry is wrong (a bare /mcp returns 404 route not found rather than triggering the flow), or a static headers.Authorization is configured on the entry, which disables Claude Code’s OAuth flow for that server.
• “does not support dynamic client registration” — Keycloak refused the registration. The realm’s Trusted-Hosts policy admits loopback redirect URIs only (argocd/app-configs/keycloak/dcr-trusted-hosts-job.yaml); a client registering with any other redirect URI is rejected. Use the --client-id mcp-public fallback above and file the shape you saw.
• Registration refused with 403 and “It’s allowed to have max 500 clients per realm” — the realm’s anonymous Max Clients Limit policy counts every client in the realm, not only registrations, and dcr-trusted-hosts-job.yaml sets it to 500. The nightly keycloak-dcr-reaper CronJob (argocd/app-configs/keycloak/dcr-client-reaper-cronjob.yaml) keeps headroom by deleting public, UUID-named clients that have no active or offline session. Check that it is running and succeeding with kubectl -n keycloak get cronjob keycloak-dcr-reaper and the logs of its most recent Job.

engram / kubernetes: re-authentication stops at Keycloak “Client not found”

The reaper deleted the client this entry registered, because it no longer held an active or offline session. Claude Code re-registers by itself when the token endpoint rejects the client_id. When it holds the registration without tokens, it replays the deleted client_id at the authorize endpoint instead; the rejection happens in the browser and never reaches Claude Code (anthropics/claude-code#84614).

Fix: /mcp → the server → Clear authentication, then Authenticate. Or run claude mcp remove <name> and re-add it with the same claude mcp add command from The identity routes.

Key-scoped route: 401/403 from the gateway

The virtual key is wrong, not sent, sent in the wrong header, or sent without the Bearer scheme prefix inside x-litellm-api-key. Re-run the Verify curl; a 200 with an mcp-session-id confirms the key and endpoint.

Claude Desktop server fails to authenticate

Almost always the header-quoting gotcha — confirm the --header arg is x-litellm-api-key:${LITELLM_MCP_KEY} (no space after the colon) and that env.LITELLM_MCP_KEY holds Bearer <key>. Quit and relaunch after any edit.

Related

• LiteLLM operations — the operator runbook: key minting, restart and replica semantics, database-outage behaviour, and the decommission rollback
• AI Tooling — general MCP server configuration
• Services catalog — LiteLLM architecture and the route inventory
• Secrets & Vault Paths — Vault path conventions
• ADR 2026-09-08 — Keycloak DCR deny removed — why a client can register directly against Keycloak (D-69)
• LiteLLM MCP route declarations: argocd/app-configs/litellm-chart/values.yaml (proxy_config.mcp_servers — the estate’s only sanctioned MCP registration point)
• kubernetes-mcp discovery config: argocd/app-configs/kubernetes-mcp/config.yaml