MCP Gateway — Client Setup
This page connects Claude Code and Claude Desktop to the cluster’s MCP plane. LiteLLM at
llm.fzymgc.house fronts twelve upstream MCP servers behind one host and exposes each at its own
path. One header admits a client on every route. On the three identity routes, engram,
kubernetes and dns, a second header carries the user’s own Keycloak token, which the client
obtains through OAuth discovery, dynamic client registration and PKCE. llm.fzymgc.house is the only MCP
host. To add a new identity route, read MCP Identity Routes.
Quick reference
Section titled “Quick reference”LiteLLM prefixes every tool name with the server name. resolve-library-id on context7 is
context7-resolve-library-id here. Anything that pins a tool name by string, such as an
allowlist, a disallowedTools entry, an agent prompt or a hook, needs the prefix.
| Property | Value |
|---|---|
| Gateway host | https://llm.fzymgc.house |
| MCP endpoint (per server) | https://llm.fzymgc.house/<server>/mcp, with no trailing slash |
| Servers | Twelve routes in three dispositions, listed below |
| Admission | A per-user LiteLLM virtual key in the x-litellm-api-key header, value Bearer <key>, on every route |
Identity (engram, kubernetes, dns) |
Authorization: Bearer <Keycloak access token> beside the admission header. The client obtains it through discovery, registration and PKCE. The dns upstream also accepts a static dnsk_ key |
| Tool names | Prefixed with the server name: context7-, exa-, firecrawl-, fal-, deepwiki-, firewalla_ro-, clickhouse_ro-, clickhouse_rw-, unifi_ro-, engram-, kubernetes-, dns- |
| Key scoping | The key’s object_permission.mcp_servers names the servers it reaches. An access group that lists the server id also grants the route. LiteLLM refuses a key with no grant for a route at initialize with HTTP 403. The exception is a key of a proxy_admin user with no object_permission list, which reaches every route. An explicit list binds a proxy_admin key too, even an empty list |
| Write surface | clickhouse_rw is the only route on this host that can change state. It needs a key whose object_permission.mcp_servers names clickhouse_rw. clickhouse_ro is the read-only instance and is what almost every client wants |
| Upstream credential | Held by LiteLLM and injected outbound. It never reaches the client |
| Claude Code | Native. It handles OAuth and custom headers on all twelve routes |
| Claude Desktop | The nine key-scoped routes through the mcp-remote bridge. The three identity routes are not reachable, because its connector cannot send the second header |
The twelve routes and what each does with your Authorization header
Section titled “The twelve routes and what each does with your Authorization header”The routes are declared once, under proxy_config.mcp_servers in
argocd/app-configs/litellm-chart/values.yaml.
| Route | Upstream | Disposition | What the client sends |
|---|---|---|---|
context7 |
mcp.context7.com (SaaS) |
overwrite: LiteLLM composes its own Authorization from a secret it holds |
x-litellm-api-key only |
exa |
mcp.exa.ai (SaaS, three-tool allowlist in the URL) |
overwrite | x-litellm-api-key only |
firecrawl |
mcp.firecrawl.dev (SaaS) |
overwrite | x-litellm-api-key only |
fal |
mcp.fal.ai (SaaS) |
overwrite | x-litellm-api-key only |
firewalla_ro |
firewalla-mcp (in-cluster, read-only) |
overwrite | x-litellm-api-key only |
clickhouse_ro |
clickhouse-mcp-ro (in-cluster, read-only) |
overwrite | x-litellm-api-key only |
clickhouse_rw |
clickhouse-mcp-rw (in-cluster, read-write) |
overwrite | x-litellm-api-key only, on a key scoped to clickhouse_rw |
engram |
memory-mcp (in-cluster) |
relay: oauth_passthrough: true, the Keycloak token goes to the upstream verbatim |
x-litellm-api-key and Authorization: Bearer <Keycloak token> |
kubernetes |
kubernetes-mcp (in-cluster) |
relay, and the upstream validates the token itself | x-litellm-api-key and Authorization: Bearer <Keycloak token> |
dns |
dns-control (in-cluster, port 8081) |
relay, and the upstream validates the token itself | x-litellm-api-key and Authorization: Bearer <Keycloak token or dnsk_ static key>. The key reaches dns through object_permission.mcp_servers or through the access group mcp-dns. See dns-control → The MCP path |
deepwiki |
mcp.deepwiki.com (SaaS) |
credential-free: no upstream secret and no relay flag | x-litellm-api-key only |
unifi_ro |
unifi-mcp (in-cluster, read-only) |
credential-free: no upstream secret and no relay flag. The key scope and the namespace network policy are the boundary | x-litellm-api-key only |
Server names use the underscore spelling (firewalla_ro, clickhouse_ro, clickhouse_rw,
unifi_ro).
LiteLLM rejects a hyphen in an MCP server name at configuration load, because - is the
separator it puts between server and tool. That rejection takes the whole proxy down, not one
route. When you scope a virtual key, use the underscore form. That is the name LiteLLM matches
object_permission.mcp_servers against.
The admission credential goes in x-litellm-api-key and not in Authorization. LiteLLM accepts
either header for admission and reads x-litellm-api-key first. Using it on every route keeps the
virtual key out of the header that the three identity routes relay to their upstream. A client
configuration then needs no change on the day a route gains an identity relay.
The x-litellm-api-key value must carry the Bearer scheme prefix. A bare key in that header is
refused with 401 Malformed API Key passed in. The refusal fails closed. x-litellm-api-key
takes precedence over Authorization, so a bare key there is refused even when a valid
Authorization: Bearer <key> is also present. On the dual-header routes a wrong prefix looks
like a bad key.
The three identity routes send TWO headers
Section titled “The three identity routes send TWO headers”engram, kubernetes and dns are the only routes on this host that relay the caller’s own
credential to the upstream. On the seven overwriting routes LiteLLM injects a secret it owns, and
the caller’s identity stops at the gateway. On deepwiki LiteLLM holds no upstream credential
and carries no relay flag. On the three identity routes the credential is the caller’s own Keycloak
token, and LiteLLM relays it untouched. That is what lets engram attribute a memory to a person
and lets the Kubernetes MCP reach the k3s API as that person rather than as a shared service
account. It also lets dns-control log each tool call under the identity of the caller.
| Header | Carries | Purpose |
|---|---|---|
x-litellm-api-key: Bearer <virtual key> |
the LiteLLM virtual key | admission, route scoping, rate limits, budget, spend attribution |
Authorization: Bearer <Keycloak access token> |
the person | relayed verbatim to the upstream, which validates it |
If a client puts its virtual key in Authorization on these three routes, LiteLLM forwards it to
the upstream. There is no redirect, no 401 and no log line, and the route keeps working. This is
why x-litellm-api-key is used on all twelve routes rather than only where a relay exists. A client
that never puts a virtual key in Authorization cannot leak one.
Each route needs a Keycloak token with a different audience. Keycloak stamps each as a fixed string rather than deriving it from the URL the client calls:
| Route | Required token audience (aud) |
Where it is set |
|---|---|---|
engram |
mcp-public |
audience in the memory-mcp values, argocd/cluster-app/templates/agent-memory.yaml |
kubernetes |
https://llm.fzymgc.house/kubernetes/mcp |
two mappers in tf/keycloak/mcp_kubernetes.tf, oauth_audience in argocd/app-configs/kubernetes-mcp/config.yaml, and k3s_oidc_audiences in ansible/roles/k3s-common/defaults/main.yml |
dns |
https://llm.fzymgc.house/dns/mcp |
the audience mapper of the scope mcp-dns in tf/keycloak/mcp_dns.tf, and MCP_AUDIENCE of dns-control, whose default in apps/dns-control/internal/config/config.go holds this value |
The audience identifies the protected resource, not the host the client reaches it through. You do not obtain these tokens yourself. Claude Code does, through discovery, registration and PKCE. See The identity routes.
LiteLLM does not validate the token it relays. The upstream does, and it fails closed. A bad,
expired or wrong-audience token therefore produces an error from the upstream, which arrives as a
successful HTTP response that carries a JSON-RPC error, not as a LiteLLM 401. If a call is
refused while tools/list works, read the token’s aud and exp before you suspect the route.
Write the endpoint path with no trailing slash. https://llm.fzymgc.house/context7/mcp serves.
The form /context7/mcp/ answers 307 to the slash-less URL, and a client that does not repeat a
POST after a redirect loses its body. Make sure that the path has no trailing slash:
curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' -X POST https://llm.fzymgc.house/context7/mcp/ \ -H 'content-type: application/json' -d '{}' # 307 https://llm.fzymgc.house/context7/mcpThe aggregate /mcp/ endpoint has the opposite rule, where the slash is required. Do not carry
the habit across.
If a client needs an exact tool list for a route, read it from that route’s tools/list rather
than deriving it. The prefix is predictable, and the underlying names are the vendor’s.
How authentication works
Section titled “How authentication works”The dual-header contract
Section titled “The dual-header contract”One header admits the client on every route. A second header carries the person, on three routes. The model is uniform on purpose.
| Header | Sent on | Carries | Who reads it |
|---|---|---|---|
x-litellm-api-key: Bearer <virtual key> |
all twelve routes, always | the LiteLLM virtual key | LiteLLM: admission, route scoping, rate limits, budget, spend attribution |
Authorization: Bearer <Keycloak access token> |
engram, kubernetes and dns only |
the person | the upstream, which validates it. LiteLLM relays it verbatim and validates nothing |
Three authorization dispositions, not a two-way safety split
Section titled “Three authorization dispositions, not a two-way safety split”Twelve entries render under mcp_servers, and each falls into exactly one disposition:
-
Seven routes (
context7,exa,firecrawl,fal,firewalla_ro,clickhouse_roandclickhouse_rw) carryauth_type: bearer_tokenwith anauthentication_token. LiteLLM composes its ownAuthorizationfrom that secret, so whatever a caller put in that header is overwritten before the request leaves the gateway. The upstream sees the gateway, not the person, and the upstream secret never reaches the client. -
Three routes (
engram,kubernetesanddns) carryextra_headers: ["Authorization"]together withoauth_passthrough: true, and relay the caller’s header to the upstream. LiteLLM holds no upstream credential for them. The person’s own Keycloak token is the credential. This is what makes a virtual key placed inAuthorizationan exfiltration on these routes.kubernetesvalidates the relayed token itself before the k3s API server validates it a second time: issuerhttps://id.fzymgc.house/realms/fzymgc, signature against the realm’s JWKS, audiencehttps://llm.fzymgc.house/kubernetes/mcp. A token minted for another audience is refused by kubernetes-mcp with401 invalid_token. A Keycloak outage answers401 temporarily_unavailableon this route.dnsalso validates the relayed token itself: issuerhttps://id.fzymgc.house/realms/fzymgc, audiencehttps://llm.fzymgc.house/dns/mcpand thedns-readerrole. It also accepts a staticdnsk_key for a service that is not an agent. See dns-control → Who can call. -
Two routes (
deepwikiandunifi_ro) carry neither an upstream credential noroauth_passthrough. The upstream ofdeepwikiis a third party. What LiteLLM does with a caller’sAuthorizationheader on it is not observable from this estate, which is one more reason to keep the virtual key out of that header everywhere. The upstream ofunifi_rois the in-clusterunifi-mcpService, which takes no credential at all. The key scope at LiteLLM and the Cilium policy of theunifi-mcpnamespace, which admits only thelitellmnamespace, are its boundary.
Read the dispositions back from the proxy with the master key:
curl -sS https://llm.fzymgc.house/v1/mcp/server \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ | jq -r '.[] | "\(.server_name) oauth_passthrough=\(.oauth_passthrough)"' # true on engram, kubernetes and dns onlyNever collapse the two headers into one, and never move the virtual key into Authorization
because it also works. On the seven overwriting routes it works and costs nothing, because the
upstream credential replaces whatever the client sent. On the three relay routes it silently
exfiltrates the key.
How the identity routes discover Keycloak
Section titled “How the identity routes discover Keycloak”An unauthenticated request to engram or kubernetes answers 401 with
WWW-Authenticate: Bearer resource_metadata="https://llm.fzymgc.house/.well-known/oauth-protected-resource/<server>/mcp".
That document is the upstream’s own RFC 9728 protected-resource document. LiteLLM relays it with
only resource rewritten to the LiteLLM URL. LiteLLM never synthesizes one, and it returns 502
when the upstream has none. The document names Keycloak
(https://id.fzymgc.house/realms/fzymgc) as the authorization server and lists the scopes the
route wants. The client then registers itself against Keycloak (RFC 7591, allowed for loopback
redirect URIs by the realm’s Trusted-Hosts policy), runs PKCE in the browser, and presents the
resulting bearer. Nothing is handed over out of band, and no static Keycloak client id is
required.
# What a conforming client reads first. resource == the LiteLLM URL; authorization_servers[0] == Keycloak.curl -sS https://llm.fzymgc.house/.well-known/oauth-protected-resource/kubernetes/mcp | jq .# scopes_supported: mcp-kubernetes, groups, offline_accesscurl -sS https://llm.fzymgc.house/.well-known/oauth-protected-resource/engram/mcp | jq .# scopes_supported: offline_accesscurl -sS https://llm.fzymgc.house/.well-known/oauth-protected-resource/dns/mcp | jq .# scopes_supported: mcp-dns, offline_accessdns-control serves its document on port 8081, and LiteLLM relays it in the same way.
kubernetes-mcp publishes its server_url as authorization_servers, so that key holds the
Keycloak issuer. The comment in argocd/app-configs/kubernetes-mcp/config.yaml explains the
misuse.
SSE is served at the aggregate path only
Section titled “SSE is served at the aggregate path only”LiteLLM has no per-server SSE endpoint. The legacy transport is mounted once, at the aggregate
path /mcp/sse, and no path segment selects a server on it. /{server}/mcp/sse, /{server}/sse
and a bare /sse all return 404. Do not construct a per-server SSE URL.
Streamable HTTP at /{server}/mcp is the transport every client on this page uses. The aggregate
lane exists for clients that speak only the older transport, and it is not a working substitute
for a per-server route.
Getting a client virtual key
Section titled “Getting a client virtual key”One key per (user, server set)
Section titled “One key per (user, server set)”A LiteLLM virtual key is minted against /key/generate with the master key, and it carries three
things that matter: who the key is for, the servers it reaches, and its limits.
curl -sS -X POST https://llm.fzymgc.house/key/generate \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' \ -d '{"key_alias":"<name>-mcp", "user_id":"<name>", "object_permission":{"mcp_servers":["context7","exa"]}, "rpm_limit":120, "max_budget":10}' \ | jq -r '.key'| Field | What it does |
|---|---|
object_permission.mcp_servers |
The route names this key reaches. This list is the entire boundary. Use the underscore spellings (clickhouse_ro, not clickhouse-ro) |
user_id |
Whose key it is. Spend and the identity attribute on every MCP span are attributed to it |
key_alias |
The label that appears on the span and in spend reports |
rpm_limit / max_budget |
Per-key limits. Optional, and worth setting on anything shared |
A key with no object_permission list reaches nothing, unless its user has the role
proxy_admin. For every other user, that is the intended default. Mint per
consumer, scoped to the set that consumer needs. One broad key shared around is a larger blast
radius and is harder to attribute.
Plan 03-13 measured the scope rules of LiteLLM v1.103.0-rc.1:
- A key with no
object_permissionlist whose user has the roleproxy_adminreaches every route. - A key with an explicit list reaches only the routes on that list. This rule holds when the user
is a
proxy_admin, and it holds for an empty list[]. - An access group that holds the key adds its routes to the list of the key.
- The scope refusal of LiteLLM is HTTP 403 with
The key is not allowed to access the requested MCP servers.
Scope is a real boundary. A key scoped to a subset sees only that subset’s tools in tools/list,
and tools/call on any other route is refused. This is the only thing that separates
clickhouse_ro from clickhouse_rw. Both instances validate the same upstream token, so nothing
about the secret distinguishes them. Adding clickhouse_rw to a key grants the write surface.
The master key lives in Vault at secret/fzymgc-house/cluster/litellm-keys/master, property
master_key. The workstation key for Claude Code is api_key on
secret/fzymgc-house/cluster/litellm-keys/claude-code. See
LiteLLM → Minting and scoping virtual keys
for the mint bodies and the budget rules.
object_permission.mcp_servers accepts the server name on the way in, and GET /key/info reads
it back as the server’s internal id. That is the same scope, recorded canonically. Do not
“correct” it.
On LiteLLM v1.103.0-rc.1, a key with no grant for a route gets HTTP 403 at initialize. The
body is {"detail":{"error":"The key is not allowed to access the requested MCP servers: <server>"}}.
A 403 at initialize is a scope refusal by LiteLLM, and the upstream never saw the request.
A Terraform-owned key and an access group
Section titled “A Terraform-owned key and an access group”The ncecere/litellm provider has no argument for object_permission. A key in tf/litellm
therefore gets its MCP routes from a LiteLLM unified access group. tf/litellm/access_groups.tf
declares each group with litellm_unified_access_group. access_mcp_server_ids lists the server
ids, and assigned_key_ids lists the management ids of the keys.
The group mcp-dns grants the dns route to the dns-agent-triage key only. The group
mcp-life-agent grants context7, exa, firecrawl, fal, deepwiki and engram to the
life-agent key only. The engram grant admits the key. The identity of the caller is the
engram static token that the life agent sends in Authorization. No route sets
allow_all_keys, so a key reaches a route only through its own grant or through a group. The HCP
workspace main-cluster-litellm applies a group change on merge. Read a group back with the
master key:
curl -sS https://llm.fzymgc.house/v1/access_group \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq .The server id of a route
Section titled “The server id of a route”A key grant and a group grant store the server id, not the route name. LiteLLM derives the id of
a route in values.yaml from its name, url, transport, auth_type and alias, unless the
route pins server_id. If you edit one of those fields on a route with no pin, the route gets a
new id. Every grant that stores the old id then stops matching.
Every route pins server_id. The dns route pins server_id: dns. Each other route pins the id
that its grants already stored, so the pins changed no grant. Read the ids back with the master
key:
curl -sS https://llm.fzymgc.house/v1/mcp/server \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ | jq -r '.[] | "\(.server_name) \(.server_id)"'Claude Code
Section titled “Claude Code”LiteLLM serves all twelve servers. Add only the ones you need. In claude mcp add, the option
--header takes one or more values, so it takes every word after it up to the next flag. The
server name and the URL therefore come before --header. --scope user makes the server
available across all projects (--scope local limits it to the current repository).
The nine key-scoped routes take the virtual key in a header:
claude mcp add --transport http context7 https://llm.fzymgc.house/context7/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope userclaude mcp add --transport http exa https://llm.fzymgc.house/exa/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope userclaude mcp add --transport http firecrawl https://llm.fzymgc.house/firecrawl/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope userclaude mcp add --transport http fal https://llm.fzymgc.house/fal/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope userclaude mcp add --transport http deepwiki https://llm.fzymgc.house/deepwiki/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope userclaude mcp add --transport http firewalla_ro https://llm.fzymgc.house/firewalla_ro/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope userclaude mcp add --transport http clickhouse_ro https://llm.fzymgc.house/clickhouse_ro/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope userclaude mcp add --transport http unifi_ro https://llm.fzymgc.house/unifi_ro/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope userclickhouse_rw is not in that list on purpose. It is the write instance of the telemetry store,
and adding it is a decision rather than a paste:
# Only if this client genuinely needs to WRITE to ClickHouse. The key must also# name clickhouse_rw in object_permission.mcp_servers, or every tools/call is# refused (see the scoping note below).claude mcp add --transport http clickhouse_rw https://llm.fzymgc.house/clickhouse_rw/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>" --scope userThe identity routes
Section titled “The identity routes”engram, kubernetes and dns authenticate the person through OAuth. There is no second static
header. The virtual key still admits the client to LiteLLM, in x-litellm-api-key. Claude Code
obtains the Keycloak token itself, through the protected-resource document on
llm.fzymgc.house, dynamic client registration against Keycloak, and PKCE in the browser. The
three entries use different fixed callback ports so their OAuth callbacks never collide:
# kubernetes: kubernetes-mcp publishes its own discovery document and validates# the token itself; Keycloak is the issuer. The token's aud carries# https://llm.fzymgc.house/kubernetes/mcp and the k3s API authorizes it as# `oidc:<your-email>`.claude mcp add --transport http --scope user --callback-port 51337 \ kubernetes https://llm.fzymgc.house/kubernetes/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>"# then in Claude Code: /mcp → kubernetes → Authenticate (browser: Keycloak) → tools appear
# engram: the same shape. memory-mcp publishes its own discovery document and# validates the token against audience mcp-public.claude mcp add --transport http --scope user --callback-port 51338 \ engram https://llm.fzymgc.house/engram/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>"# then: /mcp → engram → Authenticate (browser: Keycloak) → tools appear
# dns: dns-control publishes its own discovery document and validates the# token itself against audience https://llm.fzymgc.house/dns/mcp. The token# needs the dns-reader role, which the Keycloak group dns-readers grants.claude mcp add --transport http --scope user --callback-port 51339 \ dns https://llm.fzymgc.house/dns/mcp \ --header "x-litellm-api-key: Bearer <your-virtual-key>"# then: /mcp → dns → Authenticate (browser: Keycloak) → tools appearDo not put a static Authorization header on any of these entries. Claude Code uses a configured
headers.Authorization as the server’s credential and disables its OAuth flow for that server,
and on these three routes LiteLLM relays whatever that header carries to the upstream. The virtual
key goes in x-litellm-api-key and nowhere else.
If Keycloak refuses the dynamic registration (the symptom reads like “does not support dynamic
client registration”), add --client-id mcp-public to the command. That is the realm’s static
public PKCE client (tf/keycloak/mcp_public.tf). For kubernetes, also set that entry’s
oauth.scopes in ~/.claude.json to mcp-kubernetes groups offline_access, so the token still
carries the resource audience. For dns, add --client-id mcp-dns instead, and set the entry’s
oauth.scopes to mcp-dns offline_access.
A person’s virtual key reaches dns by the rules in
One key per (user, server set). A key of a proxy_admin user with
no object_permission list reaches dns with no change. Any other key needs dns in its
object_permission.mcp_servers, and an explicit list binds a proxy_admin key too. The access
group mcp-dns grants the route to the dns-agent-triage key only.
Claude Code registers a client through Keycloak DCR (dynamic client registration). Keycloak
creates that anonymous client with full scope turned off. A role then reaches the token only
through a role scope mapping of a client scope. The client scope mcp-dns maps the dns-control
role dns-reader and emits preferred_username (tf/keycloak/mcp_dns.tf). The person must be
in the Keycloak group dns-readers.
Scoping notes
Section titled “Scoping notes”The two ClickHouse routes present the same upstream credential. Both MCP instances validate one
token, so nothing about the secret separates them. The only thing that does is which server your
key is scoped to. A key scoped to clickhouse_ro sees none of the write instance’s tools.
The four in-cluster routes reach Kubernetes Services rather than internet vendors. That is
invisible from the client side. A slower first response from firewalla_ro is the appliance
being busy, not a gateway fault.
unifi_ro registers its tools lazily. The server lists seven tools at initialize: five
meta-tools, which start with unifi_ro-unifi_, and two read tools. Find a tool with
unifi_ro-unifi_tool_index, and run it with unifi_ro-unifi_execute and the tool name. Do not
call unifi_ro-unifi_load_tools. It loads a tool into the server and announces it with a
tools/list_changed notification, which LiteLLM does not relay, so the loaded tool never appears
in the client. Every create, update and delete tool answers success: false with
is disabled by policy, because the deployment keeps every policy gate off.
The client header is identical on all nine key-scoped routes, including deepwiki and
unifi_ro. deepwiki
needs no upstream credential, but every route on llm.fzymgc.house still requires an admitted
virtual key, and a key not scoped to deepwiki is refused it like any other.
Mint the virtual key scoped to the servers this client needs. See
Getting a client virtual key for the /key/generate body and
what each field does.
Claude Desktop
Section titled “Claude Desktop”Claude Desktop has no field for a static header on remote connectors, so the mcp-remote stdio
bridge injects it. The value carries the Bearer scheme prefix and therefore contains a space,
so keep the value in an environment variable and write --header with no space after the colon.
Edit ~/Library/Application Support/Claude/claude_desktop_config.json:
{ "mcpServers": { "context7": { "command": "npx", "args": [ "-y", "mcp-remote", "https://llm.fzymgc.house/context7/mcp", "--header", "x-litellm-api-key:${LITELLM_MCP_KEY}" ], "env": { "LITELLM_MCP_KEY": "Bearer <your-virtual-key>" } }, "exa": { "command": "npx", "args": [ "-y", "mcp-remote", "https://llm.fzymgc.house/exa/mcp", "--header", "x-litellm-api-key:${LITELLM_MCP_KEY}" ], "env": { "LITELLM_MCP_KEY": "Bearer <your-virtual-key>" } } }}Repeat the exa block for firecrawl, fal, deepwiki, firewalla_ro, clickhouse_ro and
unifi_ro,
and swap the server name in both the key and the /<server>/mcp path. The same virtual key works
for every server it is scoped to. env is per entry in this file, so the value is repeated.
Add a clickhouse_rw block only if this client must write to the telemetry store, and only with
a key scoped to clickhouse_rw. Without that scope, LiteLLM refuses the entry at initialize
with HTTP 403.
The identity routes (engram, kubernetes, dns) are not configured here. A native Claude Desktop
connector runs the OAuth flow but cannot send the x-litellm-api-key admission header beside it.
The bridge sends a static header but has no OAuth flow. Neither produces the two-header shape
these routes require. Claude Code is the client for those three routes.
Claude Desktop’s argument parser mangles header values that contain spaces, such as
Bearer <key>. Put the value in an environment variable and write the --header flag with no
space after the colon.
Fully quit Claude Desktop (⌘Q) and relaunch. It reads the configuration only at startup.
Verify
Section titled “Verify”Smoke-test a key-scoped route with the virtual key alone. The path shape, the missing trailing
slash and the Bearer prefix inside x-litellm-api-key are the three things that are easy to
get wrong, and each one surfaces as a 401 or a 307 rather than a useful message:
curl -sS -X POST https://llm.fzymgc.house/exa/mcp \ -H "x-litellm-api-key: Bearer <your-virtual-key>" \ -H 'content-type: application/json' \ -H 'accept: application/json, text/event-stream' \ -d '{"jsonrpc":"2.0","method":"initialize","id":1, "params":{"protocolVersion":"2025-06-18","capabilities":{}, "clientInfo":{"name":"curl","version":"0"}}}' -D-Run initialize first, because LiteLLM issues an mcp-session-id response header that
tools/list needs. LiteLLM refuses a key with no grant for the route at initialize, with
HTTP 403. See Getting a client virtual key for the body of the
refusal.
An identity route, unauthenticated, must answer the discovery challenge:
curl -s -D - -o /dev/null -X POST https://llm.fzymgc.house/kubernetes/mcp \ -H 'content-type: application/json' -d '{}' | rg -i '^HTTP/|^www-authenticate'# HTTP/2 401# www-authenticate: Bearer resource_metadata="https://llm.fzymgc.house/.well-known/oauth-protected-resource/kubernetes/mcp"The connected state from the client side:
claude mcp get kubernetes # shows connected state + tool count after OAuthWhich routes the proxy serves, independent of any client:
curl -sS https://llm.fzymgc.house/v1/mcp/server \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -r '.[].server_name'Rollback
Section titled “Rollback”A client-side problem is fixed on the client. There is no second host to repoint at. A wrong header, a wrong path spelling or a key not scoped to the route is a client configuration fix, diagnosed under Troubleshooting.
A LiteLLM-side regression on one route is a GitOps revert of that route’s commit. A route lands
as one change: its mcp_servers entry in argocd/app-configs/litellm-chart/values.yaml, its
extraEnvVars entry in the same file, and its key in argocd/app-configs/litellm/secrets.yaml.
Reverting the commit removes all three together. Nothing on the client changes.
Troubleshooting
Section titled “Troubleshooting”If every MCP behind llm.fzymgc.house fails at Claude Code startup while public MCPs connect,
the fault is on the Mac’s LAN side, not in the gateway. On a Mac that is dual-homed onto the Main
network, a rejected on-link route or a missing Local Network permission for the responsible app
blocks the connection before any header is sent. Read the route first, then the permission:
route -n get llm.fzymgc.house | rg 'gateway|flags' # a "!" flag marks a reject routeGrant Local Network to the app that owns the connection in System Settings, Privacy & Security, Local Network, then retry. Only after both read clean is the gateway the suspect.
LiteLLM: the upstream returns 401 and it looks like a bad secret
Section titled “LiteLLM: the upstream returns 401 and it looks like a bad secret”Symptom: initialize succeeds, tools/list returns tools, and tools/call comes back as an
HTTP 200 that carries a JSON-RPC error with the upstream’s own “unauthorized” or
“invalid token” wording. Nothing in LiteLLM’s own response is a 401, and rotating the upstream
secret does not help.
Cause: a wrong header name, not a wrong secret. Two shapes produce this, and both survive a credential rotation:
- Client side, on
engram,kubernetesordns. The virtual key was placed inAuthorizationinstead ofx-litellm-api-key. Those three routes relayAuthorizationto the upstream verbatim, so the upstream is handed a LiteLLM virtual key and rejects it. Fix: move the virtual key intox-litellm-api-key: Bearer <key>and reserveAuthorizationfor the Keycloak token. The key reached the upstream, so rotate it. - Configuration side, on any credentialed route. The Vault property was stored with an
Authorization: Bearer …prefix already baked in. LiteLLM’sauth_type: bearer_tokencomposes the header itself, so a pre-prefixed value goes out asBearer Bearer <token>. The upstream sees a malformed credential and returns401. Fix: store the bare token.
Distinguish them in one step. If you can reproduce it with curl that sends only
x-litellm-api-key, it is shape 2, a configuration fault. If sending the headers correctly makes
it go away, it was shape 1.
LiteLLM: initialize answers 403
Section titled “LiteLLM: initialize answers 403”Symptom: the client does not connect. initialize returns HTTP 403 with
The key is not allowed to access the requested MCP servers: <server>.
Cause: the key has no grant for that server. object_permission.mcp_servers on the virtual key
does not name the route, and no access group that holds the key lists the server id of the route.
The upstream never saw the request.
Fix: read the key’s scope back and compare it to the route name, in the underscore spelling:
curl -sS "https://llm.fzymgc.house/key/info" \ -H "Authorization: Bearer <your-virtual-key>" | jq '.info.object_permission'Authenticate as the key. Never put it in the query string. With no key parameter, /key/info
reports the key in the Authorization header. ?key=<value> lands the credential in LiteLLM’s
access log, in Traefik’s log on the way through the public ingress, in the url.full attribute of
the OTel span, and in /proc/<pid>/cmdline for the life of the curl. See
Verifying a minted key for the same rule with its
upstream reference. A key restricted to llm_api_routes cannot call /key/info at all and
returns 401. Use the /key/list and /budget/list pair described there, and rotate any key you
have already exposed in a URL.
Then mint or update a key that names that server. GET /key/info reads the scope back as
internal ids rather than names. That is canonical storage, not drift.
LiteLLM: something that used to arrive stopped arriving, with no error
Section titled “LiteLLM: something that used to arrive stopped arriving, with no error”Symptom: tools/list and tools/call still work. What stopped is anything that needs session
continuity, such as a server-initiated notification or a sampling or elicitation round trip. The
symptom of a lost session is the missing notification.
Cause: MCP session state is process-local. A proxy restart, or a request that lands on the other replica, drops it, and LiteLLM downgrades to stateless service rather than erroring.
Do not debug this from the client. The behaviour, what does and does not survive, and the two conditions that do produce errors are documented once, in LiteLLM → MCP session semantics across restarts and replicas.
LiteLLM: an identity route’s upstream is down and nothing says so
Section titled “LiteLLM: an identity route’s upstream is down and nothing says so”Symptom: engram, kubernetes or dns fails at tools/call with no proxy-side warning. No failing
health check, no route marked unhealthy, no metric. The first signal is a client’s failed call.
Cause: the three identity routes carry auth_type: none and an Authorization entry in
extra_headers. LiteLLM reads that shape as per-user auth and skips its health probe, because it
has no credential to probe the upstream on its own behalf. GET /v1/mcp/server/health reports
these routes as unknown. This is correct.
If one of these routes reads unhealthy, make sure that its entry in
argocd/app-configs/litellm-chart/values.yaml still sets auth_type: none. LiteLLM skips the
probe only for none. If auth_type is absent, the probe sends no credential and gets 401.
Do not give those routes a credential for the probe. LiteLLM applies static_headers after the
relayed header, so a static Authorization ends the per-user identity relay that the routes exist
to provide.
Fix: look at the upstream Deployment directly, memory-mcp in agent-memory, kubernetes-mcp
in kubernetes-mcp or dns-control in dns-control. LiteLLM probes the nine other routes
normally.
engram / kubernetes: client never connects, or no OAuth prompt
Section titled “engram / kubernetes: client never connects, or no OAuth prompt”The client only starts its OAuth flow if discovery works. Read the relayed document first:
curl -sS -o /dev/null -w '%{http_code}\n' \ https://llm.fzymgc.house/.well-known/oauth-protected-resource/engram/mcp502: the upstream is not serving its own protected-resource document, or LiteLLM’s negative cache holds an earlier failure. Forkubernetes, make sure that theauthorization_urlandserver_urlblock in the kubernetes-mcp ConfigMap is what the pod runs. A ConfigMap-only change does not roll the pod, so runkubectl -n kubernetes-mcp rollout restart deploy/kubernetes-mcp. Forengram, make sure that the memory-mcp pod is up. After the upstream serves the document, restart LiteLLM to clear the negative cache (kubectl rollout restart deployment/litellm -n litellm).404: the URL is wrong. The document lives at/.well-known/oauth-protected-resource/<server>/mcpon the host root. The server root and the/mcp/<server>order both miss.200but the client still shows no prompt: the server URL in the client entry is wrong (a bare/mcpreturns404 route not foundrather than triggering the flow), or a staticheaders.Authorizationis configured on the entry, which disables Claude Code’s OAuth flow for that server.- “does not support dynamic client registration”: Keycloak refused the registration. The realm’s
Trusted-Hosts policy admits loopback redirect URIs only
(
argocd/app-configs/keycloak/dcr-trusted-hosts-job.yaml). A client that registers with any other redirect URI is rejected. Use the--client-id mcp-publicfallback above. - Registration refused with
403and “It’s allowed to have max 500 clients per realm”: the realm’s anonymous Max Clients Limit policy counts every client in the realm, not only registrations, anddcr-trusted-hosts-job.yamlsets it to 500. The nightlykeycloak-dcr-reaperCronJob (argocd/app-configs/keycloak/dcr-client-reaper-cronjob.yaml) keeps headroom by deleting public, UUID-named clients that have no active or offline session. Make sure that it runs and succeeds withkubectl -n keycloak get cronjob keycloak-dcr-reaperand the logs of its most recent Job.
engram / kubernetes: re-authentication stops at Keycloak “Client not found”
Section titled “engram / kubernetes: re-authentication stops at Keycloak “Client not found””The reaper deleted the client this entry registered, because it no longer held an active or offline session. Claude Code re-registers by itself when the token endpoint rejects the client id. When it holds the registration without tokens, it replays the deleted client id at the authorize endpoint instead. The rejection happens in the browser and never reaches Claude Code (anthropics/claude-code#84614).
Fix: /mcp, the server, Clear authentication, then Authenticate. Or run
claude mcp remove <name> and re-add it with the same claude mcp add command from
The identity routes.
dns: the server rejected the new credentials on reconnect
Section titled “dns: the server rejected the new credentials on reconnect”Symptom: Claude Code reports that it got new credentials, but dns rejected them on reconnect.
Cause: dns-control refused the Keycloak token. dns-control answers 401 with
error="invalid_token" in WWW-Authenticate. A refusal by LiteLLM answers 403 instead, as in
LiteLLM: initialize answers 403. dns-control refuses a token
without dns-reader in resource_access.dns-control.roles, and a token for another audience.
Fix:
-
Decode the claims of the token without printing the token. With the access token in
TOKEN, run the command below. Make sure thatresource_access.dns-control.rolesholdsdns-readerand thataudholdshttps://llm.fzymgc.house/dns/mcp.Terminal window printf '%s' "$TOKEN" | cut -d. -f2 | tr '_-' '/+' \| jq -R '@base64d | fromjson | {aud, azp, preferred_username, resource_access}' -
Re-authenticate:
/mcp,dns, Clear authentication, then Authenticate. -
If the role is still absent, run
claude mcp remove dns --scope user. Add the entry again with the command in The identity routes, and sign in. The sign-in registers a new DCR client.
If the role is absent after a new registration, make sure that the person is in the Keycloak
group dns-readers.
Key-scoped route: 401/403 from the gateway
Section titled “Key-scoped route: 401/403 from the gateway”The virtual key is wrong, not sent, sent in the wrong header, or sent without the Bearer scheme
prefix inside x-litellm-api-key. Re-run the Verify curl. A 200 with an mcp-session-id
confirms the key and the endpoint.
Claude Desktop server fails to authenticate
Section titled “Claude Desktop server fails to authenticate”Almost always the header-quoting problem. Make sure that the --header argument is
x-litellm-api-key:${LITELLM_MCP_KEY} with no space after the colon, and that
env.LITELLM_MCP_KEY holds Bearer <key>. Quit and relaunch after any edit.
Related
Section titled “Related”- LiteLLM operations: the operator runbook for key minting, restart and replica semantics, and database-outage behaviour
- AI Tooling: general MCP server configuration
- Services catalog: LiteLLM architecture and the route inventory
- Secrets & Vault Paths: Vault path conventions
- ADR Keycloak DCR deny removed: why a client can register directly against Keycloak
- LiteLLM MCP route declarations:
argocd/app-configs/litellm-chart/values.yaml(proxy_config.mcp_servers, the estate’s only sanctioned MCP registration point) - kubernetes-mcp discovery configuration:
argocd/app-configs/kubernetes-mcp/config.yaml - dns-control → The MCP path: the six
dns-tools, the callers and the static key procedure, indocs/operations/dns-control.md