Skip to content

MCP Gateway — Client Setup

This page connects Claude Code and Claude Desktop to the cluster’s MCP plane. LiteLLM at llm.fzymgc.house fronts twelve upstream MCP servers behind one host and exposes each at its own path. One header admits a client on every route. On the three identity routes, engram, kubernetes and dns, a second header carries the user’s own Keycloak token, which the client obtains through OAuth discovery, dynamic client registration and PKCE. llm.fzymgc.house is the only MCP host. To add a new identity route, read MCP Identity Routes.

LiteLLM prefixes every tool name with the server name. resolve-library-id on context7 is context7-resolve-library-id here. Anything that pins a tool name by string, such as an allowlist, a disallowedTools entry, an agent prompt or a hook, needs the prefix.

Property Value
Gateway host https://llm.fzymgc.house
MCP endpoint (per server) https://llm.fzymgc.house/<server>/mcp, with no trailing slash
Servers Twelve routes in three dispositions, listed below
Admission A per-user LiteLLM virtual key in the x-litellm-api-key header, value Bearer <key>, on every route
Identity (engram, kubernetes, dns) Authorization: Bearer <Keycloak access token> beside the admission header. The client obtains it through discovery, registration and PKCE. The dns upstream also accepts a static dnsk_ key
Tool names Prefixed with the server name: context7-, exa-, firecrawl-, fal-, deepwiki-, firewalla_ro-, clickhouse_ro-, clickhouse_rw-, unifi_ro-, engram-, kubernetes-, dns-
Key scoping The key’s object_permission.mcp_servers names the servers it reaches. An access group that lists the server id also grants the route. LiteLLM refuses a key with no grant for a route at initialize with HTTP 403. The exception is a key of a proxy_admin user with no object_permission list, which reaches every route. An explicit list binds a proxy_admin key too, even an empty list
Write surface clickhouse_rw is the only route on this host that can change state. It needs a key whose object_permission.mcp_servers names clickhouse_rw. clickhouse_ro is the read-only instance and is what almost every client wants
Upstream credential Held by LiteLLM and injected outbound. It never reaches the client
Claude Code Native. It handles OAuth and custom headers on all twelve routes
Claude Desktop The nine key-scoped routes through the mcp-remote bridge. The three identity routes are not reachable, because its connector cannot send the second header

The twelve routes and what each does with your Authorization header

Section titled “The twelve routes and what each does with your Authorization header”

The routes are declared once, under proxy_config.mcp_servers in argocd/app-configs/litellm-chart/values.yaml.

Route Upstream Disposition What the client sends
context7 mcp.context7.com (SaaS) overwrite: LiteLLM composes its own Authorization from a secret it holds x-litellm-api-key only
exa mcp.exa.ai (SaaS, three-tool allowlist in the URL) overwrite x-litellm-api-key only
firecrawl mcp.firecrawl.dev (SaaS) overwrite x-litellm-api-key only
fal mcp.fal.ai (SaaS) overwrite x-litellm-api-key only
firewalla_ro firewalla-mcp (in-cluster, read-only) overwrite x-litellm-api-key only
clickhouse_ro clickhouse-mcp-ro (in-cluster, read-only) overwrite x-litellm-api-key only
clickhouse_rw clickhouse-mcp-rw (in-cluster, read-write) overwrite x-litellm-api-key only, on a key scoped to clickhouse_rw
engram memory-mcp (in-cluster) relay: oauth_passthrough: true, the Keycloak token goes to the upstream verbatim x-litellm-api-key and Authorization: Bearer <Keycloak token>
kubernetes kubernetes-mcp (in-cluster) relay, and the upstream validates the token itself x-litellm-api-key and Authorization: Bearer <Keycloak token>
dns dns-control (in-cluster, port 8081) relay, and the upstream validates the token itself x-litellm-api-key and Authorization: Bearer <Keycloak token or dnsk_ static key>. The key reaches dns through object_permission.mcp_servers or through the access group mcp-dns. See dns-control → The MCP path
deepwiki mcp.deepwiki.com (SaaS) credential-free: no upstream secret and no relay flag x-litellm-api-key only
unifi_ro unifi-mcp (in-cluster, read-only) credential-free: no upstream secret and no relay flag. The key scope and the namespace network policy are the boundary x-litellm-api-key only

Server names use the underscore spelling (firewalla_ro, clickhouse_ro, clickhouse_rw, unifi_ro). LiteLLM rejects a hyphen in an MCP server name at configuration load, because - is the separator it puts between server and tool. That rejection takes the whole proxy down, not one route. When you scope a virtual key, use the underscore form. That is the name LiteLLM matches object_permission.mcp_servers against.

The admission credential goes in x-litellm-api-key and not in Authorization. LiteLLM accepts either header for admission and reads x-litellm-api-key first. Using it on every route keeps the virtual key out of the header that the three identity routes relay to their upstream. A client configuration then needs no change on the day a route gains an identity relay.

The x-litellm-api-key value must carry the Bearer scheme prefix. A bare key in that header is refused with 401 Malformed API Key passed in. The refusal fails closed. x-litellm-api-key takes precedence over Authorization, so a bare key there is refused even when a valid Authorization: Bearer <key> is also present. On the dual-header routes a wrong prefix looks like a bad key.

The three identity routes send TWO headers

Section titled “The three identity routes send TWO headers”

engram, kubernetes and dns are the only routes on this host that relay the caller’s own credential to the upstream. On the seven overwriting routes LiteLLM injects a secret it owns, and the caller’s identity stops at the gateway. On deepwiki LiteLLM holds no upstream credential and carries no relay flag. On the three identity routes the credential is the caller’s own Keycloak token, and LiteLLM relays it untouched. That is what lets engram attribute a memory to a person and lets the Kubernetes MCP reach the k3s API as that person rather than as a shared service account. It also lets dns-control log each tool call under the identity of the caller.

Header Carries Purpose
x-litellm-api-key: Bearer <virtual key> the LiteLLM virtual key admission, route scoping, rate limits, budget, spend attribution
Authorization: Bearer <Keycloak access token> the person relayed verbatim to the upstream, which validates it

If a client puts its virtual key in Authorization on these three routes, LiteLLM forwards it to the upstream. There is no redirect, no 401 and no log line, and the route keeps working. This is why x-litellm-api-key is used on all twelve routes rather than only where a relay exists. A client that never puts a virtual key in Authorization cannot leak one.

Each route needs a Keycloak token with a different audience. Keycloak stamps each as a fixed string rather than deriving it from the URL the client calls:

Route Required token audience (aud) Where it is set
engram mcp-public audience in the memory-mcp values, argocd/cluster-app/templates/agent-memory.yaml
kubernetes https://llm.fzymgc.house/kubernetes/mcp two mappers in tf/keycloak/mcp_kubernetes.tf, oauth_audience in argocd/app-configs/kubernetes-mcp/config.yaml, and k3s_oidc_audiences in ansible/roles/k3s-common/defaults/main.yml
dns https://llm.fzymgc.house/dns/mcp the audience mapper of the scope mcp-dns in tf/keycloak/mcp_dns.tf, and MCP_AUDIENCE of dns-control, whose default in apps/dns-control/internal/config/config.go holds this value

The audience identifies the protected resource, not the host the client reaches it through. You do not obtain these tokens yourself. Claude Code does, through discovery, registration and PKCE. See The identity routes.

LiteLLM does not validate the token it relays. The upstream does, and it fails closed. A bad, expired or wrong-audience token therefore produces an error from the upstream, which arrives as a successful HTTP response that carries a JSON-RPC error, not as a LiteLLM 401. If a call is refused while tools/list works, read the token’s aud and exp before you suspect the route.

Write the endpoint path with no trailing slash. https://llm.fzymgc.house/context7/mcp serves. The form /context7/mcp/ answers 307 to the slash-less URL, and a client that does not repeat a POST after a redirect loses its body. Make sure that the path has no trailing slash:

Terminal window
curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' -X POST https://llm.fzymgc.house/context7/mcp/ \
-H 'content-type: application/json' -d '{}' # 307 https://llm.fzymgc.house/context7/mcp

The aggregate /mcp/ endpoint has the opposite rule, where the slash is required. Do not carry the habit across.

If a client needs an exact tool list for a route, read it from that route’s tools/list rather than deriving it. The prefix is predictable, and the underlying names are the vendor’s.

One header admits the client on every route. A second header carries the person, on three routes. The model is uniform on purpose.

Header Sent on Carries Who reads it
x-litellm-api-key: Bearer <virtual key> all twelve routes, always the LiteLLM virtual key LiteLLM: admission, route scoping, rate limits, budget, spend attribution
Authorization: Bearer <Keycloak access token> engram, kubernetes and dns only the person the upstream, which validates it. LiteLLM relays it verbatim and validates nothing

Three authorization dispositions, not a two-way safety split

Section titled “Three authorization dispositions, not a two-way safety split”

Twelve entries render under mcp_servers, and each falls into exactly one disposition:

  • Seven routes (context7, exa, firecrawl, fal, firewalla_ro, clickhouse_ro and clickhouse_rw) carry auth_type: bearer_token with an authentication_token. LiteLLM composes its own Authorization from that secret, so whatever a caller put in that header is overwritten before the request leaves the gateway. The upstream sees the gateway, not the person, and the upstream secret never reaches the client.

  • Three routes (engram, kubernetes and dns) carry extra_headers: ["Authorization"] together with oauth_passthrough: true, and relay the caller’s header to the upstream. LiteLLM holds no upstream credential for them. The person’s own Keycloak token is the credential. This is what makes a virtual key placed in Authorization an exfiltration on these routes.

    kubernetes validates the relayed token itself before the k3s API server validates it a second time: issuer https://id.fzymgc.house/realms/fzymgc, signature against the realm’s JWKS, audience https://llm.fzymgc.house/kubernetes/mcp. A token minted for another audience is refused by kubernetes-mcp with 401 invalid_token. A Keycloak outage answers 401 temporarily_unavailable on this route.

    dns also validates the relayed token itself: issuer https://id.fzymgc.house/realms/fzymgc, audience https://llm.fzymgc.house/dns/mcp and the dns-reader role. It also accepts a static dnsk_ key for a service that is not an agent. See dns-control → Who can call.

  • Two routes (deepwiki and unifi_ro) carry neither an upstream credential nor oauth_passthrough. The upstream of deepwiki is a third party. What LiteLLM does with a caller’s Authorization header on it is not observable from this estate, which is one more reason to keep the virtual key out of that header everywhere. The upstream of unifi_ro is the in-cluster unifi-mcp Service, which takes no credential at all. The key scope at LiteLLM and the Cilium policy of the unifi-mcp namespace, which admits only the litellm namespace, are its boundary.

Read the dispositions back from the proxy with the master key:

Terminal window
curl -sS https://llm.fzymgc.house/v1/mcp/server \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
| jq -r '.[] | "\(.server_name) oauth_passthrough=\(.oauth_passthrough)"' # true on engram, kubernetes and dns only

Never collapse the two headers into one, and never move the virtual key into Authorization because it also works. On the seven overwriting routes it works and costs nothing, because the upstream credential replaces whatever the client sent. On the three relay routes it silently exfiltrates the key.

An unauthenticated request to engram or kubernetes answers 401 with WWW-Authenticate: Bearer resource_metadata="https://llm.fzymgc.house/.well-known/oauth-protected-resource/<server>/mcp". That document is the upstream’s own RFC 9728 protected-resource document. LiteLLM relays it with only resource rewritten to the LiteLLM URL. LiteLLM never synthesizes one, and it returns 502 when the upstream has none. The document names Keycloak (https://id.fzymgc.house/realms/fzymgc) as the authorization server and lists the scopes the route wants. The client then registers itself against Keycloak (RFC 7591, allowed for loopback redirect URIs by the realm’s Trusted-Hosts policy), runs PKCE in the browser, and presents the resulting bearer. Nothing is handed over out of band, and no static Keycloak client id is required.

Terminal window
# What a conforming client reads first. resource == the LiteLLM URL; authorization_servers[0] == Keycloak.
curl -sS https://llm.fzymgc.house/.well-known/oauth-protected-resource/kubernetes/mcp | jq .
# scopes_supported: mcp-kubernetes, groups, offline_access
curl -sS https://llm.fzymgc.house/.well-known/oauth-protected-resource/engram/mcp | jq .
# scopes_supported: offline_access
curl -sS https://llm.fzymgc.house/.well-known/oauth-protected-resource/dns/mcp | jq .
# scopes_supported: mcp-dns, offline_access

dns-control serves its document on port 8081, and LiteLLM relays it in the same way.

kubernetes-mcp publishes its server_url as authorization_servers, so that key holds the Keycloak issuer. The comment in argocd/app-configs/kubernetes-mcp/config.yaml explains the misuse.

LiteLLM has no per-server SSE endpoint. The legacy transport is mounted once, at the aggregate path /mcp/sse, and no path segment selects a server on it. /{server}/mcp/sse, /{server}/sse and a bare /sse all return 404. Do not construct a per-server SSE URL.

Streamable HTTP at /{server}/mcp is the transport every client on this page uses. The aggregate lane exists for clients that speak only the older transport, and it is not a working substitute for a per-server route.

A LiteLLM virtual key is minted against /key/generate with the master key, and it carries three things that matter: who the key is for, the servers it reaches, and its limits.

Terminal window
curl -sS -X POST https://llm.fzymgc.house/key/generate \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' \
-d '{"key_alias":"<name>-mcp",
"user_id":"<name>",
"object_permission":{"mcp_servers":["context7","exa"]},
"rpm_limit":120,
"max_budget":10}' \
| jq -r '.key'
Field What it does
object_permission.mcp_servers The route names this key reaches. This list is the entire boundary. Use the underscore spellings (clickhouse_ro, not clickhouse-ro)
user_id Whose key it is. Spend and the identity attribute on every MCP span are attributed to it
key_alias The label that appears on the span and in spend reports
rpm_limit / max_budget Per-key limits. Optional, and worth setting on anything shared

A key with no object_permission list reaches nothing, unless its user has the role proxy_admin. For every other user, that is the intended default. Mint per consumer, scoped to the set that consumer needs. One broad key shared around is a larger blast radius and is harder to attribute.

Plan 03-13 measured the scope rules of LiteLLM v1.103.0-rc.1:

  • A key with no object_permission list whose user has the role proxy_admin reaches every route.
  • A key with an explicit list reaches only the routes on that list. This rule holds when the user is a proxy_admin, and it holds for an empty list [].
  • An access group that holds the key adds its routes to the list of the key.
  • The scope refusal of LiteLLM is HTTP 403 with The key is not allowed to access the requested MCP servers.

Scope is a real boundary. A key scoped to a subset sees only that subset’s tools in tools/list, and tools/call on any other route is refused. This is the only thing that separates clickhouse_ro from clickhouse_rw. Both instances validate the same upstream token, so nothing about the secret distinguishes them. Adding clickhouse_rw to a key grants the write surface.

The master key lives in Vault at secret/fzymgc-house/cluster/litellm-keys/master, property master_key. The workstation key for Claude Code is api_key on secret/fzymgc-house/cluster/litellm-keys/claude-code. See LiteLLM → Minting and scoping virtual keys for the mint bodies and the budget rules.

object_permission.mcp_servers accepts the server name on the way in, and GET /key/info reads it back as the server’s internal id. That is the same scope, recorded canonically. Do not “correct” it.

On LiteLLM v1.103.0-rc.1, a key with no grant for a route gets HTTP 403 at initialize. The body is {"detail":{"error":"The key is not allowed to access the requested MCP servers: <server>"}}. A 403 at initialize is a scope refusal by LiteLLM, and the upstream never saw the request.

The ncecere/litellm provider has no argument for object_permission. A key in tf/litellm therefore gets its MCP routes from a LiteLLM unified access group. tf/litellm/access_groups.tf declares each group with litellm_unified_access_group. access_mcp_server_ids lists the server ids, and assigned_key_ids lists the management ids of the keys.

The group mcp-dns grants the dns route to the dns-agent-triage key only. The group mcp-life-agent grants context7, exa, firecrawl, fal, deepwiki and engram to the life-agent key only. The engram grant admits the key. The identity of the caller is the engram static token that the life agent sends in Authorization. No route sets allow_all_keys, so a key reaches a route only through its own grant or through a group. The HCP workspace main-cluster-litellm applies a group change on merge. Read a group back with the master key:

Terminal window
curl -sS https://llm.fzymgc.house/v1/access_group \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq .

A key grant and a group grant store the server id, not the route name. LiteLLM derives the id of a route in values.yaml from its name, url, transport, auth_type and alias, unless the route pins server_id. If you edit one of those fields on a route with no pin, the route gets a new id. Every grant that stores the old id then stops matching.

Every route pins server_id. The dns route pins server_id: dns. Each other route pins the id that its grants already stored, so the pins changed no grant. Read the ids back with the master key:

Terminal window
curl -sS https://llm.fzymgc.house/v1/mcp/server \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
| jq -r '.[] | "\(.server_name) \(.server_id)"'

LiteLLM serves all twelve servers. Add only the ones you need. In claude mcp add, the option --header takes one or more values, so it takes every word after it up to the next flag. The server name and the URL therefore come before --header. --scope user makes the server available across all projects (--scope local limits it to the current repository).

The nine key-scoped routes take the virtual key in a header:

Terminal window
claude mcp add --transport http context7 https://llm.fzymgc.house/context7/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http exa https://llm.fzymgc.house/exa/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http firecrawl https://llm.fzymgc.house/firecrawl/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http fal https://llm.fzymgc.house/fal/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http deepwiki https://llm.fzymgc.house/deepwiki/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http firewalla_ro https://llm.fzymgc.house/firewalla_ro/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http clickhouse_ro https://llm.fzymgc.house/clickhouse_ro/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user
claude mcp add --transport http unifi_ro https://llm.fzymgc.house/unifi_ro/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user

clickhouse_rw is not in that list on purpose. It is the write instance of the telemetry store, and adding it is a decision rather than a paste:

Terminal window
# Only if this client genuinely needs to WRITE to ClickHouse. The key must also
# name clickhouse_rw in object_permission.mcp_servers, or every tools/call is
# refused (see the scoping note below).
claude mcp add --transport http clickhouse_rw https://llm.fzymgc.house/clickhouse_rw/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>" --scope user

engram, kubernetes and dns authenticate the person through OAuth. There is no second static header. The virtual key still admits the client to LiteLLM, in x-litellm-api-key. Claude Code obtains the Keycloak token itself, through the protected-resource document on llm.fzymgc.house, dynamic client registration against Keycloak, and PKCE in the browser. The three entries use different fixed callback ports so their OAuth callbacks never collide:

Terminal window
# kubernetes: kubernetes-mcp publishes its own discovery document and validates
# the token itself; Keycloak is the issuer. The token's aud carries
# https://llm.fzymgc.house/kubernetes/mcp and the k3s API authorizes it as
# `oidc:<your-email>`.
claude mcp add --transport http --scope user --callback-port 51337 \
kubernetes https://llm.fzymgc.house/kubernetes/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>"
# then in Claude Code: /mcp → kubernetes → Authenticate (browser: Keycloak) → tools appear
# engram: the same shape. memory-mcp publishes its own discovery document and
# validates the token against audience mcp-public.
claude mcp add --transport http --scope user --callback-port 51338 \
engram https://llm.fzymgc.house/engram/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>"
# then: /mcp → engram → Authenticate (browser: Keycloak) → tools appear
# dns: dns-control publishes its own discovery document and validates the
# token itself against audience https://llm.fzymgc.house/dns/mcp. The token
# needs the dns-reader role, which the Keycloak group dns-readers grants.
claude mcp add --transport http --scope user --callback-port 51339 \
dns https://llm.fzymgc.house/dns/mcp \
--header "x-litellm-api-key: Bearer <your-virtual-key>"
# then: /mcp → dns → Authenticate (browser: Keycloak) → tools appear

Do not put a static Authorization header on any of these entries. Claude Code uses a configured headers.Authorization as the server’s credential and disables its OAuth flow for that server, and on these three routes LiteLLM relays whatever that header carries to the upstream. The virtual key goes in x-litellm-api-key and nowhere else.

If Keycloak refuses the dynamic registration (the symptom reads like “does not support dynamic client registration”), add --client-id mcp-public to the command. That is the realm’s static public PKCE client (tf/keycloak/mcp_public.tf). For kubernetes, also set that entry’s oauth.scopes in ~/.claude.json to mcp-kubernetes groups offline_access, so the token still carries the resource audience. For dns, add --client-id mcp-dns instead, and set the entry’s oauth.scopes to mcp-dns offline_access.

A person’s virtual key reaches dns by the rules in One key per (user, server set). A key of a proxy_admin user with no object_permission list reaches dns with no change. Any other key needs dns in its object_permission.mcp_servers, and an explicit list binds a proxy_admin key too. The access group mcp-dns grants the route to the dns-agent-triage key only.

Claude Code registers a client through Keycloak DCR (dynamic client registration). Keycloak creates that anonymous client with full scope turned off. A role then reaches the token only through a role scope mapping of a client scope. The client scope mcp-dns maps the dns-control role dns-reader and emits preferred_username (tf/keycloak/mcp_dns.tf). The person must be in the Keycloak group dns-readers.

The two ClickHouse routes present the same upstream credential. Both MCP instances validate one token, so nothing about the secret separates them. The only thing that does is which server your key is scoped to. A key scoped to clickhouse_ro sees none of the write instance’s tools.

The four in-cluster routes reach Kubernetes Services rather than internet vendors. That is invisible from the client side. A slower first response from firewalla_ro is the appliance being busy, not a gateway fault.

unifi_ro registers its tools lazily. The server lists seven tools at initialize: five meta-tools, which start with unifi_ro-unifi_, and two read tools. Find a tool with unifi_ro-unifi_tool_index, and run it with unifi_ro-unifi_execute and the tool name. Do not call unifi_ro-unifi_load_tools. It loads a tool into the server and announces it with a tools/list_changed notification, which LiteLLM does not relay, so the loaded tool never appears in the client. Every create, update and delete tool answers success: false with is disabled by policy, because the deployment keeps every policy gate off.

The client header is identical on all nine key-scoped routes, including deepwiki and unifi_ro. deepwiki needs no upstream credential, but every route on llm.fzymgc.house still requires an admitted virtual key, and a key not scoped to deepwiki is refused it like any other.

Mint the virtual key scoped to the servers this client needs. See Getting a client virtual key for the /key/generate body and what each field does.

Claude Desktop has no field for a static header on remote connectors, so the mcp-remote stdio bridge injects it. The value carries the Bearer scheme prefix and therefore contains a space, so keep the value in an environment variable and write --header with no space after the colon. Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
"mcpServers": {
"context7": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://llm.fzymgc.house/context7/mcp",
"--header", "x-litellm-api-key:${LITELLM_MCP_KEY}"
],
"env": {
"LITELLM_MCP_KEY": "Bearer <your-virtual-key>"
}
},
"exa": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://llm.fzymgc.house/exa/mcp",
"--header", "x-litellm-api-key:${LITELLM_MCP_KEY}"
],
"env": {
"LITELLM_MCP_KEY": "Bearer <your-virtual-key>"
}
}
}
}

Repeat the exa block for firecrawl, fal, deepwiki, firewalla_ro, clickhouse_ro and unifi_ro, and swap the server name in both the key and the /<server>/mcp path. The same virtual key works for every server it is scoped to. env is per entry in this file, so the value is repeated.

Add a clickhouse_rw block only if this client must write to the telemetry store, and only with a key scoped to clickhouse_rw. Without that scope, LiteLLM refuses the entry at initialize with HTTP 403.

The identity routes (engram, kubernetes, dns) are not configured here. A native Claude Desktop connector runs the OAuth flow but cannot send the x-litellm-api-key admission header beside it. The bridge sends a static header but has no OAuth flow. Neither produces the two-header shape these routes require. Claude Code is the client for those three routes.

Claude Desktop’s argument parser mangles header values that contain spaces, such as Bearer <key>. Put the value in an environment variable and write the --header flag with no space after the colon.

Fully quit Claude Desktop (⌘Q) and relaunch. It reads the configuration only at startup.

Smoke-test a key-scoped route with the virtual key alone. The path shape, the missing trailing slash and the Bearer prefix inside x-litellm-api-key are the three things that are easy to get wrong, and each one surfaces as a 401 or a 307 rather than a useful message:

Terminal window
curl -sS -X POST https://llm.fzymgc.house/exa/mcp \
-H "x-litellm-api-key: Bearer <your-virtual-key>" \
-H 'content-type: application/json' \
-H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","method":"initialize","id":1,
"params":{"protocolVersion":"2025-06-18","capabilities":{},
"clientInfo":{"name":"curl","version":"0"}}}' -D-

Run initialize first, because LiteLLM issues an mcp-session-id response header that tools/list needs. LiteLLM refuses a key with no grant for the route at initialize, with HTTP 403. See Getting a client virtual key for the body of the refusal.

An identity route, unauthenticated, must answer the discovery challenge:

Terminal window
curl -s -D - -o /dev/null -X POST https://llm.fzymgc.house/kubernetes/mcp \
-H 'content-type: application/json' -d '{}' | rg -i '^HTTP/|^www-authenticate'
# HTTP/2 401
# www-authenticate: Bearer resource_metadata="https://llm.fzymgc.house/.well-known/oauth-protected-resource/kubernetes/mcp"

The connected state from the client side:

Terminal window
claude mcp get kubernetes # shows connected state + tool count after OAuth

Which routes the proxy serves, independent of any client:

Terminal window
curl -sS https://llm.fzymgc.house/v1/mcp/server \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -r '.[].server_name'

A client-side problem is fixed on the client. There is no second host to repoint at. A wrong header, a wrong path spelling or a key not scoped to the route is a client configuration fix, diagnosed under Troubleshooting.

A LiteLLM-side regression on one route is a GitOps revert of that route’s commit. A route lands as one change: its mcp_servers entry in argocd/app-configs/litellm-chart/values.yaml, its extraEnvVars entry in the same file, and its key in argocd/app-configs/litellm/secrets.yaml. Reverting the commit removes all three together. Nothing on the client changes.

If every MCP behind llm.fzymgc.house fails at Claude Code startup while public MCPs connect, the fault is on the Mac’s LAN side, not in the gateway. On a Mac that is dual-homed onto the Main network, a rejected on-link route or a missing Local Network permission for the responsible app blocks the connection before any header is sent. Read the route first, then the permission:

Terminal window
route -n get llm.fzymgc.house | rg 'gateway|flags' # a "!" flag marks a reject route

Grant Local Network to the app that owns the connection in System Settings, Privacy & Security, Local Network, then retry. Only after both read clean is the gateway the suspect.

LiteLLM: the upstream returns 401 and it looks like a bad secret

Section titled “LiteLLM: the upstream returns 401 and it looks like a bad secret”

Symptom: initialize succeeds, tools/list returns tools, and tools/call comes back as an HTTP 200 that carries a JSON-RPC error with the upstream’s own “unauthorized” or “invalid token” wording. Nothing in LiteLLM’s own response is a 401, and rotating the upstream secret does not help.

Cause: a wrong header name, not a wrong secret. Two shapes produce this, and both survive a credential rotation:

  1. Client side, on engram, kubernetes or dns. The virtual key was placed in Authorization instead of x-litellm-api-key. Those three routes relay Authorization to the upstream verbatim, so the upstream is handed a LiteLLM virtual key and rejects it. Fix: move the virtual key into x-litellm-api-key: Bearer <key> and reserve Authorization for the Keycloak token. The key reached the upstream, so rotate it.
  2. Configuration side, on any credentialed route. The Vault property was stored with an Authorization: Bearer … prefix already baked in. LiteLLM’s auth_type: bearer_token composes the header itself, so a pre-prefixed value goes out as Bearer Bearer <token>. The upstream sees a malformed credential and returns 401. Fix: store the bare token.

Distinguish them in one step. If you can reproduce it with curl that sends only x-litellm-api-key, it is shape 2, a configuration fault. If sending the headers correctly makes it go away, it was shape 1.

Symptom: the client does not connect. initialize returns HTTP 403 with The key is not allowed to access the requested MCP servers: <server>.

Cause: the key has no grant for that server. object_permission.mcp_servers on the virtual key does not name the route, and no access group that holds the key lists the server id of the route. The upstream never saw the request.

Fix: read the key’s scope back and compare it to the route name, in the underscore spelling:

Terminal window
curl -sS "https://llm.fzymgc.house/key/info" \
-H "Authorization: Bearer <your-virtual-key>" | jq '.info.object_permission'

Authenticate as the key. Never put it in the query string. With no key parameter, /key/info reports the key in the Authorization header. ?key=<value> lands the credential in LiteLLM’s access log, in Traefik’s log on the way through the public ingress, in the url.full attribute of the OTel span, and in /proc/<pid>/cmdline for the life of the curl. See Verifying a minted key for the same rule with its upstream reference. A key restricted to llm_api_routes cannot call /key/info at all and returns 401. Use the /key/list and /budget/list pair described there, and rotate any key you have already exposed in a URL.

Then mint or update a key that names that server. GET /key/info reads the scope back as internal ids rather than names. That is canonical storage, not drift.

LiteLLM: something that used to arrive stopped arriving, with no error

Section titled “LiteLLM: something that used to arrive stopped arriving, with no error”

Symptom: tools/list and tools/call still work. What stopped is anything that needs session continuity, such as a server-initiated notification or a sampling or elicitation round trip. The symptom of a lost session is the missing notification.

Cause: MCP session state is process-local. A proxy restart, or a request that lands on the other replica, drops it, and LiteLLM downgrades to stateless service rather than erroring.

Do not debug this from the client. The behaviour, what does and does not survive, and the two conditions that do produce errors are documented once, in LiteLLM → MCP session semantics across restarts and replicas.

LiteLLM: an identity route’s upstream is down and nothing says so

Section titled “LiteLLM: an identity route’s upstream is down and nothing says so”

Symptom: engram, kubernetes or dns fails at tools/call with no proxy-side warning. No failing health check, no route marked unhealthy, no metric. The first signal is a client’s failed call.

Cause: the three identity routes carry auth_type: none and an Authorization entry in extra_headers. LiteLLM reads that shape as per-user auth and skips its health probe, because it has no credential to probe the upstream on its own behalf. GET /v1/mcp/server/health reports these routes as unknown. This is correct.

If one of these routes reads unhealthy, make sure that its entry in argocd/app-configs/litellm-chart/values.yaml still sets auth_type: none. LiteLLM skips the probe only for none. If auth_type is absent, the probe sends no credential and gets 401.

Do not give those routes a credential for the probe. LiteLLM applies static_headers after the relayed header, so a static Authorization ends the per-user identity relay that the routes exist to provide.

Fix: look at the upstream Deployment directly, memory-mcp in agent-memory, kubernetes-mcp in kubernetes-mcp or dns-control in dns-control. LiteLLM probes the nine other routes normally.

engram / kubernetes: client never connects, or no OAuth prompt

Section titled “engram / kubernetes: client never connects, or no OAuth prompt”

The client only starts its OAuth flow if discovery works. Read the relayed document first:

Terminal window
curl -sS -o /dev/null -w '%{http_code}\n' \
https://llm.fzymgc.house/.well-known/oauth-protected-resource/engram/mcp
  • 502: the upstream is not serving its own protected-resource document, or LiteLLM’s negative cache holds an earlier failure. For kubernetes, make sure that the authorization_url and server_url block in the kubernetes-mcp ConfigMap is what the pod runs. A ConfigMap-only change does not roll the pod, so run kubectl -n kubernetes-mcp rollout restart deploy/kubernetes-mcp. For engram, make sure that the memory-mcp pod is up. After the upstream serves the document, restart LiteLLM to clear the negative cache (kubectl rollout restart deployment/litellm -n litellm).
  • 404: the URL is wrong. The document lives at /.well-known/oauth-protected-resource/<server>/mcp on the host root. The server root and the /mcp/<server> order both miss.
  • 200 but the client still shows no prompt: the server URL in the client entry is wrong (a bare /mcp returns 404 route not found rather than triggering the flow), or a static headers.Authorization is configured on the entry, which disables Claude Code’s OAuth flow for that server.
  • “does not support dynamic client registration”: Keycloak refused the registration. The realm’s Trusted-Hosts policy admits loopback redirect URIs only (argocd/app-configs/keycloak/dcr-trusted-hosts-job.yaml). A client that registers with any other redirect URI is rejected. Use the --client-id mcp-public fallback above.
  • Registration refused with 403 and “It’s allowed to have max 500 clients per realm”: the realm’s anonymous Max Clients Limit policy counts every client in the realm, not only registrations, and dcr-trusted-hosts-job.yaml sets it to 500. The nightly keycloak-dcr-reaper CronJob (argocd/app-configs/keycloak/dcr-client-reaper-cronjob.yaml) keeps headroom by deleting public, UUID-named clients that have no active or offline session. Make sure that it runs and succeeds with kubectl -n keycloak get cronjob keycloak-dcr-reaper and the logs of its most recent Job.

engram / kubernetes: re-authentication stops at Keycloak “Client not found”

Section titled “engram / kubernetes: re-authentication stops at Keycloak “Client not found””

The reaper deleted the client this entry registered, because it no longer held an active or offline session. Claude Code re-registers by itself when the token endpoint rejects the client id. When it holds the registration without tokens, it replays the deleted client id at the authorize endpoint instead. The rejection happens in the browser and never reaches Claude Code (anthropics/claude-code#84614).

Fix: /mcp, the server, Clear authentication, then Authenticate. Or run claude mcp remove <name> and re-add it with the same claude mcp add command from The identity routes.

dns: the server rejected the new credentials on reconnect

Section titled “dns: the server rejected the new credentials on reconnect”

Symptom: Claude Code reports that it got new credentials, but dns rejected them on reconnect.

Cause: dns-control refused the Keycloak token. dns-control answers 401 with error="invalid_token" in WWW-Authenticate. A refusal by LiteLLM answers 403 instead, as in LiteLLM: initialize answers 403. dns-control refuses a token without dns-reader in resource_access.dns-control.roles, and a token for another audience.

Fix:

  1. Decode the claims of the token without printing the token. With the access token in TOKEN, run the command below. Make sure that resource_access.dns-control.roles holds dns-reader and that aud holds https://llm.fzymgc.house/dns/mcp.

    Terminal window
    printf '%s' "$TOKEN" | cut -d. -f2 | tr '_-' '/+' \
    | jq -R '@base64d | fromjson | {aud, azp, preferred_username, resource_access}'
  2. Re-authenticate: /mcp, dns, Clear authentication, then Authenticate.

  3. If the role is still absent, run claude mcp remove dns --scope user. Add the entry again with the command in The identity routes, and sign in. The sign-in registers a new DCR client.

If the role is absent after a new registration, make sure that the person is in the Keycloak group dns-readers.

Key-scoped route: 401/403 from the gateway

Section titled “Key-scoped route: 401/403 from the gateway”

The virtual key is wrong, not sent, sent in the wrong header, or sent without the Bearer scheme prefix inside x-litellm-api-key. Re-run the Verify curl. A 200 with an mcp-session-id confirms the key and the endpoint.

Claude Desktop server fails to authenticate

Section titled “Claude Desktop server fails to authenticate”

Almost always the header-quoting problem. Make sure that the --header argument is x-litellm-api-key:${LITELLM_MCP_KEY} with no space after the colon, and that env.LITELLM_MCP_KEY holds Bearer <key>. Quit and relaunch after any edit.

  • LiteLLM operations: the operator runbook for key minting, restart and replica semantics, and database-outage behaviour
  • AI Tooling: general MCP server configuration
  • Services catalog: LiteLLM architecture and the route inventory
  • Secrets & Vault Paths: Vault path conventions
  • ADR Keycloak DCR deny removed: why a client can register directly against Keycloak
  • LiteLLM MCP route declarations: argocd/app-configs/litellm-chart/values.yaml (proxy_config.mcp_servers, the estate’s only sanctioned MCP registration point)
  • kubernetes-mcp discovery configuration: argocd/app-configs/kubernetes-mcp/config.yaml
  • dns-control → The MCP path: the six dns- tools, the callers and the static key procedure, in docs/operations/dns-control.md