LiteLLM
Operational guide for the LiteLLM proxy at llm.fzymgc.house: what exists, where each part is
defined, and the procedures that touch its credentials and keys.
What this is
Section titled “What this is”LiteLLM is the LLM data plane. It is an OpenAI-compatible proxy in the litellm namespace, fronted
by a Traefik IngressRoute that publishes API paths only. It keeps virtual keys, spend and budget
state in a CloudNativePG database (main, in the postgres namespace) and shares its auth cache
through a dedicated Valkey cluster in its own namespace.
People and off-cluster clients use https://llm.fzymgc.house. In-cluster workloads (octopus,
karakeep, mealie and life-agent) call the Service directly. The litellm-proxy-ingress CiliumNetworkPolicy in
argocd/app-configs/litellm/networkpolicy.yaml must admit each such workload pod on port 4000. A
direct caller skips the route exclusions in the IngressRoute. The allowed_routes value on its
virtual key still limits it to the LLM routes.
| Property | Value |
|---|---|
| Public host | https://llm.fzymgc.house |
| In-cluster Service | http://litellm.litellm.svc.cluster.local:4000 |
| Namespace | litellm |
| Image and chart | argocd/app-configs/litellm-chart/values.yaml (image tag and digest) and kustomization.yaml (chart version) |
| Database | CNPG cluster main, database litellm, reached at main-rw.postgres.svc.cluster.local:5432 |
| Database credential | secret/fzymgc-house/cluster/postgres/users/main-litellm (Vault) |
| Master key | master_key on secret/fzymgc-house/cluster/litellm-keys/master (Vault) |
| Provider and MCP upstream credentials | secret/fzymgc-house/cluster/litellm (Vault) |
| Virtual key values | secret/fzymgc-house/cluster/litellm-keys/<consumer> (Vault), see Which tool owns each key |
| Virtual key scopes | tf/litellm/main.tf, applied by the HCP Terraform workspace main-cluster-litellm |
| Manifests | argocd/app-configs/litellm/ (secrets, ingress, network policy, Valkey) and argocd/app-configs/litellm-chart/ (the chart values) |
| Seed script | scripts/seed-litellm-vault.sh (operator-run) |
| Auth cache | the litellm-valkey ValkeyCluster in the litellm namespace, password on secret/fzymgc-house/cluster/litellm-valkey |
| Telemetry | OTLP to the ClickStack collector, service.name=litellm |
The lanes a client can name are the entries of proxy_config.model_list in values.yaml. The MCP
servers behind the proxy are the entries of proxy_config.mcp_servers in the same file. The
services catalog lists both sets, and the
MCP client runbook covers the client side of the MCP routes.
Database credential and the single-source contract
Section titled “Database credential and the single-source contract”The password lives in exactly one place: the password property at
secret/fzymgc-house/cluster/postgres/users/main-litellm in Vault. Everything else that carries it
is a derivation:
| Object | What it is |
|---|---|
postgres/main-litellm-credentials (Kubernetes Secret) |
CNPG’s own copy, materialized from the Vault property by External Secrets |
litellm/litellm-secrets (Kubernetes Secret) |
the proxy’s DATABASE_URL, composed from the same Vault property |
The connection string is composed in the cluster and exists nowhere else. The ExternalSecret at
argocd/app-configs/litellm/secrets.yaml reads the username and the password as two separate
properties and assembles DATABASE_URL in its spec.target.template.data. The assembled string is
never written to Vault, never committed to git, and never present on the operator’s workstation.
The escaping is correct because of the generator, not because of the escaper. The template pipes
the password through urlquery, which is the Go text/template query escaper. It renders a space
as a plus sign. Inside a URL userinfo component a plus is a literal plus, so a password that
contains a space produces a wrong credential with no parse error anywhere. The only symptom is an
authentication failure at the database.
scripts/seed-litellm-vault.sh generates from an alphabet that contains no space and no non-ASCII
character. Nothing checks the coupling between that alphabet and this escaping. Three constraints
keep it correct:
- The password comes from the generator. Do not type one by hand.
- Do not widen the alphabet without a check that every added character survives
urlqueryinside a URL userinfo component. - Keep
urlqueryon the password and not on the username in the DSN template.
Changing the database credential in the required order
Section titled “Changing the database credential in the required order”A change to the shape of this credential is two steps across a merge boundary. The Vault write is out of band from ArgoCD, so it cannot be folded into the same change as the manifest.
-
Merge the manifest change and let ArgoCD sync it, so nothing in the cluster still references the property that is removed.
-
Make sure that the rendered Secret is correct before you delete anything:
Terminal window kubectl --context fzymgc-house get externalsecret -n litellmlitellm-secretsmust readSecretSynced. Then make sure that the renderedDATABASE_URLstill authenticates. A proxy pod that serves requests and a migration Job that completed are the evidence. -
Only then remove the retired property. Write back a version that names only the properties that remain. Piping the current version through
jqcarries the live values across unchanged and keeps them out of argv and shell history:Terminal window vault kv get -format=json secret/fzymgc-house/cluster/litellm \| jq '.data.data | del(.retired_property)' \| vault kv put secret/fzymgc-house/cluster/litellm -vault kv putwrites a new version that contains exactly the keys given, so the write itself removes the retired property.vault kv patchmerges into the current version and leaves the property in place. It is the wrong verb here.
Do not run scripts/seed-litellm-vault.sh with no flag to do this. The bare path generates a fresh
CNPG password, a fresh master key, a fresh Valkey password and a fresh value for every workload
virtual key, and writes all of them. On a live estate that is a full rotation, not a cleanup. If
you do run the bare path, treat it as a full-estate rotation: restart the proxy, re-export
LITELLM_MASTER_KEY, and re-register every workload key.
What happens if you do it in the reverse order
Section titled “What happens if you do it in the reverse order”If you remove the Vault property while an ExternalSecret still references it, the Kubernetes Secret is not deleted. External Secrets deletes a target only when the provider returns no data at all. A missing property inside an existing secret is a fetch error, not an empty data map. So:
- The ExternalSecret goes to
SecretSyncedErrorand stops refreshing. - The last-synced Secret persists, so running pods keep working and there is no outage to notice.
- Changes to the master key or the provider keys stop propagating.
- Any event that recreates the Secret leaves it with no source at all.
The warning sign is kubectl get externalsecret -n litellm showing anything other than
SecretSynced.
Purging the history is a separate step
Section titled “Purging the history is a separate step”vault kv put does not erase history. A property removed by a new version remains readable in the
prior versions of that path. To purge it, destroy the old versions:
vault kv destroy -versions=<n>,<n> secret/fzymgc-house/cluster/litellmDo not run vault kv metadata delete on this path. It removes every version, the current one
included, so it destroys the live openrouter_api_key. The seed script takes that value from the
environment, and nothing in this repository can reproduce it. If you are not holding it, it has to
be re-issued at OpenRouter. If you do run vault kv metadata delete, re-seed at once. Until you do,
the ExternalSecret behind litellm/litellm-secrets has no source.
Rotating the database password
Section titled “Rotating the database password”The only part of this rotation that this repository owns is the urlquery escaping, and it holds
by construction as long as the password comes from the generator. The rest is CNPG applying a
changed passwordSecret, LiteLLM reconnecting, and virtual-key rows surviving the reconnect. Those
are third-party behaviour, and nothing in this repository asserts them.
Rotate when you have a reason: a compromise, a departure, or a policy clock.
-
Generate the new password and write it to the single source. The generator is the same one in
scripts/seed-litellm-vault.shunder “Generate the CNPG password”. Keep the two alphabets identical. Nothing checks that they agree. The block is unindented because thePYterminator must start at column zero:Terminal window NEW_PGPASS="$(python3 - <<'PY'import secrets, stringalphabet = string.ascii_letters + string.digitsspecials = "@:/?#[]!$&'()*+,;=%"pw = [secrets.choice(specials) for _ in range(3)] + [secrets.choice(alphabet) for _ in range(29)]secrets.SystemRandom().shuffle(pw)print("".join(pw))PY)"jq -n --arg username litellm --arg password "$NEW_PGPASS" \'{username: $username, password: $password}' \| vault kv put secret/fzymgc-house/cluster/postgres/users/main-litellm -unset NEW_PGPASSjqhands the value tovaulton stdin, so the password reaches neither argv nor shell history. The path carries exactlyusernameandpassword, and thejq -nreconstructs both. The recovery path from a failed rotation is another write with the previous password.vault kv geton the prior version supplies it as long as that version is not destroyed. -
CNPG applies the password to the role on its own.
litellmis a managed role inargocd/app-configs/cnpg/postgres-cluster.yamlwithpasswordSecret: main-litellm-credentials. That Secret is materialized from the same Vault property by themain-litellm-credentialsExternalSecret inargocd/app-configs/cnpg/users-litellm.yaml, whose target carries thecnpg.io/reload: "true"label. There is no manualALTER ROLE. -
Both ExternalSecrets refresh on their own clocks.
main-litellm-credentials(namespacepostgres) andlitellm-secrets(namespacelitellm) each carryrefreshInterval: 5m. Either order has a transient window in which the proxy holds one password and the role holds the other. Expect authentication failures in the proxy log inside it. Make sure that both readSecretSynced:Terminal window kubectl --context fzymgc-house get externalsecret -n postgres main-litellm-credentialskubectl --context fzymgc-house get externalsecret -n litellm litellm-secrets -
The proxy restart is automatic.
DATABASE_URLreaches the pod as asecretKeyRefenv var, which is read at container start, so the Deployment carriesreloader.stakater.com/auto: "true"invalues.yamland Stakater Reloader restarts it. The manual fallback:Terminal window kubectl --context fzymgc-house rollout restart deployment/litellm -n litellm -
Make sure that the proxy reconnected and that the database state survived.
GET /health/readinesson a pod must reportdb: connected. A bare 200 from/health/livelinessis not the reconnect, becauseallow_requests_on_db_unavailablekeeps completions flowing on cached auth (see When the database is unavailable). Then make sure that an existing workload key still authenticates, thatGET /key/inforeports the same models, aliases, limits and budgets, and that its spend total did not go backwards.
Make sure that the litellm-db-failure alert is not firing before you start. It is the signal if
the reconnect does not happen.
Minting and scoping virtual keys
Section titled “Minting and scoping virtual keys”Every workload that calls LiteLLM holds its own virtual key. This section is how one gets made, scoped, delivered and checked. Nine credentials exist: six in-cluster consumer workloads, one agent identity that reads its key from Vault at run time, and two credentials that the operator holds on a workstation.
Which tool owns each key
Section titled “Which tool owns each key”A virtual key has two halves: the secret value and the scope. The scope is the allow-list, the
aliases, the limits, the budgets and the metadata. Vault holds every value, and
scripts/seed-litellm-vault.sh generates every value. The scope has two owners.
| Consumer name | key_alias |
Vault path and property | Scope owner |
|---|---|---|---|
octopus |
octopus |
cluster/litellm-keys/octopus api_key |
Terraform, tf/litellm/main.tf |
karakeep |
karakeep |
cluster/litellm-keys/karakeep api_key |
Terraform, tf/litellm/main.tf |
mealie |
mealie |
cluster/litellm-keys/mealie api_key |
Terraform, tf/litellm/main.tf |
agent-memory |
engram |
cluster/litellm-keys/agent-memory api_key |
Terraform, tf/litellm/main.tf |
claude-code |
claude-code |
cluster/litellm-keys/claude-code api_key |
Terraform, tf/litellm/main.tf |
openrouter-passthrough |
openrouter-passthrough |
cluster/litellm-keys/openrouter-passthrough api_key |
Terraform, tf/litellm/main.tf |
dns-agent-triage |
dns-agent-triage |
cluster/litellm-keys/dns-agent-triage api_key |
Terraform, tf/litellm/main.tf |
life-agent |
life-agent |
cluster/litellm-keys/life-agent api_key |
Terraform, tf/litellm/main.tf |
<consumer>-mcp |
<consumer>-mcp |
none | An operator, by hand, per MCP client keys |
All Vault paths in the table are below secret/fzymgc-house/. The master key is master_key on
cluster/litellm-keys/master. CONSUMER_TABLE in the seed script is the single source of each
consumer’s path and property pair. Keep each vault_path and vault_property in
tf/litellm/main.tf equal to its CONSUMER_TABLE row. Nothing checks it.
The dedicated key paths
Section titled “The dedicated key paths”Each Terraform-owned value lives alone, as the property api_key, on its own path under
secret/fzymgc-house/cluster/litellm-keys/. The master key lives alone on
cluster/litellm-keys/master. A path under that prefix holds exactly one credential.
The layout narrows the Vault access of the Terraform run. KV has no per-property access control, so
a grant on a path is a grant on every property of that path. The run role tfc-litellm reads
secret/data/fzymgc-house/cluster/litellm-keys/* and the OpenRouter management key at
infrastructure/openrouter/terraform. It writes one path, cluster/litellm/openrouter-zdr
(tf/vault/policy-litellm.tf). It cannot read the other credentials on a consumer’s path, or the
provider keys on the bare path cluster/litellm.
The HCP Terraform workspace main-cluster-litellm applies tf/litellm. It reads each value from
Vault as an ephemeral value and sends it to LiteLLM once, as the write-only argument key_wo. The
value never enters the plan or the state. The state holds only the SHA-256 hash of the value, which
is also the token that LiteLLM stores. tf/litellm/imports.tf adopts a key that exists in LiteLLM
under a configured alias, and Terraform creates a key that does not exist from the value in Vault.
The MCP client keys stay out of Terraform because the provider cannot express their scope. Every
MCP client key carries object_permission.mcp_servers, and the ncecere/litellm provider has no
argument for it. A Terraform-owned key gets its MCP routes from a litellm_unified_access_group in
tf/litellm/access_groups.tf, and the dns-agent-triage key is the one case today.
The Terraform-managed keys differ from a hand-minted key in two ways:
- The provider cannot send
key_type. It sendsallowed_routes: ["llm_api_routes"], which is the restriction thatkey_type: "llm_api"sets. A key that Terraform creates readskey_type: "default", and it is restricted in the same way. - An in-place apply sends every argument the key has in its state. The
metadataobject replaces the live one as a whole. The module ignores changes tobudget_durationandduration, so an imported key never receives them in an update. A key that Terraform created holds both in its state. Each in-place apply on that key restarts its 30-day budget window, and for a workstation key it also restarts the 90-day expiry.
The HCP agent pods must reach the proxy Service at http://litellm.litellm.svc.cluster.local:4000.
The litellm-proxy-ingress policy admits the namespace hcp-terraform. If it does not, a plan
stops at the litellm_keys data source with a timeout.
Removing the old properties later
Section titled “Removing the old properties later”Each moved credential still has a copy at its original location, kept as a rollback:
| Credential | Original location | Current location |
|---|---|---|
| master key | cluster/litellm master_key |
cluster/litellm-keys/master master_key |
octopus |
cluster/octopus litellm_api_key |
cluster/litellm-keys/octopus api_key |
karakeep |
cluster/karakeep litellm_api_key |
cluster/litellm-keys/karakeep api_key |
mealie |
cluster/mealie litellm_api_key |
cluster/litellm-keys/mealie api_key |
agent-memory |
cluster/agent-memory litellm_api_key |
cluster/litellm-keys/agent-memory api_key |
claude-code |
cluster/litellm claude_code_api_key |
cluster/litellm-keys/claude-code api_key |
openrouter-passthrough |
cluster/litellm openrouter_passthrough_api_key |
cluster/litellm-keys/openrouter-passthrough api_key |
To remove the copies, remove only the properties that moved, and leave every other property on those paths in place:
-
Make sure that every ExternalSecret reads
SecretSynced, and that no manifest, script or document still names an original location. -
For each consumer path, write a new version without
litellm_api_key. Build the body from the current version, so the other values stay out of argv and shell history:Terminal window vault kv get -format=json secret/fzymgc-house/cluster/octopus \| jq '.data.data | del(.litellm_api_key)' \| vault kv put secret/fzymgc-house/cluster/octopus -Do the same for
cluster/litellm, and deletemaster_key,claude_code_api_keyandopenrouter_passthrough_api_keyin one write. Read the result back withvault kv get -format=json <path> | jq '.data.data | keys', which prints names only. -
The removed values are still readable in the prior versions of each path. To purge them, use the version-scoped
vault kv destroy -versions=<list> <path>, and name only versions older than the current one. Do not usevault kv metadata delete. See Purging the history is a separate step.
Seeding the estate is one step
Section titled “Seeding the estate is one step”scripts/seed-litellm-vault.sh --all-consumersThis command owns seeding. It writes each of the nine values into Vault. Every row is
Terraform-owned, so it registers nothing. The next main-cluster-litellm apply registers them. The command touches no core credential. It
is safe to re-run: a consumer that is seeded and authenticating is skipped, and a scripted consumer
whose value was seeded but never registered is repaired without a new Vault version. A
Terraform-owned consumer whose value is in Vault but not live reports the verdict terraform. Read
completion off the per-consumer result table, not off the exit code.
Two preconditions, both checkable before you run it:
- The lanes are live.
GET /v1/modelsmust list every lane inmodel_list. A key registered against a lane that does not exist is a 403 at the consumer’s first call. - The proxy answers.
GET /health/livelinessreturns 200. Do not read a 401 fromGET /healthas a broken proxy. That route is authenticated, and with the master key attached it answers 200. The command performs this check itself and exits without a write if the proxy does not answer.
The bare invocation is not the seeding step. scripts/seed-litellm-vault.sh with no flag rotates
the CNPG password, the master key and the Valkey password alongside every workload virtual key. On
a live estate that is an outage. --consumer <name> seeds one value without registering it, for a
single repair.
Where each half of a key lives
Section titled “Where each half of a key lives”Vault owns the value. LiteLLM’s Postgres owns the scope. key is a first-class field on
/key/generate: it must start with sk- and be at least 16 characters, or the call returns 400.
| Fact about a key | Authoritative store | Changed by |
|---|---|---|
| the secret value itself | Vault, on the consumer’s own path | scripts/seed-litellm-vault.sh --all-consumers (all nine) or --consumer <name> (one). On a live key, follow Rotating a virtual key. Not /key/regenerate, which is Enterprise-gated on this estate and returns HTTP 500 |
| models allow-list, aliases, metadata | LiteLLM Postgres | For the nine Terraform-owned keys, a merged change to tf/litellm/main.tf. For the MCP client keys, /key/generate and /key/update |
| budgets, rate limits | LiteLLM Postgres | The same owner as the allow-list |
| spend | LiteLLM Postgres | normal traffic |
| delivery into the pod | Vault to ExternalSecret to Kubernetes Secret | that consumer’s ExternalSecret |
After Terraform owns a key, change its scope only through a merged change to tf/litellm/main.tf.
Do not change it with /key/update, the Admin UI or the seed script. The next apply sends the
configured scope again and reverts the change. A change to metadata outside Terraform is worse:
the plan does not show it, and the next in-place update of that key deletes it, because the
provider sends the configured metadata object as a whole.
Seeding the value first keeps the two stores from drifting. The value in Vault is the value, and registration only attaches scope to it.
The procedure for one credential
Section titled “The procedure for one credential”This is the mechanism that --all-consumers performs for the scripted key. Follow it by hand when
you repair exactly one scripted credential.
For a Terraform-owned key, the procedure is shorter. If the key does not exist in LiteLLM, run
step 1, then let the next main-cluster-litellm apply create the key from the value in Vault. If
the key exists, follow Rotating a virtual key. Do not register a
Terraform-owned value with /key/generate. The next plan imports that key and updates it to the
configured scope, so the scope you sent lasts only until that apply.
-
Seed the value in Vault:
Terminal window scripts/seed-litellm-vault.sh --consumer <name>--consumerwrites exactly one property withvault kv patch -method=rw, touches no core credential and no other consumer, and refuses to overwrite a property that exists unless you pass--rotate. -
Read the seeded value into a shell variable, never onto the screen:
Terminal window VAULT_ADDR=https://vault.fzymgc.houseSEEDED_KEY="$(vault kv get -field=<property> secret/fzymgc-house/<path>)" # both from the row -
Register it with
/key/generate, with the seeded value askeyand the scope from the JSON body of the row. -
Verify, per Verifying a minted key.
-
Only then edit the consumer’s manifests, per the next section.
The Vault property must exist before any manifest references it. An ExternalSecret that points at
a property that is not there goes SecretSyncedError and stops refreshing, while the running pods
keep working on their old Secret. That is a stale-configuration failure with no outage to notice.
Staging a consumer’s credential
Section titled “Staging a consumer’s credential”- Seed the Vault property (
--consumer <name>) and register the key. For a Terraform-owned key, the nextmain-cluster-litellmapply registers it. - Add the new Secret key alongside the existing one in that consumer’s ExternalSecret. Do not replace it.
- Wait one refresh interval (
refreshInterval: 5m) and make sure that the live Secret carries both keys. - Only then flip the workload’s
secretKeyRef.keyand its base URL together, in one change.
This prevents a workload from restarting onto a credential that the plane it still calls does not accept. The old Secret key stays populated for the whole bake, so a rollback is a key-name flip against a Secret that already holds both values.
The allow-list is checked before the key’s aliases resolve
Section titled “The allow-list is checked before the key’s aliases resolve”Every key’s models list must contain both the string the client sends and the lane name. LiteLLM
runs the access check against the model taken from the request, and the per-key alias rewrite
happens later. Per-key aliases are not consulted by the access check at all.
If you list only the lane name, the result is a 403 that names a model the client never sent. That
symptom reads like a routing bug and sends you to model_list, which is not where the fault is.
Removing the aliases of a Terraform-owned key
Section titled “Removing the aliases of a Terraform-owned key”Provider ncecere/litellm 2.1.0 sends aliases in /key/update only when the map has entries.
Thus a Terraform apply cannot remove the aliases of a key. With aliases = null, the apply works,
and LiteLLM keeps the aliases. With aliases = {}, LiteLLM keeps the aliases, and the apply fails
with an inconsistent result. LiteLLM replaces aliases on /key/update. It does not merge them as
it merges metadata. To remove the aliases of a key:
- Set
aliases = nullfor the key intf/litellm/main.tf, and remove the alias names from itsmodelslist. Merge the change, and let themain-cluster-litellmworkspace apply it. - In a
litellmpod, use theLITELLM_MASTER_KEYof the pod to find thetokenof the key withGET /key/list?key_alias=<alias>&return_full_object=trueonhttp://127.0.0.1:4000. Do not print the master key. - In the same pod, send
POST /key/updatewith the body{"key": "<token>", "aliases": {}}. - Start a refresh-only run of
main-cluster-litellm. The workspace applies it automatically, and the state then hasaliases = {}for the key. - Start a plan-only run, and make sure that it shows no change and no drift for the key.
Budget doctrine
Section titled “Budget doctrine”soft_budget is the signal and max_budget is a blast-radius stop, not a cost control. Spend is
flushed to the database on a delay of several seconds, so enforcement is a bound with overshoot,
not a ceiling. A max_budget close to expected spend is a fuzzy limit and, on a bursty consumer,
an availability risk. Put soft_budget near expected spend and max_budget well above it.
Per-model budgets inside a key (model_max_budget) are Enterprise-gated, which is why budgets
land per key rather than per lane.
Embedding lanes record a cost when they declare input_cost_per_token in model_list. A failed
call records $0. A lane that is absent from the LiteLLM cost map and declares no
input_cost_per_token records $0.
Spend telemetry comes from spans, not from the cost metric. LiteLLM exports gen_ai.usage.cost only
as a cumulative histogram, and a sum over a histogram’s Sum column adds running totals together.
So the spend tiles and the spend alert on the LiteLLM — LLM & key usage dashboard read the Traces
source. Each billed request writes its cost once, as SpanAttributes['litellm.cost.total'] on one
CLIENT span named <operation> <lane>, for example chat engram-summarize. No other span in the
trace repeats it, so a sum over spans does not double count.
| Tile | Select | Where | Group by |
|---|---|---|---|
litellm-key-spend |
sum(toFloat64OrZero(SpanAttributes['litellm.cost.total'])) |
ServiceName = 'litellm' AND mapContains(SpanAttributes, 'litellm.cost.total') |
SpanAttributes['litellm.metadata.user_api_key_alias'] |
litellm-lane-cost |
the same | the same | SpanAttributes['gen_ai.request.model'], which is the lane name on a span |
The alert LiteLLM key spend runaway in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml
fires when one key alias reaches 3 USD in one hour. This query gives the hourly figures:
SELECT SpanAttributes['litellm.metadata.user_api_key_alias'] AS alias, toStartOfHour(Timestamp) AS hour, sum(toFloat64OrZero(SpanAttributes['litellm.cost.total'])) AS usdFROM default.otel_tracesWHERE Timestamp > now() - INTERVAL 14 DAY AND ServiceName = 'litellm' AND mapContains(SpanAttributes, 'litellm.cost.total')GROUP BY alias, hour ORDER BY usd DESC LIMIT 20For a Terraform-owned key, change soft_budget and max_budget in tf/litellm/main.tf. The
provider sends both in one /key/update and reads soft_budget back from litellm_budget_table.
If LiteLLM discards the value, the apply fails with an inconsistent-result error, so the failure is
not silent. In that case, move the budget row by id. /budget/ is not published by the
IngressRoute, so this is an in-cluster hop:
kubectl -n litellm port-forward svc/litellm 18400:4000 &curl -sS -X POST http://127.0.0.1:18400/budget/update \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \ -d '{"budget_id":"<.info.budget_id from /key/info>","soft_budget":50.0}'For a scripted key, POST /key/update moves max_budget, which lives on the key row. Read
soft_budget back from /key/info at .info.litellm_budget_table.soft_budget, never from the
update response. If the budget row did not move, use /budget/update as above.
Fields every mint body carries
Section titled “Fields every mint body carries”The nine Terraform-owned keys carry these fields as arguments in tf/litellm/main.tf, with the
exceptions this table names. key becomes the write-only key_wo.
| Field | Why it is not optional |
|---|---|
key |
the value seeded in Vault. Omit it and LiteLLM mints its own, and the two stores diverge at once |
key_alias |
the only human-readable per-key label on the spend metrics. Without it every dashboard groups by an opaque hash |
key_type: "llm_api" |
a data-plane key. /key/, /team/ and /user/ are published at the ingress, so a leaked workload credential without this can mint more keys. A Terraform-owned key sends allowed_routes: ["llm_api_routes"] instead, which is the restriction this field sets |
models |
both strings per lane, per the section above |
aliases |
maps what the client sends to its lane, so most consumers need only a base-URL change |
rpm_limit, tpm_limit |
a runaway loop is bounded in requests before it is bounded in dollars |
soft_budget, max_budget, budget_duration |
per the doctrine above |
duration |
TTL. null on every workload key |
Workload keys carry no TTL. A duration on a key held by a pod is a scheduled outage: nothing in
this estate re-mints a key on expiry, so the workload starts failing authentication at a time
nobody chose. The two workstation credentials carry one, because a human is present to re-mint
them and an abandoned workstation credential must stop working.
octopus
Section titled “octopus”The scope is the octopus entry in tf/litellm/main.tf. The octopus-app ExternalSecret in
argocd/app-configs/octopus/secrets.yaml reads the value.
octopus’s soft_budget and max_budget pair is sized from its own demand and departs from the
ratio the other rows carry. Do not normalize the pair onto the other rows.
Most of octopus’s model strings are not in its Deployment. Its Deployment carries one model env
var, OCTOPUS_UTILITY_MODEL, for the utility lane. The review and embedding models are rows in the
available_models table of its Postgres database:
P=$(kubectl --context fzymgc-house -n postgres get cluster main -o jsonpath='{.status.currentPrimary}')kubectl --context fzymgc-house exec -n postgres "$P" -c postgres -- \ psql -d octopus -c 'SELECT "modelId", category, "isActive" FROM available_models;'ollama/bge-m3 is the platform-default embedding row. LiteLLM matches the client-sent string, so
every active row in that table must appear in models. Re-derive the list from the active rows of
that table, never from the Deployment.
The key holds these names and aliases:
models = [ "ollama/bge-m3", "bge-m3", "octopus-embed", "text-embedding-3-large", "octopus-review", "octopus-utility", local.decisions_model,]aliases = { "ollama/bge-m3" = "octopus-embed" "bge-m3" = "octopus-embed" "text-embedding-3-large" = "octopus-embed"}LiteLLM refuses a request of the key for any other model with HTTP 403 and the error
key_model_access_denied.
The chat rows octopus-review and octopus-utility are the names of LiteLLM lanes in
argocd/app-configs/litellm-chart/values.yaml. When octopus sends one of these names, LiteLLM
routes the call to the lane of the same name, so the key holds no alias for them. The lane sets
the model, the reasoning effort and the provider rule.
The lane octopus-review also names providers in provider.order. OpenRouter tries those
providers first, in that order. It uses the other ZDR providers only when the named providers
fail. Without the order, OpenRouter prefers the provider with the lowest price, and the cheapest
ZDR providers of the review model are slow. Octopus drops a review call after 300 seconds.
Each provider in the order must meet two conditions:
-
It has a ZDR endpoint for the model of the lane. This command lists those providers. Set the model id to the model of the lane:
Terminal window curl -s https://openrouter.ai/api/v1/endpoints/zdr \| jq -r '.data[] | select(.model_id == "deepseek/deepseek-v4.1-flash") | .provider_name' \| sort -u -
It produces the output budget of
OCTOPUS_REVIEW_MAX_TOKENSin under 300 seconds. With a margin of 30 seconds, a budget of 49152 needs 183 output tokens per second on a large review.
To see which provider serves the lane, read the provider field of a reply:
export VAULT_ADDR=https://vault.fzymgc.houseKEY="$(vault kv get -field=api_key secret/fzymgc-house/cluster/litellm-keys/octopus)"curl -s https://llm.fzymgc.house/v1/chat/completions \ -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ -d '{"model":"octopus-review","max_completion_tokens":2048, "messages":[{"role":"user","content":"Reply with the word OK."}]}' \ | jq '{provider, finish: .choices[0].finish_reason}'The reply must name the first provider of the order. If it names a provider outside the order, the named providers failed for that call. If that repeats, review calls run at the speed of the fallback provider, and the long ones time out.
text-embedding-3-large is not in that table either, and octopus sends it. It is octopus’s
hard-coded OpenAI default on a code path that never consults available_models. Its alias must
point at octopus-embed: that lane is bge-m3 at 1024 dimensions, which matches
EMBEDDING_DIM=1024 and the code_chunks collection. A real 3072-dimension model would write
vectors the collection cannot hold. Make sure with jq '.data[0].embedding | length', which must
return 1024.
A consumer’s model strings can come from a configuration table and from hard-coded defaults. Enumerate from both.
mealie
Section titled “mealie”The scope is the mealie entry in tf/litellm/main.tf. The mealie-app-secrets ExternalSecret in
argocd/app-configs/mealie/secrets.yaml reads the value. mealie calls the LiteLLM Service directly.
mealie and karakeep have no dedicated model_list entry. The openrouter/* wildcard serves them.
Their keys name the resolved model, not the wildcard. Listing openrouter/* in models would
grant the whole OpenRouter catalogue. Naming the resolved model keeps the grant to one model while
the access check still passes on both the client-sent string and the resolved one.
karakeep
Section titled “karakeep”The scope is the karakeep entry in tf/litellm/main.tf. The karakeep-secrets ExternalSecret
in argocd/app-configs/karakeep/secrets.yaml reads the value. karakeep calls the LiteLLM Service
directly, and the same wildcard reasoning as mealie applies.
karakeep sends two model strings, so it needs two pairs. INFERENCE_IMAGE_MODEL and
INFERENCE_TEXT_MODEL in argocd/app-configs/karakeep/deployment.yaml name them, and both
auto-tagging and auto-summarization are on. To check the text pair, POST /v1/chat/completions
with karakeep’s own key and {"model": "or-deepseek-v4-pro"} must return 200, not 403.
engram
Section titled “engram”The scope is the agent-memory entry in tf/litellm/main.tf. Three lanes: the summarize lane and
both embed lanes. The memory-mcp-litellm ExternalSecret in
argocd/app-configs/agent-memory/external-secret.yaml reads the value into the Secret key
litellm_api_key.
Two consumers can send the same client string and reach different lanes, because each key
aliases the string on its own. One model_list entry per lane, resolved per key, is what
attributes spend to the consumer rather than to a shared upstream. Do not merge two consumers’
lanes into one.
gemini-embed is listed but not aliased, so the reindex and gap-fill Jobs can name the general lane
directly while ordinary traffic on gemini-embedding-2 resolves to engram’s own lane.
The two Gemini lanes read GEMINI_LANE_API_KEY. The litellm-secrets ExternalSecret builds it from
the Vault property gemini_api_key at secret/fzymgc-house/cluster/litellm. If the env entry is
missing, os.environ/GEMINI_LANE_API_KEY resolves to None without an error, and both lanes fail
at request time while every other lane keeps working.
life-agent
Section titled “life-agent”The scope is the life-agent entry in tf/litellm/main.tf. The life-agent-gateway
ExternalSecret in argocd/app-configs/life-agent-gateway/gateway-secret.yaml reads the value from
fzymgc-house/cluster/litellm-keys/life-agent, property api_key.
The Hermes configuration of the agent (config/config.yaml in seanb4t/life-agent) names the
models on the ZDR wildcard, and the key permits these names:
| Name | Use |
|---|---|
openrouter-zdr/anthropic/claude-sonnet-5.5 |
chat and cron. Hermes sends reasoning_effort: high |
openrouter-zdr/anthropic/claude-haiku-5.5 |
the Hermes auxiliary tasks |
openrouter-zdr/anthropic/claude-opus-5.5 |
a switch with /model |
The key needs no lane and no alias for these names. To change a model, change config.yaml in
seanb4t/life-agent and the models list of the key.
No ZDR endpoint of Claude Haiku 5.5 takes temperature, and the Hermes vision, approval and title
tasks send it. With require_parameters: true, OpenRouter then answers 404. The bundled cost map
also has no row for the model. An entry with the exact name openrouter-zdr/anthropic/claude-haiku-5.5
in argocd/app-configs/litellm-chart/values.yaml drops temperature and declares the price. LiteLLM
resolves the wildcard only for a name that no entry names, so the entry wins for this model.
claude-code
Section titled “claude-code”A workstation credential with no consumer manifest. It lives alone on
secret/fzymgc-house/cluster/litellm-keys/claude-code, property api_key. Read it into a shell
variable, never onto the screen:
LITELLM_API_KEY="$(vault kv get -field=api_key secret/fzymgc-house/cluster/litellm-keys/claude-code)"The scope is the claude-code entry in tf/litellm/main.tf. No alias map. Workstation clients
send native OpenRouter slugs through the openrouter/* wildcard. If a curated name is wanted, add
it to this key’s aliases and list both strings in models.
The 90-day TTL is deliberate. Terraform sends duration: "90d" when it creates or replaces this
key, and each in-place apply moves the expiry to 90 days after that apply.
openrouter-passthrough
Section titled “openrouter-passthrough”The generic passthrough key. It lives alone on
secret/fzymgc-house/cluster/litellm-keys/openrouter-passthrough, property api_key:
OPENROUTER_PASSTHROUGH_KEY="$(vault kv get -field=api_key \ secret/fzymgc-house/cluster/litellm-keys/openrouter-passthrough)"The scope is the openrouter-passthrough entry in tf/litellm/main.tf. It is scoped to two
wildcards, openrouter/* and openrouter-zdr/*, never left empty. An empty models list means
unrestricted: every lane the proxy serves, every consumer’s ZDR lane included.
TypeSafe Jev decisions
Section titled “TypeSafe Jev decisions”Jev is a TypeSafe model that returns typed decisions: a choice, a score, or a yes or no probability.
OpenRouter serves it as typesafe/jev-1.13 at /api/alpha/decisions only. It rejects the model on
/chat/completions, so no OpenRouter model_list entry can reach it. The proxy exposes a
pass-through route for it, which the octopus key uses. engram uses the native lane in the
subsection “The native lane to TypeSafe” below.
| Item | Value |
|---|---|
| Client URL | POST https://llm.fzymgc.house/openrouter/alpha/decisions |
| Upstream | https://openrouter.ai/api/alpha/decisions, exact, with include_subpath: false |
| Credential | the proxy’s OPENROUTER_LANE_API_KEY. The client sends only its own virtual key |
| Configuration | general_settings.pass_through_endpoints in argocd/app-configs/litellm-chart/values.yaml |
| Spend | LiteLLM records $0. The real cost is in usage.cost in the response body |
The route is deny by default, and a key needs two things to call it:
| Control | Refusal without it |
|---|---|
The route grant in the key’s metadata |
HTTP 403 Key/team not allowed to access passthrough route /openrouter/alpha/decisions |
typesafe/jev-1.13 in the key’s models list |
HTTP 403 key_model_access_denied, “The requested model ‘typesafe/jev-1.13’ is not available for this API key” |
LiteLLM checks the model field of the request body against the key’s models list since
v1.104.0-rc.1, which prices typesafe/jev-1.13 on this route. Before that release, the grant was
the only control.
Only the octopus key holds the grant. Its entry in tf/litellm/main.tf carries it in
metadata through local.decisions_route_metadata, so a replacement keeps it. The entry also
lists local.decisions_model. The engram key has no grant, because it calls the native lane
below. It still lists local.decisions_model, because that is the lane name. The provider also
has a top-level allowed_passthrough_routes argument. Do not use it. LiteLLM treats that
request field as an Enterprise feature and refuses it on this proxy. The grant in metadata is not
gated.
To grant a Terraform-owned key access to the route, set its metadata to
local.decisions_route_metadata, add local.decisions_model to its models list in
tf/litellm/main.tf, and merge the change.
To grant a scripted key access to the route:
- Read the current metadata of the key with
GET /key/info?key=<key>. - Add
"allowed_passthrough_routes": ["/openrouter/alpha/decisions"]to that metadata. - Send the full metadata with
POST /key/update. The call replaces the metadata object, so keep every existing field. - Add
typesafe/jev-1.13to the key’smodelslist withPOST /key/update. Send the full list.
curl -sS -X POST "https://llm.fzymgc.house/key/update" \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \ --data-binary '{"key": "<key>", "metadata": {"allowed_passthrough_routes": ["/openrouter/alpha/decisions"]}}'A request body looks like this. The model field is the OpenRouter slug:
{ "model": "typesafe/jev-1.13", "state": "Help! My payouts have been failing for 3 days.", "questions": { "department": { "type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "Payments, refunds", "technical": "Bugs, outages", "sales": "Pricing"} } }}Do not set TYPESAFE_API_KEY or TYPESAFE_API_BASE on the proxy to reach OpenRouter. The built-in
/typesafe/* route forwards every subpath to that base URL, and every llm_api key can call it
without a grant. With the OpenRouter key behind it, every workload key can call any OpenRouter
model outside its models list, and LiteLLM records $0 spend.
The native lane to TypeSafe
Section titled “The native lane to TypeSafe”The proxy also serves Jev on the native decisions route POST https://llm.fzymgc.house/v1/systemone.
engram calls this route. The lane typesafe/jev-1.13 in model_list forwards the call to the TypeSafe API as
typesafe/jev-1.13.0, with no OpenRouter hop. The body is the same System One body as above, and
the model field is the lane name. The route goes through the normal model pipeline, so the key
allow-list, the budgets and the spend tracking apply.
| Item | Value |
|---|---|
| Credential | the proxy’s TYPESAFE_LANE_API_KEY, from the Vault property typesafe_api_key at secret/fzymgc-house/cluster/litellm |
| Access | a key that lists typesafe/jev-1.13 in its models list |
| Response | answers, model and usage with the token counts. There is no id, provider or usage.cost |
If the env entry is missing, os.environ/TYPESAFE_LANE_API_KEY resolves to None without an error,
and only this lane fails at request time.
Provider keys use names that LiteLLM does not read
Section titled “Provider keys use names that LiteLLM does not read”LiteLLM has built-in pass-through routes, for example /gemini/*, /openrouter/*, /fal_ai/*
and /typesafe/*. Each one forwards every subpath with a provider key that it finds under a
standard env name, such as GEMINI_API_KEY, OPENROUTER_API_KEY or FAL_AI_API_KEY. Every llm_api key can call these
routes without a grant. If the proxy set the standard names, every workload key can spend on the
proxy’s provider accounts with no model or budget limit.
The proxy therefore sets OPENROUTER_LANE_API_KEY, OPENROUTER_ZDR_LANE_API_KEY,
GEMINI_LANE_API_KEY and TYPESAFE_LANE_API_KEY, and every lane and the configured pass-through name them explicitly. The built-in routes find no credential. Do not add an
env var with a standard provider name to extraEnvVars. Use <PROVIDER>_LANE_API_KEY.
To make sure that the routes stay closed after a change, send a plain workload key to each route
in the table below. Each route reads the key from its own place, so send it where the route reads
it. The Gemini route reads x-goog-api-key or ?key=. With Authorization: Bearer, the request
fails at admission with 401 ... expected to start with 'sk-', which looks like a pass and tests
nothing.
| Route | Where the key goes | Answer when the route is closed |
|---|---|---|
GET /gemini/v1beta/models |
x-goog-api-key: <key> |
500. The proxy log shows Required 'GEMINI_API_KEY'/'GOOGLE_API_KEY' in environment |
GET /openrouter/v1/key |
Authorization: Bearer <key> |
401 Missing Authentication header, from OpenRouter |
GET /fal_ai/<any path> |
Authorization: Bearer <key> |
401 FAL_AI_API_KEY is not set |
A 200 from any row means that the route found a provider credential. Find the standard env name in
extraEnvVars and rename it to the <PROVIDER>_LANE_API_KEY form.
The ZDR key
Section titled “The ZDR key”The ZDR key is an OpenRouter key for the lanes that must reach only providers with zero data
retention (ZDR). The key lives in the OpenRouter workspace litellm-zdr. The default guardrail of
that workspace enforces ZDR for all five model groups, on every request of every key in the
workspace. The zdr field of a request cannot remove that rule, so a provider block from a
caller cannot reach a provider without ZDR.
| Item | Where | Owner |
|---|---|---|
Workspace litellm-zdr |
OpenRouter | tf/litellm/openrouter.tf |
Key litellm-zdr-lanes |
OpenRouter, in litellm-zdr |
tf/litellm/openrouter.tf |
Default guardrail of litellm-zdr |
OpenRouter | the management API call below |
| Management key | secret/fzymgc-house/infrastructure/openrouter/terraform, property management_key |
Sean, by hand |
| ZDR key value | secret/fzymgc-house/cluster/litellm/openrouter-zdr, property api_key |
tf/litellm/openrouter.tf |
Keep only ZDR keys in litellm-zdr. The plain lane key litellm lives in the workspace
fzymgc-litellm, which has no ZDR guardrail.
The proxy reads the ZDR key as OPENROUTER_ZDR_LANE_API_KEY. These lanes name it:
openrouter-zdr/*, engram-summarize, octopus-embed, octopus-utility, octopus-review and
the probe lane zdr-probe-embed. The lanes openrouter/* and gpt-4o-mini and the decisions route
use OPENROUTER_LANE_API_KEY. Their callers can choose a provider without ZDR.
Do not set TF_LOG on the workspace main-cluster-litellm. At DEBUG, the OpenRouter provider logs
whole response bodies, and the body of a new key holds the key.
Set the default guardrail
Section titled “Set the default guardrail”Provider OpenRouterTeam/openrouter 0.3.19 cannot manage the default guardrail of a workspace. Until
a release can, an operator sets it with one call. The output openrouter_zdr_default_guardrail_id of
tf/litellm gives the same id that this lookup finds:
export VAULT_ADDR=https://vault.fzymgc.houseMK="$(vault kv get -field=management_key secret/fzymgc-house/infrastructure/openrouter/terraform)"GID="$(curl -s https://openrouter.ai/api/v1/workspaces -H "Authorization: Bearer $MK" \ | jq -r '.data[] | select(.slug == "litellm-zdr") | .default_guardrail_id')"curl -s -X PATCH "https://openrouter.ai/api/v1/guardrails/$GID" \ -H "Authorization: Bearer $MK" -H 'Content-Type: application/json' \ -d '{"enforce_zdr_anthropic":true,"enforce_zdr_openai":true,"enforce_zdr_google":true, "enforce_zdr_xai":true,"enforce_zdr_other":true}' \ | jq -c '(.data // .) | {name, zdr: [.enforce_zdr_anthropic, .enforce_zdr_openai, .enforce_zdr_google, .enforce_zdr_xai, .enforce_zdr_other]}'unset MKThe reply must show five times true. No plan shows a later change to this guardrail. After each
change to the OpenRouter account, send the request below again.
Prove the key
Section titled “Prove the key”A request that asks for StreamLake, a provider of deepseek/deepseek-v4.1-flash without ZDR, must not
reach it:
ZDR_KEY="$(vault kv get -field=api_key secret/fzymgc-house/cluster/litellm/openrouter-zdr)"curl -s https://openrouter.ai/api/v1/chat/completions \ -H "Authorization: Bearer $ZDR_KEY" -H 'Content-Type: application/json' \ -d '{"model":"deepseek/deepseek-v4.1-flash","max_tokens":256, "provider":{"only":["StreamLake"],"zdr":false}, "messages":[{"role":"user","content":"Reply with the word OK."}]}' \ | jq -c '{provider, error: .error.message}'unset ZDR_KEYThe expected answer is HTTP 404 with “ZDR violation (guardrail)” in the message. If the reply names StreamLake, the guardrail is not in force. Set it again with the call above.
Prove the ZDR lanes
Section titled “Prove the ZDR lanes”The request above proves the key. This one proves the lanes, because LiteLLM passes a provider
block from the caller to OpenRouter in place of the block of the lane. Run it from a workstation with
Vault access:
export VAULT_ADDR=https://vault.fzymgc.housekey() { vault kv get -field=api_key "secret/fzymgc-house/cluster/litellm-keys/$1"; }ask() { curl -s https://llm.fzymgc.house/v1/chat/completions \ -H "Authorization: Bearer $1" -H 'Content-Type: application/json' \ -d "{\"model\":\"$2\",\"max_completion_tokens\":256,\"provider\":$3, \"messages\":[{\"role\":\"user\",\"content\":\"Reply with the word OK.\"}]}" \ | jq -c --arg m "$2" '{model: $m, provider, error: (.error.message // "" | .[0:90])}'}OCTOPUS="$(key octopus)"; PASS="$(key openrouter-passthrough)"ask "$OCTOPUS" octopus-review '{"only":["StreamLake"],"zdr":false}'ask "$PASS" openrouter-zdr/deepseek/deepseek-v4.1-flash '{"only":["StreamLake"],"zdr":false}'ask "$PASS" openrouter/deepseek/deepseek-v4.1-flash '{"only":["StreamLake"]}'The first two lines must not name StreamLake as provider. The third line must name StreamLake: it
proves that StreamLake serves the model and that the plain wildcard does not use the ZDR key. If the
third line does not name StreamLake, the first two prove nothing.
Each refused request reaches LiteLLM as a 404, and LiteLLM takes a deployment out of service for 5 seconds after one 404. A ZDR lane has one deployment, so for those 5 seconds the lane answers every caller with HTTP 429 “No deployments available”. Wait 5 seconds before you send a plain request to the lane.
Run this after each change to the OpenRouter account, after each change to the lanes in
values.yaml, and after each LiteLLM version bump.
Rotate the keys
Section titled “Rotate the keys”To rotate the ZDR key, raise zdr_key_version in tf/litellm/openrouter.tf and merge. The apply
creates the new key, writes it to Vault and deletes the old key. The ZDR lanes fail until the proxy
has the new value: ESO refreshes litellm-secrets within five minutes, and Stakater Reloader then
restarts the proxy. To shorten that time, force the refresh as in step 5 of the procedure for a
scripted key.
To rotate the management key, create a new one in the OpenRouter dashboard, write it with
vault kv put to its path, and delete the old one in the dashboard. The path holds only
management_key, so kv put loses nothing.
MCP client keys
Section titled “MCP client keys”A different shape from every key above. The consumer keys are scoped by models, which LLM lanes
they can reach. An MCP client key is scoped by object_permission.mcp_servers, which MCP routes it
can reach. The two scopes are independent. A key that must do both carries both. The measured
scope rules for MCP routes are in MCP Identity Routes.
They are virtual keys like any other: the same /key/generate, the same GET /key/info, the same
revocation path, the same spend attribution.
{ "key_alias": "<consumer>-mcp", "user_id": "<consumer>", "key_type": "llm_api", "object_permission": {"mcp_servers": ["context7", "exa", "clickhouse_ro"]}, "rpm_limit": 120, "max_budget": 10.0, "budget_duration": "30d", "duration": null}| Field | Why it is there |
|---|---|
object_permission.mcp_servers |
the route names this key can reach, the entire boundary. Use underscore spellings (clickhouse_ro), the names LiteLLM matches against |
user_id |
whose key it is. MCP tool-call spans carry it, which is what makes per-caller attribution work on the MCP plane |
key_alias |
the human-readable label on the span and in spend reports |
If the user of the key is not a proxy_admin, omitting object_permission reaches nothing, and
that is the intended default. It is the inverse of models, where omitting it means unrestricted.
A key of a proxy_admin user with no object_permission list reaches every MCP route. An
explicit list binds the key of a proxy_admin user too, even an empty list []. Plan 03-13
measured both rules on LiteLLM v1.103.0-rc.1.
If the user of the key is not a proxy_admin, LiteLLM refuses a key with no object_permission
at initialize with HTTP 403, so each key needs a grant for each route. The dns-agent-triage key is Terraform-owned and cannot carry
object_permission. The access group mcp-dns in tf/litellm/access_groups.tf grants it the
dns route only. Its models list names no lane, so the key serves no chat request. dns-control
still verifies the Keycloak bearer of every caller. See
A Terraform-owned key and an access group.
clickhouse_rw is the estate’s only MCP write surface, and both ClickHouse MCP instances validate
the same upstream token, so this list is the only thing that separates read from write.
The two identity routes need no extra field on the key. engram and kubernetes relay the
caller’s own Keycloak token to the upstream. The virtual key admits and scopes, and the identity
rides a second header the client sends. See
How authentication works.
Verify with GET /key/info as for any other key. It reads the scope back as internal ids rather
than names, which is canonical storage, not drift:
curl -sS "https://llm.fzymgc.house/key/info" \ -H "Authorization: Bearer <minted-key>" | jq '.info.object_permission'Authenticate as the key, per Verifying a minted key step 1.
Verifying a minted key
Section titled “Verifying a minted key”Four steps, in this order.
-
Scope.
GET /key/infoshows the expectedmodels,aliases, limits, budgets andkey_type:Terminal window curl -sS "https://llm.fzymgc.house/key/info" \-H "Authorization: Bearer $SEEDED_KEY" \| jq '.info | {models, aliases, key_type, rpm_limit, tpm_limit,max_budget, budget_duration, expires,soft_budget: .litellm_budget_table.soft_budget}'Authenticate as the key. Never put it in the query string.
?key=<value>lands the credential in LiteLLM’s access log, in Traefik’s, and in/proc/<pid>/cmdlinefor the life of thecurl. With nokeyparameter,/key/inforeports the key in theAuthorizationheader. A 401 here means the value is not registered at all, which is itself the answer to step 3.soft_budgetis not on the key row. A/key/generatethat carriessoft_budgetcreates a row inlitellm_budgettableand links it bybudget_id. The key’s ownsoft_budgetfield staysnull. Read it fromlitellm_budget_table, as the filter above does. -
Reachability. One probe call with the client-sent model string returns 200. Use the string the workload sends, not the lane name. That is what exercises the pre-alias allow-list.
-
Seed and register match. The value LiteLLM holds for this key and the value at that consumer’s Vault path must be the same value. The check is an authenticated round trip.
/key/infodoes not return the registered value, so there is no second value to digest. A value seeded but never registered returns 401, and a value registered under a different secret returns 401 for this one. Carry the digest as an identifier, never print the value:Terminal window VAULT_DIGEST="$(printf '%s' "$SEEDED_KEY" | shasum -a 256 | cut -c1-8)"PROBE_STATUS="$(curl -sS -o /dev/null -w '%{http_code}' \-H "Authorization: Bearer $SEEDED_KEY" \https://llm.fzymgc.house/v1/models)"RESOLVED_ALIAS="$(curl -sS "https://llm.fzymgc.house/key/info" \-H "Authorization: Bearer $SEEDED_KEY" | jq -r '.info.key_alias // "none"')"if [ "$PROBE_STATUS" = "200" ] && [ "$RESOLVED_ALIAS" = "octopus" ]; thenecho "coupled: vault value ${VAULT_DIGEST} authenticates and resolves to alias ${RESOLVED_ALIAS}"elseecho "NOT COUPLED vault=${VAULT_DIGEST} probe=${PROBE_STATUS} alias=${RESOLVED_ALIAS}"fiSubstitute the consumer’s own expected alias for
octopus.shasum -a 256rather thansha256sum, because macOS ships the former.On a mismatch, re-register from Vault. For a scripted key, call
/key/generateagain with the Vault value. Do not re-seed, which produces a third distinct value. For a Terraform-owned key, do not call/key/generate. Raise itskey_wo_versionwithout a new Vault write, and the apply replaces the key with the value Vault holds. -
Record the result, the value’s digest, the observed statuses and the resolved alias, in the pull request that made the change.
The probe lane is granted to nobody
Section titled “The probe lane is granted to nobody”zdr-probe-embed is a deliberately unsatisfiable harness lane. It serves no consumer, maps to no
route, and its guaranteed failure is the point. No credential lists it.
Two controls hold that. A static check over the mint bodies extracts every models array in this
document and in the Terraform module, and asserts the probe lane is a member of none of them. The
check is over the models arrays, not the whole file, because the live denial below names the lane
in order to deny it. Extracting zero models arrays is a failure, not a pass:
rg -oP '"models":\s*\[[^\]]*\]' docs/operations/litellm.mdrg -U -oP 'models\s*=\s*\[[^\]]*\]' tf/litellm/main.tfA live denial, because a runbook is documentation and not access control. With the octopus key, issue
one embedding request that names zdr-probe-embed:
curl -sS -o /dev/null -w '%{http_code}\n' \ -X POST https://llm.fzymgc.house/v1/embeddings \ -H "Authorization: Bearer $SEEDED_KEY" \ -H 'Content-Type: application/json' \ -d '{"model": "zdr-probe-embed", "input": "denial probe"}'The expected outcome is a model-access denial that names the model, not an authentication failure and not a 200. If it returns 200, a harness-only lane is reachable by a production workload.
What actually revokes a key
Section titled “What actually revokes a key”POST /key/delete, and it is not instant. Budget about 60 seconds before the credential is dead on
every replica.
Narrowing a key’s models with POST /key/update is not a revocation mechanism.
Every key mutation has two halves that converge at different speeds:
| Half | When it changes | How you observe it |
|---|---|---|
| the authoritative row in Postgres | at once, before the 200 returns | GET /key/info reports the new shape at once |
| what each replica enforces | the replica that handled the call changes inline. Every other replica keeps its cached copy until that copy expires | only a per-replica probe through kubectl port-forward can see this |
The cache that expires is LiteLLM’s user_api_key_cache. Its in-memory layer has a 60 second TTL.
replicaCount in values.yaml sets how many replicas hold such a cache. The Redis layer, which
enable_redis_auth_cache turns on, is shared. A replica that misses in memory reads the current
state from Valkey.
v1.104 added instant eviction across replicas over Redis pub/sub. On v1.104.0-rc.1 that path was
off for the Valkey cluster client, and every start logged auth cache invalidation subscriber disabled: cluster redis client has no pub/sub support. Upstream commit d33d36ce86
(BerriAI/litellm#43110) adds a cluster-aware subscriber, and v1.105.0-rc.3 includes it. The
v1.105.0-rc.3 pods do not log that line at start. No test on this cluster shows yet that an
eviction reaches every replica. Until one does, use the 60 second
in-memory TTL as the window.
If the credential is believed compromised, do not wait out the TTL. Delete the key, then
kubectl rollout restart deployment/litellm -n litellm. A pod restart discards every in-memory
cache and is the only mechanism here that revokes faster than 60 s.
A deleted key goes on spending for the length of that window. The spend ledger keeps accruing against it until every replica’s cache expires, and the restart above is the only lever that ends it sooner.
Concurrency: what is and is not guaranteed
Section titled “Concurrency: what is and is not guaranteed”After POST /key/update returns 200, exactly one thing is guaranteed: the authoritative row has
changed, and any later GET /key/info reports the new shape. Enforcement is eventually consistent
across replicas with the 60 s bound above.
Two admin mutations that race on the same key have no stated ordering guarantee. Serialize admin mutations on a single key: complete one and verify it before you issue the next.
Reading Valkey directly: VALKEYCLI_AUTH shadows REDISCLI_AUTH
Section titled “Reading Valkey directly: VALKEYCLI_AUTH shadows REDISCLI_AUTH”The valkey-litellm-valkey-* pods set VALKEYCLI_AUTH in the container environment for the
operator’s own user, and valkey-cli prefers VALKEYCLI_AUTH over REDISCLI_AUTH. Supplying the
correct default password in REDISCLI_AUTH therefore authenticates as the operator user with the
default user’s password and reports WRONGPASS invalid username-password pair. Unset it first:
kubectl get secret -n litellm litellm-valkey-auth -o jsonpath='{.data.password}' | base64 -d \| kubectl exec -i -n litellm valkey-litellm-valkey-0-1-0 -c server -- sh -c \ 'unset VALKEYCLI_AUTH; REDISCLI_AUTH=$(cat); export REDISCLI_AUTH; exec valkey-cli --no-auth-warning --user default "$@"' sh INFO statsThe password reaches the pod on stdin and becomes an environment variable in the pod’s shell. Do
not use valkey-cli -a <password>. That places the credential in argv, where /proc/<pid>/cmdline
and any ps on that node can read it.
A key’s cache entry is named for the SHA-256 of the key value, so
valkey-cli --scan --pattern '*<sha256-of-key>*' on the primary (confirm with INFO replication)
lists it alongside its spend:key:* and {api_key:*}:tokens counters.
Rotating a virtual key
Section titled “Rotating a virtual key”In-place rotation is not available on this estate. POST /key/regenerate and
POST /key/{key}/regenerate are Enterprise-gated in LiteLLM. This deployment sets no
LITELLM_LICENSE, so both route spellings return HTTP 500. The supported procedure is revoke and
re-mint, and it has a short outage. No test asserts the gate. It is a vendor’s licensing decision,
and a test on it would go red when LiteLLM changes their product, not when we make a mistake. If
you set a LITELLM_LICENSE, this section is what has to change. See
If a licence is ever acquired.
Key aliases are globally unique. A second /key/generate that carries an alias LiteLLM already
holds is rejected with Key with alias '<name>' already exists. So the replacement cannot be minted
under the consumer’s alias while the old key exists. The order is forced: delete first, then mint.
The interval between them, plus the time the new value takes to reach the running pod, is the
outage. Keeping the alias is what keeps every spend query, tile and alert pointing at one
continuous series across the rotation.
There is no overlap window. Two consequences for a key minted with a duration:
- Rotating restarts the clock. The replacement is a fresh
/key/generate, so itsdurationstarts at the re-mint. - A key you let expire costs the same outage as one you rotate, at a time you did not pick. Rotate
the TTL-bearing keys ahead of their expiry, in a quiet window. The two workstation credentials
claude-codeandopenrouter-passthroughcarry"duration": "90d". Read their expiry withGET /key/info(.info.expires).
The outage: how long, what breaks, what to do about it
Section titled “The outage: how long, what breaks, what to do about it”The window runs from /key/delete until the workload’s pod runs with the new value. It has four
legs, and only the last two are worth managing:
| Leg | Duration | Notes |
|---|---|---|
/key/delete to the new value serving on the consumer’s lane |
under a second | the outage starts at the delete, not at the Vault write |
/key/delete to the old value refused everywhere |
up to the 60 s auth cache TTL | see What actually revokes a key. A fast 401 through the ingress means “you can re-mint now”, not “the old value is dead” |
| Vault write to the Secret carrying it | about a second if you force the sync, up to refreshInterval: 5m if you do not |
the leg you can shorten, step 5 below |
| Secret updated to the pod running with it | about a minute for a rollout, and never without one | see Propagation is ESO plus a pod restart |
Run the commands back to back and the outage is a little over a minute. Wait out a refresh interval instead and it is about six.
What breaks: every call the consumer makes to https://llm.fzymgc.house/v1 is refused with 401, an
authentication failure. Nothing else in the estate is affected.
Decide how to handle the consumer before you start:
- The consumer retries.
karakeep,mealieandoctopusare background or interactive-with-retry workloads. A failed enrichment or chat turn is retried, and the window closes on its own. - Queue-shaped work drains or pauses first. If the consumer is mid-batch (an
engramcorpus reindex, anoctopusreview sweep), a 401 storm burns through its retry budget and marks work failed. Let the batch finish, or pause it, before you delete the key.
Rotate during a quiet window. No mechanism makes the outage smaller. The only lever is when it lands.
Rotating a Terraform-owned key
Section titled “Rotating a Terraform-owned key”The keys in tf/litellm/main.tf rotate through Terraform. The mechanism is the same delete
and re-create under the same alias. key_wo_version forces a replacement, and Terraform deletes
the old key before it creates the new one. The new key carries the value that Vault holds when the
apply runs. Do not add create_before_destroy to the resource. LiteLLM keeps aliases unique, so the
new key cannot exist beside the old one.
The apply reads Vault when it runs, so the order of the Vault write and the merge matters. If you write the new value earlier, ESO can deliver it to the pod before any key accepts it, and the consumer is down from that moment.
-
Decide how to handle the consumer during the outage, per the list above. Make sure that the
litellm-db-failurealert is not firing. The first caveat below explains why. -
Open a PR that raises
key_wo_versionof that key by one intf/litellm/main.tf. Change nothing else in the module. -
Read the speculative plan of
main-cluster-litellmon the PR. It must show exactly one resource that must be replaced, withkey_wo_versionmarked# forces replacement, and no other change. The value is write-only and does not appear in the plan. -
Immediately before you merge, write the new value into Vault:
Terminal window scripts/seed-litellm-vault.sh --consumer <name> --rotate<name>is the consumer name from Which tool owns each key, for exampleagent-memoryfor theengramalias. The command writes the value and does not register it. -
Merge the PR. The workspace applies automatically. Watch the run in the HCP Terraform UI or with the read-only
run-detailstool. Do not trigger a run through the API or the MCP tools. -
When the apply is complete, force the ExternalSecret to sync. This is step 5 of the scripted procedure below, and its table names each ExternalSecret.
-
Verify, per step 7 of the scripted procedure below. The shape must match the entry in
tf/litellm/main.tf, andtokenon the new row must equal the hash of the value in the pod.
If the PR cannot merge after step 4, the consumer receives a value that no key accepts within one
refresh interval. Restore the prior value at once with vault kv rollback -version=<prior version>
on that key’s dedicated path. The path holds only api_key, so the rollback restores nothing else.
If the apply fails after the delete and before the create, the key is gone and the consumer is down. Read the run error. When the cause is fixed, start a new run from the HCP Terraform UI. The run creates the key from the value in Vault.
The procedure for a scripted key
Section titled “The procedure for a scripted key”This procedure applies to the MCP client keys. For a Terraform-owned key, follow the procedure above. Steps 5 to 7 below are shared.
-
Capture the key’s current shape before you delete anything. This is what you re-mint against and what you diff afterwards. Read it with the master key, by alias. The consumer’s own key cannot read it, because a
key_type: llm_apicredential is refused on/key/infowith 403:Terminal window ALIAS=<consumer>-mcp # the LiteLLM key_aliascurl -sS "https://llm.fzymgc.house/key/list?key_alias=${ALIAS}&return_full_object=true" \-H "Authorization: Bearer $LITELLM_MASTER_KEY" \| jq '.keys[0] | {key_alias, models, aliases, rpm_limit, tpm_limit,max_budget, budget_duration, key_type, budget_id, spend}' \| tee /tmp/rotate-${ALIAS}-before.json# soft_budget lives in the budget table and /key/list does NOT join it: the row# reports `soft_budget: null` and `litellm_budget_table: null` even when one is set.BUDGET_ID="$(jq -r .budget_id /tmp/rotate-${ALIAS}-before.json)"curl -sS "https://llm.fzymgc.house/budget/list" \-H "Authorization: Bearer $LITELLM_MASTER_KEY" \| jq --arg id "$BUDGET_ID" '.[] | select(.budget_id == $id) | {soft_budget}'Cross-check it against that consumer’s mint body under Minting and scoping virtual keys. Step 4 re-uses that body verbatim.
-
Delete the old key. The outage starts now:
Terminal window printf '%s' "$OLD_KEY" \| python3 -c 'import json, sys; print(json.dumps({"keys": [sys.stdin.read().strip()]}))' \| curl -sS -X POST "https://llm.fzymgc.house/key/delete" \-H "Authorization: Bearer $LITELLM_MASTER_KEY" \-H "Content-Type: application/json" --data-binary @-python3builds the body from stdin.jq -n --arg keywould put the credential in jq’s argument vector. -
Write the new value into that consumer’s Vault target first, the same path and property its delivery already reads. Vault is the origin of the value, and step 4 attaches scope to it. If step 4 fails, the value is safely in Vault and the fix is to retry step 4. Derive the path and property from
CONSUMER_TABLErather than typing them, and hand the value tovaulton stdin:Terminal window # CONSUMER is the name the seed script accepts (`--consumer <name>` prints the# accepted set on an unknown one). For engram that name is `agent-memory`; the# alias `engram` is a LiteLLM label, not a Vault path.read -r VK_PATH VK_PROPERTY <<<"$(rg -N "(^|CONSUMER_TABLE=')${CONSUMER}\|" scripts/seed-litellm-vault.sh \| sed "s/^CONSUMER_TABLE='//" \| awk -F'|' '{ print $2, $3 }')"[ -n "$VK_PATH" ] && [ -n "$VK_PROPERTY" ] || echo "no CONSUMER_TABLE row for '${CONSUMER}'"# The same generator new_virtual_key() uses in scripts/seed-litellm-vault.sh:# VIRTUAL_KEY_PREFIX + secrets.token_urlsafe(VIRTUAL_KEY_TOKEN_BYTES). Keep the# value in a shell variable; do not print it.NEW_KEY="$(python3 -c 'import secrets; print("sk-" + secrets.token_urlsafe(32))')"printf '%s' "$NEW_KEY" | vault kv patch "$VK_PATH" "${VK_PROPERTY}=-"vault kv patchprints a recommendation to add thepatchcapability to your ACL policy and then falls back to read-then-write. That is a warning, not a failure. Read the exit status.printfrather than a here-string, because<<<appends a trailing newline to the value. The path out of the table already carries itssecret/mount, so no-mountflag. -
Register that value with
/key/generate, under the samekey_alias, with that consumer’s recorded mint body and"key"set to the value Vault now holds. Do not invent a new alias. A changed alias re-points every spend query and the spend alert at a series that starts at zero. If this call fails, the value is still in Vault. Retry it. -
Force the ExternalSecret to sync rather than wait out
refreshInterval: 5m:Terminal window kubectl annotate externalsecret ${EXTERNAL_SECRET} -n ${NAMESPACE} \force-sync=$(date +%s) --overwriteWatch
kubectl get externalsecret ${EXTERNAL_SECRET} -n ${NAMESPACE} -o jsonpath='{.status.refreshTime}', not the clock. The annotation does not put the ExternalSecret out of sync in ArgoCD.Consumer ExternalSecret Namespace karakeepkarakeep-secretskarakeepmealiemealie-app-secretsmealieoctopusoctopus-appoctopusmemory-mcp(engram)memory-mcp-litellmagent-memory -
Make sure that the pod restarted with the new value. See Propagation is ESO plus a pod restart. Stakater Reloader does it, and you are confirming, not causing.
-
Verify with a real call on the consumer’s own lanes, then diff the key’s shape. Re-run step 1’s
/key/listand/budget/listpair. Its output must match/tmp/rotate-${ALIAS}-before.jsonon alias,models,aliases, limits and budgets.token,created_atandspenddiffer. Then call every lane the consumer uses, not just one. A re-mint that dropped a singlemodelsentry leaves a credential that works until the consumer reaches for the missing lane.Cross-check that the pod holds the registered credential. LiteLLM stores
tokenas the SHA-256 of the key value, so thetokenon the row you read must equal the hash of the value in the workload’s environment.
Three caveats, each a way the procedure looks fine and is not
Section titled “Three caveats, each a way the procedure looks fine and is not”You cannot rotate or revoke during a database outage, and the outage is what makes it dangerous.
/key/delete and /key/generate are both database writes, while completions keep flowing on
cached auth because allow_requests_on_db_unavailable stays true. A rotation started during a
masked Postgres outage can delete the old key and then fail to mint its replacement, and the proxy
looks healthy the whole time. The signal is the litellm-db-failure alert. If it is firing, or
fired recently, wait. See When the database is unavailable.
Revocation is fast on the pod you talked to and slow everywhere else. A fast 401 through the ingress is the ingress routing you back to the replica that handled the delete. Read it as “you can re-mint now”, and plan revocation against the 60 s bound in What actually revokes a key.
A deleted key goes on spending for the length of that window, attributed to its alias under the
key id that was deleted. If you revoke because a credential is believed compromised, the only
lever that ends it sooner is kubectl rollout restart deployment/litellm -n litellm.
The alias is continuous across a rotation and the key id is not. A spend query grouped on
user_api_key_alias sees one unbroken series across the rotation. A query grouped on the key id
sees a series that ends and a new one that starts at zero. The key row’s own spend field resets
with the key. That is why the tiles and the spend alert are keyed on the alias.
Propagation is ESO plus a pod restart
Section titled “Propagation is ESO plus a pod restart”A Vault write that reaches the Kubernetes Secret is not the workload picking it up. Every consumer
reads its LiteLLM credential through env.valueFrom.secretKeyRef, and an environment variable is
injected once, at container start. The full chain is: Vault write, ESO refresh (forced, or
refreshInterval: 5m), Secret updated, pod restarted, new value in use.
The restart comes from Stakater Reloader, which acts only on workloads that carry the annotation:
| Workload | reloader.stakater.com/auto |
Rotation restarts it automatically? |
|---|---|---|
karakeep/karakeep |
true |
yes |
mealie/mealie |
true |
yes |
octopus/octopus |
true |
yes |
agent-memory/memory-mcp |
true |
yes |
What the seed script is for, and what it is not
Section titled “What the seed script is for, and what it is not”No mode of scripts/seed-litellm-vault.sh is the rotation path on its own:
- A bare re-run regenerates everything at once: the CNPG password, the master key, the Valkey password and all eight workload virtual keys. It has no overwrite refusal.
--consumer <name>regenerates exactly one consumer’s value, and refuses to overwrite an existing value without--rotate.
Either way the new value is dead in LiteLLM until it is registered. Vault before registration is the correct order. What disqualifies the script as a rotation is that it never deletes the live key: the old value stays registered while Vault changes underneath it, so the next ESO refresh can hand the workload a value LiteLLM has never seen, with no bounded window and no signal. The procedures above delete first so that the outage has a beginning and an end.
One exception: step 4 of Rotating a Terraform-owned key runs
--consumer <name> --rotate on a live key. That is safe only because the merge that follows
deletes the old key and registers the new value within the same few minutes.
If a licence is ever acquired
Section titled “If a licence is ever acquired”Setting LITELLM_LICENSE makes POST /key/regenerate available, with a grace_period that keeps
the old value valid across the Vault write and the ESO refresh. Three things to know before you
adopt it:
- The overlap is one key with two values, not two keys. Spend, budget and rate limits are shared during the window.
- The overlap lookup is a database read, so it fails in the masked-outage window described in the first caveat above.
- A positive in-memory cache sits in front of that lookup, so grace-period revocation is eventual within about a minute in both directions.
Nothing automated tells you to revisit this section when the licence is set. Whoever sets
LITELLM_LICENSE owns rewriting the procedures above around grace_period.
Rotating the master key
Section titled “Rotating the master key”The master key is not just an API credential. The Admin UI session token is a JWT signed with it, so a rotation signs out every logged-in operator at once. Treat a master-key rotation as a visible operator-facing event, and do not schedule one in the middle of an incident worked through the UI.
Update every place the master key is held in the same change: the Vault property master_key on
secret/fzymgc-house/cluster/litellm-keys/master, and whatever the operator workstation exports as
LITELLM_MASTER_KEY. The litellm-secrets ExternalSecret delivers it to the proxy, and Reloader
restarts the proxy.
The master-key password form must stay unroutable
Section titled “The master-key password form must stay unroutable”LiteLLM does not fail closed when its SSO environment is missing or wrong. It renders an HTML
username and password form whose POST authenticates against the master key itself. Publishing
/login or /fallback/login puts a master-key password prompt on the internet-facing host.
The proxy also refuses the login itself. general_settings in
argocd/app-configs/litellm-chart/values.yaml sets disable_env_credential_login: true, which
turns off the UI login with the master key, and disable_password_login_when_sso_enabled: true,
which rejects every username and password UI login while SSO is configured. API calls with
Authorization: Bearer <master key> do not go through the UI login and keep working. If Keycloak is
down, administer the proxy over the API with the master key.
Two controls hold the routing, and they cover different halves:
| # | Control | What it proves | Where |
|---|---|---|---|
| 1 | The exclusions in the IngressRoute rule | Nothing on its own. It is the mechanism. Both exclusion matchers must stay written in the rule, and nothing checks that they do | argocd/app-configs/litellm/ingress.yaml |
| 2 | The router-side check below | That Traefik refuses the paths, including after a deleted matcher syncs | this section, run by hand |
Confirming the master-key password form is unroutable
Section titled “Confirming the master-key password form is unroutable”Run it after any change to argocd/app-configs/litellm/ingress.yaml, once ArgoCD reports the
IngressRoute synced, after a Traefik upgrade, and once a quarter whether or not anything changed.
Run it from any shell on the LAN that reaches the host through the Traefik LoadBalancer VIP,
outside Kubernetes, with no port-forward. No browser is needed, so a scheduled job can run it.
curl -sS -i https://llm.fzymgc.house/logincurl -sS -i https://llm.fzymgc.house/fallback/loginPass: both answer Traefik’s own 404. Status 404, content-type text/plain, no server header,
body exactly:
404 page not foundFail: anything served by LiteLLM instead of refused by Traefik. A JSON body, a server: uvicorn
response header, or a 200 text/html carrying a login form. Any of those means the exclusion is
not honoured and the master-key password prompt is on the network. Treat it as an incident.
A Traefik 404 from this host means an exclusion was hit, not a missing prefix. The root is published, so there are no other 404s of that shape.
Client addresses behind Traefik
Section titled “Client addresses behind Traefik”Every connection to the proxy arrives from a pod. People and off-cluster clients come through a
Traefik pod, and in-cluster workloads call the Service. The peer address is therefore never the
real client. Traefik puts the real client address in the X-Forwarded-For header, because its
Service runs externalTrafficPolicy: Local.
Three keys in general_settings in argocd/app-configs/litellm-chart/values.yaml make LiteLLM
read that header. All three name the same range, the Cilium pod pool 10.245.0.0/16. That is also
the value of FORWARDED_ALLOW_IPS in extraEnvVars.
| Key | Effect |
|---|---|
use_x_forwarded_for: true |
Spend rows and the auth error log record the address from the header, not the Traefik pod. Without it, every request logs Received a request with an X-Forwarded-For header but use_x_forwarded_for is not enabled |
trusted_proxy_ranges |
Turns on the per-address limit on failed Admin UI sign-ins. Without it, the proxy counts failures per user name only, and every start logs trusted_proxy_ranges is not set or not a valid list of ranges |
mcp_trusted_proxy_ranges |
Lets the MCP routes resolve the client address from the header. With use_x_forwarded_for and no range, that lookup fails closed and the first request logs a warning |
The sign-in limit uses the LiteLLM defaults. One address gets 10 failed attempts in 60 seconds,
then a block of 300 seconds. One user name from one address gets 5. A blocked attempt gets 429
before the password check. The Admin UI accepts only SSO sign-ins, and the ingress hides the
password form, so the limit guards a path that is already closed. The setting keeps the count
correct if that form ever opens.
An in-cluster workload that calls the Service directly is inside the range too, so it can put any
address in its own X-Forwarded-For. It already holds a virtual key, so a false address changes
only what its rows record. Do not widen the range past the pod pool, and never use 0.0.0.0/0.
Prompt and response logging
Section titled “Prompt and response logging”LiteLLM stores the request body and the response of each call in the LiteLLM_SpendLogs table.
Two keys in general_settings in argocd/app-configs/litellm-chart/values.yaml control this:
| Key | Value | Effect |
|---|---|---|
store_prompts_in_spend_logs |
true |
Fills the proxy_server_request and response columns |
maximum_spend_logs_retention_period |
"7d" |
A daily job deletes spend-log rows older than seven days |
To read a call, open the Admin UI at https://llm.fzymgc.house/ui and go to the Logs page. You can
filter by key, model and time. The OTel spans in ClickStack carry token counts and cost only. They
do not carry prompt or response text.
LiteLLM cuts each string field to 8192 characters before it writes the row. The
MAX_STRING_LENGTH_PROMPT_IN_DB environment variable in extraEnvVars sets this limit. Without it,
the limit is 2048 characters.
The rows contain everything that the callers send, for example the memories that engram summarizes.
The same rows go into the CNPG backups. Do not remove the retention period while
store_prompts_in_spend_logs is true. Without it, the table grows without a limit in a database
that other tenants share.
The retention job deletes whole rows, so the per-request history in LiteLLM_SpendLogs covers
seven days. It also deletes rows from LiteLLM_SpendLogToolIndex. Budgets and the Usage pages keep
their history. LiteLLM keeps budget spend on the key, team and user rows, and the Usage pages read
the daily spend tables, which the job does not touch.
When the database is unavailable
Section titled “When the database is unavailable”LiteLLM runs with allow_requests_on_db_unavailable: true, which produces a lopsided failure mode.
Knowing the shape in advance stops an operator from diagnosing two faults where there is one.
The four behaviours
Section titled “The four behaviours”Completions keep flowing. Auth resolves from cache, so requests on keys that the proxy already saw continue to work through a Postgres outage. The readiness and startup probes do not check the database, so the pods stay in the Service.
The outage is loud anyway. The DB-failure hook emits an error span, and the litellm-db-failure
alert in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml fires on it. That alert, not
a completion failure, is the signal. Keep the tile on the trace source, filtered on all three of
ServiceName = 'litellm', StatusCode = 'Error' and
SpanAttributes['db.system.name'] = 'postgresql', on a strict > 0. Keep
service_callback: ["otel"] and allow_requests_on_db_unavailable together in values.yaml, so a
later change cannot buy the signal by removing the degrade. Nothing checks either.
The Admin UI login fails while completions succeed. The SSO path requires a database connection and returns 403 during a full outage. Read a failing login beside healthy completions as confirmation of the database outage, not as a second fault in the UI or in Keycloak.
Key rotation and key revocation are both unavailable during an outage. /key/delete and
/key/generate are database writes. A rotation started during a masked outage can delete the old
key and then fail to mint its replacement. See the first caveat under
Rotating a virtual key. Wait for the database before you rotate or
revoke anything.
The probes do not check the database
Section titled “The probes do not check the database”The chart default points the readiness and startup probes at /health/readiness. That path returns
HTTP 503 when the database is disconnected. With the default, a Postgres outage makes every replica
NotReady within about 30 seconds, the Service has no backends, and every lane stops at the ingress,
even though the proxy can serve from cached auth.
argocd/app-configs/litellm-chart/values.yaml therefore sets both probes to /health/liveliness:
| Probe | Path |
|---|---|
readinessProbe |
/health/liveliness |
startupProbe |
/health/liveliness |
/health/liveliness returns 200 while the process runs. A pod that starts during an outage also
becomes ready. The litellm-db-failure alert, not the probes, reports the outage.
LITELLM_DISABLE_ACCESS_LOG_PATHS in extraEnvVars names /health/liveliness, so the probe calls
do not appear in the access log. The proxy still answers them. The status code is the evidence, not
a log line.
Do not point the probes back at /health/readiness, and do not set
allow_requests_on_db_unavailable to false. Either change makes Postgres a hard dependency for
every completion.
To see the database state of one pod during an outage, call GET /health/readiness on the pod
with kubectl port-forward. A call through the Service or the ingress reaches a random replica.
Blast radius: CNPG main is shared
Section titled “Blast radius: CNPG main is shared”argocd/app-configs/cnpg/postgres-cluster.yaml declares cluster main in namespace postgres
with instances: 3, and it serves the keycloak, litellm, mealie, miniflux, octopus and
temporal databases. A Postgres outage here is an estate event, and keycloak sharing the cluster
puts SSO in its blast radius too. Never degrade the cluster to test a tenant’s behaviour.
If you are reading this during a real outage
Section titled “If you are reading this during a real outage”Capture these before it ends. It takes about two minutes:
kubectl port-forward -n litellm pod/<one pod> 4000:4000, a pod, not the Service, and record which pod.GET /health/readinesson it. Record the body and the status code.GET /health/livelinesson it. Expected 200.- A cheap completion with a key that the pod did not see in the last 60 seconds (cold). Record the
status code verbatim. 503 is the expected result. 401 is a blocking finding: a caller with a
valid credential was told their credential is bad, which is what
allow_requests_on_db_unavailableexists to prevent. - The same completion with a key that the pod saw inside the last 60 seconds (warm). Expected 200. Without this pair, a blanket 503 would satisfy the cold-key criterion while proving the degrade broken.
- Whether
litellm-db-failurefired, and how long after the first error span.
Mint, rotate and delete nothing while capturing. Never induce this outage by kubectl apply.
MCP session semantics across restarts and replicas
Section titled “MCP session semantics across restarts and replicas”The symptom of a lost MCP session is a missing notification, not an error message. There is nothing to grep for.
The four things an operator needs
Section titled “The four things an operator needs”-
Session state is process-local, so it is lost on restart and unknown to every other replica. Every MCP session lives in per-process dictionaries on the pod that issued it. Nothing is in Postgres and nothing is in Valkey.
replicaCountinvalues.yamlsets the replica count, and there is no session affinity. With more than one replica, a request that carries anmcp-session-idlands on a replica that does not know it part of the time, which is the same downgrade a restart produces. -
An unknown session id is not an error. It is a silent downgrade to stateless service. LiteLLM strips an unrecognized
mcp-session-idfrom the request and serves it statelessly. There is no400 session ID not foundto look for. Any runbook, alert or dashboard that watches for one is watching for a response this proxy does not emit. -
Plain
tools/listandtools/callkeep working. Session continuity stops. What breaks is anything that requires the server to speak first or to resume a conversation: server-initiated notifications, sampling, and elicitation. Those never arrive. -
Two conditions produce errors, and they are the only two:
Condition Response Meaning A live stateful session driven by someone other than its creator 403Sessions are owner-bound. A mismatch is refused, not silently served A caller holding too many concurrent stateful sessions 429A per-caller cap on held stateful sessions. Not a rate limit on requests, so it clears as sessions are released
Concurrency within one session is serialized. Two concurrent tools/call on the same session
complete one after the other. A client that fans out on one session gets correctness, not
throughput. If it wants throughput, it needs separate sessions.
What this means during a rollout
Section titled “What this means during a rollout”Nothing needs draining, and nothing needs a maintenance window for MCP’s sake. A rolling restart
drops every session. Every client that only does tools/list and tools/call continues without
noticing, and the clients this estate runs are all in that class. If a future client depends on
notifications, sampling or elicitation, re-establish that client after a rollout, and detect the
failure at the client, by noticing that the expected message never arrived.
If you observe a real MCP session failure, capture:
- Which replica served the request (
kubectl get pod -n litellm -o wide). - The full request headers,
mcp-session-idincluded, and the status code. A400here contradicts point 2 above and is a blocking finding. - Whether a
403or429was involved. Those are the designed errors. - What the client expected to receive and did not. For a lost session this is the only signal.
Supply chain
Section titled “Supply chain”Adoption delay
Section titled “Adoption delay”Every LiteLLM version bump reaches this cluster through a Renovate pull request that waits two
days and requires a named human reviewer. LiteLLM publishes several prereleases a week, and a fix
this cluster needs often ships only in the newest one. A longer floor kept the cluster behind those
fixes. The reviewer is the main control, and the two days give time for a bad release to be pulled
before the pull request opens. The rule is the last element of packageRules in
.github/renovate.json, group slug litellm-data-plane. It is last because Renovate applies
packageRules in array order with later matches overriding earlier ones, and three general groups
(container images, helm charts and argocd manifests) automerge these files on shorter floors. Keep
the litellm-data-plane rule the last element of packageRules.
Two files carry a LiteLLM version that a Renovate manager reports:
| Pin | File | Manager that sees it |
|---|---|---|
| Image tag plus digest | argocd/app-configs/litellm-chart/values.yaml |
helm-values |
| OCI chart version | argocd/app-configs/litellm-chart/kustomization.yaml |
kustomize |
argocd/app-configs/litellm/** carries no version pin. The rule matches it as forward coverage,
so that a pin that appears there is already covered. Do not describe that tree as guarded.
The rule matches by file name, never by dependency name. A name-based rule can cover nothing and emit no signal that it covers nothing.
Both automerge keys and an explicit reviewer list are required. The root configuration sets
automerge: true and platformAutomerge: true. Disabling one key leaves the other in force, and
omitting reviewers leaves nobody named on the pull request the delay exists to put a human in
front of.
A vulnerability-remediation pull request bypasses the two-day floor, because Renovate merges
config.vulnerabilityAlerts last, over the resolved package configuration. It does not merge
itself: .vulnerabilityAlerts.automerge is false repo-wide, so the pull request opens at once and
waits for the named reviewer. A supply-chain compromise published as an ordinary release is not a
vulnerability alert. It travels the ordinary rule path, where the floor and the reviewer apply. The
remaining exposure is a fix version that is itself compromised, which arrives with no release-age
floor. See Renovate for why a narrower, per-dependency control is not
available in repository configuration.
The model cost map comes from the image
Section titled “The model cost map comes from the image”The model cost map is the LiteLLM file that lists each model, its price per token and the
parameters that it accepts. LITELLM_LOCAL_MODEL_COST_MAP is True in
argocd/app-configs/litellm-chart/values.yaml, so the proxy reads the copy bundled in the image.
The scheduled reload, which the Admin UI configures, reads the same bundled file. The map changes
only when the image pin changes, through the pull request that Adoption delay describes.
Without the flag, the proxy downloads the map from the main branch of BerriAI/litellm at
startup and on each reload. An upstream edit to that file then changes the proxy with no pull
request in this repository, and a restart does not undo it. Issue #2918 is an example. Upstream
removed each openrouter/ row. Up to v1.105.0-rc.3 at least, LiteLLM accepts reasoning_effort
for an OpenRouter model only when the row of that model has supports_reasoning: true. The
octopus-review and octopus-utility lanes send reasoning_effort, so each Octopus review failed
with UnsupportedParamsError.
A model that is newer than the image has no row in the bundled map. A request with
reasoning_effort to that model fails in the same way.
Before each version bump, run this check on a host with Docker. Put the new digest in place of
<digest>. If the Octopus lanes use a different model, put it in place of
deepseek/deepseek-v4.1-flash.
docker run --rm --platform linux/arm64 -e LITELLM_LOCAL_MODEL_COST_MAP=True \ --entrypoint /app/.venv/bin/python ghcr.io/berriai/litellm@<digest> -c 'from litellm.utils import get_optional_paramsfor effort in ("low", "none"): params = get_optional_params(model="deepseek/deepseek-v4.1-flash", custom_llm_provider="openrouter", reasoning_effort=effort) print(effort, params["reasoning_effort"])'The command prints low low and none none. If it stops with UnsupportedParamsError, do not
merge the bump. The bump breaks each Octopus review. A version that includes BerriAI/litellm#45640
accepts the reasoning parameters for each OpenRouter model, with or without a row.
Security advisory subscription
Section titled “Security advisory subscription”The GitHub security-advisory watch on BerriAI/litellm is a setting on the operator’s GitHub
account. Nothing in this repository asserts it or checks it. It can be switched off, or the account
can lose access, and nothing here reports that.
The advisory watch and the Renovate reviewer are different accounts. The watch is on the
operator’s personal account, and the Renovate rule assigns reviewers: ["fzymgc"]. A security
advisory for LiteLLM arrives at one inbox, and the review request on a LiteLLM Renovate pull
request arrives at the other.