Skip to content

LiteLLM

Operational guide for the LiteLLM proxy at llm.fzymgc.house: what exists, where each part is defined, and the procedures that touch its credentials and keys.

LiteLLM is the LLM data plane. It is an OpenAI-compatible proxy in the litellm namespace, fronted by a Traefik IngressRoute that publishes API paths only. It keeps virtual keys, spend and budget state in a CloudNativePG database (main, in the postgres namespace) and shares its auth cache through a dedicated Valkey cluster in its own namespace.

People and off-cluster clients use https://llm.fzymgc.house. In-cluster workloads (octopus, karakeep, mealie and life-agent) call the Service directly. The litellm-proxy-ingress CiliumNetworkPolicy in argocd/app-configs/litellm/networkpolicy.yaml must admit each such workload pod on port 4000. A direct caller skips the route exclusions in the IngressRoute. The allowed_routes value on its virtual key still limits it to the LLM routes.

Property Value
Public host https://llm.fzymgc.house
In-cluster Service http://litellm.litellm.svc.cluster.local:4000
Namespace litellm
Image and chart argocd/app-configs/litellm-chart/values.yaml (image tag and digest) and kustomization.yaml (chart version)
Database CNPG cluster main, database litellm, reached at main-rw.postgres.svc.cluster.local:5432
Database credential secret/fzymgc-house/cluster/postgres/users/main-litellm (Vault)
Master key master_key on secret/fzymgc-house/cluster/litellm-keys/master (Vault)
Provider and MCP upstream credentials secret/fzymgc-house/cluster/litellm (Vault)
Virtual key values secret/fzymgc-house/cluster/litellm-keys/<consumer> (Vault), see Which tool owns each key
Virtual key scopes tf/litellm/main.tf, applied by the HCP Terraform workspace main-cluster-litellm
Manifests argocd/app-configs/litellm/ (secrets, ingress, network policy, Valkey) and argocd/app-configs/litellm-chart/ (the chart values)
Seed script scripts/seed-litellm-vault.sh (operator-run)
Auth cache the litellm-valkey ValkeyCluster in the litellm namespace, password on secret/fzymgc-house/cluster/litellm-valkey
Telemetry OTLP to the ClickStack collector, service.name=litellm

The lanes a client can name are the entries of proxy_config.model_list in values.yaml. The MCP servers behind the proxy are the entries of proxy_config.mcp_servers in the same file. The services catalog lists both sets, and the MCP client runbook covers the client side of the MCP routes.

Database credential and the single-source contract

Section titled “Database credential and the single-source contract”

The password lives in exactly one place: the password property at secret/fzymgc-house/cluster/postgres/users/main-litellm in Vault. Everything else that carries it is a derivation:

Object What it is
postgres/main-litellm-credentials (Kubernetes Secret) CNPG’s own copy, materialized from the Vault property by External Secrets
litellm/litellm-secrets (Kubernetes Secret) the proxy’s DATABASE_URL, composed from the same Vault property

The connection string is composed in the cluster and exists nowhere else. The ExternalSecret at argocd/app-configs/litellm/secrets.yaml reads the username and the password as two separate properties and assembles DATABASE_URL in its spec.target.template.data. The assembled string is never written to Vault, never committed to git, and never present on the operator’s workstation.

The escaping is correct because of the generator, not because of the escaper. The template pipes the password through urlquery, which is the Go text/template query escaper. It renders a space as a plus sign. Inside a URL userinfo component a plus is a literal plus, so a password that contains a space produces a wrong credential with no parse error anywhere. The only symptom is an authentication failure at the database.

scripts/seed-litellm-vault.sh generates from an alphabet that contains no space and no non-ASCII character. Nothing checks the coupling between that alphabet and this escaping. Three constraints keep it correct:

  1. The password comes from the generator. Do not type one by hand.
  2. Do not widen the alphabet without a check that every added character survives urlquery inside a URL userinfo component.
  3. Keep urlquery on the password and not on the username in the DSN template.

Changing the database credential in the required order

Section titled “Changing the database credential in the required order”

A change to the shape of this credential is two steps across a merge boundary. The Vault write is out of band from ArgoCD, so it cannot be folded into the same change as the manifest.

  1. Merge the manifest change and let ArgoCD sync it, so nothing in the cluster still references the property that is removed.

  2. Make sure that the rendered Secret is correct before you delete anything:

    Terminal window
    kubectl --context fzymgc-house get externalsecret -n litellm

    litellm-secrets must read SecretSynced. Then make sure that the rendered DATABASE_URL still authenticates. A proxy pod that serves requests and a migration Job that completed are the evidence.

  3. Only then remove the retired property. Write back a version that names only the properties that remain. Piping the current version through jq carries the live values across unchanged and keeps them out of argv and shell history:

    Terminal window
    vault kv get -format=json secret/fzymgc-house/cluster/litellm \
    | jq '.data.data | del(.retired_property)' \
    | vault kv put secret/fzymgc-house/cluster/litellm -

    vault kv put writes a new version that contains exactly the keys given, so the write itself removes the retired property. vault kv patch merges into the current version and leaves the property in place. It is the wrong verb here.

Do not run scripts/seed-litellm-vault.sh with no flag to do this. The bare path generates a fresh CNPG password, a fresh master key, a fresh Valkey password and a fresh value for every workload virtual key, and writes all of them. On a live estate that is a full rotation, not a cleanup. If you do run the bare path, treat it as a full-estate rotation: restart the proxy, re-export LITELLM_MASTER_KEY, and re-register every workload key.

What happens if you do it in the reverse order

Section titled “What happens if you do it in the reverse order”

If you remove the Vault property while an ExternalSecret still references it, the Kubernetes Secret is not deleted. External Secrets deletes a target only when the provider returns no data at all. A missing property inside an existing secret is a fetch error, not an empty data map. So:

  • The ExternalSecret goes to SecretSyncedError and stops refreshing.
  • The last-synced Secret persists, so running pods keep working and there is no outage to notice.
  • Changes to the master key or the provider keys stop propagating.
  • Any event that recreates the Secret leaves it with no source at all.

The warning sign is kubectl get externalsecret -n litellm showing anything other than SecretSynced.

vault kv put does not erase history. A property removed by a new version remains readable in the prior versions of that path. To purge it, destroy the old versions:

Terminal window
vault kv destroy -versions=<n>,<n> secret/fzymgc-house/cluster/litellm

Do not run vault kv metadata delete on this path. It removes every version, the current one included, so it destroys the live openrouter_api_key. The seed script takes that value from the environment, and nothing in this repository can reproduce it. If you are not holding it, it has to be re-issued at OpenRouter. If you do run vault kv metadata delete, re-seed at once. Until you do, the ExternalSecret behind litellm/litellm-secrets has no source.

The only part of this rotation that this repository owns is the urlquery escaping, and it holds by construction as long as the password comes from the generator. The rest is CNPG applying a changed passwordSecret, LiteLLM reconnecting, and virtual-key rows surviving the reconnect. Those are third-party behaviour, and nothing in this repository asserts them.

Rotate when you have a reason: a compromise, a departure, or a policy clock.

  1. Generate the new password and write it to the single source. The generator is the same one in scripts/seed-litellm-vault.sh under “Generate the CNPG password”. Keep the two alphabets identical. Nothing checks that they agree. The block is unindented because the PY terminator must start at column zero:

    Terminal window
    NEW_PGPASS="$(python3 - <<'PY'
    import secrets, string
    alphabet = string.ascii_letters + string.digits
    specials = "@:/?#[]!$&'()*+,;=%"
    pw = [secrets.choice(specials) for _ in range(3)] + [secrets.choice(alphabet) for _ in range(29)]
    secrets.SystemRandom().shuffle(pw)
    print("".join(pw))
    PY
    )"
    jq -n --arg username litellm --arg password "$NEW_PGPASS" \
    '{username: $username, password: $password}' \
    | vault kv put secret/fzymgc-house/cluster/postgres/users/main-litellm -
    unset NEW_PGPASS

    jq hands the value to vault on stdin, so the password reaches neither argv nor shell history. The path carries exactly username and password, and the jq -n reconstructs both. The recovery path from a failed rotation is another write with the previous password. vault kv get on the prior version supplies it as long as that version is not destroyed.

  2. CNPG applies the password to the role on its own. litellm is a managed role in argocd/app-configs/cnpg/postgres-cluster.yaml with passwordSecret: main-litellm-credentials. That Secret is materialized from the same Vault property by the main-litellm-credentials ExternalSecret in argocd/app-configs/cnpg/users-litellm.yaml, whose target carries the cnpg.io/reload: "true" label. There is no manual ALTER ROLE.

  3. Both ExternalSecrets refresh on their own clocks. main-litellm-credentials (namespace postgres) and litellm-secrets (namespace litellm) each carry refreshInterval: 5m. Either order has a transient window in which the proxy holds one password and the role holds the other. Expect authentication failures in the proxy log inside it. Make sure that both read SecretSynced:

    Terminal window
    kubectl --context fzymgc-house get externalsecret -n postgres main-litellm-credentials
    kubectl --context fzymgc-house get externalsecret -n litellm litellm-secrets
  4. The proxy restart is automatic. DATABASE_URL reaches the pod as a secretKeyRef env var, which is read at container start, so the Deployment carries reloader.stakater.com/auto: "true" in values.yaml and Stakater Reloader restarts it. The manual fallback:

    Terminal window
    kubectl --context fzymgc-house rollout restart deployment/litellm -n litellm
  5. Make sure that the proxy reconnected and that the database state survived. GET /health/readiness on a pod must report db: connected. A bare 200 from /health/liveliness is not the reconnect, because allow_requests_on_db_unavailable keeps completions flowing on cached auth (see When the database is unavailable). Then make sure that an existing workload key still authenticates, that GET /key/info reports the same models, aliases, limits and budgets, and that its spend total did not go backwards.

Make sure that the litellm-db-failure alert is not firing before you start. It is the signal if the reconnect does not happen.

Every workload that calls LiteLLM holds its own virtual key. This section is how one gets made, scoped, delivered and checked. Nine credentials exist: six in-cluster consumer workloads, one agent identity that reads its key from Vault at run time, and two credentials that the operator holds on a workstation.

A virtual key has two halves: the secret value and the scope. The scope is the allow-list, the aliases, the limits, the budgets and the metadata. Vault holds every value, and scripts/seed-litellm-vault.sh generates every value. The scope has two owners.

Consumer name key_alias Vault path and property Scope owner
octopus octopus cluster/litellm-keys/octopus api_key Terraform, tf/litellm/main.tf
karakeep karakeep cluster/litellm-keys/karakeep api_key Terraform, tf/litellm/main.tf
mealie mealie cluster/litellm-keys/mealie api_key Terraform, tf/litellm/main.tf
agent-memory engram cluster/litellm-keys/agent-memory api_key Terraform, tf/litellm/main.tf
claude-code claude-code cluster/litellm-keys/claude-code api_key Terraform, tf/litellm/main.tf
openrouter-passthrough openrouter-passthrough cluster/litellm-keys/openrouter-passthrough api_key Terraform, tf/litellm/main.tf
dns-agent-triage dns-agent-triage cluster/litellm-keys/dns-agent-triage api_key Terraform, tf/litellm/main.tf
life-agent life-agent cluster/litellm-keys/life-agent api_key Terraform, tf/litellm/main.tf
<consumer>-mcp <consumer>-mcp none An operator, by hand, per MCP client keys

All Vault paths in the table are below secret/fzymgc-house/. The master key is master_key on cluster/litellm-keys/master. CONSUMER_TABLE in the seed script is the single source of each consumer’s path and property pair. Keep each vault_path and vault_property in tf/litellm/main.tf equal to its CONSUMER_TABLE row. Nothing checks it.

Each Terraform-owned value lives alone, as the property api_key, on its own path under secret/fzymgc-house/cluster/litellm-keys/. The master key lives alone on cluster/litellm-keys/master. A path under that prefix holds exactly one credential.

The layout narrows the Vault access of the Terraform run. KV has no per-property access control, so a grant on a path is a grant on every property of that path. The run role tfc-litellm reads secret/data/fzymgc-house/cluster/litellm-keys/* and the OpenRouter management key at infrastructure/openrouter/terraform. It writes one path, cluster/litellm/openrouter-zdr (tf/vault/policy-litellm.tf). It cannot read the other credentials on a consumer’s path, or the provider keys on the bare path cluster/litellm.

The HCP Terraform workspace main-cluster-litellm applies tf/litellm. It reads each value from Vault as an ephemeral value and sends it to LiteLLM once, as the write-only argument key_wo. The value never enters the plan or the state. The state holds only the SHA-256 hash of the value, which is also the token that LiteLLM stores. tf/litellm/imports.tf adopts a key that exists in LiteLLM under a configured alias, and Terraform creates a key that does not exist from the value in Vault.

The MCP client keys stay out of Terraform because the provider cannot express their scope. Every MCP client key carries object_permission.mcp_servers, and the ncecere/litellm provider has no argument for it. A Terraform-owned key gets its MCP routes from a litellm_unified_access_group in tf/litellm/access_groups.tf, and the dns-agent-triage key is the one case today.

The Terraform-managed keys differ from a hand-minted key in two ways:

  • The provider cannot send key_type. It sends allowed_routes: ["llm_api_routes"], which is the restriction that key_type: "llm_api" sets. A key that Terraform creates reads key_type: "default", and it is restricted in the same way.
  • An in-place apply sends every argument the key has in its state. The metadata object replaces the live one as a whole. The module ignores changes to budget_duration and duration, so an imported key never receives them in an update. A key that Terraform created holds both in its state. Each in-place apply on that key restarts its 30-day budget window, and for a workstation key it also restarts the 90-day expiry.

The HCP agent pods must reach the proxy Service at http://litellm.litellm.svc.cluster.local:4000. The litellm-proxy-ingress policy admits the namespace hcp-terraform. If it does not, a plan stops at the litellm_keys data source with a timeout.

Each moved credential still has a copy at its original location, kept as a rollback:

Credential Original location Current location
master key cluster/litellm master_key cluster/litellm-keys/master master_key
octopus cluster/octopus litellm_api_key cluster/litellm-keys/octopus api_key
karakeep cluster/karakeep litellm_api_key cluster/litellm-keys/karakeep api_key
mealie cluster/mealie litellm_api_key cluster/litellm-keys/mealie api_key
agent-memory cluster/agent-memory litellm_api_key cluster/litellm-keys/agent-memory api_key
claude-code cluster/litellm claude_code_api_key cluster/litellm-keys/claude-code api_key
openrouter-passthrough cluster/litellm openrouter_passthrough_api_key cluster/litellm-keys/openrouter-passthrough api_key

To remove the copies, remove only the properties that moved, and leave every other property on those paths in place:

  1. Make sure that every ExternalSecret reads SecretSynced, and that no manifest, script or document still names an original location.

  2. For each consumer path, write a new version without litellm_api_key. Build the body from the current version, so the other values stay out of argv and shell history:

    Terminal window
    vault kv get -format=json secret/fzymgc-house/cluster/octopus \
    | jq '.data.data | del(.litellm_api_key)' \
    | vault kv put secret/fzymgc-house/cluster/octopus -

    Do the same for cluster/litellm, and delete master_key, claude_code_api_key and openrouter_passthrough_api_key in one write. Read the result back with vault kv get -format=json <path> | jq '.data.data | keys', which prints names only.

  3. The removed values are still readable in the prior versions of each path. To purge them, use the version-scoped vault kv destroy -versions=<list> <path>, and name only versions older than the current one. Do not use vault kv metadata delete. See Purging the history is a separate step.

Terminal window
scripts/seed-litellm-vault.sh --all-consumers

This command owns seeding. It writes each of the nine values into Vault. Every row is Terraform-owned, so it registers nothing. The next main-cluster-litellm apply registers them. The command touches no core credential. It is safe to re-run: a consumer that is seeded and authenticating is skipped, and a scripted consumer whose value was seeded but never registered is repaired without a new Vault version. A Terraform-owned consumer whose value is in Vault but not live reports the verdict terraform. Read completion off the per-consumer result table, not off the exit code.

Two preconditions, both checkable before you run it:

  1. The lanes are live. GET /v1/models must list every lane in model_list. A key registered against a lane that does not exist is a 403 at the consumer’s first call.
  2. The proxy answers. GET /health/liveliness returns 200. Do not read a 401 from GET /health as a broken proxy. That route is authenticated, and with the master key attached it answers 200. The command performs this check itself and exits without a write if the proxy does not answer.

The bare invocation is not the seeding step. scripts/seed-litellm-vault.sh with no flag rotates the CNPG password, the master key and the Valkey password alongside every workload virtual key. On a live estate that is an outage. --consumer <name> seeds one value without registering it, for a single repair.

Vault owns the value. LiteLLM’s Postgres owns the scope. key is a first-class field on /key/generate: it must start with sk- and be at least 16 characters, or the call returns 400.

Fact about a key Authoritative store Changed by
the secret value itself Vault, on the consumer’s own path scripts/seed-litellm-vault.sh --all-consumers (all nine) or --consumer <name> (one). On a live key, follow Rotating a virtual key. Not /key/regenerate, which is Enterprise-gated on this estate and returns HTTP 500
models allow-list, aliases, metadata LiteLLM Postgres For the nine Terraform-owned keys, a merged change to tf/litellm/main.tf. For the MCP client keys, /key/generate and /key/update
budgets, rate limits LiteLLM Postgres The same owner as the allow-list
spend LiteLLM Postgres normal traffic
delivery into the pod Vault to ExternalSecret to Kubernetes Secret that consumer’s ExternalSecret

After Terraform owns a key, change its scope only through a merged change to tf/litellm/main.tf. Do not change it with /key/update, the Admin UI or the seed script. The next apply sends the configured scope again and reverts the change. A change to metadata outside Terraform is worse: the plan does not show it, and the next in-place update of that key deletes it, because the provider sends the configured metadata object as a whole.

Seeding the value first keeps the two stores from drifting. The value in Vault is the value, and registration only attaches scope to it.

This is the mechanism that --all-consumers performs for the scripted key. Follow it by hand when you repair exactly one scripted credential.

For a Terraform-owned key, the procedure is shorter. If the key does not exist in LiteLLM, run step 1, then let the next main-cluster-litellm apply create the key from the value in Vault. If the key exists, follow Rotating a virtual key. Do not register a Terraform-owned value with /key/generate. The next plan imports that key and updates it to the configured scope, so the scope you sent lasts only until that apply.

  1. Seed the value in Vault:

    Terminal window
    scripts/seed-litellm-vault.sh --consumer <name>

    --consumer writes exactly one property with vault kv patch -method=rw, touches no core credential and no other consumer, and refuses to overwrite a property that exists unless you pass --rotate.

  2. Read the seeded value into a shell variable, never onto the screen:

    Terminal window
    VAULT_ADDR=https://vault.fzymgc.house
    SEEDED_KEY="$(vault kv get -field=<property> secret/fzymgc-house/<path>)" # both from the row
  3. Register it with /key/generate, with the seeded value as key and the scope from the JSON body of the row.

  4. Verify, per Verifying a minted key.

  5. Only then edit the consumer’s manifests, per the next section.

The Vault property must exist before any manifest references it. An ExternalSecret that points at a property that is not there goes SecretSyncedError and stops refreshing, while the running pods keep working on their old Secret. That is a stale-configuration failure with no outage to notice.

  1. Seed the Vault property (--consumer <name>) and register the key. For a Terraform-owned key, the next main-cluster-litellm apply registers it.
  2. Add the new Secret key alongside the existing one in that consumer’s ExternalSecret. Do not replace it.
  3. Wait one refresh interval (refreshInterval: 5m) and make sure that the live Secret carries both keys.
  4. Only then flip the workload’s secretKeyRef.key and its base URL together, in one change.

This prevents a workload from restarting onto a credential that the plane it still calls does not accept. The old Secret key stays populated for the whole bake, so a rollback is a key-name flip against a Secret that already holds both values.

The allow-list is checked before the key’s aliases resolve

Section titled “The allow-list is checked before the key’s aliases resolve”

Every key’s models list must contain both the string the client sends and the lane name. LiteLLM runs the access check against the model taken from the request, and the per-key alias rewrite happens later. Per-key aliases are not consulted by the access check at all.

If you list only the lane name, the result is a 403 that names a model the client never sent. That symptom reads like a routing bug and sends you to model_list, which is not where the fault is.

Removing the aliases of a Terraform-owned key

Section titled “Removing the aliases of a Terraform-owned key”

Provider ncecere/litellm 2.1.0 sends aliases in /key/update only when the map has entries. Thus a Terraform apply cannot remove the aliases of a key. With aliases = null, the apply works, and LiteLLM keeps the aliases. With aliases = {}, LiteLLM keeps the aliases, and the apply fails with an inconsistent result. LiteLLM replaces aliases on /key/update. It does not merge them as it merges metadata. To remove the aliases of a key:

  1. Set aliases = null for the key in tf/litellm/main.tf, and remove the alias names from its models list. Merge the change, and let the main-cluster-litellm workspace apply it.
  2. In a litellm pod, use the LITELLM_MASTER_KEY of the pod to find the token of the key with GET /key/list?key_alias=<alias>&return_full_object=true on http://127.0.0.1:4000. Do not print the master key.
  3. In the same pod, send POST /key/update with the body {"key": "<token>", "aliases": {}}.
  4. Start a refresh-only run of main-cluster-litellm. The workspace applies it automatically, and the state then has aliases = {} for the key.
  5. Start a plan-only run, and make sure that it shows no change and no drift for the key.

soft_budget is the signal and max_budget is a blast-radius stop, not a cost control. Spend is flushed to the database on a delay of several seconds, so enforcement is a bound with overshoot, not a ceiling. A max_budget close to expected spend is a fuzzy limit and, on a bursty consumer, an availability risk. Put soft_budget near expected spend and max_budget well above it.

Per-model budgets inside a key (model_max_budget) are Enterprise-gated, which is why budgets land per key rather than per lane.

Embedding lanes record a cost when they declare input_cost_per_token in model_list. A failed call records $0. A lane that is absent from the LiteLLM cost map and declares no input_cost_per_token records $0.

Spend telemetry comes from spans, not from the cost metric. LiteLLM exports gen_ai.usage.cost only as a cumulative histogram, and a sum over a histogram’s Sum column adds running totals together. So the spend tiles and the spend alert on the LiteLLM — LLM & key usage dashboard read the Traces source. Each billed request writes its cost once, as SpanAttributes['litellm.cost.total'] on one CLIENT span named <operation> <lane>, for example chat engram-summarize. No other span in the trace repeats it, so a sum over spans does not double count.

Tile Select Where Group by
litellm-key-spend sum(toFloat64OrZero(SpanAttributes['litellm.cost.total'])) ServiceName = 'litellm' AND mapContains(SpanAttributes, 'litellm.cost.total') SpanAttributes['litellm.metadata.user_api_key_alias']
litellm-lane-cost the same the same SpanAttributes['gen_ai.request.model'], which is the lane name on a span

The alert LiteLLM key spend runaway in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml fires when one key alias reaches 3 USD in one hour. This query gives the hourly figures:

SELECT SpanAttributes['litellm.metadata.user_api_key_alias'] AS alias,
toStartOfHour(Timestamp) AS hour,
sum(toFloat64OrZero(SpanAttributes['litellm.cost.total'])) AS usd
FROM default.otel_traces
WHERE Timestamp > now() - INTERVAL 14 DAY AND ServiceName = 'litellm'
AND mapContains(SpanAttributes, 'litellm.cost.total')
GROUP BY alias, hour ORDER BY usd DESC LIMIT 20

For a Terraform-owned key, change soft_budget and max_budget in tf/litellm/main.tf. The provider sends both in one /key/update and reads soft_budget back from litellm_budget_table. If LiteLLM discards the value, the apply fails with an inconsistent-result error, so the failure is not silent. In that case, move the budget row by id. /budget/ is not published by the IngressRoute, so this is an in-cluster hop:

Terminal window
kubectl -n litellm port-forward svc/litellm 18400:4000 &
curl -sS -X POST http://127.0.0.1:18400/budget/update \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
-d '{"budget_id":"<.info.budget_id from /key/info>","soft_budget":50.0}'

For a scripted key, POST /key/update moves max_budget, which lives on the key row. Read soft_budget back from /key/info at .info.litellm_budget_table.soft_budget, never from the update response. If the budget row did not move, use /budget/update as above.

The nine Terraform-owned keys carry these fields as arguments in tf/litellm/main.tf, with the exceptions this table names. key becomes the write-only key_wo.

Field Why it is not optional
key the value seeded in Vault. Omit it and LiteLLM mints its own, and the two stores diverge at once
key_alias the only human-readable per-key label on the spend metrics. Without it every dashboard groups by an opaque hash
key_type: "llm_api" a data-plane key. /key/, /team/ and /user/ are published at the ingress, so a leaked workload credential without this can mint more keys. A Terraform-owned key sends allowed_routes: ["llm_api_routes"] instead, which is the restriction this field sets
models both strings per lane, per the section above
aliases maps what the client sends to its lane, so most consumers need only a base-URL change
rpm_limit, tpm_limit a runaway loop is bounded in requests before it is bounded in dollars
soft_budget, max_budget, budget_duration per the doctrine above
duration TTL. null on every workload key

Workload keys carry no TTL. A duration on a key held by a pod is a scheduled outage: nothing in this estate re-mints a key on expiry, so the workload starts failing authentication at a time nobody chose. The two workstation credentials carry one, because a human is present to re-mint them and an abandoned workstation credential must stop working.

The scope is the octopus entry in tf/litellm/main.tf. The octopus-app ExternalSecret in argocd/app-configs/octopus/secrets.yaml reads the value.

octopus’s soft_budget and max_budget pair is sized from its own demand and departs from the ratio the other rows carry. Do not normalize the pair onto the other rows.

Most of octopus’s model strings are not in its Deployment. Its Deployment carries one model env var, OCTOPUS_UTILITY_MODEL, for the utility lane. The review and embedding models are rows in the available_models table of its Postgres database:

Terminal window
P=$(kubectl --context fzymgc-house -n postgres get cluster main -o jsonpath='{.status.currentPrimary}')
kubectl --context fzymgc-house exec -n postgres "$P" -c postgres -- \
psql -d octopus -c 'SELECT "modelId", category, "isActive" FROM available_models;'

ollama/bge-m3 is the platform-default embedding row. LiteLLM matches the client-sent string, so every active row in that table must appear in models. Re-derive the list from the active rows of that table, never from the Deployment.

The key holds these names and aliases:

models = [
"ollama/bge-m3", "bge-m3", "octopus-embed", "text-embedding-3-large",
"octopus-review", "octopus-utility",
local.decisions_model,
]
aliases = {
"ollama/bge-m3" = "octopus-embed"
"bge-m3" = "octopus-embed"
"text-embedding-3-large" = "octopus-embed"
}

LiteLLM refuses a request of the key for any other model with HTTP 403 and the error key_model_access_denied.

The chat rows octopus-review and octopus-utility are the names of LiteLLM lanes in argocd/app-configs/litellm-chart/values.yaml. When octopus sends one of these names, LiteLLM routes the call to the lane of the same name, so the key holds no alias for them. The lane sets the model, the reasoning effort and the provider rule.

The lane octopus-review also names providers in provider.order. OpenRouter tries those providers first, in that order. It uses the other ZDR providers only when the named providers fail. Without the order, OpenRouter prefers the provider with the lowest price, and the cheapest ZDR providers of the review model are slow. Octopus drops a review call after 300 seconds.

Each provider in the order must meet two conditions:

  • It has a ZDR endpoint for the model of the lane. This command lists those providers. Set the model id to the model of the lane:

    Terminal window
    curl -s https://openrouter.ai/api/v1/endpoints/zdr \
    | jq -r '.data[] | select(.model_id == "deepseek/deepseek-v4.1-flash") | .provider_name' \
    | sort -u
  • It produces the output budget of OCTOPUS_REVIEW_MAX_TOKENS in under 300 seconds. With a margin of 30 seconds, a budget of 49152 needs 183 output tokens per second on a large review.

To see which provider serves the lane, read the provider field of a reply:

Terminal window
export VAULT_ADDR=https://vault.fzymgc.house
KEY="$(vault kv get -field=api_key secret/fzymgc-house/cluster/litellm-keys/octopus)"
curl -s https://llm.fzymgc.house/v1/chat/completions \
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-d '{"model":"octopus-review","max_completion_tokens":2048,
"messages":[{"role":"user","content":"Reply with the word OK."}]}' \
| jq '{provider, finish: .choices[0].finish_reason}'

The reply must name the first provider of the order. If it names a provider outside the order, the named providers failed for that call. If that repeats, review calls run at the speed of the fallback provider, and the long ones time out.

text-embedding-3-large is not in that table either, and octopus sends it. It is octopus’s hard-coded OpenAI default on a code path that never consults available_models. Its alias must point at octopus-embed: that lane is bge-m3 at 1024 dimensions, which matches EMBEDDING_DIM=1024 and the code_chunks collection. A real 3072-dimension model would write vectors the collection cannot hold. Make sure with jq '.data[0].embedding | length', which must return 1024.

A consumer’s model strings can come from a configuration table and from hard-coded defaults. Enumerate from both.

The scope is the mealie entry in tf/litellm/main.tf. The mealie-app-secrets ExternalSecret in argocd/app-configs/mealie/secrets.yaml reads the value. mealie calls the LiteLLM Service directly.

mealie and karakeep have no dedicated model_list entry. The openrouter/* wildcard serves them. Their keys name the resolved model, not the wildcard. Listing openrouter/* in models would grant the whole OpenRouter catalogue. Naming the resolved model keeps the grant to one model while the access check still passes on both the client-sent string and the resolved one.

The scope is the karakeep entry in tf/litellm/main.tf. The karakeep-secrets ExternalSecret in argocd/app-configs/karakeep/secrets.yaml reads the value. karakeep calls the LiteLLM Service directly, and the same wildcard reasoning as mealie applies.

karakeep sends two model strings, so it needs two pairs. INFERENCE_IMAGE_MODEL and INFERENCE_TEXT_MODEL in argocd/app-configs/karakeep/deployment.yaml name them, and both auto-tagging and auto-summarization are on. To check the text pair, POST /v1/chat/completions with karakeep’s own key and {"model": "or-deepseek-v4-pro"} must return 200, not 403.

The scope is the agent-memory entry in tf/litellm/main.tf. Three lanes: the summarize lane and both embed lanes. The memory-mcp-litellm ExternalSecret in argocd/app-configs/agent-memory/external-secret.yaml reads the value into the Secret key litellm_api_key.

Two consumers can send the same client string and reach different lanes, because each key aliases the string on its own. One model_list entry per lane, resolved per key, is what attributes spend to the consumer rather than to a shared upstream. Do not merge two consumers’ lanes into one.

gemini-embed is listed but not aliased, so the reindex and gap-fill Jobs can name the general lane directly while ordinary traffic on gemini-embedding-2 resolves to engram’s own lane.

The two Gemini lanes read GEMINI_LANE_API_KEY. The litellm-secrets ExternalSecret builds it from the Vault property gemini_api_key at secret/fzymgc-house/cluster/litellm. If the env entry is missing, os.environ/GEMINI_LANE_API_KEY resolves to None without an error, and both lanes fail at request time while every other lane keeps working.

The scope is the life-agent entry in tf/litellm/main.tf. The life-agent-gateway ExternalSecret in argocd/app-configs/life-agent-gateway/gateway-secret.yaml reads the value from fzymgc-house/cluster/litellm-keys/life-agent, property api_key.

The Hermes configuration of the agent (config/config.yaml in seanb4t/life-agent) names the models on the ZDR wildcard, and the key permits these names:

Name Use
openrouter-zdr/anthropic/claude-sonnet-5.5 chat and cron. Hermes sends reasoning_effort: high
openrouter-zdr/anthropic/claude-haiku-5.5 the Hermes auxiliary tasks
openrouter-zdr/anthropic/claude-opus-5.5 a switch with /model

The key needs no lane and no alias for these names. To change a model, change config.yaml in seanb4t/life-agent and the models list of the key.

No ZDR endpoint of Claude Haiku 5.5 takes temperature, and the Hermes vision, approval and title tasks send it. With require_parameters: true, OpenRouter then answers 404. The bundled cost map also has no row for the model. An entry with the exact name openrouter-zdr/anthropic/claude-haiku-5.5 in argocd/app-configs/litellm-chart/values.yaml drops temperature and declares the price. LiteLLM resolves the wildcard only for a name that no entry names, so the entry wins for this model.

A workstation credential with no consumer manifest. It lives alone on secret/fzymgc-house/cluster/litellm-keys/claude-code, property api_key. Read it into a shell variable, never onto the screen:

Terminal window
LITELLM_API_KEY="$(vault kv get -field=api_key secret/fzymgc-house/cluster/litellm-keys/claude-code)"

The scope is the claude-code entry in tf/litellm/main.tf. No alias map. Workstation clients send native OpenRouter slugs through the openrouter/* wildcard. If a curated name is wanted, add it to this key’s aliases and list both strings in models.

The 90-day TTL is deliberate. Terraform sends duration: "90d" when it creates or replaces this key, and each in-place apply moves the expiry to 90 days after that apply.

The generic passthrough key. It lives alone on secret/fzymgc-house/cluster/litellm-keys/openrouter-passthrough, property api_key:

Terminal window
OPENROUTER_PASSTHROUGH_KEY="$(vault kv get -field=api_key \
secret/fzymgc-house/cluster/litellm-keys/openrouter-passthrough)"

The scope is the openrouter-passthrough entry in tf/litellm/main.tf. It is scoped to two wildcards, openrouter/* and openrouter-zdr/*, never left empty. An empty models list means unrestricted: every lane the proxy serves, every consumer’s ZDR lane included.

Jev is a TypeSafe model that returns typed decisions: a choice, a score, or a yes or no probability. OpenRouter serves it as typesafe/jev-1.13 at /api/alpha/decisions only. It rejects the model on /chat/completions, so no OpenRouter model_list entry can reach it. The proxy exposes a pass-through route for it, which the octopus key uses. engram uses the native lane in the subsection “The native lane to TypeSafe” below.

Item Value
Client URL POST https://llm.fzymgc.house/openrouter/alpha/decisions
Upstream https://openrouter.ai/api/alpha/decisions, exact, with include_subpath: false
Credential the proxy’s OPENROUTER_LANE_API_KEY. The client sends only its own virtual key
Configuration general_settings.pass_through_endpoints in argocd/app-configs/litellm-chart/values.yaml
Spend LiteLLM records $0. The real cost is in usage.cost in the response body

The route is deny by default, and a key needs two things to call it:

Control Refusal without it
The route grant in the key’s metadata HTTP 403 Key/team not allowed to access passthrough route /openrouter/alpha/decisions
typesafe/jev-1.13 in the key’s models list HTTP 403 key_model_access_denied, “The requested model ‘typesafe/jev-1.13’ is not available for this API key”

LiteLLM checks the model field of the request body against the key’s models list since v1.104.0-rc.1, which prices typesafe/jev-1.13 on this route. Before that release, the grant was the only control.

Only the octopus key holds the grant. Its entry in tf/litellm/main.tf carries it in metadata through local.decisions_route_metadata, so a replacement keeps it. The entry also lists local.decisions_model. The engram key has no grant, because it calls the native lane below. It still lists local.decisions_model, because that is the lane name. The provider also has a top-level allowed_passthrough_routes argument. Do not use it. LiteLLM treats that request field as an Enterprise feature and refuses it on this proxy. The grant in metadata is not gated.

To grant a Terraform-owned key access to the route, set its metadata to local.decisions_route_metadata, add local.decisions_model to its models list in tf/litellm/main.tf, and merge the change.

To grant a scripted key access to the route:

  1. Read the current metadata of the key with GET /key/info?key=<key>.
  2. Add "allowed_passthrough_routes": ["/openrouter/alpha/decisions"] to that metadata.
  3. Send the full metadata with POST /key/update. The call replaces the metadata object, so keep every existing field.
  4. Add typesafe/jev-1.13 to the key’s models list with POST /key/update. Send the full list.
Terminal window
curl -sS -X POST "https://llm.fzymgc.house/key/update" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
--data-binary '{"key": "<key>", "metadata": {"allowed_passthrough_routes": ["/openrouter/alpha/decisions"]}}'

A request body looks like this. The model field is the OpenRouter slug:

{
"model": "typesafe/jev-1.13",
"state": "Help! My payouts have been failing for 3 days.",
"questions": {
"department": {
"type": "choice",
"instructions": "Which team should handle this?",
"criteria": {"billing": "Payments, refunds", "technical": "Bugs, outages", "sales": "Pricing"}
}
}
}

Do not set TYPESAFE_API_KEY or TYPESAFE_API_BASE on the proxy to reach OpenRouter. The built-in /typesafe/* route forwards every subpath to that base URL, and every llm_api key can call it without a grant. With the OpenRouter key behind it, every workload key can call any OpenRouter model outside its models list, and LiteLLM records $0 spend.

The proxy also serves Jev on the native decisions route POST https://llm.fzymgc.house/v1/systemone. engram calls this route. The lane typesafe/jev-1.13 in model_list forwards the call to the TypeSafe API as typesafe/jev-1.13.0, with no OpenRouter hop. The body is the same System One body as above, and the model field is the lane name. The route goes through the normal model pipeline, so the key allow-list, the budgets and the spend tracking apply.

Item Value
Credential the proxy’s TYPESAFE_LANE_API_KEY, from the Vault property typesafe_api_key at secret/fzymgc-house/cluster/litellm
Access a key that lists typesafe/jev-1.13 in its models list
Response answers, model and usage with the token counts. There is no id, provider or usage.cost

If the env entry is missing, os.environ/TYPESAFE_LANE_API_KEY resolves to None without an error, and only this lane fails at request time.

Provider keys use names that LiteLLM does not read

Section titled “Provider keys use names that LiteLLM does not read”

LiteLLM has built-in pass-through routes, for example /gemini/*, /openrouter/*, /fal_ai/* and /typesafe/*. Each one forwards every subpath with a provider key that it finds under a standard env name, such as GEMINI_API_KEY, OPENROUTER_API_KEY or FAL_AI_API_KEY. Every llm_api key can call these routes without a grant. If the proxy set the standard names, every workload key can spend on the proxy’s provider accounts with no model or budget limit.

The proxy therefore sets OPENROUTER_LANE_API_KEY, OPENROUTER_ZDR_LANE_API_KEY, GEMINI_LANE_API_KEY and TYPESAFE_LANE_API_KEY, and every lane and the configured pass-through name them explicitly. The built-in routes find no credential. Do not add an env var with a standard provider name to extraEnvVars. Use <PROVIDER>_LANE_API_KEY.

To make sure that the routes stay closed after a change, send a plain workload key to each route in the table below. Each route reads the key from its own place, so send it where the route reads it. The Gemini route reads x-goog-api-key or ?key=. With Authorization: Bearer, the request fails at admission with 401 ... expected to start with 'sk-', which looks like a pass and tests nothing.

Route Where the key goes Answer when the route is closed
GET /gemini/v1beta/models x-goog-api-key: <key> 500. The proxy log shows Required 'GEMINI_API_KEY'/'GOOGLE_API_KEY' in environment
GET /openrouter/v1/key Authorization: Bearer <key> 401 Missing Authentication header, from OpenRouter
GET /fal_ai/<any path> Authorization: Bearer <key> 401 FAL_AI_API_KEY is not set

A 200 from any row means that the route found a provider credential. Find the standard env name in extraEnvVars and rename it to the <PROVIDER>_LANE_API_KEY form.

The ZDR key is an OpenRouter key for the lanes that must reach only providers with zero data retention (ZDR). The key lives in the OpenRouter workspace litellm-zdr. The default guardrail of that workspace enforces ZDR for all five model groups, on every request of every key in the workspace. The zdr field of a request cannot remove that rule, so a provider block from a caller cannot reach a provider without ZDR.

Item Where Owner
Workspace litellm-zdr OpenRouter tf/litellm/openrouter.tf
Key litellm-zdr-lanes OpenRouter, in litellm-zdr tf/litellm/openrouter.tf
Default guardrail of litellm-zdr OpenRouter the management API call below
Management key secret/fzymgc-house/infrastructure/openrouter/terraform, property management_key Sean, by hand
ZDR key value secret/fzymgc-house/cluster/litellm/openrouter-zdr, property api_key tf/litellm/openrouter.tf

Keep only ZDR keys in litellm-zdr. The plain lane key litellm lives in the workspace fzymgc-litellm, which has no ZDR guardrail.

The proxy reads the ZDR key as OPENROUTER_ZDR_LANE_API_KEY. These lanes name it: openrouter-zdr/*, engram-summarize, octopus-embed, octopus-utility, octopus-review and the probe lane zdr-probe-embed. The lanes openrouter/* and gpt-4o-mini and the decisions route use OPENROUTER_LANE_API_KEY. Their callers can choose a provider without ZDR.

Do not set TF_LOG on the workspace main-cluster-litellm. At DEBUG, the OpenRouter provider logs whole response bodies, and the body of a new key holds the key.

Provider OpenRouterTeam/openrouter 0.3.19 cannot manage the default guardrail of a workspace. Until a release can, an operator sets it with one call. The output openrouter_zdr_default_guardrail_id of tf/litellm gives the same id that this lookup finds:

Terminal window
export VAULT_ADDR=https://vault.fzymgc.house
MK="$(vault kv get -field=management_key secret/fzymgc-house/infrastructure/openrouter/terraform)"
GID="$(curl -s https://openrouter.ai/api/v1/workspaces -H "Authorization: Bearer $MK" \
| jq -r '.data[] | select(.slug == "litellm-zdr") | .default_guardrail_id')"
curl -s -X PATCH "https://openrouter.ai/api/v1/guardrails/$GID" \
-H "Authorization: Bearer $MK" -H 'Content-Type: application/json' \
-d '{"enforce_zdr_anthropic":true,"enforce_zdr_openai":true,"enforce_zdr_google":true,
"enforce_zdr_xai":true,"enforce_zdr_other":true}' \
| jq -c '(.data // .) | {name, zdr: [.enforce_zdr_anthropic, .enforce_zdr_openai,
.enforce_zdr_google, .enforce_zdr_xai, .enforce_zdr_other]}'
unset MK

The reply must show five times true. No plan shows a later change to this guardrail. After each change to the OpenRouter account, send the request below again.

A request that asks for StreamLake, a provider of deepseek/deepseek-v4.1-flash without ZDR, must not reach it:

Terminal window
ZDR_KEY="$(vault kv get -field=api_key secret/fzymgc-house/cluster/litellm/openrouter-zdr)"
curl -s https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $ZDR_KEY" -H 'Content-Type: application/json' \
-d '{"model":"deepseek/deepseek-v4.1-flash","max_tokens":256,
"provider":{"only":["StreamLake"],"zdr":false},
"messages":[{"role":"user","content":"Reply with the word OK."}]}' \
| jq -c '{provider, error: .error.message}'
unset ZDR_KEY

The expected answer is HTTP 404 with “ZDR violation (guardrail)” in the message. If the reply names StreamLake, the guardrail is not in force. Set it again with the call above.

The request above proves the key. This one proves the lanes, because LiteLLM passes a provider block from the caller to OpenRouter in place of the block of the lane. Run it from a workstation with Vault access:

Terminal window
export VAULT_ADDR=https://vault.fzymgc.house
key() { vault kv get -field=api_key "secret/fzymgc-house/cluster/litellm-keys/$1"; }
ask() {
curl -s https://llm.fzymgc.house/v1/chat/completions \
-H "Authorization: Bearer $1" -H 'Content-Type: application/json' \
-d "{\"model\":\"$2\",\"max_completion_tokens\":256,\"provider\":$3,
\"messages\":[{\"role\":\"user\",\"content\":\"Reply with the word OK.\"}]}" \
| jq -c --arg m "$2" '{model: $m, provider, error: (.error.message // "" | .[0:90])}'
}
OCTOPUS="$(key octopus)"; PASS="$(key openrouter-passthrough)"
ask "$OCTOPUS" octopus-review '{"only":["StreamLake"],"zdr":false}'
ask "$PASS" openrouter-zdr/deepseek/deepseek-v4.1-flash '{"only":["StreamLake"],"zdr":false}'
ask "$PASS" openrouter/deepseek/deepseek-v4.1-flash '{"only":["StreamLake"]}'

The first two lines must not name StreamLake as provider. The third line must name StreamLake: it proves that StreamLake serves the model and that the plain wildcard does not use the ZDR key. If the third line does not name StreamLake, the first two prove nothing.

Each refused request reaches LiteLLM as a 404, and LiteLLM takes a deployment out of service for 5 seconds after one 404. A ZDR lane has one deployment, so for those 5 seconds the lane answers every caller with HTTP 429 “No deployments available”. Wait 5 seconds before you send a plain request to the lane.

Run this after each change to the OpenRouter account, after each change to the lanes in values.yaml, and after each LiteLLM version bump.

To rotate the ZDR key, raise zdr_key_version in tf/litellm/openrouter.tf and merge. The apply creates the new key, writes it to Vault and deletes the old key. The ZDR lanes fail until the proxy has the new value: ESO refreshes litellm-secrets within five minutes, and Stakater Reloader then restarts the proxy. To shorten that time, force the refresh as in step 5 of the procedure for a scripted key.

To rotate the management key, create a new one in the OpenRouter dashboard, write it with vault kv put to its path, and delete the old one in the dashboard. The path holds only management_key, so kv put loses nothing.

A different shape from every key above. The consumer keys are scoped by models, which LLM lanes they can reach. An MCP client key is scoped by object_permission.mcp_servers, which MCP routes it can reach. The two scopes are independent. A key that must do both carries both. The measured scope rules for MCP routes are in MCP Identity Routes.

They are virtual keys like any other: the same /key/generate, the same GET /key/info, the same revocation path, the same spend attribution.

{
"key_alias": "<consumer>-mcp",
"user_id": "<consumer>",
"key_type": "llm_api",
"object_permission": {"mcp_servers": ["context7", "exa", "clickhouse_ro"]},
"rpm_limit": 120,
"max_budget": 10.0,
"budget_duration": "30d",
"duration": null
}
Field Why it is there
object_permission.mcp_servers the route names this key can reach, the entire boundary. Use underscore spellings (clickhouse_ro), the names LiteLLM matches against
user_id whose key it is. MCP tool-call spans carry it, which is what makes per-caller attribution work on the MCP plane
key_alias the human-readable label on the span and in spend reports

If the user of the key is not a proxy_admin, omitting object_permission reaches nothing, and that is the intended default. It is the inverse of models, where omitting it means unrestricted. A key of a proxy_admin user with no object_permission list reaches every MCP route. An explicit list binds the key of a proxy_admin user too, even an empty list []. Plan 03-13 measured both rules on LiteLLM v1.103.0-rc.1.

If the user of the key is not a proxy_admin, LiteLLM refuses a key with no object_permission at initialize with HTTP 403, so each key needs a grant for each route. The dns-agent-triage key is Terraform-owned and cannot carry object_permission. The access group mcp-dns in tf/litellm/access_groups.tf grants it the dns route only. Its models list names no lane, so the key serves no chat request. dns-control still verifies the Keycloak bearer of every caller. See A Terraform-owned key and an access group.

clickhouse_rw is the estate’s only MCP write surface, and both ClickHouse MCP instances validate the same upstream token, so this list is the only thing that separates read from write.

The two identity routes need no extra field on the key. engram and kubernetes relay the caller’s own Keycloak token to the upstream. The virtual key admits and scopes, and the identity rides a second header the client sends. See How authentication works.

Verify with GET /key/info as for any other key. It reads the scope back as internal ids rather than names, which is canonical storage, not drift:

Terminal window
curl -sS "https://llm.fzymgc.house/key/info" \
-H "Authorization: Bearer <minted-key>" | jq '.info.object_permission'

Authenticate as the key, per Verifying a minted key step 1.

Four steps, in this order.

  1. Scope. GET /key/info shows the expected models, aliases, limits, budgets and key_type:

    Terminal window
    curl -sS "https://llm.fzymgc.house/key/info" \
    -H "Authorization: Bearer $SEEDED_KEY" \
    | jq '.info | {models, aliases, key_type, rpm_limit, tpm_limit,
    max_budget, budget_duration, expires,
    soft_budget: .litellm_budget_table.soft_budget}'

    Authenticate as the key. Never put it in the query string. ?key=<value> lands the credential in LiteLLM’s access log, in Traefik’s, and in /proc/<pid>/cmdline for the life of the curl. With no key parameter, /key/info reports the key in the Authorization header. A 401 here means the value is not registered at all, which is itself the answer to step 3.

    soft_budget is not on the key row. A /key/generate that carries soft_budget creates a row in litellm_budgettable and links it by budget_id. The key’s own soft_budget field stays null. Read it from litellm_budget_table, as the filter above does.

  2. Reachability. One probe call with the client-sent model string returns 200. Use the string the workload sends, not the lane name. That is what exercises the pre-alias allow-list.

  3. Seed and register match. The value LiteLLM holds for this key and the value at that consumer’s Vault path must be the same value. The check is an authenticated round trip. /key/info does not return the registered value, so there is no second value to digest. A value seeded but never registered returns 401, and a value registered under a different secret returns 401 for this one. Carry the digest as an identifier, never print the value:

    Terminal window
    VAULT_DIGEST="$(printf '%s' "$SEEDED_KEY" | shasum -a 256 | cut -c1-8)"
    PROBE_STATUS="$(curl -sS -o /dev/null -w '%{http_code}' \
    -H "Authorization: Bearer $SEEDED_KEY" \
    https://llm.fzymgc.house/v1/models)"
    RESOLVED_ALIAS="$(curl -sS "https://llm.fzymgc.house/key/info" \
    -H "Authorization: Bearer $SEEDED_KEY" | jq -r '.info.key_alias // "none"')"
    if [ "$PROBE_STATUS" = "200" ] && [ "$RESOLVED_ALIAS" = "octopus" ]; then
    echo "coupled: vault value ${VAULT_DIGEST} authenticates and resolves to alias ${RESOLVED_ALIAS}"
    else
    echo "NOT COUPLED vault=${VAULT_DIGEST} probe=${PROBE_STATUS} alias=${RESOLVED_ALIAS}"
    fi

    Substitute the consumer’s own expected alias for octopus. shasum -a 256 rather than sha256sum, because macOS ships the former.

    On a mismatch, re-register from Vault. For a scripted key, call /key/generate again with the Vault value. Do not re-seed, which produces a third distinct value. For a Terraform-owned key, do not call /key/generate. Raise its key_wo_version without a new Vault write, and the apply replaces the key with the value Vault holds.

  4. Record the result, the value’s digest, the observed statuses and the resolved alias, in the pull request that made the change.

zdr-probe-embed is a deliberately unsatisfiable harness lane. It serves no consumer, maps to no route, and its guaranteed failure is the point. No credential lists it.

Two controls hold that. A static check over the mint bodies extracts every models array in this document and in the Terraform module, and asserts the probe lane is a member of none of them. The check is over the models arrays, not the whole file, because the live denial below names the lane in order to deny it. Extracting zero models arrays is a failure, not a pass:

Terminal window
rg -oP '"models":\s*\[[^\]]*\]' docs/operations/litellm.md
rg -U -oP 'models\s*=\s*\[[^\]]*\]' tf/litellm/main.tf

A live denial, because a runbook is documentation and not access control. With the octopus key, issue one embedding request that names zdr-probe-embed:

Terminal window
curl -sS -o /dev/null -w '%{http_code}\n' \
-X POST https://llm.fzymgc.house/v1/embeddings \
-H "Authorization: Bearer $SEEDED_KEY" \
-H 'Content-Type: application/json' \
-d '{"model": "zdr-probe-embed", "input": "denial probe"}'

The expected outcome is a model-access denial that names the model, not an authentication failure and not a 200. If it returns 200, a harness-only lane is reachable by a production workload.

POST /key/delete, and it is not instant. Budget about 60 seconds before the credential is dead on every replica.

Narrowing a key’s models with POST /key/update is not a revocation mechanism.

Every key mutation has two halves that converge at different speeds:

Half When it changes How you observe it
the authoritative row in Postgres at once, before the 200 returns GET /key/info reports the new shape at once
what each replica enforces the replica that handled the call changes inline. Every other replica keeps its cached copy until that copy expires only a per-replica probe through kubectl port-forward can see this

The cache that expires is LiteLLM’s user_api_key_cache. Its in-memory layer has a 60 second TTL. replicaCount in values.yaml sets how many replicas hold such a cache. The Redis layer, which enable_redis_auth_cache turns on, is shared. A replica that misses in memory reads the current state from Valkey.

v1.104 added instant eviction across replicas over Redis pub/sub. On v1.104.0-rc.1 that path was off for the Valkey cluster client, and every start logged auth cache invalidation subscriber disabled: cluster redis client has no pub/sub support. Upstream commit d33d36ce86 (BerriAI/litellm#43110) adds a cluster-aware subscriber, and v1.105.0-rc.3 includes it. The v1.105.0-rc.3 pods do not log that line at start. No test on this cluster shows yet that an eviction reaches every replica. Until one does, use the 60 second in-memory TTL as the window.

If the credential is believed compromised, do not wait out the TTL. Delete the key, then kubectl rollout restart deployment/litellm -n litellm. A pod restart discards every in-memory cache and is the only mechanism here that revokes faster than 60 s.

A deleted key goes on spending for the length of that window. The spend ledger keeps accruing against it until every replica’s cache expires, and the restart above is the only lever that ends it sooner.

Concurrency: what is and is not guaranteed

Section titled “Concurrency: what is and is not guaranteed”

After POST /key/update returns 200, exactly one thing is guaranteed: the authoritative row has changed, and any later GET /key/info reports the new shape. Enforcement is eventually consistent across replicas with the 60 s bound above.

Two admin mutations that race on the same key have no stated ordering guarantee. Serialize admin mutations on a single key: complete one and verify it before you issue the next.

Reading Valkey directly: VALKEYCLI_AUTH shadows REDISCLI_AUTH

Section titled “Reading Valkey directly: VALKEYCLI_AUTH shadows REDISCLI_AUTH”

The valkey-litellm-valkey-* pods set VALKEYCLI_AUTH in the container environment for the operator’s own user, and valkey-cli prefers VALKEYCLI_AUTH over REDISCLI_AUTH. Supplying the correct default password in REDISCLI_AUTH therefore authenticates as the operator user with the default user’s password and reports WRONGPASS invalid username-password pair. Unset it first:

Terminal window
kubectl get secret -n litellm litellm-valkey-auth -o jsonpath='{.data.password}' | base64 -d \
| kubectl exec -i -n litellm valkey-litellm-valkey-0-1-0 -c server -- sh -c \
'unset VALKEYCLI_AUTH; REDISCLI_AUTH=$(cat); export REDISCLI_AUTH;
exec valkey-cli --no-auth-warning --user default "$@"' sh INFO stats

The password reaches the pod on stdin and becomes an environment variable in the pod’s shell. Do not use valkey-cli -a <password>. That places the credential in argv, where /proc/<pid>/cmdline and any ps on that node can read it.

A key’s cache entry is named for the SHA-256 of the key value, so valkey-cli --scan --pattern '*<sha256-of-key>*' on the primary (confirm with INFO replication) lists it alongside its spend:key:* and {api_key:*}:tokens counters.

In-place rotation is not available on this estate. POST /key/regenerate and POST /key/{key}/regenerate are Enterprise-gated in LiteLLM. This deployment sets no LITELLM_LICENSE, so both route spellings return HTTP 500. The supported procedure is revoke and re-mint, and it has a short outage. No test asserts the gate. It is a vendor’s licensing decision, and a test on it would go red when LiteLLM changes their product, not when we make a mistake. If you set a LITELLM_LICENSE, this section is what has to change. See If a licence is ever acquired.

Key aliases are globally unique. A second /key/generate that carries an alias LiteLLM already holds is rejected with Key with alias '<name>' already exists. So the replacement cannot be minted under the consumer’s alias while the old key exists. The order is forced: delete first, then mint. The interval between them, plus the time the new value takes to reach the running pod, is the outage. Keeping the alias is what keeps every spend query, tile and alert pointing at one continuous series across the rotation.

There is no overlap window. Two consequences for a key minted with a duration:

  • Rotating restarts the clock. The replacement is a fresh /key/generate, so its duration starts at the re-mint.
  • A key you let expire costs the same outage as one you rotate, at a time you did not pick. Rotate the TTL-bearing keys ahead of their expiry, in a quiet window. The two workstation credentials claude-code and openrouter-passthrough carry "duration": "90d". Read their expiry with GET /key/info (.info.expires).

The outage: how long, what breaks, what to do about it

Section titled “The outage: how long, what breaks, what to do about it”

The window runs from /key/delete until the workload’s pod runs with the new value. It has four legs, and only the last two are worth managing:

Leg Duration Notes
/key/delete to the new value serving on the consumer’s lane under a second the outage starts at the delete, not at the Vault write
/key/delete to the old value refused everywhere up to the 60 s auth cache TTL see What actually revokes a key. A fast 401 through the ingress means “you can re-mint now”, not “the old value is dead”
Vault write to the Secret carrying it about a second if you force the sync, up to refreshInterval: 5m if you do not the leg you can shorten, step 5 below
Secret updated to the pod running with it about a minute for a rollout, and never without one see Propagation is ESO plus a pod restart

Run the commands back to back and the outage is a little over a minute. Wait out a refresh interval instead and it is about six.

What breaks: every call the consumer makes to https://llm.fzymgc.house/v1 is refused with 401, an authentication failure. Nothing else in the estate is affected.

Decide how to handle the consumer before you start:

  • The consumer retries. karakeep, mealie and octopus are background or interactive-with-retry workloads. A failed enrichment or chat turn is retried, and the window closes on its own.
  • Queue-shaped work drains or pauses first. If the consumer is mid-batch (an engram corpus reindex, an octopus review sweep), a 401 storm burns through its retry budget and marks work failed. Let the batch finish, or pause it, before you delete the key.

Rotate during a quiet window. No mechanism makes the outage smaller. The only lever is when it lands.

The keys in tf/litellm/main.tf rotate through Terraform. The mechanism is the same delete and re-create under the same alias. key_wo_version forces a replacement, and Terraform deletes the old key before it creates the new one. The new key carries the value that Vault holds when the apply runs. Do not add create_before_destroy to the resource. LiteLLM keeps aliases unique, so the new key cannot exist beside the old one.

The apply reads Vault when it runs, so the order of the Vault write and the merge matters. If you write the new value earlier, ESO can deliver it to the pod before any key accepts it, and the consumer is down from that moment.

  1. Decide how to handle the consumer during the outage, per the list above. Make sure that the litellm-db-failure alert is not firing. The first caveat below explains why.

  2. Open a PR that raises key_wo_version of that key by one in tf/litellm/main.tf. Change nothing else in the module.

  3. Read the speculative plan of main-cluster-litellm on the PR. It must show exactly one resource that must be replaced, with key_wo_version marked # forces replacement, and no other change. The value is write-only and does not appear in the plan.

  4. Immediately before you merge, write the new value into Vault:

    Terminal window
    scripts/seed-litellm-vault.sh --consumer <name> --rotate

    <name> is the consumer name from Which tool owns each key, for example agent-memory for the engram alias. The command writes the value and does not register it.

  5. Merge the PR. The workspace applies automatically. Watch the run in the HCP Terraform UI or with the read-only run-details tool. Do not trigger a run through the API or the MCP tools.

  6. When the apply is complete, force the ExternalSecret to sync. This is step 5 of the scripted procedure below, and its table names each ExternalSecret.

  7. Verify, per step 7 of the scripted procedure below. The shape must match the entry in tf/litellm/main.tf, and token on the new row must equal the hash of the value in the pod.

If the PR cannot merge after step 4, the consumer receives a value that no key accepts within one refresh interval. Restore the prior value at once with vault kv rollback -version=<prior version> on that key’s dedicated path. The path holds only api_key, so the rollback restores nothing else.

If the apply fails after the delete and before the create, the key is gone and the consumer is down. Read the run error. When the cause is fixed, start a new run from the HCP Terraform UI. The run creates the key from the value in Vault.

This procedure applies to the MCP client keys. For a Terraform-owned key, follow the procedure above. Steps 5 to 7 below are shared.

  1. Capture the key’s current shape before you delete anything. This is what you re-mint against and what you diff afterwards. Read it with the master key, by alias. The consumer’s own key cannot read it, because a key_type: llm_api credential is refused on /key/info with 403:

    Terminal window
    ALIAS=<consumer>-mcp # the LiteLLM key_alias
    curl -sS "https://llm.fzymgc.house/key/list?key_alias=${ALIAS}&return_full_object=true" \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
    | jq '.keys[0] | {key_alias, models, aliases, rpm_limit, tpm_limit,
    max_budget, budget_duration, key_type, budget_id, spend}' \
    | tee /tmp/rotate-${ALIAS}-before.json
    # soft_budget lives in the budget table and /key/list does NOT join it: the row
    # reports `soft_budget: null` and `litellm_budget_table: null` even when one is set.
    BUDGET_ID="$(jq -r .budget_id /tmp/rotate-${ALIAS}-before.json)"
    curl -sS "https://llm.fzymgc.house/budget/list" \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
    | jq --arg id "$BUDGET_ID" '.[] | select(.budget_id == $id) | {soft_budget}'

    Cross-check it against that consumer’s mint body under Minting and scoping virtual keys. Step 4 re-uses that body verbatim.

  2. Delete the old key. The outage starts now:

    Terminal window
    printf '%s' "$OLD_KEY" \
    | python3 -c 'import json, sys; print(json.dumps({"keys": [sys.stdin.read().strip()]}))' \
    | curl -sS -X POST "https://llm.fzymgc.house/key/delete" \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
    -H "Content-Type: application/json" --data-binary @-

    python3 builds the body from stdin. jq -n --arg key would put the credential in jq’s argument vector.

  3. Write the new value into that consumer’s Vault target first, the same path and property its delivery already reads. Vault is the origin of the value, and step 4 attaches scope to it. If step 4 fails, the value is safely in Vault and the fix is to retry step 4. Derive the path and property from CONSUMER_TABLE rather than typing them, and hand the value to vault on stdin:

    Terminal window
    # CONSUMER is the name the seed script accepts (`--consumer <name>` prints the
    # accepted set on an unknown one). For engram that name is `agent-memory`; the
    # alias `engram` is a LiteLLM label, not a Vault path.
    read -r VK_PATH VK_PROPERTY <<<"$(
    rg -N "(^|CONSUMER_TABLE=')${CONSUMER}\|" scripts/seed-litellm-vault.sh \
    | sed "s/^CONSUMER_TABLE='//" \
    | awk -F'|' '{ print $2, $3 }'
    )"
    [ -n "$VK_PATH" ] && [ -n "$VK_PROPERTY" ] || echo "no CONSUMER_TABLE row for '${CONSUMER}'"
    # The same generator new_virtual_key() uses in scripts/seed-litellm-vault.sh:
    # VIRTUAL_KEY_PREFIX + secrets.token_urlsafe(VIRTUAL_KEY_TOKEN_BYTES). Keep the
    # value in a shell variable; do not print it.
    NEW_KEY="$(python3 -c 'import secrets; print("sk-" + secrets.token_urlsafe(32))')"
    printf '%s' "$NEW_KEY" | vault kv patch "$VK_PATH" "${VK_PROPERTY}=-"

    vault kv patch prints a recommendation to add the patch capability to your ACL policy and then falls back to read-then-write. That is a warning, not a failure. Read the exit status. printf rather than a here-string, because <<< appends a trailing newline to the value. The path out of the table already carries its secret/ mount, so no -mount flag.

  4. Register that value with /key/generate, under the same key_alias, with that consumer’s recorded mint body and "key" set to the value Vault now holds. Do not invent a new alias. A changed alias re-points every spend query and the spend alert at a series that starts at zero. If this call fails, the value is still in Vault. Retry it.

  5. Force the ExternalSecret to sync rather than wait out refreshInterval: 5m:

    Terminal window
    kubectl annotate externalsecret ${EXTERNAL_SECRET} -n ${NAMESPACE} \
    force-sync=$(date +%s) --overwrite

    Watch kubectl get externalsecret ${EXTERNAL_SECRET} -n ${NAMESPACE} -o jsonpath='{.status.refreshTime}', not the clock. The annotation does not put the ExternalSecret out of sync in ArgoCD.

    Consumer ExternalSecret Namespace
    karakeep karakeep-secrets karakeep
    mealie mealie-app-secrets mealie
    octopus octopus-app octopus
    memory-mcp (engram) memory-mcp-litellm agent-memory
  6. Make sure that the pod restarted with the new value. See Propagation is ESO plus a pod restart. Stakater Reloader does it, and you are confirming, not causing.

  7. Verify with a real call on the consumer’s own lanes, then diff the key’s shape. Re-run step 1’s /key/list and /budget/list pair. Its output must match /tmp/rotate-${ALIAS}-before.json on alias, models, aliases, limits and budgets. token, created_at and spend differ. Then call every lane the consumer uses, not just one. A re-mint that dropped a single models entry leaves a credential that works until the consumer reaches for the missing lane.

    Cross-check that the pod holds the registered credential. LiteLLM stores token as the SHA-256 of the key value, so the token on the row you read must equal the hash of the value in the workload’s environment.

Three caveats, each a way the procedure looks fine and is not

Section titled “Three caveats, each a way the procedure looks fine and is not”

You cannot rotate or revoke during a database outage, and the outage is what makes it dangerous. /key/delete and /key/generate are both database writes, while completions keep flowing on cached auth because allow_requests_on_db_unavailable stays true. A rotation started during a masked Postgres outage can delete the old key and then fail to mint its replacement, and the proxy looks healthy the whole time. The signal is the litellm-db-failure alert. If it is firing, or fired recently, wait. See When the database is unavailable.

Revocation is fast on the pod you talked to and slow everywhere else. A fast 401 through the ingress is the ingress routing you back to the replica that handled the delete. Read it as “you can re-mint now”, and plan revocation against the 60 s bound in What actually revokes a key.

A deleted key goes on spending for the length of that window, attributed to its alias under the key id that was deleted. If you revoke because a credential is believed compromised, the only lever that ends it sooner is kubectl rollout restart deployment/litellm -n litellm.

The alias is continuous across a rotation and the key id is not. A spend query grouped on user_api_key_alias sees one unbroken series across the rotation. A query grouped on the key id sees a series that ends and a new one that starts at zero. The key row’s own spend field resets with the key. That is why the tiles and the spend alert are keyed on the alias.

A Vault write that reaches the Kubernetes Secret is not the workload picking it up. Every consumer reads its LiteLLM credential through env.valueFrom.secretKeyRef, and an environment variable is injected once, at container start. The full chain is: Vault write, ESO refresh (forced, or refreshInterval: 5m), Secret updated, pod restarted, new value in use.

The restart comes from Stakater Reloader, which acts only on workloads that carry the annotation:

Workload reloader.stakater.com/auto Rotation restarts it automatically?
karakeep/karakeep true yes
mealie/mealie true yes
octopus/octopus true yes
agent-memory/memory-mcp true yes

What the seed script is for, and what it is not

Section titled “What the seed script is for, and what it is not”

No mode of scripts/seed-litellm-vault.sh is the rotation path on its own:

  • A bare re-run regenerates everything at once: the CNPG password, the master key, the Valkey password and all eight workload virtual keys. It has no overwrite refusal.
  • --consumer <name> regenerates exactly one consumer’s value, and refuses to overwrite an existing value without --rotate.

Either way the new value is dead in LiteLLM until it is registered. Vault before registration is the correct order. What disqualifies the script as a rotation is that it never deletes the live key: the old value stays registered while Vault changes underneath it, so the next ESO refresh can hand the workload a value LiteLLM has never seen, with no bounded window and no signal. The procedures above delete first so that the outage has a beginning and an end.

One exception: step 4 of Rotating a Terraform-owned key runs --consumer <name> --rotate on a live key. That is safe only because the merge that follows deletes the old key and registers the new value within the same few minutes.

Setting LITELLM_LICENSE makes POST /key/regenerate available, with a grace_period that keeps the old value valid across the Vault write and the ESO refresh. Three things to know before you adopt it:

  • The overlap is one key with two values, not two keys. Spend, budget and rate limits are shared during the window.
  • The overlap lookup is a database read, so it fails in the masked-outage window described in the first caveat above.
  • A positive in-memory cache sits in front of that lookup, so grace-period revocation is eventual within about a minute in both directions.

Nothing automated tells you to revisit this section when the licence is set. Whoever sets LITELLM_LICENSE owns rewriting the procedures above around grace_period.

The master key is not just an API credential. The Admin UI session token is a JWT signed with it, so a rotation signs out every logged-in operator at once. Treat a master-key rotation as a visible operator-facing event, and do not schedule one in the middle of an incident worked through the UI.

Update every place the master key is held in the same change: the Vault property master_key on secret/fzymgc-house/cluster/litellm-keys/master, and whatever the operator workstation exports as LITELLM_MASTER_KEY. The litellm-secrets ExternalSecret delivers it to the proxy, and Reloader restarts the proxy.

The master-key password form must stay unroutable

Section titled “The master-key password form must stay unroutable”

LiteLLM does not fail closed when its SSO environment is missing or wrong. It renders an HTML username and password form whose POST authenticates against the master key itself. Publishing /login or /fallback/login puts a master-key password prompt on the internet-facing host.

The proxy also refuses the login itself. general_settings in argocd/app-configs/litellm-chart/values.yaml sets disable_env_credential_login: true, which turns off the UI login with the master key, and disable_password_login_when_sso_enabled: true, which rejects every username and password UI login while SSO is configured. API calls with Authorization: Bearer <master key> do not go through the UI login and keep working. If Keycloak is down, administer the proxy over the API with the master key.

Two controls hold the routing, and they cover different halves:

# Control What it proves Where
1 The exclusions in the IngressRoute rule Nothing on its own. It is the mechanism. Both exclusion matchers must stay written in the rule, and nothing checks that they do argocd/app-configs/litellm/ingress.yaml
2 The router-side check below That Traefik refuses the paths, including after a deleted matcher syncs this section, run by hand

Confirming the master-key password form is unroutable

Section titled “Confirming the master-key password form is unroutable”

Run it after any change to argocd/app-configs/litellm/ingress.yaml, once ArgoCD reports the IngressRoute synced, after a Traefik upgrade, and once a quarter whether or not anything changed. Run it from any shell on the LAN that reaches the host through the Traefik LoadBalancer VIP, outside Kubernetes, with no port-forward. No browser is needed, so a scheduled job can run it.

Terminal window
curl -sS -i https://llm.fzymgc.house/login
curl -sS -i https://llm.fzymgc.house/fallback/login

Pass: both answer Traefik’s own 404. Status 404, content-type text/plain, no server header, body exactly:

404 page not found

Fail: anything served by LiteLLM instead of refused by Traefik. A JSON body, a server: uvicorn response header, or a 200 text/html carrying a login form. Any of those means the exclusion is not honoured and the master-key password prompt is on the network. Treat it as an incident.

A Traefik 404 from this host means an exclusion was hit, not a missing prefix. The root is published, so there are no other 404s of that shape.

Every connection to the proxy arrives from a pod. People and off-cluster clients come through a Traefik pod, and in-cluster workloads call the Service. The peer address is therefore never the real client. Traefik puts the real client address in the X-Forwarded-For header, because its Service runs externalTrafficPolicy: Local.

Three keys in general_settings in argocd/app-configs/litellm-chart/values.yaml make LiteLLM read that header. All three name the same range, the Cilium pod pool 10.245.0.0/16. That is also the value of FORWARDED_ALLOW_IPS in extraEnvVars.

Key Effect
use_x_forwarded_for: true Spend rows and the auth error log record the address from the header, not the Traefik pod. Without it, every request logs Received a request with an X-Forwarded-For header but use_x_forwarded_for is not enabled
trusted_proxy_ranges Turns on the per-address limit on failed Admin UI sign-ins. Without it, the proxy counts failures per user name only, and every start logs trusted_proxy_ranges is not set or not a valid list of ranges
mcp_trusted_proxy_ranges Lets the MCP routes resolve the client address from the header. With use_x_forwarded_for and no range, that lookup fails closed and the first request logs a warning

The sign-in limit uses the LiteLLM defaults. One address gets 10 failed attempts in 60 seconds, then a block of 300 seconds. One user name from one address gets 5. A blocked attempt gets 429 before the password check. The Admin UI accepts only SSO sign-ins, and the ingress hides the password form, so the limit guards a path that is already closed. The setting keeps the count correct if that form ever opens.

An in-cluster workload that calls the Service directly is inside the range too, so it can put any address in its own X-Forwarded-For. It already holds a virtual key, so a false address changes only what its rows record. Do not widen the range past the pod pool, and never use 0.0.0.0/0.

LiteLLM stores the request body and the response of each call in the LiteLLM_SpendLogs table. Two keys in general_settings in argocd/app-configs/litellm-chart/values.yaml control this:

Key Value Effect
store_prompts_in_spend_logs true Fills the proxy_server_request and response columns
maximum_spend_logs_retention_period "7d" A daily job deletes spend-log rows older than seven days

To read a call, open the Admin UI at https://llm.fzymgc.house/ui and go to the Logs page. You can filter by key, model and time. The OTel spans in ClickStack carry token counts and cost only. They do not carry prompt or response text.

LiteLLM cuts each string field to 8192 characters before it writes the row. The MAX_STRING_LENGTH_PROMPT_IN_DB environment variable in extraEnvVars sets this limit. Without it, the limit is 2048 characters.

The rows contain everything that the callers send, for example the memories that engram summarizes. The same rows go into the CNPG backups. Do not remove the retention period while store_prompts_in_spend_logs is true. Without it, the table grows without a limit in a database that other tenants share.

The retention job deletes whole rows, so the per-request history in LiteLLM_SpendLogs covers seven days. It also deletes rows from LiteLLM_SpendLogToolIndex. Budgets and the Usage pages keep their history. LiteLLM keeps budget spend on the key, team and user rows, and the Usage pages read the daily spend tables, which the job does not touch.

LiteLLM runs with allow_requests_on_db_unavailable: true, which produces a lopsided failure mode. Knowing the shape in advance stops an operator from diagnosing two faults where there is one.

Completions keep flowing. Auth resolves from cache, so requests on keys that the proxy already saw continue to work through a Postgres outage. The readiness and startup probes do not check the database, so the pods stay in the Service.

The outage is loud anyway. The DB-failure hook emits an error span, and the litellm-db-failure alert in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml fires on it. That alert, not a completion failure, is the signal. Keep the tile on the trace source, filtered on all three of ServiceName = 'litellm', StatusCode = 'Error' and SpanAttributes['db.system.name'] = 'postgresql', on a strict > 0. Keep service_callback: ["otel"] and allow_requests_on_db_unavailable together in values.yaml, so a later change cannot buy the signal by removing the degrade. Nothing checks either.

The Admin UI login fails while completions succeed. The SSO path requires a database connection and returns 403 during a full outage. Read a failing login beside healthy completions as confirmation of the database outage, not as a second fault in the UI or in Keycloak.

Key rotation and key revocation are both unavailable during an outage. /key/delete and /key/generate are database writes. A rotation started during a masked outage can delete the old key and then fail to mint its replacement. See the first caveat under Rotating a virtual key. Wait for the database before you rotate or revoke anything.

The chart default points the readiness and startup probes at /health/readiness. That path returns HTTP 503 when the database is disconnected. With the default, a Postgres outage makes every replica NotReady within about 30 seconds, the Service has no backends, and every lane stops at the ingress, even though the proxy can serve from cached auth.

argocd/app-configs/litellm-chart/values.yaml therefore sets both probes to /health/liveliness:

Probe Path
readinessProbe /health/liveliness
startupProbe /health/liveliness

/health/liveliness returns 200 while the process runs. A pod that starts during an outage also becomes ready. The litellm-db-failure alert, not the probes, reports the outage.

LITELLM_DISABLE_ACCESS_LOG_PATHS in extraEnvVars names /health/liveliness, so the probe calls do not appear in the access log. The proxy still answers them. The status code is the evidence, not a log line.

Do not point the probes back at /health/readiness, and do not set allow_requests_on_db_unavailable to false. Either change makes Postgres a hard dependency for every completion.

To see the database state of one pod during an outage, call GET /health/readiness on the pod with kubectl port-forward. A call through the Service or the ingress reaches a random replica.

argocd/app-configs/cnpg/postgres-cluster.yaml declares cluster main in namespace postgres with instances: 3, and it serves the keycloak, litellm, mealie, miniflux, octopus and temporal databases. A Postgres outage here is an estate event, and keycloak sharing the cluster puts SSO in its blast radius too. Never degrade the cluster to test a tenant’s behaviour.

If you are reading this during a real outage

Section titled “If you are reading this during a real outage”

Capture these before it ends. It takes about two minutes:

  1. kubectl port-forward -n litellm pod/<one pod> 4000:4000, a pod, not the Service, and record which pod.
  2. GET /health/readiness on it. Record the body and the status code.
  3. GET /health/liveliness on it. Expected 200.
  4. A cheap completion with a key that the pod did not see in the last 60 seconds (cold). Record the status code verbatim. 503 is the expected result. 401 is a blocking finding: a caller with a valid credential was told their credential is bad, which is what allow_requests_on_db_unavailable exists to prevent.
  5. The same completion with a key that the pod saw inside the last 60 seconds (warm). Expected 200. Without this pair, a blanket 503 would satisfy the cold-key criterion while proving the degrade broken.
  6. Whether litellm-db-failure fired, and how long after the first error span.

Mint, rotate and delete nothing while capturing. Never induce this outage by kubectl apply.

MCP session semantics across restarts and replicas

Section titled “MCP session semantics across restarts and replicas”

The symptom of a lost MCP session is a missing notification, not an error message. There is nothing to grep for.

  1. Session state is process-local, so it is lost on restart and unknown to every other replica. Every MCP session lives in per-process dictionaries on the pod that issued it. Nothing is in Postgres and nothing is in Valkey. replicaCount in values.yaml sets the replica count, and there is no session affinity. With more than one replica, a request that carries an mcp-session-id lands on a replica that does not know it part of the time, which is the same downgrade a restart produces.

  2. An unknown session id is not an error. It is a silent downgrade to stateless service. LiteLLM strips an unrecognized mcp-session-id from the request and serves it statelessly. There is no 400 session ID not found to look for. Any runbook, alert or dashboard that watches for one is watching for a response this proxy does not emit.

  3. Plain tools/list and tools/call keep working. Session continuity stops. What breaks is anything that requires the server to speak first or to resume a conversation: server-initiated notifications, sampling, and elicitation. Those never arrive.

  4. Two conditions produce errors, and they are the only two:

    Condition Response Meaning
    A live stateful session driven by someone other than its creator 403 Sessions are owner-bound. A mismatch is refused, not silently served
    A caller holding too many concurrent stateful sessions 429 A per-caller cap on held stateful sessions. Not a rate limit on requests, so it clears as sessions are released

Concurrency within one session is serialized. Two concurrent tools/call on the same session complete one after the other. A client that fans out on one session gets correctness, not throughput. If it wants throughput, it needs separate sessions.

Nothing needs draining, and nothing needs a maintenance window for MCP’s sake. A rolling restart drops every session. Every client that only does tools/list and tools/call continues without noticing, and the clients this estate runs are all in that class. If a future client depends on notifications, sampling or elicitation, re-establish that client after a rollout, and detect the failure at the client, by noticing that the expected message never arrived.

If you observe a real MCP session failure, capture:

  1. Which replica served the request (kubectl get pod -n litellm -o wide).
  2. The full request headers, mcp-session-id included, and the status code. A 400 here contradicts point 2 above and is a blocking finding.
  3. Whether a 403 or 429 was involved. Those are the designed errors.
  4. What the client expected to receive and did not. For a lost session this is the only signal.

Every LiteLLM version bump reaches this cluster through a Renovate pull request that waits two days and requires a named human reviewer. LiteLLM publishes several prereleases a week, and a fix this cluster needs often ships only in the newest one. A longer floor kept the cluster behind those fixes. The reviewer is the main control, and the two days give time for a bad release to be pulled before the pull request opens. The rule is the last element of packageRules in .github/renovate.json, group slug litellm-data-plane. It is last because Renovate applies packageRules in array order with later matches overriding earlier ones, and three general groups (container images, helm charts and argocd manifests) automerge these files on shorter floors. Keep the litellm-data-plane rule the last element of packageRules.

Two files carry a LiteLLM version that a Renovate manager reports:

Pin File Manager that sees it
Image tag plus digest argocd/app-configs/litellm-chart/values.yaml helm-values
OCI chart version argocd/app-configs/litellm-chart/kustomization.yaml kustomize

argocd/app-configs/litellm/** carries no version pin. The rule matches it as forward coverage, so that a pin that appears there is already covered. Do not describe that tree as guarded.

The rule matches by file name, never by dependency name. A name-based rule can cover nothing and emit no signal that it covers nothing.

Both automerge keys and an explicit reviewer list are required. The root configuration sets automerge: true and platformAutomerge: true. Disabling one key leaves the other in force, and omitting reviewers leaves nobody named on the pull request the delay exists to put a human in front of.

A vulnerability-remediation pull request bypasses the two-day floor, because Renovate merges config.vulnerabilityAlerts last, over the resolved package configuration. It does not merge itself: .vulnerabilityAlerts.automerge is false repo-wide, so the pull request opens at once and waits for the named reviewer. A supply-chain compromise published as an ordinary release is not a vulnerability alert. It travels the ordinary rule path, where the floor and the reviewer apply. The remaining exposure is a fix version that is itself compromised, which arrives with no release-age floor. See Renovate for why a narrower, per-dependency control is not available in repository configuration.

The model cost map is the LiteLLM file that lists each model, its price per token and the parameters that it accepts. LITELLM_LOCAL_MODEL_COST_MAP is True in argocd/app-configs/litellm-chart/values.yaml, so the proxy reads the copy bundled in the image. The scheduled reload, which the Admin UI configures, reads the same bundled file. The map changes only when the image pin changes, through the pull request that Adoption delay describes.

Without the flag, the proxy downloads the map from the main branch of BerriAI/litellm at startup and on each reload. An upstream edit to that file then changes the proxy with no pull request in this repository, and a restart does not undo it. Issue #2918 is an example. Upstream removed each openrouter/ row. Up to v1.105.0-rc.3 at least, LiteLLM accepts reasoning_effort for an OpenRouter model only when the row of that model has supports_reasoning: true. The octopus-review and octopus-utility lanes send reasoning_effort, so each Octopus review failed with UnsupportedParamsError.

A model that is newer than the image has no row in the bundled map. A request with reasoning_effort to that model fails in the same way.

Before each version bump, run this check on a host with Docker. Put the new digest in place of <digest>. If the Octopus lanes use a different model, put it in place of deepseek/deepseek-v4.1-flash.

Terminal window
docker run --rm --platform linux/arm64 -e LITELLM_LOCAL_MODEL_COST_MAP=True \
--entrypoint /app/.venv/bin/python ghcr.io/berriai/litellm@<digest> -c '
from litellm.utils import get_optional_params
for effort in ("low", "none"):
params = get_optional_params(model="deepseek/deepseek-v4.1-flash",
custom_llm_provider="openrouter", reasoning_effort=effort)
print(effort, params["reasoning_effort"])'

The command prints low low and none none. If it stops with UnsupportedParamsError, do not merge the bump. The bump breaks each Octopus review. A version that includes BerriAI/litellm#45640 accepts the reasoning parameters for each OpenRouter model, with or without a row.

The GitHub security-advisory watch on BerriAI/litellm is a setting on the operator’s GitHub account. Nothing in this repository asserts it or checks it. It can be switched off, or the account can lose access, and nothing here reports that.

The advisory watch and the Renovate reviewer are different accounts. The watch is on the operator’s personal account, and the Renovate rule assigns reviewers: ["fzymgc"]. A security advisory for LiteLLM arrives at one inbox, and the review request on a LiteLLM Renovate pull request arrives at the other.