Operational guide for the LiteLLM proxy at llm.fzymgc.house — where its database credential
lives, and the order in which that credential may be changed.
LiteLLM is the LLM data plane: an OpenAI-compatible proxy in the litellm namespace, fronted by a
Traefik IngressRoute that publishes API paths only. It keeps virtual keys, spend and budget state in
a CloudNativePG database (main, in the postgres namespace) and shares its auth cache through a
dedicated Valkey cluster in its own namespace.
Property
Value
Public host
https://llm.fzymgc.house
Namespace
litellm
Database
CNPG cluster main, database litellm, reached at main-rw.postgres.svc.cluster.local:5432
The password lives in exactly one place: the password property at
secret/fzymgc-house/cluster/postgres/users/main-litellm in Vault. Everything else that carries it
is a derivation, not a source:
CNPG’s own copy, materialised from the Vault property by External Secrets
litellm/litellm-secrets (Kubernetes Secret)
the proxy’s DATABASE_URL, composed from the same Vault property
The connection string is composed in the cluster and exists nowhere else. The ExternalSecret at
argocd/app-configs/litellm/secrets.yaml reads the username and the password as two separate
properties and assembles DATABASE_URL in its spec.target.template.data. The assembled string is
never written to Vault, never committed to git, and never present on the operator’s workstation.
That is the whole of ARCH-02: before this change the password also existed percent-encoded inside a
connection-string property, and nothing prevented the two copies drifting apart on a rotation.
The escaping is correct because of the seed script, not because of the escaper. The template
pipes the password through urlquery, which is Go text/template’s query escaper. It renders a
space as a plus sign. Inside a URL userinfo component a plus is a literal plus, not a space — so a
password containing a space would produce a wrong credential with no parse error anywhere, and
the only symptom would be an authentication failure at the database.
scripts/seed-litellm-vault.sh generates from an 81-character alphabet that contains no space and
no non-ASCII character, and every one of those characters was measured to survive the round trip.
The coupling between that alphabet and this escaping produces no runtime signal at all, and nothing
checks this coupling. Do not widen the alphabet without re-checking that every added character
survives urlquery.
This composition is also the first piped template function in any ExternalSecret in this estate.
Every other one is a bare substitution, so the rendered Secret is inspected after a sync rather than
assumed correct.
Changing the shape of this credential is a two-step across a merge boundary. Step 3 is out-of-band
from ArgoCD — it is an operator running vault kv put — so it cannot be folded into the same
change as step 1.
Merge the manifest change and let ArgoCD sync it, so nothing in the cluster still references
the property being removed.
Verify the rendered Secret before deleting anything.
litellm-secrets must read SecretSynced. Then confirm the rendered DATABASE_URL still
authenticates — a proxy pod that is serving requests and a migration Job that completed are the
practical evidence. This step is mandatory rather than a formality: a template function inside an
ExternalSecret is unproven in this estate, and this is where a silent escaping defect surfaces.
Only then remove the retired property, by writing back a version naming only the properties
that remain. Piping the current version through jq carries the two live values across
unchanged and keeps them out of argv and shell history:
vault kv put writes a new version containing exactly the keys given, so the write itself is
what removes a retired property. vault kv patch merges into the current version and would leave
the property in place — it is the wrong verb here.
Do not re-run scripts/seed-litellm-vault.sh to do this. A bare run of that script
generates a fresh CNPG password, a fresh LiteLLM master key, a fresh Valkey password and a
fresh value for every one of the eight workload virtual keys, and writes all of them. (The
script’s own usage text says so: the bare path is “first-time seeding of an EMPTY estate”.)
Steps 1 and 2 above are only satisfiable once ArgoCD has synced and the
proxy is authenticating — that is, once the system is live — so running the script here
rotates ten live credentials to drop one unread property. The master key changes under
every admin client, the Valkey password changes under a
running cache client that will not re-authenticate until the pods restart, and every workload
holds a credential LiteLLM no longer recognises. That script’s bare path is the first-time
seeding tool; the narrow write above is what this step needs, and --all-consumers is what the
estate’s workload keys are seeded with. If you do run the bare path anyway, treat it as a
full-estate rotation: restart the proxy, re-export LITELLM_MASTER_KEY, and re-register every
workload key.
Removing the Vault property while an ExternalSecret still references it does not delete the
Kubernetes Secret. External Secrets deletes a target only when the provider returns no data at all,
and a missing property inside an existing secret is a fetch error, not an empty data map. So:
the ExternalSecret goes to SecretSyncedError and stops refreshing;
the last-synced Secret persists, so running pods keep working and there is no outage to
notice;
changes to the master key or the provider keys silently stop propagating;
any event that recreates the Secret — an ArgoCD prune and recreate, a namespace rebuild — leaves
it with no source at all.
A quiet stale-config failure is worse than a loud one. The warning sign is
kubectl get externalsecret -n litellm showing anything other than SecretSynced.
vault kv put does not erase history. A property removed by writing a new version remains readable
in prior versions of that path to anyone with read access on it. Deleting the property is
therefore a hygiene improvement, not a purge. Destroying the old versions is a deliberate, separate
action:
Read this before running either command below.vault kv metadata delete removes every
version of the path, including the current one — it destroys the live master_keyand the
live openrouter_api_key, not only the retired property you came here to purge. The master key
is regenerable: scripts/seed-litellm-vault.sh mints a new one. openrouter_api_key is not.
That script takes it from the environment (openrouter_api_key="$OPENROUTER_API_KEY",
scripts/seed-litellm-vault.sh:91), and nothing in this repository can reproduce the value. If
you are not already holding it, it has to be re-issued at OpenRouter.
Prefer the version-scoped vault kv destroy: it removes the versions you name and leaves the
current one — and therefore both live properties — intact.
Terminal window
# destroy specific versions of the path, or delete the path's metadata and all versions
If you do run vault kv metadata delete, re-seed immediately afterwards: until you do, the
ExternalSecret behind litellm/litellm-secrets has no source at all.
The deletion PR (#2049) removed agentgateway in one change: both ArgoCD Applications and their
trees, the two namespaces, the scrape job, the dashboard and alert tiles, the Terraform references in
tf/keycloak and tf/uptime-kuma, the mcp-gw entry in the k3s OIDC audiences, the CoreDNS
split-horizon rewrites, and the readers of the agentgateway Vault path (repointed at
cluster/litellm, D-74). One PR rather than three because the rollback is one operation (D-71):
git revert of that PR’s merge commit, plus a Vault undelete (D-77). There is no second plane to
repoint a client at — llm-gw.fzymgc.house and mcp-gw.fzymgc.house no longer resolve (D-72), so
“repoint the client” is not a rollback of anything. See the ADR
agentgateway is decommissioned.
Read this before running anything below.secret/fzymgc-house/cluster/agentgateway was
removed with vault kv delete — a KV v2 soft delete. The deleted version is still on the
mount and can be undeleted indefinitely: the mount runs with max_versions 0 and
delete_version_after 0s, so nothing ages it out. The vk_* accept-list keys and llm_admin_key
come back with that version. NEVER run vault kv destroy or vault kv metadata delete against
this path. Either one turns a reversible decommission into an unrecoverable one: the accept-list
values exist nowhere else, and a revert would then restore ExternalSecrets with no source.
Terminal window
# 1. the tree — restores both Applications; ArgoCD re-creates the apps, the CRDs and the namespaces
gitrevert-m1<deletion-merge-sha>
# 2. the Vault path — <version> is the version that was current at delete time; read it first
# the vk_* accept-list and llm_admin_key return with the version; ExternalSecrets resync within refreshInterval
At the time of deletion (2026-09-10) the current version was 13;
vault kv undelete -mount=secret -versions=13 fzymgc-house/cluster/agentgateway restores it.
After the revert and the undelete, check:
ArgoCD recreates agentgateway-controller (sync-wave −1: the CRDs and the controller) and
agentgateway (the Gateway, the routes, the policies, the ExternalSecrets), and both namespaces
return: kubectl get application -n argocd agentgateway agentgateway-controller reads
Synced/Healthy.
ExternalSecrets — the restored agentgateway-vkeys and agentgateway-openrouter-vkeys resync
from the undeleted version within their refreshInterval; kubectl get externalsecret -n agentgateway reads SecretSynced on every row. The readers the PR repointed at
cluster/litellm are repointed back by the revert, and both paths hold the seven shared
properties, so nothing waits on a copy.
The Terraform and Ansible halves are NOT restored by the revert — re-apply them by hand. The
revert restores the source files, but the HCP workspaces and the k3s API server hold applied
state. Re-run the main-cluster-keycloak and main-cluster-uptime-kuma applies (the mcp-gw
audience mappers, the oauth2-proxy dashboard entry and redirect URI, the uptime-kuma monitor),
then the k3s-config play (the mcp-gw entry back in k3s_oidc_audiences) — Keycloak first, then
the k3s roll, so no live token carries an audience the API server rejects.
Host DNS — the HostMapping CR in the restored tree re-registers llm-gw, mcp-gw and
openrouter-gw on the router, and the CoreDNS split-horizon rewrites return with the
coredns-custom sync. Until both have happened a LAN client still gets NXDOMAIN.
SUPERSEDED 2026-09-11 (05-04). The HostMapping half of this step is FORFEITED: Phase 05 removed the operator,
both CRDs in its API group and the router-hosts-agent Vault identity, so a reverted tree restores a CR against a
kind that no longer resolves — and #2049 had merged, so the window was already closed. That also takes step 1
with it: git revert -m 1 restores argocd/app-configs/agentgateway/hostmapping.yaml and its kustomization.yaml
entry, so the restored agentgateway Application fails sync wholesale on the unresolvable kind — the Gateway, the
routes and the ExternalSecrets do not come back. This rollback is no longer executable as written. Recovering
agentgateway now means re-authoring the tree without hostmapping.yaml before pushing; the Vault undelete (step 2),
the Terraform and Ansible re-applies (step 3) and the coredns-custom half of this step are unchanged.
This section is the undo for the decommission itself. A client-side problem is fixed on the client,
and a LiteLLM-side regression is a GitOps revert of the route’s own commit — neither needs this
section; the MCP client runbook says the same.
This procedure is UNREHEARSED. No live rotation of this credential has ever been performed on
this estate. Read that literally. Every step below is derived from the manifests, the chart values
and the seed script — each one says which — and none of it is the report of an execution. Rotate
from this document when you have a real reason to (a compromise, a departure, a policy clock), and
expect to correct a step. It is not a rehearsed drill and it is not written as one.
No rotation was staged to make it one. Plan 03-15 was scoped to rotate this password once, in
order to “prove the escaping”, and the operator descoped that rotation on 2026-08-29. The
reasoning is at the end of this section and is carried as residual R-20 in
docs/engineering/specs/2026-08-16-litellm-llm-parity-matrix.md. The short version: the only part of
this that is ours is the escaping, and it holds by construction.
The urlquery escaping holds by construction, not pending a rotation
Do not read the sections above as saying the escaping awaits a live rotation. It does not.
The escaping is also safe by construction, not by luck: urlquery mangles exactly one thing — a
space, which it renders as +, a literal plus inside a userinfo component — and the generator’s
alphabet (string.ascii_letters + string.digits + "@:/?#[]!$&'()*+,;=%") contains no space. The
seed script says so at its own point of generation: “excluding the space is precisely what makes
urlquery correct.”
Three constraints keep it that way:
The password comes from the generator. A hand-typed value is outside the alphabet entirely,
and a space in one is exactly the silent defect. Step 1 below therefore generates; it does not
prompt.
Do not widen the alphabet without re-checking that every added character survives urlquery
inside a URL userinfo component — in particular, no space and nothing non-ASCII.
Keep urlquery on the password and not on the username in the DSN template in
argocd/app-configs/litellm/secrets.yaml.
Nothing checks the second or the third.
What this procedure does NOT prove, and is not ours to prove
Three of the properties a live rotation would exercise are third-party software behaving as
documented, and nothing in this repository asserts them:
that CNPG applies a changed passwordSecret to the database role;
that LiteLLM reconnects to Postgres after the credential changes;
that virtual-key scope, budgets and spend history survive that reconnect (they live in Postgres
rows the reconnect does not touch).
They are recorded here as expectations, not as claims anything checks. A test on any of them would
go red when a vendor shipped a change, not when we made a mistake.
Step 1 — generate the new password and write it to the single source. Generate it; do not type
one. A hand-typed value is outside the generator’s alphabet, and that is the one way to
reintroduce the space defect. This generator is the same one at
scripts/seed-litellm-vault.sh § “Generate the CNPG password” — keep the two alphabets identical;
nothing checks that they agree.
The block below is unindented on purpose: the PY heredoc terminator must start at column zero, so
copy it as it stands rather than into an indented context.
Terminal window
NEW_PGPASS="$(python3-<<'PY'
import secrets, string
alphabet = string.ascii_letters + string.digits
specials = "@:/?#[]!$&'()*+,;=%"
pw = [secrets.choice(specials) for _ in range(3)] + [secrets.choice(alphabet) for _ in range(29)]
jq hands the value to vault on stdin, so — as in step 3 of the ordering contract above — the
password reaches neither argv nor shell history. Writing it as a password="<new>" argument instead
would put a live credential in /proc/<pid>/cmdline and in the operator’s history file.
vault kv put here is deliberate and correct: this path carries exactly username and password,
and the jq -n above reconstructs both, so nothing is dropped.
Hold the outgoing value before you overwrite it. The recovery path from a failed rotation is
another write, and it needs the previous password. vault kv get on the prior version supplies it
as long as that version has not been destroyed.
Step 2 — CNPG applies it to the database role on its own; there is no manual ALTER ROLE.
Derived from configuration: litellm is a CNPG managed role
(argocd/app-configs/cnpg/postgres-cluster.yaml:104-111 — ensure: present, passwordSecret: main-litellm-credentials), and that Secret is materialised from this same Vault property by the
main-litellm-credentials ExternalSecret (argocd/app-configs/cnpg/users-litellm.yaml), whose
target carries the cnpg.io/reload: "true" label. UNREHEARSED: the latency from the Vault write
to the applied role password has never been measured here.
Step 3 — both ExternalSecrets refresh independently, and the order between them is not
guaranteed.main-litellm-credentials (namespace postgres) and litellm-secrets (namespace
litellm) each carry refreshPolicy: Periodic with refreshInterval: 5m, on unrelated clocks.
Either can land first, and either order has a transient window: the proxy holding a new
DATABASE_URL before CNPG has applied the role, or the role changed before the proxy has the new
URL. Confirm both:
Both must read SecretSynced. UNREHEARSED: the width of that window is unmeasured. Expect the
proxy to log authentication failures inside it.
Step 4 — the proxy restart is automatic; do not assume you must trigger it.DATABASE_URL
reaches the pod as a secretKeyRef env var, which is snapshotted at container start, so a refreshed
Secret never reaches a running process on its own — and that is exactly why the Deployment carries
reloader.stakater.com/auto: "true" (argocd/app-configs/litellm-chart/values.yaml:41-42). Stakater
Reloader supplies the rollout. The manual fallback, if the annotation is ever removed or Reloader is
down:
Step 5 — verify what actually matters, not just liveness.GET /health/readiness reporting
db: connected is the reconnect; a bare 200 from /health is not, because
allow_requests_on_db_unavailable keeps completions flowing on cached auth (see “When the database
is unavailable”). Then confirm the state the database holds: an existing workload key still
authenticates, GET /key/info reports the same models, aliases, limits and budgets, and its spend
total has not gone backwards.
The plan-03-10 database-outage alert is the signal if the reconnect does not happen. Check it is not
already firing before you start.
Why no rotation was performed, and what that leaves open
The rotation plan (03-15) existed to prove five properties. Split by ownership, two are ours and
three are not:
Property
Ours?
Status
urlquery escaping survives the generator’s alphabet
yes
Holds by construction: the generator’s alphabet contains no space — see the escaping section above
The alphabet/escaper coupling cannot drift
yes
Holds only while the rotation block’s alphabet and scripts/seed-litellm-vault.sh’s stay identical; nothing checks that
CNPG serves a changed credential
no
Third-party; not asserted
LiteLLM reconnects after a credential change
no
Third-party; not asserted
Virtual-key state survives the reconnect
no
Third-party; not asserted
Rotating a live credential — irreversibly, against the database holding every workload key’s scope,
budget and spend — would have sampled the first two more weakly than the construction holds them, and
would have “proven” the last three only in the sense of watching a vendor’s software work once.
The plan itself half-knew this: its own reporting task required the executor to state whether the
generated password happened to contain an escape-requiring character, and to record that the
escaping was not proven if it did not. A justification written to accommodate the possibility that
it does not hold is not a justification for a one-way act on production.
What remains genuinely open is this procedure, not the escaping. Steps 2 through 4 have never
been executed here. The first operator with a real reason to rotate runs them, and corrects this
section from what they find.
Operator ruling, 2026-08-30: SC#3’s database half is met by the offline proof
Roadmap success criterion #3 asked for a rehearsed rotation of both a virtual key and the
litellm database password. The virtual-key half was rehearsed against fovea that day and its
measurements are in “Rotating a virtual key”. The database half was put
to the operator as a decision, per D-54, and the ruling was to amend the criterion rather than
perform the rotation. Recorded here because the alternative — leaving SC#3 reading as met with no
ruling behind it — is the failure this section exists to prevent.
The reason, as given. The only property in this rotation that we own is DSN urlquery escaping,
and it holds by construction: urlquery mangles only a space, and the 81-character generator
alphabet has none, where a live rotation samples a random 32. For the property we own the
construction argument is strictly stronger evidence, not weaker. The remaining three properties
are third-party behaviour we would not test even after a clean live run, and this project tests what
it owns. Against that,
the rotation is one-way and its blast radius is CNPG main in namespace postgres, shared with
mealie, temporal and octopus.
The fovea rehearsal added a figure that widens the gap between what a live run would cost and what it
would prove: the LiteLLM half of a rotation was sub-second, while the delivery half was
63.1 s and never completed without a hand-issued command. The database rotation’s equivalent
legs — the ordering window between two independently-refreshing 5m ExternalSecrets, and the CNPG
reload — are the steps marked UNREHEARSED above, and they are unmeasured for a shared cluster
rather than for a single consumer.
What the offline proof covers, and what it does not. Read this before recording SC#3 as met
anywhere:
Property
Covered by the offline proof?
A generated password survives urlquery escaping
Yes — by construction: urlquery mangles only a space, and the 81-character alphabet has none
The escaping cannot drift from the generator’s alphabet
No — nothing checks the two alphabet copies agree; keep them identical
CNPG serves a changed credential
No — unexercised
LiteLLM reconnects after a credential change
No — unexercised
Virtual-key rows survive that reconnect
No — unexercised
Steps 2 through 4 of the procedure above
No — still UNREHEARSED, and the markers stay
“SC#3 met” does not mean the database rotation has been performed. It has not. Nobody has run
these steps against this estate, and the UNREHEARSED markers above are accurate rather than stale.
What would force a rehearsal: a compromise, a staff departure, or a policy clock.
Rotating merely to rehearse spends the irreversibility for evidence about someone else’s software.
The first operator with one of those three reasons runs steps 2 through 4, and corrects this section
and the markers above from what they find.
Residual R-20 is NOT closed by this ruling and keeps its reopen condition. Amending the criterion
records what the evidence actually supports; it does not retire the residual that says the procedure
is unrehearsed.
Every workload that calls LiteLLM holds its own virtual key (D-36). This section is how one gets
made, scoped, delivered and checked. Eight credentials exist: five in-cluster consumer workloads,
one off-cluster agent profile (jarvis), plus two credentials held on the operator’s workstation
rather than by a pod.
This command owns seeding. It writes each of the eight values into Vault and registers each
one with /key/generate carrying that credential’s scope, in a single invocation. It touches no
core credential. It is safe to re-run: a consumer that is already seeded and already authenticating
is skipped, and a consumer whose value was seeded but never registered is repaired without moving
its Vault version. It prints a per-consumer result table — read completion off that table, not off
the exit code.
Two preconditions, both checkable before you run it.
The lanes are live. The model_list on main carries all eleven entries and ArgoCD has
synced them. GET /v1/models must list fovea-scout, fovea-deepdive and fovea-embed. A key
registered against a lane that does not exist is a 403 at the consumer’s first call.
The proxy answers.GET /health/liveliness (or /health/readiness) returns 200. Do not
check GET /health — it is authenticated and returns 401 by design, so a 401 there is not
evidence of a broken ingress. The command performs this same check itself and exits without
writing anything if the proxy does not answer, because a value seeded and not registered is dead
in LiteLLM and the workload holding it fails closed.
The authenticated half answers 200. With the master key attached, GET /health answers
200. That is what makes the 401 above readable: a 401 on /health means the caller was
unauthenticated, never that the proxy is down.
The bare invocation is NOT the seeding step.scripts/seed-litellm-vault.sh with no flag
rotates the CNPG password, the LiteLLM master key and the Valkey password alongside every workload
virtual key. On a live estate that is an outage, not a seed: the master key changes under every
admin client, and every workload’s value goes dead in LiteLLM
until it is re-registered. The bare path exists for first-time seeding of an empty estate. If you
are seeding a running estate, the command is --all-consumers.
Why this is written down as an owner. Plans 03-11, 03-12 and 03-13 each told the operator to
“run that consumer’s seed step” and no plan owned seeding as a whole. Nothing failed until cutover
time read cluster Vault and found no litellm_api_key on any of the consumer paths then in the
table. An unowned
side-effecting step is invisible until something reads it, so seeding is owned here, by this one
step. --consumer <name> below remains available for a single rotation or a repair — it is not the
routine path, and it seeds without registering.
Keep this section and scripts/seed-litellm-vault.sh both naming --all-consumers as the seeding
command, and keep the mint bodies below equal to the ones the script registers. Nothing checks
either.
Vault owns the VALUE. LiteLLM’s Postgres owns the SCOPE. That split is D-52, and it reverses
the earlier D-37 shape in which LiteLLM minted the value and Vault held a copy. D-37 was decided on
the stated basis that LiteLLM accepts no caller-supplied key value; that basis is false. key is a
first-class field on /key/generate, validated rather than ignored — it must start with sk- and
be at least 16 characters, or the call returns 400.
Fact about a key
Authoritative store
Changed by
the secret value itself
Vault, on the consumer’s own path
scripts/seed-litellm-vault.sh --all-consumers (all eight) or --consumer <name> (one). On a live key, neither: use revoke-and-re-mint, “Rotating a virtual key”. NOT/key/regenerate or /key/{key}/regenerate — Enterprise-gated on this estate and returns HTTP 500
models allow-list, aliases
LiteLLM Postgres
/key/generate, /key/update
budgets, rate limits, spend
LiteLLM Postgres
/key/generate, /key/update, and normal traffic
delivery into the pod
Vault → ExternalSecret → k8s Secret
editing that consumer’s ExternalSecret
Seeding the value first is what keeps the two stores from drifting. Under D-37 a regenerate that
nobody copied back into Vault left two live stores disagreeing with no signal; seeding first means
the value in Vault is the value, and registration only attaches scope to it.
This is the mechanism --all-consumers above performs for all eight at once. Follow it by hand when
you are repairing or rotating exactly one credential; do not follow it eight times as a way of
seeding the estate.
Seed the value in Vault.
Terminal window
scripts/seed-litellm-vault.sh--consumerfovea
Name the flag. A bare re-run of that script rotates the CNPG password, the master key, the
Valkey password and every workload virtual key — running it mid-cutover to seed one consumer
takes the whole plane down. --consumer writes exactly one property with
vault kv patch -method=rw, touches no core credential and no other consumer, and refuses to
overwrite a property that already exists unless you pass --rotate. That refusal covers the
--consumer path only; the bare invocation has no equivalent and is not going to grow one.
Read the seeded value into a shell variable, never onto the screen.
Register it with /key/generate, passing the seeded value as key along with the scope.
The per-credential bodies are below.
Verify, per “Verifying a minted key”.
Only then edit the consumer’s manifests, per “Staging a consumer’s credential”.
Order matters and the failure is silent. The Vault property must exist before any manifest
references it. An ExternalSecret pointing at a property that is not there goes SecretSyncedError
and stops refreshing, while the running pods keep working on their old Secret — a stale-config
failure with no outage to notice. That is why every cutover seeds first and edits manifests second.
The cutover plans all follow the same stage-then-switch sequence. It is written once here rather
than three times there.
Seed the Vault property (--consumer <name>) and register the key.
Add the new Secret key alongside the existing one in that consumer’s ExternalSecret. Do not
replace it.
Wait one refresh interval (refreshInterval: 5m) and confirm the live Secret carries both
keys.
Only then flip the workload’s secretKeyRef.key and its base URL together, in one change.
What this prevents: a workload restarting onto a credential that the plane it is still calling
does not accept. Flipping the base URL and the key separately guarantees one intermediate state
where the pod presents the wrong credential to whichever plane it reaches.
What it preserves: the old Secret key stays populated for the whole bake, so a rollback is a
key-name flip against a Secret that already holds both values — not a wait on ESO to re-sync a
property somebody deleted.
The allow-list is checked BEFORE the key’s aliases resolve
Every key’s models list must contain BOTH the string the client already sends AND the lane
name.can_key_call_model runs in the auth dependency against the model taken from the request
context; the per-key alias rewrite happens later, in add_litellm_data_to_request, and per-key
aliases are not consulted by the access check at all.
Listing only the lane name produces a 403 naming a model the client never sent — a symptom
that reads like a routing bug and sends you to model_list, which is not where the fault is.
Both strings, every time.
soft_budget is the signal and max_budget is a blast-radius stop, not a cost control (D-46).
Spend is flushed to the database on a measured 4.2–10.5 second window, so enforcement is a bound
with overshoot, not a ceiling. A max_budget set close to expected spend is a fuzzy limit and, on
a bursty consumer, an availability risk — the key starts refusing requests over an accounting lag.
Put soft_budget near expected spend and max_budget well above it.
Per-model budgets inside a key (model_max_budget) are Enterprise-gated, which is why budgets
land per key rather than per lane. budget_limits and throttle_on_budget_exceeded are ungated in
the type definitions but their runtime paths were not measured — do not rely on them here.
Embedding lanes under-report. LiteLLM records $0 for an embedding call unless the lane
declares pricing, so octopus, fovea-embed, engram-embed and gemini-embed spend does not
accrue against these budgets today. Do not read a low number on an embed-heavy key as evidence of
low usage.
Raising a budget on a live key takes two calls, not one.POST /key/update moves
max_budget, which lives on the key row. It accepts soft_budget in the same body and silently
discards it: the response echoes "soft_budget": null — the key row’s own field, which is null on
every key that has one — and the linked litellm_budgettable row keeps its previous value. Read the
result back from /key/info at .info.litellm_budget_table.soft_budget, never from the update
response, which cannot distinguish “discarded” from “not set here”.
Moving the signal means updating the budget row by id. /budget/ is not published by the
IngressRoute, so this is an in-cluster hop:
-d'{"budget_id":"<.info.budget_id from /key/info>","soft_budget":50.0}'
The two rows carry disjoint halves of the pair: the budget row holds soft_budget and reports
max_budget as null, while the key row holds the max_budget that actually enforces. Updating
either one does not move the other.
the value seeded in Vault (D-52). Omit it and LiteLLM mints its own, and the two stores diverge immediately
key_alias
the only human-readable per-key label on the spend metrics. Without it every dashboard groups by an opaque hash and KEY-03 is unanswerable
key_type: "llm_api"
a data-plane key. /key/, /team/ and /user/ are published at the ingress now, so a leaked workload credential without this can mint more keys
models
both strings per lane, per the warning above
aliases
maps what the client already sends to its lane, so most consumers need only a base-URL change
rpm_limit, tpm_limit
a runaway loop is bounded in requests before it is bounded in dollars
soft_budget, max_budget, budget_duration
per the doctrine above
duration
TTL. null on every workload key — see below
Workload keys carry no TTL, deliberately. A duration on a key held by a pod is a scheduled
outage: nothing in this estate re-mints a key on expiry, so the workload simply starts failing
authentication at a time nobody chose. The two workstation credentials do carry one, because a
human is present to re-mint them and an abandoned workstation credential should stop working.
Three lanes, so six models entries — and the document states them because “both strings” is
easiest to get wrong when there are three pairs:
or-deepseek-v4-flash-zdr, fovea-scout, or-glm-5-2-zdr, fovea-deepdive,
qwen3-embedding-8b, fovea-embed.
The seeded value reaches no argument vector, and the shape above is what makes that true.printf is a shell builtin, so it starts no process at all; python3 and curl both take the
value on stdin — python3 reads it and the body from the pipe and carries only its program
text in argv, and curl takes the assembled document with --data-binary @- rather than -d.
This is the rule scripts/seed-litellm-vault.sh follows at write_virtual_key and http_call,
for the reason it states there: argv is readable in /proc/<pid>/cmdline for the life of the
process.
This block previously used jq -n --arg key "$SEEDED_KEY" and -d "$(…)" while claiming the value
never reached argv. It reached two: --arg puts it in jq’s, and the command substitution then
put the whole rendered body in curl’s — the two constructions the seed script rejects by name
(scripts/seed-litellm-vault.sh, the write_virtual_key and http_call comments). A documented
safety property that does not hold is worse than a documented gap, because it stops the reader
looking.
Two things it still does not buy you. The master key is passed with -H and so does reach
curl’s argv; closing that needs the three-field stdin helper the seed script uses, and the hand-run
commands in this document keep the -H form for legibility. And del(.key, .token) on the
response is not cosmetic: /key/generate echoes the registered value back, so an unfiltered
response puts live credential material in the operator’s scrollback.
fovea’s guard role reuses the scout lane, so it needs no fourth pair.
octopus’s soft:max pair is sized from measured demand, not scaled from a default. It departs
from the roughly 10x ratio the other consumer rows carry, and that is deliberate: soft_budget
sits on observed 30-day spend so the signal means something the first time it fires, and
max_budget is the blast-radius stop the doctrine asks for, at the ceiling the claude-code key
already uses. A ratio preserved for its own sake would put the stop far above any spend this
consumer can plausibly reach, which is not a stop. Do not normalise the pair onto the other rows.
Seed with --consumer octopus; the value is litellm_api_key on
secret/fzymgc-house/cluster/octopus.
octopus’s model strings are NOT in its Deployment. Unlike every other
consumer here, octopus carries no model env var at all — an env-var sweep over
its Deployment returns only EMBEDDING_DIM and EMBEDDING_BATCH_ITEMS and
looks like a workload with no model config. The real list is rows in the
available_models table of its Postgres database:
psql-doctopus-c'SELECT "modelId", category, "isActive" FROM available_models;'
ollama/bge-m3 is the platform-default embedding row — a legacy alias naming
the decommissioned seattle ollama box. It worked on the retired gateway only
because its openrouter-embed-bge backend pinned provider.openai.model, which
collapsed any client string onto baai/bge-m3. LiteLLM matches the
client-sent string instead, so every row above must appear in models here.
Re-derive this list from that table, never from the Deployment, and never from
gateway telemetry — the retired gateway logged the POST-resolution model name,
so its gen_ai.request.model showed baai/bge-m3 no matter what the client sent.
text-embedding-3-large is NOT in that table either, and octopus sends it.
It is octopus’s hard-coded OpenAI default (EMBEDDING_DIM ?? 3072 // default text-embedding-3-large) on a code path that never consults available_models,
so enumerating the table alone still misses it — it surfaced only as a live 403
on a real PR review after the cutover. Its alias MUST point at octopus-embed:
that lane is bge-m3 at 1024 dimensions, matching EMBEDDING_DIM=1024 and
the existing code_chunks collection. Letting the string reach a real
3072-dimension OpenAI model would write vectors the collection cannot hold.
Verify with jq '.data[0].embedding | length' — it must return 1024.
The general lesson: a consumer’s model strings can come from a config table AND
from hard-coded defaults. Enumerate from BOTH, and treat a post-cutover 403 as
the discovery mechanism of last resort rather than the plan.
Mealie is absent from D-36’s original five-key list and is an LLM consumer, so it gets a key here —
the sixth workload key D-36’s own per-workload boundary implies once the consumer inventory is
complete. It reaches LiteLLM through headroom, which forwards the caller’s key unchanged, so this
key’s alias still appears on the spend metrics.
Why the second entry is a concrete model and not a lane name. mealie and karakeep have no
dedicated model_list entry — unlike fovea, octopus and engram, they are served by the
openrouter/* curated wildcard. The obvious reading of “both strings” would put openrouter/* in
models, which hands a recipe app the entire OpenRouter catalogue and defeats the per-workload
boundary the key exists to draw. Naming the resolved model instead keeps the grant to one model
while still satisfying the access check on both the client-sent string and the resolved one.
Same wildcard-lane reasoning as mealie above. karakeep also reaches LiteLLM through headroom.
karakeep is the one consumer that sends TWO model strings, so it needs two pairs.INFERENCE_IMAGE_MODEL is or-qwen3-vl-8b and INFERENCE_TEXT_MODEL is or-deepseek-v4-pro
(argocd/app-configs/karakeep/deployment.yaml:117-120), and both auto-tagging and
auto-summarization are on. An earlier revision of this body listed the vision pair only; measured
against the live key, or-deepseek-v4-pro returned 403 key_model_access_denied — that is,
karakeep’s text tagging and summarization would have failed closed the moment it cut over, while
image tagging kept working, which reads as a model-specific upstream fault rather than as a key
scope gap. Detection: POST /v1/chat/completions with karakeep’s own key and
{"model": "or-deepseek-v4-pro"} must return 200, not 403.
Three lanes: the summarize lane and both embed lanes.
engram and fovea send the same string to different lanes. Both clients send
or-deepseek-v4-flash-zdr; engram’s key aliases it to engram-summarize and fovea’s to
fovea-scout. That is D-31 working as intended — one model_list entry per lane, resolved
per key, so spend attributes to the consumer rather than to a shared upstream. Do not “simplify”
the two lanes into one; it would look like tidying and would silently destroy per-consumer
attribution.
gemini-embed is listed but not aliased, so the operational reindex and gap-fill Jobs can name the
general lane directly while ordinary traffic on gemini-embedding-2 resolves to engram’s own lane.
The two Gemini lanes serve as of 03-13, which wired GEMINI_API_KEY into the proxy environment
from the paid Vault property gemini_api_key — then read cross-app from the agentgateway path, so
LiteLLM and the retained rollback route presented one credential rather than two copies that could
drift; since Phase 5 (D-74) the property lives on secret/fzymgc-house/cluster/litellm with the
other six shared upstream credentials. Before 03-13 the lanes were declared and NOT usable:
os.environ/GEMINI_API_KEY resolved to None, which does not raise, so both lanes failed at request
time while every other lane kept serving. If that env entry is ever dropped, that silent lane-local
failure is the shape it takes — do not repoint an engram embedding call site at a build lacking it.
Seed with --consumer agent-memory — engram’s Vault path is
secret/fzymgc-house/cluster/agent-memory, and its memory-mcp-litellm ExternalSecret reads
litellm_api_key from that path. (Until Phase 5 it read the embedder key from a different app’s
path — the retired gateway’s vk_engram_embedder — which is why its cutover was a two-field change
rather than the one-line property: change every other consumer needed.)
The only consumer that is not a cluster workload. jarvis is a hermes agent profile running on the
seattle Mac mini (seanb4t/hermes-fleet, profiles/jarvis/config.yaml), reaching this proxy over
the tailnet.
No aliases. The node names the lane directly — model.default: jarvis in its config.yaml is
the only model string it ever sends, which is the point of the per-profile alias: repointing jarvis
at a different upstream is an edit to model_list here, applied by Argo, never an edit on the node.
All three lanes are on the allow-list, and that is load-bearing.router_settings.fallbacks
sends a failed jarvis request to jarvis-fallback-kimi then jarvis-fallback-deepseek, and the
allow-list is checked against the RESOLVED lane — so a key permitted only jarvis would get a
refusal exactly when the fallback was supposed to save it. Change the allow-list and the fallback
list together.
max_budget: 5.0 is a requirement, not a tuning knob. hermes-fleet GW-02 specifies a hard $5.00
monthly ceiling on this profile; raising it breaks the requirement rather than tuning it. The
doctrine above still applies to the pair, and it is soft_budget that carries it: at 0.5 the signal
fires at a tenth of the stop — the 10× ratio five of the other seven keys already run (fovea,
karakeep, mealie, agent-memory, openrouter-passthrough; claude-code is 8× and octopus 4×).
Tune soft_budget after the first month against real spend. If the key starts refusing before the
month ends, that is GW-02 working — take it to hermes-fleet, not to this row.
object_permission.mcp_servers scopes the key to engram alone, so the profile’s MCP config can
reach durable memory and nothing else on the gateway. A key with no object_permission reaches no
MCP server at all; see MCP Gateway Clients § “Key scoping”.
Its Vault path is secret/fzymgc-house/hermes/jarvis, and nothing creates that path for you.
Every other consumer’s path already exists because the app’s own secret created it. This one is
created by a single operator write that also carries the node’s engram_token, and it must happen
BEFORE the seed step:
1. vault kv get -format=json secret/fzymgc-house/hermes/jarvis # does it exist?
2. absent -> vault kv put secret/fzymgc-house/hermes/jarvis - # create, JSON on stdin
3. scripts/seed-litellm-vault.sh --consumer jarvis # merges litellm_api_key in
Run those as three separate commands and branch on the first one’s result yourself. Never follow a
failed patch with a put: a put replaces the whole document and would erase the sibling credential.
Chaining them as get && patch || put is the specific mistake — in POSIX shell the || arm runs
when either earlier command fails, so a permissions error on the merge falls through to the
replace. Values go in on stdin or from a 0600 file, never in argv.
The property is delivered by Vault Agent under AppRole hermes-jarvis, not by ESO: there is no
ExternalSecret and no pod. The node renders it into a 0600 .env from
deploy/seattle/env.ctmpl in seanb4t/hermes-fleet, which is the file to read when you need to know
what the node actually consumes.
A workstation credential with no consumer manifest, so it lives on the bare cluster/litellm path
under the property claude_code_api_key — not litellm_api_key.
{
"key_alias": "claude-code",
"key_type": "llm_api",
"models": ["openrouter/*"],
"rpm_limit": 120,
"tpm_limit": 1000000,
"soft_budget": 25.0,
"max_budget": 200.0,
"budget_duration": "30d",
"duration": "90d"
}
No alias map. The curated or-* names have no in-repo consumer and no model_list lane of their
own, so workstation clients send native OpenRouter slugs (openrouter/anthropic/claude-opus-4.7)
through the wildcard. If a curated name is wanted later, add it to this key’s aliases and list
both strings in models.
The 90-day TTL is deliberate: a workstation credential that outlives the workstation should stop
working, and a human is present to re-mint it.
The generic passthrough key, replacing the retired openrouter-gw.fzymgc.house host (hl-v4vo,
superseded 2026-09-08). It lives on
cluster/litellm under openrouter_passthrough_api_key.
{
"key_alias": "openrouter-passthrough",
"key_type": "llm_api",
"models": ["openrouter/*", "openrouter-zdr/*"],
"rpm_limit": 60,
"tpm_limit": 500000,
"soft_budget": 10.0,
"max_budget": 100.0,
"budget_duration": "30d",
"duration": "90d"
}
Scoped to two wildcards, not left empty. An empty models list means UNRESTRICTED — every
lane the proxy serves, including every consumer’s ZDR lane. The two wildcards are a real bound:
they fail closed against a future non-OpenRouter deployment, and they keep this credential isolated
from every workload key, exactly as the lane’s dedicated accept-list did on the retired gateway.
A different shape from every key above, and the difference is the point. The consumer keys in
this section are scoped by models — which LLM lanes they may reach. An MCP client key is scoped by
object_permission.mcp_servers — which MCP routes it may reach. The two scopes are independent:
models says nothing about MCP, and object_permission says nothing about lanes. A key that must
do both carries both.
They live in this section rather than in a second list because they are virtual keys like any other:
the same /key/generate, the same GET /key/info, the same revocation path, the same spend
attribution. Keeping one list of keys is what makes “who holds a credential against this proxy” a
question with one answer.
the route names this key may reach — the entire boundary. Use underscore spellings (clickhouse_ro), the names LiteLLM matches against
user_id
whose key it is. MCP tool-call spans carry it, which is what makes per-caller attribution work on the MCP plane at all
key_alias
the human-readable label on the span and in spend reports, same as every key above
Omitting object_permission reaches nothing, and that is the intended default — the inverse of
models, where omitting it means UNRESTRICTED. Do not carry the models habit across: an empty
models is the dangerous shape, an empty object_permission is the safe one.
clickhouse_rw is the estate’s only MCP write surface, and both ClickHouse MCP instances
validate the same upstream token — so this list is the only thing separating read from write.
Naming clickhouse_rw “to be safe” grants the write surface and nothing downstream objects.
The two identity routes need no extra field on the key.engram and kubernetes relay the
caller’s own Keycloak token to the upstream; the virtual key still admits and scopes, and the
identity rides a second header the client sends. Scoping a key to engram grants reach, not
identity. See
MCP Gateway — Client Setup.
Verify with GET /key/info exactly as for any other key — note it reads the scope back as internal
ids rather than names, which is canonical storage rather than drift:
Authenticate as the key, per “Verifying a minted key” step 1 —
?key=<value> writes the credential into LiteLLM’s access log, Traefik’s, the OTel span’s
url.full attribute at 30-day ClickStack retention, and /proc/<pid>/cmdline. This document
already states that rule; it is applied here rather than restated.
The two workstation properties are distinct on purpose
Five consumer paths carry one litellm_api_key each. One path — cluster/litellm — carries two
purpose-named properties, matching the convention that path already uses for master_key and
openrouter_api_key.
They cannot both be litellm_api_key. A KV path holds one value per property, so the second write
would silently overwrite the first and leave one of the two workstation keys registered in LiteLLM
against a value nobody holds. It would surface much later and far away, as an authentication
failure that reads like a Vault problem. Do not “normalise” these two onto the consumer convention.
Authenticate AS the key; never put it in the query string.?key=<value> lands the
credential in LiteLLM’s access log, in Traefik’s on the way through the public ingress, and in
/proc/<pid>/cmdline for the life of the curl. scripts/seed-litellm-vault.sh refuses that
shape by name at key_authenticates for exactly this reason. With no key parameter,
/key/info reports the key in the Authorization header — key = key or user_api_key_dict.api_key in litellm/proxy/management_endpoints/key_management_endpoints.py
at the v1.96.2 tag this estate runs, whose docstring documents the header form alongside the
query form. The response is the same {key, info} envelope, litellm_budget_table included, so
the jq filter is unchanged. A 401 here means the value is not registered at all, which is
itself the answer to step 3.
soft_budget is NOT on the key row — read it from litellm_budget_table. A
/key/generate carrying soft_budget creates a row in litellm_budgettable and links it by
budget_id; the key’s own soft_budget field stays null forever. Selecting the top-level
field — which this document did until 03-11 measured it — shows "soft_budget": null on a key
that has one, and reads as a dropped field on a mint that was correct. Measured on the fovea
key: top-level soft_budgetnull, litellm_budget_table.soft_budget5.0, which is the
value the mint body sent.
Reachability. One probe call using the client-sent model string returns 200. Use the
string the workload actually sends, not the lane name — that is what exercises the pre-alias
allow-list.
Seed↔register match. The value LiteLLM holds for this key and the value currently at that
consumer’s Vault path must be the same value. Nothing else proves this: a correctly-formatted
value that was never registered, or a value seeded in Vault with a different value registered
in LiteLLM, fails only at the consumer’s first real call, where it presents as an ESO problem.
The check is an authenticated round trip, not a digest comparison — and the difference
matters./key/info does not return the registered value, so there is no second value to
digest. This document previously printed a comparison of VAULT_DIGEST against a LIVE_DIGEST
computed from the same shell variable: both sides were sha256(vault value), the branch could
not take its else arm, and the digest half proved nothing. 03-11 measured that and replaced it.
The real evidence is that the Vault value authenticates and that LiteLLM resolves it to the
expected alias and scope — a value seeded but never registered returns 401, and a value
registered under a different secret returns 401 for this one.
Carry the digest as an identifier for the SUMMARY, not as a comparison. Never print the
value: a procedure that has an operator eyeball two secrets side by side puts both into a
terminal scrollback and a shell history.
Substitute the consumer’s own expected alias for fovea. shasum -a 256 rather than
sha256sum: macOS ships the former and not the latter, and the operator running a cutover is on
a workstation.
Both calls authenticate as the key and neither puts it in a URL — see step 1. This step is
the one place a live workload credential is most certainly present, so it is the worst place to
hand it to an access log.
On MISMATCH, re-register from Vault — call /key/generate again carrying the Vault value.
Do not re-seed: that produces a third distinct value and leaves you further from a match.
Record the result — the value’s digest, the observed statuses and the resolved alias — in
the cutover plan’s SUMMARY. An unrecorded check is indistinguishable from a skipped one.
zdr-probe-embed is a deliberately unsatisfiable harness lane. It serves no consumer, maps to no
route, and its guaranteed failure is the point. No credential minted by this document lists it.
Two controls hold that, and they are different in kind.
A static check over the mint bodies. Extract every models array in this document and assert
the probe lane is a member of none of them.
Scope matters here: the check is over the models arrays, not the whole file. A whole-file
match count of zero is not the property and cannot be satisfied — the live denial step below names
the lane in order to deny it, which is the opposite of granting it. Two criteria that cannot both
hold get one of them quietly dropped, and the one dropped would be this one.
Carry a vacuity guard: extracting zeromodels arrays is a FAILURE, not a pass. An absence
check that ran before any models array existed examined nothing and proved nothing.
A live denial, because a runbook is documentation and not access control. With fovea’s key,
issue one embedding request naming zdr-probe-embed:
The expected outcome is a model-access denial naming the model — not an authentication failure,
and not a 200. Record the observed status and error type. If it returns 200, a harness-only lane is
reachable by a production workload and that lane’s guaranteed failure has become that workload’s
failure.
Short answer: POST /key/delete, and it is not instant — budget about 60 seconds before the
credential is dead on every replica.
Narrowing a key’s models with POST /key/update is not a revocation mechanism here, and
nothing in this document tells you to use it as one.
Why a 200 from an admin mutation is not “applied”. Every key mutation here has two halves that
converge at different speeds:
Half
When it changes
How you observe it
the authoritative row in Postgres
immediately, before the 200 returns
GET /key/info reports the new shape at once
what each replica enforces
the replica that handled the call changes inline; every other replica keeps its own cached copy until that copy expires
only a per-replica probe can see this
replicaCount: 2 (argocd/app-configs/litellm-chart/values.yaml), and the cache that expires is
litellm’s user_api_key_cache. Its in-memory layer has a 60 second TTL — litellm 1.96.2 builds
it as UserApiKeyCache(default_in_memory_ttl=UserAPIKeyCacheTTLEnum.in_memory_cache_ttl.value)
(proxy/proxy_server.py:2002), and that enum member is 60 (proxy_server.py:1394-1395). It is
not the 5 second DEFAULT_IN_MEMORY_TTL, which governs different objects; assuming otherwise
gives a bound this deployment never honours.
Measured on this deployment, 2026-08-30, per replica through kubectl port-forward to each pod
rather than through the ingress:
Mutation
Handling replica
The other replica
What the caller sees at the end
POST /key/update narrowing models
—
59.3 s / 60.4 s / 60.2 s over three runs
HTTP 403, a model-access denial
POST /key/delete
0.3 s
60.3 s / 60.7 s over two runs
HTTP 401, an authentication failure
Both are bounded by the same 60 second TTL. /key/delete is still the mechanism to use — it removes
the row, so the credential cannot come back, and its refusal is an unambiguous 401 rather than a
model-scope decision — but the window is ~60 s, not the ~2 s figure quoted in the rotation
procedure below. That 2 s is measured through the ingress, which routes to whichever replica
answers and on this deployment repeatedly picks the one that just handled the delete. It is a useful
figure for “can I re-mint yet”; it is the wrong figure for “is the old credential dead”.
If the credential is believed compromised, do not wait out the TTL. Delete the key, then
kubectl rollout restart deployment/litellm -n litellm — pod restart discards every in-memory cache
and is the only mechanism here that revokes faster than 60 s. That restart is a deployment-level
action.
Residual R-10 is closed by measurement rather than waived: the roadmap’s “applies cluster-wide”
claim is TRUE, only its implied immediacy was false.
After POST /key/update returns 200, exactly one thing is guaranteed: the authoritative row has
changed, and any subsequent GET /key/info reports the new shape. Enforcement is eventually
consistent across replicas with the ~60 s bound above. A caller that reads the 200 as “this scope is
now enforced everywhere” is wrong for up to a minute.
Two admin mutations racing on the same key have no stated ordering guarantee on this deployment
— not in litellm’s documentation and not measured here. An update and a delete, or two updates,
issued concurrently against one key may be applied in either order, and because each is invalidated
lazily per replica, two replicas can briefly disagree about which one won. Nothing in this estate
depends on that ordering, and nothing asserts it. Serialise admin mutations on a single key:
complete one and verify it before issuing the next.
If you ever need to inspect the auth cache in Valkey, know this first, because it costs an hour
otherwise. The valkey-litellm-valkey-* pods set VALKEYCLI_AUTH in the container environment
for the operator’s own user, and valkey-cli 9.0 prefers VALKEYCLI_AUTH over REDISCLI_AUTH.
Supplying the correct default password in REDISCLI_AUTH therefore authenticates as the operator
user with the default user’s password and reports WRONGPASS invalid username-password pair — a
message that says the password is wrong when it is right. Unset it first:
The password reaches the pod on stdin and becomes an environment variable in the pod’s shell. Do
not use valkey-cli -a <password>: that places the credential in argv, where /proc/<pid>/cmdline
and any ps on that node can read it for the life of the process. Findings WR-04 and WR-05
in this phase were both exactly that class of mistake.
A key’s cache entry is named for the sha256 of the key value, so
valkey-cli --scan --pattern '*<sha256-of-key>*' on the primary
(valkey-litellm-valkey-0-1-0; confirm with INFO replication) lists it alongside its
spend:key:* and {api_key:*}:tokens counters.
In-place rotation is NOT available on this estate.POST /key/regenerate and
POST /key/{key}/regenerate are Enterprise-gated in litellm 1.96.2. The check lives in the
running image at
/app/.venv/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py:4771-4778
— if premium_user is not True and not is_master_key_regeneration: raise ValueError(...) — and it
fires before any regeneration work, so nothing in the request body ever reaches the
deprecated-token path. This deployment sets no LITELLM_LICENSE, so both route spellings return
HTTP 500. Measured 2026-08-29.
Decided 2026-08-29: in-place rotation is descoped rather than bought. A licence was not
acquired and a mint-new-then-expire-old overlap was rejected, because global alias uniqueness would
force every overlap to run under two aliases and split KEY-03’s per-alias spend for the duration.
The supported procedure is revoke-and-re-mint, and it has a short outage. That outage is the
cost the operator accepted; it is documented below rather than elided. Recorded as residual R-9
in docs/engineering/specs/2026-08-16-litellm-llm-parity-matrix.md.
No test asserts this gate, deliberately. It is a third-party vendor’s licensing decision — not
our configuration, not our code, nothing we could break. A test on it would go red when LiteLLM
changes their product, not when we make a mistake. This paragraph and R-9 are
where the constraint lives. If you set a LITELLM_LICENSE, this section is what has to change, and
the paragraph you are reading is what tells you so.
Why there is an outage at all. Key aliases are globally unique: a second /key/generate
carrying an alias LiteLLM already holds is rejected with Key with alias '<name>' already exists.
So the replacement cannot be minted under the consumer’s real alias while the old key still exists.
The order is forced — delete first, then mint — and the interval between them, plus the time the
new value takes to reach the running pod, is the outage. Keeping the alias is what keeps every
KEY-03 spend query, tile and alert pointing at one continuous series across the rotation.
That uniqueness is LiteLLM’s behaviour (measured 2026-08-29), not a rule we impose, so nothing in
this repository asserts it — this paragraph is where it is recorded. If a future release ever allows two live
keys to share an alias, the replacement could be staged ahead of the delete and this procedure would
no longer need an outage. Whoever notices that owns rewriting this section.
There is NO overlap window, and that changes what a key’s TTL means. Because the replacement
cannot be staged ahead of the delete, there is no interval in which both values are accepted —
nothing to widen, nothing to tune, and no way to cover an expiry gaplessly. Two consequences for a
key minted with a duration:
Rotating restarts the clock; it does not inherit the remainder. The replacement is a fresh
/key/generate from the recorded mint body, so its duration is measured from the re-mint. Rotate
a "duration": "90d" key with a week left and the new key expires 90 days from that moment.
A key you let expire costs you the same outage as one you rotate — you just do not get to pick
when. So rotate the TTL-bearing keys ahead of their expiry, in a quiet window, rather than
discovering the expiry. Six of the eight consumer bodies carry "duration": null and are
unaffected; the two that are not are the workstation credentials claude-code and
openrouter-passthrough, both "duration": "90d", both currently expiring 2026-11-14 (read
2026-08-30).
The outage: how long, what breaks, what to do about it
REHEARSED 2026-08-30 against fovea. Every figure in this section is a measurement taken
during that rotation, not an estimate. fovea was chosen because it is the one consumer with no
reloader.stakater.com/auto annotation, so its manual restart — the leg the other five never
exercise — was rehearsed rather than reasoned about.
The window runs from /key/delete until the workload’s pod is running with the new value. It is not
one number; it is four legs, and only the last two are worth managing:
Leg
Measured 2026-08-30 (fovea)
Notes
/key/delete → new value serving on the consumer’s own lane
0.5 s — Vault write returned at 0.4 s, /key/generate at 0.5 s, and a completion on or-deepseek-v4-flash-zdr succeeded on the same second
the outage starts here, not when you write Vault. This is the whole LiteLLM half, and it is not the slow part
/key/delete → old value refused
0.3 s on the replica that handled the delete, 2.3 s through the ingress, 61.5 s on the other replica
the ingress figure is an artefact — see “What actually revokes a key” above. Plan revocation against the 61.5 s, and read the 2.3 s as “you may re-mint now”
Vault write → Secret carries it
0.74 s from the force-sync annotation to the ExternalSecret’s refreshTime moving; up to refreshInterval: 5m if you do not force it (step 5)
the leg you can actually shorten, and forcing it is worth ~5 minutes
Secret updated → pod running with it
63.1 s for kubectl rollout restart deployment/fovea to complete, and never without that command — measured, see below
see the propagation section below
So: the LiteLLM half is sub-second and the delivery half is the whole outage. The rehearsed
end-to-end window — /key/delete at 07:47:58 Z to the fovea pod running with the new value at
07:51:26 Z — was 3 m 28 s, and most of that was operator pace between commands rather than any
system being slow. Run the six commands back to back and it is a little over a minute; wait one
refresh interval out instead of forcing it and it is about six. For fovea it is unbounded
until you restart the deployment by hand.
The third leg was demonstrated, not assumed. With Vault and the fovea-app Secret both already
carrying the new value, the running pod’s FOVEA_GATEWAY__OPENAI__API_KEY still hashed to the OLD
value — same container id, restartCount: 0 — so for those three minutes fovea held a credential
LiteLLM had already deleted. After kubectl rollout restart the new pod’s environment hashed to the
new value, and that hash equals the token LiteLLM stores for key_alias: fovea, which is what
makes “the pod holds the registered credential” an observation rather than an inference. The
instrument was an ephemeral debug container reading /proc/1/environ (the image is distroless and
has no shell), hashing in place so no credential left the container, and it was positive-controlled
first: before the rotation, Vault, the Secret and the pod all hashed identically.
What breaks during it. Every call the consumer makes to https://llm.fzymgc.house/v1 is refused
401 — an authentication failure, not a model-access denial, so it is unambiguous in the logs.
Nothing else in the estate is affected: the proxy, the other consumers and the database are all
untouched, and there is no partial or ambiguous state to unwind. The consumer either has a working
credential or it does not.
What to do about the consumer. Decide this before you start, not while the key is deleted:
Best case — the consumer retries.karakeep, mealie and octopus are all background or
interactive-with-retry workloads; a failed enrichment or a failed chat turn is retried or reissued
and the window closes on its own. Nothing to do beyond telling anyone using it.
Queue-shaped work — drain or pause first. If the consumer is mid-batch (an engram corpus
reindex, an octopus review sweep), a 401 storm will burn through its retry budget and mark work
failed rather than pending. Let the batch finish, or pause it, before deleting the key.
fovea — schedule the restart, do not discover it.fovea has no restarter (below). Its
rotation is not “delete, mint, wait”; it is “delete, mint, write, force-sync, restart, verify”,
and the restart is a step you perform, not one that happens to you.
Rotate during a quiet window. There is no mechanism here that makes the outage smaller, so the only
lever is choosing when it lands.
Capture the key’s current shape BEFORE you delete anything. This is what you re-mint against
and what you diff afterwards; once the key is gone the row is gone with it. Read it with the
master key, by alias — the consumer’s own credential cannot read it, see below:
Terminal window
ALIAS=fovea# the LiteLLM key_alias, not necessarily the Vault path segment
GET /key/info authenticated as the consumer’s own key returns HTTP 403 here, and this step
used to tell you to do exactly that. Every one of the eight consumer keys is minted with
"key_type": "llm_api", and LiteLLM restricts such a key to LLM routes:
{"detail":"Virtual key is not allowed to call this route. Only allowed to call routes: ['llm_api_routes']. Tried to call route: /key/info"}. Piped through the old jq '.info | …'
that error produced a file of nulls — and the very next step deletes the key, so the shape
was gone with no way to get it back. Measured 2026-08-30 during the fovea rehearsal; the
/key/list + /budget/list pair above is what actually works.
No credential goes in a URL, and none needs to. The old form kept the value out of the query
string by putting it in an Authorization header; this form does not handle the consumer’s value
at all. ?key=<value> would put the credential in LiteLLM’s access log and in Traefik’s — see
“Verifying a minted key” step 1 for the mechanism and the upstream reference.
Cross-check it against that consumer’s mint body under
“Minting and scoping virtual keys” — that section holds the
authoritative body for each of the eight consumers, and step 3 re-uses it verbatim.
The body is built by python3 reading the value on stdin, not by jq -n --arg key, which
would put the credential in jq’s argument vector — see “fovea” above for why that is
the same hole in a different process.
Write the new value into that consumer’s Vault target FIRST — the same path and property its
ExternalSecret already reads. Vault is the origin of the value and LiteLLM registers it
(D-52, which reversed D-37); step 4 then attaches scope to the value Vault already holds by
passing it as /key/generate’s explicit key field.
This step and the next one used to be the other way round, with /key/generate minting a
value and this step copying it into Vault afterwards. That is the D-37 order D-52 reversed, and
it contradicted both “Where each half of a key lives” and the
recorded mint bodies this procedure tells you to re-use — every one of which already reads its
value out of Vault. Corrected 2026-08-30 during the fovea rehearsal, which followed the D-52
order. The practical difference is what a failure leaves behind: mint-first and a failed Vault
write strands a registered key whose value exists only in your shell; Vault-first and a failed
/key/generate leaves the value safely in Vault, and the fix is to retry step 4.
Derive the path and property from CONSUMER_TABLE in the seed script rather than typing
them, and hand the value to vault on stdin so it reaches neither argv nor shell history:
Terminal window
# CONSUMER is the name the seed script accepts (`--consumer <name>` prints the
# accepted set on an unknown one). For engram that name is `agent-memory`; the
# alias `engram` is a LiteLLM label, not a Vault path.
vault kv patch prints a recommendation to add the patch capability to your ACL policy and
then falls back to read-then-write. That is a warning, not a failure — it wrote. The seed
script passes -method=rw to take the same path deliberately. Confirm with the command’s exit
status rather than the absence of output.
Four of the eight do not follow cluster/<name> + litellm_api_key, which is exactly why
this step derives instead of hardcoding. The previous version of this step wrote
litellm_api_key on secret/fzymgc-house/cluster/${CONSUMER} for all seven. For those four it
wrote a property nothing reads, on a path nothing reads — and because step 2 has already deleted
the old key, following it left the credential revoked in LiteLLM and unrotated in Vault. It also
contradicted “The two workstation properties are distinct on
purpose” in this document and the
CONSUMER_TABLE header in the script. That table is the single source of truth for the
path/property pair; a second list here is the defect plan 03-17 exists to prevent, so this step
reads that one.
printf rather than a here-string: <<< appends a trailing newline to the value, and whether
that newline ends up inside the stored credential is not a question worth having mid-rotation.
The path out of the table already carries its secret/ mount, so no -mount flag.
Register that value with /key/generate, under the SAME key_alias, using that consumer’s
recorded mint body with "key" set to the value Vault now holds. The recorded bodies under
“Minting and scoping virtual keys” already read the value
out of Vault and set key from it, so re-use one verbatim — for fovea that is
“fovea”, and it is the exact body the 2026-08-30 rehearsal replayed.
Do not invent a new alias to “be safe”. A changed alias silently re-points every KEY-03 spend
query and the D-47 spend alert at a series that starts at zero, which reads as a consumer that
stopped working. Keeping the alias is what makes the spend series continuous across a rotation —
see the KEY-03 observation below.
Measured 2026-08-30: /key/delete, the Vault write and this call together took 0.5 s, and a
completion on fovea’s own lane succeeded on the same second. If this call fails, the value is
still in Vault — retry it; nothing is lost.
Force the ExternalSecret to sync rather than waiting out refreshInterval: 5m. This is the
one leg of the outage you can shorten, and it costs one command:
Measured 2026-08-30 on fovea-app: 0.74 s from the annotation to the ExternalSecret’s
status.refreshTime moving, with the Secret carrying the new value by then. Watch that field,
not the clock — kubectl get externalsecret ${EXTERNAL_SECRET} -n ${NAMESPACE} -o jsonpath='{.status.refreshTime}'. Adding this annotation does not put the ExternalSecret
out of sync in ArgoCD; the fovea Application stayed Synced/Healthy throughout.
Consumer
ExternalSecret
Namespace
fovea
fovea-app
fovea
karakeep
karakeep-secrets
karakeep
mealie
mealie-app-secrets
mealie
octopus
octopus-app
octopus
memory-mcp (engram)
memory-mcp-litellm
agent-memory
Confirm the pod actually restarted with the new value — see “Propagation is ESO plus a pod
restart” below. For fovea this step is an explicit
kubectl rollout restart deployment/fovea -n fovea; for everything else Stakater Reloader does it
and you are confirming, not causing. Measured 2026-08-30: 63.1 s for
kubectl rollout status deployment/fovea -n fovea to report the rollout complete, and the pod
demonstrably did not hold the new value before that command was issued.
Verify with a real call on the consumer’s own lanes, then diff the key’s shape. Re-run
step 1’s /key/list + /budget/list pair — notGET /key/info as the key, which returns
403 for a key_type: llm_api credential (step 1). Its output must match
/tmp/rotate-${ALIAS}-before.json on alias, models, aliases, limits and budgets; token,
created_at and spend are expected to differ, and that is what makes this an allow-list diff
rather than a whole-row comparison. A field that silently changed here is a mis-typed re-mint,
and it will surface later as a model-access denial nobody connects to the rotation.
Then call every lane the consumer actually uses, not just one. A re-mint that dropped a
single models entry leaves a credential that works — until the consumer reaches for the lane
that went missing. For fovea the 2026-08-30 rehearsal ended with three calls: chat on
or-deepseek-v4-flash-zdr and or-glm-5-2-zdr, and an embedding on qwen3-embedding-8b, all
200.
Cross-check that the pod holds the registered credential. LiteLLM stores token as the
SHA-256 of the key value, so the token on the row you just read must equal the hash of the
value in the workload’s environment. That turns “the pod has the new key” into a comparison
rather than a belief — and it never requires printing the credential.
Three caveats, each a way the procedure looks fine and is not
You cannot rotate — or revoke — during a database outage, and the outage is what makes it
dangerous./key/delete and /key/generate are both database writes, while completions keep
flowing on cached auth because allow_requests_on_db_unavailable stays true. So a rotation started
during a masked Postgres outage can delete the old key and then fail to mint its replacement,
turning a two-minute outage into one that lasts as long as the database is down — and the proxy will
look healthy the whole time. The signal is the database-outage alert (plan 03-10). If it is firing,
or has fired recently enough that you are unsure, wait. See
“When the database is unavailable”.
Revocation is fast on the pod you talked to and slow everywhere else./key/delete converges to
a 401 in about 2 seconds through the ingress (2.3 s measured 2026-08-30) — but that is the
ingress routing you back to the replica that just handled the delete, and on the other replica the
deleted value kept working for 61.5 s, bounded by the 60 s user_api_key_cache in-memory TTL.
An earlier version of this paragraph attributed the ~2 s to “the 5 s general in-memory TTL”; that
constant (DEFAULT_IN_MEMORY_TTL) governs different objects and never bounded this. Read the ~2 s
as “you may re-mint now”, poll rather than concluding from one call, and plan revocation against
the ~60 s — see “What actually revokes a key”.
A deleted key goes on spending for the length of that window. During the 2026-08-30 rehearsal
the revoked value wrote 30 further spend rows over the 59 seconds after its own deletion, every
one of them attributed to user_api_key_alias: fovea under the key id that had just been deleted.
If you are revoking because a credential is believed compromised, the spend ledger will keep
accruing against it until every replica’s cache expires, and the only lever that ends it sooner is
kubectl rollout restart deployment/litellm -n litellm.
History (until 2026-09, Phase 5). octopus, karakeep and mealie were also accepted on the
retired gateway’s llm-gw listener through OVERLAP entries in its accept-list, added by plan 03-12
as the retained rollback plane and reading each consumer’s litellm_api_key straight from Vault.
While those entries existed a revocation was not complete until the entry was removed in the same
change, and a rotation carried the value along automatically. The plane and the entries went out
with the decommission (D-71, D-48, D-49). There is one plane:
what LiteLLM thinks of a key is what every consumer gets.
Spend attribution across a rotation: what was observed, once
The alias is continuous across a rotation and the key id is not — observed, not gated. This is
the reason step 4 insists on the same key_alias, so it is worth having one recorded observation
behind the claim rather than an argument. Read from LiteLLM_SpendLogs either side of the
2026-08-30 fovea rehearsal:
metadata.user_api_key_alias
key id (first 8 of the SHA-256)
rows
window
fovea
6dfc5f63 (deleted at 07:47:58 Z)
478,099
2026-08-16 15:31 → 2026-08-30 07:48:57
fovea
45249923 (minted at 07:47:58 Z)
from 1
2026-08-30 07:47:58 →
So a KEY-03 query grouped on user_api_key_alias sees one unbroken fovea series across the
rotation; a query grouped on the key id sees a series that ends and a new one that starts at zero.
The key row’s own spend field resets with the key — fovea’s read 1.8397 before and 0.0 after —
which is why the alias is what the tiles and the D-47 alert are keyed on.
Note the overlap in that table: the deleted key’s last row is at 07:48:57, fifty-nine seconds
after it was deleted. That is the same 60 s cache window, showing up in the ledger.
This is an observation, not a gate, deliberately. Asserting it would mean standing a test over
LiteLLM’s asynchronous spend-write timing — a third party’s behaviour, on a path where a red would
report their scheduling rather than our mistake.
Propagation is ESO plus a pod restart, and one consumer has no restarter
A Vault write reaching the Kubernetes Secret is not the workload picking it up. Every consumer
in this estate reads its LiteLLM credential through env.valueFrom.secretKeyRef, and an environment
variable is injected once, at container start. Updating the Secret changes nothing inside a
running pod. The full chain is:
Vault write → ESO refresh (forced, or refreshInterval: 5m) → Secret updated → pod restarted →
new value in use.
The restart comes from Stakater Reloader, which is deployed cluster-wide (kube-system/reloader-reloader)
and acts only on workloads that ask for it. Measured 2026-08-29:
Workload
reloader.stakater.com/auto
Rotation restarts it automatically?
karakeep/karakeep
true
yes
mealie/mealie
true
yes
octopus/octopus
true
yes
agent-memory/memory-mcp
true
yes
fovea/fovea
absent
NO
fovea is the trap, and revoke-and-re-mint makes it worse rather than better. fovea consumes
litellm_api_key as the env var FOVEA_GATEWAY__OPENAI__API_KEY from Secret fovea-app and carries
no reloader annotation, so steps 4 and 5 update Vault and the Secret while the running pod keeps
presenting the value it started with. Under the old grace-period mechanism that produced an overlap
expiring against a pod that never moved. Under this procedure the old value is already deleted,
so the pod is holding a credential that authenticates nowhere and fovea stays down until someone
restarts it — with every intermediate step an operator would think to check reporting success.
Until fovea carries the annotation, a fovea rotation must include
Terminal window
kubectlrolloutrestartdeployment/fovea-nfovea
as step 6, and step 7’s verification must reach the value the pod actually holds, not just the
value you minted. Recorded as residual R-11.
Observed, 2026-08-30. During the rehearsal Vault and the fovea-app Secret both carried the new
value while the running pod’s FOVEA_GATEWAY__OPENAI__API_KEY still hashed to the old one — same
container id, restartCount: 0, a pod that had been up ten hours — for the three minutes between
the ESO sync and the restart. Every intermediate check an operator would think to run was green the
whole time: the ExternalSecret read SecretSynced/Ready, the Secret held the new value, the
Application was Synced/Healthy, and the pod was 1/1 Running. Nothing in the estate reports
this state. After kubectl rollout restart the new pod’s environment matched, and that hash equals
LiteLLM’s stored token for key_alias: fovea.
How to check it, on an image with no shell.fovea is distroless, so there is no kubectl exec -- sh. An ephemeral debug container joined to the target’s PID namespace can read the value the
process was started with and hash it in place, so nothing leaves the container:
'tr "\0" "\n" < /proc/1/environ | sed -n "s/^FOVEA_GATEWAY__OPENAI__API_KEY=//p" \
| tr -d "\n" | sha256sum | cut -c1-8'
kubectllogs-nfovea ${POD} -cenvprobe
Compare that against the token on the key row from step 1 — LiteLLM stores token as the SHA-256
of the value, so they must be equal. Run it once before you start, while Vault, the Secret and
the pod should all agree: a probe that has never been seen to report a match is not evidence when it
reports a mismatch.
No mode of scripts/seed-litellm-vault.sh is the rotation path, and the two modes fail
differently, so a half-remembered rule is worse than none:
A bare re-run regenerates everything at once — the CNPG password, the master key, the Valkey
password and all eight workload virtual keys. It has no overwrite refusal and will not grow one.
--consumer <name> regenerates exactly one consumer’s value, and refuses to overwrite an
existing litellm_api_key without --rotate for precisely this reason.
Either way the new value is dead in LiteLLM until it is re-registered with /key/generate.
Vault-before-registration is the correct order — it is D-52, and step 3 above follows it — so that
is not what disqualifies the script here. What disqualifies it is that it never deletes the live
key: the old value stays registered while Vault changes underneath it, so the next ESO refresh can
hand the workload a value LiteLLM has never seen, with no bounded window and no signal. The procedure
above deletes first precisely so the outage has a beginning and an end. --rotate exists for a key
that was never registered, or for a cold rebuild — not for rotating a live credential.
Setting LITELLM_LICENSE makes POST /key/regenerate available, and with it a grace_period that
keeps the old value valid across the Vault write and the ESO refresh — no outage, and no alias
churn, because it rotates the value in place under the existing row. Three things to know before
adopting it, all read from litellm 1.96.2’s source:
The overlap is one key with two values, not two keys._insert_deprecated_key writes the old
hash against the NEW key’s active_token_id, so spend, budget and RPM/TPM are SHARED
during the window. An operator who reads it as two independent keys will misread every number the
window produces.
The overlap lookup is a database read, through the deprecated-token table, so it fails in
exactly the masked-outage window described in the first caveat above. The database rule does not
relax under a licence.
A 60-second positive in-memory cache (_DEPRECATED_KEY_CACHE_TTL_SECONDS) sits in front of
that lookup, so grace-period revocation is eventual within a minute in both directions — much
slower than the 2 s of an ordinary delete.
Setting the licence is the trigger to revisit this section — nothing automated will tell you.
That is a deliberate choice, not an oversight: the gate belongs to LiteLLM, so there is nothing of
ours whose breakage a test could report. Whoever sets LITELLM_LICENSE owns rewriting the procedure
above around grace_period, updating residual R-9, and re-opening KEY-02 in
.planning/workstreams/litellm-return/REQUIREMENTS.md.
The master key is not just an API credential: the Admin UI session token is a JWT signed with
it. Rotating the master key therefore invalidates every Admin UI session — every logged-in
operator is signed out at once, and anything holding a UI token must log in again. Treat a master-key
rotation as a visible operator-facing event rather than as an ordinary credential change, and do not
schedule one in the middle of an incident being worked through the UI.
Every place the master key is held must be updated in the same change: the Vault property
master_key on secret/fzymgc-house/cluster/litellm and whatever the operator workstation exports
as LITELLM_MASTER_KEY.
LiteLLM does not fail closed when its SSO environment is missing or wrong. It renders an HTML
username/password form (ui_sso.py:992-1004) whose POST authenticates against the master key
itself (proxy_server.py:13662-13679). Measured in-cluster 2026-08-29: GET /fallback/login
returned 200 text/html, POST /login returned 401. Publishing either path puts a master-key
password prompt on the internet-facing host.
Two controls hold this, and they cover different halves. Neither replaces the other.
#
Control
What it proves
Where
1
The exclusions in the IngressRoute rule
Nothing on its own — it is the mechanism. Both exclusion matchers must stay written in the rule; nothing checks that they do
Owner: the cluster operator — as reviewer of the change for the change-driven triggers below,
and on their own initiative for the quarterly run.
When to run: at minimum, after any change to argocd/app-configs/litellm/ingress.yaml, once
ArgoCD reports the IngressRoute synced. Also after a Traefik upgrade, and after any change that adds
a route or a priority to that file.
And once a quarter, whether or not anything changed. Every trigger above is change-driven, so a
quarter with no ingress edit and no Traefik upgrade produces no check at all — and control 2 is the
only proof this property has. A
quarter is chosen because the property only moves when the ingress or Traefik moves, and both of
those already have their own trigger; the schedule is a backstop that bounds how long a regression
from anything else could sit unnoticed, not a second attempt at catching what the triggers catch.
Shorten it if that bound ever stops feeling like enough.
Where to run it from: any shell on the LAN that reaches the host through the Traefik
LoadBalancer VIP — outside Kubernetes, with no port-forward. That is the entire vantage requirement,
so no browser and no operator at a keyboard is needed: an agent or a scheduled job can honour the
quarterly run as cheaply as a person can. Last measured this way on 2026-08-30 from 192.168.20.99,
a LAN client outside the cluster.
Terminal window
curl-sS-ihttps://llm.fzymgc.house/login
curl-sS-ihttps://llm.fzymgc.house/fallback/login
PASS — both answer Traefik’s own 404, which is the discriminator the ingress header records:
status 404, content-type text/plain, no server header, body exactly:
404 page not found
FAIL — anything served by LiteLLM instead of refused by Traefik. The tell is the same
discriminator read the other way: a JSON body, or a server: uvicorn response header, or a
200 text/html carrying a login form. Any of those means the exclusion is not being honoured and
the master-key password prompt is on the network. Treat it as an incident: the ingress is the only
thing standing between that form and the internet.
A Traefik 404 from this host now means an exclusion was hit, not a missing prefix — the root is
published, so there are no other 404s of that shape to confuse it with.
LiteLLM runs with allow_requests_on_db_unavailable: true, which produces a deliberately lopsided
failure mode. Knowing the shape in advance is what stops an operator diagnosing two faults where
there is one.
Read the labels before acting on anything here. Every claim below is marked OBSERVED,
FROM SOURCE or PREDICTED, and they are not interchangeable:
Label
Means
OBSERVED
Someone watched this happen on this cluster, on the date given.
FROM SOURCE
Read out of the running image or the rendered manifest, and cited below. Nobody watched it.
PREDICTED
Neither. Believed on reasoning alone — the first real outage is the first test.
No Postgres outage has ever been induced or observed on this estate. Nothing in this section is
OBSERVED. That is a deliberate 2026-08-30 ruling, recorded with its residual at the end of the
section, not an oversight.
Completions keep flowing — PREDICTED at the proxy, and FALSE at the ingress. Auth resolves from
cache, so requests on already-seen keys should continue to serve through a Postgres outage. That is
the point of the setting. But see “The readiness trap” below: the pods are removed from the Service
while this is happening, so through llm.fzymgc.house the completions do not keep flowing. The
degrade is real at the proxy and invisible behind the ingress.
The outage is loud anyway — the alert RULE exists, its FIRING is unobserved. The DB-failure
hook emits an error span and the plan-03-10 alert (litellm-db-failure) fires on it. That alert —
not a completion failure — is the signal. What is enforced, and what is not:
A constraint (ours). Keep the litellm-db-failure tile built through trcfg so it reads the
trace source, filtered on all three of ServiceName = 'litellm', StatusCode = 'Error' and
SpanAttributes['db.system.name'] = 'postgresql', and named by a litellmTileAlert(...) on a
strict > 0. Keep service_callback: ["otel"] and allow_requests_on_db_unavailabletogether, so a later change cannot buy the signal by removing the degrade. Nothing checks
either.
FROM SOURCE, not gated (LiteLLM’s). That LiteLLM actually emits that span on a database
failure. Read from the running 1.96.2 image: proxy/utils.py:2046 calls
async_service_failure_hook, _service_logger.py:282,309 fans it out over
litellm.service_callback to the OTel logger, integrations/otel/logger.py:491 sets
error_override, and integrations/otel/emitter.py:102 sets
Status(StatusCode.ERROR, …). integrations/otel/model/spans.py maps service postgres (and
batch_write_to_db) to db.system.name = "postgresql" on a CLIENT DB_CALL span named
"{service} {call_type}" — hence postgres <call_type>, which is why the tile groups by
SpanName.
Never seen. The alert has never been observed transitioning to firing. It is the first tile
alert in bootstrap-script.yaml bound to a trace source rather than a metric one, and 03-10
recorded that evaluation as asserted rather than measured. It still is.
The Admin UI login FAILS while completions succeed — PREDICTED. The SSO path requires a database
connection (_raise_if_sso_exceeds_free_user_limit raises db_not_connected_error, HTTP 403, when
the Prisma client is absent), so login should return 403 during a full outage even while the data
plane is fine. An operator who reaches a failing login while completions are healthy should read it
as confirmation of the database outage, not as a second, separate fault in the UI or in Keycloak.
Note that during a total outage the ingress is down too (below), so this is the shape to expect
when the database is degraded rather than gone.
Key rotation and key revocation are both unavailable during an outage — PREDICTED, for the same
reason: /key/delete and /key/generate are database writes, while completions keep serving from
cache. Because rotation here is revoke-and-re-mint, a rotation started during a masked outage can
delete the old key and then fail to mint its replacement, leaving the consumer down for as long
as the database is — with the proxy looking healthy throughout. See the first caveat under
“Rotating a virtual key”. Wait for the database before rotating or
revoking anything.
The readiness trap: a Postgres outage takes the ingress down
FROM SOURCE — read 2026-08-30 out of pod litellm-56956b8677-9vtkc and the rendered Deployment.
This is the most operationally important thing in this section and it is not what the four
predictions above imply.
The rendered probes on Deployment litellm, container litellm:
Probe
Path
Period
Failure threshold
Timeout
readinessProbe
/health/readiness
10s
3
5s
startupProbe
/health/readiness (same path)
10s
30
5s
livenessProbe
/health/liveliness
15s
5
5s
/health/readinessreturns HTTP 503 when the database is disconnected. In
litellm/proxy/health_endpoints/_health_endpoints.py, health_readiness delegates to
_resolve_public_readiness_db, whose body ends:
_db_health_readiness_check caches only the connected result, and only for 15 seconds; a
disconnected result is never cached, so every probe re-checks and none of them recovers while the
database is down.
The consequence, derived from those two facts. Readiness 503s → after failureThreshold: 3 at
periodSeconds: 10 each pod is NotReady within roughly half a minute → both replicas fail
together, because they share the one database → the litellm EndpointSlice empties → the Service has
no backends → the ingress serves nothing. So a Postgres outage is not a “warm keys keep serving”
degrade as seen by a consumer. It is a full data-plane outage of every LLM lane, and the cached
auth that allow_requests_on_db_unavailable preserves is only reachable by
kubectl port-forwarddirectly to a pod, bypassing the Service.
Two corollaries worth having in advance:
The pods do not restart-loop.health_liveliness returns 200 unconditionally unless the
process is shutting down, so the liveness probe stays green and Kubernetes leaves the pods running.
Recovery is a readiness flip, not a rollout.
Any probe you take during an outage must address a POD, never the Service. A Service-level
reading during an outage measures the ingress, not the proxy, and cannot answer what the proxy did.
This is the same trap that made the published /key/delete revocation figure an ingress artefact
(see “Rotating a virtual key”).
Do not “fix” this by flipping allow_requests_on_db_unavailable to false. D-51’s rationale is
that a Postgres blip should degrade to cached auth rather than 401 every caller; flipping it reverses
T-01-07 and makes Postgres a hard dependency for every completion. If the readiness coupling is to be
changed, the thing to change is the probe, not the degrade.
argocd/app-configs/cnpg/postgres-cluster.yaml declares cluster main in namespace postgres with
instances: 3, and it serves six databases: mealie, temporal, octopus, miniflux,
keycloak and litellm (:64-107). A Postgres outage here is an estate event, and keycloak
sharing the cluster means SSO is in its blast radius too. Never degrade the cluster to test a
tenant’s behaviour.
Why the degrade is documented and not measured (2026-08-30)
A scoped, reversible drill was designed to induce this outage: an allow-list egress NetworkPolicy in
the litellm namespace severing only the path to postgres, merged and reverted through ArgoCD. It
was not run. The operator’s ruling, verbatim:
we MUST STOP TESTING functionality we do not own.
This was a scope ruling, not a risk ruling — the distinction matters, because the drill’s design
had by then been made demonstrably safer (single-replica severing, an agent-reversible disarm, a
pre-opened revert PR) and none of that made it in scope. The 503-vs-401 degrade is LiteLLM’s
implementation, and the deciding branch had already been read out of the running image and is
quoted above. Owning the switch (allow_requests_on_db_unavailable, D-51) is not the same as owning
what it switches. Inducing a production outage to watch a vendor branch already read from source is
the thing this estate keeps correcting in itself.
The residual is carried, not waived. Declining the measurement does not decline the risk:
What is unmeasured
The cold-key status code during an outage (503 expected, 401 would mean the degrade is inverted); the paired warm-key 200; the degraded /health/readiness body; litellm-db-failure observed firing; signal-to-alert latency.
Reopen condition
(a) A real unplanned Postgres outage — capture it then, for free, per the list below; or (b) a LiteLLM version bump that changes _resolve_public_readiness_db, the service_callback fan-out, or the DB_CALL span mapping, since the FROM SOURCE claims above are pinned to 1.96.2.
Owner
Whoever is handling the outage when (a) happens; whoever reviews the Renovate PR when (b) does.
If you are reading this DURING a real outage, capture these before it ends — it is the only
free opportunity to close the gap, and it takes about two minutes:
kubectl port-forward -n litellm pod/<one pod> 4000:4000 — a POD, not the Service, and record
which pod.
GET /health/readiness on it. Record the body and the status code. This estate has never seen
it in anything but {"status": "healthy", "db": "connected"}.
GET /health/liveliness on it. Expected 200.
A cheap completion with a key that has not been used recently on that pod (cold). Record the
status code verbatim. 503 is the expected result; 401 is a blocking finding — it would mean a
caller with a valid credential was told their credential is bad, which is exactly what
allow_requests_on_db_unavailable exists to prevent.
The same completion with a key that has been used on that pod inside the last 60 seconds
(warm — the virtual-key auth cache TTL is 60s, see
“Rotating a virtual key”). Expected 200. Without this pair, a blanket
503 would satisfy the cold-key criterion while proving the degrade broken.
Whether litellm-db-failure fired, and how long after the first error span.
Mint, rotate and delete nothing while capturing. The rotation hazard above applies to you too.
Never induce this by kubectl apply. If a future ruling ever puts an induced drill back in
scope, the mechanism is a committed manifest ArgoCD syncs, with its revert PR opened before the
severing PR is merged, and kubectl used only to observe.
MCP session semantics across restarts and replicas
The one sentence that saves the debugging session: the symptom of a lost MCP session is a missing
notification, not an error message. There is nothing to grep for. If you are looking for a status
code, you are looking for something that does not exist.
1. Session state is process-local, so it is lost on restart and unknown to the other replica.
Every MCP session lives in per-process dictionaries on the pod that issued it
(mcp_server/server.py:539-556). Nothing is in Postgres, nothing is in Valkey. The Deployment runs
two replicas with no session affinity, so a request carrying an mcp-session-id lands on the
replica that does not know it roughly half the time — the same downgrade a restart produces, on a
perfectly healthy proxy.
2. An unknown session id is not an error. It is a silent downgrade to stateless service._handle_stale_mcp_sessionstrips an unrecognised mcp-session-id from the request before it
reaches the SDK and serves the request statelessly instead (mcp_server/server.py:3549-3566). This
is a deliberate compatibility shim, not an absence of validation — and the operational consequence
is that there is no 400 session ID not found to look for. Any runbook, alert or dashboard that
watches for one is watching for a response this proxy does not emit.
3. Plain tools/list and tools/call keep working. Session continuity silently stops. The
downgrade is invisible for stateless request/response work — the overwhelming majority of MCP
traffic — because a stateless tools/call needs nothing the dropped session held. What breaks is
anything that requires the server to speak first or to resume a conversation: server-initiated
notifications, sampling, and elicitation. Those simply never arrive.
4. Two conditions DO produce errors, and they are the only two. Both are working as designed;
neither is a restart artefact:
Condition
Response
Meaning
A live stateful session driven by someone other than its creator
403 (mcp_server/server.py:4199-4216)
Sessions are owner-bound. A mismatch is refused, not silently served. Not a 200 — do not expect a session hijack to be quiet
A caller holding too many concurrent stateful sessions
429 (mcp_server/server.py:4243-4258)
A per-caller cap on concurrently-held stateful sessions. Not a 400, and not a rate limit on requests — it is a limit on held sessions, so it clears as sessions are released
Concurrency within one session is serialized, not parallel. Two concurrent tools/call on the
same session complete sequentially: a per-session asyncio.Lock serializes same-session requests
(mcp_server/server.py:550-555). Both succeed. A client that fans out on one session gets
correctness, not throughput — if throughput is what it wants, it needs separate sessions.
Nothing needs draining, and nothing needs a maintenance window for MCP’s sake. A rolling restart
drops every session; every client that only does tools/list and tools/call continues without
noticing, and the clients this estate runs are all in that class today. If a future client depends
on notifications, sampling or elicitation, that is the client to re-establish deliberately after
a rollout — and the way to detect the failure is at the client, by noticing the expected message
never arrived.
Why this is documented from source and not measured
Everything above is read from the deployed v1.96.2 image, with file and line citations, and no
drill was run to reproduce it.
The reason is scope, and it is the same ruling that governs
the database-unavailable degrade:
MCP session handling is LiteLLM’s implementation, not ours. Rule afja1qt48h puts third-party
behaviour outside what this estate tests. A gate asserting the stale-session downgrade, the owner
403 or the session cap would go red when a vendor ships, not when we break something — and a
check that reddens on someone else’s release is a check about them, not about us. We own the
decision to run two replicas without affinity; we do not own what LiteLLM does with a session id it
does not recognise.
The residual is carried, not waived.
What is unmeasured
That a restart produces exactly the downgrade described rather than an error; the observed cap value behind the 429; whether a notification-dependent client degrades gracefully or hangs
Reopen condition
(a) A client arrives that genuinely needs notifications, sampling or elicitation — then the behaviour is ours to characterise because the dependency is ours; or (b) a LiteLLM version bump touching _handle_stale_mcp_session, the owner binding, or the session cap, since every claim above is pinned to 1.96.2
Owner
Whoever onboards that client when (a) happens; whoever reviews the Renovate PR when (b) does
If you observe a real MCP session failure, capture it — it is the free measurement:
Which replica served the request (kubectl get pod -n litellm -o wide, and the pod name from the
response if available).
The full request headers, mcp-session-id included, and the status code. A 400 here would
contradict claim 2 above and is a blocking finding.
Whether a 403 or 429 was involved — those are the designed errors and identify the condition
immediately.
What the client expected to receive and did not. For a lost session this is the only signal.
Every LiteLLM version bump reaches this cluster through a Renovate pull request, and that pull
request now waits seven days and requires a named human reviewer. The rule that does it is the last
element of packageRules in .github/renovate.json, group slug litellm-data-plane. It is last
because Renovate applies packageRules in array order with later matches overriding earlier ones,
and three general groups — container images (ghcr), helm charts and argocd manifests — automerge
these files on floors of two to three days. Position is behaviour here, not tidiness: keep the
litellm-data-plane rule the last element of packageRules, because nothing after it may loosen
what it sets.
Exactly two files carry a LiteLLM version that a Renovate manager actually reports:
argocd/app-configs/litellm/** carries no version pin today. It is matched as forward coverage, so
that the day a pin appears there the rule already reaches it. Do not describe that tree as
guarded.
The rule matches by file name, never by dependency name. The key-affecting encoders rule
immediately above it in the same file carries a written caveat that one of its own four dependency
names is matched by no manager at all — a name-based rule can silently cover nothing, and emits no
signal that it is covering nothing. File-name matching is the mechanism the existing
argocd manifests group already uses in this config, so it is proven here rather than novel.
Seven days is the estate’s existing handle-with-care tier — the same floor already applied to
major version bumps, kubernetes components and terraform core. It is not a number invented for
LiteLLM. At LiteLLM’s release cadence it is roughly two to four releases of lag, and that lag is the
price being paid: the proxy runs a few releases behind the newest, in exchange for other people
meeting a bad release first.
Both automerge keys and an explicit reviewer list are required. The root configuration sets
automerge: true, platformAutomerge: true and reviewers: []. Disabling one automerge key leaves
the other in force, and omitting reviewers leaves nobody named on the very pull request the delay
exists to put a human in front of.
The residual, stated narrowly — and narrowed. Renovate merges config.vulnerabilityAlerts
last, over the fully-resolved package configuration — measured in Renovate’s source
(lib/workers/repository/updates/flatten.ts), not inferred from the documentation, which does not
answer it. A vulnerability-remediation pull request therefore bypasses the seven-day floor, which is
still forced to null on that lane. It no longer bypasses both automerge keys:
platformAutomerge: false was never overridden, and .vulnerabilityAlerts.automerge is now false
repo-wide, so the pull request opens at once and waits for the named reviewer. That reverses the
earlier acceptance that the vulnerability path stays fast estate-wide with LiteLLM not carved out of
it — see Renovate Operations for why a narrower, per-dependency control is
not available in repository configuration.
The broad reading of that residual — “vulnerability pull requests bypass the gate, so the gate is
porous” — overstates it, so state it precisely. The fast path fires only for a GitHub or OSV
vulnerability alert against a detected dependency. A supply-chain compromise published as an
ordinary release is not a vulnerability alert, and that is the incident class on record for this
package: it travels the ordinary rule path, where the seven-day floor and the named reviewer do
apply. The genuine exposure is narrower and worth naming exactly — a fix version that is itself
compromised arrives with no release-age floor, though it can no longer merge itself.
Attestation, 2026-08-15: the GitHub security-advisory watch on BerriAI/litellm is enabled on the account seanb4t — operator statement, NOT machine-verified.
That is deliberately one unwrapped line so the date, the account and the disclaimer stay together
for anyone reading it or searching for it. It came from a human looking at the account, and nothing
in this repository asserts it.
Nothing verifies it, deliberately. Verifying it would need a credentialed gh api call, and it
would assert third-party account state that changes no line of our configuration. So this line is worth exactly what a dated human
statement is worth, and it goes stale silently — a watch can be switched off, or the account can
lose access, and this document will go on saying it is on. Re-attest it when this section is next
touched rather than trusting the date.
The advisory watch and the Renovate reviewer are different accounts. The watch is on seanb4t;
the Renovate rule assigns reviewers: ["fzymgc"]. Those are two notification inboxes: a GitHub
security advisory for LiteLLM arrives at one, the review request on a LiteLLM Renovate pull request
arrives at the other. This is an observed fact about the configuration as it stands — both values
are deliberate — not a defect being reported. But do not read this section as evidence that one
person is guaranteed to see both signals.