LiteLLM LLM-02 Parity Matrix

Compiled: 2026-08-30 (recompiled from artifacts; first compiled 2026-08-29 from prose) Requirement: LLM-02 — parity verified against the previous gateway’s behaviour and documented in the milestone record. Phase: 03-llm-key-management, plan 03-18 (gap closure over 03-16’s close-out). Parity session: 6e953b82e4e34d358eb0ddccb87b4f00 — eleven rows issued two-sided on 2026-08-30 with all four credentials present. Every artifact carries that id as parity_session_id.

---

What parity means here, and what it deliberately does not

Parity in this document is structural and routing parity (decision D-45), asserted per lane:

1. the same upstream model is reached,
2. the same provider flags travel on the wire (including zdr),
3. the response shape and finish_reason are the same,
4. token accounting is comparable, and
5. errors map the same way.

Content parity is explicitly out of scope. Completions are non-deterministic, and the model’s prose is not the gateway’s to hold constant. A matrix defined on output text would measure the model, drift on every upstream release, and say nothing about the gateway. A matrix defined on the five dimensions above stays meaningful when the model changes, because those five are the full set of things a gateway is actually responsible for.

Where the cells come from

Every cell below is transcribed from a durable, machine-readable artifact. That was not true of the first compilation of this document, and the difference is the whole of residual R-13’s closure.

• test_route_structural_parity writes one results/llm02-parity-<route>.json per row, carrying that row’s evidence_class and its per-lane observation of every dimension asserted.
• Eleven such files exist, one per legacy route, all eleven carrying parity_session_id: 6e953b82e4e34d358eb0ddccb87b4f00 — the marker plan 03-18 added for exactly this reason. results/ is gitignored and ACCUMULATES, so a count over the llm02-parity-*.json glob is satisfied by a mixture of sessions; the count that matters is over that field.
• A row that is issued two-sided and FAILS now writes its artifact too, under the TWO-SIDED-FAILED class. Before plan 03-18 a failing row wrote nothing, so “measured and different” was indistinguishable from “never issued” — which is precisely how the 2026-08-29 session’s three failed rows had to be reconstructed from prose.

No cell is inferred from a neighbouring cell, and no 03-11/03-12/03-13 prose observation is carried forward into a cell the new artifact supersedes. Where the artifact records a dimension it did not observe, the cell says NOT OBSERVED with the reason.

Two dimensions are still not observed by the parity module and are marked as such per row rather than filled from elsewhere: the error-mapping dimension is sampled by test_error_mapping_parity over only three representative routes (ERROR_MAPPING_ROUTES, one per credential accept-list), and the ZDR provider-pool attestation behind dimension (b) is asserted by test_zdr_wire.py, a different module with its own residuals (R-1, R-3, R-4). Dimension (b)’s cells below carry the in-band signal the parity artifact does record: the response body’s provider field on each lane.

Evidence classes, and the four rules that govern them

tools/litellm-verify/tests/test_llm_lane_parity.py declares four classes. Each one-sided class names the side that WAS observed, and the two are easy to transpose:

Class	Meaning
PARITY	Both lanes were issued and compared on all five dimensions, and every dimension agreed.
TWO-SIDED-FAILED	Both lanes were issued and compared, and a dimension DIFFERED. The comparison happened; its outcome was a difference.
AGENTGATEWAY-ONLY	Lane B was issued; the LiteLLM side has no destination to issue against.
LITELLM-ONLY	The LiteLLM side was issued; lane B could not be.

Four rules, applied without exception:

1. A row under either one-sided class is a RESIDUAL, never a parity result, and appears in the residual register as well as in the table.
2. A one-sided row must carry the named residual recorded with it. A class with no reason attached is not transcribable.
3. The class must match the side the recorded note says was observed. A class disagreeing with its own recorded residual is a blocking finding, not a discrepancy to smooth over.
4. Neither ZDR route (llm-chat-zdr, openrouter-generic-zdr) may carry either one-sided class. Both are real header-selected HTTPRoutes with real lane-B paths, and the lane whose whole purpose is a privacy guarantee is the last one that may go without two-sided evidence.

TWO-SIDED-FAILED was added by plan 03-18 and is a real class, not a state written into the column by a compiler: the module emits it, checks.fail records it, and the row stays RED. One further state appears in the Evidence column and is not an evidence class:

• NOT MEASURED — the parity row was never issued, so no class was emitted. The harness module is explicit that a missing credential blocks a row and “never earns an evidence class: we could not authenticate is a precondition that was not met, not a measurement.”

No row in this matrix carries LITELLM-ONLY, AGENTGATEWAY-ONLY or NOT MEASURED. All eleven were issued two-sided in one session.

The matrix — eleven rows, one per legacy route

#	Legacy route	(a) upstream model reached	(b) provider flags on the wire	(c) shape and finish_reason	(d) token accounting	(e) error mapping	Evidence
1	llm-chat	openrouter/google/gemini-2.5-flash (LiteLLM, from the spend record) vs google/gemini-2.5-flash (lane B body) — MATCH	provider: Google on BOTH lanes	1 choice, finish_reason: stop on BOTH lanes	7 / 1 / 8 on BOTH lanes; cost 4.6e-06 and upstream_inference_cost populated on both	NOT OBSERVED — this route is not in ERROR_MAPPING_ROUTES	PARITY — llm02-parity-llm-chat.json
2	llm-chat-zdr	openrouter/deepseek/deepseek-v4-flash vs deepseek/deepseek-v4-flash — MATCH	provider: Novita (LiteLLM) vs SiliconFlow (lane B) — different providers within the same OpenRouter pool; pool-membership attestation is R-4’s, not this module’s	1 choice on both; finish_reason: length (LiteLLM) vs stop (lane B) — both OpenAI-compatible values, differing because the two providers spent the 8-token budget differently	11 / 8 / 19 (LiteLLM) vs 90 / 10 / 100 (lane B) — an unexplained 8× prompt-token difference on the same body, recorded as residual R-22	NOT OBSERVED — this route is not in ERROR_MAPPING_ROUTES	PARITY — llm02-parity-llm-chat-zdr.json
3	llm-embeddings	openrouter/baai/bge-m3 vs BAAI/bge-m3 — MATCH this session. The SAME row read parasail-bge-m3 on lane B 40 minutes earlier and FAILED: lane B echoes the serving provider’s own naming, so this cell is provider-dependent (R-21)	provider absent on the LiteLLM leg (litellm 1.96.2 drops it on embeddings, R-1) vs DeepInfra on lane B	1 vector of 1024 dimensions on BOTH lanes	12 / 12 on BOTH lanes; lane B additionally reports cost 1.2e-07	NOT OBSERVED — this route is not in ERROR_MAPPING_ROUTES	PARITY — llm02-parity-llm-embeddings.json; see R-21
4	llm-summarize-engram	openrouter/deepseek/deepseek-v4-flash vs deepseek/deepseek-v4-flash — MATCH	provider: Novita (LiteLLM) vs Venice (lane B)	1 choice, finish_reason: length on BOTH lanes	11 / 8 / 19 (LiteLLM) vs 11 / 9 / 20 (lane B) — same prompt count, one completion token apart	NOT OBSERVED — this route is not in ERROR_MAPPING_ROUTES	PARITY — llm02-parity-llm-summarize-engram.json
5	llm-fovea-scout	openrouter/deepseek/deepseek-v4-flash vs deepseek/deepseek-v4-flash — MATCH. This row FAILED this dimension on 2026-08-29 for the R-5 read-path defect and is now green on the same assertion	provider: Mancer 2 (LiteLLM) vs DeepInfra (lane B)	1 choice on both; finish_reason: length (LiteLLM) vs stop (lane B)	11 / 8 / 19 (LiteLLM) vs 11 / 2 / 13 (lane B) — same prompt count	TWO-SIDED, FAILED — LiteLLM refuses with a JSON error object; agentgateway’s 401 body is bare text (R-6)	PARITY on the structural row — llm02-parity-llm-fovea-scout.json; the error row fails separately (R-6)
6	llm-fovea-deepdive	openrouter/z-ai/glm-5.2 vs z-ai/glm-5.2 — MATCH. Also previously RED on R-5	provider: Decart on BOTH lanes	1 choice, finish_reason: length on BOTH lanes	19 / 8 / 27 on BOTH lanes, with identical cost 2.2116e-05 and identical cost breakdown	NOT OBSERVED — this route is not in ERROR_MAPPING_ROUTES	PARITY — llm02-parity-llm-fovea-deepdive.json
7	llm-fovea-embed	openrouter/qwen/qwen3-embedding-8b vs Qwen/Qwen3-Embedding-8B — MATCH under case-folded containment. Also previously RED on R-5; also provider-dependent (R-21)	provider absent on the LiteLLM leg (R-1) vs Nebius on lane B	1 vector of 4096 dimensions on BOTH lanes	8 / 8 on BOTH lanes; lane B additionally reports cost 8e-08	NOT OBSERVED — this route is not in ERROR_MAPPING_ROUTES	PARITY — llm02-parity-llm-fovea-embed.json; see R-21
8	llm-embeddings-engram-v2	gemini/gemini-embedding-2 vs gemini-embedding-2 — MATCH	provider absent on BOTH lanes — Google’s OpenAI-compat surface carries no such field	1 vector of 3072 dimensions on BOTH lanes, matching the deployed memory_v4 corpus	7 / 7 on the LiteLLM leg; lane B carries no usage at all, which is expected difference 2 below and is now OBSERVED rather than predicted	NOT OBSERVED — this route is not in ERROR_MAPPING_ROUTES	PARITY — llm02-parity-llm-embeddings-engram-v2.json. Two-sided, and explicitly NOT AGENTGATEWAY-ONLY
9	llm-embeddings-gemini	gemini/gemini-embedding-2 vs gemini-embedding-2 — MATCH	provider absent on BOTH lanes	1 vector of 3072 dimensions on BOTH lanes	7 / 7 on the LiteLLM leg; lane B carries no usage (expected difference 2, OBSERVED)	TWO-SIDED, FAILED — same cause as row 5: agentgateway’s 401 body carries no JSON error object (R-6). Measured here on the agentgateway-gemini-vkeys accept-list	PARITY on the structural row — llm02-parity-llm-embeddings-gemini.json. Two-sided, NOT AGENTGATEWAY-ONLY
10	openrouter-generic	openrouter/openai/gpt-4o-mini vs openai/gpt-4o-mini — MATCH	provider: Azure (LiteLLM) vs OpenAI (lane B) — the same model served by two different OpenRouter upstreams	1 choice, finish_reason: stop on BOTH lanes	14 / 2 / 16 (LiteLLM) vs 14 / 1 / 15 (lane B) — same prompt count	TWO-SIDED, FAILED — R-6 again, measured here on the agentgateway-openrouter-vkeys accept-list	PARITY on the structural row — llm02-parity-openrouter-generic.json
11	openrouter-generic-zdr	openrouter/deepseek/deepseek-v4-flash vs deepseek/deepseek-v4-flash — MATCH. This row was BLOCKED at every prior attempt (R-14)	provider: DigitalOcean on BOTH lanes	1 choice, finish_reason: stop on BOTH lanes	11 / 2 / 13 (LiteLLM) vs 11 / 3 / 14 (lane B) — same prompt count	NOT OBSERVED — this route is not in ERROR_MAPPING_ROUTES	PARITY — llm02-parity-openrouter-generic-zdr.json. The ZDR passthrough lane now carries two-sided evidence

Token-accounting cells read prompt / completion / total. Every row’s counts came from the artifact’s usage block, and every row’s dimension (a) came from the artifact’s resolved_model block, whose resolved_model_source names the per-lane read path: the /spend/logs record on the LiteLLM leg, the response body on lane B.

A note on lane_diff_characterisation.choices_len. Several chat rows record a: 0, b: 1 or a: 0, b: 0 there while dimension (c) records one choice on each lane. That is not a contradiction: lane_diff counts only choices whose message.content is a string, and a reasoning model that spends its whole 8-token budget in reasoning_content returns a choice with empty content. lane_diff is characterisation and nothing branches on it; dimension (c) is the assertion.

Eleven rows onto nine LiteLLM destinations — read the count difference correctly

The eleven legacy routes map onto nine LiteLLM destinations, because two destinations are each reached by two routes:

LiteLLM destination	Reached by
openrouter/*	llm-chat (row 1) and openrouter-generic (row 10)
openrouter-zdr/*	llm-chat-zdr (row 2) and openrouter-generic-zdr (row 11)
octopus-embed	llm-embeddings (row 3)
engram-summarize	llm-summarize-engram (row 4)
fovea-scout	llm-fovea-scout (row 5)
fovea-deepdive	llm-fovea-deepdive (row 6)
fovea-embed	llm-fovea-embed (row 7)
engram-embed	llm-embeddings-engram-v2 (row 8)
gemini-embed	llm-embeddings-gemini (row 9)

Eleven-onto-nine is a routing fact about the NEW plane. It is not evidence that the two legacy paths behaved the same. The two rows in each pair differ in host, path, credential accept-list and, for the ZDR pair, in header selection: rows 10 and 11 live on openrouter-gw.fzymgc.house under /api/v1/... and authenticate against agentgateway-openrouter-vkeys, while rows 1 and 2 live on llm-gw.fzymgc.house under /v1/... and authenticate against agentgateway-vkeys. Plan 03-01 issues eleven two-sided comparisons for exactly this reason. Consolidating the table to nine rows would discard eleven measurements the phase set out to take, and would read as a dropped lane rather than as a fold. Do not consolidate.

The harness module’s own docstring names only the openrouter-zdr/* fold when it explains eleven-onto-nine. The openrouter/* fold is equally real and is visible in the module’s own ROUTES table (rows 1 and 10 both declare model_name = "openrouter/*"). The table above is transcribed from ROUTES, which is authoritative over the prose above it.

Rows 8 and 9 are NOT AGENTGATEWAY-ONLY, and recording them so would be false

The one-sided AGENTGATEWAY-ONLY branch exists for exactly these two Google embed rows, and it fires only when the LiteLLM destination is absent from /v1/models — which is how plan 03-06 records an Open Question 3 dimension verdict of MISMATCH or NOT MEASURED.

The verdict shipped as MATCH (3072 produced against a 3072-dimension deployed corpus, recorded in docs/engineering/specs/2026-08-16-litellm-lane-parameter-measurements.md and re-observed live on engram-embed before 03-13 Task 2 was written). Plan 03-06 therefore wrote both engram-embed and gemini-embed, and 03-13 cut engram’s embedding-bearing call sites over onto them. Both LiteLLM destinations exist, so the one-sided branch does not apply and both rows are two-sided by construction. Until 2026-08-30 they had simply never been issued.

Recording rows 8 and 9 as AGENTGATEWAY-ONLY would publish the inverse of the estate’s actual shape: it would claim the LiteLLM side has nowhere to send the request, on the two lanes engram is serving production embeddings through today.

Now settled by measurement rather than by argument. Both rows were issued two-sided in the 2026-08-30 session and both carry PARITY, with 3072 dimensions produced on each lane. The one-sided branch was never reached, because the LiteLLM destinations are present in /v1/models — which is exactly what the argument above predicted.

Blocking findings — both CLOSED by the 2026-08-30 session

BLOCKING FINDING 1 — openrouter-generic-zdr carries no two-sided evidence — CLOSED

Row 11’s recorded state was BLOCKED: AGENTGATEWAY_OPENROUTER_API_KEY was absent from every recorded run, and the harness blocks the row rather than degrading it to a one-sided comparison. The lane whose entire purpose is a privacy guarantee was one of the two the phase had measured least.

Closed. The credential was read from Vault property vk_openrouter_passthrough into the process environment for the 2026-08-30 session, and row 11 issued two-sided and passed all five dimensions: llm02-parity-openrouter-generic-zdr.json, evidence_class: PARITY. Both ZDR rows now carry PARITY, which is what rule 4 above requires. Residual R-14 closes with it.

BLOCKING FINDING 2 — six of eleven parity rows were never issued — CLOSED

Rows 1, 3, 4, 8, 9 and 10 had no recorded parity run: the single recorded session (plan 03-11, 2026-08-16) selected -k "fovea or zdr or spend_attribution", matching five of eleven parametrised route ids, and no later session ran the parity module at all.

Closed. One credentialed session on 2026-08-30 (LITELLM_REQUIRE_LIVE=1 task litellm-verify:live -- -k "route_structural_parity or error_mapping_parity") issued all eleven rows two-sided and wrote eleven artifacts, all carrying parity_session_id: 6e953b82e4e34d358eb0ddccb87b4f00. Every structural row passed. Residual R-13 closes with it.

What is NOT closed by that session, stated here rather than left to be inferred: the error-mapping dimension still fails two-sided on all three sampled routes, for a cause that is entirely on the legacy leg and retires with it (R-6); dimension (a) on the two OpenRouter embedding rows is provider-dependent and was observed both green and red within one hour (R-21); and lane B reports an unexplained 8× prompt-token count on llm-chat-zdr (R-22).

Consumers — the middlebox property, asserted rather than footnoted

octopus, karakeep and mealie do not reach LiteLLM directly. They reach it through headroom, which forwards the caller’s key unchanged; headroom’s own OPENAI_TARGET_API_URL was repointed to https://llm.fzymgc.house/v1 in plan 03-12.

A middlebox is only acceptable under KEY-03 (per-key spend attribution) if each consumer still lands on its own key alias on the far side of the hop. Measured, and it holds. Plan 03-12‘s test_attribution_survives_the_headroom_hop asserts distinctness, not presence — three cost rows arriving proves only that traffic flowed, while three rows sharing one alias is exactly the D-53 failure and is invisible to any per-row check. Each of the three consumers’ own delivered credential was used, not a disposable key.

Three distinct key aliases observed across the middlebox hop:

Alias	Consumer	Lane
octopus	octopus	octopus-embed (baai/bge-m3)
karakeep	karakeep	openrouter/* (deepseek/deepseek-v4-pro)
mealie	mealie	openrouter/* (google/gemini-2.5-flash)

Each carries its own metadata.user_api_key_alias on its own gen_ai.usage.cost rows. The narrow headroom repoint is therefore sufficient under KEY-03, and the backlogged “fold headroom into LiteLLM” evaluation stays optional rather than becoming required.

For completeness, fovea calls LiteLLM directly and produced three lane rows under a single fovea alias (03-11); engram calls it directly too, under alias-engram.

Expected differences — declared as differences, not scored as regressions

1. LiteLLM adds a usage-include field to every OpenRouter chat request

This is deliberate and beneficial. It is how provider-attested cost comes back instead of being estimated from a cost table, and it is why chat spend is now more accurate than it was under agentgateway. Evidenced by cost_details.upstream_inference_cost being populated alongside cost — a field that is absent when the include is not sent — observed on fovea-scout, fovea-deepdive (03-11), and the karakeep and mealie chat lanes (03-12).

A future reader comparing wire captures will find this field on the LiteLLM leg and not on the agentgateway leg. That is the improvement, not a divergence to correct.

2. Lane B carries no usage on the two Gemini rows

On rows 8 and 9 the agentgateway side is expected to omit usage entirely. agentgateway serves those lanes in Passthrough mode, precisely because Google’s OpenAI-compat surface omits usage (argocd/app-configs/agentgateway/llm-backends.yaml:138-147). The harness records this as an expected difference and asserts token accounting on the LiteLLM side only for those two rows, rather than failing or skipping.

OBSERVED, 2026-08-30. Both rows were issued two-sided and both artifacts record usage.agentgateway: {} against a populated LiteLLM-side usage of 7 prompt / 7 total tokens. The prediction held exactly; the exception is now a measurement rather than a forecast.

3. Path prefixes do not survive the cutover

Under agentgateway a lane was selected by URL path (/fovea-scout/v1/..., /engram/v1/..., /gemini-embed/v1/...). Under LiteLLM a lane is selected by the request body’s model, so every consumer collapses onto the same /v1 surface and lane identity moves into the key’s aliases. A mechanical host-swap that preserved the path segment 404s at the ingress, which matches Host(llm.fzymgc.house) && PathPrefix(/v1/). This is a routing difference by design, recorded so that a reader does not read the missing prefixes as a lost lane.

4. The LiteLLM-side parity request carries two measurement instruments lane B does not

Added by plan 03-18 and declared here rather than left to be discovered in a wire capture. Both are on the LiteLLM leg only, and neither reaches a provider:

• metadata.spend_logs_metadata.parity_run_id — the correlation marker. Its value is the session’s PARITY_SESSION_ID joined to a per-row uuid4 hex. LiteLLM copies spend_logs_metadata verbatim into the SpendLogs row’s metadata column, which is how dimension (a) finds this request’s spend record instead of “the most recent row”. A body-id correlation was rejected because the four embedding rows answer an OpenAI embeddings envelope with no id field at all, so it would have silently covered seven of eleven rows.
• cache: {"no-cache": true} — a measured necessity, not tidiness. The probe body is a fixed string, so the second and every later run of a row is served from this proxy’s response cache (observed 2026-08-30: 0.03 s wall and a gen-… id minted minutes earlier, against 2-4 s and a fresh id with the field set). A cache hit writes no spend row, so without this the dimension-(a) read would block on every re-run — a measurement that works exactly once is not a measurement. It also makes the comparison fairer: lane B has no such cache, so bypassing ours is what puts both lanes on the same footing of actually reaching the provider.

Residual register

Every entry carries four fields: what it is, why it is accepted, who owns closing it, and the concrete detection that goes red on its own. A residual whose only enforcement is a name in a document does not close, so no entry here has an owner without a detection.

R-1 — the embed lanes’ ZDR evidence is a fail-closed negative control, not attestation

• What: fovea-embed, octopus-embed, engram-embed and gemini-embed cannot be provider-attested. LiteLLM 1.96.2’s embedding responses carry no provider field at all, and the metric’s gen_ai.provider.name is empty for these lanes. The ZDR evidence is instead a fail-closed negative control: the harness-only zdr-probe-embed lane is refused 403 key_model_access_denied to a real workload key.
• Why accepted: the control is real evidence and the 403 is real evidence. Neither is attestation, and reporting them as attestation would claim coverage the estate does not have.
• Owner: whoever runs the live lane.
• Detection: test_zdr_embed_lane_negative_control against the deployed zdr-probe-embed lane, on each operator-run live lane. It PASSED in the 03-11 run.

R-2 — no CI sentinel guards the provider overrides, and the window is unbounded

• What: nothing in CI asserts that the provider: {zdr: true, ...} blocks are still present on the ZDR lanes in argocd/app-configs/litellm-chart/values.yaml. A merged config change that drops one is undetected until the next operator-run live lane, and there is no schedule that forces one — the window is unbounded, not merely long.
• Why accepted: knowingly, under D-35. The assistant raised the CI sentinel; the operator declined it in favour of live assertion only, on the grounds that a config read proves our config and not the provider’s behaviour. Recorded as accepted, not re-litigated.
• Owner: whoever runs the live lane.
• Detection: the operator-run live lane (task litellm-verify:live), with the window named as unbounded rather than glossed as “until the next run”.

R-3 — the plain-wildcard ZDR negative assertion is advisory

• What: test_wildcard_lane_carries_no_zdr_override is marked ADVISORY in its own recorded note. Provider identity cannot distinguish an unconstrained request from a constrained one served by a ZDR-pool provider — an unconstrained call that happens to land on a ZDR-capable provider is indistinguishable, by provider name alone, from a constrained one.
• Why accepted: the test’s deterministic half still carries a real property, and it asserts that half separately: the plain and ZDR wildcards must resolve to distinct deployment ids, so the plain wildcard silently resolving to the ZDR entry is caught. Only the provider-identity leg is advisory.
• Owner: whoever runs the live lane.
• Detection, and where the strong form lives instead: the deterministic distinct-deployment-id assertion inside the same test; the 03-06 render-parsing criterion over the plain openrouter/* entry; and test_zdr_embed_lane_negative_control. The test records all three in its own note.

R-4 — ZDR_CAPABLE_PROVIDERS is a maintained frozenset, and a stale one presents as a red a human must classify

• What: tools/litellm-verify/tests/test_zdr_wire.py declares ZDR_CAPABLE_PROVIDERS = frozenset({"nebius", "deepinfra"}) and test_zdr_chat_lanes_are_provider_attested asserts membership unconditionally. The soft “present and non-empty” fallback was removed because it stayed green with the ZDR constraint dropped — on the assertion ROADMAP SC#2 leans on. Observed on ZDR lanes to date: DeepInfra, CoreWeave, Novita, Sail Research, Azure, Together, DigitalOcean. nebius has never been observed and OpenRouter lists no Nebius endpoint for deepseek/deepseek-v4-flash at all, so half the declared pool cannot serve the model it is declared for.
• Why accepted: the cost is that OpenRouter adding a provider to the ZDR pool produces a failure indistinguishable at first glance from a real privacy defect. A red requiring human classification is strictly better than a green that cannot fail.
• Owner: whoever runs the live lane.
• Detection: the test itself. Its failure message names the observed provider and the two legitimate resolutions, and it records the observed provider with checks.note on every run, so the set’s accuracy is re-derivable from evidence rather than from memory.

R-5 — the structural-parity dimension (a) read the wrong field — CLOSED 2026-08-30

• What it was: route_structural_parity failed on all three fovea rows because LiteLLM’s response model echoes the string the caller sent — the lane name for a master-key call, the client alias for a workload key — never the upstream id. The dimension was real; the field was wrong. Re-measured directly on 2026-08-30: a fovea-scout completion answered "model": "fovea-scout" in the body while the same response’s x-litellm-model-name header and its spend record both said openrouter/deepseek/deepseek-v4-flash.
• Status: CLOSED. _resolved_upstream_model now owns the read path — the response body on the agentgateway leg (a forwarding proxy’s model IS the provider’s), the /spend/logs record on the LiteLLM leg, selected by the per-request marker declared as expected difference 4 above. There is no fallback to the body: a spend record that does not arrive within 30 s BLOCKS the row, naming the route and the marker. _model_matches was NOT changed; the fix moved where the observed string comes from, never what equality means.
• The closure rests on a measurement, not on the absence of a failure. Recorded red-proof: with llm-fovea-scout’s declared upstream_model temporarily set to openai/gpt-4o-mini, the same selector failed with “llm-fovea-scout on litellm: the resolved model ‘openrouter/deepseek/deepseek-v4-flash’ does not name the expected upstream model ‘openai/gpt-4o-mini’” — the message naming the spend-record model. Reverted; the row is green on the same assertion that was red on 2026-08-29.
• Owner: closed; no owner outstanding.
• Detection going forward: test_structural_parity_never_reads_the_response_model_on_the_litellm_lane in tools/litellm-verify/tests/test_phase03_config_gates.py. It parses the parity module with ast and fails if any read of the response payload inside _resolved_upstream_model sits outside the branch guarded on the agentgateway lane, if the helper stops calling client.spend_logs(...), or if test_route_structural_parity passes anything but the helper’s bound result as the observed argument of _model_matches. It lives in the gates lane because that is the only lane CI runs, needs no cluster and no credentials, and was itself red-proofed by restoring the body read.

R-6 — the 401 error-mapping asymmetry is entirely on the legacy leg

• What: error_mapping_parity fails two-sided. agentgateway’s 401 body is api key authentication failure: no API Key found, with no JSON error object. LiteLLM’s 401 and 403 bodies do carry one (auth_error, key_model_access_denied).
• Widened by measurement, 2026-08-30: the asymmetry is not specific to one route or one accept-list. All three representative routes failed identically in the same session, with the same verbatim body — llm-fovea-scout (agentgateway-vkeys), llm-embeddings-gemini (agentgateway-gemini-vkeys) and openrouter-generic (agentgateway-openrouter-vkeys). It is a property of the agentgateway listener, not of any lane.
• Why accepted: the asymmetry is a deficiency of the retiring gateway, not of the new plane, and it retires with llm-gw in Phase 5. Changing agentgateway’s error body during the bake would edit the rollback path.
• Owner: Phase 5, as part of removing the listener.
• Detection: the test itself; it goes green when lane B ceases to exist and the row is removed with the route, and red today.

R-7 — the CI equality gate cannot detect its own removal

• What: test_ci_gates_lane_matches_the_gates_task (plan 03-04) asserts that .github/workflows/litellm-verify.yml’s pytest step and the litellm-verify:gates task name the same module set. A change that removes the pytest invocation altogether stops the gate running, so it cannot fire on it. This is a property of any gate guarding its own runner, not a gap in this one.
• Why accepted: the alternative is a second runner outside the repository to watch the first, which would itself need a watcher.
• Owner: whoever reviews workflow changes.
• Detection, two mechanisms: .github/workflows/litellm-verify.yml is itself a paths: entry, so any edit to it triggers a run and appears in the diff; and the schedule: trigger (plan 03-07, cron: '23 7 * * *') runs the whole gates lane daily against the default branch, so a lane that has stopped reporting is visible as an absence of daily runs rather than only as a missing check on one pull request.

R-8 — the three agentgateway overlap accept-list entries

• What: octopus-litellm, karakeep-litellm and mealie-litellm in the agentgateway-vkeys ExternalSecret (argocd/app-configs/agentgateway/secrets.yaml), added by plan 03-12. State the exposure precisely: while they exist, those three sk- credentials authenticate on a second plane carrying NO models allow-list, NO budget, NO rpm/tpm limit, NO TTL, NO revocation path and NO alias attribution. A key revoked or budget-exhausted in LiteLLM keeps spending through llm-gw. This is the KEY-02/KEY-03 control set this phase exists to build, disapplied on a parallel path. It is the phase’s one genuine control-plane bypass.

• Why accepted: removing them earlier breaks D-43’s one-line rollback, which lands the three consumers back on agentgateway still holding those sk- values. That trades a security residual for an incident residual. Shortening the window was considered and rejected on that ground.

• Owner: Phase 5, with the rest of the accept-list.

• Detection, mechanical rather than nominal:

  • test_agentgateway_overlap_bake_window_not_expired parses the # OVERLAP-BAKE-ENDS: 2026-09-15 marker in argocd/app-configs/agentgateway/secrets.yaml and fails once that date is past, and also fails if the marker is missing or unparseable, so an unbounded window cannot pass.
  • test_agentgateway_overlap_entries_present is the presence half of a retained-with-inversion pair — it asserts all six ids (the three vk_* originals and their three -litellm siblings), because an overlap entry that has quietly replaced its sibling has destroyed the rollback plane in the change meant to protect it. Phase 5 inverts this gate rather than deleting it.

• The two lanes those gates actually run on, stated because a claim about reach is part of the detection and this one was wrong for two review cycles:

  1. on a pull request matching the workflow’s paths: filter, and
  2. daily on the schedule: trigger against the default branch, with no path filter (GitHub applies paths: to pull_request only) — the lane that lets the gate fire on the day the window expires rather than on the next unrelated pull request.

  Both depend on the workflow’s run: line naming test_phase03_config_gates.py, which it did not before plan 03-04. The scheduled lane has its own limit: GitHub disables scheduled workflows after 60 days of repository inactivity. The bake ceiling was set to 30 days for exactly that reason — a 90-day ceiling would not reliably still have a daily lane running when it expired.

• Rotation coupling, from the 03-14 runbook: revoking one of those three keys is incomplete until its overlap entry is removed in the same change, because the entry reads the consumer’s Vault property directly and has no notion of revocation.

R-9 — in-place virtual-key rotation is DESCOPED; rotation is revoke-and-re-mint

• What: POST /key/regenerate is Enterprise-gated in litellm 1.96.2. The gate is in the running image at litellm/proxy/management_endpoints/key_management_endpoints.py:4771-4778 and fires before any regeneration work; the deployment sets no LITELLM_LICENSE, so both route spellings return HTTP 500. Plan 03-14’s rotation rehearsal could not run and was halted rather than simulated.

• Why accepted — DECIDED 2026-08-29, option C of the three on record in 03-14. (A) acquiring a LITELLM_LICENSE was rejected: it puts an Enterprise dependency under a self-hosted estate, and a lapse silently removes the rotation mechanism again. (B) mint-new-then-expire-old on caller-supplied key values was rejected: global alias uniqueness forces every overlap to run under two aliases, so KEY-03’s per-alias tiles and the D-47 spend alert would be wrong-by-construction during every rotation. (C) in-place rotation is descoped. A compromised or rotated key is handled by revoke-and-re-mint under the same alias, accepting a short outage — roughly one to two minutes with a forced ESO sync, up to about six waiting a refresh interval out, and unbounded for fovea until it is restarted by hand (R-11). The procedure, its outage and its caveats are documented in docs/operations/litellm.md section “Rotating a virtual key”.

• What this costs, stated plainly: there is no zero-downtime rotation path on this estate. A key that must be revoked now takes the consumer down for the length of the propagation chain. That is acceptable for these consumers — background and interactive-with-retry workloads whose failed calls are reissued — and would not be for a latency-critical one. A new consumer that cannot tolerate a two-minute credential outage re-opens this decision.

• Owner: the operator. KEY-02 is NOT marked complete. Its scoping, limits, budget and TTL halves are delivered and gated; its rotation half is delivered as a different, cruder mechanism than the requirement text names, and the requirement is left open rather than reworded to match what shipped.

• Detection — this residual is carried by DOCUMENTATION, and that is the correct home for it. The constraint is a third-party vendor’s licensing decision: not our configuration, not our code, nothing of ours that could break. A test asserting the gate is still in force would go red when LiteLLM changes their product rather than when we do something wrong, which is not a signal the harness exists to give. So this entry and the runbook section are the detection, and test_rotation_regenerate_with_grace_period was deleted rather than inverted — a test for a capability we deliberately dropped is dead weight, and deleting it is not weakening the suite because the capability left scope with this decision.

  What IS asserted is the mechanism we own: test_rotation_revoke_and_remint_preserves_the_key_contract in tools/litellm-verify/tests/test_key_lifecycle.py covers that a delete revokes, that a re-mint under the same alias authenticates, and that it reproduces the allow-list, per-key aliases, limits and budgets it replaces. It has not been run live yet — it lands on the next operator-run live lane.

  What re-opens this residual is a person, not a gate. Whoever sets a LITELLM_LICENSE, or onboards a consumer that cannot tolerate a two-minute credential outage, owns rewriting the runbook section and this entry. Nothing automated will prompt them.

R-10 — /key/update cluster-wide propagation — CORRECTED 2026-08-30

• What this entry used to say, and what was wrong with it. It read “/key/update does not propagate cluster-wide”: test_rotation_key_update_propagates_within_the_in_memory_ttl_bound failed live, neither litellm replica dropped a narrowed allow-list within a 10 s bound (converged so far: {}) reasoned from a 5 s DEFAULT_IN_MEMORY_TTL, and the conclusion drawn was that ROADMAP SC#5’s “applies cluster-wide” claim is false as deployed — plausibly D-07’s Valkey MovedError surfacing as a broken shared auth-cache invalidation. That reading is wrong, and it is corrected in place rather than deleted, because the failing run was real and a future reader is owed what was believed from it. The propagation was never broken. The bound was.
• The correction, measured in plan 03-19 on 2026-08-30. The 10 s bound was reasoned from DEFAULT_IN_MEMORY_TTL = 5, a constant that does not govern the virtual-key auth cache. litellm 1.96.2 builds that cache as UserApiKeyCache(default_in_memory_ttl=UserAPIKeyCacheTTLEnum.in_memory_cache_ttl.value) (proxy_server.py:2002), and that enum member is 60 (proxy_server.py:1394-1395); user_api_key_cache.default_in_memory_ttl = 60 was read back off a running replica. Convergence, measured per replica over four independent samples: 59.3 s, 60.4 s, 60.2 s, 59.7 s. PROPAGATION_DEADLINE_S was re-based 10.0 → 90.0, a ceiling over the measured worst case rather than a restatement of a constant, and red-proved by halving it to 45.0 — 1 failed at 45.2 s naming the stale replica — then reverted. /key/update does apply cluster-wide. Only its implied immediacy was false, and only the criterion’s wording needed amending (03-24).
• D-07 was ELIMINATED as the cause, on a read with discriminating power — not on a /cache/ping. A key warmed on replica A was served by replica B with Valkey keyspace_hits +7, against a measured noise floor of 0 over 8 s idle and a positive control of +6; the Valkey primary carries a key named for that key’s sha256. Across default.otel_logs for the litellm namespace over 24 h, MovedError, CLUSTERDOWN and Connection reset are all 0, against positive controls of 63,670 and 293 rows in the same window. The shared cache is working; D-13 held.
• The security-relevant half, which the original entry did not carry. /key/delete is bounded by the same ~60 s per-replica TTL: 0.3 s on the replica that handled the mutation, 60.3 s and 60.7 s on the other. The published ~2 s revocation figure is an INGRESS ARTEFACT — the ingress routes to whichever replica answers, and on this deployment that is repeatedly the pod that just handled the call, so an ingress-side stopwatch cannot see the stale replica at all. The operative consequence, and the reason this belongs in a register and not only in a test: narrowing a key’s scope via /key/update is NOT a revocation mechanism. Revocation is /key/delete, and inside the ~60 s window the only faster answer for a suspected compromise is delete followed by a deployment restart. REVOCATION_DEADLINE_S = 30.0 would not hold per replica; it is annotated where it lives rather than changed here (WINDOWS #198, closed by 03-20, which owns that assertion).
• Why this correction and not a configuration change. Option D — setting general_settings.user_api_key_cache_ttl — was surfaced to the operator and DECLINED, not silently omitted. It would make the criterion true by changing production to fit the criterion’s wording, and its roughly 12× increase in per-replica Postgres auth reads would have landed immediately before 03-22 deliberately induced a Postgres outage, coupling two measurements that had to stay independent. argocd/app-configs/litellm-chart/values.yaml was not edited: zero lines changed, verified against HEAD.
• Owner: closed by measurement; no owner outstanding for the propagation claim. What re-opens it is a person, not a gate: a requirement that narrowing a key’s scope act as a revocation mechanism inside a window shorter than 60 s. Whoever carries that requirement owns re-opening Option D and this entry. Nothing automated will prompt them.
• Detection: the same test, now green, on every live run — task litellm-verify:live -- -k rotation_key_update_propagates. It asserts a ceiling over the 2026-08-30 measurement and its failure message cites that measurement and its date, never the config constant the original bound was reasoned from, so a convergence that drifts past 90 s goes red and reads as a change in behaviour rather than as a mis-copied constant. There is deliberately no gate on the 60 s figure itself: that is litellm’s own cache TTL, and a test asserting it would go red when the vendor changes their product rather than when we do something wrong.

R-11 — fovea has no restarter, so a rotation would strand its running pod

• What: every consumer reads its LiteLLM credential through env.valueFrom.secretKeyRef, and an environment variable is injected once, at container start. Stakater Reloader supplies the restart for karakeep, mealie, octopus and memory-mcp via reloader.stakater.com/auto. fovea carries no such annotation and reads litellm_api_key as FOVEA_GATEWAY__OPENAI__API_KEY, so a rotation would update Vault and the Secret while the running pod keeps the old value indefinitely, having looked complete at every step an operator would check.
• R-9’s descope makes this worse, not better. Under the grace-period mechanism this produced an overlap expiring against a pod that never moved. Under revoke-and-re-mint the old value is already deleted by the time Vault is written, so a fovea rotation that omits the restart leaves the pod holding a credential that authenticates nowhere: fovea stays down until someone restarts it, rather than merely losing an overlap.
• OBSERVED 2026-08-30, no longer hypothetical. The first fovea rotation was performed (plan 03-20) and this is exactly what it did. For 3 m 28 s — between the ExternalSecret sync and the hand-issued restart — the running pod’s FOVEA_GATEWAY__OPENAI__API_KEY still hashed to a value LiteLLM had already deleted, on the same container id with restartCount: 0. Every check an operator would run read green for the whole of it: ExternalSecret SecretSynced/Ready, Secret carrying the new value, ArgoCD Application Synced/Healthy, pod 1/1 Running. Nothing in the estate reports that state. That is the observability gap, and it is what this residual is now about — not the rotation, which the runbook step handles.
• Why accepted for now: the runbook makes the explicit kubectl rollout restart deployment/fovea -n fovea a numbered step for fovea, with a real call on every one of fovea’s lanes and a pod-environment hash comparison as the verification, and that procedure was rehearsed successfully. The annotation was deliberately not added in 03-20 — the manual restart is precisely what roadmap SC#3 asked to have rehearsed, so adding it would have removed the thing under test.
• DISPOSITION — filed as follow-up by operator ruling, 2026-08-30 (plan 03-20, decision 2). Adding reloader.stakater.com/auto to fovea/fovea, matching karakeep, mealie, octopus and memory-mcp, is this residual’s own stated closing condition and the estate-consistent fix. It is tracked, not accepted: ledger entry WINDOWS #193 stays open rather than waived, because the ruling was to file the work, not to accept the cost.
• Owner: whoever adds the annotation. The “first fovea key rotation” half of this owner line is spent — that rotation happened on 2026-08-30.
• Detection: kubectl get deploy -n fovea -o json | rg reloader.stakater.com/auto returns nothing; the runbook step is the procedural guard until the annotation lands. Note that this detection is a check on the manifest, and the 3 m 28 s window above is invisible to it and to everything else — which is the argument for closing the residual rather than relying on it.

R-12 — the reverse rollback gap on engram’s key

• What: the documented reverse rollback cannot run through engram’s key. fovea-embed (qwen3-embedding-8b, 4096-dim) returns 403 for it: engram’s key admits only its own five lanes, none at 4096.
• Why accepted: while the bake is open the retained agentgateway credential (litellm_ollama_key, still populated from cluster/agentgateway) remains the other way back, so a rollback path exists — it is the reverse gap-fill that does not.
• Owner: whoever performs a reverse gap-fill, before doing so.
• Detection: the 403 itself; a reverse gap-fill fails closed and loudly rather than writing wrong-dimension vectors. Adding the lane to the allow-list or using a different key is the fix.

R-13 — the phase produced no durable parity artifact, and six of eleven rows were never issued — CLOSED 2026-08-30

• What it was: results/ is gitignored and no llm02-parity-*.json existed for any of the eleven routes; the only recorded parity session selected five of eleven rows. This matrix was second-hand by construction — transcribed from prose written at observation time.
• Status: CLOSED. Eleven results/llm02-parity-<route>.json artifacts now exist, one per legacy route, all eleven carrying parity_session_id: 6e953b82e4e34d358eb0ddccb87b4f00. Every cell in the table above is transcribed from them, and the “where the cells come from” section says so. Two harness gaps found while closing it were fixed rather than worked around: a row that failed two-sided wrote no artifact (so a measured failure was indistinguishable from an unissued row — now the TWO-SIDED-FAILED class), and the artifact recorded only two of the five dimensions (so three columns could only have said “asserted equal” — now shape, provider_flag and full usage are recorded per lane).
• Count over the field, never over the glob. results/ accumulates, so ls results/llm02-parity-*.json | wc -l is satisfied by a mixture of sessions — three blocked rows plus eight stale files add to eleven and read as closed. The count that means anything is over parity_session_id matched together with its value, because a search pattern built from an empty session id matches every artifact and returns eleven unarmed.
• Owner: closed; no owner outstanding.
• Detection going forward, and it is still time-bounded: LITELLM_REQUIRE_LIVE=1 task litellm-verify:live -- -k route_structural_parity with all four credentials present writes one artifact per row; the absence of a file is the signal. A RE-measurement is only ever possible while llm-gw still exists — Phase 5 removes lane B, after which no two-sided comparison can be taken at all, which is why the OVERLAP-BAKE-ENDS: 2026-09-15 date remains the deadline for anything that would re-open this.

R-14 — the openrouter-gw ZDR lane has no parity evidence — CLOSED 2026-08-30

• What it was: route_structural_parity[openrouter-generic-zdr] reported BLOCKED — AGENTGATEWAY_OPENROUTER_API_KEY absent. Blocking rather than degrading to a one-sided run is the harness behaving correctly, but the lane was left with no evidence of any kind, and it is one of the two whose whole purpose is a privacy guarantee.
• Status: CLOSED. The credential is Vault property vk_openrouter_passthrough at secret/fzymgc-house/cluster/agentgateway, a member of the agentgateway-openrouter-vkeys accept-list. Read into the process environment for the 2026-08-30 session — never into argv and never into a query string (WR-04 / WR-05) — the row issued two-sided and passed all five dimensions: llm02-parity-openrouter-generic-zdr.json, evidence_class: PARITY, provider: DigitalOcean on both lanes. Nothing about the lane ever prevented the measurement; the credential was simply not exported.
• Owner: closed; no owner outstanding.
• Detection going forward: the BLOCKED verdict itself under task litellm-verify:evidence, or a hard FAIL under task litellm-verify:live, where LITELLM_REQUIRE_LIVE=1 converts every block into a failure. Same Phase 5 deadline as R-13.

R-15 — a large share of estate-wide spend carries an empty key alias

• What: a majority of gen_ai.usage.cost datapoints carry an empty metadata.user_api_key_alias — 29 728 datapoints on deepseek/deepseek-chat alone in a 2-hour window — that is, traffic from keys minted without key_alias. KEY-03 is answerable for the seven workload keys and not for that traffic.
• Why accepted: the traffic is outside this phase’s seven-consumer scope; the seven workload keys all carry aliases and all attribute correctly.
• Owner: whoever owns the non-workload keys on this proxy.
• Detection: the alias-grouped ClickHouse query over otel_metrics_histogram; an "" row is the signature.

R-16 — no lane’s cost figure is unobserved, but two were observed only through telemetry

• What: every embed lane this phase cut over records a strictly positive cost, so RESEARCH Pitfall 4’s silent-zero does not apply: fovea-embed at 9e-08 per request (two independent sources agreeing exactly) and octopus-embed at 0.0432 cumulative over 133 requests during a live repository index. engram-embed and gemini-embed have no recorded cost observation, and neither does engram-summarize.
• Why accepted: the two fovea/octopus figures were what the cutover gates required; the engram and Gemini lanes were verified by dimension and recall rather than by cost.
• Owner: whoever runs the next operator live lane.
• Detection: test_embed_cost_nonzero_on_every_embed_lane, parametrised per embed lane, under task litellm-verify:live -- -k spend_attribution. It passes today for fovea-embed and octopus-embed.

R-17 — the harness leaks the LiteLLM master key on every failing live run

• What: LiteLLMClient (tools/litellm-verify/litellm_verify/client.py) has no redacting __repr__, so every failing live test prints api_key='sk-…' — the master key — in its fixture repr, into results.xml and the operator’s scrollback. Third recurrence; twelve copies in the 03-11 session alone.
• Why accepted: contained today — results.xml is gitignored and CI runs only the credential-free static lane. It is not accepted as safe: every cutover plan instructs an operator to run the live lane, and R-13 above asks for another one.
• Owner: whoever next revises litellm_verify/client.py.
• Detection: rg -c 'sk-' tools/litellm-verify/results.xml after a live run; a non-zero count is the signature. It returned 0 after the 03-14 run, which had no client-fixture failures.

R-18 — three spend checks could never report their own verdict

• What: pyproject.toml sets pytest timeout = 60 globally while tests/test_spend_attribution.py polls for METRIC_ARRIVAL_TIMEOUT_S = 180. pytest kills the test 120 s before it can decide, so those checks report Failed: Timeout (>60.0s) regardless of whether the row arrived. Half of this residual is closed — METRIC_VALUE_TABLES now reads all three metric tables through measured per-table value columns, after 03-12 measured that gen_ai.usage.cost lands only in otel_metrics_histogram (Sum and Count columns, AggregationTemporality: 2, so it must be read with max() and never sum()). The timeout half is not closed.
• Why accepted: not in any cutover plan’s files_modified; fixing it there would have been a scope breach.
• Owner: whoever next revises tools/litellm-verify/pyproject.toml.
• Detection: run -k spend_attribution with CLICKHOUSE_PASSWORD set; a Failed: Timeout (>60.0s) at _await_cost_rows is the signature.

R-19 — a CiliumNetworkPolicy egress rule cannot be reasoned about from the URL

• What: socket-LB DNATs a LoadBalancer VIP to a backend pod, on its container port, before policy evaluation, so an egress rule permitting port 443 to the world never sees the VIP — it sees a traefik pod on 8443. This broke every consumer on the 03-12 headroom repoint and was fixed in a follow-up. A toCIDR rule would not have helped either: the post-DNAT destination carries a pod identity and this cluster leaves policy-cidr-match-mode unset.
• Why accepted: it is a cluster-wide networking property, not a LiteLLM one.
• Owner: whoever performs the next consumer repoint.
• Detection: kubectl exec into the policed pod and curl the new URL before merging a repoint. 401 means reachable; a TIMEOUT means the policy dropped it. No test enforces this; it is a procedural check, and it is recorded as such rather than dressed up as a gate.

R-20 — the live database-password rotation is DESCOPED; the escaping is proven by construction

• What: the litellm CNPG password has never been rotated on this estate, and plan 03-15 — which existed to rotate it once behind a human gate — was descoped by operator decision on 2026-08-29. No Vault property was written, no database role was altered, and scripts/seed-litellm-vault.sh was not run in any mode. The risk this closes was carried as “the ExternalSecret template, the urlquery escaping and the CNPG reconnect path are untested end to end until something actually changes the value”, deferred twice — out of Phase 1 into Phase 2, and out of Phase 2 into Phase 3 as “the first place a credential actually changes”. It is now resolved by analysis, not carried forward a third time.

• Why accepted — the part we own is already proven, and proven more strongly than a rotation could prove it. Split the five properties a rotation would exercise by ownership. Two are ours: that a generated password survives the urlquery escaping in argocd/app-configs/litellm/secrets.yaml:50, and that the escaping cannot drift from the generator’s alphabet in scripts/seed-litellm-vault.sh. Both are held by test_seed_password_alphabet_round_trips, which reads the alphabet out of the seed script and round-trips every one of its 81 characters singly and concatenated, with test_urlquery_round_trip_detects_space_bearing_alphabet as its committed negative case. A live rotation samples 32 of those same 81 characters at random and might contain no escape-requiring character at all — weaker evidence, at the cost of an irreversible write against the database holding every workload key’s scope, budget and spend. The escaping is safe by construction: urlquery mangles only a space, and the alphabet contains none.

  The remaining three properties are not ours: that CNPG serves a changed credential, that LiteLLM reconnects after one, and that virtual-key rows survive that reconnect. Those are third-party software behaving as documented. Under the same rule that retired the Enterprise-gate and alias-uniqueness assertions in R-9, no test asserts them and none should — such a test reports a vendor shipping a change, not us making a mistake.

  The plan half-knew this. Its own reporting task required the executor to state whether the generated password happened to contain an escape-requiring character and, if it did not, to record that the escaping was not proven and that this residual stands. A justification written to accommodate the possibility that it does not hold is not a justification for a one-way act on production.

• Environment equivalence, asserted rather than assumed (the condition under which an offline render is evidence about production): the gate reads the same alphabet the generator emits from the generator’s own file; urllib.parse.quote_plus and Go text/template’s urlquery are both the query escaper and both render a space as +; and the gate composes the same DSN shape the ExternalSecret composes, which test_dsn_template_applies_urlquery independently pins. Four things would invalidate that equivalence, and they are named at the point of use in docs/operations/litellm.md § “Why an offline round trip is evidence about the in-cluster render”: a password that did not come from the generator (the runbook therefore generates rather than prompts), a divergence between quote_plus and urlquery, a widened alphabet, and a changed template shape. The middle one is an assumption with no detection; the other three are gated.

• What this leaves genuinely open is the PROCEDURE, not the escaping. Steps 2 through 4 of docs/operations/litellm.md § “Rotating the database password” have never been executed here and are marked UNREHEARSED in the document, including an ordering window between two independently-refreshing 5m ExternalSecrets that has never been observed.

• Owner: the first operator with a real reason to rotate this credential — a compromise, a departure, a policy clock. Rotating merely to rehearse would spend the irreversibility for evidence about someone else’s software.

• Detection: test_seed_password_alphabet_round_trips and test_urlquery_round_trip_detects_space_bearing_alphabet in tools/litellm-verify/tests/test_fnd_static_gates.py, plus test_dsn_template_applies_urlquery, all on every task litellm-verify:gates run and in CI. They go red if the alphabet widens, if a character stops surviving the round trip, if the detector stops detecting, or if the template stops escaping. No detection exists for the unrehearsed procedure, and none is proposed: it is a document, and the check on it is a human running it. That is recorded here as procedural rather than dressed up as a gate.

• OPERATOR RULING 2026-08-30 (plan 03-20, decision 1) — amend-with-offline-proof. Roadmap SC#3 asked for a rehearsed rotation of both a virtual key and this database password. The virtual-key half was rehearsed against fovea that day; the database half was put to the operator behind D-54’s gate and ruled: amend SC#3 to record this residual’s offline proof as sufficient, rather than perform the rotation. The reason given, verbatim in 03-20-SUMMARY.md, is that the one property we own is proven exhaustively over all 81 generator characters where a live rotation samples a random 32 — strictly stronger evidence for that property — while the other three are third-party behaviour we would not gate on even after a clean live run, set against a one-way act whose blast radius is CNPG main, shared with mealie, temporal and octopus. The fovea rehearsal supplied a supporting figure: a rotation’s LiteLLM leg was sub-second and its delivery leg 63.1 s, never completing without a hand-issued command — and this procedure’s equivalent legs are unmeasured for a shared cluster rather than a single consumer.

• THIS RULING DOES NOT CLOSE R-20. Nothing was rotated, no Vault property was written, and the UNREHEARSED markers in docs/operations/litellm.md are untouched and accurate. The residual keeps its reopen condition exactly as stated in its Owner line: the first operator with a real reason — a compromise, a departure, or a policy clock. Amending a criterion to state what the evidence supports is not the same act as retiring the residual that says the procedure is unrehearsed, and reading “SC#3 met” as “the database rotation has been performed” would be wrong.

R-21 — dimension (a) on the two OpenRouter embedding rows is provider-dependent, and flipped inside one hour

• What: lane B echoes the serving provider’s own naming of the model, and OpenRouter fans a single embedding slug across providers that name it differently. Measured on 2026-08-30, on the same row, forty minutes apart: llm-embeddings answered parasail-bge-m3 on lane B (FAILED _model_matches against the declared baai/bge-m3, recorded as TWO-SIDED-FAILED) and then BAAI/bge-m3 (PASSED). llm-fovea-embed passed on Qwen/Qwen3-Embedding-8B in the same session and is exposed to the identical mechanism.
• Why it is recorded rather than fixed: the honest reading is that the green cell in the table above is a sample, not a guarantee — the cell says so, and this entry is why. The tempting fix is to loosen _model_matches until every provider spelling passes, which would buy a permanently green cell with an instrument that can no longer tell a wrong lane from a differently-named one. The second tempting fix is to pin the provider, which would assert a third party’s routing decision. Neither is taken.
• What it is NOT: it is not evidence that the two lanes reach different models. Dimensions (c) and (d) agreed exactly on both observations — 1024 dimensions and 12/12 tokens on llm-embeddings in both — so the model reached was the same; only the string naming it differed.
• Owner: whoever next runs the parity module. A red on either embedding row’s dimension (a) is to be classified — provider renaming (this residual) or a genuinely wrong lane — and not silenced.
• Detection: the assertion itself, on any live parity run, plus the durable artifact: a failing row now writes llm02-parity-<route>.json with evidence_class: TWO-SIDED-FAILED and the failing dimension verbatim in observations.failed_dimension, so a flip is citable after the fact rather than living only in a terminal that has scrolled.

R-22 — lane B reports an 8× prompt-token count on llm-chat-zdr for the same body

• What: the 2026-08-30 session sent the identical 5-word probe to both lanes on llm-chat-zdr and the artifact recorded 11 prompt tokens on the LiteLLM leg and 90 on lane B, with total tokens 19 against 100 and cost 3.78e-06 against 1.45e-05. Every other chat row agreed on prompt tokens exactly (rows 4, 5, 10 and 11 all match to the token). The two lanes were served by different providers on this row — Novita against SiliconFlow.
• Why it is recorded and not chased here: dimension (d) as D-45 defines it is token accounting is comparable, which the harness asserts as present and non-zero on both lanes; it passed. The 79-token gap is a real observation that the dimension as defined does not fail on, and inventing a tighter numeric bound at compile time would be a threshold chosen to match one measurement. The plausible causes — a provider-side system prompt on the SiliconFlow route, or a different tokenizer — are both third-party behaviour, and this estate does not assert those.
• Consequence if real: ZDR chat spend under agentgateway would have been over-reported relative to LiteLLM for this lane, which makes the cutover look more expensive than it was. That is a reporting question, not a correctness one.
• Owner: whoever next reconciles per-lane cost between the two planes, before Phase 5 removes lane B and the comparison becomes impossible.
• Detection: the artifact itself — llm02-parity-llm-chat-zdr.json records both lanes’ full usage blocks, so a re-run either reproduces the gap or does not. There is deliberately no gate: a numeric bound here would assert a provider’s tokenizer.

R-23 — the bake-window time box SC#2 attributes to R-21 and R-22 lives in roadmap prose only

• What: ROADMAP SC#2 states that R-21 and R-22 “carry the same time box” as R-6 — OVERLAP-BAKE-ENDS: 2026-09-15. Neither entry carries a date, and neither references the bake window. R-21’s Detection is the live assertion plus the durable artifact; R-22’s is the artifact with deliberately no gate; both Owners are named by role rather than by date. Only R-6 and R-8 carry that marker mechanically — R-8 via test_agentgateway_overlap_bake_window_not_expired, which parses the # OVERLAP-BAKE-ENDS: marker in argocd/app-configs/agentgateway/secrets.yaml. So the time box on R-21 and R-22 exists in one sentence of the roadmap and nowhere else, and nothing fires when it expires.
• Why it is recorded rather than fixed here: wiring R-8’s gate to these two would make it fire on the date for a reason the gate cannot check. Whether R-21 and R-22 have been reconciled is a human judgement — provider renaming against a genuinely wrong lane in one case, per-lane cost reconciliation in the other — not a parseable fact, and a gate that goes red on a date without being able to say what would make it green is a calendar reminder wearing a test’s clothes. The alternative fix is to soften SC#2’s wording, which is the roadmap author’s call and not this register’s. Both are left to a person, and the claim is recorded as prose-only rather than dressed up as equivalence with R-6.
• Owner: whoever next touches SC#2’s wording or these two entries. The two acceptable outcomes are named: add the bake-window reference to R-21 and R-22, or soften SC#2 to “are expected to retire in the same window”.
• Detection: test_agentgateway_overlap_bake_window_not_expired going red on OVERLAP-BAKE-ENDS: 2026-09-15 is the signal that actually reaches a human. It fails for R-6 and R-8’s sake, and this entry is what tells that human R-21 and R-22 ride the same date without riding the same gate — which is the whole reason it is written down here rather than left in the roadmap sentence. REOPEN CONDITION: that red, or any earlier edit to SC#2 or to either entry. If 2026-09-15 passes with R-21 and R-22 still undated and still carrying no bake-window reference, SC#2’s “carry the same time box” is false as written, and the choice above stops being optional.

R-24 — the eleven parity artifacts backing SC#2 are gitignored, so SC#2’s evidence is not reproducible from a clean checkout

• What: SC#2 cites “eleven citable results/llm02-parity-<route>.json artifacts”. tools/litellm-verify/.gitignore:4 excludes results/, confirmed by git check-ignore -v tools/litellm-verify/results/. The artifacts are real and were read first-hand by 03-18 and by the phase-03 verifier — in the working tree that produced them. A clean checkout has none of them, so on a fresh clone the primary evidence for LLM-02 is the matrix prose above. 03-18’s own coverage entry D3 already records that the transcription from artifact to table is a human-judgment claim that no gate asserts; this entry records that the artifact side of that transcription is not durable either.
• Why it is accepted rather than fixed: this is a DESIGN question the operator deliberately deferred, not an oversight. The .gitignore line is itself a decision with a stated reason (T-02-01: run output carries endpoint names, pod names and live error text from a cluster, and the raw files are regenerated by re-running the suite). Committing evidence artifacts reverses that decision for one class of file, and whether an estate’s evidence artifacts belong in git — with their provenance, their staleness once the estate moves under them, and their leak surface — is a larger question than this matrix should settle by taking a convenient exception for itself. It is left open on purpose.
• Owner: whoever first has to defend SC#2 from a checkout that is not the working tree that produced the artifacts.
• Detection: git check-ignore -v tools/litellm-verify/results/ names the excluding rule and its file:line in one command, and that command is this entry’s check — it goes “red” the moment someone expects the artifacts to be in git and finds the rule that says they are not. There is deliberately no test: an assertion that the eleven files are present would be red on every clean checkout by construction, which is precisely the state the .gitignore intends, so it would report the design rather than a defect. REOPEN CONDITION: either (a) SC#2 has to be defended from a clean checkout — a reviewer, an audit, or this worktree being removed — or (b) Phase 5 retires lane B, after which the artifacts are the only surviving record of the two-lane comparison and cannot be regenerated, because the comparison itself no longer exists. (b) is the one with a real clock on it: it arrives with Phase 5’s removal of the llm-gw and openrouter-gw listeners.

Phase 5 handoff — the only enumeration of its inherited scope that exists

Phase 5 currently reads Plans: TBD in the roadmap. This list is therefore the only place its inherited scope is written down. Each item names the gate that makes it visible.

Inherited item	Where	Gate that surfaces it
The llm-gw and openrouter-gw listeners and their eleven routes	argocd/app-configs/agentgateway/{llm-routes,llm-policies,openrouter-generic}.yaml	test_rollback_path_intact — asserts all eleven routes are still declared (EXPECTED_ROLLBACK_ROUTE_COUNT = 11)
The agentgateway scrape job	argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml	test_agentgateway_scrape_retained — inverted in Phase 5, not deleted (D-48)
The agentgateway dashboards	tools/hyperdx/dashboards/agentgateway-llm-mcp-usage.json	test_agentgateway_dashboard_retained — inverted in Phase 5, not deleted
The three overlap accept-list entries	argocd/app-configs/agentgateway/secrets.yaml	test_agentgateway_overlap_entries_present (inverted in Phase 5) and test_agentgateway_overlap_bake_window_not_expired (R-8)
The CoreDNS rewrite for the old host	argocd/app-configs/coredns-custom/coredns-custom.yaml	The file is entry 4 of CONSUMER_COMMENT_ALLOW_LIST in test_no_consumer_points_at_llm_gw, whose set equality fails when the mention is removed without the list being updated

Phase 5 also inherits the eight-entry CONSUMER_COMMENT_ALLOW_LIST itself: every entry names Phase 5 as its remover, and the list is asserted by set equality in both directions, so removing an entry’s mention without removing the entry is as red as adding an unlisted one.

Two residuals above have a Phase 5 deadline rather than a Phase 5 owner: R-13 and R-14 can only be closed while lane B still exists. Removing the listeners closes the window on ever measuring them.

Rollback — what remains in place, and until when

The previous gateway’s listeners and all eleven routes remain declared, and its scrape path remains in place. agentgateway serves no consumer traffic — every consumer manifest under argocd/ outside the retained agentgateway/ directory was verified to carry no llm-gw value, by test_no_consumer_points_at_llm_gw’s scan A over a 391-file git ls-files corpus — but the plane is up, and it is up deliberately, as the D-43 rollback path through the bake.

That is what makes the rollback observable for as long as it exists: the routes are asserted present by test_rollback_path_intact, the scrape job by test_agentgateway_scrape_retained, and the three overlap credentials by test_agentgateway_overlap_entries_present. The one-line D-43 rollback is reverting headroom’s upstream URL, which lands octopus, karakeep and mealie back on a plane that still accepts what their pods hold.

All of it is removed in Phase 5, not here. ROADMAP SC#1’s final clause — that llm-gw is removed in Phase 3 — is superseded by D-43 and by LLM-02’s own “after bake” wording. The bake ceiling is 2026-09-15, machine-checked.