LiteLLM Lane Parameter Measurements

Measured: 2026-08-16 Artifact under measurement: litellm==1.96.2 — the exact GOV-02-pinned version, obtained with uv pip install 'litellm==1.96.2' into an isolated virtualenv. Nothing was installed into the repo; no dependency enters tools/. Purpose: answer RESEARCH Open Questions 1, 2 and 3 by measurement, in the exact key/value form plan 03-06 will apply, and before 03-13 moves engram.

Provenance convention

This document uses the same convention as 03-RESEARCH.md. Every claim is one of:

• [MEASURED: <module>:<lines>] — that file was opened and read at those lines. Module paths are relative to the installed litellm package root.
• [MEASURED: executed] — proven by running litellm 1.96.2; the captured output is quoted.
• [NOT MEASURED: <what blocked it>] — no measurement was made. A recommendation with no measurement behind it is labelled RECOMMENDATION, never stated as a finding.

Repo claims cite path:lines and quote verbatim.

Environment: macOS aarch64, CPython 3.13.5, uv 0.12.3. The retry/timeout probes ran against a local capture server on loopback (no provider traffic). The Open Question 3 probes issued real calls to Google’s embeddings API — see that section’s disclosure.

---

Open Question 1 — what replaces llm-embeddings-retry and llm-summarize-engram-retry?

The property under test

The engram summarize lane’s tuning is measured and load-bearing. From argocd/app-configs/agentgateway/llm-routes.yaml:90-100, verbatim:

# This lane had NO timeout and NO retry — the worst offender (7.1% of
# summary calls timed out at the 30.0s wall, ClickStack otel_traces
# 2026-06-28). Same per-try/overall budget as the embed lanes above:
# backendRequest=8s per-try (engages with llm-summarize-engram-retry in
# llm-policies.yaml) + 28s overall < engram's 30s client deadline.

with timeouts: { request: 28s, backendRequest: 8s } at llm-routes.yaml:100, and the paired retry policy attempts: 3, backoff: 1s, codes: [503] at argocd/app-configs/agentgateway/llm-policies.yaml:170-174. The client deadline it is sized against is ENGRAM_SUMMARY_TIMEOUT: 30s at argocd/app-configs/agent-memory/summarize-cronjob.yaml:83-84.

The sibling embed policy llm-embeddings-retry is attempts: 4, backoff: 1s, codes: [503] at llm-policies.yaml:146-150, targeting three routes at llm-policies.yaml:141-145: llm-embeddings (octopus), llm-embeddings-engram-v2 (engram) and llm-fovea-embed (fovea).

So the question is whether LiteLLM can express: a per-attempt bound sitting under an overall bound, with a bounded attempt count, on a chosen status code.

(a) num_retries — per-deployment, Router-level, or both? BOTH, plus per-request.

num_retries is a per-deployment litellm_params field [MEASURED: litellm/types/router.py:417] — it appears in LiteLLMParamsTypedDict alongside timeout, stream_timeout and max_retries at litellm/types/router.py:384-386.

It is also a Router constructor argument [MEASURED: litellm/router.py:346], defaulted at [MEASURED: litellm/router.py:594-599], verbatim:

        if num_retries is not None:
            self.num_retries = num_retries
        elif litellm.num_retries is not None:
            self.num_retries = litellm.num_retries
        else:
            self.num_retries = openai.DEFAULT_MAX_RETRIES

litellm.num_retries defaults to None and openai.DEFAULT_MAX_RETRIES is 2 [MEASURED: executed]. This is the single most consequential default in this section: a lane that declares nothing still retries twice — three attempts. Today most agentgateway routes carry no retry policy at all, so transcribing them without an explicit num_retries is a silent behaviour change, not a no-op. See the CONCLUSION table’s “lanes that must declare num_retries: 0” row.

Composition, in precedence order [MEASURED: litellm/router.py:6457-6462, 6486-6493]:

1. A per-request num_retries in the call kwargs wins outright (request_num_retries).
2. Otherwise the failing deployment’s litellm_params.num_retries wins — it is attached to the exception at litellm/router.py:2943-2958 and read back at litellm/router.py:6486-6493.
3. Otherwise the Router-level value, else 0.
4. A matching retry_policy entry overrides all of the above — see (c).

Verbatim [MEASURED: litellm/router.py:6488-6493]:

            if (
                request_num_retries is None
                and deployment_num_retries is not None
                and isinstance(deployment_num_retries, int)
            ):
                num_retries = deployment_num_retries

Confirmed by execution — probes A and B below produce three upstream attempts from the deployment field and from the Router field respectively.

(b) Does timeout bound one attempt or the whole request? Per-attempt: YES. Overall: NO.

This is the load-bearing answer, and it is the inverse of the failure RESEARCH feared. LiteLLM does have a per-attempt bound. What it does not have is agentgateway’s overall request: bound.

Every timeout knob — deployment timeout, deployment request_timeout, litellm_settings.request_timeout, and router_settings.timeout — resolves through one precedence chain into a single value that is passed to a single attempt [MEASURED: litellm/router.py:3168-3178], verbatim:

        timeout = (
            kwargs.get("timeout", None)  # the params dynamically set by user
            or kwargs.get("request_timeout", None)  # the params dynamically set by user
            or data.get("timeout", None)  # timeout set on litellm_params for this deployment
            or data.get("request_timeout", None)  # timeout set on litellm_params for this deployment
            or self.request_timeout  # litellm_settings.request_timeout (per-attempt)
            or self.timeout  # timeout set on router (router_settings.timeout)
            or self.default_litellm_params.get("timeout", None)
        )

They are alternatives, not nested bounds. The retry loop at [MEASURED: litellm/router.py:6556-6571] re-invokes make_call with the same resolved timeout on each attempt and applies no wall-clock budget across attempts.

[MEASURED: executed] — probe F sets router_settings.timeout=2 with num_retries=2 against an upstream that sleeps 5s:

F : hits=3 offsets=[0.0, 2.94, 6.38] gaps=[2.94, 3.44] total=10.74s outcome=Timeout

Three attempts, each cut at 2s, total wall 10.74s. If router_settings.timeout were an overall bound the call would have failed at ~2s. It is not.

PER-ATTEMPT TIMEOUT EXISTS: YES — the deployment’s timeout bounds a single attempt. OVERALL REQUEST TIMEOUT EXISTS: NO — no Router setting bounds the sum of attempts.

Consequence for the engram lane, in its own terms. The agentgateway shape was per-try 8s under an overall 28s under the client’s 30s. Under LiteLLM the middle term disappears. The effective wall becomes (num_retries + 1) × timeout + Σ backoff + overhead, which is computed, not enforced — nothing in the proxy stops it being exceeded. That is a named residual, recorded in the CONCLUSION table, not a solved problem: if engram’s client deadline is ever shortened, or a lane’s timeout is raised, no LiteLLM-side guard notices.

The 7.1% regression does not return, because the per-try property that fixed it survives intact. What is lost is the belt-and-braces overall cut.

(c) Which status codes retry, and is a 503-only policy expressible? No — 503-only is NOT expressible.

Default retryable statuses [MEASURED: litellm/utils.py:6346-6372], verbatim:

def _should_retry(status_code: int):
    """
    Retries on 408, 409, 429 and 500 errors.
    ...
    """

with if status_code >= 500: return True at litellm/utils.py:6369-6370. Gate applied at [MEASURED: litellm/router.py:6702-6706].

So LiteLLM retries 408, 409, 429 and every 5xx. The agentgateway policies are codes: [503] only. The LiteLLM policy is strictly broader.

retry_policy does not narrow it, because retry_policy is keyed by exception type, not status code [MEASURED: litellm/types/router.py:95-109]:

class RetryPolicy(BaseModel):
    BadRequestErrorRetries: Optional[int] = None
    AuthenticationErrorRetries: Optional[int] = None
    TimeoutErrorRetries: Optional[int] = None
    RateLimitErrorRetries: Optional[int] = None
    ContentPolicyViolationErrorRetries: Optional[int] = None
    InternalServerErrorRetries: Optional[int] = None

and it is resolved at [MEASURED: litellm/router_utils/get_retry_from_policy.py:17-52], which consults AuthenticationError, Timeout, RateLimitError, ContentPolicyViolationError and BadRequestError — and never InternalServerErrorRetries.

RetryPolicy.InternalServerErrorRetries is declared but never read. It is the only field of the model with no branch in the resolver. A 5xx-scoped retry policy written against it is inert and fails silently — the Router falls through to the plain num_retries.

[MEASURED: executed] — probe E sets retry_policy={"InternalServerErrorRetries": 1} with num_retries=3 against an upstream returning 503:

E : hits=4 offsets=[0.0, 0.55, 1.79, 3.86] gaps=[0.55, 1.24, 2.07] total=8.15s outcome=ServiceUnavailableError

Four attempts — 1 + num_retries, i.e. the policy did nothing. Contrast probe D, where a field the resolver does consult takes effect: retry_policy={"TimeoutErrorRetries": 1} with num_retries=3 against a slow upstream:

D : hits=2 offsets=[0.0, 2.53] gaps=[2.53] total=5.86s outcome=Timeout

Two attempts — the policy overrode num_retries=3.

Practical reading: transcribe codes: [503] as “retry on transient upstream failure” and accept the wider set. For the embed lanes this is arguably an improvement (a 502 or 504 from OpenRouter is just as transient as a 503). The one status worth thinking about is 429, which is retryable and is the one status that can trigger a cooldown — see (d).

(d) allowed_fails and cooldowns — can a retry policy cool a lane out of rotation? Only if you set allowed_fails. Do not.

D-31 gives one model_list entry per lane, so every lane here is a single-deployment model group. That matters, because the current (“v2”) cooldown logic deliberately protects single-deployment groups [MEASURED: litellm/router_utils/cooldown_handlers.py:193-238]: the 429 branch and the error-rate branch are both guarded by not is_single_deployment_model_group, and the remaining branch needs total_requests_this_minute >= SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD, which is 1000 [MEASURED: litellm/constants.py:75-77].

But that whole block is reached only when allowed_fails is unset. Setting it switches to the legacy path [MEASURED: litellm/router_utils/cooldown_handlers.py:198-201, 239-244], which cools a deployment out after allowed_fails failures with no single-deployment guard at all [MEASURED: litellm/router_utils/cooldown_handlers.py:381-409]:

    if updated_fails > allowed_fails:
        return True

litellm.allowed_fails defaults to 3, and Router.allowed_fails is treated as “set” only when it differs from that module default [MEASURED: litellm/router_utils/cooldown_handlers.py:412-426].

[MEASURED: executed] — six consecutive 503s at a single-deployment lane, with and without allowed_fails:

H  allowed_fails NOT set (v2 logic), single-deployment lane, 6 consecutive 503s
H : outcomes=['ServiceUnavailableError', ...x6]
H : cooldown_list=[]

I  allowed_fails=2 set on router (switches to legacy v1 logic), same 6x 503s
I : outcomes=['ServiceUnavailableError', 'ServiceUnavailableError', 'ServiceUnavailableError', 'COOLED', 'COOLED', 'COOLED']
I : cooldown_list=['cd80bb0942ebaff010f63470ec03f5370239695a079af2f1fe6f388b4d341005']

Without allowed_fails, six failures leave the cooldown list empty — the lane stays in rotation. With allowed_fails=2, the lane is removed from rotation after three failures and subsequent requests fail with no deployment available, which on a single-deployment lane is a total outage of that lane, not a failover.

RECOMMENDATION: do not set allowed_fails, allowed_fails_policy or cooldown_time on these lanes, and do not set them at router_settings either — the switch is global. With them unset, a retry policy cannot silently cool a lane out of rotation. Default cooldown duration, for reference, is 5s [MEASURED: litellm/constants.py:32].

(e) Backoff shape — exponential with jitter, partly configurable

[MEASURED: litellm/router.py:6780-6823] delegates to _calculate_retry_after [MEASURED: litellm/utils.py:6409-6432], verbatim:

    # Add some jitter (default JITTER is 0.75 - so upto 0.75s)
    jitter = JITTER * random.random()
    ...
    # Calculate exponential backoff
    num_retries = max_retries - remaining_retries
    sleep_seconds = INITIAL_RETRY_DELAY * pow(2.0, num_retries)

    # Make sure sleep_seconds is boxed between min_timeout and MAX_RETRY_DELAY
    sleep_seconds = max(sleep_seconds, min_timeout)
    sleep_seconds = min(sleep_seconds, MAX_RETRY_DELAY)

    return sleep_seconds + jitter

with INITIAL_RETRY_DELAY = 0.5, MAX_RETRY_DELAY = 8.0, JITTER = 0.75 [MEASURED: litellm/constants.py:329-331]. min_timeout is the Router’s retry_after [MEASURED: litellm/router.py:6813, 6820].

So the shape is exponential, jittered, floored at retry_after, capped at 8s — not agentgateway’s constant backoff: 1s. It is configurable only through retry_after (a floor) and the INITIAL_RETRY_DELAY / MAX_RETRY_DELAY / JITTER environment variables, which are process-global and therefore not a per-lane knob.

Two behaviours worth recording:

• An upstream Retry-After header, when present and 0 < value <= 60, overrides the calculation [MEASURED: litellm/utils.py:6420-6422].
• If other healthy deployments exist in the same model group, the backoff is skipped entirely and the retry is immediate [MEASURED: litellm/router.py:6796-6800]. Under D-31’s one-entry-per-lane design there are none, so the exponential backoff always applies here.

Measured probe results

All probes ran against a loopback capture server; no provider traffic. hits counts upstream requests actually received.

Probe	Configuration	Upstream behaviour	Attempts	Total wall	Finding
A	deployment timeout: 2, deployment num_retries: 2, router num_retries=0	sleeps 5s	3	11.38s	per-deployment num_retries honoured; timeout is per-attempt
B	deployment timeout: 2, router num_retries=2	sleeps 5s	3	10.50s	Router-level num_retries honoured identically
C	deployment timeout: 2, deployment num_retries: 2	503 immediately	3	5.17s	503 is retried by default, no policy needed
D	retry_policy={"TimeoutErrorRetries": 1}, router num_retries=3	sleeps 5s	2	5.86s	retry policy overrides num_retries for a consulted exception type
E	retry_policy={"InternalServerErrorRetries": 1}, router num_retries=3	503	4	8.15s	policy inert — field never read by the resolver
F	router_settings.timeout=2, router num_retries=2	sleeps 5s	3	10.74s	no overall wall — router timeout is also per-attempt
H	single-deployment lane, allowed_fails unset	6× 503	—	—	cooldown list stays empty; lane stays in rotation
I	single-deployment lane, allowed_fails=2	6× 503	—	—	lane cooled out of rotation after 3 failures

Sizing the engram lane against its real client deadline

Because there is no overall wall, the engram lane’s budget has to be measured, not derived — and the arithmetic under-predicts. Against a permanently-slow upstream, worst of two runs each:

Candidate	Attempts	Measured worst wall	Headroom vs ENGRAM_SUMMARY_TIMEOUT: 30s
timeout: 8, num_retries: 2	3	29.07s	+0.93s
timeout: 7, num_retries: 2	3	25.72s	+4.28s
timeout: 6, num_retries: 2	3	23.10s	+6.90s
timeout: 8, num_retries: 1	2	18.52s	+11.48s

[MEASURED: executed]. The obvious transcription — carry agentgateway’s per-try 8s straight across — lands at 29.07s against a 30s client deadline: under 1 second of margin, and no overall wall to enforce even that. It would have looked correct by arithmetic (3 × 8 + 2 backoffs ≈ 27s) and been wrong in practice; the measured overhead beyond attempts-plus-backoff is a consistent ~2.5–3s, which this document records as observed rather than explained.

timeout: 7, num_retries: 2 is the recommended sizing: it keeps agentgateway’s three attempts and a near-identical per-try cut, and restores a real margin (4.28s) under the client deadline.

---

CONCLUSION — the exact litellm_params plan 03-06 must write

One row per agentgateway policy being replaced. Values are transcription-ready.

Replaced policy	LiteLLM lane(s)	litellm_params to write	Basis
llm-summarize-engram-retry (attempts: 3, backoff: 1s, codes: [503] + route request: 28s, backendRequest: 8s)	engram-summarize	timeout: 7 num_retries: 2	3 attempts preserved; per-attempt bound preserved (b); sizing measured against the real 30s deadline, not derived
llm-embeddings-retry (attempts: 4, backoff: 1s, codes: [503]) — target llm-embeddings, route request: 120s, backendRequest: 60s	octopus-embed	timeout: 60 num_retries: 1	agentgateway’s own 120s overall wall already capped this at ~2 full 60s attempts; 2 attempts is the faithful reproduction, not a reduction
llm-embeddings-retry — target llm-embeddings-engram-v2, route request: 60s, no per-try	engram-embed	timeout: 60 num_retries: 3	keeps the 60s single-attempt budget and all 4 attempts. Residual: with the 60s overall wall gone, a slow-path worst case is ~4×60s rather than 60s. If 03-12 or 03-13 observe a long tail, num_retries: 1 caps it at ~123s
llm-embeddings-retry — target llm-fovea-embed, route request: 60s, backendRequest: 45s	fovea-embed	timeout: 45 num_retries: 1	the 60s wall admitted one full 45s attempt plus a fragment; 2 attempts is the closest reproduction that still retries
(no retry policy today) — route llm-embeddings-gemini, request: 60s	gemini-embed	timeout: 60 num_retries: 0	this route is not in llm-embeddings-retry’s targetRefs (llm-policies.yaml:141-145). num_retries: 0 must be explicit — the Router default is 2 (a)
(no retry policy today) — chat routes llm-chat, llm-chat-zdr, llm-fovea-scout, llm-fovea-deepdive, openrouter-generic*	openrouter/*, openrouter-zdr/*, fovea-scout, fovea-deepdive	num_retries: 0	Silent-default trap. These carry no retry today. Omitting num_retries gives them 3 attempts, changing behaviour and multiplying spend on a retried completion
llm-max-buffer (32Mi frontend buffer)	—	no LiteLLM equivalent written	No measured analogue exists. See Open Question 2

Do not set allowed_fails, allowed_fails_policy or cooldown_time on any lane or in router_settings — measured in (d) to remove a single-deployment lane from rotation entirely.

Do not express the 503-only constraint via retry_policy — InternalServerErrorRetries is inert (c). Accept the wider default set (408/409/429/5xx).

Named residuals carried out of Open Question 1

1. No overall request bound exists. Every lane’s wall is (num_retries + 1) × timeout + Σ backoff + ~2.5–3s overhead, computed and unenforced. A future change to any client deadline has no LiteLLM-side guard. Carried into the parity matrix as a known behavioural difference from agentgateway, not as a defect to fix in this phase.
2. Retry surface is wider than 503. 408, 409, 429 and all 5xx retry. Deliberate, accepted, and recorded here so a parity diff does not score it as a regression.
3. Backoff shape changes from constant 1s to exponential-with-jitter (0.5s, 1.0s, … capped 8s). Absorbed into the measured sizing above.
4. engram-embed’s slow-path worst case grows from a 60s wall to ~4×60s. Watch during 03-13.

---

Open Question 2 — is a 32Mi body-buffer equivalent needed?

Status: [NOT MEASURED — no LiteLLM analogue was found to measure, and the failure it guards against cannot be reproduced without production-scale batched embeds.]

What the policy does today

argocd/app-configs/agentgateway/llm-policies.yaml:112-128, verbatim:

# Raise the in-memory body buffer above the 2 MiB default. The AI processing
# path buffers the full LLM response to parse usage/tokens; a batched embedding
# response can exceed 2 MiB and is rejected with 503 "response was too large",
# failing octopus repo indexing (hl-0we). Gateway-scoped so it covers the llm-gw
# listener — both octopus (baai/bge-m3, 1024-dim) and engram (qwen3-embedding-8b,
# 4096-dim, the larger vectors). 32Mi = headroom.

It exists because agentgateway buffers the whole response to parse usage, and a batched embeddings response blew a 2 MiB default. LiteLLM also parses usage, so the same class of limit could exist — but no equivalent knob was found in the Router or proxy configuration surface, and no buffer-and-reject behaviour was measured. This section therefore records a disposition and a test, not a finding.

The deliberate test

The test is the octopus cutover in plan 03-12, which carries the largest batched embeds in the estate. Octopus batches 64 items per embeddings request — argocd/app-configs/octopus/deployment.yaml:81-85, verbatim:

            # Batch size carried over from the seattle lane. Keeps each embeddings
            # response well under the gateway's 32Mi buffer; OpenRouter/DeepInfra
            # is faster than the old seattle box, so this is comfortably conservative.
            - name: EMBEDDING_BATCH_ITEMS
              value: "64"

EMBEDDING_BATCH_ITEMS: "64" is therefore not an arbitrary number — its comment ties it directly to the 32Mi buffer this question is about. It is the dial to turn if the failure appears.

The failure signature to watch for

An opaque 503 on a batched embeddings response — the hl-0we failure the policy was created to fix, in which a single failed embed aborts an entire octopus repo index. It presents as an embeddings-only failure that scales with batch size: small batches succeed, full 64-item batches fail. A 503 that is uniform across batch sizes is an upstream transient and a different problem.

The rollback

One line: revert headroom’s upstream URL — OPENAI_TARGET_API_URL at argocd/app-configs/headroom-apps/deployment.yaml:49-50 — back to https://llm-gw.fzymgc.house/v1. Per D-43 the llm-gw listener stays up through the bake precisely so this revert is available; it is a merge, not a live edit (root CLAUDE.md: ArgoCD manages deployments).

If the signature appears and the rollback is taken, the second dial before re-attempting is EMBEDDING_BATCH_ITEMS, lowered from 64.

---

Open Question 3 — which LiteLLM provider serves the Gemini embed lanes?

Why this is the corpus-safety question

The agentgateway lane deliberately runs in Passthrough rather than Embeddings mode. From argocd/app-configs/agentgateway/llm-backends.yaml:139-147, verbatim:

        # Passthrough (NOT Embeddings): return Google's response verbatim.
        # agentgateway <=v1.3.1's OpenAI embeddings parser requires a `usage`
        # field that Google's /v1beta/openai/embeddings omits -> 503 "missing
        # field usage". Fixed upstream (#2455) but unreleased as of
        # v1.4.0-alpha.1.

That usage omission is the property this question had to re-measure against LiteLLM, because it is what would zero the lane’s spend. The dimension is the more dangerous half. The reindex Job records the hazard in its own words — argocd/app-configs/agent-memory/reindex-memory-v4-job.yaml:60-61, verbatim:

#     Running the reverse gap-fill with the Gemini env (3072) CORRUPTS the 4096
#     memory_v3 rollback target — Qdrant rejects the dim mismatch (Pitfall 4).

and pins the contract at reindex-memory-v4-job.yaml:114-115, verbatim:

            - name: ENGRAM_EMBED_DIM
              value: "3072"  # the dim memory_v4 is created at; MUST equal the returned vector length

“MUST equal the returned vector length” is the acceptance bar. It is answered below by a captured vector length, not by an asserted number.

Disclosure — the probe that was issued

Two real calls per option were issued to Google’s embeddings API from the measurement environment (T-03-05-C, accepted). The probe input is a fixed non-sensitive string authored for this measurement:

litellm lane dimension probe: fixed non-sensitive string

No estate data was used. The credential was read from Vault (secret/fzymgc-house/cluster/agentgateway, property gemini_api_key — the same paid key the live lane already uses per argocd/app-configs/agentgateway/secrets.yaml:186-191), held only in the probe process environment, and deleted afterwards. It appears nowhere in this document or in the repository. The Gemini lane is already a paid, non-ZDR production lane, so this adds no new exposure class.

The two options, measured

Dimension of comparison	Option 1 — gemini/gemini-embedding-2	Option 2 — openai/ + api_base shim
Endpoint actually called	.../v1beta/models/gemini-embedding-2:batchEmbedContents	.../v1beta/openai/embeddings
Request shape	native Gemini requests[].content.parts[].text	OpenAI {"input": [...], "model": ...}
usage on the raw upstream response	NO — but usageMetadata.promptTokenCount is present	NO — and nothing replaces it
usage on the litellm response	YES, populated (prompt_tokens=11)	YES, but zero-filled (prompt_tokens=0)
Vector dimension produced	3072	3072
Explicit dimension parameter	accepted and honoured	not measured
Resolves in litellm’s cost map	PRESENT — mode=embedding, input_cost_per_token=2e-07, output_vector_size=3072	ABSENT
Computed spend on the probe call	2.2e-06	raises This model isn't mapped yet

[MEASURED: executed] for every cell except the one marked not measured.

usage present on the chosen option’s response: YES. The precise finding is worth stating carefully, because the two surfaces fail differently and only one is recoverable:

• Neither Google surface returns an OpenAI-shaped usage object. Confirmed by raw capture below.
• The native surface returns usageMetadata.promptTokenCount, which litellm maps into a populated usage on its own response. Token accounting survives, so spend is real.
• The OpenAI-compat surface returns no token accounting at all. litellm still constructs a usage object, so the field is structurally present — but zero-filled. Combined with the slug being absent from the cost map, this lane would report $0 spend while billing real money to Google. That is the KEY-03 silent-zero failure of Pitfall 4, on the estate’s only paid embed lane.

This settles the provider choice on its own, before the dimension question: Option 1, the native gemini/gemini-embedding-2 route.

Observed evidence

Captured from the run, not retyped. Vector values are redacted; the length is the measurement, and pasting 3072 floats into a committed document would serve nothing.

Invocation issued (Option 1, native route):

litellm.embedding(model='gemini/gemini-embedding-2', input=['litellm lane dimension probe: fixed non-sensitive string'], api_key=<REDACTED>)

Endpoint the invocation actually reached, and the emitted request body:

ENDPOINT CALLED: https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-2:batchEmbedContents
REQUEST BODY: {"requests": [{"model": "models/gemini-embedding-2", "content": {"parts": [{"text": "litellm lane dimension probe: fixed non-sensitive string"}]}}]}

Response excerpt as litellm returned it (floats redacted, length preserved):

{
  "object": "list",
  "model": "gemini-embedding-2",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": "<3072 floats REDACTED>"
    }
  ],
  "usage": {
    "completion_tokens": 0,
    "prompt_tokens": 11,
    "total_tokens": 11,
    "completion_tokens_details": null,
    "prompt_tokens_details": null
  }
}

Raw upstream response bodies, so usage presence is shown rather than asserted. Native surface:

HTTP 200
top-level keys: ['embeddings', 'usageMetadata']
'usage' in response body?: NO
{
  "embeddings": [
    {
      "values": "<3072 floats REDACTED>"
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 12,
    "promptTokenDetails": [
      {
        "modality": "TEXT",
        "tokenCount": 12
      }
    ]
  }
}

OpenAI-compat surface — note the absent usage key, exactly as the agentgateway comment records:

HTTP 200
top-level keys: ['data', 'model', 'object']
'usage' in response body?: NO
{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": "<3072 floats REDACTED>"
    }
  ],
  "model": "gemini-embedding-2"
}

The captured vector length, on its own line in a fixed machine-readable shape:

OBSERVED len(embedding) = 3072

(One observed variance, recorded rather than smoothed: the same probe string was counted as 11 prompt tokens through litellm’s native request construction and 12 by a hand-rolled call to the same endpoint. The cause was not investigated; it does not bear on the dimension, and it is noted here so a later reader does not mistake the two captures for a contradiction.)

Dimension parameter, accepted or honoured? Asking for the default 3072 and receiving 3072 proves nothing, so a non-default value was requested. Setting dimensions: 1536 in a Router model_list entry’s litellm_params — the exact config layer 03-06 writes — emitted outputDimensionality: 1536 on the upstream request and returned a vector of length 1536 [MEASURED: executed]. The parameter is therefore accepted, carried from litellm_params, and honoured, not silently ignored.

The verdict

The produced dimension is the captured length above. The deployed engram corpus dimension is ENGRAM_EMBED_DIM: "3072" at reindex-memory-v4-job.yaml:114-115.

DIMENSION VERDICT: produced dimension 3072, deployed engram corpus dimension 3072 — MATCH

Reproduced at the point of decision, from reindex-memory-v4-job.yaml:60-61:

#     Running the reverse gap-fill with the Gemini env (3072) CORRUPTS the 4096
#     memory_v3 rollback target — Qdrant rejects the dim mismatch (Pitfall 4).

The contract for plan 03-13. The branch that moves a live embedder is reachable only from a positive measurement. A NOT MEASURED verdict is treated by 03-13 exactly as MISMATCH — both block the Gemini lane cutover in this phase rather than deferring the question. There is no third state in which a number appears on the verdict line without a capture behind it: the produced dimension on that line is required to equal the OBSERVED len(embedding) capture, and if the two ever disagree the document is not usable and the probe must be re-run.

On this run the verdict is MATCH, so 03-13’s embed cutover is cleared on the dimension question. It remains gated on everything else 03-13 asserts.

A durable guard, given the hazard. Because a wrong dimension corrupts the corpus silently, the lane entries below pin dimensions: 3072 explicitly rather than relying on Google’s default remaining 3072. The parameter is measured as honoured, so pinning converts a silent future drift into a loud, immediate mismatch at the lane.

The exact model_list entries plan 03-06 must write

Each field traces to a measurement above. Retry and timeout values come from the Open Question 1 CONCLUSION table.

  model_list:
    # engram's live embed lane (replaces route llm-embeddings-engram-v2).
    # Native gemini/ route, NOT an openai/-with-api_base shim: the OpenAI-compat
    # surface returns no token accounting and the slug is absent from litellm's
    # cost map, so that option reports $0 spend on the estate's only paid embed
    # lane (measured 2026-08-16).
    - model_name: engram-embed
      litellm_params:
        model: gemini/gemini-embedding-2
        api_key: os.environ/GEMINI_API_KEY
        # Pinned, not defaulted. MUST equal ENGRAM_EMBED_DIM and the dim
        # memory_v4 was created at; a mismatch corrupts the corpus silently.
        dimensions: 3072
        timeout: 60
        num_retries: 3
    # Permanent provider-named Google embeddings lane (replaces route
    # llm-embeddings-gemini). NOT covered by llm-embeddings-retry today, so
    # num_retries is 0 EXPLICITLY — the Router default is 2.
    - model_name: gemini-embed
      litellm_params:
        model: gemini/gemini-embedding-2
        api_key: os.environ/GEMINI_API_KEY
        dimensions: 3072
        timeout: 60
        num_retries: 0

input_cost_per_token is NOT required on either entry. The slug resolves in litellm’s cost map at input_cost_per_token=2e-07 with mode=embedding, and a probe call computed a non-zero cost of 2.2e-06 [MEASURED: executed]. This is the one embed lane in the estate that does not need declared pricing — Pitfall 4’s remedy applies to the OpenRouter embed slugs, not to this one.

Both lanes are single-deployment model groups, so the Open Question 1 rule holds: do not set allowed_fails on them.