fovea LLM role routing via agentgateway to ZDR open models

Design bead: hl-3w5i Related: hl-uh07 (fovea deploy epic), hl-uh07.9 (verify + go-live), hl-z5fc (mesh egress) Status: Design Date: 2026-06-25

Summary

fovea (0.4.x) reviews PRs in a scout → gate → deepdive pipeline. It currently runs Anthropic-direct (claude-sonnet-4-6 for every role). This spec routes fovea’s two LLM roles through the cluster’s agentgateway (llm-gw.fzymgc.house) to zero-data-retention (ZDR) open models on OpenRouter, and uses fovea’s 0.4.x multi-aspect scout to tier models per aspect:

• scout role → DeepSeek V4-Flash (cheap, high-volume scan). Reuses the existing openrouter-zdr lane.
• deepdive role → GLM-5.2 (agentic verifier emitting a strict-JSON verdict). New dedicated ZDR backend.
• Aspect tiering: correctness + security scout aspects route to the deepdive role (GLM-5.2); types + tests stay on the scout role (Flash).
• claude-sonnet-4-6 is retained as a documented break-glass fallback (not built).

This reverses the prior “fovea is intentionally Anthropic-direct” decision. The driver is cost (open models are 5–80× cheaper per output token than sonnet) and consolidation onto the cluster’s existing gateway, while keeping private PR diffs on ZDR endpoints. fovea stays in shadow mode (publish_enabled: false) throughout; go-live remains the separate hl-uh07.9 gate.

Goals

• Route both fovea LLM roles through agentgateway (single OpenAI-compatible dialect).
• Keep all code-bearing traffic on ZDR endpoints (matches the octopus/engram precedent).
• Use per-aspect tiering to spend GLM-5.2 only where it matters and Flash everywhere else.
• Preserve the response_format: json_schema + tool-calling guarantees deepdive depends on.
• No behaviour change for users: ship in shadow mode behind the existing go-live gate.

Non-goals

• Flipping publish_enabled: true (separate, gated by hl-uh07.9).
• Istio/mesh egress for fovea (hl-z5fc).
• Building the sonnet ZDR fallback backend (documented, deferred — YAGNI).
• Retrieval/embedding routing (retrieval.enabled: false today; out of scope).
• Any upstream fovea change (e.g. N-tier roles); we design within current capabilities.

Grounded facts & constraints

These shape the design; full grounding traces are on bead hl-3w5i.

1. fovea role schema (internal/config/config.go). gateway.roles is a map[string]RoleModel; each RoleModel = {provider (anthropic|openai, default openai), model, base_url (optional per-role openai endpoint override), api_key}. Required roles are scout and deepdive; each must set model. Per-role provider drives which credential is required; a provider: openai role with no base_url (role or gateway.openai.base_url) fails config validation at boot (this caused the 0.4.3 CrashLoopBackOff, fixed in #1380).

2. Closed role set → two-tier only (internal/core/aspect.go, internal/eyes/scout/registry.go, internal/llm/provider.go). Built-in scout aspects are correctness, security, types, tests. The per-aspect scout.aspects[<aspect>].model override is an llm.Role, not a model id, and validRole() accepts only scout or deepdive (unknown → warn + fall back to scout). So per-aspect tiering is bounded to the two role models; a third distinct scout model is impossible without an upstream change. The “strong scout tier” is therefore necessarily the deepdive role model.

3. Structured output (ADR fovea-fl2). deepdive’s verdict uses response_format: {type: json_schema, strict} on Chat Completions (schema from a Go struct via invopop/jsonschema). The gateway translates it to each backend’s native mechanism. The model’s OpenRouter endpoint must support structured_outputs + tools, or the request fails.

4. Single dialect (ADR fovea-m7c). All LLM calls go through one OpenAI-compatible gateway dialect — so the sonnet fallback also routes through the gateway (provider: openai), not a second Anthropic-native path.

5. OpenRouter mechanics. require_parameters: true forces routing only to provider endpoints that support the requested params (response_format + tools) and fails loud otherwise — the structural safety net against a silent re-run of the #17 verdict-parse failure. All chosen models advertise tools + structured_outputs + ZDR endpoints (verified live against GET /api/v1/models and ?zdr=true). Fusion and Auto Router are rejected: Fusion reports tools:false, structured_outputs:false, makes external web calls (violates ZDR), and is dynamically priced (~4–5× a completion).

6. agentgateway ZDR is backend-global (llm-policies.yaml, llm-backends.yaml). backend.ai.overrides (zdr, require_parameters) has no per-model selector, so each ZDR model needs its own dedicated AgentgatewayBackend (pinning provider.openai.model collapses aliases to it). Today the only ZDR chat lane is openrouter-zdr → deepseek/deepseek-v4-flash, reached via an x-agentgateway-backend header.

7. fovea is the cluster’s first ZDR chat client. Proven ZDR lanes today are all embeddings (octopus bge, engram qwen3). The existing ZDR chat route is header-gated and was left as a “reachability via real clients” live-verify gate. fovea’s RoleModel exposes per-role base_url but no custom-header field, so it cannot use the header route — driving the header-free, path-based routes below. The header-free path mechanism itself is proven: llm-embeddings-engram already serves a custom prefix /engram-embed/v1/embeddings → a dedicated ZDR backend (no rewrite filter, longest-prefix-matched). fovea mirrors that pattern for chat.

Model selection & rationale

Role / use	Model	OpenRouter (in/out per 1M)	Why
scout (types, tests aspects)	DeepSeek V4-Flash	$0.09 / $0.18	Cheapest; reliable structured output + tool use (TAU-bench 95); 3× faster than M3; reuses existing ZDR lane. Scout is recall-oriented with downstream verification.
deepdive + strong scout (correctness, security)	GLM-5.2	$0.95 / $3.00	Best open agentic/structured reliability (Terminal-Bench 81, MCP-Atlas 77, FrontierSWE 74); ~5× cheaper than sonnet.
break-glass fallback	claude-sonnet-4-6	$3.00 / $15.00	Known-good; the config #17 was verified against. Documented flip, not built.

MiniMax M3 was evaluated and rejected for the cheap tier: its ~6.7× output premium buys capability concentrated in domains scout does not exercise (browsing, long-horizon terminal agentics), while Flash matches or beats it on the scout-relevant axes and reuses the existing lane.

Architecture

1. agentgateway (argocd/app-configs/agentgateway/)

• New backend openrouter-zdr-glm in llm-policies.yaml (beside openrouter-zdr, where ZDR backends live): pins provider.openai.model: z-ai/glm-5.2, alias or-glm-5-2-zdr, overrides: {zdr: true, require_parameters: true}, auth agentgateway-openrouter.
• Reuse openrouter-zdr (deepseek-v4-flash) for the cheap tier — no new backend.
• Two header-free HTTPRoutes in llm-routes.yaml (where routes live), directly mirroring the proven llm-embeddings-engram route, which already serves a custom prefix /engram-embed/v1/embeddings → a dedicated ZDR backend with no URL-rewrite filter (the backend’s pathPrefix: /api/v1 does the upstream rewrite; longest-prefix-matched away from the canonical route):
  • llm-fovea-scout — /fovea-scout/v1/chat/completions → openrouter-zdr (Flash)
  • llm-fovea-deepdive — /fovea-deepdive/v1/chat/completions → openrouter-zdr-glm (GLM-5.2) These prefixes do not overlap /v1/chat/completions, so there is no match ambiguity. Modify (not create) the existing llm-apikey AgentgatewayPolicy in llm-policies.yaml to add llm-fovea-scout and llm-fovea-deepdive to its targetRefs, so the vkey gate covers them.
• New virtual key vk_fovea — see §3 (stored in both the agentgateway accept-list path and fovea’s own client path, per the karakeep/mealie pattern).

2. fovea config (argocd/cluster-app/templates/fovea.yaml valuesObject)

config:
  gateway:
    # global default endpoint; per-role base_url overrides below pin the routes
    openai: { base_url: "https://llm-gw.fzymgc.house/fovea-scout/v1" }
    roles:
      scout:
        provider: openai
        model: or-deepseek-v4-flash-zdr     # backend pins the model; string nominal
        base_url: "https://llm-gw.fzymgc.house/fovea-scout/v1"
      deepdive:
        provider: openai
        model: or-glm-5-2-zdr
        base_url: "https://llm-gw.fzymgc.house/fovea-deepdive/v1"
  scout:
    aspects:
      correctness: { model: deepdive }      # → GLM-5.2 (strong tier)
      security:    { model: deepdive }      # → GLM-5.2 (strong tier)
      # types, tests inherit the scout role (Flash) — no override needed

• The model strings are nominal: each ZDR backend pins its model, so the alias collapses regardless. They are kept meaningful for readability and to satisfy fovea’s “model is required” validation.
• Add openaiKey: { name: fovea-app, key: openai_api_key } → FOVEA_GATEWAY__OPENAI__API_KEY = the vk_fovea value. Retain anthropicKey (present but unused while both roles are provider: openai, so fovea requires no anthropic credential) — this keeps the break-glass flip to Anthropic-direct a one-line config change with no secret churn.
• Chart grounding for openaiKey: verified against the 0.4.3 chart — helm show values declares openaiKey (→ FOVEA_GATEWAY__OPENAI__API_KEY, “optional; gateway may be keyless”), and helm template oci://ghcr.io/seanb4t/charts/fovea --version 0.4.3 renders it to a secretKeyRef env (name: fovea-app, key: openai_api_key). The 2026-06-09 deploy spec’s three-key list (privateKeySecret/webhookSecret/anthropicKey) predates 0.4.0, which introduced the gateway/openai config surface (#23).
• gateway.openai.base_url is the global default endpoint; both roles set their own base_url, so the global is only a fallback (pointed at the scout lane). deepdive overrides it with its own lane. Intended semantics: default = scout lane; deepdive explicitly overrides.

3. Secrets / ESO

The vkey value is stored in two Vault locations, mirroring karakeep/mealie (client own-path openai_api_key value = the vkey; matching accept-list entry on the gateway path):

• Client credential — store the vk_fovea value as openai_api_key under Vault fzymgc-house/cluster/fovea. fovea’s ExternalSecret (argocd/app-configs/fovea/external-secret.yaml) reads it into fovea-app.openai_api_key, surfaced via the chart’s openaiKey → FOVEA_GATEWAY__OPENAI__API_KEY.
• Accept-list — store the SAME value as property vk_fovea under Vault fzymgc-house/cluster/agentgateway (where every vkey lives), and add a vk_fovea entry to the agentgateway-vkeys ExternalSecret (argocd/app-configs/agentgateway/secrets.yaml): a template.data line — fovea: '{"key":"{{ .vk_fovea }}","metadata":{"group":"service"}}' — plus a data entry (secretKey: vk_fovea, remoteRef → fzymgc-house/cluster/agentgateway property vk_fovea).
• The ESO reader policy already wildcards secret/data/* — no Vault policy change.

4. Network

• Confirm/permit fovea ns → llm-gw VIP (192.168.20.155) on 443, the same reachability every gateway client uses. No toFQDNs (broken on this cluster, hl-bdr); the VIP is an in-cluster MetalLB address.

5. Fallback (documented, deferred)

Break-glass for a GLM-5.2 deepdive regression: flip gateway.roles.deepdive back to provider: anthropic with the retained Anthropic key (one-line revert), or later add an openrouter-zdr-sonnet backend + /fovea-deepdive repoint. Not built now (YAGNI until verification fails).

Cost picture (order-of-magnitude)

Per review, roughly: scout = 4 aspect calls (2 on GLM-5.2, 2 on Flash) + ≤5 deepdives on GLM-5.2. Versus the all-sonnet baseline this cuts the high-volume scout output cost ~80× (types/tests on Flash) and ~5× on the GLM-routed work, while the absolute per-review delta between Flash and M3 for the cheap aspects is sub-cent — confirming model choice is driven by fit, not headline price.

Rollout & verification

1. Ship all changes with publish_enabled: false (shadow).
2. After sync: confirm fovea pod 1/1, ExternalSecrets synced, and a test PR drives scout fan-out across both tiers (Flash + GLM-5.2 visible in ClickStack telemetry) and deepdive reaches the verified funnel stage on GLM-5.2 with no require_parameters/structured-output failures.
3. Only then the separate hl-uh07.9 go-live flip.

Acceptance: Application Synced/Healthy; pod 1/1; vk_fovea authenticates (no 401 at the gateway); a real PR shows tiered scout models + a GLM-5.2 verified verdict in telemetry; ZDR confirmed — agentgateway telemetry/access logs show fovea’s requests routed to the openrouter-zdr / openrouter-zdr-glm backends (which carry the zdr: true + require_parameters: true override), cross-checked against the OpenRouter activity dashboard (ZDR endpoint, params honored).

Risks

• First ZDR chat client (R-1). The header-free path-prefix mechanism is itself proven (llm-embeddings-engram serves /engram-embed/v1/embeddings → a dedicated ZDR backend, no rewrite filter), so fovea mirrors a working pattern; the residual unknown is chat (vs embeddings) over a ZDR lane, covered by the live-verify step. Fallback: the header-based llm-chat-zdr route exists.
• Open-model verdict reliability (R-2). GLM-5.2 deepdive structured output is unproven in-cluster; require_parameters: true fails loud rather than silent, and the sonnet break-glass is one flip away.
• Model/alias drift (R-3). OpenRouter model ids or pricing can change; the backend pins explicit ids and the design is revertible per-role.
• Reasoning-mode cost (R-4). fovea has no per-role thinking-mode knob; GLM-5.2 / DeepSeek default reasoning effort applies. Watch deepdive max_tokens (8000) and wall_clock (90s) under live load; tune in a follow-up if budgets are exceeded.

Change surfaces

File	Change
argocd/app-configs/agentgateway/llm-policies.yaml	new openrouter-zdr-glm backend; modify llm-apikey policy targetRefs to add the two fovea routes
argocd/app-configs/agentgateway/llm-routes.yaml	two header-free fovea HTTPRoutes — llm-fovea-scout + llm-fovea-deepdive (mirror llm-embeddings-engram)
argocd/app-configs/agentgateway/secrets.yaml	add vk_fovea to the agentgateway-vkeys ExternalSecret (template.data + data)
argocd/cluster-app/templates/fovea.yaml	gateway roles + per-role base_url + aspect tiering; add openaiKey (retain anthropicKey unused)
argocd/app-configs/fovea/external-secret.yaml	add openai_api_key (the vk_fovea value)
Vault	…/cluster/fovea#openai_api_key (client) and …/cluster/agentgateway#vk_fovea (accept-list) — same value
docs/reference/services.md, secrets.md	document fovea→agentgateway routing + new secret

Out of scope

Go-live flip (hl-uh07.9); mesh egress (hl-z5fc); retrieval/embedding routing; upstream fovea N-tier roles; the sonnet ZDR backend.