Skip to content
fzymgc-house Cluster Docs

Split model routing: OpenRouter reviews + in-cluster bge-m3 embeddings (1024-dim)

Date: 2026-06-06 Status: Accepted Decision: hl-6nm Deciders: Sean Brandt

Context

Section titled “Context”

Octopus needs both a chat-completion model (code review) and an embedding model (Qdrant code indexing). Upstream hard-codes OpenAI text-embedding-3-large (3072-dim) and OPENAI_API_KEY is required even when reviewing with Claude. The cluster’s agentgateway (ADR hl-hv3) exposes a unified OpenAI-compatible endpoint for both OpenRouter and in-cluster models, but the two model categories have different ZDR, cost, and latency profiles.

Decision

Section titled “Decision”

Register two provider=openai availableModel aliases in Octopus’s admin DB: one mapping to an OpenRouter frontier model (ZDR alias) for reviews, one mapping to the in-cluster agentgateway bge-m3 alias for embeddings. Set EMBEDDING_DIM=1024 and patch qdrant.ts so the Qdrant collection vector size is configurable. Both calls traverse agentgateway via OPENAI_BASE_URL + one virtual key.

Rationale

Section titled “Rationale”
  • ZDR applies to OpenRouter-routed calls; keeping embeddings in-cluster avoids transmitting private-repo code to a third-party embedding API.
  • agentgateway’s unified endpoint means both aliases share one virtual key, auth model, and OTel pipeline — no per-model credential management.
  • 1024 is a hard Qdrant collection-schema constraint: once collections exist at this dimension, all future embeddings must be 1024-dim or the collections must be dropped and rebuilt.
  • bge-m3 is already served in-cluster for engram, so no new model-serving infrastructure is required.

Alternatives Considered

Section titled “Alternatives Considered”
  • OpenAI text-embedding-3-large for both — no fork needed, but embedding traffic egresses to OpenAI (breaks ZDR intent) and adds per-token cost + an external dependency for a workload that can be served in-cluster.
  • A single self-hosted model (Ollama) for reviews and embeddings — zero egress, lowest cost, but in-cluster frontier-class review quality is not available; review quality would degrade materially.

Consequences

Section titled “Consequences”
  • Positive: embedding traffic never leaves the cluster; reviews get the ZDR guarantee; unified observability via agentgateway provider/model labels; reuses the existing bge-m3 deployment.
  • Negative: EMBEDDING_DIM=1024 is a migration-breaking constraint on Qdrant collections — switching embedders later requires dropping and rebuilding all 7 collections; requires the qdrant.ts fork patch until upstream merges it.
  • Neutral: the availableModel entries are set via the Octopus admin UI (DB-backed), not GitOps manifests — model selection is an operational, not deployment, artifact.