Skip to content

Design: the Octopus review lane

See openspec/changes/archive/2026-10-05-octopus-review-lane/proposal.md for the problem and for the decisions that Sean made on 2026-10-04.

Octopus is a fork (seanb4t/octopus, branch upgrade/v1.2.14-analysis) that sends every LLM call to the LiteLLM Service with the octopus virtual key. These facts shape the design:

  • The fork sends only a model name and an output token budget on a chat call. It sends no reasoning effort and no provider rule. A lane in LiteLLM is the only place that can set them.
  • LiteLLM checks the models list of a key before it applies an alias of the key. A name that Octopus sends must be on the list. If the name is itself a lane, it needs no alias.
  • The review call does not stream. Node’s fetch drops a call that sends no headers in 300 seconds, and the OpenAI client of the fork then sends the same call two more times.
  • A reply that ends because it used the whole token budget makes the check run fail. Octopus treats a finish_reason other than stop as an incomplete review.
  • The review model of an organization comes from the Octopus database. The order is the pin of the repository (repositories.reviewModelId), then the pin of the organization (organizations.defaultModelId), then the row of available_models that has isPlatformDefault true.
  • Octopus reads the provider of a model from available_models and holds the result for five minutes. A model name that has no row goes to the Anthropic client and fails.
  • The model that Octopus calls the review model also serves smaller calls: the analyzer (a budget of 8192 tokens), the request for missing findings (4096) and the Slack reply (2048).
  • LiteLLM names the span of a call after the lane. A call on a named lane has the span name chat <lane>, and the attribute litellm.provider.model holds the model at OpenRouter. A call on the wildcard lane has the model in its span name.

This change adds no address and no prefix, so docs/reference/network.md does not change.

Goals:

  • One place holds the model, the reasoning effort and the provider rule of Octopus.
  • A review call ends before the 300-second limit in all but rare cases.
  • The content of a review call and of a utility call reaches ZDR providers only.
  • The switch to the new lane needs no deployment, and a person can reverse it in one statement.

Non-Goals:

  • A change to the fork. The transport of the review call (streaming, explicit timeouts, no second attempt after a length reply) stays a separate piece of work in seanb4t/octopus.
  • A change to the karakeep key or the mealie key.
  • A change to the embedding lane or to the embedding names of the key.
  • A timeout on the new lane. OpenRouter bills a canceled call that does not stream in full, so a timeout saves no money.

D-1. Octopus sends role names, and each role name is a lane

Section titled “D-1. Octopus sends role names, and each role name is a lane”

Octopus sends octopus-review for the review model and octopus-utility for the utility model. Both are model_name entries in argocd/app-configs/litellm-chart/values.yaml. The octopus key lists both names and holds no alias for them.

The new lane:

- model_name: "octopus-review"
litellm_params:
model: "openrouter/deepseek/deepseek-v4.1-flash"
api_key: os.environ/OPENROUTER_LANE_API_KEY
reasoning_effort: "low"
provider:
zdr: true
require_parameters: true
order:
- Together
- NextBit
allow_fallbacks: true
num_retries: 0

D-8 gives the reason for the provider order.

The octopus key at the end of the change:

models = [
"ollama/bge-m3", "bge-m3", "octopus-embed", "text-embedding-3-large",
"octopus-review", "octopus-utility",
local.decisions_model,
]
aliases = {
"ollama/bge-m3" = "octopus-embed"
"bge-m3" = "octopus-embed"
"text-embedding-3-large" = "octopus-embed"
}

The key is the resource litellm_key.virtual_key["octopus"]. This change edits two arguments that Terraform manages today, models and aliases. It adds no argument. The metadata object of the key stays as it is.

A model change after this change is one edit of model: in the lane.

Alternative: keep the name or-deepseek-v4-pro in Octopus and point its alias at the lane, as issue #2740 planned. Rejected. The Octopus UI then shows a model that the lane does not serve, and each later model change makes the name more wrong.

Alternative: the fork sends the reasoning effort, and the Octopus UI owns the model. Rejected by Sean on 2026-10-04. See openspec/changes/archive/2026-10-05-octopus-review-lane/proposal.md.

D-2. The first review model is deepseek-v4.1-flash at low effort

Section titled “D-2. The first review model is deepseek-v4.1-flash at low effort”

OpenRouter lists three effort levels for this model: max, high and low. The default is high.

The evidence is public and not ours. MacroscopeBench (a code review benchmark, read on 2026-10-04) scores the model 70.7 at low effort, 70.5 at high and 72.0 at max. The same source gives a precision of about 65%, which is about 1.4 true findings for each false one. The list price at the first-party provider is $0.15 for a million input tokens and $0.60 for a million output tokens. The current model costs $0.66 and $1.98. OpenRouter shows a median of 92 tokens per second across the providers of the new model, against 50 for the current model.

The model has no measurement on our pull requests. The check after the switch is the query in the Migration Plan, which counts findings per review. The baseline for the current model at high effort is 678 reviews from 2026-09-25 to 2026-10-04 with 0.59 findings per review: 1 critical, 13 high, 35 medium and 21 low.

The first day on the lane and the test of D-8 gave the first measurement. Low effort does not bound the reasoning on a large diff. A review of a diff with 85,000 input tokens used 27,600 to 31,100 reasoning tokens in five runs, which is about 92% of the output. One run at high effort used the same number of reasoning tokens as one run at low effort. The model and the effort stay, and D-8 changes the provider.

The way back is one line: set model: of the lane to openrouter/deepseek/deepseek-v4-pro-0813. The effort stays low, which that model also accepts.

Alternative: the current model at low effort first, then a trial of the new model, as issue #2740 planned. Rejected. With D-1, each step of that plan is the same one-line edit, so the order has no cost, and the new model has the better evidence at low effort.

Alternative: google/gemini-3.8-flash. It found more defects than the current model in a second benchmark (DiffDojo), but it costs more than the current model and has no data at low effort.

D-3. Both chat lanes carry the ZDR provider rule

Section titled “D-3. Both chat lanes carry the ZDR provider rule”

The new lane sets provider.zdr: true and provider.require_parameters: true, as the lanes engram-summarize and octopus-embed do. The lane octopus-utility gains the same block. Sean decided on 2026-10-04 that the utility lane gets the rule. On that day, OpenRouter listed 22 ZDR providers for the model of both lanes, with 24 endpoints.

A test on 2026-08-16 showed that the rule of the lane openrouter-zdr/* won over a provider block in the request. A test on 2026-10-04, on LiteLLM v1.104.0-rc.1, showed the opposite. The provider block of a request replaces the block of the lane, so a caller can reach a provider without ZDR. Octopus sends no provider block, so its own calls stay on ZDR providers. The OpenSpec change openrouter-zdr-guardrail enforces ZDR at OpenRouter for every ZDR lane.

require_parameters: true limits a lane to the providers that support each parameter of a request. A chat call of Octopus sends a model, a token budget and the messages, and the lane adds the reasoning effort. Each of the 24 endpoints supports the token budget and the reasoning parameter.

The rule of the review lane has no effect before the switch. The rule of the utility lane takes effect at the sync of pull request 1, because Octopus uses that lane today. So an operator proves the request shape of a utility call before the merge. The proof goes through the lane openrouter-zdr/*, which holds the same provider block. Step 1 of the Migration Plan gives the request.

D-4. The catalog holds the two role names, and one of them is the platform default

Section titled “D-4. The catalog holds the two role names, and one of them is the platform default”

The table available_models gains two rows with the provider openai and the category llm: octopus-review and octopus-utility. The row octopus-review becomes the only platform default of the category. Each model pin of an organization or a repository changes to octopus-review. After the last pull request, every other chat row becomes inactive, which removes it from the Octopus UI. The rows stay in the table, so a way back needs no insert.

Both rows carry the list price of deepseek-v4.1-flash. Octopus uses the price only for its own cost display. The spend of the octopus key in LiteLLM is the cost of record, so the price in the catalog can lag a model change.

An organization without a pin gets the platform default. For such an organization, the fork can send a diff that it classes as mechanical to the utility model. An organization with a pin never does that. On 2026-09-25, two organizations held a pin and one did not.

Alternative: clear every pin and use the platform default alone. Rejected for now. It changes which lane serves a mechanical diff for two organizations, and nothing in this change needs that.

D-5. The switch is a change in the Octopus database

Section titled “D-5. The switch is a change in the Octopus database”

The new lane (pull request 1) and its name on the key (pull request 2) have no effect while no caller sends the name. The order of those two pull requests does not matter. The ZDR rule of the utility lane is the one part of pull request 1 that takes effect at its sync (D-3).

The switch is then two steps by an operator. The first step inserts the two rows. The second step comes at least five minutes later, because Octopus holds its provider list for five minutes, and it moves the platform default and the pins. A person reverses the switch with one statement, and the old names stay on the key until pull request 4.

Alternative: move the alias in Terraform, as issue #2740 planned. Rejected. If the alias goes live before the lane, every review fails, and the way back is a second apply.

D-6. The output token budget comes from one day of calls on the lane

Section titled “D-6. The output token budget comes from one day of calls on the lane”

OCTOPUS_REVIEW_MAX_TOKENS stays 32768 through the switch. A lower budget does not shorten a call. In the test of D-8, a call with a budget of 12288 used all 12288 tokens for reasoning and returned no review text.

With the provider order of D-8, the budget rises to 49152. At the lowest token rate that the test of D-8 measured at Together, 250 tokens per second, a call produces 49152 tokens in 197 seconds.

The rise needs no day of calls first. A review that needs fewer tokens than the old budget ends when it is done, so the higher budget does not change it. A review that needs more ended with length under the old budget. With the higher budget, it ends in time at a fast provider. At a slow provider it fails under both budgets. Sean decided this on 2026-10-05.

The query below checks the budget after the fact. Run it after a day that held a large diff. The budget of the query is the number of tokens that a slow call produces in 270 seconds, with an upper limit of 49152. A slow call here is the 10th percentile of the token rate. The budget must stay at or above the 99th percentile of the output of that day. The query:

SELECT count() AS calls,
round(quantile(0.99)(o)) AS p99_out,
round(quantile(0.1)(o / d), 1) AS p10_rate,
least(49152, floor(quantile(0.1)(o / d) * 270 / 1024) * 1024) AS budget,
max(i) AS max_in
FROM (
SELECT toFloat64OrZero(SpanAttributes['gen_ai.usage.output_tokens']) AS o,
toFloat64OrZero(SpanAttributes['gen_ai.usage.input_tokens']) AS i,
Duration / 1e9 AS d
FROM default.otel_traces
WHERE Timestamp > now() - INTERVAL 1 DAY
AND ServiceName = 'litellm'
AND SpanName = 'chat octopus-review'
AND SpanAttributes['gen_ai.request.max_tokens'] = '49152'
AND toFloat64OrZero(SpanAttributes['gen_ai.usage.output_tokens']) > 1000
)

The reading counts only from a day that held a large diff. If max_in is under 60000, wait for a day that holds one. The slow calls come from large diffs, so a day without one always passes.

If budget is under p99_out, the lane is too slow for its own replies. In that case, do not lower the budget. Tell Sean, who decides between other providers in the order of D-8 and another model.

The first reading was on 2026-10-05, after 12.6 hours on the lane and before the provider order. Its filter held the budget of that day, 32768. It gave 35 calls, a p99_out of 32768, a p10_rate of 61.7 and a budget of 16384 under the old upper limit of 32768. The budget was under p99_out. Sean decided on 2026-10-05 to keep the model and to name fast providers first (D-8).

Alternative: the 12288 of issue #2740. Rejected. That number came from calls at high effort on another model.

Alternative: an upper limit of 32768. Rejected. In the test of D-8, one of two runs on the largest diff used all 32768 tokens at a fast provider. At 250 tokens per second, a call produces 49152 tokens in 197 seconds.

D-7. Queries name the lane, and the model comes from an attribute

Section titled “D-7. Queries name the lane, and the model comes from an attribute”

Each query in this change and in the runbook filters on SpanName = 'chat octopus-review'. A query that needs the model reads SpanAttributes['litellm.provider.model']. A filter on a span name that holds the model returns no rows for a named lane, and a threshold on no rows passes.

The runbook docs/operations/octopus.md gains a section on the 300-second limit. It gives the cause, the three span fields that show it (gen_ai.usage.output_tokens, gen_ai.response.finish_reasons and the span duration), the query of D-6 and the steps of a model change.

D-8. The review lane names fast providers first

Section titled “D-8. The review lane names fast providers first”

The lane octopus-review sets provider.order to Together and then NextBit, with allow_fallbacks: true. OpenRouter tries those providers first. The other ZDR providers stay as the fallback, so a review still runs when both named providers fail. The ZDR rule of D-3 stays on the lane.

Without the order, OpenRouter prefers the provider with the lowest price. On 2026-10-05, the 10th percentile of the token rate on the lane was 61.7 tokens per second. A call that produces the budget of 32768 tokens in 270 seconds needs 122.

A test on 2026-10-05 sent 20 requests through the lane openrouter-zdr/* with the model of the lane. Each request held the system prompt of the fork and the diff of a merged pull request. The budget was 32768 unless the table says otherwise. The results for the diff of pull request #2765, with 85,000 input tokens:

Variant Provider Seconds Output tokens (reasoning) Finish
low, no order DeepInfra 368 30,774 (28,670) stop
low, no order Together 109 30,060 (27,644) stop
high DeepInfra 732 32,444 (28,670) stop
reasoning.max_tokens: 4000 DeepInfra 287 23,541 (21,206) stop
low, budget 12288 Together 39 12,288 (12,288) length, no text
none Together 10 2,284 (0) stop
low, sort: throughput, run 1 DeepInfra 221 23,457 (21,098) stop
low, sort: throughput, run 2 DeepInfra 250 25,217 (22,722) stop
low, the order, run 1 Together 119 31,861 (29,435) stop
low, the order, run 2 Together 124 32,768 (31,118) length

On the large calls of the test, Together produced 250 to 315 output tokens per second in seven calls. DeepInfra produced 44 to 106 in six calls. The order reached Together in four of four calls. A large review costs about $0.05 at Together and about $0.013 at DeepInfra.

The speed numbers come from one day. The proof request of the Migration Plan and the query of D-6 show the speed of the lane in production.

Alternative: reasoning off (reasoning_effort: "none"). Rejected. Each of four reviews ended in under 25 seconds. But two of two findings that were compared with the repository were false, and one of them had the severity high. Two of two compared findings at low effort were true. The sample is small.

Alternative: a reasoning token cap (reasoning.max_tokens). Rejected. It has no effect on this model.

Alternative: a lower output budget. Rejected. See D-6.

Alternative: provider.sort: "throughput". Rejected. It reached DeepInfra in two of three calls.

Alternative: another model. Not tested. It is the next step if the lane does not meet the time limit of the spec with the order.

  • [The new model at low effort finds fewer defects on our pull requests] → The findings query of the Migration Plan compares one week with the baseline of D-2. Issue #2790 holds that comparison, outside this change. The way back is the one-line edit of D-2.
  • [The new model reports more false findings] → The public precision is about 65%. No query measures this. Sean reads the reviews of the first week and decides.
  • [A slow provider still pushes a call past 300 seconds] → The limit stays in the fork. D-8 names fast providers first, D-6 bounds the slow case, and the acceptance allows 1% of calls. The transport work in the fork removes the limit.
  • [Together and NextBit slow down, or one of them loses its ZDR endpoint] → The fallback keeps reviews running at the speed of the other ZDR providers. The query of D-6 shows the token rate, and a reply names its provider. The fix is an edit of the order.
  • [A large review costs more with the provider order] → It costs about $0.05, against about $0.013 at the cheapest provider. The old review model cost $0.044 for each review. The spend of the octopus key shows the result.
  • [require_parameters leaves no provider for a request shape that Octopus sends] → The proof request in step 3 of the Migration Plan runs before the switch. After the switch, a failed review shows in the check run within minutes, and the way back is one statement.
  • [The lane serves the smaller calls of the review model too] → The spend of the lane holds the analyzer, the request for missing findings and the Slack reply. A query on reviews filters on the token budget.
  • [An organization without a pin sends a mechanical diff to octopus-utility] → That lane has reasoning off. This is the behavior of today for that organization, and D-3 gives the lane the ZDR rule.
  • [The ZDR rule leaves no provider for a utility call] → The proof request in step 1 of the Migration Plan runs before the merge. After the sync, a failed utility call shows as a span chat octopus-utility with an error status. The way back is a pull request that removes the provider block from the lane octopus-utility.
  • [The Octopus UI and its cost display no longer show the real model] → The lane in values.yaml is the record of the model, and the LiteLLM spend is the record of the cost. The runbook says so.
  • [Two applies restart the budget window of the key] → This happens only if Terraform created the octopus key. The plan of each pull request shows no budget argument, so read the spend of the key before the merge if the window matters.

Each step names its way back.

  1. Before the merge of pull request 1, an operator proves the request shape of a utility call on ZDR providers from a workstation:

    Terminal window
    export VAULT_ADDR=https://vault.fzymgc.house
    KEY="$(vault kv get -field=api_key \
    secret/fzymgc-house/cluster/litellm-keys/openrouter-passthrough)"
    curl -s https://llm.fzymgc.house/v1/chat/completions \
    -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
    -d '{"model":"openrouter-zdr/deepseek/deepseek-v4.1-flash","reasoning_effort":"none",
    "max_completion_tokens":256,
    "messages":[{"role":"user","content":"Reply with the word OK."}]}' \
    | jq '{provider, finish: .choices[0].finish_reason}'

    The reply must name a provider and end with stop.

    Pull request 1 then adds the lane octopus-review and the provider block of the lane octopus-utility. ArgoCD syncs litellm, and the proxy restarts with the new configuration. Way back before the switch: revert the pull request. Way back after the switch, for the rule of the utility lane only: a pull request that removes its provider block.

  2. Pull request 2 adds octopus-review to the key. The workspace main-cluster-litellm applies on merge. The plan must change only litellm_key.virtual_key["octopus"]. Way back: revert the pull request.

  3. An operator proves the lane from a workstation:

    Terminal window
    export VAULT_ADDR=https://vault.fzymgc.house
    KEY="$(vault kv get -field=api_key secret/fzymgc-house/cluster/litellm-keys/octopus)"
    curl -s https://llm.fzymgc.house/v1/chat/completions \
    -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
    -d '{"model":"octopus-review","max_completion_tokens":2048,
    "messages":[{"role":"user","content":"Reply with the word OK."}]}' \
    | jq '{provider, finish: .choices[0].finish_reason, usage}'

    The reply must name a provider and end with stop. The provider must be in the ZDR list:

    Terminal window
    curl -s https://openrouter.ai/api/v1/endpoints/zdr \
    | jq -r '.data[] | select(.model_id == "deepseek/deepseek-v4.1-flash") | .provider_name' \
    | sort -u

    The operator then sends the request again with "provider":{"only":["StreamLake"],"zdr":false} added to the body. StreamLake serves the model without ZDR. The provider field of the reply must not be StreamLake.

  4. The operator inserts the two rows. psql runs on the Postgres pod of the main cluster:

    Terminal window
    kubectl --context fzymgc-house exec -i -n postgres main-16 -c postgres -- psql -d octopus <<'SQL'
    INSERT INTO available_models
    (id, "modelId", "displayName", provider, category, "inputPrice", "outputPrice",
    "isActive", "isPlatformDefault", "sortOrder", "updatedAt")
    VALUES
    ('octopus-review', 'octopus-review', 'Review (LiteLLM lane)', 'openai', 'llm',
    0.15, 0.60, true, false, 0, now()),
    ('octopus-utility', 'octopus-utility', 'Utility (LiteLLM lane)', 'openai', 'llm',
    0.15, 0.60, true, false, 1, now());
    SQL

    Way back: DELETE FROM available_models WHERE "modelId" IN ('octopus-review', 'octopus-utility');

  5. At least five minutes later, the operator saves the pins of today and then moves them:

    Terminal window
    kubectl --context fzymgc-house exec -i -n postgres main-16 -c postgres -- psql -d octopus <<'SQL'
    SELECT 'organization' AS kind, id, "defaultModelId" AS pin FROM organizations
    WHERE "defaultModelId" IS NOT NULL
    UNION ALL
    SELECT 'repository', id, "reviewModelId" FROM repositories WHERE "reviewModelId" IS NOT NULL
    UNION ALL
    SELECT 'platform default', id, "modelId" FROM available_models
    WHERE category = 'llm' AND "isPlatformDefault";
    BEGIN;
    UPDATE available_models SET "isPlatformDefault" = ("modelId" = 'octopus-review')
    WHERE category = 'llm';
    UPDATE organizations SET "defaultModelId" = 'octopus-review'
    WHERE "defaultModelId" IS NOT NULL;
    UPDATE repositories SET "reviewModelId" = 'octopus-review'
    WHERE "reviewModelId" IS NOT NULL;
    COMMIT;
    SQL

    The operator keeps the output of the SELECT. It is the record for the way back. Way back, if each pin was or-deepseek-v4-pro:

    UPDATE organizations SET "defaultModelId" = 'or-deepseek-v4-pro'
    WHERE "defaultModelId" = 'octopus-review';
    UPDATE repositories SET "reviewModelId" = 'or-deepseek-v4-pro'
    WHERE "reviewModelId" = 'octopus-review';

    The platform default stays on octopus-review in the way back. An organization with a pin does not read it.

  6. The operator starts one review: change an open pull request to draft and back to ready for review. The check run Octopus Review must end as success.

  7. Three pull requests follow the switch. The way back of each one is a revert.

    Pull request 3 sets OCTOPUS_UTILITY_MODEL to octopus-utility and updates the runbook. The old utility name stays on the key and in the catalog until step 8.

    Pull request 4 adds the provider order of D-8 to the lane octopus-review. After the ArgoCD sync of litellm, the operator sends the first request of step 3 again. The reply must name Together and end with stop.

    Pull request 5 sets OCTOPUS_REVIEW_MAX_TOKENS to 49152 (D-6). It opens after pull request 4 synced. After a day that held a large diff, the query of D-6 checks the budget.

  8. The operator makes sure that Octopus can send no old name: each model pin and the platform default hold octopus-review or no value, and OCTOPUS_UTILITY_MODEL is octopus-utility. The operator then makes the old rows inactive, on the primary of the main cluster:

    UPDATE available_models SET "isActive" = false
    WHERE category = 'llm' AND "modelId" NOT IN ('octopus-review', 'octopus-utility');

    Pull request 6 then removes the old names from the key. Way back: revert the pull request, and set "isActive" = true on the rows that the way back needs.

The findings query, for the week after step 5. Set the interval to cover the period, and set the token budget in spans to the value of that period:

WITH spans AS (
SELECT 1 AS k, Timestamp + toIntervalNanosecond(Duration) - toIntervalSecond(1) AS t0
FROM default.otel_traces
WHERE Timestamp > now() - INTERVAL 7 DAY AND ServiceName = 'litellm'
AND SpanName = 'chat octopus-review'
AND SpanAttributes['gen_ai.request.max_tokens'] = '32768'
AND Duration < 295e9
), generated AS (
SELECT 1 AS k, Timestamp AS lts,
toUInt32OrZero(extract(Body, 'chars, ([0-9]+) findings')) AS findings
FROM default.otel_logs
WHERE Timestamp > now() - INTERVAL 7 DAY AND ServiceName = 'octopus'
AND startsWith(Body, '[reviewer] Review generated:')
), split AS (
SELECT 1 AS k, Timestamp AS sts, extract(Body, 'severities: (.*)$') AS sev
FROM default.otel_logs
WHERE Timestamp > now() - INTERVAL 7 DAY AND ServiceName = 'octopus'
AND startsWith(Body, '[reviewer] Split:')
), sg AS (
SELECT s.t0, g.lts, g.findings, 1 AS k
FROM spans s ASOF LEFT JOIN generated g ON s.k = g.k AND g.lts >= s.t0
WHERE g.lts > toDateTime64('2000-01-01 00:00:00', 9)
AND g.lts <= s.t0 + toIntervalSecond(4)
), j AS (
SELECT sg.findings,
if(p.sts > toDateTime64('2000-01-01 00:00:00', 9)
AND p.sts <= sg.lts + toIntervalSecond(120), p.sev, '') AS sev
FROM sg ASOF LEFT JOIN split p ON sg.k = p.k AND p.sts >= sg.lts
)
SELECT count() AS reviews,
round(avg(findings), 2) AS avg_findings,
sum(countSubstrings(sev, '🔴')) AS critical,
sum(countSubstrings(sev, '🟠')) AS high,
sum(countSubstrings(sev, '🟡')) AS medium,
sum(countSubstrings(sev, '🔵')) AS low
FROM j

The query matches each review call to the log line Review generated that follows it within 4 seconds, and that line to the next Split line within 120 seconds.