Octopus: recency-aware, doc-skeptical repo analysis

• Design bead: hl-9d8 (epic context: hl-0we)
• Date: 2026-06-07
• Status: Draft
• Repos: seanb4t/octopus (fork code), fzymgc-house/selfhosted-cluster (deploy + docs)
• Related: ADR hl-78b (fork-and-contribute), docs/operations/octopus.md (fork state, embedding limits)

Problem

Octopus’s repository analyzer (apps/web/lib/analyzer.ts) builds its entire picture of a repo from getRepoChunks(repoId, 80) — an unordered Qdrant scroll that returns an effectively random 80 of N chunks (weft: 781). Documentation and design docs are indexed alongside code (.md/.mdx/.txt are in CODE_EXTENSIONS) and enter the analysis prompt with the same authority as live code. Three structural gaps:

1. Selection is blind. No ordering, no doc-vs-code distinction, no path diversity. A handful of stale design docs can dominate the sample.
2. No recency signal exists. Chunk payloads carry only indexedAt — the indexing-run timestamp, identical for every chunk. Nothing records when a file last changed, so no prompt wording can distinguish a week-old ADR from a two-year-old design doc.
3. The prompt treats all chunks as equal truth. Doc claims are repeated as fact; the resulting analysis (stored on repository.analysis, surfaced in the repo UI, chat context, and Slack responder) describes architecture that may no longer exist.

The same blind scroll feeds summarizer.ts (40 chunks), and the separate review-time retrieval path (searchSimilarChunks + Cohere rerank in reviewer.ts:1241–1262) is equally recency-blind — semantic similarity happily surfaces an outdated design doc that shares vocabulary with the diff.

Decision summary

Three stacked fork branches (each one upstream-PR-sized, per ADR hl-78b conventions), stacked on the current tip feat/optional-review-branding:

#	Branch	Env-gated?	What it adds
1	feat/file-recency-metadata	Yes — INDEX_FILE_RECENCY (default off → byte-identical behavior)	Per-file lastModifiedAt chunk payload field via a bare blob-less git history scan at index time
2	feat/doc-aware-analysis	No — straight quality fix	Structured chunk selection (doc cap, path diversity, recency preference) + “Documentation Accuracy” analysis section + code-is-source-of-truth framing
3	feat/recency-aware-review-context	No — straight quality fix	Review-time context: age annotations, stale-doc demotion post-rerank, prefer-code-over-docs framing

Branches 2 and 3 consume the metadata from branch 1 but degrade gracefully when it is absent (pre-existing chunks, scan disabled, scan failure) — missing lastModifiedAt means “age unknown”: no annotation, no demotion.

Gating rationale: branch 1 costs network (a clone per index run), so it follows the fork’s env-gated default-unchanged convention. Branches 2–3 are behavior-changing by nature (better selection, better prompt); env-gating a prompt fix would be awkward in an upstream PR — they are presented upstream as unconditional improvements.

1. feat/file-recency-metadata

Environment

• INDEX_FILE_RECENCY — unset/falsy (default): no scan, no new payload field, behavior byte-identical to upstream. Truthy: scan runs during every full index.
• INDEX_FILE_RECENCY_WINDOW — optional history cap passed to --shallow-since (e.g. 2 years ago; default unbounded). Files whose last commit predates the window do not appear in the shallow log at all, so they are absent from the recency map; the chunk-payload step writes lastModifiedAt: null for any indexed file missing from the map, and consumers label those “older than the scan window”. (Map-absent ≡ null ≡ “age unknown” — one rule for window-cutoff, scan-disabled, and scan-failure alike.)

Both documented in the fork’s .env.example.

Mechanism

During indexing (all three provider flows — GitHub API flow, Bitbucket, GitLab):

git clone --bare --filter=blob:none [--shallow-since=<window>] --single-branch \
  --branch <defaultBranch> <authedUrl> <tmpdir>
git -C <tmpdir> log --format='%H %cI' --name-status

• Bare + --filter=blob:none transfers commits and trees only — no file contents, no checkout, no lazy blob fetches. Cost scales with commit count, not repo size.
• GitHub: this clone is net-new infrastructure. The GitHub full-index flow is REST-only today (tree + blob fetches — it never clones); the recency scan introduces the first git invocation on that path. Auth follows the codebase’s existing header-based convention (indexer.ts:191–196 deliberately never embeds tokens in clone URLs, since git surfaces the command in error logs): pass http.extraHeader=Authorization: Basic base64("x-access-token:" + installationToken) via GIT_CONFIG_* env, exactly as the BB/GL content clone does. BB/GL flows reuse their existing clone URL + auth header. git is already in the runner image (apps/web/Dockerfile:50) and the indexer already shells out via execFileAsync.
• The scan helper lives in indexer.ts (alongside the existing clone code) with signature collectFileRecency(authedUrl, branch, onLog, signal): Promise<Map<string, string>>, threading the indexer’s existing OnLog callback for warnings.
• One pass over git log --name-status output builds Map<path, lastCommitISO>: first time a path is seen (log is newest-first) wins. Rename entries (R<score>\told\tnew) record the new path; the old path continues accumulating from earlier commits but is irrelevant (it no longer exists in the tree).
• The temp clone is removed in a finally block, mirroring the existing clone cleanup.

Payload

New optional chunk payload field lastModifiedAt: string | null (ISO 8601), set from the map for every chunk of that file. Qdrant payloads are schemaless — purely additive, no collection migration, no dimension change, the embedding-limits envelope (docs/operations/octopus.md) is untouched.

Incremental indexing (incrementalIndex, indexer.ts:648 — webhook-driven re-index of changed files; REST-only, no clone, and the recency scan does not run there) gains an optional headCommitDate?: string parameter plumbed from the push webhook payload’s head-commit timestamp; chunks of changed files get lastModifiedAt: headCommitDate. If the caller can’t supply it, the field is null (“age unknown”) — never now(), which would fabricate freshness.

Failure mode

Any scan error (clone failure, git missing, parse error) → one warning via the indexer’s OnLog callback (onLog(..., "warning")) plus console.warn, and the index proceeds without the field. The scan can never fail or block an index run.

2. feat/doc-aware-analysis

Selection: selectAnalysisChunks(repoId, limit)

New function in apps/web/lib/qdrant.ts, replacing the blind getRepoChunks scroll for both consumers — analyzer (80 chunks) and summarizer (40 chunks). getRepoChunks is removed: analyzer and summarizer are its only callers (verified by repo-wide search), so it is orphaned once both migrate. selectAnalysisChunks needs no logging callback — selection has no user-meaningful failure mode; Qdrant errors propagate to the caller exactly as getRepoChunks’s did.

1. Scroll all payloads for the repo, paginated via next_page_offset with a per-page limit of 256, with_vector: false (payload-only; cheap even at 10k+ chunks).
2. Classify each chunk by extension: doc (.md, .mdx, .txt) vs code (everything else, including config).
3. Budget: docs capped at 20% of limit (analyzer: ≤16 of 80). Code fills the rest; spare doc budget spills over to code.
4. Diversity: round-robin across top-level directories; max 3 chunks per file.
5. Seeds: always include root README* chunks (current top-level intent) and the root manifest (package.json, go.mod, Cargo.toml, pyproject.toml, …) when present.
6. Recency: within each bucket, prefer newest lastModifiedAt; chunks without the field sort after dated ones. Compute the median code-file age as the repo’s staleness reference point and return it alongside the chunks.

Returned shape: { chunks: AnalysisChunk[], medianCode: string | null } where AnalysisChunk = { text, filePath, kind: "doc" | "code", lastModifiedAt }.

Context formatting (analyzer.ts)

Code chunks first, then docs in a dedicated, explicitly-framed block:

<code chunks, each headed by: // src/auth/session.ts (last modified 2026-05-28)>

--- Documentation excerpts (prose may be outdated; the code is the source of truth) ---

// docs/design.md (last modified 2024-08-12 — 14 months older than typical code in this repo)
...

Age annotations appear only when lastModifiedAt exists; the relative phrase appears only when the median code age exists and the doc is stale. Staleness threshold (shared with §3): a doc chunk is stale when its lastModifiedAt is more than 6 months older than the median code-file date. The absolute gap is preferred over a relative one (e.g. 2× median age) because relative thresholds misfire on young repos — with a 2-week median, a month-old doc would be flagged. Confirm the constant at plan time; define it once and import it in both §2 and §3 code.

Prompt changes (analyzer.ts)

• New 7th section in the required output structure:

  ## Documentation Accuracy Compare documentation claims against the code. List claims the code contradicts or that appear outdated (cite the doc file and the contradicting code). Note which docs appear current and trustworthy. If no documentation was provided, say so in one line.

• One framing line added to the instruction preamble: “Documentation excerpts describe intent at the time they were written. Verify claims against the code; never repeat a documentation claim as fact without code evidence.”

• maxTokens 4096 → 5120 (headroom for the extra section; working estimate — confirm against real output length at plan/E2E time).

Summarizer (summarizer.ts)

Uses selectAnalysisChunks(repoId, 40) and gains the framing line only — no 7th section (summaries stay short).

3. feat/recency-aware-review-context

• searchSimilarChunks (qdrant.ts) additionally returns lastModifiedAt from the payload (additive change to its return type).
• Scope — both bulk-context callers: the webhook PR flow (reviewer.ts:1241–1262) and the local-review/API flow (review-core.ts:215), which assembles the same 50-chunk + rerank context. The demotion/annotation logic is implemented once as a shared helper (e.g. applyRecencyToContextChunks next to the rerank utilities) and called from both. The limit-3 targeted symbol lookups in review-validation.ts (gatherCrossFileContext, gatherVerificationContext) are explicitly out of scope: they are precision queries where demotion could drop the only result, and a stale doc surfacing there is answering a direct question, not framing the review.
• In the (shared) context assembly (post-rerank, pre-prompt):
  • Demote stale docs: doc-classified chunks that are stale per the shared §2 threshold (> 6 months older than the median code-file date) are dropped unless their rerank score ≥ 0.6 (they earned their place). Median computed from the retrieved set’s code chunks — no extra Qdrant round-trip.
  • Annotate: surviving doc chunks get (documentation — verify against code) in their // path:Lx-Ly header; all chunks with known dates get last modified <date>.
• One line added to the review prompt’s codebase-context framing: “Where documentation and code conflict, the code is authoritative.”
• Missing lastModifiedAt anywhere → no demotion, no annotation (pre-recency chunks behave exactly as today).

Error handling & compatibility

Condition	Behavior
INDEX_FILE_RECENCY unset	Branch 1 inert; branches 2–3 still improve selection/prompting, just without age data
Recency scan fails	Warning logged; index completes without lastModifiedAt
Chunks predate this work	Treated as “age unknown” — no annotation/demotion; full benefits arrive on next re-index
Repo has no docs	Doc budget spills to code; Documentation Accuracy section reports “no documentation provided”
Tiny repos (< limit chunks)	Selection returns everything; caps/diversity are no-ops

Nothing here touches embeddings, vector dimensions, batch sizes, or the gateway — the hard-won embedding pipeline envelope is out of scope and unmodified.

Testing

• Unit (fork; bun test, tests in apps/web/lib/__tests__/*.test.ts — upstream’s existing convention, e.g. incremental-index.test.ts):
  • git log --name-status parser: adds/modifies, renames (R100), merge commits, window-bounded logs.
  • selectAnalysisChunks: doc cap, per-file cap, dir round-robin, README/manifest seeding, recency ordering, missing-lastModifiedAt ordering, tiny-repo passthrough.
  • Reviewer demotion: stale doc dropped, high-score stale doc kept, undated chunks untouched.
• E2E (cluster, after deploy): Cancel + re-index weft (expect the known envelope: ~16 batches, ~160 s, zero 503s), then re-run analysis and verify (a) age annotations in pod-logged context (or via a debug log), (b) the ## Documentation Accuracy section appears in the stored analysis, (c) a PR review on weft still completes.

Rollout

1. Fork: three branches stacked on feat/optional-review-branding; add each to the branches: list in .github/workflows/build-image.yaml; push → multi-arch build; watch with gh run watch <id> --repo seanb4t/octopus --exit-status.
2. Cluster PR: repoint both digests (octopus + octopus-migrate) in argocd/app-configs/octopus/deployment.yaml from the new tip build; add INDEX_FILE_RECENCY: "true" to the Deployment env.
3. Verify pod digest + env via kubectl, then re-index weft (recency backfill) and re-run analysis.
4. Docs: fork .env.example (new envs); cluster docs/operations/octopus.md — extend the fork-state table with the three branches/envs and add an ops note: re-index a repo to backfill recency metadata; analysis improvements work without it but age-awareness needs the re-index.
5. Upstream (extends hl-0we.4 batch): three more upstream PRs once validated.

Out of scope

• Knowledge chunks (getKnowledgeChunksByOrg) — org-curated guidelines, deliberately authoritative; staleness there is a human curation problem.
• Re-ranking model changes, embedding model/dimension changes.
• hl-0we.17 (pubby/ES replacement) — unrelated private-fork track.

Open questions (plan-time)

• Confirm the 6-month staleness constant against weft’s real file-date distribution during E2E (the mechanism and sharing between §2/§3 are settled above; only the constant is tunable).