Skip to content
fzymgc-house Cluster Docs

Serve llms.txt artifacts from the build; enforce coverage via custom CI gate

Date: 2026-06-07 Status: Accepted Decision: hl-25z Deciders: Sean Brandt

Context

Section titled “Context”

AI agents currently consume hand-maintained CLAUDE.md path pointers that rot and are not machine-consumable. The Starlight migration adopts the starlight-llms-txt plugin, which emits llms.txt (index), llms-full.txt (concatenated), and llms-small.txt (minified) at build time. Two questions: commit generated artifacts to the repo or serve them from the build, and how to guarantee every page is included (the plugin has no built-in coverage gate).

Decision

Section titled “Decision”

Do not commit llms.txt artifacts; serve them from the Cloudflare Pages build (authenticated agents fetch via a Cloudflare Access service token). Add a custom post-build coverage script in Phase 3 CI that enumerates the docs collection’s page slugs and asserts each appears in llms-full.txt, failing CI on any gap.

Rationale

Section titled “Rationale”
  • The plugin has no built-in coverage gate (verified against its v0.10.0 config docs) — a custom check is needed regardless.
  • Committed generated artifacts go stale between builds and add churn to every docs PR; standard generated-file hygiene argues against committing.
  • Cloudflare Access service-token auth already exists for agent infra (agentgateway), so build-served artifacts are reachable by the agents that need them.
  • Layering the coverage assertion on the build output keeps it independent of plugin internals/version.

Alternatives Considered

Section titled “Alternatives Considered”
  • Commit generated llms.txt to the repo — available offline and diffs visible in PRs, but stale between builds and noisy. Rejected.
  • Plugin-only, no coverage assertion — zero custom code, but coverage gaps are silent; an “all clear” artifact missing pages is worse than none. Rejected.
  • Public unauthenticated /llms.txt path* — maximizes crawler reach but leaks private homelab doc content. Rejected.

Consequences

Section titled “Consequences”
  • Positive: agents always get a fresh, complete knowledge artifact; CI enforces no page is silently excluded; the repo stays clean of generated files; llms-full.txt can supersede the fragile CLAUDE.md path-pointer routing layer.
  • Negative: a custom coverage script must be written and maintained (Phase 3); failures surface only post-build.
  • Neutral: the customSets per-area bundle taxonomy remains an open item resolved during Phase 3 planning.