Docs Migration to Astro Starlight + Aggressive Grooming & Validation

Date: 2026-06-07 Status: Draft Design bead: hl-bit Supersedes (tooling): 2025-12-29 docs-site design (MkDocs Material)

Overview

Migrate the cluster documentation site from MkDocs Material to Astro Starlight, aggressively groom the ~194 markdown files (most of which are orphaned from navigation), and stand up a multi-dimensional validation harness. Both the current and target stacks emit static HTML deployed to Cloudflare Pages behind Cloudflare Access, so the deployment target is unchanged — only the builder and the surrounding validation/AI tooling change.

The work is sequenced validate → groom → migrate across three independently shippable phases, drawn on a risk/reversibility boundary: Phase 1 is read-only, Phase 2 is destructive but runs on the stable MkDocs site, Phase 3 is the only cutover and is gated by a Cloudflare Pages preview deploy.

Problem Statement

• Content sprawl. 194 markdown files under docs/; the MkDocs nav: surfaces only ~40. docs/plans/ (96 files) and docs/engineering/ (43 files) are almost entirely orphaned — present on disk, absent from navigation and search-weighting intent.
• No validation. No internal-link checking, no orphan detection, no external-link liveness, and no reconciliation of reference docs (services.md, secrets.md, network.md) against the running cluster.
• AI routing is fragile. The 2025-12-29 design made CLAUDE.md files a “routing layer” of hand-maintained docs/... path pointers. These rot and do not produce a machine-consumable knowledge artifact.

Goals

1. Replace the MkDocs builder with Astro Starlight, preserving the Cloudflare Pages + Cloudflare Access deployment and per-branch preview deploys.
2. Aggressively prune dead/superseded historical plans and specs; keep only living docs.
3. Stand up validation across four dimensions: internal links/anchors, orphan/coverage, external-link liveness, and live-cluster cross-check.
4. Generate llms.txt / llms-full.txt / llms-small.txt so AI agents consume a canonical artifact instead of CLAUDE.md path pointers.

Non-Goals

• Changing the deployment target (stays Cloudflare Pages) or the Access policy.
• A full live-cluster reconciler. The cross-check is bounded to four reference docs and is advisory, never a CI gate.
• Restructuring CLAUDE.md beyond updating doc pointers / adding the llms.txt reference.
• Visual redesign beyond Starlight defaults (dark theme, brand accent later).

Decisions (from brainstorming)

Decision	Choice
Orphan fate	Aggressively prune — delete superseded/dead, keep living
Cutover	Big-bang single PR (Phase 3), preview deploy as safety net
Validation scope	All four dimensions
Sequencing	Validate → groom → migrate
Content layout	Isolated website/ Astro subproject reading repo-root docs/ via loader glob base
llms.txt consumption	Build-served + coverage gate; do not commit generated artifacts
Prune autonomy	Auto-delete clear-dead by criteria; file beads for ambiguous

Architecture

Layout

selfhosted-cluster/
├── docs/                      # UNCHANGED location — single source of truth
│   ├── getting-started/  operations/  reference/  architecture/  adr/
│   └── plans/  superpowers/   # groomed in Phase 2
├── website/                   # NEW — isolated Astro/Starlight subproject
│   ├── package.json  pnpm-lock.yaml  astro.config.mjs
│   ├── src/content.config.ts  # custom glob loader → ../docs + docsSchema()
│   ├── src/loaders/docs.ts     # wraps astro/loaders glob() with external base
│   └── src/  public/
├── tools/docs-audit/          # NEW — Phase 1 validation harness
└── .github/workflows/docs.yml # REWIRED in Phase 3 (build step only)

Rationale: website/ keeps Node tooling out of the Ansible/Terraform/k8s repo root, while docs/ staying at root preserves every CLAUDE.md pointer, edit_uri, and the cross-check paths. The starlight-llms-txt plugin operates on Starlight’s rendered page graph (by slug), downstream of the loader, so it imposes no constraint on physical file location.

Content loader (grounded). Starlight’s stock docsLoader() hardwires its base to srcDir + content/ + <collection> (via getCollectionPathFromRoot), so it cannot read ../docs directly. The supported mechanism is a thin custom loader that wraps Astro’s glob() from astro/loaders with an external base and pairs it with docsSchema():

website/src/loaders/docs.ts

import { glob } from 'astro/loaders';
import type { Loader, LoaderContext } from 'astro/loaders';

export function externalDocsLoader(externalPath: string): Loader {
  return {
    name: 'external-docs-loader',
    load: (ctx: LoaderContext) =>
      glob({
        base: new URL(externalPath, ctx.config.root),
        pattern: '**/*.{md,mdx}',
      }).load(ctx),
  };
}

website/src/content.config.ts

import { defineCollection } from 'astro:content';
import { docsSchema } from '@astrojs/starlight/schema';
import { externalDocsLoader } from './loaders/docs';

export const collections = {
  docs: defineCollection({ loader: externalDocsLoader('../docs'), schema: docsSchema() }),
};

Default generateId() slugifies file paths, so ../docs/operations/vault.md → /operations/vault. This is a documented pattern (not a gamble); Risk #1 below verifies slug/sidebar/edit-link fidelity rather than feasibility. Symlinking src/content/docs to ../docs is explicitly rejected — Astro does not officially support symlinked content collections and behavior is inconsistent across OS/build environments.

Validation’s two lifecycles

Validation appears twice with the same conceptual checks but different lifecycles:

• One-shot audit (Phase 1) — tooling-agnostic, runs against raw docs/ markdown, emits a report that drives grooming. Exists before Starlight.
• Permanent gates (Phase 3) — starlight-links-validator (internal links/anchors, build-failing) and the llms-full.txt coverage assertion run in CI. External-link liveness and the cluster cross-check remain scheduled/local, never PR-blocking.

Phase 1 — VALIDATE (read-only audit harness)

Ships a PR adding tools/docs-audit/ plus a committed baseline docs-audit.md report. No content changes.

Dimension	Mechanism	Notes
Internal links + anchors	lychee --offline (or markdown-link-check) over docs/**	Resolves relative links + #anchor fragments
Orphan / coverage	Script: enumerate docs/**/*.md, diff vs MkDocs nav reachability	Produces the ranked unreferenced-file list that drives Phase 2
External link liveness	lychee (online, rate-limited, cached)	Run on demand; flaky, never a hard gate
Live-cluster cross-check	Bespoke script + read-only k8s MCP / kubectl + Vault read	services.md vs running Services; secrets.md vs Vault paths; version pins vs deployed images

Hard constraint: the cluster cross-check is local/scheduled only — GitHub Actions runners cannot reach the private cluster, and the k8s MCP is read-only/local. It ships as a make docs-audit-cluster target and optionally an in-cluster CronJob that posts findings to ntfy. It is never wired into CI as a build gate.

Output: a single docs-audit.md with sections per dimension, each finding classified keep / delete-candidate / uncertain / drift.

Phase 2 — GROOM (aggressive prune)

Driven entirely by the Phase 1 report. Still on MkDocs — the site keeps building throughout, so every grooming PR is independently shippable.

• Auto-delete clear-dead by criteria. Candidate dead-criteria (finalized against actual audit output, but scaffolded here so the planner isn’t starting from zero):

  • Lives under plans/archive/ (already self-declared archived).
  • Frontmatter or heading marks Status: superseded|completed|abandoned, or the plan references a PR/bead that is merged/closed.
  • A spec whose paired plan is fully implemented and whose decisions are now captured in an ADR under docs/adr/.
  • Zero inbound links from any living doc (per the Phase 1 orphan check) and older than a staleness threshold (e.g. last git-touch > N months).

  The grooming script deletes files matching these directly (git history is the backup).

• File beads for ambiguous. Anything not matching clear-dead criteria gets a bead under the design epic for human triage, rather than being deleted or silently kept.

• Reconcile cross-check drift. Fix stale services.md / secrets.md / network.md, or file beads for genuine infra drift surfaced by the audit.

• Define target IA. Establish the sidebar group structure Starlight will autogenerate from in Phase 3.

Phase 3 — MIGRATE (big-bang cutover)

One PR:

1. Scaffold website/ — Astro + @astrojs/starlight + starlight-links-validator + starlight-llms-txt. Add the custom externalDocsLoader('../docs') (see Architecture) so the docs collection reads repo-root docs/.
2. Add frontmatter — title (required; derived from existing H1), sidebar.order / sidebar.label / sidebar.hidden as needed. Largest mechanical task; scripted from existing H1s + the MkDocs nav ordering.
3. Translate MkDocs syntax → Starlight — !!! note / !!! warning admonitions → :::note / :::caution asides; pymdownx.tabbed → <Tabs> / <TabItem>; snippets/emoji as needed. Fenced code blocks work as-is via Expressive Code. Delivered as a scripted codemod plus a visual diff review on the preview deploy.
4. Rewire .github/workflows/docs.yml — three coordinated changes:
   • Build step: replace uvx --with mkdocs-material ... mkdocs build with pnpm install + pnpm --dir website build (Node/pnpm setup via setup-node + pnpm/action-setup, replacing setup-uv).
   • Deploy path: the wrangler command’s positional output dir changes from site (MkDocs) to website/dist (Astro’s default output). The --project-name=cluster-docs and preview-branch flags stay the same, so the Cloudflare Access policy and preview-URL PR comment are preserved — but the deploy command is not literally “unchanged.”
   • Trigger paths: on.push.paths / on.pull_request.paths change from docs/** + mkdocs.yml to docs/** + website/** + the workflow file (drop the now-deleted mkdocs.yml). Without website/**, changes to astro.config.mjs or the loader would not trigger a build.
5. Wire permanent gates — starlight-links-validator fails the build on broken internal links/anchors (plugin-provided). A custom coverage step (small post-build script) enumerates the docs collection’s page slugs and asserts each appears in the generated llms-full.txt, failing CI on a gap. Coverage is our check, not a starlight-llms-txt feature.
6. Delete mkdocs.yml; update CLAUDE.md routing pointers and add the llms-full.txt reference for AI consumption.

llms.txt

Configured via starlight-llms-txt (grounded against the plugin’s published config docs): projectName, description, optional customSets for per-area bundles (e.g. operations, reference), and minify/exclude for llms-small.txt. The plugin emits llms.txt (index), llms-full.txt (concatenated), and llms-small.txt (minified) at build, served by the site; authenticated agent infra (agentgateway) fetches via a Cloudflare Access service token. Artifacts are not committed to the repo. The custom coverage gate (Phase 3 step 5) reads the build’s llms-full.txt to confirm every page is included — this is our assertion layered on top of the plugin, not a plugin capability.

Risks & Spikes

#	Risk	Mitigation
1	Out-of-project content base (../docs) — handled by the custom externalDocsLoader (grounded; reference impl in Architecture). Residual risk is fidelity, not feasibility: do generated slugs, autogenerated sidebar order, and edit_uri links match expectations?	30-min spike builds a handful of pages through the custom loader and verifies slugs/sidebar/edit-links. Fallback if fidelity fails: Approach A (Astro at repo root, content still in docs/). Symlinks are rejected (unsupported by Astro)
2	Syntax-translation fidelity (admonitions, tabs)	Scripted codemod + visual diff on preview deploy before cutover
3	Cluster cross-check scope creep	Bound to the four reference docs; advisory only; resist building a reconciler
4	Frontmatter title collisions / missing H1s	Audit reports files lacking a clean H1; handle in Phase 3 prep
5	Cloudflare Pages build env (Node vs Python)	Build runs in GitHub Actions, not CF’s builder; only the build command changes, deploy step is unchanged

Success Criteria

• pnpm --dir website build produces a Starlight site from ../docs with no broken internal links (validator passes)
• Cloudflare Pages production + preview deploys work; Access policy unchanged
• llms.txt / llms-full.txt / llms-small.txt generated and served; coverage check passes
• Orphaned file count reduced aggressively; clear-dead auto-deleted, ambiguous tracked as beads
• docs-audit.md reproducible via a documented command; cluster cross-check runnable locally
• mkdocs.yml removed; CLAUDE.md pointers updated
• Each phase landed as its own PR. Phases 1 and 2 are independently shippable (read-only, then MkDocs-still-building); Phase 3 is sequentially dependent — it requires the Phase 1 loader-fidelity spike to pass and the Phase 2 IA to be defined before its PR opens

Grounding references

Consulted during brainstorming (also recorded as bd note traces on hl-bit):

• probe + read — docs/plans/archive/2025-12-29-docs-site-design.md: MkDocs chosen to replace Git+Notion; site behind Cloudflare Access (Authentik/GitHub/ OTP, domain-restricted); CLAUDE.md files intentionally a routing layer.
• probe + read — mkdocs.yml, .github/workflows/docs.yml: current builder is uvx mkdocs build → output site/ → wrangler pages deploy site --project-name=cluster-docs; triggers on docs/** + mkdocs.yml.
• context7 / deepwiki — withastro/starlight: docsLoader() base is hardwired via getCollectionPathFromRoot; external content requires a custom glob() loader with base + docsSchema() (reference impl in Architecture); autogenerate sidebar; frontmatter title (required), sidebar.hidden/order/ label, draft, _-prefix ignored; starlight-links-validator fails build on broken internal links/anchors.
• firecrawl — delucis/starlight-llms-txt config docs (v0.10.0): emits llms.txt / llms-full.txt / llms-small.txt; options projectName, description, customSets, promote/demote/exclude, minify, rawContent, pageSeparator. No built-in coverage gate (custom, per Phase 3).

Open Items (resolve in planning)

• Final dead-criteria for auto-delete (decided against real audit output)
• customSets taxonomy for llms.txt (which doc areas become bundles)
• Whether the cluster cross-check CronJob ships in this effort or as a follow-up