Docs Starlight Migration — Phase 1 (Validation/Audit Harness) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a Python audit harness under tools/docs-audit/ that inventories the docs across four dimensions (internal links/anchors, orphan/coverage, external-link liveness, live-cluster cross-check) and emits a committed docs-audit.md report that drives Phase 2 grooming.

Architecture: A small importable package (docs_audit) of pure-logic modules (nav parsing, orphan detection, frontmatter/H1 parsing, lychee-JSON parsing, report assembly) unit-tested with pytest, plus thin integration wrappers that shell out to lychee and talk to the cluster via the already-pinned kubernetes/hvac clients. A single audit.py CLI orchestrates checks. No content changes ship in Phase 1.

Tech Stack: Python 3.13 + uv (repo convention), pytest (new dev-dep), PyYAML/kubernetes/hvac (already pinned in pyproject.toml), lychee (external Rust binary) for link checking.

Design bead: hl-bit · Spec: docs/engineering/specs/2026-06-07-docs-starlight-migration-design.md

Model labels: All Phase 1 tasks are mechanical TDD scaffold work — materialize each bead with model:haiku (per the Rule 5 --labels model:<value> convention).

File Structure

Path	Responsibility
`tools/docs-audit/docs_audit/__init__.py`	Package marker
`tools/docs-audit/docs_audit/nav.py`	Parse `mkdocs.yml` `nav:` → set of reachable doc paths (pure)
`tools/docs-audit/docs_audit/orphans.py`	Enumerate `docs/*/.md`, diff vs reachable → orphans (pure)
`tools/docs-audit/docs_audit/frontmatter.py`	Split YAML frontmatter, extract first H1 (pure; Phase 3 title prep)
`tools/docs-audit/docs_audit/lychee.py`	Shell out to `lychee`; parse its `--format json` fail map (parser is pure)
`tools/docs-audit/docs_audit/cluster.py`	Extract namespaces/Vault-paths from reference docs (pure) + live k8s/Vault checks (integration)
`tools/docs-audit/docs_audit/report.py`	Assemble the `docs-audit.md` markdown report (pure)
`tools/docs-audit/audit.py`	CLI entrypoint orchestrating the checks
`tools/docs-audit/tests/test_*.py`	Unit tests for the pure modules
`tools/docs-audit/docs-audit.md`	Generated, committed baseline report (Phase 2 input)
`tools/docs-audit/README.md`	How to run the harness
`pyproject.toml`	Add `pytest` dev-dep + `[tool.pytest.ini_options]`
`.github/workflows/docs-audit.yml`	Informational offline audit on docs PRs

Pure modules hold all the testable logic; integration is isolated to two thin seams (lychee.run_* and cluster.live_*) so the rest is deterministic.

Task 1: Scaffold package + pytest config

Files:

Modify: pyproject.toml
Create: tools/docs-audit/docs_audit/__init__.py
Create: tools/docs-audit/tests/__init__.py
Create: tools/docs-audit/README.md
Test: tools/docs-audit/tests/test_smoke.py
Step 1: Add pytest dev-dep and pytest config to pyproject.toml

In the [project.optional-dependencies] dev = [ ... ] list, add:

  "pytest>=8.3,<9",

Append a new section at the end of pyproject.toml:

[tool.pytest.ini_options]
pythonpath = ["tools/docs-audit"]
testpaths = ["tools/docs-audit/tests"]

Step 2: Create the package + test package markers and README

Create tools/docs-audit/docs_audit/__init__.py:

"""docs-audit: a read-only audit harness for the cluster docs."""

Create tools/docs-audit/tests/__init__.py (empty file).

Create tools/docs-audit/README.md:

# docs-audit

Read-only audit harness for `docs/`. Emits `docs-audit.md`.

## Run

```bash
uv sync --extra dev
# offline checks (links + orphans + frontmatter):
uv run python tools/docs-audit/audit.py --check offline --out tools/docs-audit/docs-audit.md
# add external link liveness (slow, online):
uv run python tools/docs-audit/audit.py --check external
# add live-cluster cross-check (needs kubeconfig + VAULT_ADDR/VAULT_TOKEN):
uv run python tools/docs-audit/audit.py --check cluster
```

Prerequisite: `lychee` (`brew install lychee` or `cargo install lychee`).

## Tests

```bash
uv run pytest tools/docs-audit/tests -q
```

Step 3: Write a smoke test

Create tools/docs-audit/tests/test_smoke.py:

import docs_audit


def test_package_imports():
    assert docs_audit is not None

Step 4: Sync and run the smoke test

Run: uv sync --extra dev && uv run pytest tools/docs-audit/tests -q Expected: PASS (1 passed).

Step 5: Commit

jj commit -m "feat(docs-audit): scaffold audit package + pytest config [hl-bit]" (VCS-appropriate per references/vcs-preamble.md.)

Task 2: `nav.py` — reachable docs from the MkDocs nav

Files:

Create: tools/docs-audit/docs_audit/nav.py
Test: tools/docs-audit/tests/test_nav.py
Step 1: Write the failing test

Create tools/docs-audit/tests/test_nav.py:

from docs_audit.nav import reachable_docs


def test_reachable_docs_collects_nested_paths():
    nav = [
        {"Home": "index.md"},
        {"Operations": [
            "operations/index.md",
            {"Vault": "operations/vault.md"},
            {"Incidents": [{"One": "operations/incidents/a.md"}]},
        ]},
    ]
    assert reachable_docs(nav) == {
        "index.md",
        "operations/index.md",
        "operations/vault.md",
        "operations/incidents/a.md",
    }


def test_reachable_docs_ignores_non_md_strings():
    assert reachable_docs([{"Ext": "https://example.com"}]) == set()

Step 2: Run test to verify it fails

Run: uv run pytest tools/docs-audit/tests/test_nav.py -q Expected: FAIL with ModuleNotFoundError: No module named 'docs_audit.nav'.

Step 3: Write minimal implementation

Create tools/docs-audit/docs_audit/nav.py:

from pathlib import Path

import yaml


def parse_nav(mkdocs_path: Path) -> list:
    """Return the raw `nav:` list from a mkdocs.yml file (empty if absent)."""
    data = yaml.safe_load(Path(mkdocs_path).read_text(encoding="utf-8"))
    return (data or {}).get("nav", []) or []


def reachable_docs(nav: list) -> set[str]:
    """Collect every `*.md` path referenced anywhere in a mkdocs nav tree.

    Paths are returned exactly as written in mkdocs.yml (relative to docs/).
    """
    found: set[str] = set()

    def walk(node) -> None:
        if isinstance(node, str):
            if node.endswith(".md"):
                found.add(node)
        elif isinstance(node, dict):
            for value in node.values():
                walk(value)
        elif isinstance(node, list):
            for item in node:
                walk(item)

    walk(nav)
    return found

Step 4: Run test to verify it passes

Run: uv run pytest tools/docs-audit/tests/test_nav.py -q Expected: PASS (2 passed).

Step 5: Commit

jj commit -m "feat(docs-audit): nav reachable-docs parser [hl-bit]"

Task 3: `orphans.py` — orphan/coverage detection

Files:

Create: tools/docs-audit/docs_audit/orphans.py
Test: tools/docs-audit/tests/test_orphans.py
Step 1: Write the failing test

Create tools/docs-audit/tests/test_orphans.py:

from pathlib import Path

from docs_audit.orphans import all_markdown, find_orphans, group_by_top_dir


def test_all_markdown_relative(tmp_path: Path):
    (tmp_path / "a.md").write_text("x")
    sub = tmp_path / "ops"
    sub.mkdir()
    (sub / "b.md").write_text("y")
    assert all_markdown(tmp_path) == {"a.md", "ops/b.md"}


def test_find_orphans_excludes_reachable_and_excluded():
    all_md = {"index.md", "a.md", "plans/old.md", "README.md"}
    reachable = {"index.md", "a.md"}
    assert find_orphans(all_md, reachable, exclude={"README.md"}) == ["plans/old.md"]


def test_group_by_top_dir():
    assert group_by_top_dir(["plans/a.md", "plans/b.md", "ops/c.md"]) == {
        "plans": ["plans/a.md", "plans/b.md"],
        "ops": ["ops/c.md"],
    }

Step 2: Run test to verify it fails

Run: uv run pytest tools/docs-audit/tests/test_orphans.py -q Expected: FAIL with ModuleNotFoundError: No module named 'docs_audit.orphans'.

Step 3: Write minimal implementation

Create tools/docs-audit/docs_audit/orphans.py:

from pathlib import Path


def all_markdown(docs_dir: Path) -> set[str]:
    """All `*.md` files under docs_dir, as POSIX paths relative to docs_dir."""
    docs_dir = Path(docs_dir)
    return {p.relative_to(docs_dir).as_posix() for p in docs_dir.rglob("*.md")}


def find_orphans(
    all_md: set[str], reachable: set[str], exclude: set[str] | None = None
) -> list[str]:
    """Markdown files present on disk but not referenced by the nav."""
    exclude = exclude or set()
    return sorted(all_md - reachable - exclude)


def group_by_top_dir(paths: list[str]) -> dict[str, list[str]]:
    """Group relative paths by their first path segment for reporting."""
    groups: dict[str, list[str]] = {}
    for p in paths:
        top = p.split("/", 1)[0] if "/" in p else "."
        groups.setdefault(top, []).append(p)
    return groups

Step 4: Run test to verify it passes

Run: uv run pytest tools/docs-audit/tests/test_orphans.py -q Expected: PASS (3 passed).

Step 5: Commit

jj commit -m "feat(docs-audit): orphan/coverage detection [hl-bit]"

Task 4: `frontmatter.py` — frontmatter split + first H1

Files:

Create: tools/docs-audit/docs_audit/frontmatter.py
Test: tools/docs-audit/tests/test_frontmatter.py
Step 1: Write the failing test

Create tools/docs-audit/tests/test_frontmatter.py:

from docs_audit.frontmatter import first_h1, split_frontmatter, title_status


def test_split_frontmatter_present():
    fm, body = split_frontmatter("---\ntitle: X\n---\n# Heading\n")
    assert fm == "title: X"
    assert body.strip() == "# Heading"


def test_split_frontmatter_absent():
    fm, body = split_frontmatter("# Just heading\n")
    assert fm is None
    assert body == "# Just heading\n"


def test_first_h1():
    assert first_h1("intro\n# My Title\nmore") == "My Title"
    assert first_h1("no heading here") is None


def test_title_status_reports_missing_title():
    # No frontmatter, has H1 -> migratable (title derivable from H1)
    assert title_status("# Vault\n") == {"has_frontmatter": False, "h1": "Vault", "needs_attention": False}
    # No frontmatter, no H1 -> needs attention (Starlight requires a title)
    assert title_status("plain text") == {"has_frontmatter": False, "h1": None, "needs_attention": True}

Step 2: Run test to verify it fails

Run: uv run pytest tools/docs-audit/tests/test_frontmatter.py -q Expected: FAIL with ModuleNotFoundError: No module named 'docs_audit.frontmatter'.

Step 3: Write minimal implementation

Create tools/docs-audit/docs_audit/frontmatter.py:

import re

_FM_RE = re.compile(r"^---\n(.*?)\n---\n?(.*)$", re.DOTALL)
_H1_RE = re.compile(r"^#\s+(.+?)\s*$", re.MULTILINE)


def split_frontmatter(text: str) -> tuple[str | None, str]:
    """Return (frontmatter_block_or_None, body)."""
    match = _FM_RE.match(text)
    if not match:
        return None, text
    return match.group(1), match.group(2)


def first_h1(body: str) -> str | None:
    """First level-1 ATX heading text, or None."""
    match = _H1_RE.search(body)
    return match.group(1) if match else None


def title_status(text: str) -> dict:
    """Phase-3 readiness: Starlight requires a `title`. A file with no
    frontmatter but a clean H1 is migratable; one with neither needs a human."""
    fm, body = split_frontmatter(text)
    h1 = first_h1(body)
    return {
        "has_frontmatter": fm is not None,
        "h1": h1,
        "needs_attention": fm is None and h1 is None,
    }

Step 4: Run test to verify it passes

Run: uv run pytest tools/docs-audit/tests/test_frontmatter.py -q Expected: PASS (4 passed).

Step 5: Commit

jj commit -m "feat(docs-audit): frontmatter + H1 title-readiness check [hl-bit]"

Task 5: `lychee.py` — internal-link check + JSON parser

Files:

Create: tools/docs-audit/docs_audit/lychee.py
Test: tools/docs-audit/tests/test_lychee.py
Step 1: Write the failing test

Create tools/docs-audit/tests/test_lychee.py:

from docs_audit.lychee import build_command, parse_failures


def test_parse_failures_flattens_error_map():
    # Real lychee --format json shape: top-level `error_map`, each entry has a
    # `url` and a nested `status` object with `text`/`code`.
    raw = (
        '{"total":3,"successful":1,"errors":2,'
        '"error_map":{"docs/a.md":['
        '{"url":"./missing.md","status":{"text":"Cannot find file","code":null}},'
        '{"url":"#nope","status":{"text":"Missing fragment","code":null}}]}}'
    )
    assert parse_failures(raw) == [
        {"source": "docs/a.md", "url": "./missing.md", "status": "Cannot find file"},
        {"source": "docs/a.md", "url": "#nope", "status": "Missing fragment"},
    ]


def test_parse_failures_includes_timeouts():
    raw = (
        '{"timeout_map":{"docs/b.md":['
        '{"url":"https://slow.example","status":{"text":"Timeout"}}]}}'
    )
    assert parse_failures(raw) == [
        {"source": "docs/b.md", "url": "https://slow.example", "status": "Timeout"},
    ]


def test_parse_failures_empty_when_no_error_map():
    assert parse_failures('{"total":1,"successful":1}') == []


def test_build_command_offline_includes_fragments():
    cmd = build_command(["docs"], offline=True)
    assert cmd[0] == "lychee"
    assert "--offline" in cmd
    assert "--include-fragments" in cmd
    assert "--format" in cmd and "json" in cmd


def test_build_command_online_omits_offline():
    cmd = build_command(["docs"], offline=False)
    assert "--offline" not in cmd

Step 2: Run test to verify it fails

Run: uv run pytest tools/docs-audit/tests/test_lychee.py -q Expected: FAIL with ModuleNotFoundError: No module named 'docs_audit.lychee'.

Step 3: Write minimal implementation

Create tools/docs-audit/docs_audit/lychee.py:

import json
import shutil
import subprocess


def build_command(paths: list[str], offline: bool) -> list[str]:
    """Construct the lychee invocation. JSON output; `--include-fragments`
    (defaults to `anchor-only`) makes offline runs catch missing `#anchor`
    targets in local files."""
    cmd = ["lychee", "--format", "json", "--no-progress"]
    if offline:
        cmd += ["--offline", "--include-fragments"]
    cmd += paths
    return cmd


def parse_failures(raw: str) -> list[dict]:
    """Flatten lychee's `error_map` and `timeout_map` into {source, url, status}.

    lychee --format json nests the human-readable reason under `status.text`
    (an object, not a bare string); each failure entry's link is `url`."""
    data = json.loads(raw)
    failures: list[dict] = []
    for map_key in ("error_map", "timeout_map"):
        for source, entries in (data.get(map_key) or {}).items():
            for entry in entries:
                status = entry.get("status") or {}
                text = status.get("text", "") if isinstance(status, dict) else str(status)
                failures.append(
                    {"source": source, "url": entry.get("url", ""), "status": text}
                )
    return failures


def run(paths: list[str], offline: bool) -> list[dict]:
    """Integration seam: execute lychee and return parsed failures.

    Raises RuntimeError if lychee is not installed so the gap is loud, never
    silently treated as 'no broken links'."""
    if shutil.which("lychee") is None:
        raise RuntimeError("lychee not found on PATH; install via `brew install lychee`")
    proc = subprocess.run(
        build_command(paths, offline), capture_output=True, text=True
    )
    # lychee exits non-zero when it finds broken links; that is expected.
    return parse_failures(proc.stdout)

Step 4: Run test to verify it passes

Run: uv run pytest tools/docs-audit/tests/test_lychee.py -q Expected: PASS (5 passed).

Step 5: Commit

jj commit -m "feat(docs-audit): lychee wrapper + error-map parser [hl-bit]"

Task 6: `cluster.py` — reference-doc extractors + live cross-check

Files:

Create: tools/docs-audit/docs_audit/cluster.py
Test: tools/docs-audit/tests/test_cluster.py
Step 1: Write the failing test

Create tools/docs-audit/tests/test_cluster.py:

from docs_audit.cluster import (
    diff_sets,
    extract_namespaces,
    extract_vault_paths,
)

SERVICES_MD = """# Services Catalog

| Service | URL | Namespace | Category |
|---------|-----|-----------|----------|
| [Vault](#vault) | `vault.fzymgc.house` | vault | Platform |
| [ArgoCD](#argocd) | `argocd.fzymgc.house` | argocd | Platform |
| [K8s OIDC RBAC](#k8s-oidc-rbac) | Internal | N/A | Platform |
"""

SECRETS_MD = """# Secrets

| `secret/fzymgc-house/cluster/authentik` | Authentik | `x` |
| `secret/fzymgc-house/cluster/vault/*` | Vault | `y` |
"""


def test_extract_namespaces_skips_na():
    assert extract_namespaces(SERVICES_MD) == {"vault", "argocd"}


def test_extract_vault_paths_skips_wildcards():
    assert extract_vault_paths(SECRETS_MD) == {"secret/fzymgc-house/cluster/authentik"}


def test_diff_sets():
    only_doc, only_live = diff_sets({"a", "b"}, {"b", "c"})
    assert only_doc == ["a"]
    assert only_live == ["c"]

Step 2: Run test to verify it fails

Run: uv run pytest tools/docs-audit/tests/test_cluster.py -q Expected: FAIL with ModuleNotFoundError: No module named 'docs_audit.cluster'.

Step 3: Write minimal implementation

Create tools/docs-audit/docs_audit/cluster.py:

import re

_VAULT_PATH = re.compile(r"`(secret/[^`*]+)`")


def _cells(row: str) -> list[str]:
    """Split a markdown table row into trimmed cell values."""
    return [c.strip() for c in row.strip().strip("|").split("|")]


def extract_namespaces(services_md: str) -> set[str]:
    """Namespaces from the services table whose header contains 'Namespace'.

    Anchors on the header row (locating the Namespace column by name), then
    reads that column from subsequent rows until the table ends. This avoids
    matching the `|---|---|` separator row or unrelated tables in the file."""
    out: set[str] = set()
    ns_col: int | None = None
    for line in services_md.splitlines():
        if not line.lstrip().startswith("|"):
            ns_col = None  # table ended
            continue
        cells = _cells(line)
        if "Namespace" in cells:  # header row → lock onto the column
            ns_col = cells.index("Namespace")
            continue
        if ns_col is None or ns_col >= len(cells):
            continue
        ns = cells[ns_col]
        if not ns or ns == "N/A" or set(ns) <= {"-"}:  # skip blanks/N/A/separator
            continue
        out.add(ns)
    return out


def extract_vault_paths(secrets_md: str) -> set[str]:
    """`secret/...` KV paths in secrets.md.

    Excludes wildcard (`*`) entries, `<placeholder>` templates, and the raw
    `secret/data/...` KV-v2 API form (which `live_vault_paths_exist` can't
    resolve via the high-level hvac client)."""
    out: set[str] = set()
    for path in _VAULT_PATH.findall(secrets_md):
        path = path.rstrip("/")
        if "<" in path or path.startswith("secret/data/"):
            continue
        out.add(path)
    return out


def diff_sets(doc: set[str], live: set[str]) -> tuple[list[str], list[str]]:
    """Return (only-in-doc, only-in-live), each sorted."""
    return sorted(doc - live), sorted(live - doc)


def live_namespaces() -> set[str]:
    """Integration seam: namespaces from the current kubeconfig context."""
    from kubernetes import client, config

    config.load_kube_config()
    return {ns.metadata.name for ns in client.CoreV1Api().list_namespace().items}


def live_vault_paths_exist(paths: set[str]) -> dict[str, bool]:
    """Integration seam: whether each KV-v2 path reads back from Vault.

    Uses VAULT_ADDR / VAULT_TOKEN from the environment."""
    import os

    import hvac

    vault = hvac.Client(url=os.environ["VAULT_ADDR"], token=os.environ["VAULT_TOKEN"])
    result: dict[str, bool] = {}
    for path in paths:
        sub = path.removeprefix("secret/")
        try:
            vault.secrets.kv.v2.read_secret_version(path=sub, mount_point="secret")
            result[path] = True
        except Exception:  # noqa: BLE001 - advisory check; absence is the signal
            result[path] = False
    return result

Step 4: Run test to verify it passes

Run: uv run pytest tools/docs-audit/tests/test_cluster.py -q Expected: PASS (3 passed).

Step 5: Commit

jj commit -m "feat(docs-audit): cluster cross-check extractors + live seams [hl-bit]"

Task 7: `report.py` — assemble the `docs-audit.md` report

Files:

Create: tools/docs-audit/docs_audit/report.py
Test: tools/docs-audit/tests/test_report.py
Step 1: Write the failing test

Create tools/docs-audit/tests/test_report.py:

from docs_audit.report import render_report


def test_render_report_has_sections_and_counts():
    md = render_report(
        orphans_by_dir={"plans": ["plans/a.md", "plans/b.md"]},
        internal_failures=[{"source": "docs/x.md", "url": "#z", "status": "Missing fragment"}],
        external_failures=[],
        title_attention=["reference/weird.md"],
        cluster_findings={"namespaces_only_in_doc": ["ghost"], "namespaces_only_in_cluster": []},
    )
    assert "# Docs Audit Report" in md
    assert "## Orphans (2)" in md
    assert "plans/a.md" in md
    assert "## Internal link failures (1)" in md
    assert "Missing fragment" in md
    assert "## Files needing a title (1)" in md
    assert "ghost" in md

Step 2: Run test to verify it fails

Run: uv run pytest tools/docs-audit/tests/test_report.py -q Expected: FAIL with ModuleNotFoundError: No module named 'docs_audit.report'.

Step 3: Write minimal implementation

Create tools/docs-audit/docs_audit/report.py:

def _count(groups: dict[str, list[str]]) -> int:
    return sum(len(v) for v in groups.values())


def render_report(
    *,
    orphans_by_dir: dict[str, list[str]],
    internal_failures: list[dict],
    external_failures: list[dict],
    title_attention: list[str],
    cluster_findings: dict | None,
) -> str:
    lines: list[str] = ["# Docs Audit Report", ""]

    lines.append(f"## Orphans ({_count(orphans_by_dir)})")
    lines.append("")
    for top in sorted(orphans_by_dir):
        lines.append(f"### {top}")
        lines += [f"- `{p}`" for p in orphans_by_dir[top]]
        lines.append("")

    lines.append(f"## Internal link failures ({len(internal_failures)})")
    lines += [f"- `{f['source']}` → `{f['url']}` ({f['status']})" for f in internal_failures]
    lines.append("")

    lines.append(f"## External link failures ({len(external_failures)})")
    lines += [f"- `{f['source']}` → `{f['url']}` ({f['status']})" for f in external_failures]
    lines.append("")

    lines.append(f"## Files needing a title ({len(title_attention)})")
    lines += [f"- `{p}`" for p in title_attention]
    lines.append("")

    if cluster_findings is not None:
        lines.append("## Cluster cross-check")
        for key, values in cluster_findings.items():
            lines.append(f"- **{key}**: {', '.join(values) if values else '(none)'}")
        lines.append("")

    return "\n".join(lines) + "\n"

Step 4: Run test to verify it passes

Run: uv run pytest tools/docs-audit/tests/test_report.py -q Expected: PASS (1 passed).

Step 5: Commit

jj commit -m "feat(docs-audit): report renderer [hl-bit]"

Task 8: `audit.py` — CLI entrypoint + generate baseline report

Files:

Create: tools/docs-audit/audit.py
Create: tools/docs-audit/docs-audit.md (generated, committed)
Step 1: Write the CLI

Create tools/docs-audit/audit.py:

import argparse
from pathlib import Path

from docs_audit import cluster, frontmatter, lychee, nav, orphans, report

REPO = Path(__file__).resolve().parents[2]
DOCS = REPO / "docs"
MKDOCS = REPO / "mkdocs.yml"


def _offline_sections() -> dict:
    reachable = nav.reachable_docs(nav.parse_nav(MKDOCS))
    all_md = orphans.all_markdown(DOCS)
    orphan_list = orphans.find_orphans(all_md, reachable, exclude={"README.md"})

    attention: list[str] = []
    for rel in sorted(all_md):
        status = frontmatter.title_status((DOCS / rel).read_text(encoding="utf-8"))
        if status["needs_attention"]:
            attention.append(rel)

    return {
        "orphans_by_dir": orphans.group_by_top_dir(orphan_list),
        "internal_failures": lychee.run([str(DOCS)], offline=True),
        "title_attention": attention,
    }


def _cluster_section() -> dict:
    services_md = (DOCS / "reference" / "services.md").read_text(encoding="utf-8")
    secrets_md = (DOCS / "reference" / "secrets.md").read_text(encoding="utf-8")
    doc_ns = cluster.extract_namespaces(services_md)
    only_doc, only_live = cluster.diff_sets(doc_ns, cluster.live_namespaces())
    vault_exist = cluster.live_vault_paths_exist(cluster.extract_vault_paths(secrets_md))
    return {
        "namespaces_only_in_doc": only_doc,
        "namespaces_only_in_cluster": only_live,
        "vault_paths_missing": sorted(p for p, ok in vault_exist.items() if not ok),
    }


def main() -> None:
    parser = argparse.ArgumentParser(description="Audit the cluster docs.")
    parser.add_argument(
        "--check",
        choices=["offline", "external", "cluster", "all"],
        default="offline",
    )
    parser.add_argument("--out", type=Path, default=None)
    args = parser.parse_args()

    sections = _offline_sections()
    external_failures: list[dict] = []
    cluster_findings = None

    if args.check in {"external", "all"}:
        external_failures = lychee.run([str(DOCS)], offline=False)
    if args.check in {"cluster", "all"}:
        cluster_findings = _cluster_section()

    md = report.render_report(
        orphans_by_dir=sections["orphans_by_dir"],
        internal_failures=sections["internal_failures"],
        external_failures=external_failures,
        title_attention=sections["title_attention"],
        cluster_findings=cluster_findings,
    )
    if args.out:
        args.out.write_text(md, encoding="utf-8")
        print(f"wrote {args.out}")
    else:
        print(md)


if __name__ == "__main__":
    main()

Step 2: Verify the CLI parses and runs offline

Run: uv run python tools/docs-audit/audit.py --check offline --out tools/docs-audit/docs-audit.md Expected: prints wrote tools/docs-audit/docs-audit.md; the file exists and contains an ## Orphans (N) header with N large (the ~150 unreferenced files). (Requires lychee installed; if absent, the run raises a clear RuntimeError — install it first.)

Step 3: Sanity-check the report

Run: rg -c '^- ' tools/docs-audit/docs-audit.md Expected: a count in the dozens-to-hundreds (orphans + any link failures), confirming the harness actually traversed the tree.

Step 4: Lint the generated report

Run: rumdl check tools/docs-audit/docs-audit.md || rumdl fmt tools/docs-audit/docs-audit.md Expected: clean (or auto-fixed). tools/ is outside docs/, so the repo-root .rumdl.toml applies — do not pass --config docs/.rumdl.toml here.

Step 5: Commit

jj commit -m "feat(docs-audit): CLI entrypoint + committed baseline report [hl-bit]"

Task 9: CI workflow — informational offline audit on docs PRs

Files:

Create: .github/workflows/docs-audit.yml
Step 1: Write the workflow

Create .github/workflows/docs-audit.yml:

name: Docs Audit

on:
  pull_request:
    paths:
      - 'docs/**'
      - 'tools/docs-audit/**'
      - '.github/workflows/docs-audit.yml'
  workflow_dispatch:

permissions:
  contents: read
  pull-requests: write

jobs:
  audit:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64
    steps:
      - uses: runs-on/action@d141ef83eb66d096ce8afc767e09115a65c63b60 # v2.1.2
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          fetch-depth: 0
      - name: Install uv
        uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
      - name: Install lychee
        uses: lycheeverse/lychee-action@8646ba30535128ac92d33dfc9133794bfdd9b411 # v2.8.0
        with:
          args: --version
      - name: Run offline audit (informational)
        run: |
          uv run python tools/docs-audit/audit.py --check offline \
            --out tools/docs-audit/docs-audit.md
      - name: Upload report artifact
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: docs-audit
          path: tools/docs-audit/docs-audit.md

Step 2: Lint the workflow YAML

Run: yamllint -c .yamllint.yaml .github/workflows/docs-audit.yml Expected: clean (no errors).

Step 3: Verify pinned action SHAs resolve

Run: rg 'uses:' .github/workflows/docs-audit.yml Expected: every uses: is pinned to a 40-char SHA with a # vX comment. The SHAs above were resolved via the GitHub API during planning (lychee-action v2.8.0 = 8646ba30…, upload-artifact v4.6.2 = ea165f8d…; runs-on/checkout/setup-uv copied from the in-repo docs.yml). Re-verify only if a newer major has shipped:

gh api repos/lycheeverse/lychee-action/git/refs/tags/v2.8.0 --jq '.object.sha'
gh api repos/actions/upload-artifact/git/refs/tags/v4.6.2 --jq '.object.sha'

Step 4: Commit

jj commit -m "ci(docs-audit): informational offline audit on docs PRs [hl-bit]"

Rule 7 Grounding (executed during planning)

File paths: docs/, mkdocs.yml, docs/reference/services.md, docs/reference/secrets.md, pyproject.toml, .github/workflows/ all verified present; tools/ confirmed absent (created by Task 1). The runs-on/setup-uv action SHAs are copied from the existing .github/workflows/docs.yml (verified present and current in-repo).
Library APIs: PyYAML (yaml.safe_load), kubernetes (config.load_kube_config, CoreV1Api().list_namespace), hvac (secrets.kv.v2.read_secret_version) are all pinned in pyproject.toml; signatures are stable across the pinned majors. lychee flags (--offline, --include-fragments [anchor-only default], --format json) and the JSON shape (error_map/timeout_map, entries {url, status:{text,code}}) verified against the lychee source via Context7. Action SHAs (lychee-action v2.8.0, upload-artifact v4.6.2) resolved via the GitHub API during planning; runs-on/checkout/setup-uv SHAs copied from the in-repo docs.yml.

Phase 2 Roadmap — GROOM (detailed after Phase 1 audit output exists)

Phase 2 is intentionally not decomposed into bite-sized tasks here: its dead-criteria thresholds and triage are driven by the real docs-audit.md. Once Phase 1 lands and the report is generated, a follow-up plan will cover:

A grooming script that consumes docs-audit.md + git last-touch dates and classifies each orphan delete / keep / bead by the candidate dead-criteria in the spec (archive-prefix, Status: superseded|completed|abandoned, implemented-plan-with-ADR, zero-inbound + staleness threshold).
Auto-delete of clear-dead files; bd create (under epic hl-bit) for each ambiguous file.
Reconcile cross-check drift in services.md / secrets.md / network.md.
Define the target IA (sidebar group structure) for Phase 3.

Phase 3 Roadmap — MIGRATE (specifiable now; detailed in a follow-up plan)

Does not depend on audit output, but sequenced after grooming to migrate less. A follow-up plan will cover:

Loader spike (Risk #1): build a few pages through externalDocsLoader('../docs'); verify slugs/sidebar/edit-links. Fallback: Astro at repo root.
Scaffold website/ (Astro + Starlight + starlight-links-validator + starlight-llms-txt).
Frontmatter automation: script-add title from H1 (reusing frontmatter.first_h1), driven by the Phase 1 title_attention list.
Syntax codemod: !!! note → :::note, pymdownx.tabbed → <Tabs>/<TabItem>; visual-diff on preview deploy.
Rewire .github/workflows/docs.yml: build pnpm --dir website build; deploy dir site → website/dist; trigger paths docs/** + website/** (drop mkdocs.yml).
Permanent gates: starlight-links-validator (build-failing) + custom llms-full.txt coverage check. Replaces the informational Phase 1 workflow.
Delete mkdocs.yml; update CLAUDE.md pointers; add llms-full.txt reference.

Self-Review

Spec coverage: Phase 1’s four validation dimensions each map to a task — internal links (Task 5/8), orphans (Task 3/8), external links (Task 8 --check external), cluster cross-check (Task 6/8 --check cluster). Frontmatter/title-readiness (Task 4) covers a Phase-3 prerequisite the spec named under Risk #4. Phases 2–3 are roadmapped per the agreed scope.
Placeholders: none — every code step contains runnable code; roadmap sections are explicitly labelled as deferred, not as ### Task N: tasks.
Type consistency: reachable_docs/find_orphans/group_by_top_dir/title_status/parse_failures/render_report signatures are used identically in audit.py as defined in their tasks.

Docs Starlight Migration — Phase 1 (Validation/Audit Harness) Implementation Plan

File Structure

Task 1: Scaffold package + pytest config

Task 2: nav.py — reachable docs from the MkDocs nav

Task 3: orphans.py — orphan/coverage detection

Task 4: frontmatter.py — frontmatter split + first H1

Task 5: lychee.py — internal-link check + JSON parser

Task 6: cluster.py — reference-doc extractors + live cross-check

Task 7: report.py — assemble the docs-audit.md report

Task 8: audit.py — CLI entrypoint + generate baseline report

Task 9: CI workflow — informational offline audit on docs PRs

Rule 7 Grounding (executed during planning)

Phase 2 Roadmap — GROOM (detailed after Phase 1 audit output exists)

Phase 3 Roadmap — MIGRATE (specifiable now; detailed in a follow-up plan)

Self-Review

Task 2: `nav.py` — reachable docs from the MkDocs nav

Task 3: `orphans.py` — orphan/coverage detection

Task 4: `frontmatter.py` — frontmatter split + first H1

Task 5: `lychee.py` — internal-link check + JSON parser

Task 6: `cluster.py` — reference-doc extractors + live cross-check

Task 7: `report.py` — assemble the `docs-audit.md` report

Task 8: `audit.py` — CLI entrypoint + generate baseline report