Cilium PR4 — Decommission Calico + Production Cutover Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Remove Calico entirely (Tigera operator, CRDs, ClusterRoles, three namespaces, Ansible role) and flip Cilium from migration mode to production mode (cni.customConf: false, policyEnforcementMode: default, operator.unmanagedPodWatcher.restart: true). This is the project’s point of no return.

Architecture: Two-phase cutover. Phase A stops and removes Calico while Cilium still has cni.customConf: true (per-node CiliumNodeConfig overrides drive each node’s exclusive mode). Phase B flips the global Cilium values so the per-node overrides become moot; delete CiliumNodeConfig; restore default operator behavior. Phase order is critical — flipping Phase B values before Phase A is complete leaves a window where Cilium evicts Calico’s CNI config while calico-node pods still expect it.

Tech Stack: kubectl, helm, ansible, Velero, ArgoCD.

Reference spec: docs/engineering/specs/2026-05-10-calico-to-cilium-design.md (commit fe05a87c).

File Structure

Files modified:

ansible/roles/cilium/defaults/main.yml — flip migration toggles to production values (4 lines)
ansible/k3s-playbook.yml — remove Phase 5 (Calico) block
ansible/CLAUDE.md — remove calico row from Roles Reference + Phases tables
argocd/CLAUDE.md — Velero categories (remove Calico ns, no Cilium entries needed since cilium lives in kube-system which is already excluded)
argocd/app-configs/velero/backup-schedule.yaml — update excludedNamespaces (remove calico/tigera entries)
README.md, docs/index.md, docs/architecture/overview.md, docs/architecture/index.md, docs/reference/network.md, docs/reference/technologies.md, docs/reference/services.md — Calico → Cilium textual updates
mkdocs.yml — add nav entries for new ops docs; remove merlin nav entry

Files created:

docs/operations/cilium.md — evergreen Cilium operations runbook
(docs/operations/cilium-migration.md was started in PR2 and appended throughout PR3; finalize a “PR4 — Decommission” section)

Files deleted:

ansible/roles/calico/ (entire directory: defaults/, meta/, tasks/)
docs/operations/merlin.md (stale; merlin removed in PR #908)

Files NOT touched in this PR:

argocd/app-configs/cf-ssh-bastion/calico-network-policy.yaml — this file becomes a silent no-op once Calico CRDs are deleted (its apiVersion: projectcalico.org/v3 won’t resolve). PR5 deletes the file and removes the kustomization entry.

Cluster state changes:

All Calico/Tigera CRDs removed (~31 total)
All Calico/Tigera ClusterRoles + ClusterRoleBindings removed (~11 + matched bindings)
3 namespaces deleted: calico-system, calico-apiserver, tigera-operator
Cilium Helm release upgraded; new values flip cni.customConf → false, policyEnforcementMode → default, operator.unmanagedPodWatcher.restart → true
CiliumNodeConfig cilium-default deleted
Migration labels (io.cilium.migration/cilium-default=true) removed from all 8 nodes
system-upgrade-controller scaled back to 1 replica

Tasks

Task 1: Pre-flight (in order)

Files: none (read-only verification + backup)

Step 1: Take the pre-removal Velero backup

BACKUP_NAME="pre-calico-removal-$(date +%Y%m%d-%H%M%S)"
velero backup create "${BACKUP_NAME}" --wait
echo "Backup: ${BACKUP_NAME}"

Expected: backup completes Successful. This is the closest thing PR4 has to a rollback target. Per the spec, this only restores app data — Cilium state in kube-system and the bastion’s new policy in cf-ssh-bastion are both Velero-excluded.

Step 2: Verify all PR3 work is complete

# 1. No non-host-network pods on Calico CIDR
COUNT=$(kubectl get pods -A -o wide -o json \
  | jq '[.items[] | select(.spec.hostNetwork != true) | select(.status.podIP // "" | startswith("10.42."))] | length')
echo "Pods still on Calico CIDR: ${COUNT}"
# Expected: 0

If ${COUNT} > 0, HALT. Investigate via kubectl get pods -A -o wide | rg '10\.42\.' and complete PR3 (or rollback the offending pod’s node) before proceeding.

# 2. All 8 CiliumNodes registered
kubectl get ciliumnodes -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | wc -l
# Expected: 8

# 3. Cilium endpoint count within tolerance of non-host-network pod count
EP_COUNT=$(cilium endpoint list 2>/dev/null | tail -n +3 | wc -l)
POD_COUNT=$(kubectl get pods -A -o json \
  | jq '[.items[] | select(.spec.hostNetwork != true) | select(.status.phase == "Running")] | length')
DIFF=$((EP_COUNT - POD_COUNT))
ABS_DIFF=${DIFF#-}  # absolute value (bash)
echo "Cilium endpoints: ${EP_COUNT}, non-host-network running pods: ${POD_COUNT}, diff: ${DIFF}"
if [ "${ABS_DIFF}" -gt 5 ]; then
  echo "FAIL: endpoint/pod mismatch (${ABS_DIFF}) exceeds tolerance of 5"
  echo "Pods without matching endpoint:"
  comm -23 \
    <(kubectl get pods -A -o json | jq -r '.items[] | select(.spec.hostNetwork != true) | select(.status.phase == "Running") | "\(.metadata.namespace)/\(.metadata.name)"' | sort) \
    <(cilium endpoint list -o json | jq -r '.[] | "\(.status.namespace // "?")/\(.status["pod-name"] // "?")"' | sort)
  exit 1
fi
echo "PASS: endpoint/pod alignment within tolerance"

Expected: diff ≤ 5 (small race-window variance is OK; larger gaps indicate missed pods).

# 4. cilium status --wait
cilium status --wait

Expected: OK.

# 5. Hubble flows from every node (use CiliumNode count as authoritative)
kubectl get ciliumnodes --no-headers | wc -l
# Expected: 8

Step 3: Verify no new NetworkPolicy/CNP merged since PR2

# Enumerate every networking-policy-type resource
echo "=== networking.k8s.io NetworkPolicies (should be none) ==="
kubectl get networkpolicy -A
echo "=== Calico NetworkPolicies (should be only cf-ssh-bastion entries from PR2) ==="
kubectl get networkpolicies.crd.projectcalico.org -A
echo "=== CiliumNetworkPolicies (should be only cf-ssh-bastion entries from PR2) ==="
kubectl get ciliumnetworkpolicies.cilium.io -A
echo "=== CiliumClusterwideNetworkPolicies (should be none) ==="
kubectl get ciliumclusterwidenetworkpolicies.cilium.io
echo "=== AdminNetworkPolicies (newer K8s ANP API; Cilium 1.19 supports) ==="
kubectl get adminnetworkpolicies 2>/dev/null || echo "(no ANP CRD installed — expected)"

Expected: only the cf-ssh-bastion entries in the Calico/Cilium sections; ANP either absent CRD or no resources. If anything unexpected appears, HALT and resolve.

Task 2: Phase A — Stop the Tigera operator

Files: none (cluster state)

Step 1: Scale tigera-operator Deployment to 0

kubectl scale -n tigera-operator deploy/tigera-operator --replicas=0
kubectl wait --for=delete pod -l name=tigera-operator -n tigera-operator --timeout=2m

Expected: operator pod terminates within 2 minutes. Verify:

kubectl get pods -n tigera-operator

Expected: no pods running (No resources found in tigera-operator namespace.).

Task 3: Phase A — Delete Tigera operator-managed CRs

Files: none (cluster state)

Step 1: Delete in dependency order — Whisker first

kubectl delete whisker default --wait=true

If stuck (>2min): force-clear finalizers:

kubectl patch whisker default --type=merge -p '{"metadata":{"finalizers":[]}}'
kubectl delete whisker default --grace-period=0 --force

Step 2: Delete Goldmane

kubectl delete goldmane default --wait=true
# Same finalizer-patch escape hatch if stuck

Step 3: Delete APIServer

kubectl delete apiserver default --wait=true

Step 4: Delete Installation (this triggers calico-system workload teardown)

kubectl delete installation default --wait=true

This deletion triggers the operator-style finalizer cleanup — calico-node, calico-typha, calico-kube-controllers, calico-apiserver pods all get terminated. Since the operator is scaled to 0 (Task 2), nothing fights the deletion.

If stuck (>5min):

kubectl patch installation default --type=merge -p '{"metadata":{"finalizers":[]}}'
kubectl delete installation default --grace-period=0 --force

Step 5: Verify Calico workloads are gone

kubectl get pods -n calico-system
kubectl get pods -n calico-apiserver

Expected: both return No resources found.

Task 4: Phase A — Delete the three Calico namespaces

Files: none (cluster state)

Step 1: Delete each namespace; cascade removes any stragglers

kubectl delete namespace calico-system --wait=true --timeout=5m
kubectl delete namespace calico-apiserver --wait=true --timeout=5m
kubectl delete namespace tigera-operator --wait=true --timeout=5m

Expected: each completes within 5 minutes.

Step 2: Verify namespaces are gone

! kubectl get ns calico-system 2>/dev/null && \
! kubectl get ns calico-apiserver 2>/dev/null && \
! kubectl get ns tigera-operator 2>/dev/null && \
echo "all gone"

Expected: prints all gone.

Task 5: Phase A — Delete cluster-scoped Calico/Tigera CRDs

Files: none (cluster state)

Step 1: Enumerate and verify current CRD list matches spec

kubectl get crd -o name | rg 'projectcalico\.org|operator\.tigera\.io' | sort | tee /tmp/calico-crds-current.txt
wc -l /tmp/calico-crds-current.txt

Expected: ~31 lines. Compare against the spec’s authoritative list (PR4 section, step 4); any CRDs that appeared since spec write should also be deleted.

Step 2: Delete all listed CRDs

# Delete each CRD with a per-resource timeout, capturing failures
FAILED_CRDS=()
while IFS= read -r crd; do
  if ! kubectl delete "${crd}" --wait=true --timeout=2m 2>&1; then
    echo "FAILED to delete ${crd} cleanly"
    FAILED_CRDS+=("${crd}")
  fi
done < /tmp/calico-crds-current.txt

# Summary
if [ ${#FAILED_CRDS[@]} -eq 0 ]; then
  echo "All CRDs deleted cleanly"
else
  echo ""
  echo "==== STUCK CRDs (${#FAILED_CRDS[@]} of $(wc -l < /tmp/calico-crds-current.txt)) ===="
  printf '  %s\n' "${FAILED_CRDS[@]}"
  echo ""
  echo "For each stuck CRD, remove finalizers manually using patch (NOT apply — apply merges"
  echo "based on last-applied-configuration and may not strip finalizers):"
  echo "  for crd in \"\${FAILED_CRDS[@]}\"; do"
  echo "    kubectl patch \"\${crd}\" --type=merge -p '{\"metadata\":{\"finalizers\":[]}}'"
  echo "    kubectl delete \"\${crd}\" --wait=true --timeout=30s --ignore-not-found"
  echo "  done"
  echo ""
  echo "If the CRD's instances (CRs) have their own finalizers blocking CRD deletion,"
  echo "those need force-clearing first."
  echo ""
  echo "WARNING: bulk-stripping finalizers from CR instances bypasses controller cleanup."
  echo "This is acceptable HERE because PR4 Phase A step 1 already scaled the Tigera"
  echo "operator to 0 replicas, so the controller is no longer reconciling. Do NOT apply"
  echo "this pattern when a controller is still running — it would orphan resources the"
  echo "controller was responsible for cleaning up (volumes, cloud LB rules, certificates)."
  echo ""
  echo "Force-clear recipe (operator scaled to 0, safe to use here):"
  echo ""
  echo "  for crd_kind in <stuck-CRD-kinds>; do"
  echo "    kubectl get \"\${crd_kind}\" -A -o name 2>/dev/null \\"
  echo "      | xargs -I{} kubectl patch {} --type=merge -p '{\"metadata\":{\"finalizers\":[]}}'"
  echo "  done"
  exit 1
fi

Step 3: Verify cleanup

kubectl get crd -o name | rg 'projectcalico\.org|operator\.tigera\.io' | wc -l

Expected: 0.

rm /tmp/calico-crds-current.txt

Task 6: Phase A — Delete cluster-scoped Calico/Tigera RBAC

Files: none (cluster state)

Step 1: Enumerate ClusterRoles and ClusterRoleBindings

kubectl get clusterrole,clusterrolebinding -o name \
  | rg -i 'calico|tigera' \
  | sort \
  | tee /tmp/calico-rbac.txt
wc -l /tmp/calico-rbac.txt

Expected: 11 ClusterRoles + matched ClusterRoleBindings (per spec). The actual count depends on the cluster — may be slightly higher if the operator created any since spec write.

Step 2: Delete each

FAILED_RBAC=()
while IFS= read -r resource; do
  if ! kubectl delete "${resource}" 2>&1; then
    echo "FAILED to delete ${resource}"
    FAILED_RBAC+=("${resource}")
  fi
done < /tmp/calico-rbac.txt

if [ ${#FAILED_RBAC[@]} -ne 0 ]; then
  echo ""
  echo "==== STUCK RBAC resources (${#FAILED_RBAC[@]}) ===="
  printf '  %s\n' "${FAILED_RBAC[@]}"
  exit 1
fi

Step 3: Verify cleanup

kubectl get clusterrole,clusterrolebinding -o name | rg -i 'calico|tigera' | wc -l

Expected: 0.

rm /tmp/calico-rbac.txt

Step 4: Verify no admission webhooks (sanity)

kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration -o name | rg -i 'calico|tigera' | wc -l

Expected: 0.

Task 7: Phase B — Flip Cilium values in Ansible defaults

Files:

Modify: ansible/roles/cilium/defaults/main.yml
Step 1: Edit the defaults file — flip 4 migration toggles

Open ansible/roles/cilium/defaults/main.yml. Find this block:

# Migration toggles - PR2 values shown; PR4 flips a subset per spec Phase B
cilium_cni_custom_conf: true                          # PR2: true; PR4: false (the cutover toggle)
cilium_cni_uninstall: false                           # PR2: false; PR4: true (default)
cilium_policy_enforcement_mode: "never"               # PR2: "never"; PR4: "default"
cilium_operator_unmanaged_pod_watcher_restart: false  # PR2: false; PR4: true
cilium_bpf_host_legacy_routing: true                  # PR2: true; PR4: STAYS true (flip is deferred follow-up)

Replace with:

# Migration toggles - flipped to production values in PR4 (was PR2 migration mode)
cilium_cni_custom_conf: false                         # was true in PR2; cutover toggle
cilium_cni_uninstall: true                            # was false in PR2; chart default
cilium_policy_enforcement_mode: "default"             # was "never" in PR2; enforces CiliumNetworkPolicies now
cilium_operator_unmanaged_pod_watcher_restart: true   # was false in PR2; chart default
cilium_bpf_host_legacy_routing: true                  # UNCHANGED; flip to false is a deferred follow-up project

Step 2: Validate yamllint

Run: uvx --from yamllint yamllint ansible/roles/cilium/defaults/main.yml Expected: clean.

Step 3: Commit

Commit message: feat(cilium): flip Cilium to production values (PR4/5)

Task 8: Phase B — Run the ansible playbook to apply new values

Files: none (operational)

Step 1: Re-run cilium-install (now triggers helm upgrade with new values)

ansible-playbook \
  -i ansible/inventory/hosts.yml \
  ansible/k3s-playbook.yml \
  --tags cilium-install \
  -v

Expected: the Install or upgrade Cilium via Helm task reports changed=true (helm upgrade with new values). Subsequent tasks report ok (CiliumNodeConfig still applied, DaemonSet still Ready).

Step 2: Restart the Cilium DaemonSet so all agents read new config

kubectl -n kube-system rollout restart ds/cilium
kubectl -n kube-system rollout status ds/cilium --timeout=10m

Expected: rollout completes within 10 minutes.

Step 3: Verify the new config is in effect

# Verify production-mode values are now in cilium-config
EP=$(kubectl -n kube-system get cm cilium-config -o jsonpath='{.data.enable-policy}')
WC=$(kubectl -n kube-system get cm cilium-config -o jsonpath='{.data.write-cni-conf-when-ready}')
CC=$(kubectl -n kube-system get cm cilium-config -o jsonpath='{.data.custom-cni-conf}')

echo "enable-policy: ${EP}"              # Expected: default
echo "write-cni-conf-when-ready: ${WC}"  # Expected: empty (global) or /host/etc/cni/net.d/05-cilium.conflist
echo "custom-cni-conf: ${CC}"            # Expected: false

[ "${EP}" = "default" ] || { echo "FAIL: enable-policy not flipped to default"; exit 1; }
[ "${CC}" = "false" ] || [ -z "${CC}" ] || { echo "FAIL: custom-cni-conf still ${CC} (expected false or unset)"; exit 1; }

cilium status --wait
# Expected: OK

Task 9: Phase B — Delete CiliumNodeConfig and migration labels

Files: none (cluster state)

Step 1: Delete the CiliumNodeConfig (it’s no longer needed)

kubectl delete -n kube-system ciliumnodeconfig cilium-default

Step 2: Remove migration labels from all 8 nodes

kubectl label nodes -l io.cilium.migration/cilium-default=true io.cilium.migration/cilium-default-

Step 3: Verify

kubectl get nodes -l io.cilium.migration/cilium-default=true --no-headers | wc -l
# Expected: 0
kubectl get ciliumnodeconfig -A
# Expected: No resources found

Task 10: Phase B — Restore system-upgrade-controller

Files: none (operational)

Step 1: Scale system-upgrade-controller back to 1

kubectl scale -n system-upgrade deploy/system-upgrade-controller --replicas=1
kubectl wait --for=condition=Available --timeout=5m deploy/system-upgrade-controller -n system-upgrade

Expected: pod becomes Ready.

Task 11: Phase B — Verify policy enforcement

Files: none (verification)

Step 1: Verify CiliumNetworkPolicies now enforce

The bastion’s policies should now be enforcing. Use paired negative + positive tests to confirm.

# NEGATIVE TEST: Try a flow the bastion's CiliumNetworkPolicy explicitly drops.
# Bastion's allow-list permits port 22 (SSH) on 192.168.20.142, but NOT port 80.
# Pre-PR4 (policyEnforcementMode: never), the connection would attempt and fail
# with "connection refused" (port not listening) OR succeed. Post-PR4, Cilium
# drops the packet before TCP handshake completes — manifests as TIMEOUT.
set +e
OUT=$(kubectl exec -n cf-ssh-bastion deploy/cf-ssh-bastion -- \
  timeout 5 curl -sm 5 -v http://192.168.20.142:80/ 2>&1)
RC=$?
set -e
echo "Exit code: ${RC}"
echo "--- curl output (first 10 lines) ---"
echo "${OUT}" | head -10

# Verify the failure mode matches "policy drop" (timeout or ICMP unreachable), NOT "no listener" (refused):
if echo "${OUT}" | rg -q 'Operation timed out|Connection timed out|Failed to connect.*timed out|No route to host|Host is unreachable'; then
  echo "PASS: drop-consistent failure mode observed"
else
  echo "WARN: expected timeout/unreachable (policy drop), got different failure mode"
  echo "  Could be a relaxed-policy scenario OR an unexpected error string."
  echo "  Consulting Hubble for the authoritative verdict:"
  sleep 5
  DROPPED_COUNT=$(cilium hubble observe --since 1m --pod cf-ssh-bastion/cf-ssh-bastion --verdict DROPPED -o json 2>/dev/null \
    | jq -r '.flow | select(.destination.IP == "192.168.20.142") | .verdict' | wc -l)
  if [ "${DROPPED_COUNT}" -gt 0 ]; then
    echo "  Hubble shows ${DROPPED_COUNT} DROPPED verdict(s) to 192.168.20.142 — policy IS enforcing; the curl error was just a different presentation. PASS."
  else
    echo "FAIL: no DROPPED verdict in Hubble either. Policy may not be enforcing."
    exit 1
  fi
fi

# Hubble cross-check: verify Cilium recorded the DROPPED verdict
sleep 5
cilium hubble observe --since 1m --pod cf-ssh-bastion/cf-ssh-bastion --verdict DROPPED -o json \
  | jq -r '.flow | select(.destination.IP == "192.168.20.142") | "\(.time) DROPPED \(.destination.IP):\(.l4.TCP.destination_port // "?")"' \
  | head -3
# Expected: at least one DROPPED line to 192.168.20.142

# POSITIVE TEST: kube-apiserver healthz (allowed by bastion-allow-k8s-api / toEntities: kube-apiserver)
kubectl exec -n cf-ssh-bastion deploy/cf-ssh-bastion -- \
  timeout 3 curl -fsk https://kubernetes.default.svc/healthz
# Expected: prints "ok"

# POSITIVE TEST: kube-dns resolution (allowed by bastion-allow-kube-dns)
kubectl exec -n cf-ssh-bastion deploy/cf-ssh-bastion -- \
  timeout 3 getent hosts kube-dns.kube-system.svc.cluster.local
# Expected: an A record

# Hubble cross-check: verify Cilium recorded the FORWARDED verdict on positive paths
sleep 5
cilium hubble observe --since 1m --pod cf-ssh-bastion/cf-ssh-bastion --verdict FORWARDED -o json \
  | jq -r '.flow | "\(.time) FORWARDED \(.destination.namespace // "?")/\(.destination.pod_name // "?") :\(.l4.TCP.destination_port // .l4.UDP.destination_port // "?")"' \
  | head -5
# Expected: at least one FORWARDED line each to default/kubernetes-svc (or apiserver) AND kube-system/kube-dns

Task 12: Repository cleanup — remove Calico Ansible role

Files:

Delete: ansible/roles/calico/ (entire directory)
Step 1: Remove the directory

git rm -r ansible/roles/calico/

Step 2: Verify deletion

ls ansible/roles/ | rg -c calico
# Expected: 0

Step 3: Commit

Commit message: feat(cilium): remove ansible/roles/calico (PR4/5)

Task 13: Repository cleanup — remove Phase 5 from k3s-playbook.yml

Files:

Modify: ansible/k3s-playbook.yml
Step 1: Remove the Phase 5 (Calico) block

Open ansible/k3s-playbook.yml. Delete this block entirely:

# Phase 5: Install Calico CNI
- name: Install Calico CNI
  hosts: tp_cluster_controlplane[0]
  gather_facts: false
  roles:
    - role: calico
      tags:
        - k3s-calico
        - calico

Phase 5b (Cilium) stays. Optionally rename “Phase 5b” to “Phase 5” for cleanliness — recommended:

Change the comment line:

# Phase 5b: Install Cilium CNI (live migration mode through PR3; flips to
# production mode in PR4 when migration is complete and Calico is removed)

To:

# Phase 5: Install Cilium CNI

Step 2: Syntax check

Run: ansible-playbook -i ansible/inventory/hosts.yml ansible/k3s-playbook.yml --syntax-check Expected: clean.

Step 3: Commit

Commit message: feat(cilium): remove Phase 5 (Calico) from k3s-playbook (PR4/5)

Task 14: Update ansible/CLAUDE.md

Files:

Modify: ansible/CLAUDE.md
Step 1: Remove calico row from Roles Reference table

Find the row beginning with | `calico` | and DELETE it.

Step 2: Update the cilium row description (no longer “until PR4”)

Find the row beginning with | `cilium` | (current description: “CNI installation via Helm (live migration mode until PR4)”).

Replace its description column with: CNI installation via Helm.

Step 3: Update the Phases table

Find and DELETE the row beginning with | 5. Calico CNI |.

Rename the row beginning with | 5b. Cilium CNI (migration mode) | to start with | 5. Cilium CNI | (keep the tags column unchanged).

Step 4: rumdl check

Run: uvx --from rumdl rumdl check ansible/CLAUDE.md Expected: clean.

Step 5: Commit

Commit message: docs(cilium): remove Calico from Ansible CLAUDE.md (PR4/5)

Task 15: Update argocd/CLAUDE.md and Velero excludes

Files:

Modify: argocd/CLAUDE.md
Modify: argocd/app-configs/velero/backup-schedule.yaml
Step 1: Update argocd/CLAUDE.md Velero categories table

In argocd/CLAUDE.md, find the “Velero Backup Strategy” section’s table. The Networking row should no longer mention calico-system, calico-apiserver, tigera-operator. Edit the row to read:

| Networking | traefik, metallb | Stateless, config in Git |

(Cilium lives in kube-system which is already excluded as Kubernetes core, so no Cilium entry is needed.)

Step 2: Update argocd/app-configs/velero/backup-schedule.yaml

In argocd/app-configs/velero/backup-schedule.yaml, find the excludedNamespaces: lists (one for daily, one for weekly). Remove these three lines from each:

        - calico-system
        - calico-apiserver
        - tigera-operator

(Don’t remove kube-system — Cilium lives there; we explicitly exclude it for the same kube-system reason as before.)

Step 3: Verify kustomize render

kubectl --context fzymgc-house kustomize argocd/app-configs/velero/ > /tmp/velero-rendered.yaml
rg 'calico|tigera' /tmp/velero-rendered.yaml
# Expected: empty (no calico/tigera references)
rm /tmp/velero-rendered.yaml

Step 4: rumdl + yamllint

Run:

uvx --from rumdl rumdl check argocd/CLAUDE.md
uvx --from yamllint yamllint argocd/app-configs/velero/backup-schedule.yaml

Both: clean.

Step 5: Commit

Commit message: feat(velero): remove Calico namespaces from excludes (PR4/5)

Task 16: Update top-level docs

Files:

Modify: README.md, docs/index.md, docs/architecture/overview.md, docs/architecture/index.md, docs/reference/network.md, docs/reference/technologies.md, docs/reference/services.md
Step 1: For each file, find Calico mentions and replace with Cilium-equivalent

Use rg -li calico to find candidates:

rg -li 'calico' README.md docs/

For each file, the typical changes are:

“Calico” → “Cilium”
“Tigera operator” → “Cilium Helm chart” or just remove (no operator)
“Whisker/Goldmane” (Calico observability) → “Hubble”
Pod CIDR 10.42.0.0/16 → 10.245.0.0/16
VXLANCrossSubnet → “VXLAN tunnel”

Make the edits file by file. Read each file’s full context first; don’t blindly find/replace.

Step 2: docs/reference/network.md — also remove merlin mentions

Find and remove the row in the namespaces-using-Calico-policy table for merlin (merlin was removed in PR #908; the row is stale).

Find and remove the “Calico Network Policies” section’s mention of merlin if present.

Step 3: rumdl all modified docs

uvx --from rumdl rumdl check README.md docs/

Expected: clean.

Step 4: Commit

Commit message: docs(cilium): update Calico references to Cilium (PR4/5)

Task 17: Delete stale merlin doc

Files:

Delete: docs/operations/merlin.md
Step 1: Remove the file

git rm docs/operations/merlin.md

Step 2: Commit

Commit message: docs: delete stale merlin operations doc (removed in PR #908) (PR4/5)

Task 18: Create docs/operations/cilium.md (evergreen ops runbook)

Files:

Create: docs/operations/cilium.md
Step 1: Create the evergreen operations runbook

# Cilium Operations Runbook

Evergreen operational guide for the cluster's Cilium 1.19.3 CNI deployment.
For the one-time Calico→Cilium migration log, see
[`cilium-migration.md`](/operations/cilium-migration/).

## Deployment

- **Installed by:** `ansible/roles/cilium` (Phase 5 of `ansible/k3s-playbook.yml`)
- **Chart:** `cilium/cilium` from `https://helm.cilium.io/`
- **Version:** 1.19.3 (pinned in `roles/cilium/defaults/main.yml`)
- **Namespace:** `kube-system`
- **Pod CIDR:** `10.245.0.0/16` (IPv4) + `fd00:10:245::/48` (IPv6)
- **Service CIDR:** `10.43.0.0/16` (k3s default; unchanged from Calico era)
- **Routing:** tunnel mode, VXLAN protocol, port 8473
- **kube-proxy:** still running (Cilium `kubeProxyReplacement: "false"`); replacement is a follow-up project
- **Gateway API:** v1.4.1 Standard CRDs installed; Cilium gateway controller enabled; GatewayClass is `cilium`. No HTTPRoutes/Gateways exist yet.

## k3s --cluster-cidr note

`k3s-common/defaults/main.yml` still references `10.42.0.0/16` as the k3s
cluster CIDR. This is **intentionally retained** post-Calico-removal:

- Cilium owns IPAM via `ipam.mode: cluster-pool` and writes pod IPs from `10.245.0.0/16` to `CiliumNode.spec.ipam.podCIDRs`.
- k3s passes `--cluster-cidr=10.42.0.0/16` to kube-controller-manager-equivalent, which populates `Node.spec.podCIDR` from that range.
- Cilium IGNORES `Node.spec.podCIDR` when in cluster-pool mode; the two coexist harmlessly.
- Changing the k3s value would require a server restart on every control-plane node — out of scope for any non-emergency operation.

## Common commands

### cilium-cli (workstation)

```bash
# Cluster health
cilium status --wait
cilium connectivity test     # full e2e suite — takes ~5 minutes
cilium connectivity test --hubble=false  # quick smoke

# Hubble flows
cilium hubble observe --last 100
cilium hubble observe --since 5m --verdict DROPPED  # see policy drops
cilium hubble observe --pod NAMESPACE/POD-NAME
cilium hubble ui            # opens browser to internal UI via port-forward
```

### kubectl

```bash
# Per-node CNI state
kubectl get ciliumnodes
kubectl get ciliumendpoints -A | head -20

# Policy resources
kubectl get ciliumnetworkpolicies -A
kubectl get ciliumclusterwidenetworkpolicies

# Agent diagnostics
kubectl exec -n kube-system ds/cilium -- cilium-dbg status
kubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint list
```

## Hubble UI access

Hubble UI is internal-only (no IngressRoute by design).

```bash
cilium hubble ui  # auto-port-forwards and opens browser
# Or manually:
kubectl port-forward -n kube-system svc/hubble-ui 12000:80
```

## Pod CIDR allocation per node

Each node gets a `/24` from the cluster pool. View allocation:

```bash
kubectl get ciliumnodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}'
```

## NetworkPolicy enforcement

Cluster-wide `policyEnforcementMode: default` — pods without matching CiliumNetworkPolicy are allowed by default; pods with matching policies are restricted to what the policies permit.

Currently enforced policies live in:

- `argocd/app-configs/cf-ssh-bastion/cilium-network-policy.yaml`

## Known follow-ups

- **`bpf.hostLegacyRouting: true` → `false`** — currently `true` because flipping requires a scheduled low-traffic window (upstream warns of brief packet loss). Plan a future maintenance window.
- **kube-proxy replacement (`kubeProxyReplacement: "true"`)** — separate follow-up project; interacts with MetalLB and requires k3s `--disable=kube-proxy`.
- **Gateway API route migration** — Cilium gateway controller is installed but unused; future project to migrate Traefik IngressRoutes to HTTPRoutes (much bigger scope).
- **Native routing (`autoDirectNodeRoutes: true`)** — viable since all nodes are on the same L2 segment; would eliminate VXLAN encapsulation overhead. Future single-PR optimization.

## References

- [Cilium 1.19 docs](https://docs.cilium.io/en/v1.19/)
- [Cilium 1.19 Helm values reference](https://docs.cilium.io/en/v1.19/helm-reference/)
- [Hubble docs](https://docs.cilium.io/en/v1.19/observability/hubble/)
- Repo: `ansible/roles/cilium/` (role source)
- Repo: `docs/engineering/specs/2026-05-10-calico-to-cilium-design.md` (original design)
- Repo: `docs/operations/cilium-migration.md` (one-time migration log)

Step 2: rumdl check

Run: uvx --from rumdl rumdl check docs/operations/cilium.md Expected: clean.

Step 3: Commit

Commit message: docs(cilium): add evergreen operations runbook (PR4/5)

Task 19: Finalize cilium-migration.md with PR4 entry

Files:

Modify: docs/operations/cilium-migration.md
Step 1: Append the PR4 section

## PR4 — Decommission Calico

- **Date:** <fill in>
- **Operator:** <fill in>
- **Pre-removal Velero backup:** `pre-calico-removal-<timestamp>`
- **Phase A (stop Calico):**
  - tigera-operator scaled to 0 ✓
  - Whisker / Goldmane / APIServer / Installation deleted ✓
  - Namespaces calico-system / calico-apiserver / tigera-operator deleted ✓
  - 22 projectcalico.org CRDs + 9 operator.tigera.io CRDs deleted ✓
  - 11 ClusterRoles + matched ClusterRoleBindings deleted ✓
  - 0 admission webhooks (confirmed) ✓
- **Phase B (Cilium cutover):**
  - Values flipped: cni.customConf=false, cni.uninstall=true, policyEnforcementMode=default, operator.unmanagedPodWatcher.restart=true
  - bpf.hostLegacyRouting: STAYS true (deferred follow-up)
  - helm upgrade + rollout restart ds/cilium completed ✓
  - CiliumNodeConfig cilium-default deleted ✓
  - Migration labels removed from all 8 nodes ✓
  - system-upgrade-controller restored to 1 replica ✓
- **Repo changes:**
  - ansible/roles/calico/ deleted
  - k3s-playbook.yml Phase 5 (Calico) removed; Phase 5b renamed to Phase 5
  - ansible/CLAUDE.md, argocd/CLAUDE.md, Velero backup-schedule.yaml updated
  - docs/operations/merlin.md deleted (stale)
  - docs/operations/cilium.md created (evergreen runbook)
  - Top-level docs updated to reference Cilium / 10.245 / VXLAN / Hubble
- **Verification:**
  - CiliumNetworkPolicy enforcement confirmed via test drop
  - cilium status --wait OK
  - No Calico CRDs / ClusterRoles / Namespaces remain

## Project complete

The Calico → Cilium migration is complete. PR5 (the small follow-up) deletes the obsolete `cf-ssh-bastion/calico-network-policy.yaml` file (which is now a silent no-op since the Calico CRD is gone) and the kustomization entry referencing it.

Follow-up projects (filed as beads):

- bpf.hostLegacyRouting flip (scheduled maintenance window)
- kube-proxy replacement
- Hubble external exposure (optional)
- Gateway API route migration off Traefik
- Native routing (autoDirectNodeRoutes) optimization

Step 2: rumdl check

Run: uvx --from rumdl rumdl check docs/operations/cilium-migration.md Expected: clean.

Step 3: Commit

Commit message: docs(cilium): finalize migration log with PR4 entry (PR4/5)

Task 20: Update mkdocs.yml nav

Files:

Modify: mkdocs.yml
Step 1: Inspect current Operations nav

Run: rg -n -A 30 '^nav:' mkdocs.yml | head -60

Note where Operations entries live.

Step 2: Add the two new operations docs and remove merlin

Under the Operations: section (or equivalent):

ADD: - Cilium: operations/cilium.md
ADD: - Cilium Migration Log: operations/cilium-migration.md
REMOVE: any line referencing operations/merlin.md (left over from the PR #908 cleanup gap)
Step 3: Render the docs site locally to verify

Run: uvx --with mkdocs-material mkdocs build --strict 2>&1 | tail -10

Expected: build succeeds with no “missing reference” warnings. The --strict flag fails on broken links — this catches both the merlin nav-entry-with-deleted-doc AND any other broken links.

Step 4: rumdl + yamllint

Run: uvx --from yamllint yamllint mkdocs.yml Expected: clean.

Step 5: Commit

Commit message: docs: update mkdocs nav for Cilium docs (PR4/5)

Task 21: Final verification + open PR

Files: none (PR creation)

Step 1: Cluster-state final check

# 1. No Calico CRDs
kubectl get crd | rg -i 'calico|tigera' | wc -l
# Expected: 0

# 2. No Calico namespaces
kubectl get ns | rg -i 'calico|tigera' | wc -l
# Expected: 0

# 3. No Calico ClusterRoles
kubectl get clusterrole,clusterrolebinding -o name | rg -i 'calico|tigera' | wc -l
# Expected: 0

# 4. Cilium enforces policies
kubectl get cm -n kube-system cilium-config -o jsonpath='{.data.enable-policy}'; echo ""
# Expected: default

# 5. CiliumNodeConfig is gone
kubectl get ciliumnodeconfig -A
# Expected: No resources found

# 6. cilium status
cilium status --wait

# 7. system-upgrade-controller restored
kubectl get deploy -n system-upgrade system-upgrade-controller -o jsonpath='{.status.readyReplicas}'; echo ""
# Expected: 1

Step 2: Repo lint

lefthook run pre-commit

Expected: clean.

Step 3: Open the PR

git push -u origin feat/cilium-migration  # or PR4-specific branch
gh pr create \
  --title "feat(cilium): decommission Calico + production cutover (PR4/5)" \
  --body "$(cat <<'EOF'
## Summary

The project's point of no return. Decommissions Calico entirely (operator, CRDs, ClusterRoles, namespaces, Ansible role) and flips Cilium from migration mode to production mode (policyEnforcementMode=default, cni.customConf=false, etc.).

## Cluster impact

- Calico CRDs, ClusterRoles, namespaces removed
- Cilium values flipped; CiliumNetworkPolicies now enforce
- CiliumNodeConfig cilium-default deleted; migration labels removed
- system-upgrade-controller restored to 1 replica

## bpf.hostLegacyRouting decision

Kept at `true` permanently in this PR. Flipping to `false` requires a scheduled maintenance window (upstream warns of brief packet loss). Filed as follow-up beads issue.

## Repo cleanup

- Deleted `ansible/roles/calico/`
- Removed Phase 5 (Calico) from `k3s-playbook.yml`
- Updated `ansible/CLAUDE.md`, `argocd/CLAUDE.md`, `argocd/app-configs/velero/backup-schedule.yaml`
- Deleted stale `docs/operations/merlin.md`
- Created `docs/operations/cilium.md` (evergreen runbook)
- Finalized `docs/operations/cilium-migration.md` with PR4 entry
- Updated top-level docs and `mkdocs.yml` nav

## Spec / prior PR

- Spec: `docs/engineering/specs/2026-05-10-calico-to-cilium-design.md`
- PR3: #<PR3-number>
- Next: PR5 (delete the obsolete calico-network-policy.yaml file)

## Test plan

- [x] Phase A merge gate passed (no pods on Calico CIDR, all 8 CiliumNodes registered)
- [x] All Calico/Tigera CRDs, ClusterRoles, namespaces deleted
- [x] Helm upgrade + rollout restart completed
- [x] CiliumNetworkPolicy enforcement verified (test drop)
- [x] system-upgrade-controller restored
- [x] mkdocs --strict builds without broken links
- [x] cilium status --wait OK

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Self-Review Checklist

Spec coverage — Phase A (Tasks 2-6: scale operator, delete CRs, delete namespaces, delete CRDs, delete RBAC) and Phase B (Tasks 7-11: flip values, helm upgrade, delete CiliumNodeConfig, restore system-upgrade, verify enforcement). Repo cleanup (Tasks 12-20) covers role deletion, playbook trim, CLAUDE.md updates, Velero excludes, doc updates, merlin deletion, cilium.md creation, migration-log finalization, mkdocs nav.
Placeholder scan — <fill in> markers in migration log are for operator-recorded values. Every command is explicit.
Type/symbol consistency — variable names match across Tasks 7 and 8; helm chart name, namespace, version all consistent.
No invented references — every namespace name, CRD prefix, ClusterRole pattern, and kubectl command refers to real resources verified during spec authoring.

Notes for the implementer

Phase A goes BEFORE Phase B, no exceptions. Flipping cni.customConf: false while calico-node pods are still running evicts their CNI conf files; behavior is undefined. The plan’s ordering (Task 2→6 = Phase A, Task 7→11 = Phase B) preserves this.
The Tigera operator scale-to-0 (Task 2) MUST complete before deleting Installation (Task 3 step 4). Otherwise the operator races to recreate calico-system pods you’re trying to delete.
The merge gate (Task 1) is a HARD gate. If ANY check fails, halt and resolve before continuing. PR4 is the point of no return.
cilium status --wait mid-PR4 may briefly report degraded as Helm upgrade rolls agents. This is expected. Wait for the rollout to complete before re-running status.
The bastion’s calico-network-policy.yaml file is intentionally left in tree during PR4. After Phase A deletes the CRD, the file is a silent no-op. PR5 removes it.

Execution handoff

Plan complete and saved to docs/engineering/plans/2026-05-11-cilium-pr4-decommission-calico.md. Two execution options:

Subagent-Driven — fresh subagent per task. Strongly recommended for the document-editing tasks (12-20); operator should be in-the-loop for cluster tasks (1-11).
Inline Execution (recommended for cluster tasks) — operational nature requires continuous oversight. Pair this plan with the spec’s “Order of operations” subsection for cross-reference.

Which approach?