Cilium PR3 — Per-Node Migration Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Migrate all 8 cluster nodes from Calico-managed networking to Cilium-managed networking via upstream’s per-node procedure. Each node is independently labeled, drained, agent-restarted, rebooted, and verified. After all 8 nodes pass verification, the PR is opened with the migration runbook as its repository artifact.

Architecture: Iterative procedure with two stability gates: (1) after node 1, validate IPv6 dual-stack works end-to-end (upstream marks family-changes as untested); (2) after all 5 workers, soak 24h before touching control plane. Control-plane nodes migrate sequentially (alpha-3 → alpha-2 → alpha-1) to preserve etcd quorum. The repository artifact of PR3 is docs/operations/cilium-migration.md (started in PR2) with one log entry appended per node.

Tech Stack: kubectl, cilium-cli, ansible (for ansible.builtin.reboot), Velero, kubernetes MCP for verification queries, beads for per-node task tracking.

Reference spec: docs/engineering/specs/2026-05-10-calico-to-cilium-design.md (commit fe05a87c).

---

File Structure

Files modified:

• docs/operations/cilium-migration.md — append one log entry per node + stability-gate entries (8+ appends total).

Cluster state changes:

• 8 nodes labeled io.cilium.migration/cilium-default=true (cumulative — one label flip per node)
• Pod IPs cluster-wide shift from 10.42.0.0/16 (Calico) to 10.245.0.0/16 (Cilium) as nodes migrate
• DaemonSet pod sandboxes rebuilt via reboot on each migrated node
• system-upgrade-controller scaled to 0 throughout PR3 (restored in PR4)

No code files modified in this PR. Tracking artifacts live in beads.

---

Tasks

Task 1: Pre-flight (executed once before the first node)

Files: none (read-only checks + state changes that are restored later)

• Step 1: Take a fresh Velero backup if PR2’s is older than 24h

LATEST_BACKUP=$(velero backup get -o json | jq -r '.items[] | select(.metadata.name | startswith("pre-cilium-install-")) | .metadata.creationTimestamp' | sort -r | head -1)
echo "Latest pre-PR2-install backup: ${LATEST_BACKUP}"

# If older than 24h, take a fresh one:
BACKUP_NAME="pre-pr3-migration-$(date +%Y%m%d-%H%M%S)"
velero backup create "${BACKUP_NAME}" --wait
echo "PR3 pre-flight backup: ${BACKUP_NAME}"

Expected: backup completes Successful.

• Step 2: Capture pre-PR3 cluster state for audit

kubectl --context fzymgc-house cluster-info dump --output-directory=/tmp/pre-pr3-cluster-state --all-namespaces
tar czf /tmp/pre-pr3-cluster-state.tar.gz -C /tmp pre-pr3-cluster-state
rm -rf /tmp/pre-pr3-cluster-state
echo "Saved /tmp/pre-pr3-cluster-state.tar.gz ($(du -h /tmp/pre-pr3-cluster-state.tar.gz | cut -f1))"

Retain this archive for the duration of PR3+PR4 in case post-migration audit is needed.

• Step 3: Enumerate PodDisruptionBudgets and identify stall risks

kubectl get pdb -A -o json \
  | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name) minAvailable=\(.spec.minAvailable // "-") maxUnavailable=\(.spec.maxUnavailable // "-")"' \
  | tee /tmp/pre-pr3-pdb-inventory.txt

Review the output:

• PDBs with minAvailable: 1 AND single-replica workloads → drain will block. Most likely candidates: Longhorn replicas (cluster-aware; usually OK), CNPG primary (1 of 3 instances — OK), Dolt (StatefulSet), Temporal (StatefulSet).
• Document the inventory in the migration log (Task 11 below).

Mitigation if any node’s drain stalls > 10 min during PR3: fall back to kubectl drain --disable-eviction (uses delete API, bypasses PDBs). Document the reason in the migration log.

• Step 4: Scale system-upgrade-controller to 0

kubectl scale -n system-upgrade deploy/system-upgrade-controller --replicas=0
kubectl wait --for=delete pod -n system-upgrade -l app.kubernetes.io/name=system-upgrade-controller --timeout=2m
kubectl get deploy -n system-upgrade system-upgrade-controller -o jsonpath='{.spec.replicas}'
echo ""

Expected: deployment shows 0 replicas; no pod exists. PR4 restores this to 1.

• Step 5: Verify CiliumNodeConfig is healthy

kubectl get ciliumnodeconfig -n kube-system cilium-default -o yaml | head -20

Expected: resource exists, nodeSelector.matchLabels is io.cilium.migration/cilium-default: "true", defaults.cni-exclusive: "true".

• Step 6: Verify pre-PR3 cluster state

# 1. No nodes are labeled with the migration label
kubectl get nodes -l 'io.cilium.migration/cilium-default=true' --no-headers | wc -l
# Expected: 0

# 2. policyEnforcementMode is "never" (confirms PR2 install, not PR4 cutover)
kubectl -n kube-system get cm cilium-config -o jsonpath='{.data.enable-policy}'
echo ""
# Expected: never

# 3. No pods are on Cilium CIDR (confirms PR3 hasn't started yet)
kubectl get pods -A -o json \
  | jq '[.items[] | select(.spec.hostNetwork != true) | select(.status.podIP // "" | startswith("10.245."))] | length'
# Expected: 0

# All three together confirm this is genuinely pre-PR3 state, not post-PR4 cleanup.

• Step 7: Verify operator workstation prerequisites

cilium version --client    # cilium-cli installed
kubectl version --client   # kubectl available
ansible --version          # ansible CLI installed (for ansible.builtin.reboot)
helm version --short       # helm CLI installed (for any manual helm queries)

All four should report versions.

• Step 8: Verify SSH connectivity (used by ansible.builtin.reboot)

ansible -i ansible/inventory/hosts.yml -m ping all --limit tp_cluster_nodes

Expected: all 8 nodes return pong. If any fails, fix before proceeding.

• Step 9: Append pre-flight entry to migration log

Append to docs/operations/cilium-migration.md:

## PR3 — Per-node migration

### Pre-flight

- **Date:** <fill in>
- **Operator:** <fill in>
- **Pre-flight Velero backup:** `pre-pr3-migration-<timestamp>`
- **Cluster state archive:** `/tmp/pre-pr3-cluster-state.tar.gz` (retain through PR4)
- **system-upgrade-controller:** scaled to 0 replicas (restored in PR4)
- **PDB inventory:**

  ```text
  <paste contents of /tmp/pre-pr3-pdb-inventory.txt>
  ```

- **Migration order:**
  1. tpi-beta-1 (worker)
  2. **First-node validation gate** — IPv6 + stability soak
  3. tpi-beta-2, tpi-beta-3 (workers)
  4. tpi-beta-4, tpi-alpha-4 (workers; bastion's host node migrates here)
  5. **All-workers stability gate** — 24h soak
  6. tpi-alpha-3, tpi-alpha-2, tpi-alpha-1 (control plane reverse order)

Commit message: docs(cilium): record PR3 pre-flight (PR3/5).

---

Task 2: Create beads epic + 8 per-node child issues

Files: none (beads tracking)

• Step 1: Create the parent epic

EPIC_ID=$(bd create \
  --title "Cilium PR3: per-node migration (8 nodes)" \
  --description "Tracks the iterative per-node migration from Calico to Cilium per docs/engineering/specs/2026-05-10-calico-to-cilium-design.md and docs/engineering/plans/2026-05-11-cilium-pr3-per-node-migration.md." \
  --type epic \
  --priority 1 \
  --json | jq -r '.id')
echo "Epic: ${EPIC_ID}"

• Step 2: Create one child issue per node, in migration order

NODES=(tpi-beta-1 tpi-beta-2 tpi-beta-3 tpi-beta-4 tpi-alpha-4 tpi-alpha-3 tpi-alpha-2 tpi-alpha-1)
for node in "${NODES[@]}"; do
  CHILD_ID=$(bd create \
    --title "Migrate ${node} from Calico to Cilium" \
    --description "Execute per-node procedure from docs/engineering/plans/2026-05-11-cilium-pr3-per-node-migration.md Task 3 for node ${node}." \
    --type task \
    --priority 1 \
    --parent "${EPIC_ID}" \
    --json | jq -r '.id')
  echo "  Child ${node}: ${CHILD_ID}"
done

Expected: 8 child issues created under the epic. Use bd ready to find the next one to work.

• Step 3: Add dependencies so beads tracks the order

# Each subsequent node depends on the prior one
# (Use bd dep add or similar - check bd help)
bd ready  # confirm tpi-beta-1 is the only one initially ready

---

Task 3: Per-node migration procedure (REPEAT for each of the 8 nodes)

This task is the same for every node. The ${NODE} variable is set per iteration.

Files: none (cluster state operations)

For each node in order: tpi-beta-1 → tpi-beta-2 → tpi-beta-3 → tpi-beta-4 → tpi-alpha-4 → tpi-alpha-3 → tpi-alpha-2 → tpi-alpha-1.

(After tpi-beta-1, run Task 4 (first-node validation gate) before continuing. After the 5 workers, run Task 5 (all-workers stability gate) before touching the CP nodes.)

• Step 1: Set the node variable and claim the beads issue

export NODE="tpi-beta-1"  # change for each iteration
bd update <child-id-for-this-node> --claim
bd update <child-id-for-this-node> --status in_progress

• Step 2: Cordon and drain the node

kubectl cordon ${NODE}
kubectl drain --ignore-daemonsets --delete-emptydir-data ${NODE}

Expected: drain completes within 5 minutes. If it stalls (likely candidates: a PDB blocking eviction of a single-replica StatefulSet pod), wait up to 10 minutes total. Then:

• Investigate which pod is blocking: kubectl get events -A --field-selector type=Warning | rg -i 'PDB\|disruption' | head.
• Fallback: kubectl drain --ignore-daemonsets --delete-emptydir-data --disable-eviction ${NODE} (uses delete API; document in log).
• Step 3: Apply the migration label

kubectl label node ${NODE} --overwrite "io.cilium.migration/cilium-default=true"

• Step 4: Restart the Cilium agent on this node

kubectl -n kube-system delete pod \
  --field-selector spec.nodeName=${NODE} \
  -l k8s-app=cilium

• Step 5: Wait for the new Cilium agent pod to be Ready

kubectl -n kube-system rollout status ds/cilium -w --timeout=5m

Expected: rollout completes within 5 minutes.

If the rollout times out (error: timed out waiting for the condition):

• Inspect the Cilium agent pod: kubectl describe pod -n kube-system -l k8s-app=cilium --field-selector spec.nodeName=${NODE} for the CrashLoopBackOff/ImagePullBackOff/Init pending reason.
• Common causes: image pull on slow link (wait longer), CiliumNodeConfig label-propagation race (re-run step 4 to delete the agent pod again), node still has Calico’s CNI conf overriding (manually kubectl debug node/${NODE} -- chroot /host ls /etc/cni/net.d/).
• Do NOT proceed to step 6 until the rollout succeeds.

---

• Step 6: Verify the new CNI conf was written via kubectl debug

kubectl debug node/${NODE} --image=busybox -- chroot /host ls -1 /etc/cni/net.d/

Expected output (lexical order): 05-cilium.conflist BEFORE any 10-calico.conflist line. If 05-cilium.conflist is missing, the agent didn’t pick up the CiliumNodeConfig — investigate before proceeding.

• Step 7: Reboot the node via ansible.builtin.reboot

ansible -i ansible/inventory/hosts.yml \
  --become \
  --limit ${NODE} \
  -m ansible.builtin.reboot \
  -a 'reboot_timeout=600 post_reboot_delay=30' \
  all

Expected: ansible reports SUCCESS for the node within ~3-5 minutes (TPI compute modules boot slower than bare-metal). Don’t worry about the “60-90s” estimate from the spec — reboot_timeout=600 gives 10 minutes.

• Step 8: Wait for the node to be Ready in k8s

kubectl wait --for=condition=Ready --timeout=10m node/${NODE}

Expected: node/<NODE> condition met.

• Step 9: cilium-cli verification post-reboot

cilium status --wait

Expected: OK. May show “Cluster Pods: / managed by Cilium” with the count incrementing as more nodes migrate.

• Step 10: Uncordon the node

kubectl uncordon ${NODE}

• Step 11: Smoke test — verify-network pod

Note: ghcr.io/nicolaka/netshoot:v0.8 is pinned; netshoot ships rg and curl in this version. If a future version drops those tools, the smoke test will fail confusingly — verify image contents before bumping.

kubectl -n kube-system run --attach --rm --restart=Never verify-network \
  --overrides='{"spec": {"nodeName": "'${NODE}'", "tolerations": [{"operator": "Exists"}]}}' \
  --image=ghcr.io/nicolaka/netshoot:v0.8 -- \
  /bin/bash -c 'ip -br addr | rg "10\\.245\\." \
  && curl -fsk https://$KUBERNETES_SERVICE_HOST/healthz \
  && curl -fs https://api.github.com -o /dev/null \
  && echo OK'

Expected: stdout contains an IP from 10.245.0.0/16, then ok from the apiserver healthz, then OK. Exit 0.

The api.github.com check validates DNS resolution, default route, NAT, and TLS — the full user-visible workload path.

• Step 12: Cross-CNI smoke test (skip on tpi-alpha-1 — no Calico peer remains)

Find a still-Calico pod (one whose IP is from 10.42.0.0/16):

CALICO_POD=$(kubectl get pods -A -o json \
  | jq -r '[.items[] | select(.spec.hostNetwork != true) | select(.status.podIP // "" | startswith("10.42.")) | {ns: .metadata.namespace, name: .metadata.name, ip: .status.podIP}][0]')
echo "${CALICO_POD}"
CALICO_NS=$(echo "${CALICO_POD}" | jq -r '.ns')
CALICO_NAME=$(echo "${CALICO_POD}" | jq -r '.name')

Find a freshly-Cilium pod on this node:

CILIUM_POD=$(kubectl get pods -A --field-selector spec.nodeName=${NODE} -o json \
  | jq -r '[.items[] | select(.spec.hostNetwork != true) | select(.status.podIP // "" | startswith("10.245.")) | {ns: .metadata.namespace, name: .metadata.name, ip: .status.podIP}][0]')
echo "${CILIUM_POD}"

Ping between them (use kubectl exec; adjust ports based on what’s running):

# Cilium pod -> Calico pod (TCP smoke, e.g., to a pod on port 80)
kubectl exec -n <CILIUM_NS> <CILIUM_NAME> -- timeout 3 nc -zv <CALICO_IP> <PORT> 2>&1 | head

# Calico pod -> Cilium pod
kubectl exec -n ${CALICO_NS} ${CALICO_NAME} -- timeout 3 nc -zv <CILIUM_IP> <PORT> 2>&1 | head

Expected: both directions succeed. If either fails, this is a real problem — investigate before next node.

• Step 13: Verify all pods on ${NODE} are Cilium-managed (no straggler Calico pods)

kubectl get pods -A -o wide --field-selector spec.nodeName=${NODE} --no-headers \
  | rg -v '\s(192\.168\.20\.|<none>|10\.245\.|fd00:10:245)'

Expected: empty output. Any line indicates a still-Calico-IP pod — investigate (likely a DS pod whose sandbox didn’t rebuild; force-delete: kubectl delete pod -n <NS> <NAME>).

• Step 14: Append per-node entry to migration log

Append to docs/operations/cilium-migration.md:

### ${NODE} — migrated <YYYY-MM-DD HH:MM UTC>

- Drain duration: <X> min (`--disable-eviction` used: yes/no — if yes, list PDBs bypassed: <ns/name list> and resulting brief unavailability: <ns/workload>)
- Reboot duration: <X> min
- Verify-network pod IP: 10.245.x.y / 10.245.<sample>
- API server healthz: OK
- Cross-CNI smoke (Cilium → Calico): OK / N/A (final node)
- Cross-CNI smoke (Calico → Cilium): OK / N/A (final node)
- Pods on node post-migration: <total>, all on Cilium CIDR
- Agent restart count post-migration: 0
- Anomalies: <none / describe>

Commit message: docs(cilium): record ${NODE} migration (PR3/5).

• Step 15: Close the beads issue for this node

bd close <child-id-for-this-node>
bd ready  # confirms the next node's issue is now ready

---

Task 4: First-node validation gate (after tpi-beta-1, before tpi-beta-2)

Files: none (verification + log entry)

• Step 1: IPv6 dual-stack validation

Schedule a verify pod on tpi-beta-1 and verify it has an IPv6 address from fd00:10:245::/48:

kubectl -n kube-system run --attach --rm --restart=Never verify-ipv6 \
  --overrides='{"spec": {"nodeName": "tpi-beta-1", "tolerations": [{"operator": "Exists"}]}}' \
  --image=ghcr.io/nicolaka/netshoot:v0.8 -- \
  /bin/bash -c 'ip -6 -br addr && echo "---" && ip -6 route'

Expected: at least one address from fd00:10:245::/48 on a Cilium-managed interface.

Test pod-to-pod IPv6 connectivity (4-check sequence — all must pass):

# 1. Find a Calico-side v6 pod (excludes Cilium v6 range fd00:10:245)
CALICO_V6_POD=$(kubectl get pods -A -o json \
  | jq -c '[.items[]
      | select(.spec.hostNetwork != true)
      | select(.status.podIPs // [] | any(.ip | startswith("fd00:") and (startswith("fd00:10:245") | not)))
      ] | .[0] // empty')
test -n "${CALICO_V6_POD}" \
  || { echo "FAIL: no Calico-side v6 pod found; cannot run v6 reverse-direction validation"; exit 1; }
CALICO_NS=$(echo "${CALICO_V6_POD}" | jq -r '.metadata.namespace')
CALICO_NAME=$(echo "${CALICO_V6_POD}" | jq -r '.metadata.name')
CALICO_V6_IP=$(echo "${CALICO_V6_POD}" | jq -r '.status.podIPs[] | select(.ip | startswith("fd00:") and (startswith("fd00:10:245") | not)) | .ip')
echo "Calico-side v6 peer: ${CALICO_NS}/${CALICO_NAME} ${CALICO_V6_IP}"

# 2. Cilium → Calico v6 ping
kubectl -n kube-system run --attach --rm --restart=Never verify-v6-out \
  --overrides='{"spec": {"nodeName": "tpi-beta-1", "tolerations": [{"operator": "Exists"}]}}' \
  --image=ghcr.io/nicolaka/netshoot:v0.8 -- \
  /bin/bash -c "ping6 -c 3 ${CALICO_V6_IP}"
# Expected: 3 successful pings

# 3. Calico → Cilium v6 ping (reverse direction)
#    Get the Cilium-side pod v6 IP
CILIUM_V6_IP=$(kubectl get pods -A --field-selector spec.nodeName=tpi-beta-1 -o json \
  | jq -r '[.items[] | select(.spec.hostNetwork != true) | select(.status.podIPs[]?.ip // "" | startswith("fd00:10:245"))] | .[0].status.podIPs[] | select(.ip | startswith("fd00:10:245")) | .ip' | head -1)
echo "Cilium-side v6: ${CILIUM_V6_IP}"
kubectl exec -n "${CALICO_NS}" "${CALICO_NAME}" -- ping6 -c 3 "${CILIUM_V6_IP}"
# Expected: 3 successful pings

# 4. v6 DNS resolution from Cilium side
kubectl -n kube-system run --attach --rm --restart=Never verify-v6-dns \
  --overrides='{"spec": {"nodeName": "tpi-beta-1", "tolerations": [{"operator": "Exists"}]}}' \
  --image=ghcr.io/nicolaka/netshoot:v0.8 -- \
  /bin/bash -c "getent ahostsv6 kubernetes.default.svc.cluster.local | head -3"
# Expected: AAAA records returned

# 5. v6 to kube-apiserver via the v6 Service ClusterIP
V6_API=$(kubectl get svc -n default kubernetes -o jsonpath='{.spec.clusterIPs[*]}' | tr ' ' '\n' | rg ':' | head -1)
test -n "${V6_API}" || { echo "FAIL: no v6 ClusterIP found on default/kubernetes service. Expected dual-stack v4+v6 per spec; check kube-apiserver --service-cluster-ip-range includes both v4 and v6 CIDRs."; exit 1; }
echo "v6 kube-apiserver: ${V6_API}"

V6_API_OUT=$(kubectl -n kube-system run --attach --rm --restart=Never verify-v6-api \
  --overrides='{"spec": {"nodeName": "tpi-beta-1", "tolerations": [{"operator": "Exists"}]}}' \
  --image=ghcr.io/nicolaka/netshoot:v0.8 -- \
  /bin/bash -c "curl -fsk -6 https://[${V6_API}]/healthz && echo v6-API-OK" 2>&1)
echo "${V6_API_OUT}"
echo "${V6_API_OUT}" | rg -q 'v6-API-OK' || { echo "FAIL: v6 kube-apiserver check did not print v6-API-OK"; exit 1; }

If ANY of these checks fails, halt and execute the IPv6 fallback runbook below.

• Step 2: If IPv6 fails — execute IPv6 fallback (see spec)

WARNING: This fallback DISABLES IPv6 cluster-wide. Any workload depending on IPv6 (e.g., router-hosts AAAA records, Tailscale-internal v6, services that bind to :: and expect v6 clients) will stop working until the follow-up “Re-enable Cilium IPv6 dual-stack” project completes. File the follow-up beads issue BEFORE executing this fallback.

If the IPv6 validation fails:

1. File a follow-up beads issue: “Re-enable Cilium IPv6 dual-stack post-migration.”

2. Update ansible/roles/cilium/templates/values.yaml.j2:

   • Change ipv6: { enabled: true } to ipv6: { enabled: false } (currently hardcoded)
   • Remove the clusterPoolIPv6PodCIDRList and clusterPoolIPv6MaskSize lines from ipam.operator

   (Optionally: refactor to a cilium_ipv6_enabled Ansible variable for cleaner future toggling. This refactor is out of scope for the fallback — the inline change above is the minimum-viable disable.)

3. Re-run ansible-playbook --tags cilium-install (this triggers a helm upgrade).

4. Re-run the per-node procedure (Task 3) starting from step 4 (delete Cilium pod) on tpi-beta-1 to pick up the new config.

5. Document the fallback in the migration log.

6. Resume Task 3 for tpi-beta-2.

• Step 3: Stability soak — 4h minimum, 24h preferred before tpi-beta-2

Wait 4-24h. Check periodically:

# Agent restart count on the migrated node
kubectl get pods -n kube-system -l k8s-app=cilium \
  --field-selector spec.nodeName=tpi-beta-1 \
  -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}'
echo ""

# Pod restart events on the node
kubectl get events -A --field-selector type=Warning -o json \
  | jq -r --arg node tpi-beta-1 \
    '.items[] | select(.involvedObject.kind=="Node" and .involvedObject.name==$node) | "\(.lastTimestamp) \(.reason) \(.message)"' \
  | sort | tail -20

Expected: restart count 0; no recent NetworkNotReady events.

• Step 4: Append validation-gate entry to migration log

### First-node validation gate (post-tpi-beta-1)

- **IPv6 dual-stack:** OK / FALLBACK APPLIED (link to fallback notes)
- **Stability soak duration:** <Xh>
- **Agent restart count on tpi-beta-1:** 0
- **NetworkNotReady events:** none
- **Decision:** proceed to tpi-beta-2

Commit and proceed.

---

Task 5: All-workers stability gate (after tpi-alpha-4, before tpi-alpha-3)

Files: none (verification)

After all 5 workers (tpi-beta-1..4, tpi-alpha-4) have migrated, wait 24h before touching the control plane.

• Step 1: Run the 5-signal stability check on every migrated worker

For each of the 5 migrated workers:

for NODE in tpi-beta-1 tpi-beta-2 tpi-beta-3 tpi-beta-4 tpi-alpha-4; do
  echo "=== ${NODE} ==="
  # Agent restart count
  echo -n "agent restarts: "
  kubectl get pods -n kube-system -l k8s-app=cilium \
    --field-selector spec.nodeName=${NODE} \
    -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}'
  echo ""
  # BPFMapPressure / drop warnings in last 24h of agent logs
  echo -n "agent warnings: "
  # Find the cilium-agent pod on the target node, then check its recent logs for warnings/errors
  POD=$(kubectl get pods -n kube-system -l k8s-app=cilium -o json \
    --field-selector spec.nodeName=${NODE} \
    | jq -r '.items[0].metadata.name')
  test -n "${POD}" || { echo "ERROR: no cilium-agent pod on ${NODE}"; exit 1; }

  # Capture logs once so we can detect kubectl-logs failures separately from rg-no-matches
  LOG=$(kubectl logs -n kube-system "${POD}" --since=24h 2>&1) \
    || { echo "ERROR: kubectl logs failed for ${POD}"; exit 1; }

  # Match both text format (level=warning) AND JSON format ("level":"warning"), plus specific markers
  COUNT=$(printf '%s\n' "${LOG}" | rg -c '(level=|"level":")(warning|error)|BPFMapPressure|FATAL' || true)
  echo "${COUNT:-0}"
  # NetworkNotReady events
  echo -n "networknotready events: "
  SINCE=$(date -u -d '24 hours ago' +%s 2>/dev/null || date -u -v-24H +%s)
  kubectl get events -A --field-selector type=Warning -o json \
    | jq -r --arg node "${NODE}" --argjson since "${SINCE}" \
      '[.items[] | select(.reason=="NetworkNotReady") | select(.involvedObject.kind=="Node" and .involvedObject.name==$node) | select((.lastTimestamp // .eventTime | fromdateiso8601) > $since)] | length'
done

Expected: every node shows 0 for all three counts.

• Step 2: Verify Hubble flows from all 5 migrated nodes

# For each migrated node, confirm its agent is actively exporting flows.
# If a node is genuinely quiet, generate traffic from it first.
for MIGRATED_NODE in tpi-beta-1 tpi-beta-2 tpi-beta-3 tpi-beta-4 tpi-alpha-4; do
  COUNT=$(cilium hubble observe --node "${MIGRATED_NODE}" --since 5m --last 100 -o json 2>/dev/null | wc -l)
  if [ "${COUNT}" -eq 0 ]; then
    # Generate traffic from this node; verify pod actually scheduled + completed
    POD_NAME="traffic-$(echo ${MIGRATED_NODE} | tr . -)"
    kubectl -n kube-system run "${POD_NAME}" \
      --overrides='{"spec": {"nodeName": "'${MIGRATED_NODE}'", "tolerations": [{"operator": "Exists"}]}}' \
      --image=ghcr.io/nicolaka/netshoot:v0.8 --restart=Never \
      --command -- curl -fsk -o /dev/null https://kubernetes.default.svc/healthz

    if ! kubectl wait --for=jsonpath='{.status.phase}=Succeeded' --timeout=90s "pod/${POD_NAME}" -n kube-system; then
      echo "--- pod state ---"
      kubectl get pod "${POD_NAME}" -n kube-system -o wide
      echo "--- recent events ---"
      kubectl describe pod "${POD_NAME}" -n kube-system | tail -20
      kubectl delete pod "${POD_NAME}" -n kube-system --ignore-not-found --grace-period=1
      echo "FAIL: synthetic traffic pod did not complete on ${MIGRATED_NODE}"
      exit 1
    fi
    kubectl delete pod "${POD_NAME}" -n kube-system --ignore-not-found --grace-period=1
    sleep 10  # Hubble flow flush window (>= 5s; conservative)
    COUNT=$(cilium hubble observe --node "${MIGRATED_NODE}" --since 1m --last 100 -o json 2>/dev/null | wc -l)
  fi
  echo "${MIGRATED_NODE}: ${COUNT} flows in window"
  [ "${COUNT}" -gt 0 ] || { echo "FAIL: agent on ${MIGRATED_NODE} not exporting flows"; exit 1; }
done

Expected: every migrated node reports > 0 flows in the observation window.

• Step 3: Append gate entry to migration log

### All-workers stability gate

- **Soak duration:** <Xh>
- **Per-worker agent restarts:** all 0
- **Per-worker agent warnings (pressure/throttle/drop):** all 0
- **NetworkNotReady events in last 24h:** all 0
- **Hubble flows observed from migrated nodes:** <count of distinct node names>
- **Decision:** proceed to control-plane migration (tpi-alpha-3 first)

Commit and proceed.

---

Task 6: Final-node caveat acknowledgment

Files: none (gate)

Before migrating tpi-alpha-1 (the final node), explicitly acknowledge:

• Step 1: Confirm earlier 7 nodes have all been stable ≥24h

Re-run Task 5’s checks across all 7 migrated nodes. All zeros.

• Step 2: Take a fresh Velero backup

velero backup create pre-final-node-$(date +%Y%m%d-%H%M%S) --wait

• Step 3: Acknowledge in the migration log that tpi-alpha-1 is a one-way step

### Final-node gate (pre-tpi-alpha-1)

- **All 7 prior nodes stable for 24h+:** confirmed
- **Final Velero backup:** `pre-final-node-<timestamp>`
- **Acknowledgment:** migrating tpi-alpha-1 is one-way. Rollback at this point requires `helm uninstall cilium` and would orphan 7 already-migrated nodes. Proceeding.

Commit, then proceed with Task 3 (one more iteration) for tpi-alpha-1.

---

Task 7: Open the PR

Files: none (PR creation)

After the 8th node (tpi-alpha-1) migration is logged:

• Step 1: Final cluster-wide sanity check

# 1. All non-host-network pods are on Cilium CIDR
kubectl get pods -A -o wide --field-selector 'status.phase=Running' -o json \
  | jq -r '.items[] | select(.spec.hostNetwork != true) | "\(.metadata.namespace)/\(.metadata.name) \(.status.podIP)"' \
  | rg '10\.42\.'
# Expected: empty

# 2. All 8 nodes appear in CiliumNode resources
kubectl get ciliumnodes --no-headers | wc -l
# Expected: 8

# 3. cilium status overall
cilium status --wait

• Step 2: Close the beads epic

bd close ${EPIC_ID}

• Step 3: Commit + push + open PR

git push -u origin feat/cilium-migration  # or PR3-specific branch
gh pr create \
  --title "feat(cilium): per-node migration runbook (PR3/5)" \
  --body "$(cat <<'EOF'
## Summary

All 8 cluster nodes migrated from Calico to Cilium via upstream's official per-node procedure. Migration log appended iteratively to `docs/operations/cilium-migration.md`.

## Cluster impact

- All non-host-network pods now have IPs from `10.245.0.0/16` (Cilium CIDR).
- Calico is still installed (`calico-system` namespace and CRDs exist) but no longer manages any pod.
- `policyEnforcementMode: "never"` remains in effect cluster-wide until PR4.

## Notes

- IPv6 dual-stack: [validated successfully / fallback applied — see migration log]
- system-upgrade-controller still scaled to 0 (restored in PR4)
- No new NetworkPolicy / CiliumNetworkPolicy may merge until PR4

## Spec / prior PR

- Spec: `docs/engineering/specs/2026-05-10-calico-to-cilium-design.md`
- PR2: #<PR2-number>
- Next: PR4 (decommission Calico, flip Cilium to production mode)

## Test plan

- [x] All 8 nodes migrated; per-node verification in migration log
- [x] No pods on Calico CIDR `10.42.x.y`
- [x] `kubectl get ciliumnodes` shows 8 nodes
- [x] cross-CNI smoke tests passed (or N/A for final node)
- [x] No agent restarts on migrated nodes during stability gates
- [x] Final cluster-wide check before PR open: clean

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

---

Self-Review Checklist

• Spec coverage — pre-flight Velero (T1.1), system-upgrade scale-down (T1.4), PDB inventory (T1.3), per-node procedure (T3 with all 13 substeps), first-node IPv6 gate (T4), all-workers stability gate (T5), final-node caveat (T6), runbook log appends (T3.14 + gate logs).
• Placeholder scan — <fill in> markers in the migration log template are values the operator records at execution, not author gaps. Every kubectl/cilium/ansible command is explicit. The <NODE> placeholder in Task 3’s pre-loop framing is documented as a template-var to be substituted.
• Type/symbol consistency — label io.cilium.migration/cilium-default=true, selector k8s-app=cilium, CIDR 10.245.0.0/16 all match spec. The migration order in T1.9 matches T2.2’s node array matches T3’s iteration order.
• No invented references — every CLI flag (--disable-eviction, --ignore-daemonsets, reboot_timeout, post_reboot_delay) is real.

---

Notes for the implementer

• PR3 is iterative and may span multiple sessions. Use beads to resume across sessions. After each per-node iteration, commit the migration log entry — that way if the session crashes, the next session knows exactly where to resume.
• Each per-node iteration takes ~15-30 minutes (drain + label + restart + reboot + verify). Plus the first-node 4-24h soak and all-workers 24h soak. Total PR3 duration: 2-4 days minimum.
• If you hit the IPv6 fallback path, document it thoroughly. The follow-up beads issue should reference the migration log entry that triggered it.
• DO NOT skip the first-node validation gate. Catching IPv6 problems on node 1 (where rollback is cheap) is far better than catching them on node 4 (where rollback orphans 3 nodes’ worth of pods).
• DO NOT migrate two control-plane nodes in parallel. etcd quorum loss risk; the migration order (tpi-alpha-3 → tpi-alpha-2 → tpi-alpha-1) is sequential by design.

---

Execution handoff

Plan complete and saved to docs/engineering/plans/2026-05-11-cilium-pr3-per-node-migration.md. Two execution options:

1. Subagent-Driven — can dispatch per-node subagents but each one is operational against live cluster; you should be reviewing every step’s output before proceeding.

2. Inline Execution (recommended for this PR) — operational nature means continuous human oversight is appropriate. Use beads + this plan as the runbook; tick checkboxes as steps complete.

Which approach?