Skip to content
fzymgc-house Cluster Docs

Cilium Migration Log

This document is the chronological log of the Calico→Cilium migration described in docs/engineering/specs/2026-05-10-calico-to-cilium-design.md. It is updated iteratively during PR3 (per-node migration) and finalized in PR4.

After PR4 merges, this log is retained as a historical artifact so future debugging can reference what was done and when.

PR2 — Cilium installed alongside Calico

Section titled “PR2 — Cilium installed alongside Calico”
  • Date: 2026-05-11
  • Operator: Sean
  • Velero backup: pre-cilium-install-20260511-154646 (succeeded; default storage location)
  • Cilium version: 1.19.3
  • Cluster state pre-install: 8 nodes on Calico CNI; the gateway-api ArgoCD Application’s resources were partially orphaned post-PR1 merge (Application pruned, but 13 CRDs + safe-upgrades policy remained — cleaned up manually via kubectl per the PR2 pre-flight before the Ansible run).
  • Install duration: ~20 min including diagnosis of two surfaced bugs (see below).

Bugs surfaced during PR2 execution

Section titled “Bugs surfaced during PR2 execution”

  1. CEL regex escape in safe-upgrades-policy.yaml — YAML double-quoted \\. parsed to \. which CEL rejected as an unrecognized escape. Fixed in this PR by switching every \\ escape to \\\\ (5 backslash escapes total in the bundle-version regex: two \\. instances and one \\d) so YAML parses each to \\ and CEL parses each to literal \ in the regex. Final regex passed to CEL: v1\.[0-3]\.\d+.

  2. Operator nodeSelector value mismatch — node-role.kubernetes.io/control-plane: "" (kubeadm convention) didn’t match k3s’s "true" value; operator pods stayed Pending, blocking agent CRD registration. Fixed in this PR by changing the selector value to "true".

  3. kubernetes.core.helm 6.2.0 incompatible with Helm v4 — error “too many values to unpack (expected 3)”. Root cause: kubernetes.core 6.2.0 (Oct 2025) parses helm CLI output assuming the v3 format; Helm v4 changed the column layout. The collection’s 6.3.0 release acknowledged this by explicitly capping support at <4.0.0; 6.4.0 (Apr 2026) added v4 compatibility across the Helm modules (issue #1038).

    Resolution (follow-up PR after PR2 merge): bumped ansible/requirements.yml floor from >=5.0.0 to >=6.4.0 and upgraded the controller’s local collection. Re-ran the role end-to-end against the post-PR2 cluster as a smoke test — clean run, zero failed tasks, helm release advanced from revision 2 failed to revision 3 deployed. Tracked in beads hl-3y4.

    Workaround used during PR2 install (kept for reference; superseded by the 6.4.0 bump):

    Terminal window
    # Render the role's values template to a tmpfile, then run helm directly
    ansible localhost -m template \
      -a "src=ansible/roles/cilium/templates/values.yaml.j2 dest=/tmp/cilium-values.yaml" \
      -e "@ansible/roles/cilium/defaults/main.yml" \
      -e "k8s_context=fzymgc-house"
    helm upgrade cilium cilium/cilium \
      --version 1.19.3 \
      --namespace kube-system \
      --values /tmp/cilium-values.yaml \
      --wait --timeout 15m
    # And manually apply CiliumNodeConfig (Phase 3) since Phase 2 module
    # failure short-circuited the rest of the role:
    ansible localhost -m template \
      -a "src=ansible/roles/cilium/templates/cilium-node-config.yaml.j2 dest=/tmp/cilium-node-config.yaml" \
      -e "@ansible/roles/cilium/defaults/main.yml" \
      -e "k8s_context=fzymgc-house"
    kubectl apply -f /tmp/cilium-node-config.yaml

  4. Hubble Relay stuck with stale Calico IP — pod was scheduled during the broken-install window before Cilium agents were Ready, picked up a Calico IP from the still-active Calico CNI. Resolved by kubectl delete pod to force recreation once operator + agents were Ready. See the generalizable runbook in “PR3 troubleshooting” below.

Post-install verification (2026-05-11 ~16:35 ET / 20:35 UTC)

Section titled “Post-install verification (2026-05-11 ~16:35 ET / 20:35 UTC)”
CheckResult
cilium status --wait (post-fixes)OK
Cilium DaemonSet8/8 Ready
cilium-operator2/2 Ready (both on CP nodes: tpi-alpha-1, tpi-alpha-2)
Hubble Relay1/1 Ready
Hubble UI1/1 Ready
CiliumNodeConfig cilium-defaultApplied; matches 0 nodes (correct for pre-PR3)
Pods on Cilium CIDR0 (correct — cni.customConf: true in migration mode)
cilium-config.enable-policynever (migration mode)
Workload pod restarts attributable to install0 (9 pods started in the install window were all velero/*-kopia-maintain-job-* CronJob runs — see “Velero/Kopia churn during install” below)
6 bastion CiliumNetworkPolicy resources created via ArgoCD(will land after this PR merges and ArgoCD syncs)

Velero/Kopia churn during install

Section titled “Velero/Kopia churn during install”

The 9 pods that started during the install window were all scheduled CronJobs in the velero namespace, unrelated to the Cilium install:

PodNamespaceStart (UTC)
argocd-default-kopia-maintain-job-*-cxh7svelero2026-05-11T20:11:11Z
grafana-default-kopia-maintain-job-*-6c5ncvelero2026-05-11T20:11:23Z
longhorn-system-default-kopia-maintain-job-*-7nh5svelero2026-05-11T20:11:32Z
vault-default-kopia-maintain-job-*-k2xtpvelero2026-05-11T20:20:22Z
velero-default-kopia-maintain-job-*-wjj5wvelero2026-05-11T20:20:32Z
kube-system-default-kopia-maintain-job-*-t92s6velero2026-05-11T20:20:41Z
metallb-default-kopia-maintain-job-*-27xdgvelero2026-05-11T20:20:50Z
traefik-default-kopia-maintain-job-*-zrddlvelero2026-05-11T20:21:00Z
grafana-operator-default-kopia-maintain-job-*-rhv5svelero2026-05-11T20:21:09Z

All have the kopia-maintain-job- pattern indicating Velero’s Kopia maintenance scheduled jobs (per-app-namespace backup repository upkeep). None were workload pods; none were restarts of existing pods.

PR3 troubleshooting — generalizable patterns

Section titled “PR3 troubleshooting — generalizable patterns”

Two recurring patterns from PR2 install that operators should expect during per-node migration:

  1. Cilium-managed pod stuck with Calico IP (Hubble Relay, controllers, etc.)

    When a pod is scheduled during a transient window where Cilium hasn’t yet claimed CNI ownership on its target node, kubelet picks up Calico’s CNI conf and the pod gets a Calico-range IP (10.42.x.y). Even after Cilium completes its setup on that node, the existing pod keeps its Calico IP and may fail to communicate with Cilium-managed services.

    Detection:

    Terminal window
    # Find Cilium-namespace pods whose IPs are NOT in the Cilium pod CIDR
    kubectl get pods -n kube-system -o wide | rg '10\.42\.'

    Resolution:

    Terminal window
    # Force pod recreation; new pod will pick up Cilium IP from current CNI
    kubectl delete pod -n <namespace> <pod-name>

    This applied to hubble-relay during PR2 install. Expect it to apply to any non-DaemonSet Cilium-component pod (hubble-ui, cilium-operator) if scheduled during a node-migration transient.

Constraint reminder

Section titled “Constraint reminder”

Between PR2 merge and PR4 merge, no new NetworkPolicy or CiliumNetworkPolicy resources may merge to argocd/. During this window, policyEnforcementMode: "never" makes Cilium ignore all policies cluster-wide, so newly-added policies would silently fail to enforce and create a security regression. The bastion’s pre-staged policies (this PR) are exempt because they’re the migration target, not new functionality.

PR3 — Per-node migration

Section titled “PR3 — Per-node migration”

Pre-flight

Section titled “Pre-flight”

  • Date: 2026-05-11

  • Operator: Sean

  • Pre-flight Velero backup: pre-pr3-migration-20260511-195355 (Completed; default storage location)

  • Cluster state archive: /tmp/pre-pr3-cluster-state.tar.gz (17 MB; retain through PR4)

  • system-upgrade-controller: scaled to 0 replicas (restored in PR4)

  • Pre-PR3 invariants verified:

    • 0 nodes labeled with io.cilium.migration/cilium-default=true
    • cilium-config.enable-policy = never (migration mode)
    • 0 workload pods on Cilium CIDR 10.245.0.0/16

  • SSH connectivity: all 8 nodes return SUCCESS via ansible -m ping

  • PDB inventory (drain risk assessment):

    calico-apiserver/calico-apiserver       minAvailable=-  maxUnavailable=1
    calico-system/calico-typha              minAvailable=-  maxUnavailable=1
    cert-manager/cert-manager               minAvailable=1  maxUnavailable=-
    cert-manager/cert-manager-cainjector    minAvailable=1  maxUnavailable=-
    cert-manager/cert-manager-webhook       minAvailable=1  maxUnavailable=-
    loki/loki-backend                       minAvailable=-  maxUnavailable=1
    loki/loki-read                          minAvailable=-  maxUnavailable=1
    loki/loki-write                         minAvailable=-  maxUnavailable=1
    longhorn-system/csi-attacher            minAvailable=1  maxUnavailable=-
    longhorn-system/csi-provisioner         minAvailable=1  maxUnavailable=-
    longhorn-system/instance-manager-*      minAvailable=1  maxUnavailable=-   (8 per-node singletons)
    nats/nats                               minAvailable=-  maxUnavailable=1
    postgres/main                           minAvailable=1  maxUnavailable=-
    postgres/main-primary                   minAvailable=1  maxUnavailable=-
    traefik/traefik                         minAvailable=1  maxUnavailable=-
    vault/vault                             minAvailable=-  maxUnavailable=1

    Likely drain blockers: Longhorn instance-manager-* pods are per-node singletons with minAvailable=1 and are not DaemonSets, so a drain targeting their node will trip the PDB. The documented mitigation is kubectl drain --disable-eviction (delete API bypass). Note in each per-node log entry whether --disable-eviction was needed.

Migration order:

  1. tpi-beta-1 (worker)
  2. First-node validation gate — IPv6 dual-stack + stability soak (4-24 h)
  3. tpi-beta-2, tpi-beta-3 (workers)
  4. tpi-beta-4, tpi-alpha-4 (workers; bastion’s host node migrates here)
  5. All-workers stability gate — 24 h soak before touching control plane
  6. tpi-alpha-3, tpi-alpha-2, tpi-alpha-1 (control plane, reverse order, preserves etcd quorum)

(Per-node entries appended below as each migration completes.)

tpi-beta-1 — migrated 2026-05-12 00:13 UTC

Section titled “tpi-beta-1 — migrated 2026-05-12 00:13 UTC”
  • Drain duration: 35 s (under the 5-min “expected” window). One PDB block hit on longhorn-system/instance-manager-5ed5af1b5f1... — Longhorn rescheduled the replica during the retry loop and the fourth eviction attempt succeeded. --disable-eviction NOT used. Other PDB-protected workloads (postgres/main-16, loki/loki-write-2, nats/nats-1) evicted cleanly thanks to their maxUnavailable=1 PDBs permitting one-at-a-time eviction. 29 pods evicted total, 0 stuck.
  • Cilium agent restart on labeled node: cilium pod replaced within ~30 s; new pod Ready after ~60 s.
  • CNI conf flip confirmed via cilium-agent’s host mount: 05-cilium.conflist written; 10-calico.conflist renamed to 10-calico.conflist.cilium_bak (the cni-exclusive: true behavior).
  • Reboot duration: 2 m 3 s (ansible.builtin.reboot, reboot_timeout=600 post_reboot_delay=30).
  • Post-reboot transient: cilium-envoy-f5xwq showed 0/1 Ready for ~60 s while its xDS gRPC stream reconnected to the freshly-started agent. Resolved on its own; cilium status --wait returned OK shortly after.
  • Smoke test (modified — see “Anomalies” below):
    • Verify-network pod IPs: 10.245.5.191/32 (v4) and fd00:10:245:5::1776/128 (v6) — both from Cilium’s cluster pools.
    • API server reachability: TLS handshake to https://10.43.0.1:443/healthz completes in 8 ms (returns 401 from the auth wall — connectivity is healthy).
    • External egress: ping 1.1.1.1 returns 7 ms.
    • Cross-CNI Cilium→Calico (v4 ICMP, 10.245.5.x → 10.42.180.76): 3/3 received, 0.5-0.8 ms RTT.
    • Cross-CNI Calico→Cilium (v4 ICMP, 10.42.x.x → 10.245.5.54): 3/3 received, 0.5-0.8 ms RTT.
  • Pods on node post-migration: 16 total (8 with Cilium pod IPs, 7 hostNetwork including kept Calico-node DS pod, 1 leftover node-debugger). 0 stragglers (no pod with a stale 10.42.x.x IP).
  • Agent restart count post-migration: 1 (the single reboot-induced restart; no flapping).
  • Anomalies (plan-text bugs, none migration-caused):
    1. Plan’s smoke test uses rg but ghcr.io/nicolaka/netshoot:v0.8 ships grep only, not rg. Substituted grep -F "10.245.". The plan’s “if a future version drops those tools” disclaimer was already obsolete at write time. Track for future plan revisions.
    2. Plan’s curl -fs https://api.github.com test fails with rc=6 (couldn’t resolve host) on any node in this cluster because of a musl libc + cluster search-domain (fzymgc.house) interaction that returns an empty authoritative response, short-circuiting musl’s domain iteration. Reproduced on a still-Calico node (tpi-beta-2) for confirmation. The plan’s test is a false-negative on this cluster regardless of CNI.
    3. Cluster’s Calico install is single-stack v4 (259 v4 pod IPs, 0 v6 pod IPs cluster-wide pre-migration). Cilium is ADDING v6 capability, not preserving it. This means cross-CNI v6 ping is N/A — no Calico-side v6 peer exists. v6 cross-node tunneling can only be validated after node 2 migrates.

First-node validation gate — 2026-05-12 00:25 UTC

Section titled “First-node validation gate — 2026-05-12 00:25 UTC”
  • IPv6 dual-stack: WORKING within scope.
    • Cilium pod v6 address from pool fd00:10:245::/48: ✓ (fd00:10:245:5::29af/128 sample).
    • v6 default route present: ✓ (default via fd00:10:245:5::ee96 dev eth0 metric 1024 mtu 1360).
    • v6 pod-to-pod within Cilium on same node: ✓ (3/3 ping6, 73-143 μs RTT — kernel-only, doesn’t exercise VXLAN).
    • v6 DNS resolution (AAAA for external host via cluster DNS): ✓ (8 Google AAAA records returned via dig @10.43.0.10).
    • v6 cross-CNI: N/A (Calico is v4-only; no Calico v6 peer exists in the cluster).
    • v6 to kube-apiserver: N/A (kube-apiserver Service is single-stack v4 per ipFamilies: ["IPv4"]; cluster’s control-plane addressing was always v4 and Cilium doesn’t change that).
    • Decision: v6 is adding capability without breaking anything, so the IPv6 fallback (disable v6 cluster-wide) is NOT invoked. Full v6 cross-node validation is deferred until after node 2’s migration when there will be two Cilium-managed nodes available.
  • Stability soak duration: ~20 min (intentionally shortened from the plan’s 4-24 h target). Rationale: operator judgment to proceed with tpi-beta-2 immediately so that cross-node Cilium↔Cilium validation (v4 AND v6 VXLAN tunneling) could be added to the evidence set. The soak gate’s purpose is to catch latent stability issues; functional cross-node evidence is arguably more decisive for proving the migration design works. Mitigation: tpi-beta-1’s agent restart count was monitored during and after tpi-beta-2’s migration; remained at 1 throughout (no destabilization).
  • Agent restart count on tpi-beta-1 at gate entry: 1 (single reboot-induced restart).
  • NetworkNotReady events on tpi-beta-1: none.
  • Cluster-wide collateral check: all 8 nodes Ready; 38 ArgoCD applications Synced/Healthy; 2 preexisting unhealthy pods (karakeep/meilisearch CrashLoopBackOff 26 h, temporal-worker- controller-manager ImagePullBackOff 7 h 31 m) — both predate the migration and are not regressions.
  • Decision (taken 2026-05-12 ~00:45 UTC): proceeded to tpi-beta-2 without the longer soak; functional cross-node evidence (see tpi-beta-2 entry below) supersedes the time-based soak as validation that the migration design works.

tpi-beta-2 — migrated 2026-05-12 00:55 UTC

Section titled “tpi-beta-2 — migrated 2026-05-12 00:55 UTC”
  • Drain wall-clock 5 m 10 s. First-pass kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=5m hit the global timeout while retrying eviction of longhorn-system/instance-manager-9e303193f6af67a4962641022b0ed2ac (Longhorn did NOT self-resolve the PDB this time, unlike on tpi-beta-1). Other 29 pods evicted within ~30 s. Fallback kubectl drain ... --disable-eviction completed in 7 s by using the delete API to bypass the PDB. --disable-eviction used — documented per plan Task 3 Step 2 fallback policy. No workload impact (Longhorn replicas remain healthy on other nodes).
  • CNI conf flip confirmed: 05-cilium.conflist written; 10-calico.conflist renamed to .cilium_bak.
  • Reboot duration: 2 m 2 s.
  • Cilium global health: post-reboot showed “1 warning” momentarily; resolved on its own within ~30 s (cilium-envoy xDS reconnect pattern, same as tpi-beta-1).
  • Smoke test: verify-network pod got 10.245.6.235/32 (v4) and fd00:10:245:6::4bc2/128 (v6) — different /24 and /64 from tpi-beta-1’s pool slice, confirming per-node IPAM allocation. External egress works (ping 1.1.1.1: 7 ms RTT, 0% loss).
  • CRITICAL CROSS-NODE VALIDATION (only possible with 2 Cilium nodes):
    • v4 tpi-beta-2 → tpi-beta-1 (10.245.6.x → 10.245.5.54): 3/3, 0.5-0.8 ms RTT — VXLAN tunneling works.
    • v6 tpi-beta-2 → tpi-beta-1 (fd00:10:245:6:: → fd00:10:245:5::9fdf): 3/3, 0.6-1.6 ms RTT — IPv6 VXLAN tunneling works across nodes.
    • v4 reverse (b1 → b2): 3/3, 0.5-0.9 ms RTT.
    • v6 reverse (b1 → b2): 3/3, 0.7-1.3 ms RTT.
    • Cross-CNI still works (b2 Cilium → Calico pod on tpi-alpha-3): 3/3, 0.6-1.4 ms RTT.
  • Pods on node post-migration: 15 total (10 Cilium-managed, 5 hostNetwork). 0 stragglers.
  • tpi-beta-1 stability through tpi-beta-2 migration: agent restart count remained at 1 (no destabilization from second migration). Confirms node migrations are independent.
  • Cluster-wide Cilium-managed pods: 8 → 32 of 221 (the jump is the full set of newly-Cilium pods on tpi-beta-2 PLUS the workload pods that migrated to tpi-beta-2 from tpi-beta-1’s earlier drain rescheduling, e.g., nats/nats-1, mosquitto-*-7lwhm).
  • Anomalies: same 3 plan-text bugs as tpi-beta-1 entry (no new surprises). --disable-eviction was needed this time (it wasn’t for tpi-beta-1) — confirms Longhorn’s PDB self-resolution on tpi-beta-1 was favorable timing, not a reliable pattern.

tpi-beta-3 — migrated 2026-05-12 ~01:20 UTC

Section titled “tpi-beta-3 — migrated 2026-05-12 ~01:20 UTC”

  • Two operational adventures this time (worth recording because they may recur on remaining nodes):

    1. User initiated kubectl drain without --timeout. Drain hung indefinitely on the Longhorn instance-manager-1b8c1c4d55fc... PDB block (same pattern as tpi-beta-2). Resolution: user Ctrl-C the stuck drain, then kubectl drain --disable-eviction completed in 5 s. Lesson: future per-node migrations should run kubectl drain --disable-eviction directly from the start (skip the timeout-then-retry dance). Plan revision should reflect this.
    2. vault-1 got bounced across multiple nodes during the long drain window and its data Longhorn volume got stuck. Sequence: vault-1 was on tpi-beta-3 originally; drain evicted it; k8s scheduled vault-1 to tpi-beta-2; pod stuck ContainerCreating ~10 min with FailedAttachVolume: pvc-3008cdde.... While the pod was bouncing, Longhorn’s volume currentNodeID stayed at tpi-beta-3 because the OLD kubelet on tpi-beta-3 still held an active mount reference. Once tpi-beta-3 rebooted, the volume detached cleanly — but by then vault-1’s pod had been moved to tpi-alpha-4 by k8s, while the volume reattached to tpi-beta-2 (mismatch). Resolution: kubectl delete pod -n vault vault-1 to force a StatefulSet recreation. New vault-1 landed on tpi-beta-3 (which had finished rebooting); Longhorn detached the volume from tpi-beta-2 and reattached to tpi-beta-3 in ~10 s; vault-1 reached Running 1/1 ~95 s after recreation. Lesson: pod-volume mismatch from drain-bouncing resolves itself if you delete the stuck pending pod and let the StatefulSet controller recreate. No manual volume.spec.nodeID patch dance was needed.

  • Drain duration: 5 s (post-Ctrl-C, with --disable-eviction).

  • Reboot duration: 2 m 3 s.

  • CNI flip confirmed: 05-cilium.conflist written; 10-calico.conflist renamed to .cilium_bak.

  • Note on cordon state: tpi-beta-3 appeared uncordoned post- reboot, but this was Sean uncordoning manually outside the scripted command sequence — NOT automatic k3s behavior. Confirmed during tpi-alpha-4 verification: cordons DO persist through reboot. Add explicit kubectl uncordon ${NODE} to every per-node sequence. (Earlier draft of this log claimed “k3s clears unschedulable on reboot” — that was wrong.)

  • Smoke test:

    • Verify-network pod: 10.245.7.59/32 (v4) and fd00:10:245:7::41a0/128 (v6). External egress (ping 1.1.1.1) 6 ms.
    • Cross-node b3 → b1 v4: 1.6-2.5 ms; v6: 0.59-0.61 ms.
    • Cross-node b3 → b2 v4: 13-14 ms; v6: 11-15 ms. (Higher RTT than b3 → b1; likely a transient or physical-layer factor — 0% loss is what matters.)

  • Pods on node post-migration: 14 total (11 Cilium-managed including vault/vault-1, 3 hostNetwork). 0 stragglers.

  • All-prior-nodes stability through tpi-beta-3 migration: tpi-beta-1 restart count = 1, tpi-beta-2 restart count = 1. Cluster Cilium pod count: 32 → 52 / 221.

  • Notes for remaining nodes:

    • Run kubectl drain --disable-eviction directly (skip the standard drain attempt).
    • Don’t forget to kubectl uncordon ${NODE} after the node is Ready post-reboot. Cordon persists through reboot; DaemonSets schedule on cordoned nodes (they tolerate the NoSchedule taint), so the node will appear healthy even when cordoned. Workload pods won’t reschedule there.
    • Watch for StatefulSet pods (vault, postgres, dolt, temporal, loki-write, nats, prometheus) bouncing during drain. If any pod is stuck ContainerCreating for >5 min with FailedAttachVolume, delete the pod and let StatefulSet recreate.

tpi-beta-4 — migrated 2026-05-12 ~01:29 UTC

Section titled “tpi-beta-4 — migrated 2026-05-12 ~01:29 UTC”
  • Applied lessons from tpi-beta-3 — went directly to kubectl drain --disable-eviction --timeout=3m. Drain 36 s (no PDB delay). 14 workload pods evicted (mostly short-lived Velero Kopia maintenance jobs).
  • Reboot 2 m 5 s.
  • 7 Cilium-managed pods on tpi-beta-4 (pool /24 10.245.0.x — Cilium IPAM grabbed the first free slot from pool start, not sequential after b3’s 10.245.7.x). 0 stragglers.
  • Cross-node b4 → {b1, b2, b3} v4 and v6: all OK, RTT 0.6-11 ms.
  • No StatefulSet pods on tpi-beta-4 (the only non-DS storage- using pods elsewhere) → no volume-bounce drama. Clean run.
  • cf-ssh-bastion not affected (it’s on tpi-alpha-4, which is next).
  • All-prior-nodes stability check: tpi-beta-1, tpi-beta-2, tpi-beta-3 agent restart counts still 1 each — cumulative stability through 4 migrations.
  • Cluster Cilium pod count: 52 → 75 / 220.
  • Note for migrations after this: longhorn instance-manager-* isn’t immediately rescheduled back to a newly-migrated node; Longhorn schedules it when needed for volume hosting.

tpi-alpha-4 — migrated 2026-05-12 ~01:43 UTC

Section titled “tpi-alpha-4 — migrated 2026-05-12 ~01:43 UTC”
  • This is the LAST worker before the all-workers stability gate. Also hosts cf-ssh-bastion (the only external SSH path into the cluster).
  • Drain 47 s (--disable-eviction direct). 14 workload pods evicted including cf-ssh-bastion-*, argocd-server (one of 2 replicas), loki/loki-write-2, nats/nats-0, postgres/main-15, Authentik server/worker, hcp-terraform agents, karakeep, GHA runner listener.
  • Reboot 36 s — notably faster than other nodes’ ~2 min. May be tpi-alpha-4 boots faster (warmer caches? different storage? haven’t pinned down). One side effect: cilium agent restart count = 2 (not 1 like other nodes). Root cause: faster reboot caught the agent mid-shutdown while it was actively importing cf-ssh-bastion CiliumNetworkPolicies, leading to an additional SIGTERM-recovery cycle on top of the normal reboot restart. Agent is currently Running and healthy; not treated as a regression.
  • cf-ssh-bastion rescheduled cleanly to tpi-beta-1 with new Cilium IP 10.245.5.30. SSH access preserved via Cloudflare tunnel.
  • nats-0 saga (similar to vault-1 but worse):
    • During drain, nats-0 was evicted and bounced through tpi-alpha-4 → tpi-alpha-3 → tpi-beta-4. Its data PVC (pvc-30882cc6-...) entered a wedged state: state=detached robustness=unknown currentNodeID="" specNodeID=tpi-alpha-3. Longhorn could not attach the volume — repeated FailedAttachVolume: rpc error: code = DeadlineExceeded for >7 min.
    • Resolution: delete the PVC + pod entirely. The NATS StatefulSet recreated both with a fresh PV; nats-0 reached Running 3/3 ~1 min later. JetStream replicated the stream data from nats-1/nats-2 (running peers on Calico-side nodes), making this safe.
    • Important pattern: when a Longhorn volume reaches robustness=unknown AND attach errors are DeadlineExceeded, the simple kubectl delete pod fix won’t work — the volume itself is wedged. For HA-clustered services (NATS JetStream, CNPG, Vault Raft cluster, etc.), deleting the PVC and letting cluster replication restore data is faster than coaxing the wedged volume back to life.
  • Smoke test: pod IPs 10.245.4.39/32 (v4), fd00:10:245:4::e515/128 (v6). External ping 1.1.1.1: 6-7 ms. Cross-node a4 → {b1, b2}: v4 0.9-2.1 ms, v6 0.7-1.5 ms.
  • Pods on node post-migration: 7 Cilium-managed (DS pods that rescheduled here automatically). 0 stragglers.

All-workers stability gate — 2026-05-12 ~01:43 UTC

Section titled “All-workers stability gate — 2026-05-12 ~01:43 UTC”
  • All 5 workers migrated: tpi-beta-1, tpi-beta-2, tpi-beta-3, tpi-beta-4, tpi-alpha-4.
  • Cluster Cilium pod count: 94/221 (~43%). Remaining 127 pods are still on Calico CIDR (10.42.0.0/16), pinned to the 3 still-Calico control-plane nodes (tpi-alpha-1, tpi-alpha-2, tpi-alpha-3).
  • Agent restart counts: 4 nodes at 1, tpi-alpha-4 at 2 (explained above). All agents currently Running and healthy.
  • Stability soak required: 24 hours minimum before migrating any control-plane node. The CP migrations are higher-risk than worker migrations because:
    • kube-apiserver lives on CP nodes (loss of one → 2/3 etcd quorum survives, but rolling reboots must be sequential).
    • cilium-operator replicas are pinned to control-plane nodes (currently on tpi-alpha-1 and tpi-alpha-2). Migration of these two nodes requires the operator to reschedule to a peer; the third CP (tpi-alpha-3) must be migrated FIRST so it’s eligible to host an operator before tpi-alpha-1 and tpi-alpha-2 migrate.
  • Notes captured during workers phase, applied to remaining 3 CP nodes:
    • Run kubectl drain --disable-eviction --timeout=3m directly.
    • Reboot via ansible.builtin.reboot reboot_timeout=600 post_reboot_delay=30.
    • Watch for StatefulSet pods stuck ContainerCreating with FailedAttachVolume >5 min — kubectl delete pod first; if volume is robustness=unknown with DeadlineExceeded, delete the PVC and let HA service replicate data back.
    • Run kubectl uncordon ${NODE} explicitly after the node is Ready post-reboot — cordons DO persist through reboot (confirmed during tpi-alpha-4 migration; Sean caught my earlier omission). DaemonSets schedule on cordoned nodes so the node will look healthy even when no workload pods can reschedule there.
  • Pending soak: wait minimum 24 h before migrating tpi-alpha-3 (next: hl-51o.6).

Cross-CNI NetworkPolicy intervention — 2026-05-12 02:55-03:21 UTC

Section titled “Cross-CNI NetworkPolicy intervention — 2026-05-12 02:55-03:21 UTC”

Between the all-workers gate and the first CP migration, a real problem surfaced and was resolved via two hotfix PRs to main:

  • Symptom: vault-1 (migrated to Cilium tpi-alpha-4) entered a restart loop returning HTTP 474 from /v1/sys/health. Cluster showed ~10 ExternalSecret UpdateFailed events in 10 minutes.
  • Root cause: the vault Helm chart’s auto-generated NetworkPolicy had from: [namespaceSelector: {}] ingress rules. Calico’s enforcer on still-Calico vault-0/vault-2 evaluated this by looking the source pod up in its WorkloadEndpoints cache; Cilium-managed pods (10.245.0.0/16) are NOT in Calico’s cache, so the rule didn’t match and Calico defaulted to deny. Cross-CNI raft heartbeats from vault-1 timed out.
  • Misinterpretation worth recording: HTTP 474 in Vault 1.20.4 is “standby + HA connection unhealthy” (per the full JSON body’s ha_connection_healthy: false), NOT “removed from cluster” as earlier documentation suggested. Verify root cause by reading the full response body, not just the status code. An earlier speculative raft-state wipe + PVC delete were detours that didn’t help (and were ultimately reverted by deleting the vault-1 PVCs anyway for unrelated reasons).
  • Fixes (in order):
    1. PR #1015 (a8818079) — disable vault chart’s NP via Helm values. ArgoCD synced; vault-1 reached Running 1/1 restartCount=0 within 90 seconds.
    2. Cascade mitigation (commit 63c61e7d on PR3 branch) — re-point ESO ClusterSecretStore from vault-internal (round-robin all 3 vault pods) to vault-active (leader-only). This is independent of the vault-1 fix and prevents future similar cascades. Live cluster patched directly; TF source updated for next manual apply.
    3. PR #1016 (fe13f7ce) — same fix for NATS NP. NATS pods were healthy at the time, but the same pattern would have broken NATS clustering during the last CP migration (tpi-alpha-1, hosts nats-2). Removed pre-emptively.
  • Systemic risk audit: other namespaces with namespaceSelector: {} / podSelector NPs are at risk during cross-CNI transitions. After audit: only vault + NATS were migration-critical. The 3 calico-system NPs (allow-apiserver, goldmane, whisker) and 6 vestigial cf-ssh-bastion Calico NPs don’t intersect migration traffic — they go away with Calico in PR4 / are swept in PR5.

tpi-alpha-3 — migrated 2026-05-12 12:08 UTC

Section titled “tpi-alpha-3 — migrated 2026-05-12 12:08 UTC”
  • First CP node migration. Audit pre-drain: 59 pods total including 4 StatefulSets (vault/vault-2, loki/loki-backend-0, dolt/dolt-0 — singleton, no HA peer — and tailscale/ts-home-network-tz7kn-0), one of two cilium-operator replicas, and 6 Longhorn volumes attached.
  • Drain 45 s (--disable-eviction direct). 59 pods evicted, 0 stuck. Reboot 2 m 3 s.
  • Cilium operator handoff worked cleanly: the operator pod on tpi-alpha-3 (...ls5ff) rescheduled to tpi-alpha-2 (...n9jdp) within 70 s of eviction. Both operators stayed Running through the migration; the podAntiAffinity: required constraint forced the move to the only valid CP slot.
  • All 4 StatefulSets recovered cleanly via SS controller reschedule — NO volume-bounce intervention needed:
    • vault-2 → tpi-beta-4 (Cilium IP 10.245.0.60), Running 1/1 in <2 min.
    • loki-backend-0 → tpi-alpha-4, Running 1/1.
    • dolt-0 → tpi-alpha-4, Running 1/1 (singleton; brief downtime acceptable).
    • tailscale/ts-home-network-tz7kn-0 → tpi-beta-4, Running 1/1.
  • This is the validation that the NP-removal hotfixes worked. Previously vault-1 needed PVC delete + raft wipe attempts. Now vault-2 reattached its volume and joined raft cleanly — same cross-CNI path, but without the Calico NP blocking the heartbeat. Confirmed via /v1/sys/seal-status: sealed:false, removed_from_cluster:false for both vault-1 and vault-2.
  • Vault HA preserved. All 3 pods 1/1 Running throughout: vault-0 Calico (tpi-alpha-1), vault-1 Cilium (tpi-alpha-4), vault-2 fresh Cilium (tpi-beta-4).
  • Smoke test: pod IPs 10.245.3.135/32 (v4) and fd00:10:245:3::7d60/128 (v6). External ping 1.1.1.1: 5-7 ms RTT.
  • 0 stragglers on tpi-alpha-3 post-migration.
  • No ESO cascade (PR #1015 + the vault-active routing prevented it).
  • All prior Cilium agents stable through this migration: tpi-beta-1..4 restart=1 each (unchanged), tpi-alpha-4 restart=2 (unchanged from prior anomaly), tpi-alpha-3 restart=1 (single reboot-induced restart, clean).
  • Cluster Cilium pod count: 94 → 146 / 221 (~66%). Biggest single-migration jump so far.
  • Cilium operator placement now: tpi-alpha-1 + tpi-alpha-2. Next migration (tpi-alpha-2) will move its operator to either tpi-alpha-3 (now Cilium-eligible) or tpi-alpha-1.

tpi-alpha-2 — migrated 2026-05-12 12:58 UTC

Section titled “tpi-alpha-2 — migrated 2026-05-12 12:58 UTC”
  • Second CP node migration. Audit pre-drain: 33 pods total (much lighter than alpha-3’s 59). Drain targets included one of two cilium-operator replicas (...n9jdp, freshly placed there during alpha-3’s migration 50 min earlier), vault/vault-agent-injector, loki/loki-read, and 3 StatefulSets (loki-chunks-cache-0, loki-results-cache-0, prometheus/alertmanager-0). Only 1 Longhorn volume attached.
  • Drain 39 s (--disable-eviction direct). Reboot 2 m 3 s.
  • Cilium operator danced again: the operator on tpi-alpha-2 (...n9jdp) rescheduled to tpi-alpha-3 (...jxqjp) within 65 s of eviction. The podAntiAffinity: required constraint forced it; tpi-alpha-3 was the only valid CP slot (alpha-1 still has the original operator). Both operators stayed Running.
  • All 3 StatefulSets recovered cleanly via SS controller reschedule (no volume-bounce intervention needed):
    • loki-chunks-cache-0 → tpi-alpha-3, Running 1/1.
    • loki-results-cache-0 → tpi-alpha-4, Running 1/1.
    • prometheus/alertmanager-0 → tpi-alpha-4, Running 1/1.
  • Smoke test: pod IPs 10.245.2.99/32 (v4) and fd00:10:245:2::e2d6/128 (v6). External ping 1.1.1.1: 5.8-6.3 ms RTT.
  • 0 stragglers on tpi-alpha-2 post-migration.
  • All prior Cilium agents stable through this migration: beta-1..4 = 1 each, alpha-3 = 1, alpha-4 = 2 (prior anomaly), alpha-2 = 1 (single reboot-induced restart, clean).
  • Cluster Cilium pod count: 146 → 171 / 221 (~77%).
  • Cilium operator placement now: tpi-alpha-1 + tpi-alpha-3. Migrating tpi-alpha-1 next will move that operator to tpi-alpha-2 (now Cilium).

tpi-alpha-1 — migrated 2026-05-12 13:10 UTC (FINAL CP)

Section titled “tpi-alpha-1 — migrated 2026-05-12 13:10 UTC (FINAL CP)”
  • The big one. 58 pods total including:
    • vault-0 (active Raft leader at drain time)
    • nats-2 (last Calico NATS replica)
    • external-secrets-operator-* (single replica)
    • cilium-operator-7744fd4c4b-hl829 (16h-old, the original-placement operator)
    • kube-vip-tpi-alpha-1 (DaemonSet — owns the cluster API VIP)
    • 5 StatefulSets (vault-0, nats-2, loki-write-0, argocd/argocd-application-controller-0, tailscale ts-home-network-tz7kn-1)
    • 8 Longhorn volumes attached
    • postgres/main-15 (one CNPG instance)
    • replicator-kubernetes-replicator (cross-namespace secret replicator used for vault CA chain etc.)
  • Drain 52 s (--disable-eviction direct). Reboot 2 m 3 s.
  • Vault leader election succeeded: vault-0’s eviction triggered an election; vault-1 became the new leader within seconds (verified via /v1/sys/leader on vault-2 reporting leader_cluster_address: vault-1.vault-internal:8201). vault-0 came back fresh on tpi-alpha-2 with Cilium IP 10.245.2.211, restartCount=0. ESO routing via vault-active Service made the failover transparent — no cascade events, ExternalSecrets continued reconciling.
  • kube-vip VIP failover transparent: the API VIP 192.168.20.140 was reachable throughout (verified post-reboot: ping 192.168.20.140 shows 0% loss, 0.8 ms RTT). VRRP-based leader election handled the failover automatically.
  • Cilium operator dance (final iteration): hl829 on alpha-1 → hrzr5 on tpi-alpha-2 (78 s after eviction). Both operators stayed Running. Placement post-migration: tpi-alpha-2 + tpi-alpha-3 (both Cilium).
  • StatefulSets that moved:
    • vault-0 → tpi-alpha-2 (Cilium IP 10.245.2.211)
    • argocd-application-controller-0 → tpi-alpha-2 (Cilium)
    • loki-write-0 → tpi-alpha-2 (Cilium)
    • tailscale ts-home-network-tz7kn-1 → tpi-alpha-3 (Cilium)
    • nats-2 → tpi-alpha-1 (rescheduled BACK after reboot; needed one kubectl delete pod to clear a transient volume-attach bounce — same proven pattern as nats-0 earlier in the migration. Recovered in 40 s after delete.)
  • etcd quorum preserved through reboot: kubectl get --raw='/readyz' → [+]etcd ok. tpi-alpha-1’s etcd member rejoined post-reboot; 2/3 quorum maintained throughout.
  • 0 stragglers on tpi-alpha-1 post-migration.
  • No ESO cascade (the vault-active routing did its job).
  • Cluster Cilium pod count: 171 → 221 / 221 (100% Cilium-managed).
  • All 8 Cilium agent restart counts: tpi-alpha-1..3 = 1 each (clean single reboot-induced restart per node), tpi-alpha-4 = 2 (prior anomaly, unchanged), tpi-beta-1..4 = 1 each. No flapping at any point during the 8-node migration.

PR3 — complete

Section titled “PR3 — complete”
  • All 8 nodes Cilium-managed. 221/221 cluster pods on Cilium CIDR 10.245.0.0/16 (dual-stack with fd00:10:245::/48).
  • Total wall-clock for PR3: ~18 hours (workers ~3 h, all- workers gate ~10 h, 3 CP nodes ~30 min total).
  • Two hotfix PRs merged to main during the migration:
    • PR #1015 (a8818079) — disabled vault chart’s NetworkPolicy. Root cause: Calico’s enforcer couldn’t evaluate namespaceSelector: {} against Cilium-side source pods, blocking cross-CNI vault raft heartbeats.
    • PR #1016 (fe13f7ce) — disabled NATS NetworkPolicy pre-emptively (same pattern; NATS clustering would have degraded during the last CP migration).
    • Plus a commit on this PR3 branch (63c61e7d) repointing ESO ClusterSecretStore from vault-internal (round-robin) to vault-active (leader-only), preventing future cascades of the form seen with vault-1’s initial restart loop.
  • Cluster state after PR3:
    • Calico runs in parallel (its DaemonSets, calico-apiserver, etc. are still up — they don’t manage CNI but consume cluster resources)
    • policyEnforcementMode: never on Cilium (no CiliumNetworkPolicy enforcement yet)
    • Bastion CiliumNetworkPolicies are stored but not enforced
  • Next: PR4 decommissions Calico (calico and calico-system namespaces removed) and flips Cilium’s policyEnforcementMode from never to default. After PR4, the bastion’s CiliumNetworkPolicies and any other CNPs go live. PR5 sweeps the legacy calico-network-policy.yaml file from argocd/app-configs/cf-ssh-bastion/.

PR4 — Decommission Calico

Section titled “PR4 — Decommission Calico”
  • Date: 2026-05-12
  • Operator: Sean
  • Pre-removal Velero backup: pre-calico-removal-20260512-092425 (status: PartiallyFailed — only failure was nats-js-nats-2 PVC hitting Longhorn’s snapshotMaxCount: 10 limit after PR3 churn. Acceptable for rollback since NATS is HA-replicated. Follow-up: clean up old snapshots on that PVC.)

Phase A — Stop and remove Calico

Section titled “Phase A — Stop and remove Calico”
  1. Scaled tigera-operator Deployment to 0 (Task 2).
  2. Deleted Tigera-managed CRs in dependency order: Whisker → Goldmane → APIServer → Installation. Installation CR’s finalizer required manual patch because the operator was scaled to 0 (no finalizer worker). (Task 3)
  3. Deleted three Calico namespaces: calico-system, calico-apiserver, tigera-operator. calico-system required manual tigera.io/cni-protector finalizer removal from two ServiceAccounts (calico-cni-plugin, calico-node) for the namespace to terminate. (Task 4)
  4. Deleted all 31 Calico/Tigera CRDs cleanly. (Task 5)
  5. Deleted 4 ClusterRoles + 3 ClusterRoleBindings. The two CRs owned by the Installation (calico-cni-plugin, calico-node clusterroles and clusterrolebindings) carried the same tigera.io/cni-protector finalizer and required manual removal. (Task 6)

Phase B — Flip Cilium to production mode

Section titled “Phase B — Flip Cilium to production mode”
  1. Updated ansible/roles/cilium/defaults/main.yml (Task 7):
    • cilium_cni_custom_conf: true → false
    • cilium_cni_uninstall: false → true
    • cilium_policy_enforcement_mode: "never" → "default"
    • cilium_operator_unmanaged_pod_watcher_restart: false → true
  2. Rendered values via ansible localhost -m template and ran helm upgrade cilium ... directly (the ansible play header hosts: tp_cluster_controlplane[0] couldn’t connect because the local SSH agent had issues; all role tasks delegate_to: localhost so manual helm upgrade had identical effect). Revision 4 deployed. (Task 8)
  3. Deleted CiliumNodeConfig cilium-default and removed the io.cilium.migration/cilium-default=true label from all 8 nodes. (Task 9)
  4. Scaled system-upgrade-controller back to 1 replica. (Task 10)

Verification

Section titled “Verification”
  • cilium status --wait: OK across the board (8/8 agents, 8/8 envoy, 2/2 operators, Hubble Relay OK).
  • kubectl -n kube-system get configmap cilium-config -o jsonpath='{.data.enable-policy}' returns default.
  • Zero new unhealthy pods after enabling enforcement (only the 2 preexisting issues remain: karakeep/meilisearch CrashLoop, temporal-worker-controller-manager ImagePullBackOff — both predate the migration).
  • Bastion smoke test: kube-DNS query from bastion pod succeeded — CNP rules are enforcing as designed.

Repo cleanup

Section titled “Repo cleanup”
  • Deleted ansible/roles/calico/ (entire directory).
  • Removed Phase 5 (Calico) block from ansible/k3s-playbook.yml; renumbered Phase 5b → Phase 5.
  • Updated ansible/CLAUDE.md, argocd/CLAUDE.md, README.md, docs/index.md, docs/architecture/overview.md, docs/architecture/index.md, docs/reference/network.md, docs/reference/technologies.md, docs/reference/services.md to reflect Cilium as the sole CNI.
  • Removed Calico/Tigera namespaces from argocd/app-configs/velero/backup-schedule.yaml excludedNamespaces.
  • Deleted docs/operations/merlin.md (stale; merlin removed in PR #908).
  • Created docs/operations/cilium.md evergreen operations runbook.

Outstanding

Section titled “Outstanding”
  • PR5 sweeps argocd/app-configs/cf-ssh-bastion/calico-network-policy.yaml (now a silent no-op since Calico CRDs are deleted).
  • Follow-up Velero hygiene: pvc-2b4682ff-... (nats-2 data) has 10 Longhorn snapshots — at the limit. Periodically clean up old snapshots or raise the snapshotMaxCount setting on that volume.

Migration complete

Section titled “Migration complete”

All 5 PRs of the Calico→Cilium migration project landed between 2026-05-09 and 2026-05-12. Cluster runs Cilium 1.19.3 in production mode with policyEnforcementMode: default.