Calico → Cilium Migration

Status: Design approved, ready for implementation planning Date: 2026-05-10 Owner: Sean Workspace: .worktrees/feat-cilium-migration

Summary

Replace Calico (currently the cluster CNI, installed via Tigera operator from the ansible/roles/calico role) with Cilium, using Cilium’s official live per-node migration path. The migration is zero-downtime: Cilium is installed alongside Calico, each node is drained and migrated one at a time, then Calico is decommissioned. The only Calico-specific resource in the repo applied to the cluster is argocd/app-configs/cf-ssh-bastion/calico-network-policy.yaml, which is rewritten as a CiliumNetworkPolicy. (Note: docs/operations/merlin.md references Calico policies, but merlin was removed in PR #908 — those docs are stale and get deleted as part of PR4’s documentation cleanup.)

This project ships Cilium as a like-for-like CNI replacement plus Hubble + Gateway API CRDs. It does NOT replace kube-proxy and does NOT migrate any Traefik IngressRoute to HTTPRoute — those are deliberate follow-up projects unlocked by this work.

Goals

Cilium as the sole CNI, installed by ansible/roles/cilium mirroring the Calico role’s lifecycle position.
Zero-downtime migration via Cilium’s k8s live install migration workflow.
Hubble (relay + UI) installed, cluster-internal only.
Gateway API CRDs + Cilium gateway controller installed (not used yet — Traefik remains the ingress).
Calico fully removed: Tigera operator, all projectcalico.org CRDs, three Calico namespaces, the ansible/roles/calico directory, and references in Velero excludes / docs / CLAUDE.md files.
The cf-ssh-bastion network policy is translated to CiliumNetworkPolicy with behavioral parity (no security regression, no scope creep).

Non-goals

kube-proxy replacement (kubeProxyReplacement: true + --disable=kube-proxy in k3s). Has independent risk surface (MetalLB interaction, k3s node restarts); deferred to a follow-up project. The cilium_kube_proxy_replacement variable is wired through defaults/main.yml for a single-line future flip.
Traefik retirement / IngressRoute → HTTPRoute migration. Out of scope. Gateway API CRDs are installed in PR2 so this can happen later without chart reconfiguration.
Native routing (autoDirectNodeRoutes: true). Single-subnet topology makes this viable without BGP, but VXLAN is the lower-risk day-1 choice. Wired as a variable for future migration. (Note: not a runtime-safe flip — switching tunnel → native requires a planned agent rollout and may briefly interrupt pod traffic; treat it as its own future project.)
Cilium ClusterMesh. Single-cluster homelab; no multi-cluster need.
L7 / DNS-based NetworkPolicy. cf-ssh-bastion is translated L3/L4 only; DNS-egress hardening (e.g., allow github.com only) is a separate security-hardening PR.
mTLS via SPIFFE. Cilium supports it; nothing in the cluster uses it.
Decommissioning kube-vip or MetalLB. Both are unaffected.
Building Cilium observability dashboards / ServiceMonitor. No Calico ServiceMonitor exists today (verified — no *calico* or *tigera* ServiceMonitor in argocd/app-configs/). Cilium’s Prometheus integration is a separate observability project that comes after this migration.
External Hubble UI exposure (Authentik-protected hubble.fzymgc.house). Internal access only, matching current Whisker usage. Future optional PR.

Constraints (from cluster topology + memory)

Cluster scope: 8 nodes (tpi-alpha-1..4 + tpi-beta-1..4), all on 192.168.20.0/24. Single L2 segment. Control plane is tpi-alpha-1..3; workers are tpi-alpha-4 + tpi-beta-1..4.
No BGP. The Firewalla router has no first-class BGP support and no UI for peer configuration; BGP-based routing modes are off the table.
No remote nodes. seattle (192.168.23.109) is referenced in the bastion’s egress allowlist but is an SSH destination, not a cluster member.
k3s bootstrap requires a CNI before Ready. Cilium MUST be installed by Ansible (not ArgoCD) at bootstrap time, like Calico is today.
Branch protection on main. All work via feature branches + PRs.
GitOps for in-cluster state. All changes that touch argocd/app-configs/* go through ArgoCD sync, not kubectl apply.
mkdocs.yml nav: is hand-curated (per memory) — any new doc under docs/operations/ MUST get an explicit nav entry in the same PR.
Cilium’s upstream migration procedure marks “changing IP families” as untested. Dual-stack (v4+v6 on both sides, no family change) is not the same as a family change, so it should work, but is a known-untested area. PR2 installs Cilium with dual-stack enabled; mid-PR3 we validate IPv6 on the first migrated node before proceeding. Fallback documented in PR3: disable v6 + accept temporary v6 pod-networking outage until a follow-up project re-enables it.
NetworkPolicy enforcement is fully disabled during migration (policyEnforcementMode: "never" per upstream). This is non-negotiable per the official procedure; otherwise Cilium drops legitimate cross-CNI flows. The bastion’s protection model therefore has two distinct phases (Calico enforces until the bastion’s node migrates, then unprotected until PR4’s policyEnforcementMode: default flip — see “Bastion policy strategy” section).
system-upgrade-controller is active (argocd/app-configs/system-upgrade/). PR3 pre-flight scales it to 0 replicas; PR4 restores it.
tigera-operator reconciles Installation — PR4 must scale it to 0 before deleting the Installation resource, otherwise the operator fights the deletion.
No new NetworkPolicy or CiliumNetworkPolicy resources may merge between PR2 and PR4. During the migration window, policyEnforcementMode: "never" makes Cilium ignore all policies cluster-wide. A NetworkPolicy merged in this window would silently fail to enforce, creating a security regression hidden until PR4. Block via PR template checkbox / reviewer attention. Add a freeze note to docs/operations/cilium-migration.md for the migration window. The bastion’s pre-staged CiliumNetworkPolicy is exempt — it was approved as part of PR2.
Cilium operator runs 2 replicas on control-plane nodes. During PR3 control-plane migration (alpha-3 → alpha-2 → alpha-1), one of those nodes hosting an operator pod will drain. The operator pod reschedules to the remaining control-plane node, briefly leaving 1 replica. This is tolerable because operator is not on the data path (only reconciles CiliumNode resources and IP pool allocation). Don’t migrate two control-plane nodes in parallel.

Gateway API ownership transfer (added during PR1 execution)

The cluster has had an independent argocd/app-configs/gateway-api/ Application installed since 2025-08, managing Gateway API v1.5.1 Experimental CRDs directly from upstream (config/crd/experimental?ref=v1.5.1). It also installs a safe-upgrades.gateway.networking.k8s.io ValidatingAdmissionPolicy that blocks any CRD install with bundle-version < v1.5.0.

Audit at PR1 execution time confirmed: zero Gateway API resources are in use anywhere in the cluster. No Gateway, GatewayClass, HTTPRoute, GRPCRoute, TLSRoute, TCPRoute, UDPRoute, ReferenceGrant, or BackendTLSPolicy resources exist. The Experimental channel installation has been pure cluster-state overhead with no consumer.

Sequencing fact: in a fresh cluster bootstrap, Cilium installs via Ansible BEFORE ArgoCD comes up. So Cilium must own Gateway API CRDs — there’s no time window where the ArgoCD-managed gateway-api Application could install them before Cilium needs them. The pre-existing setup worked only because the cluster was bootstrapped with Calico (no Gateway API dependency in the CNI), ArgoCD came up, and then Gateway API was added independently.

Decision: The Cilium role takes over Gateway API CRD ownership. The legacy argocd/app-configs/gateway-api/ Application is removed. CRD version aligns with Cilium 1.19’s documented requirement (v1.4.1 Standard). The Experimental- channel CRDs not used by Cilium (TLSRoute, TCPRoute, UDPRoute, BackendTLSPolicy, ListenerSets, XMeshes, XBackendTrafficPolicies) are intentionally dropped — none are in use; they can be re-added as a future project if needed.

Cluster-side prep (one-time, executed pre-PR2):

kubectl delete validatingadmissionpolicybinding safe-upgrades.gateway.networking.k8s.io
kubectl delete validatingadmissionpolicy safe-upgrades.gateway.networking.k8s.io
kubectl delete crd \
  listenersets.gateway.networking.k8s.io \
  xmeshes.gateway.networking.x-k8s.io \
  xbackendtrafficpolicies.gateway.networking.x-k8s.io \
  tcproutes.gateway.networking.k8s.io \
  udproutes.gateway.networking.k8s.io \
  tlsroutes.gateway.networking.k8s.io \
  backendtlspolicies.gateway.networking.k8s.io \
  gateways.gateway.networking.k8s.io \
  gatewayclasses.gateway.networking.k8s.io \
  httproutes.gateway.networking.k8s.io \
  grpcroutes.gateway.networking.k8s.io \
  referencegrants.gateway.networking.k8s.io

Why kubectl-delete and not ArgoCD-cascade: the legacy gateway-api Application lacks metadata.finalizers: [resources-finalizer.argocd.argoproj.io], so removing it from cluster-app/templates/ deletes the Application object but does NOT cascade to managed resources (per repo memory argocd-app-deletion-doesnt-cascade-without-finalizer). The CRDs would be orphaned. The kubectl-delete is the explicit, atomic cleanup.

Verification post-cleanup:

kubectl get crd | rg 'gateway\.networking' | wc -l returns 0
kubectl get validatingadmissionpolicy | rg gateway returns empty

After this prep, PR2’s Ansible run (--tags cilium-install) applies the v1.4.1 Standard CRDs via the role’s Phase 1 task without policy interference.

Update: the safe-upgrades.gateway.networking.k8s.io ValidatingAdmissionPolicy is re-vendored into ansible/roles/cilium/files/gateway-api-crds/safe-upgrades-policy.yaml with the bundle-version floor adjusted from upstream’s v1.5.0 to v1.4.0 to match Cilium 1.19’s required Gateway API version. The policy is applied by the role’s Phase 1 task, restoring the cluster’s guard against unsafe Gateway API CRD downgrades.

Decisions (locked in)

Decision	Choice
Migration approach	Cilium’s official live per-node migration (dual-CNI, drain/migrate one node at a time)
Deployment mechanism	Ansible role `ansible/roles/cilium` mirroring `ansible/roles/calico` lifecycle position
Install method	Helm via `kubernetes.core.helm` module (vs Calico’s `kubernetes.core.k8s` against operator manifests)
Cilium namespace	`kube-system` (upstream default; non-standard placement invites footguns)
Pod CIDR	New range `10.245.0.0/16` (IPv4) + `fd00:10:245::/48` (IPv6), adopted permanently — not reclaiming Calico’s `10.42.0.0/16`
Service CIDR	Unchanged (`10.43.0.0/16`, k3s default)
Routing mode	Tunnel, VXLAN protocol — closest functional match to Calico’s VXLANCrossSubnet
Cilium VXLAN tunnel port	`8473` (distinct from Calico’s default 4789, per upstream migration requirement)
`cni.customConf` during migration	`true` (prevents Cilium from writing CNI conf at install; CiliumNodeConfig writes per-node when label flips)
`cni.uninstall` during migration	`false` (preserves CNI conf on agent shutdown so `helm uninstall` doesn’t strand a node)
`policyEnforcementMode` during migration	`"never"` (upstream-mandated; otherwise Cilium drops legitimate cross-CNI traffic)
`bpf.hostLegacyRouting` during migration	`true` (required for cross-CNI pod-to-pod connectivity in the dual-CNI window)
`operator.unmanagedPodWatcher.restart` during migration	`false` (prevents Cilium from restarting Calico-managed pods)
kube-proxy replacement	`"false"` (string, not bool — Cilium 1.19 schema); separate follow-up project
Hubble UI access	Cluster-internal only (no IngressRoute)
Gateway API CRDs	Owned by the Cilium Ansible role (v1.4.1 Standard: `GatewayClass`, `Gateway`, `HTTPRoute`, `GRPCRoute`, `ReferenceGrant`) — vendored under `ansible/roles/cilium/files/gateway-api-crds/`. Cilium’s chart does NOT auto-install them; the role applies them in Phase 1 before Helm install. The legacy `argocd/app-configs/gateway-api/` Application (v1.5.1 Experimental, in repo since 2025-08, unused) is REMOVED in PR1 — see “Gateway API ownership transfer” section below.
Gateway API controller	Cilium gateway controller installed (`gatewayAPI.enabled: true`); GatewayClass is `cilium`; no routes migrated
cf-ssh-bastion policy dialect	`CiliumNetworkPolicy` (cilium.io/v2) — superset of current Calico features
PR breakdown	5 PRs (role / install-dual / migrate-iterative / decom-calico / sweep-old-policy)
Bastion policy strategy	See “Bastion policy strategy” section. Short version: Calico policy stays in place during migration (enforces on Calico nodes); Cilium policy added in PR2 but does not enforce until PR4’s `policyEnforcementMode: default` flip. Calico policy file deleted in PR5.
Node migration order in PR3	All workers first (`tpi-beta-1..4`, then `tpi-alpha-4`), then control plane reverse order (`tpi-alpha-3..1`) so kube-vip leader migrates last. Bastion’s host node migrates last among workers to minimize unprotected window.
Pre-PR4 merge gate	Mechanical: see PR4 “Merge gate” subsection (7 specific checks).

Target state (post-PR5)

Aspect	Value
CNI	Cilium (sole), installed by `ansible/roles/cilium`
Cilium version	1.19.3 (latest stable as of 2026-05-11; verified via `helm search repo cilium/cilium --versions`)
Cilium namespace	`kube-system`
Pod CIDR (IPv4)	`10.245.0.0/16`, `/24` per node
Pod CIDR (IPv6)	`fd00:10:245::/48`, `/64` per node
Service CIDR	`10.43.0.0/16` (unchanged)
Dual-stack	Preserved
Routing	Tunnel mode, VXLAN protocol, port 8473
`cni.customConf`	`false` (post-cutover; PR2 sets `true`)
`cni.uninstall`	`true` (default; PR2 sets `false`)
`policyEnforcementMode`	`default` (post-cutover; PR2 sets `"never"`)
`bpf.hostLegacyRouting`	`true` (retained permanently in PR4; flip to `false` is a follow-up project with scheduled maintenance window)
`operator.unmanagedPodWatcher.restart`	`true` (default; PR2 sets `false`)
`CiliumNodeConfig` `cilium-default`	Deleted post-cutover (no longer needed)
kube-proxy	Still running (replacement = follow-up project)
Hubble	Relay + UI enabled, cluster-internal only
Gateway API CRDs	`kubernetes-sigs/gateway-api v1.4.1` Standard channel, pre-installed by Ansible role
Gateway API safe-upgrades policy	`safe-upgrades.gateway.networking.k8s.io` `ValidatingAdmissionPolicy` installed (guards against CRD downgrades)
Gateway API controller	Cilium gateway controller installed; GatewayClass `cilium`; zero routes
Calico artifacts	None: no operator, no CRDs, no namespaces, no `ansible/roles/calico`, no Velero excludes for Calico namespaces
MetalLB	Unchanged
kube-vip	Unchanged
Traefik	Unchanged

Phase breakdown (PR series)

Each PR is independently revertable. PR3 is iterative (per-node) and may span multiple sessions; its repo artifact is the migration runbook plus final checkpoint.

PR1 — Add `ansible/roles/cilium` (code only, no install)

Add ansible/roles/cilium/{tasks,defaults,meta,handlers,templates,files}/. (Calico role today is tasks,defaults,meta only; Cilium adds handlers/ for the rare agent-restart case, templates/ for the Helm values.yaml.j2 and the CiliumNodeConfig manifest, and files/ for the pinned Gateway API v1.4.1 CRD YAML downloaded at PR1 author time. Vendoring the CRDs into the role avoids network dependencies during cluster bootstrap.)
Helm-based install via kubernetes.core.helm. Helm repo https://helm.cilium.io/, chart cilium/cilium, version pinned in defaults/main.yml.
Templated values.yaml.j2 rendering all upstream-required migration values (see “Defaults shape” section below) plus the locked feature scope.
Templated cilium-node-config.yaml.j2 — applies a CiliumNodeConfig CR with selector io.cilium.migration/cilium-default=true and defaults write-cni-conf-when-ready=/host/etc/cni/net.d/05-cilium.conflist, custom-cni-conf=false, cni-chaining-mode=none, cni-exclusive=true. This is the mechanism that toggles each node from non-exclusive to exclusive when the migration label is applied in PR3.
Gateway API CRDs (v1.4.1) vendored under ansible/roles/cilium/files/gateway-api-crds/: gatewayclasses, gateways, httproutes, grpcroutes, referencegrants from https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/*.yaml, plus safe-upgrades-policy.yaml (safe-upgrades.gateway.networking.k8s.io ValidatingAdmissionPolicy that guards against CRD downgrades).
(Optional, not done in PR1) Override cilium_* variables in ansible/inventory/group_vars/tp_cluster_nodes.yml. The role defaults are self-sufficient for PR1 through PR4; group_vars overrides are only needed if a future operator wants to deviate from the spec-locked values.
Add a new phase (“Phase 5b: Cilium CNI”) to ansible/k3s-playbook.yml, tagged cilium-install. Calico’s Phase 5 is left intact.
Update ansible/CLAUDE.md Roles Reference and Phases tables to include cilium alongside calico for the duration of the migration.
No cluster change yet.

Cluster impact: None. Rollback: git revert.

PR2 — Install Cilium alongside Calico in migration mode

Pre-flight (in order):

velero backup create pre-cilium-install-$(date +%Y%m%d-%H%M%S) --wait.
Verify no NetworkPolicy resources are pending creation (kubectl get networkpolicy,networkpolicies.crd.projectcalico.org,globalnetworkpolicies.crd.projectcalico.org -A) — the existing set continues to enforce via Calico during migration.
Verify Gateway API CRDs are NOT already installed under a different version (kubectl get crd gatewayclasses.gateway.networking.k8s.io -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}'). If present at a non-1.4.1 version, hold PR2 until reconciled with Cilium 1.19’s requirement.

Operator workstation prerequisite: install cilium-cli before PR2 begins (used by PR2 verification and PR3 stability gates). On macOS: brew install cilium-cli. On Linux: see cilium/cilium-cli releases for the version matching Cilium 1.19.x. (This prerequisite is documented in docs/operations/cilium.md once PR4 creates that file, but for PR2/PR3 execution the operator must already have it installed locally.)

Install steps (executed by ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium-install):

Apply pinned Gateway API v1.4.1 Standard channel CRDs from roles/cilium/files/gateway-api-crds/. (Pre-Cilium prerequisite — chart doesn’t auto-install.)
Helm install cilium/cilium chart version 1.19.3 into kube-system with the full migration-mode values block (see “Migration-mode Helm values” below).
Apply the CiliumNodeConfig resource (selector matches the migration label, which no nodes have yet — so it applies to nothing initially).
Wait for Cilium DaemonSet Ready on all 8 nodes (numberReady == desiredNumberScheduled).
Wait for Hubble Relay Ready (Deployment readyReplicas > 0).
Wait for cilium-operator Ready.
Add argocd/app-configs/cf-ssh-bastion/cilium-network-policy.yaml (new file) to the bastion’s kustomization. Note: this policy is created but does NOT enforce because policyEnforcementMode: "never" is set cluster-wide during migration. It exists pre-staged so that PR4’s policyEnforcementMode: default flip activates it cluster-wide atomically. See “Bastion policy strategy” section.

Migration-mode Helm values (rendered from `values.yaml.j2`)

operator:
  replicas: 2
  unmanagedPodWatcher:
    restart: false
  # Pin operator to control-plane nodes - they're stable and the operator
  # is not on the data path. Required for "don't migrate two operator-hosting
  # nodes in parallel" constraint per spec.
  nodeSelector:
    node-role.kubernetes.io/control-plane: ""
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
routingMode: tunnel
tunnelProtocol: vxlan
tunnelPort: 8473
cni:
  customConf: true
  uninstall: false
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList: ["10.245.0.0/16"]
    clusterPoolIPv4MaskSize: 24
    clusterPoolIPv6PodCIDRList: ["fd00:10:245::/48"]   # dual-stack supplement (see below)
    clusterPoolIPv6MaskSize: 64                         # dual-stack supplement (see below)
ipv4:
  enabled: true                                         # chart default (explicit)
ipv6:
  enabled: true                                         # dual-stack supplement (see below)
policyEnforcementMode: "never"
bpf:
  hostLegacyRouting: true
kubeProxyReplacement: "false"
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
    service:
      type: ClusterIP
gatewayAPI:
  enabled: true

Most values above are upstream-mandated for the migration window (operator.unmanagedPodWatcher.restart, routingMode, tunnelProtocol, tunnelPort, cni.customConf, cni.uninstall, ipam.mode, clusterPoolIPv4PodCIDRList, policyEnforcementMode, bpf.hostLegacyRouting) per Cilium’s k8s-install-migration docs.

Deliberate supplements to upstream (clearly marked # dual-stack supplement above):

ipv6.enabled: true and the clusterPoolIPv6* entries are NOT in upstream’s prescribed block. They are this project’s choice to preserve cluster-wide dual-stack. Upstream marks IP-family CHANGES as untested but is silent on family-preservation, which is what we’re doing. PR3’s first-node IPv6 validation gate + IPv6 fallback runbook contain the risk.
kubeProxyReplacement: "false", hubble.*, gatewayAPI.enabled: true are this project’s feature-scope decisions, orthogonal to the migration procedure itself.

CiliumNodeConfig (rendered from `cilium-node-config.yaml.j2`)

apiVersion: cilium.io/v2
kind: CiliumNodeConfig
metadata:
  namespace: kube-system
  name: cilium-default
spec:
  nodeSelector:
    matchLabels:
      io.cilium.migration/cilium-default: "true"
  defaults:
    write-cni-conf-when-ready: /host/etc/cni/net.d/05-cilium.conflist
    custom-cni-conf: "false"
    cni-chaining-mode: "none"
    cni-exclusive: "true"

Until PR3 starts applying the label to nodes, this CR has no effect.

Verification

Cilium DaemonSet Ready on all 8 nodes.
Hubble Relay + cilium-operator Ready.
cilium status --wait returns OK (requires cilium-cli installed locally; see PR1’s docs/operations/cilium.md for install instructions).
No workload pod restart (kubectl get pods -A | rg -v '^kube-system|^calico-' | head — none should show RESTARTS > 0 attributable to PR2).
kubectl get pods -A -o wide | rg '10\.245\.' returns empty (no pods on Cilium CIDR yet — confirms migration mode is intact).

Cluster impact: Cilium pods running on all nodes; no workload pods migrated yet. Calico remains in full enforcement of all existing NetworkPolicies.

Rollback: Only valid before any PR3 node has been labeled (once labels exist, the rollback procedure is PR3’s per-node fallback, not PR2’s).

# 1. Uninstall Cilium chart (cni.uninstall: false means no CNI conf is removed)
helm uninstall cilium -n kube-system

# 2. Delete the migration-mode CR
kubectl delete ciliumnodeconfig cilium-default -n kube-system

# 3. Delete the pre-staged bastion CiliumNetworkPolicy file from kustomization
#    (ArgoCD will then prune it from the cluster on next sync)

# 4. Delete the Gateway API CRDs that PR2 installed (otherwise they persist
#    as orphaned cluster state):
kubectl delete -f ansible/roles/cilium/files/gateway-api-crds/

Calico continues uninterrupted throughout. No node loses CNI from this rollback.

PR3 — Per-node migration (iterative)

Tracked as beads issues (one per node). Order:

tpi-beta-1 → tpi-beta-2 → tpi-beta-3 (3 workers, none host the bastion)
First-node validation gate (24h soak; see “Stability gate” below)
tpi-beta-4 → tpi-alpha-4 (remaining 2 workers; bastion’s host node migrates as part of this group, which is when the bastion is briefly unprotected)
All-workers validation gate (24h soak)
tpi-alpha-3 → tpi-alpha-2 → tpi-alpha-1 (control plane reverse order; kube-vip leader migrates last)

Pre-flight (once, before first node)

Pause system-upgrade-controller: kubectl scale -n system-upgrade deploy/system-upgrade-controller --replicas=0. Restore in PR4.
Enumerate PodDisruptionBudgets cluster-wide: kubectl get pdb -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name) minAvailable=\(.spec.minAvailable) maxUnavailable=\(.spec.maxUnavailable) selector=\(.spec.selector)"'. Review each: any PDB with minAvailable equal to the matched pod count (e.g., a 1-replica StatefulSet with minAvailable: 1) will stall kubectl drain indefinitely. Known-affected workloads in this cluster: Longhorn (replica anti-affinity may stall drain of a node hosting the only healthy replica of a volume), CNPG/Dolt/Temporal (3-replica StatefulSets — fine for single-node drain), NATS (clustered — fine). Mitigation per-node: if drain stalls > 10 min on a node, either resolve the underlying replica health, or fall back to kubectl drain --disable-eviction (uses delete API, bypasses PDB — accept the brief unavailability).
Fresh Velero backup if > 24h since PR2 backup: velero backup create pre-pr3-migration-$(date +%Y%m%d-%H%M%S) --wait.
kubectl cluster-info dump > /tmp/pre-pr3-cluster-state.yaml for post-migration audit.
Verify CiliumNodeConfig cilium-default is healthy: kubectl get ciliumnodeconfig -n kube-system cilium-default -o yaml returns the resource cleanly.
etcd-leader awareness (for control-plane node migrations only): k3s uses embedded etcd; the leader is one of tpi-alpha-1..3. Draining the leader triggers automatic re-election (typically <2s). kube-vip handles the API VIP, so external API access is uninterrupted. Do not migrate two CP nodes in parallel — that would risk quorum loss. The PR3 CP order (alpha-3 → alpha-2 → alpha-1) is sequential to honor this. No manual etcdctl move-leader is required for k3s; verify post-drain via kubectl get nodes and cilium status --wait.

Per-node procedure

All ${NODE} references below are template placeholders. Sequence matches Cilium’s upstream migration docs, including the explicit reboot step.

# Set the node name once per iteration; substitute with actual node
NODE="tpi-beta-1"   # change for each node

# 1. Cordon
kubectl cordon ${NODE}

# 2. Drain workload pods (DaemonSets stay; their pod sandboxes get rebuilt at reboot)
kubectl drain --ignore-daemonsets --delete-emptydir-data ${NODE}

# 3. Apply migration label — CiliumNodeConfig selector matches it
kubectl label node ${NODE} --overwrite "io.cilium.migration/cilium-default=true"

# 4. Restart the Cilium agent on this node so it picks up the per-node config
#    and writes /etc/cni/net.d/05-cilium.conflist
kubectl -n kube-system delete pod \
  --field-selector spec.nodeName=${NODE} \
  -l k8s-app=cilium

# 5. Wait for the new agent pod to be Ready on this node
kubectl -n kube-system rollout status ds/cilium -w --timeout=5m

# 6. Verify the new CNI conf was written, WITHOUT requiring SSH access:
#    (kubectl debug works on every k3s node; SSH is not a prerequisite.)
#    Note: NO -it flag — script context has no TTY and -it would fail.
kubectl debug node/${NODE} --image=busybox -- chroot /host ls -1 /etc/cni/net.d/
#    Expected output (lexical order): 05-cilium.conflist before any 10-calico.conflist.

# 7. REBOOT the node — upstream-required step. Use the repo's existing
#    ansible.builtin.reboot idiom (see ansible/reboot-nodes-playbook.yml)
#    which handles connection-loss-and-reconnect cleanly. A bare
#    `ssh sudo systemctl reboot` would return non-zero under `set -e`
#    because sshd kills the session mid-command.
#    Rationale: DaemonSet pod sandboxes (calico-node, alloy, longhorn, etc.)
#    were created via Calico's CNI socket and won't transition to Cilium's
#    eBPF management without kubelet rebuilding them. The reboot rebuilds
#    every pod sandbox on the node through the new (Cilium) CNI conf.
ansible -i ansible/inventory/hosts.yml --become --limit ${NODE} \
  -m ansible.builtin.reboot \
  -a 'reboot_timeout=600 post_reboot_delay=30' \
  all
#    The reboot module waits for SSH to return; supplement with a node-Ready
#    check (TPI compute modules take longer to come back than bare-metal —
#    timeout generously):
kubectl wait --for=condition=Ready --timeout=10m node/${NODE}

# 8. cilium-cli verification post-reboot
cilium status --wait

# 9. Uncordon
kubectl uncordon ${NODE}

# 10. Smoke test: schedule a verify pod onto the freshly-migrated node and
#     check it gets a Cilium IP and can reach the API server
kubectl -n kube-system run --attach --rm --restart=Never verify-network \
  --overrides='{"spec": {"nodeName": "'${NODE}'", "tolerations": [{"operator": "Exists"}]}}' \
  --image=ghcr.io/nicolaka/netshoot:v0.8 -- \
  /bin/bash -c 'ip -br addr | rg "10\\.245\\." && curl -fsk https://$KUBERNETES_SERVICE_HOST/healthz && echo OK'
# Expected: "OK" on stdout, exit 0.

# 11. Cross-CNI smoke (skip on final node — no Calico peer remains)
#     From a Calico-managed pod to a pod just rescheduled on ${NODE} (Cilium IP):
kubectl exec -n <some-still-calico-ns> <calico-pod> -- curl -fsm 3 <cilium-pod-ip>:<port>
#     And the reverse direction.

# 12. Verify all pod sandboxes on the node are now Cilium-managed.
#     Filter out: header, host-network pods (node IP 192.168.20.x),
#     pending pods (<none>), Cilium IPv4 pods (10.245.x), Cilium IPv6 pods
#     (fd00:10:245::x), and IPv6 link-local entries some pods report.
kubectl get pods -A -o wide --field-selector spec.nodeName=${NODE} --no-headers \
  | rg -v '\s(192\.168\.20\.|<none>|10\.245\.|fd00:10:245)'
# Expected: empty (all remaining pods are either host-network, pending, or on
# the Cilium pod CIDR). Any line in output is a still-Calico pod that needs
# investigation (typically: a DaemonSet pod whose sandbox didn't rebuild on
# reboot — manually delete the pod to force reschedule).

# 13. Append migration log entry with timestamp + verification output

Note on the reboot step: upstream’s migration doc shows docker restart $NODE because their example uses a kind cluster. For real k3s nodes, an OS reboot serves the same purpose. If a node can’t be rebooted (long-running stateful workloads, planned outage constraints), the alternative is to drain ALL pods (not just workload pods) via kubectl drain --force --delete-emptydir-data --ignore-daemonsets=false followed by manual deletion of DS pods on the node — but this is more disruptive than a reboot. The spec recommends reboot.

Stability gate (after node 1 and after all workers)

“Stable” means ALL of:

No Cilium agent pod restarts on migrated nodes (kubectl get pods -n kube-system -l k8s-app=cilium --field-selector spec.nodeName=${NODE} -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}' returns 0).
No BPFMapPressure warnings in Cilium agent logs (kubectl logs -n kube-system -l k8s-app=cilium --tail=1000 | rg -i 'pressure|throttl|drop' shows no recent matches).

No application pod restarts on migrated nodes attributable to networking. kubectl get events does NOT support --since; bound the window shell-side via jq (expected: empty output for the past 24h):

SINCE=$(date -u -d '24 hours ago' +%s 2>/dev/null || date -u -v-24H +%s)
kubectl get events -A --field-selector type=Warning -o json \
  | jq -r --arg node "${NODE}" --argjson since "${SINCE}" \
    '.items[]
     | select(.reason=="NetworkNotReady")
     | select(.involvedObject.kind=="Node" and .involvedObject.name==$node)
     | select((.lastTimestamp // .eventTime | fromdateiso8601) > $since)
     | .lastTimestamp'

Hubble flows from the migrated node exist (expected: flow count > 0; spot-check shows valid source/dest pairs). Note: -o jsonpath is NOT a valid cilium hubble observe output; use -o json. Filtering with --node ${NODE} returns only flows on that node:
Terminal window
```
cilium hubble observe --last 1000 --node ${NODE} -o json | wc -l
cilium hubble observe --last 50 --node ${NODE}
```
IPv6 validation (after first node only): kubectl exec from a Cilium-migrated pod, verify IPv6 pod-to-pod reachability across the cluster (one pod on the migrated node, one on a still-Calico node). If IPv6 fails, see “IPv6 fallback” below before proceeding.

IPv6 fallback (only invoked if first-node IPv6 validation fails)

Upstream marks IP-family changes as untested; dual-stack preservation should work but if it doesn’t:

Halt PR3 at the first node.
Update values.yaml.j2: ipv6.enabled: false, drop the IPv6 cluster-pool entries.
helm upgrade cilium cilium/cilium -n kube-system -f new-values.yaml.
Re-run the first-node migration. IPv6 pod connectivity is lost cluster-wide; document this in cilium-migration.md and file a follow-up issue for a future v6-re-enable project.

Per-node verification + log entry

Append to docs/operations/cilium-migration.md:

## ${NODE} — migrated ${TIMESTAMP}

- Verify-network pod IP: 10.245.x.y ✓
- API server reachable: OK ✓
- Cross-CNI smoke (Cilium ⇆ Calico): OK ✓
- IPv6 (first node only): pod-to-pod v6 ping OK / [FAIL — fallback applied]
- Agent restart count post-migration: 0
- Hubble flows: <count> from this node in last hour

Cluster impact: Pods on migrated nodes use Cilium; the rest still use Calico. NetworkPolicy enforcement is fully disabled cluster-wide (intentionally, per upstream) until PR4. Calico’s policies on Calico-managed pods are still evaluated by Calico’s typha/felix, but Cilium does NOT enforce on migrated pods.

Per-node rollback (nodes 1–7, until PR4 merges) — best-effort, not upstream-supported:

# 1. Remove the migration label so CiliumNodeConfig stops applying defaults
kubectl label node ${NODE} io.cilium.migration/cilium-default-

# 2. Drain again (workload pods will reschedule)
kubectl drain --ignore-daemonsets --delete-emptydir-data ${NODE}

# 3. Restart the Cilium agent FIRST (with label gone, CiliumNodeConfig
#    no longer matches, so the new agent pod won't re-apply the per-node
#    `cni-exclusive: true` override and won't rewrite the conflist).
#    Doing this BEFORE the rm avoids a race where the still-running agent
#    re-writes the file we just deleted.
kubectl -n kube-system delete pod \
  --field-selector spec.nodeName=${NODE} \
  -l k8s-app=cilium
kubectl -n kube-system rollout status ds/cilium -w --timeout=5m

# 4. Now remove the Cilium CNI conf so kubelet falls back to 10-calico.conflist:
#    (Use kubectl debug node/ since SSH/sudo via raw ssh has the same
#     connection-killed-mid-command issue called out in the migration procedure.)
kubectl debug node/${NODE} --image=busybox -- \
  chroot /host rm -f /etc/cni/net.d/05-cilium.conflist

# 5. Reboot via Ansible's reboot module (same idiom as the forward procedure)
#    to rebuild all pod sandboxes through Calico's CNI:
ansible -i ansible/inventory/hosts.yml --become --limit ${NODE} \
  -m ansible.builtin.reboot \
  -a 'reboot_timeout=600 post_reboot_delay=30' \
  all
kubectl wait --for=condition=Ready --timeout=10m node/${NODE}

# 6. Uncordon
kubectl uncordon ${NODE}

This procedure is NOT in upstream’s docs (upstream provides no single-node rollback). It works in principle because Calico’s calico-node DS pod has been running on the node throughout the migration (drain ignored DSes), so the Calico CNI socket and felix iptables rules are still in place. The file-deletion + agent-restart + reboot sequence puts the node back into a Calico-managed state. Validate end-to-end on a non-critical worker before trusting it as a rollback for a control-plane node.

Calico-node DS pod on migrated nodes: throughout PR3, calico-node DaemonSet pods continue running on every node (drain ignores DSes). Felix’s iptables rules persist but are harmless because pod traffic on migrated nodes is now Cilium-eBPF-managed (bypassing iptables for the pod-to-pod path). The calico-node pods are removed cluster-wide in PR4 Phase A step 3 when the calico-system namespace is deleted.

Final node (the 8th, tpi-alpha-1) has no Calico peer to fall back to. Rollback at that point is helm uninstall cilium + kubectl delete ciliumnodeconfig cilium-default, which would orphan 7 migrated nodes. Treat the final node migration as a one-way step; only proceed when all earlier nodes have passed both stability gates above.

PR4 — Decommission Calico

Pre-flight (in order)

velero backup create pre-calico-removal-$(date +%Y%m%d-%H%M%S) --wait — captures application-namespace state at the point Calico is still installed but every node is on Cilium. Per the “Velero rollback story” caveat above, this backup does NOT capture Cilium state (in kube-system) or the bastion’s new policy — those are Velero-excluded. Use it for app-data restore only.
Verify all 8 nodes have been through PR3 successfully — no nodes still on Calico CIDR (10.42.0.0/16).
Verify the all-workers stability gate from PR3 passed (no agent restarts, no BPFMapPressure warnings, IPv6 validated).
Confirm no new NetworkPolicy or CiliumNetworkPolicy resources merged between PR2 and now (other than the pre-staged bastion policy from PR2).

Merge gate (mechanical, all must pass)

kubectl get pods -A -o wide --field-selector 'status.phase=Running' -o json | jq -r '.items[] | select(.spec.hostNetwork != true) | "\(.metadata.namespace)/\(.metadata.name) \(.status.podIP)"' | rg '10\.42\.' returns empty (no non-host-network pods on the Calico CIDR).
kubectl get ciliumnodes lists all 8 nodes with Status.IPAM.PodCIDRs populated.
cilium endpoint list count ≈ count of non-host-network pods (kubectl get pods -A --field-selector 'status.phase=Running' -o json | jq '[.items[] | select(.spec.hostNetwork != true)] | length'); discrepancy means orphaned endpoints.
All workers + 7-of-8 nodes passed PR3 stability gate (last node migrates as part of PR4 if not already done).
Hubble flows from every node — but quiet nodes may not show up in a flow buffer. Use the CiliumNode resource as the authoritative per-node liveness check instead: kubectl get ciliumnodes -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | wc -l returns 8. Spot-check Hubble traffic with cilium hubble observe --last 500 -o json | jq -r '.node_name' | sort -u | wc -l (should be ≥ 6 of 8 nodes if the cluster has typical traffic).
Velero backup completed Successful.
tigera-operator Deployment health checked (will be scaled to 0 in step 1; needs to be healthy first so we can observe the scale-down).

Order of operations

The cutover happens in two distinct phases. Calico is decommissioned first (so the operator stops reconciling), then Cilium’s migration-mode values are flipped to production values. The critical insight: cni.exclusive does not function unless cni.customConf is also flipped to false — the cutover-toggle is customConf, not exclusive.

Phase A — Stop and remove Calico

Scale tigera-operator Deployment to 0 so it stops reconciling Calico resources during deletion: kubectl scale -n tigera-operator deploy/tigera-operator --replicas=0. Wait for the operator pod to terminate (kubectl wait --for=delete pod -l name=tigera-operator -n tigera-operator --timeout=2m).
Delete Tigera operator-managed operator.tigera.io resources in order (Whisker depends on Goldmane depends on APIServer depends on Installation): kubectl delete whisker default; kubectl delete goldmane default; kubectl delete apiserver default; kubectl delete installation default. Each can be patched to remove finalizers if stuck (kubectl patch <kind> default --type=merge -p '{"metadata":{"finalizers":[]}}').
Delete the three Calico namespaces: calico-system, calico-apiserver, tigera-operator. Cascade removes their workloads.
Delete cluster-scoped Calico/Tigera CRDs explicitly (namespace deletion does not cascade to CRDs). Enumerated below from the live cluster as of 2026-05-11. At PR-write time, re-run kubectl get crd -o name | rg 'projectcalico\.org|operator\.tigera\.io' and diff against this list — apply any new CRDs that have appeared:
- crd.projectcalico.org (22 CRDs): bgpconfigurations, bgpfilters, bgppeers, blockaffinities, caliconodestatuses, clusterinformations, felixconfigurations, globalnetworkpolicies, globalnetworksets, hostendpoints, ipamblocks, ipamconfigs, ipamhandles, ippools, ipreservations, kubecontrollersconfigurations, networkpolicies, networksets, stagedglobalnetworkpolicies, stagedkubernetesnetworkpolicies, stagednetworkpolicies, tiers.
- operator.tigera.io (9 CRDs): apiservers, gatewayapis, goldmanes, imagesets, installations, istios, managementclusterconnections, tigerastatuses, whiskers.
Delete cluster-scoped Calico/Tigera ClusterRoles + ClusterRoleBindings. Enumerated below from the live cluster as of 2026-05-11. At PR-write time, diff against kubectl get clusterrole,clusterrolebinding -o name | rg -i 'calico|tigera':
- ClusterRoles (11): calico-cni-plugin, calico-crds, calico-extension-apiserver-auth-access, calico-kube-controllers, calico-node, calico-tier-getter, calico-tiered-policy-passthrough, calico-typha, calico-webhook-reader, tigera-operator, tigera-operator-secrets.
- ClusterRoleBindings: matched names (same prefixes).
- Admission webhooks: none installed (kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration -o name | rg -i 'calico|tigera' returns empty as of 2026-05-11). No action required.
Verify Calico cleanup:
- kubectl get crd -o name | rg 'projectcalico\.org|operator\.tigera\.io' returns empty.
- kubectl get clusterrole,clusterrolebinding -o name | rg -i 'calico|tigera' returns empty.
- All three Calico namespaces are gone (each check returns non-zero): ! kubectl get ns calico-system 2>/dev/null && ! kubectl get ns calico-apiserver 2>/dev/null && ! kubectl get ns tigera-operator 2>/dev/null && echo "all gone" prints all gone.

Phase B — Cilium production cutover

Flip Cilium migration-mode values to production values in roles/cilium/defaults/main.yml:
- cilium_cni_custom_conf: false (was true)
- cilium_cni_uninstall: true (was false — restore default)
- cilium_policy_enforcement_mode: default (was "never")
- cilium_operator_unmanaged_pod_watcher_restart: true (was false)
- No change to cilium_bpf_host_legacy_routing — stays true. PR4 does NOT flip it. Rationale: upstream warns that flipping true → false “may cause a brief interruption.” Without a defined maintenance window and a planned packet-loss budget, flipping it in PR4 is a free hazard. File a follow-up project to schedule the flip during a low-traffic window with explicit go/no-go criteria. Leaving it at true post- migration has no functional downside (it’s a fallback path that simply isn’t exercised once everything is on Cilium-native routing).
- Note: cni.exclusive was true from PR2 onwards inside the CiliumNodeConfig per-node defaults; the GLOBAL Helm value for cni.exclusive is not what gates the cutover. The cutover is customConf: false. Once customConf is false, each agent stops respecting per-node overrides and reads global config. This is correct because by PR4 every node is already labeled.
Re-run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium-install. The role re-renders values.yaml.j2 and runs helm upgrade.
Restart the Cilium DaemonSet so all agents read the new config: kubectl -n kube-system rollout restart ds/cilium. Wait for rollout: kubectl -n kube-system rollout status ds/cilium --timeout=10m.
Delete the CiliumNodeConfig — it has no purpose after the customConf flip: kubectl delete -n kube-system ciliumnodeconfig cilium-default.
Remove the migration label from all nodes (cosmetic, no functional impact): kubectl label nodes -l io.cilium.migration/cilium-default=true io.cilium.migration/cilium-default-.
Verify cutover:
- cilium status --wait returns OK.
- kubectl exec test of an existing CiliumNetworkPolicy (the bastion’s) now enforces — verify by attempting a flow the policy should drop and confirming it is dropped.
- kubectl get networkpolicies.crd.projectcalico.org -A 2>&1 | rg -c 'not found' confirms the legacy Calico-policy file in cf-ssh-bastion is now an ArgoCD-driven no-op (CRD gone; the YAML file is cleaned up in PR5).
Restore system-upgrade-controller: kubectl scale -n system-upgrade deploy/system-upgrade-controller --replicas=1.
k3s --cluster-cidr config note: k3s’s bootstrap config (in ansible/roles/k3s-common/defaults/main.yml) still names 10.42.0.0/16 as the cluster CIDR. This is intentionally left as-is: Cilium’s ipam.mode: cluster-pool makes Cilium authoritative for pod IP assignment via the CiliumNode.spec.ipam.podCIDRs field. The k3s value feeds Node.spec.podCIDR, which Cilium-cluster-pool IPAM ignores. The two coexist harmlessly. Changing the k3s value would require a server restart on every control-plane node, which is out of scope. Document the rationale in docs/operations/cilium.md.

Repository cleanup

Remove ansible/roles/calico/ directory entirely.
Remove the Calico phase from ansible/k3s-playbook.yml.
Update ansible/CLAUDE.md Roles Reference and Phases tables (remove calico).
Update argocd/CLAUDE.md Velero categories: remove calico-system/calico-apiserver/tigera-operator from networking exclusions.
Update argocd/app-configs/velero/backup-schedule.yaml exclude list accordingly.

Documentation updates

Update CNI mention in: README.md, docs/index.md, docs/architecture/overview.md, docs/architecture/index.md, docs/reference/network.md, docs/reference/technologies.md, docs/reference/services.md.
Delete docs/operations/merlin.md (merlin was removed in PR #908; the doc is stale).
Remove the merlin row from the docs/reference/network.md “Per-namespace policy summary” table and any other stale merlin mentions.
New file: docs/operations/cilium.md (evergreen ops runbook — includes a note on the retained k3s --cluster-cidr=10.42.0.0/16 value per step 8 above).
New file: docs/operations/cilium-migration.md (migration log, written iteratively during PR3, finalized here).
Update mkdocs.yml nav: — add the two new docs under Operations, remove the merlin entry.

Cluster impact: Calico CRDs and cluster-scoped resources gone; Cilium authoritative on every node.

Rollback: Has two layers, both partial:

Velero restore from pre-calico-removal-* backup. Restores application namespace state (anything not in the Velero exclude list). Does NOT restore: Calico CRDs/operator/cluster-scoped RBAC (gone from the cluster and not in any backup), Cilium state in kube-system (Velero-excluded — argocd/app-configs/velero/backup-schedule.yaml), the new bastion CiliumNetworkPolicy (the cf-ssh-bastion namespace is also Velero-excluded).
Re-bootstrap Calico via Ansible. Re-run ansible-playbook --tags k3s-calico from a git revert of this PR’s role+playbook changes. This recreates Tigera operator + Installation, but every node would need to be drained/relabeled to flip back to Calico (Cilium owns the CNI socket and won’t release it just because Calico’s resources reappear).

In practice: this PR is the project’s point of no return. After merge, the realistic recovery path is “fix forward in Cilium,” not “roll back to Calico.” Plan accordingly.

PR5 — Remove obsolete Calico policy file

Delete argocd/app-configs/cf-ssh-bastion/calico-network-policy.yaml (already a silent no-op since PR4 removed the CRD).
Update argocd/app-configs/cf-ssh-bastion/kustomization.yaml to drop the reference.
Verify ArgoCD shows the app Synced with only the Cilium policy.

Cluster impact: None. Rollback: git revert.

Bastion policy strategy

The cf-ssh-bastion is the only namespace in the repo with NetworkPolicy resources. Its protection model has five phases:

Phase	Calico policy	CiliumNetworkPolicy	Effective enforcer
Pre-PR2 (today)	enforces	does not exist	Calico
PR2 — PR3 first half (bastion on Calico node)	enforces	exists, `policyEnforcementMode: "never"`	Calico
PR3 second half (bastion’s node migrated to Cilium)	exists but no enforcer	exists, `policyEnforcementMode: "never"`	NONE — bastion is unprotected from migration of its host node until PR4
PR4 cutover (`policyEnforcementMode: default`)	exists but no enforcer (Calico evicted)	enforces	Cilium
PR5 (Calico policy YAML deleted)	does not exist	enforces	Cilium

The unprotected window in PR3 is the cost of using Cilium’s upstream live-migration procedure (which mandates policyEnforcementMode: "never"). Mitigations:

The bastion host node migrates last among workers (per PR3 order), so the window is minimized to a single node-migration cycle plus the 24h-all-workers stability gate before control-plane migrations begin.
The bastion’s external attack surface during the window is its cloudflared-tunnel entry point only (port 2222 from cloudflared ns) — the Calico policy currently allows this exact path, so a Cilium-side policy gap is largely covered by upstream network controls (Cloudflare tunnel auth, then SSH key auth on the bastion).
We do NOT delete the Calico policy file pre-PR4. Even though it has no enforcer once Calico’s node is gone, leaving it in tree makes the PR5 diff trivially auditable.

Alternative considered + rejected: delete all NetworkPolicies before PR2 (upstream’s other suggested mitigation). Rejected because it removes protection on every namespace from PR2 onwards, not just the bastion’s host-node migration window.

Ansible role structure

ansible/roles/cilium/
├── defaults/main.yml          # cilium_version, cilium_helm_repo, CIDR, feature flags, migration toggles
├── handlers/main.yml          # restart-cilium-ds (Helm handles rollouts; rarely fires)
├── meta/main.yml              # galaxy metadata (mirror calico/meta/main.yml)
├── tasks/main.yml             # helm_repository add → helm install → wait for DS Ready → wait for Hubble Relay Ready
├── templates/
│   ├── values.yaml.j2         # Cilium Helm values, templated from defaults
│   └── cilium-node-config.yaml.j2  # CiliumNodeConfig CR for live migration
└── files/gateway-api-crds/
    ├── gatewayclasses.yaml         # vendored from kubernetes-sigs/gateway-api v1.4.1 Standard
    ├── gateways.yaml
    ├── httproutes.yaml
    ├── grpcroutes.yaml
    ├── referencegrants.yaml
    └── safe-upgrades-policy.yaml   # ValidatingAdmissionPolicy guarding against CRD downgrades

`defaults/main.yml` shape

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
cilium_version: "1.19.3"  # latest stable as of 2026-05-11
cilium_kubeconfig: "{{ lookup('env', 'HOME') }}/.kube/configs/{{ k8s_context }}-admin.yml"
cilium_namespace: kube-system
cilium_helm_repo_url: "https://helm.cilium.io/"
cilium_helm_repo_name: cilium

# IPAM
cilium_cluster_pool_ipv4_cidr: "10.245.0.0/16"
cilium_cluster_pool_ipv4_mask_size: 24
cilium_cluster_pool_ipv6_cidr: "fd00:10:245::/48"
cilium_cluster_pool_ipv6_mask_size: 64

# Routing
cilium_routing_mode: tunnel
cilium_tunnel_protocol: vxlan
cilium_tunnel_port: 8473                       # MUST differ from Calico's 4789 during dual-CNI window
cilium_auto_direct_node_routes: false

# Migration toggles — flipped at cutover (see migration-mode → production-mode below)
cilium_cni_custom_conf: true                    # migration mode: true; production mode: false (the cutover toggle)
cilium_cni_uninstall: false                     # migration mode: false; production mode: true (default)
cilium_policy_enforcement_mode: "never"         # migration mode: "never"; production mode: "default"
cilium_operator_unmanaged_pod_watcher_restart: false  # migration mode: false; production mode: true
cilium_bpf_host_legacy_routing: true            # migration mode: true; production mode: STAYS true (flip to false is a deferred follow-up)

# Feature scope (constant across migration and production modes)
# NOTE: Cilium 1.19's Helm schema accepts kubeProxyReplacement as a string
# ("true"/"false"), not a boolean. Render with quotes in values.yaml.j2.
cilium_kube_proxy_replacement: "false"
cilium_hubble_enabled: true
cilium_hubble_relay_enabled: true
cilium_hubble_ui_enabled: true
cilium_gateway_api_enabled: true
# Gateway API CRDs are pre-installed by a separate Ansible task in this role
# (chart does NOT auto-install them). Pinned version:
cilium_gateway_api_crd_version: "v1.4.1"

# Retry (mirror calico_*_retry_count pattern)
cilium_retry_count: 30
cilium_retry_delay: 10
cilium_install_retry_count: 60
cilium_install_retry_delay: 10

`tasks/main.yml` shape (sketch)

Tasks use kubernetes.core.helm_repository, kubernetes.core.helm, kubernetes.core.k8s (for CRDs and the CiliumNodeConfig apply), and kubernetes.core.k8s_info (for readiness waits) — all FQCN per ansible/CLAUDE.md. The role is delegated to localhost and runs once, matching the Calico role’s pattern.

Task order:

Pre-install: apply Gateway API v1.4.1 CRDs from roles/cilium/files/gateway-api-crds/*.yaml via kubernetes.core.k8s with state: present and server_side_apply. Tagged cilium-install and cilium-gateway-api-crds for selective re-runs.
Add the Cilium Helm repository.
Helm install/upgrade cilium/cilium chart with templated values (wait: true, wait_timeout: 15m). Helm install task uses wait: true with wait_timeout: 15m (15 min covers worst-case DaemonSet rollout on TPI compute modules; 10 min was deemed insufficient for slow eMMC boot + image pull races). operator.replicas: 2 is set explicitly (matches Cilium chart default; pinned for visibility so the “don’t migrate two operator-hosting nodes in parallel” constraint is self-documenting in the values).
Apply the CiliumNodeConfig CR from templates/cilium-node-config.yaml.j2. Selector matches io.cilium.migration/cilium-default=true. Initially applies to no nodes; PR3 adds the label.
Wait for Cilium DaemonSet Ready on all desired nodes.
Wait for Hubble Relay (Deployment readyReplicas > 0) — when cilium_hubble_relay_enabled.
Wait for cilium-operator Ready.

Playbook integration

- name: Install Cilium CNI
  hosts: tp_cluster_controlplane[0]
  gather_facts: false
  roles:
    - role: cilium
      tags:
        - k3s-cilium
        - cilium
        - cilium-install

After PR4 lands, the Phase 5 (Calico) block is deleted; the Phase 5b comment is trimmed.

Risk matrix

Phase	Worst case	Detection	Rollback
PR1	n/a (code only)	n/a	`git revert`
PR2	Cilium DaemonSet crashloops on a node	`kubectl get ds -n kube-system cilium` shows `numberReady < desired`; Hubble Relay never Ready	`helm uninstall cilium -n kube-system`. Zero workload impact.
PR2	New `CiliumNetworkPolicy` is wrong, blocks bastion traffic	Bastion’s SSH proxy starts failing	Delete the new `cilium-network-policy.yaml` from kustomization; Calico policy continues enforcing.
PR3	Node migration leaves pod stuck in `ContainerCreating` (CNI socket race)	Pod stuck > 5min after uncordon	Re-cordon, remove migration label, drain, uncordon — pod reschedules on a Calico-managed node.
PR3	Cross-CNI pod-to-pod connectivity fails	Smoke test (curl Cilium-IP pod → Calico-IP pod) fails	Pause migration; roll the one bad node back or diagnose before advancing.
PR3	Hubble or cilium-operator OOMKilled	`kubectl top pods -n kube-system`; events show OOMKill	Bump resource requests in `values.yaml.j2`; re-run `--tags cilium-install`.
PR3	IPv6 dual-stack fails on first migrated node (upstream untested)	First-node IPv6 validation step fails	”IPv6 fallback” runbook in PR3 section: disable v6 in values, accept temp v6 outage, file follow-up.
PR3	system-upgrade-controller upgrades k3s mid-migration	New k3s server pod schedules during migration	Pre-flight scales the controller to 0; PR4 restores.
PR3	CiliumNodeConfig label propagation race — agent restarts before reading new config	New agent pod starts without writing `05-cilium.conflist`; pods stay on Calico	PR3 step 4 includes a 10s sleep after labeling and `kubectl wait --for=condition=Ready node`. If migration appears to no-op (no Cilium IP on rescheduled pods), repeat steps 5–6.
PR3 (final node)	No Calico-managed peer left for fallback	Migration fails or partially completes	One-way step: only proceed when all earlier 7 nodes have passed both stability gates. Rollback = `helm uninstall cilium` (unrealistic — would orphan 7 migrated nodes).
PR4	tigera-operator fights deletion (recreates `Installation` after delete)	After `kubectl delete installation default`, calico-system pods recreated within ~30s	Phase A step 1 prevents this — scale operator to 0 BEFORE deleting Installation.
PR4	`policyEnforcementMode: default` flip drops legitimate traffic for namespaces with no CiliumNetworkPolicy	Apps in those namespaces lose traffic immediately post-flip	Pre-PR4 audit: enumerate namespaces that had Calico NetworkPolicies. Only cf-ssh-bastion has them (verified via earlier reviewer cross-check). All other namespaces had no policies so `default` mode allows-all by default. Confirm by running `kubectl get networkpolicy,networkpolicies.crd.projectcalico.org,globalnetworkpolicies.crd.projectcalico.org -A` immediately before PR4 — only cf-ssh-bastion entries should appear.
PR4	Orphan CRDs or ClusterRoles after namespace delete	`kubectl get crd \| rg 'projectcalico\\.org\|operator\\.tigera\\.io'` returns rows; same for `clusterrole \| rg -i 'calico\|tigera'`	Delete orphans by name (enumerated lists in PR4 Phase A steps 4–5).
PR4	Step order violated (Phase B run before Phase A complete)	Mid-flight chaos: Cilium tries to take over while Calico still reconciling	Reviewer-enforced via PR description checklist (no automated gate is realistic — Ansible task ordering ensures it within a single run, but cross-PR ordering relies on review attention). PR4 description MUST cite the “Order of operations” subsection and confirm each Phase A checkmark before merging the Phase B value flip.
PR4	Velero exclude misconfig backs up Cilium namespace unnecessarily	Backup size spikes	Edit `backup-schedule.yaml` excludes; non-urgent.
PR5	n/a (deleting a no-op file)	n/a	`git revert`

Mandatory pre-flight backups

Before	Command
PR2	`velero backup create pre-cilium-install-$(date +%Y%m%d-%H%M%S) --wait`
PR3 (first node)	Fresh backup if >24h since PR2 backup: `velero backup create pre-pr3-migration-$(date +%Y%m%d-%H%M%S) --wait`. Also `kubectl cluster-info dump > /tmp/pre-pr3-cluster-state.yaml` for post-migration audit.
PR4	`velero backup create pre-calico-removal-$(date +%Y%m%d-%H%M%S) --wait`

Documentation deliverables

File	Change	PR
`README.md`	Update CNI mention	PR4
`docs/index.md`	Update CNI mention	PR4
`docs/architecture/overview.md`	Update CNI in stack description	PR4
`docs/architecture/index.md`	Update CNI in stack description	PR4
`docs/reference/network.md`	Calico → Cilium; VXLAN tunnel; `10.245.0.0/16` pod CIDR	PR4
`docs/reference/technologies.md`	Replace Calico entry with Cilium	PR4
`docs/reference/services.md`	Remove Whisker/Goldmane; add Hubble	PR4
`docs/operations/merlin.md`	DELETE — stale (merlin removed in PR #908)	PR4
`docs/operations/cilium.md`	NEW evergreen Cilium ops runbook (incl. note on retained k3s `--cluster-cidr=10.42.0.0/16`)	PR4
`docs/operations/cilium-migration.md`	NEW per-node migration log (written during PR3, finalized in PR4)	PR3 + PR4
`mkdocs.yml`	Add nav entries for the two new ops docs; remove the `merlin` entry	PR4
`ansible/CLAUDE.md`	Update Roles Reference + Phases tables	PR1 (add cilium row) + PR4 (remove calico row, trim Phase 5)
`argocd/CLAUDE.md`	Update Velero categories	PR4
`argocd/app-configs/velero/backup-schedule.yaml`	Update exclude list	PR4

Open questions / deferred decisions

None. All technical decisions are resolved against the actual Cilium 1.19.3 chart, Cilium’s official k8s-install-migration docs, and the live cluster state (verified 2026-05-11):

Cilium version: 1.19.3 (helm search repo cilium/cilium --versions).
Migration-mode Helm values: locked per upstream — cni.customConf: true, cni.uninstall: false, routingMode: tunnel, tunnelProtocol: vxlan, tunnelPort: 8473, policyEnforcementMode: "never", bpf.hostLegacyRouting: true, operator.unmanagedPodWatcher.restart: false.
Production-mode Helm values (PR4 cutover): cni.customConf: false, cni.uninstall: true, policyEnforcementMode: default, operator.unmanagedPodWatcher.restart: true. bpf.hostLegacyRouting intentionally kept at true in PR4 (flip to false is a follow-up).
CiliumNodeConfig: applied in PR2, deleted in PR4 step 10. Selector + defaults specified in PR2’s “CiliumNodeConfig” subsection.
Agent pod label: k8s-app=cilium (verified by rendering 1.19.3 chart).
kubeProxyReplacement value type: string "false".
Gateway API CRDs: kubernetes-sigs/gateway-api v1.4.1 Standard channel (GatewayClass, Gateway, HTTPRoute, GRPCRoute, ReferenceGrant). Pinned by version in defaults/main.yml and vendored under roles/cilium/files/gateway-api-crds/. Cilium chart does NOT auto-install.
CRD enumeration: 22 crd.projectcalico.org + 9 operator.tigera.io (full lists in PR4 Phase A step 4).
ClusterRole enumeration: 11 named ClusterRoles (full list in PR4 Phase A step 5).
Admission webhooks: none exist for Calico/Tigera in this cluster.
Bastion policy strategy: documented in dedicated section above. Unprotected window during PR3 is accepted with mitigations.
IPv6 dual-stack: enabled in PR2 with explicit first-node validation gate and fallback runbook in PR3 (upstream marks IP-family changes as untested but family-preservation is unspecified).

PR-write time still requires a quick diff of the live cluster CRDs/ClusterRoles against these lists (PRs may be days/weeks after design approval; new resources could appear via Tigera operator reconciliation while Calico is still installed). Treat the enumerations as the authoritative starting point, not as exhaustive forever.

References

Cilium 1.19 k8s install migration — the authoritative procedure; PR2/PR3/PR4 mirror its sequence
Cilium 1.19 Gateway API — Gateway API CRD prerequisites (v1.4.1 Standard channel)
Cilium 1.19 Helm reference — values schema for the Helm chart
Gateway API v1.4.1 release — CRD manifest source
Repo memory: mkdocs-yml-is-hand-curated (relevant for PR4 nav updates). The Tigera-operator-install’s CRDs/ClusterRoles also don’t cascade from namespace deletion, which is the same anti-cascade hazard documented in the ArgoCD-Application-deletion memory entry — different tooling, same mental model.
Prior migration spec for reference: docs/engineering/specs/2026-05-10-pushover-alertmanager-migration-design.md (PR-series structure precedent)