Cilium Production-Mode Hardening — Design

Date: 2026-05-13 Author: Sean Brandt (with Claude Opus 4.7) Status: Approved — ready for implementation plan Parent epic: hl-3ed Children: hl-7uj (KPR), hl-fp8 (bpf_host_legacy_routing)

Context

The Calico → Cilium migration completed 2026-05-12 (PR #1018). At cutover, two Cilium production-mode values were intentionally deferred to keep the migration blast radius minimal:

• cilium_kube_proxy_replacement: "false" — kube-proxy stayed running alongside Cilium
• cilium_bpf_host_legacy_routing: true — host-namespace traffic kept using iptables

Both targets are flipped in this initiative. The cluster moves to Cilium upstream’s recommended production posture for 1.19.x: Cilium kube-proxy replacement (KPR) handling all Service routing via eBPF, and Cilium’s native eBPF datapath handling host-namespace traffic (no iptables fallback for known destinations).

Goals

• Migrate the 8-node k3s cluster from kube-proxy + Cilium-with-legacy-host-routing (current) to Cilium-KPR + Cilium-native-host-routing (target) in a single maintenance window with staged checkpoints.
• Single ansible PR with staged playbook runs: Stage 1 helm-upgrades Cilium with KPR enabled (kube-proxy still running as fallback), Stage 2 restarts k3s servers with --disable=kube-proxy (kube-proxy DaemonSet removed by k3s).
• Preserve the ability to roll back at each stage without code revert.

Non-Goals

• Cilium 1.20.x upgrade. Tracked separately as hl-r39, gated on 1.20 GA + Standalone DNS Proxy graduating from alpha.
• L7 DNS proxy work. Cluster has a deeper L7 DNS bug (hl-bdr) that the upstream-recommended workaround (PR #1071) did not fix; the redesign already commits to L4-only CNPs.
• Crown-jewel CNP redesign (hl-7q1). Independent of KPR.
• kube-vip replacement or modification. The VIP address 192.168.20.140 becomes Cilium’s k8sServiceHost in the new config.

Architecture

Components

Repo files changed:

File	Change
ansible/inventory/group_vars/all.yml	Add kube_vip_address: "192.168.20.140" (promote from ansible/roles/kube-vip/defaults/main.yml to inventory-wide scope so the cilium role can reference it from Phase 5 where kube-vip role is not loaded). Ansible var precedence makes this override the role default; no kube-vip behavior change.
ansible/roles/cilium/defaults/main.yml	cilium_kube_proxy_replacement: "false" → "true"; cilium_bpf_host_legacy_routing: true → false; add cilium_k8s_service_host: "{{ kube_vip_address }}"; add cilium_k8s_service_port: 6443; add cilium_bpf_masquerade: true (required when hostLegacyRouting: false — chart default for bpf.masquerade in 1.19.3 is false, verified via helm show values cilium/cilium --version 1.19.3)
ansible/roles/cilium/templates/values.yaml.j2	Add k8sServiceHost: {{ cilium_k8s_service_host }} and k8sServicePort: {{ cilium_k8s_service_port }} as top-level Helm values, near the existing kubeProxyReplacement line. Also add bpf.masquerade: {{ cilium_bpf_masquerade | lower }} to the bpf: block where hostLegacyRouting lives.
ansible/roles/cilium/tasks/main.yml	(1) On the existing helm-upgrade task: change wait: true to wait: false (or remove the param) — the rollout-restart immediately following makes the helm wait window redundant and risks a 15min wasted timeout. (2) Append a new task kubectl rollout restart daemonset/cilium -n kube-system. (3) Append a new task kubectl rollout status daemonset/cilium -n kube-system --timeout=10m that tracks observedGeneration correctly (the existing role’s k8s_info-based “Wait for Cilium DaemonSet to be Ready” check uses numberReady == desiredNumberScheduled, which is transiently true at the instant rollout-restart is issued — race condition). The new kubectl rollout status task must come BEFORE the existing readiness wait, OR the existing wait must be deleted in favor of kubectl rollout status.
ansible/roles/k3s-common/defaults/main.yml	k3s_disable: [traefik, servicelb] → [traefik, servicelb, kube-proxy]
ansible/k3s-playbook.yml (Phase 4: Worker nodes)	Add serial: 1 to limit worker k3s-agent restart concurrency. Defensive change — not required for this cutover (workers’ config.yaml is unchanged because the k3s-common template’s agent path doesn’t include k3s_disable), but cheap protection against future changes that DO touch workers.

Cluster state changes after rollout:

• cilium-config ConfigMap gains kube-proxy-replacement: true, k8s-service-host: 192.168.20.140, k8s-service-port: "6443", enable-bpf-masquerade: "true", and loses enable-host-legacy-routing: true.
• kube-system/kube-proxy DaemonSet stops (k3s removes it on next restart with --disable=kube-proxy).
• iptables KUBE-SVC-* / KUBE-SEP-* chains cleared on every node.
• Cilium agents re-program eBPF maps with Service entries that were previously handled by kube-proxy’s iptables rules.

Data Flow / Cutover Sequence

The cutover is a single ansible PR but executed in two stages with a checkpoint between them. Each stage is invoked via ansible --tags.

Stage 1: helm-upgrade Cilium
  Dry-run first (recommended; cheap defense against typos and var-scope issues):
    ansible-playbook -i ansible/inventory/hosts.yml \
      ansible/k3s-playbook.yml --tags cilium-install --check --diff
    Confirm in the diff that values.yaml.j2 renders with:
      - kubeProxyReplacement: "true"
      - k8sServiceHost: 192.168.20.140  (NOT empty)
      - k8sServicePort: 6443
      - bpf.hostLegacyRouting: false
      - bpf.masquerade: true
  Real run:
    ansible-playbook -i ansible/inventory/hosts.yml \
      ansible/k3s-playbook.yml --tags cilium-install
  Effect:
    - templates values.yaml.j2 (KPR=true, hostLegacyRouting=false, masquerade=true,
      k8sServiceHost/Port set)
    - helm upgrade cilium (wait=false, agent rollout handled by next task) → updates ConfigMap
    - kubectl rollout restart daemonset/cilium → agents reload with new config
    - kubectl rollout status daemonset/cilium --timeout=10m → wait for ALL 8 agents
      on the new ReplicaSet generation (uses observedGeneration, not transient
      numberReady=desiredNumberScheduled)
  State after Stage 1:
    - Cilium agents running KPR + native routing + bpf.masquerade
    - kube-proxy DaemonSet STILL running (acts as redundant Service routing path)
    - iptables KUBE-* chains still present (kube-proxy programs them)

CHECKPOINT 1 (manual, ~5 min):
  - kubectl get cm -n kube-system cilium-config -o jsonpath='{.data.kube-proxy-replacement}' → true
  - kubectl get cm -n kube-system cilium-config -o jsonpath='{.data.enable-host-legacy-routing}' → false
  - kubectl get pods -n kube-system -l k8s-app=cilium → all 8 fresh (Age < 5min), READY
  - kubectl get ds -n kube-system kube-proxy → still present (Ready desired=current)
  - Smoke test: curl from a pod to a Service IP (e.g., kubernetes.default.svc); verify response
  - Host-network workload health: kube-vip VIP 192.168.20.140 reachable, alloy shipping logs
  - Baseline: kubectl get externalsecret -A | grep -c True → 44

Stage 2: k3s --disable=kube-proxy (rolling, CP nodes ONLY)
  Dry-run first:
    ansible-playbook -i ansible/inventory/hosts.yml \
      ansible/k3s-playbook.yml --tags k3s-install \
      --limit tp_cluster_controlplane --check --diff
    Confirm only the k3s config.yaml on CP nodes changes (kube-proxy added to disable list).
  Real run:
    ansible-playbook -i ansible/inventory/hosts.yml \
      ansible/k3s-playbook.yml --tags k3s-install \
      --limit tp_cluster_controlplane
  CRITICAL: the --limit flag is required. Without it, Phase 4 (worker) plays also
  execute under --tags k3s-install (verified via --list-tasks). Workers' config.yaml
  content is unchanged (the k3s-common template's agent branch doesn't include
  k3s_disable), so the Restart k3s handler does not fire, BUT the playbook still
  re-runs many no-op tasks on workers — wasted time and unnecessary surface area
  during an active maintenance window.
  Effect (per CP node, rolling per Phase 2's serial:1):
    - template /etc/rancher/k3s/config.yaml (k3s_disable now includes kube-proxy)
    - file content changes → handler fires → systemctl restart k3s
    - k3s server restarts with --disable=kube-proxy → kube-proxy DS removed by k3s
  Rolling order: alpha-1 (Phase 1, single host) → alpha-2 (Phase 2 serial:1) → alpha-3
  Wall clock: ~30-45s per CP node × 3 CP nodes ≈ 2-3 minutes total
  Worker k3s-agent: NOT restarted in this Stage (per --limit)
  Worker pods: brief disruption during the kube-proxy DS Pod teardown on each worker
               (k3s removes the kube-proxy Pods cluster-wide once the server-side
               managed-resource is gone)

CHECKPOINT 2 (manual, ~5 min):
  - kubectl get ds -n kube-system kube-proxy → NotFound (k3s deleted it)
  - kubectl exec -n kube-system <any-cilium-agent> -- iptables -t nat -S | grep -c '^-N KUBE-SVC' → 0
    (use -S not -L; -L on systems with many rules can hang)
  - Service traffic via Cilium-only path: smoke test in-cluster Service curl
  - ArgoCD reachability: kubectl get application -n argocd | head still returns OK
    (ArgoCD's argocd-server ClusterIP Service now goes through Cilium KPR exclusively)
  - 44/44 ExternalSecrets Ready=True (DNS+Service health proxy)
  - cilium-dbg service list on one agent → shows expected Services from cluster
  - cilium-dbg status --verbose | grep -A 5 "KubeProxyReplacement" → reports
    "Strict" mode, datapath "LB-bpf" (known-good fingerprint)

Stage 3: documentation + bead closure (no playbook run)
  - New runbook: docs/operations/cilium-kpr.md (debugging Service issues without
    kube-proxy; cilium-dbg service / cilium-dbg lb / cilium monitor commands)
  - Update docs/reference/network.md: KPR active, kube-proxy absent, host-namespace
    traffic uses eBPF native routing
  - Update mkdocs.yml nav: add the new runbook (memory: mkdocs-nav-curated)
  - bd close hl-7uj, hl-fp8, hl-3ed

Error Handling and Rollback

Stage 1 failure (Cilium KPR doesn’t program, Services break):

helm rollback cilium -n kube-system
kubectl rollout restart daemonset/cilium -n kube-system
kubectl rollout status daemonset/cilium -n kube-system

Cluster returns to kube-proxy-only Service routing. The git revert PR can be opened later; immediate priority is restoring traffic.

Stage 2 failure (kube-proxy removed but Service traffic broken):

The canonical rollback is “revert the git change to k3s-common/defaults/main.yml and re-run ansible-playbook --tags k3s-install --limit tp_cluster_controlplane”, which keeps the template authoritative and the next normal run consistent.

If immediate restoration is needed BEFORE the revert PR is prepared (template will get re-written next playbook run, so this is emergency-only):

# Override k3s_disable via extra-vars to keep the template authoritative for THIS run:
ansible-playbook -i ansible/inventory/hosts.yml \
  ansible/k3s-playbook.yml --tags k3s-install \
  --limit tp_cluster_controlplane \
  -e "k3s_disable=['traefik','servicelb']"

This re-templates config.yaml without kube-proxy in disable, triggers the Restart k3s handler per CP node (rolling per Phase 2 serial:1), and k3s re-installs the kube-proxy DaemonSet automatically because --disable=kube-proxy is no longer in effect. DO NOT edit /etc/rancher/k3s/config.yaml directly with lineinfile — it’s a template-managed file and will be reverted by the next normal playbook run.

After service is restored, prepare and merge a git revert PR. Then a normal ansible-playbook run produces a no-op diff.

Failure modes to watch:

• Cilium agent doesn’t pick up new ConfigMap on helm-upgrade. Confirmed during PR #1071 — helm upgrade updates the ConfigMap but does not restart the DaemonSet. Mitigation: ansible task explicitly issues kubectl rollout restart daemonset/cilium after helm upgrade.
• kube-vip VIP availability briefly degraded during cutover. kube-vip runs as a static pod on every control-plane node with host network and announces the VIP via ARP/BGP. During k3s server restart on a CP node, the kube-vip static pod is terminated (kubelet kills it; no graceful pre-stop hook path for a static pod). Lease-based leader election (kube_vip_leader_election: true) means another CP node takes over leadership once the lease expires or is detected stale. Failover time is bounded by the lease parameters in kube-vip.yaml.j2; not measured precisely here. Mitigation: Phase 2 already has serial: 1, so only one CP node is in restart at a time; at least 2 of 3 kube-vip instances remain active and one of them re-announces the VIP.
• NodePort Service unreachable mid-cutover. Stage 1 leaves kube-proxy and KPR both active (KPR shadowing kube-proxy’s iptables rules). Stage 2 removes kube-proxy. Window of unique-path-via-Cilium begins at Stage 2; if Cilium isn’t programming a Service correctly, Checkpoint 2 catches it.
• DNS during k3s server restart. coredns is a Deployment, not affected by k3s server restart directly. Resolvers in pods use the cluster DNS Service IP; KPR programs that Service. Brief blip possible during the agent’s eBPF reload window (sub-second per node).
• ExternalSecret refresh failures during cutover. ESO talks to Vault via Service IP (vault-active.vault:8200). KPR must program this Service correctly. Checkpoint 1 and Checkpoint 2 both verify 44/44 ESes Ready. Failure here triggers Stage 1 or Stage 2 rollback.
• ArgoCD UI unreachable during cutover. argocd-server is a ClusterIP Service reached via kube-proxy iptables today, via Cilium KPR after Stage 1, and via Cilium KPR exclusively after Stage 2. CLI kubectl always works because it goes via the kube-vip VIP (host network, no Service IP). Mitigation: operator should rely on kubectl get application -n argocd during the window, not the UI; restore UI by checking Cilium’s Service map programming for argocd-server after each stage if needed.

Testing Strategy

Pre-flight (before Stage 1):

• Velero backup per ansible/CLAUDE.md pre-deployment checklist:

  velero backup create pre-cilium-kpr-$(date +%Y%m%d-%H%M%S) --wait

• All 44 ExternalSecrets Ready=True

• All ArgoCD apps Synced+Healthy (excluding velero, pre-existing OutOfSync)

• kube-vip VIP reachable externally: curl -k https://192.168.20.140:6443/healthz returns ok

• Per-node host-level VIP connectivity — Cilium agents hit the VIP from the host net ns post-cutover. All 8 nodes must return ok AND a direct route entry; investigate any node with an indirect route via kube-proxy before proceeding:

  ansible -i ansible/inventory/hosts.yml tp_cluster_nodes -m shell \
    -a "curl -k --max-time 5 https://192.168.20.140:6443/livez && ip route get 192.168.20.140"

• cilium-dbg status --brief on each agent → OK

• bpf.masquerade chart-default verification (final defense against Issue #9 from spec review): helm show values cilium/cilium --version 1.19.3 | grep -A 2 '^ masquerade:' — chart default should be ~ (null → effectively false). Spec sets bpf.masquerade: true explicitly to override.

Per-stage: see Cutover Sequence above.

Post-cutover 24h regression watch:

• Grafana dashboards: cluster traffic volume, Service error rates, certificate expiry, ESO sync success — all green
• Hubble drops not elevated (compare against pre-cutover baseline)
• No alerts firing for the watched namespaces (vault, cert-manager, ESO, cnpg-system, authentik)

Quality gates (CI on the PR):

• yamllint ansible/
• ansible-lint ansible/ (if configured)
• ansible-playbook --syntax-check ansible/k3s-playbook.yml

Documentation Pointers

New runbook to add as part of Stage 3:

• docs/operations/cilium-kpr.md — debugging Service issues without kube-proxy. Covers cilium-dbg service list, cilium-dbg lb, cilium monitor, symptom-to-tool mapping (e.g., “Service IP not reachable from a pod” → check Cilium eBPF Service map; “wrong backend selected” → check load-balancer affinity).

Updates to existing docs:

• docs/reference/network.md — note KPR active, kube-proxy absent, host-namespace routing on eBPF native path.
• mkdocs.yml nav: — add the new runbook (memory: mkdocs-nav-curated).

Open Questions for Implementation Time

Both questions from brainstorming resolved or accepted as operator-determined:

1. Maintenance window timing. Deferred to operator discretion. No automated trigger; user picks when to run Stage 1 and Stage 2.
2. k3s server restart concurrency. Investigated and gap found in Phase 4. Spec adds serial: 1 to Phase 4 as a defensive cleanup (cheap protection against future ansible runs that DO touch worker config). For THIS cutover specifically, workers don’t restart at all — the k3s-common config.yaml.j2 template’s agent path doesn’t include k3s_disable, so adding kube-proxy to that list changes only the server-side config. Stage 2 rolling order is alpha-1 → alpha-2 → alpha-3 (CP nodes only, enforced via --limit tp_cluster_controlplane on the Stage 2 invocation).

Bead Plan

• Update epic hl-3ed: scope expanded from “two separate cycles” to “single bundled cycle per this spec”.
• hl-7uj (KPR child): tracked in this spec under Stage 1/Stage 2 changes; no longer needs a separate spec.
• hl-fp8 (bpf_host_legacy_routing child): tracked in this spec under Stage 1; no longer needs a separate spec.
• Dependency hl-7uj blocks hl-fp8: remove. They are now part of the same cutover and land together.
• New beads if discovered during implementation: file individually.

Re-evaluation Triggers (for future hardening cycles)

This spec gets the cluster to upstream-recommended Cilium 1.19.x production posture. Future hardening cycles to consider when triggers fire:

• hl-r39 (Cilium 1.20+ Standalone DNS Proxy adoption) — when 1.20 GA and the standalone proxy graduates beyond alpha.
• hl-bdr (L7 DNS proxy bug) — if a fix lands that works on this cluster, re-evaluate the L4-only constraint and the crown-jewel CNP redesign’s L7 visibility story.