Cilium kube-proxy Replacement (KPR) — Operations Runbook

Cluster state since the hl-3ed cutover (2026-05-13): kube-proxy is no longer running. All Service routing is handled by Cilium’s eBPF datapath. This runbook covers the debugging and verification commands that replace the kube-proxy / iptables-based workflow.

TL;DR — what changed

Capability	Before (k3s + Cilium with KPR=false)	After (Cilium KPR=Strict)
Service routing	k3s embedded kube-proxy programming iptables `KUBE-SVC-*` chains	Cilium socketLB intercepts at socket layer; eBPF Service map
Source IP for pod-to-external	iptables MASQUERADE	Cilium `bpf.masquerade: true`
Host-namespace traffic routing	Linux FIB lookup + iptables	Cilium eBPF native routing (`bpf.hostLegacyRouting: false`)
API server endpoint for Cilium agents	Resolved via Service IP (kube-proxy required)	`KUBERNETES_SERVICE_HOST=192.168.20.140` (kube-vip VIP) in agent pod env
`iptables -t nat -S \| grep KUBE-SVC`	hundreds of chains	zero (only `KUBE-KUBELET-CANARY` from kubelet)

Quick reference — symptom-to-tool mapping

Symptom	First-line command
Service IP not reachable from a pod	`cilium-dbg service list \| grep <service-cluster-ip>`
Wrong backend pod selected	`cilium-dbg lb list`
Real-time packet path debugging	`kubectl exec ... cilium-dbg monitor --related-to <endpoint-id>`
Verify KPR is active on an agent	`cilium-dbg status --verbose \| grep -A 5 KubeProxyReplacement`
Confirm kube-proxy is gone	`iptables -t nat -S \| grep -c '^-N KUBE-SVC'` (expect `0`)
External traffic source IP wrong	check `bpf.masquerade` is `true` in `cilium-config`

Common tasks

Confirm KPR is active and healthy on a node

NODE=tpi-alpha-1
NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \
  | awk -v n="$NODE" '$7==n{print $1}')
kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \
  cilium-dbg status --verbose | grep -A 5 KubeProxyReplacement

Expected output includes Status: True and Socket LB: Enabled.

Confirm a Service is programmed on a node

NODE=tpi-alpha-1
NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \
  | awk -v n="$NODE" '$7==n{print $1}')
SERVICE_IP=$(kubectl get service -n <ns> <name> -o jsonpath='{.spec.clusterIP}')
kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \
  cilium-dbg service list | grep "$SERVICE_IP"

Expected: lines showing frontend (Service ClusterIP:port) and backend (pod IP:port) entries, with (active) markers on the backends.

Confirm an endpoint can reach a Service

NODE=tpi-alpha-1
NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \
  | awk -v n="$NODE" '$7==n{print $1}')
EP_ID=$(kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \
  cilium-dbg endpoint list | grep "<pod-namespace>/<pod-name>" | awk '{print $1}')
kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \
  cilium-dbg monitor --related-to "$EP_ID" --type policy-verdict --last 30

Expected: policy-verdict log lines with action ALLOWED for valid Service egress.

Debug “Service not reachable” without kube-proxy

Confirm the Service exists and has Endpoints:
Terminal window
```
kubectl get service,endpoints -n <ns> <name>
```

Confirm Cilium programmed the Service on the source node:

kubectl exec -n kube-system <agent-pod> -c cilium-agent -- \
  cilium-dbg service list | grep <service-cluster-ip>

Use cilium-dbg monitor from the source-node agent and trigger the request from the source pod. Look for policy-verdict events on the source endpoint.

Verify cilium-config carries the production-mode values

for k in kube-proxy-replacement enable-host-legacy-routing enable-bpf-masquerade; do
  v=$(kubectl get cm -n kube-system cilium-config -o jsonpath="{.data.${k}}")
  echo "$k = $v"
done

Expected:

kube-proxy-replacement = true
enable-host-legacy-routing = false
enable-bpf-masquerade = true

Verify k3s embedded kube-proxy is not running

ansible -i ansible/inventory/hosts.yml tp_cluster_controlplane -b -m shell \
  -a "grep '^disable-kube-proxy:' /etc/rancher/k3s/config.yaml"

Expected: every CP node shows disable-kube-proxy: true.

Check for stale KUBE-* iptables chains

ansible -i ansible/inventory/hosts.yml tp_cluster_controlplane -b -m shell \
  -a "iptables -t nat -S | grep -c '^-N KUBE-'"

Expected: every CP node shows 1 (just KUBE-KUBELET-CANARY, kubelet’s own health-check chain). Anything significantly higher means kube-proxy is somehow still running.

Architecture context

KPR was enabled via cilium_kube_proxy_replacement: "true" in the cilium ansible role (PR #1076).
k8sServiceHost: 192.168.20.140 (kube-vip VIP) is set so Cilium agents can reach the API server without kube-proxy. Stored in agent pod env vars (KUBERNETES_SERVICE_HOST/PORT), not in cilium-config (chart behavior).
bpf.hostLegacyRouting: false — host-namespace traffic uses eBPF native routing instead of iptables FIB lookup.
bpf.masquerade: true — preserves source-NAT for pod-to-external traffic (required when hostLegacyRouting: false).
k3s disable-kube-proxy: true in /etc/rancher/k3s/config.yaml on every control-plane node disables the embedded kube-proxy controller inside the k3s process (PR #1078). Note: the disable: LIST in the same file does NOT accept kube-proxy — must use the separate top-level flag.

Rollback (re-enable kube-proxy)

If KPR proves problematic and you need to restore kube-proxy:

Revert the relevant ansible PRs (or set k3s_disable_kube_proxy: false in ansible/roles/k3s-common/defaults/main.yml).
Run ansible-playbook -i ansible/inventory/hosts.yml ansible/k3s-playbook.yml --tags k3s-install --limit tp_cluster_controlplane.
k3s will restart on each CP, embedded kube-proxy will start again, and iptables KUBE-* chains will repopulate within ~30s.
Optionally also revert Cilium KPR via cilium_kube_proxy_replacement: "false" then ansible-playbook ... --tags cilium-install.

docs/operations/cilium.md — general Cilium operations
docs/reference/network.md — cluster network reference
Spec: docs/engineering/specs/2026-05-13-cilium-production-mode-hardening-design.md
Plan: docs/engineering/plans/2026-05-13-cilium-production-mode-hardening.md
Memory k3s-disable-list-vs-disable-flags — why disable: [..., kube-proxy] doesn’t work on k3s