Cilium kube-proxy Replacement (KPR)
kube-proxy does not run on this cluster. Cilium’s eBPF datapath handles every Service. This page states the settings that make that so, and the commands that replace the kube-proxy and iptables workflow when a Service misbehaves.
What exists
Section titled “What exists”| Setting | Value | Where |
|---|---|---|
| Service routing | Cilium socket LB intercepts at the socket layer and an eBPF Service map holds the backends | cilium_kube_proxy_replacement: "true" in ansible/roles/cilium/defaults/main.yml |
| API server endpoint for the agents | k8sServiceHost is 127.0.0.1 on port 6444, stored in the pod environment of the agent, the operator and cilium-envoy as KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, not in cilium-config. On a server that port is kube-apiserver, on an agent it is the k3s client load balancer, so no node dials the API VIP |
cilium_k8s_service_host, cilium_k8s_service_port |
| Host-namespace routing | eBPF host routing, bpf.hostLegacyRouting: false |
cilium_bpf_host_legacy_routing |
| Source NAT for pod-to-external traffic | bpf.masquerade: true, required when host legacy routing is off |
cilium_bpf_masquerade |
| k3s embedded kube-proxy | off, disable-kube-proxy: true in /etc/rancher/k3s/config.yaml on every server |
k3s_disable_kube_proxy in ansible/roles/k3s-common/defaults/main.yml |
iptables -t nat KUBE-SVC-* chains |
none, only kubelet’s KUBE-KUBELET-CANARY |
the checks below |
| LoadBalancer addresses | LB-IPAM assigns from the pool routed-vips (192.168.250.1 to .253, fddb:f665:73f7:250::1 to ::253) only to a Service with loadBalancerClass: io.cilium/bgp-control-plane in the namespace list of the pool. defaultLBServiceIPAM: none gives no address to a Service without the class |
cilium_default_lb_service_ipam, argocd/app-configs/cilium-lb/ip-pool.yaml |
| LoadBalancer announcement | The BGP control plane announces each routed VIP as a host route from every node with a ready endpoint. Each node holds one session per family from its Lab leg to the UCG-Fiber | cilium_bgp_control_plane_enabled, argocd/app-configs/cilium-lb/bgp.yaml, tf/unifi/bgp.tf |
| The API VIP | 192.168.250.254 and fddb:f665:73f7:250::254, the Service kube-system/kubernetes-api in argocd/app-configs/cilium-lb/api-vip.yaml. The cilium role applies its two EndpointSlices from the control-plane host vars, because ArgoCD excludes that kind. Only the three servers announce it |
cilium_api_vip_service_name, ansible/roles/cilium/templates/api-vip-endpointslices.yaml.j2 |
The k3s disable: list does not accept kube-proxy. The separate top-level flag is the only way
to turn it off.
Symptom to first command
Section titled “Symptom to first command”| Symptom | First command |
|---|---|
| Service IP not reachable from a pod | cilium-dbg service list | grep <service-cluster-ip> |
| Wrong backend pod selected | cilium-dbg lb list |
| Real-time packet path | kubectl exec ... cilium-dbg monitor --related-to <endpoint-id> |
| Is KPR active on an agent | cilium-dbg status --verbose | grep -A 5 KubeProxyReplacement |
| Is kube-proxy gone | iptables -t nat -S | grep -c '^-N KUBE-SVC' (expect 0) |
| External traffic has the wrong source IP | read enable-bpf-masquerade in cilium-config |
| A routed VIP does not answer | cilium-dbg bgp peers on a node with an endpoint, then vtysh -c 'show ip route bgp' on the gateway (UniFi BGP) |
Checks
Section titled “Checks”KPR is active and healthy on a node
Section titled “KPR is active and healthy on a node”NODE=tpi-alpha-1NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \ | awk -v n="$NODE" '$7==n{print $1}')kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \ cilium-dbg status --verbose | grep -A 5 KubeProxyReplacementThe output includes Status: True and Socket LB: Enabled.
A Service is programmed on a node
Section titled “A Service is programmed on a node”NODE=tpi-alpha-1NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \ | awk -v n="$NODE" '$7==n{print $1}')SERVICE_IP=$(kubectl get service -n <ns> <name> -o jsonpath='{.spec.clusterIP}')kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \ cilium-dbg service list | grep "$SERVICE_IP"The output shows the frontend (Service ClusterIP and port) and the backends (pod IP and port)
with (active) markers.
An endpoint can reach a Service
Section titled “An endpoint can reach a Service”NODE=tpi-alpha-1NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \ | awk -v n="$NODE" '$7==n{print $1}')EP_ID=$(kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \ cilium-dbg endpoint list | grep "<pod-namespace>/<pod-name>" | awk '{print $1}')kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \ cilium-dbg monitor --related-to "$EP_ID" --type policy-verdict --last 30The output shows policy-verdict log lines with action ALLOWED for valid Service egress.
cilium-config carries the production values
Section titled “cilium-config carries the production values”for k in kube-proxy-replacement enable-host-legacy-routing enable-bpf-masquerade; do v=$(kubectl get cm -n kube-system cilium-config -o jsonpath="{.data.${k}}") echo "$k = $v"donekube-proxy-replacement = trueenable-host-legacy-routing = falseenable-bpf-masquerade = trueThe k3s embedded kube-proxy is off
Section titled “The k3s embedded kube-proxy is off”ansible -i ansible/inventory/hosts.yml tp_cluster_controlplane -b -m shell \ -a "grep '^disable-kube-proxy:' /etc/rancher/k3s/config.yaml"Every server shows disable-kube-proxy: true.
No stale KUBE-* iptables chains
Section titled “No stale KUBE-* iptables chains”ansible -i ansible/inventory/hosts.yml tp_cluster_controlplane -b -m shell \ -a "iptables -t nat -S | grep -c '^-N KUBE-'"Every server shows 1, which is kubelet’s own KUBE-KUBELET-CANARY chain. A higher count means
kube-proxy is running.
Procedures
Section titled “Procedures”Debug a Service that is not reachable
Section titled “Debug a Service that is not reachable”-
Make sure that the Service exists and has endpoints:
Terminal window kubectl get service,endpoints -n <ns> <name> -
Make sure that Cilium programmed the Service on the source node with the check A Service is programmed on a node.
-
Run
cilium-dbg monitorfrom the source node’s agent and trigger the request from the source pod. Read thepolicy-verdictevents on the source endpoint.
Re-enable kube-proxy
Section titled “Re-enable kube-proxy”-
Set
k3s_disable_kube_proxy: falseinansible/roles/k3s-common/defaults/main.yml. -
Re-run the k3s servers:
Terminal window cd ansibleuv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags k3s-install --limit tp_cluster_controlplanek3s restarts on each server, the embedded kube-proxy starts, and the
KUBE-*chains return. -
To also turn off Cilium’s replacement, set
cilium_kube_proxy_replacement: "false"and re-run the Cilium role with--tags cilium-install.
Related pages
Section titled “Related pages”- Cilium operations
- Network reference
docs/engineering/specs/2026-05-13-cilium-production-mode-hardening-design.md, the design