Skip to content

Cilium kube-proxy Replacement (KPR)

kube-proxy does not run on this cluster. Cilium’s eBPF datapath handles every Service. This page states the settings that make that so, and the commands that replace the kube-proxy and iptables workflow when a Service misbehaves.

Setting Value Where
Service routing Cilium socket LB intercepts at the socket layer and an eBPF Service map holds the backends cilium_kube_proxy_replacement: "true" in ansible/roles/cilium/defaults/main.yml
API server endpoint for the agents k8sServiceHost is 127.0.0.1 on port 6444, stored in the pod environment of the agent, the operator and cilium-envoy as KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, not in cilium-config. On a server that port is kube-apiserver, on an agent it is the k3s client load balancer, so no node dials the API VIP cilium_k8s_service_host, cilium_k8s_service_port
Host-namespace routing eBPF host routing, bpf.hostLegacyRouting: false cilium_bpf_host_legacy_routing
Source NAT for pod-to-external traffic bpf.masquerade: true, required when host legacy routing is off cilium_bpf_masquerade
k3s embedded kube-proxy off, disable-kube-proxy: true in /etc/rancher/k3s/config.yaml on every server k3s_disable_kube_proxy in ansible/roles/k3s-common/defaults/main.yml
iptables -t nat KUBE-SVC-* chains none, only kubelet’s KUBE-KUBELET-CANARY the checks below
LoadBalancer addresses LB-IPAM assigns from the pool routed-vips (192.168.250.1 to .253, fddb:f665:73f7:250::1 to ::253) only to a Service with loadBalancerClass: io.cilium/bgp-control-plane in the namespace list of the pool. defaultLBServiceIPAM: none gives no address to a Service without the class cilium_default_lb_service_ipam, argocd/app-configs/cilium-lb/ip-pool.yaml
LoadBalancer announcement The BGP control plane announces each routed VIP as a host route from every node with a ready endpoint. Each node holds one session per family from its Lab leg to the UCG-Fiber cilium_bgp_control_plane_enabled, argocd/app-configs/cilium-lb/bgp.yaml, tf/unifi/bgp.tf
The API VIP 192.168.250.254 and fddb:f665:73f7:250::254, the Service kube-system/kubernetes-api in argocd/app-configs/cilium-lb/api-vip.yaml. The cilium role applies its two EndpointSlices from the control-plane host vars, because ArgoCD excludes that kind. Only the three servers announce it cilium_api_vip_service_name, ansible/roles/cilium/templates/api-vip-endpointslices.yaml.j2

The k3s disable: list does not accept kube-proxy. The separate top-level flag is the only way to turn it off.

Symptom First command
Service IP not reachable from a pod cilium-dbg service list | grep <service-cluster-ip>
Wrong backend pod selected cilium-dbg lb list
Real-time packet path kubectl exec ... cilium-dbg monitor --related-to <endpoint-id>
Is KPR active on an agent cilium-dbg status --verbose | grep -A 5 KubeProxyReplacement
Is kube-proxy gone iptables -t nat -S | grep -c '^-N KUBE-SVC' (expect 0)
External traffic has the wrong source IP read enable-bpf-masquerade in cilium-config
A routed VIP does not answer cilium-dbg bgp peers on a node with an endpoint, then vtysh -c 'show ip route bgp' on the gateway (UniFi BGP)
Terminal window
NODE=tpi-alpha-1
NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \
| awk -v n="$NODE" '$7==n{print $1}')
kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \
cilium-dbg status --verbose | grep -A 5 KubeProxyReplacement

The output includes Status: True and Socket LB: Enabled.

Terminal window
NODE=tpi-alpha-1
NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \
| awk -v n="$NODE" '$7==n{print $1}')
SERVICE_IP=$(kubectl get service -n <ns> <name> -o jsonpath='{.spec.clusterIP}')
kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \
cilium-dbg service list | grep "$SERVICE_IP"

The output shows the frontend (Service ClusterIP and port) and the backends (pod IP and port) with (active) markers.

Terminal window
NODE=tpi-alpha-1
NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \
| awk -v n="$NODE" '$7==n{print $1}')
EP_ID=$(kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \
cilium-dbg endpoint list | grep "<pod-namespace>/<pod-name>" | awk '{print $1}')
kubectl exec -n kube-system "$NODE_AGENT" -c cilium-agent -- \
cilium-dbg monitor --related-to "$EP_ID" --type policy-verdict --last 30

The output shows policy-verdict log lines with action ALLOWED for valid Service egress.

cilium-config carries the production values

Section titled “cilium-config carries the production values”
Terminal window
for k in kube-proxy-replacement enable-host-legacy-routing enable-bpf-masquerade; do
v=$(kubectl get cm -n kube-system cilium-config -o jsonpath="{.data.${k}}")
echo "$k = $v"
done
kube-proxy-replacement = true
enable-host-legacy-routing = false
enable-bpf-masquerade = true
Terminal window
ansible -i ansible/inventory/hosts.yml tp_cluster_controlplane -b -m shell \
-a "grep '^disable-kube-proxy:' /etc/rancher/k3s/config.yaml"

Every server shows disable-kube-proxy: true.

Terminal window
ansible -i ansible/inventory/hosts.yml tp_cluster_controlplane -b -m shell \
-a "iptables -t nat -S | grep -c '^-N KUBE-'"

Every server shows 1, which is kubelet’s own KUBE-KUBELET-CANARY chain. A higher count means kube-proxy is running.

  1. Make sure that the Service exists and has endpoints:

    Terminal window
    kubectl get service,endpoints -n <ns> <name>
  2. Make sure that Cilium programmed the Service on the source node with the check A Service is programmed on a node.

  3. Run cilium-dbg monitor from the source node’s agent and trigger the request from the source pod. Read the policy-verdict events on the source endpoint.

  1. Set k3s_disable_kube_proxy: false in ansible/roles/k3s-common/defaults/main.yml.

  2. Re-run the k3s servers:

    Terminal window
    cd ansible
    uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags k3s-install --limit tp_cluster_controlplane

    k3s restarts on each server, the embedded kube-proxy starts, and the KUBE-* chains return.

  3. To also turn off Cilium’s replacement, set cilium_kube_proxy_replacement: "false" and re-run the Cilium role with --tags cilium-install.