Cilium Operations Runbook
Cilium is the cluster’s CNI. This page states what exists and where it is set, the procedures that touch it, and a check for each. The Lab VLAN leg that Cilium manages as a second device is in Lab service VLAN. kube-proxy replacement has its own page.
What exists
Section titled “What exists”ansible/roles/cilium installs the chart cilium/cilium from https://helm.cilium.io, from the
play Install Cilium CNI in ansible/k3s-playbook.yml (tags cilium, cilium-install,
k3s-cilium). The role keeps every feature flag in defaults/main.yml, and there is no override
layer. templates/values.yaml.j2 renders the chart values.
| Setting | Value | Variable |
|---|---|---|
| Version | 1.20.1 |
cilium_version |
| Namespace | kube-system |
cilium_namespace |
| Pod CIDR, IPv4 | 10.245.0.0/16, one /24 per node |
cilium_cluster_pool_ipv4_cidr, cilium_cluster_pool_ipv4_mask_size |
| Pod CIDR, IPv6 | fd00:10:245::/48, one /64 per node |
cilium_cluster_pool_ipv6_cidr, cilium_cluster_pool_ipv6_mask_size |
| IPAM | cluster-pool, both families |
the template |
| Routing | tunnel mode, VXLAN, UDP port 8473 | cilium_routing_mode, cilium_tunnel_protocol, cilium_tunnel_port |
| Devices | end0, end0.1000 and end0.3000, no wildcard |
cilium_devices |
| kube-proxy replacement | "true", API endpoint k8sServiceHost = 127.0.0.1 on port 6444, kube-apiserver on a server and the k3s client load balancer on an agent |
cilium_kube_proxy_replacement, cilium_k8s_service_host, cilium_k8s_service_port |
| BPF host routing | on (hostLegacyRouting: false), BPF masquerade on |
cilium_bpf_host_legacy_routing, cilium_bpf_masquerade |
| Policy enforcement | default |
cilium_policy_enforcement_mode |
| L7 proxy | on, DNS proxy transparent mode off | cilium_l7_proxy_enabled, cilium_dns_proxy_transparent_mode |
| Hubble | agent, relay and UI on, metrics dns drop tcp flow icmp http |
cilium_hubble_* |
| Gateway API | controller on, seven Standard CRD kinds vendored | cilium_gateway_api_enabled, files/gateway-api-crds/ |
| Encryption | WireGuard, pod to pod, node encryption off | cilium_encryption_* |
The device pin keeps tailscale0 out of the NodePort and masquerade datapath, and listing
end0.1000 and end0.3000 lets tagged Lab and Main frames through Cilium’s VLAN filter on
end0. Cilium drops a tagged frame at end0 for a VLAN device that the list does not name. A
restart of the agent never adopts an unlisted device, so the list changes before a leg moves onto
a tag. Each
node’s InternalIP is its Main address, which sits on end0.3000.
The Service CIDR is 10.43.0.0/16 and fd00:10:43::/112, set as k8s_service_cidr in
ansible/inventory/group_vars/tp_cluster_nodes.yml and rendered into the k3s configuration.
The k3s cluster CIDR
Section titled “The k3s cluster CIDR”k8s_cluster_cidr in the same group_vars file reads 10.42.0.0/16,fd00:10:245::/48, and k3s
writes it as cluster-cidr. That value populates Node.spec.podCIDR. Cilium in cluster-pool
mode ignores Node.spec.podCIDR and writes pod ranges from its own pools to
CiliumNode.spec.ipam.podCIDRs. The two coexist. A change to the k3s value restarts k3s on every
server, so it stays as it is.
The Gateway API bundle
Section titled “The Gateway API bundle”The role installs the seven Standard CRDs from files/gateway-api-crds/ (gatewayclasses,
gateways, httproutes, grpcroutes, referencegrants, tlsroutes, backendtlspolicies) and
then the safe-upgrades.gateway.networking.k8s.io ValidatingAdmissionPolicy from the same
directory, before the Helm release. The kinds are listed in cilium_gateway_api_crd_files. Every
vendored file carries gateway.networking.k8s.io/bundle-version: v1.6.2, and the role variable
cilium_gateway_api_crd_version reads the same value. Before the role applies the files, an
assert task makes sure that every file carries the version the variable names. If Renovate bumps
the variable, the role fails until the files are re-vendored from the matching upstream release.
The repository declares no Gateway, GatewayClass or HTTPRoute, so Cilium’s gateway
controller serves nothing.
To re-vendor the bundle, run this from the repository root with the new tag in GAPI_VERSION:
GAPI_VERSION=v1.6.2BASE="https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/${GAPI_VERSION}/config/crd/standard"DEST=ansible/roles/cilium/files/gateway-api-crdsfor crd in gatewayclasses gateways httproutes grpcroutes referencegrants tlsroutes backendtlspolicies; do curl -fsSL "${BASE}/gateway.networking.k8s.io_${crd}.yaml" -o "${DEST}/${crd}.yaml"donecurl -fsSL "${BASE}/gateway.networking.k8s.io_vap_safeupgrades.yaml" -o "${DEST}/safe-upgrades-policy.yaml"rg -o -N --no-filename 'bundle-version: v[0-9.]+' "${DEST}" | sort -u # one line, equal to GAPI_VERSIONCommon commands
Section titled “Common commands”cilium-cli (workstation)
Section titled “cilium-cli (workstation)”# Cluster healthcilium status --waitcilium connectivity test # full e2e suite, about 5 minutescilium connectivity test --hubble=false # quick smoke
# Hubble flowscilium hubble observe --last 100cilium hubble observe --since 5m --verdict DROPPED # policy dropscilium hubble observe --pod NAMESPACE/POD-NAMEcilium hubble ui # opens the UI through a port-forwardkubectl
Section titled “kubectl”# Per-node CNI statekubectl get ciliumnodeskubectl get ciliumendpoints -A | head -20
# Policy resourceskubectl get ciliumnetworkpolicies -Akubectl get ciliumclusterwidenetworkpolicies
# Agent diagnosticskubectl exec -n kube-system ds/cilium -- cilium-dbg statuskubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint listPod CIDR allocation per node
Section titled “Pod CIDR allocation per node”kubectl get ciliumnodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}'Hubble UI
Section titled “Hubble UI”The Hubble UI is on https://hubble.fzymgc.house behind the shared oauth2-proxy forward auth.
Hubble UI operations covers access and the group that grants it. A
port-forward works without the ingress:
cilium hubble ui# orkubectl port-forward -n kube-system svc/hubble-ui 12000:80NetworkPolicy enforcement
Section titled “NetworkPolicy enforcement”The cluster-wide policyEnforcementMode is default. A pod that no CiliumNetworkPolicy selects is
allowed everything. A pod that a policy selects gets only what the policy permits, and each
direction-specific policy engages default deny for its direction on its own, so no baseline
policy is needed.
The repository declares fourteen CiliumNetworkPolicies in nine namespaces and no CiliumClusterwideNetworkPolicy:
| Namespace | Policies | Manifest |
|---|---|---|
clickhouse-mcp |
1 | argocd/app-configs/clickhouse-mcp/networkpolicy.yaml |
clickstack |
1 | argocd/app-configs/clickstack/networkpolicy.yaml |
dolt |
5 | argocd/app-configs/dolt/network-policies.yaml |
firewalla-mcp |
1 | argocd/app-configs/firewalla-mcp/networkpolicy.yaml |
headroom-agents |
1 | argocd/app-configs/headroom-agents/networkpolicy.yaml |
hl-assets |
1 | argocd/app-configs/hl-assets/networkpolicy.yaml |
kubernetes-mcp |
1 | argocd/app-configs/kubernetes-mcp/networkpolicy.yaml |
litellm |
2 | argocd/app-configs/litellm/networkpolicy.yaml |
Read what is in force with kubectl get ciliumnetworkpolicies -A. To make sure that enforcement
works, run a probe pod in a namespace without a policy and connect to a workload in a policed
namespace. cilium hubble observe --verdict DROPPED shows Policy denied for the attempt.
DNS L7 visibility (cluster-wide)
Section titled “DNS L7 visibility (cluster-wide)”hubble.metrics.enabled includes dns, so DNS flow events appear in hubble observe --type l7
and in the Hubble UI service map. Make sure that it works:
cilium hubble observe --type l7 --protocol dns --last 20No policy in the repository carries an L7 rules.dns block or a toFQDNs rule. On this cluster a
policy with an L7 DNS rule leaves the pods it selects unable to resolve names on most of the
nodes, and the agents on those nodes log nothing about it. toFQDNs depends on that rule, because
Cilium fills its FQDN cache only from DNS proxy observation. Turning dnsProxy.enableTransparentMode
on does not change which nodes fail, so it stays off. Namespaces that need egress to named hosts
use the coarse toEntities: world allow instead, and argocd/app-configs/dolt/network-policies.yaml
stays L4-only for the same reason. DNS visibility through Hubble metrics is not affected.
The standalone DNS proxy is not a workaround. The published 1.20.1 chart ships no image for it, so its DaemonSet cannot start.
A CiliumClusterwideNetworkPolicy with enableDefaultDeny: {egress: false, ingress: false}
does not engage L7 enforcement, so it cannot provide cluster-wide DNS visibility either.
Transparent encryption (WireGuard)
Section titled “Transparent encryption (WireGuard)”Pod-to-pod traffic is encrypted cluster-wide with Cilium’s WireGuard transparent encryption. That
keeps in-cluster credentials off the cleartext overlay, for example the HyperDX OTLP ingest token
that senders push to cs-otel-collector:4317 over h2c.
The role sets cilium_encryption_enabled: true, cilium_encryption_type: wireguard and
cilium_encryption_node_encryption: false. Node-to-node encryption is beta, and pod-to-pod covers
the credentials, because senders and the collector are all pods. The chart renders
enable-wireguard: "true" into cilium-config. Each agent brings up cilium_wg0 on UDP 51871,
distributes peer public keys through the CiliumNode objects, and lowers the MTU for the WireGuard
and VXLAN overhead.
Make sure that encryption is active:
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep Encryption# Encryption: Wireguard [cilium_wg0 (Pubkey: ..., Port: 51871, Peers: 7)]kubectl -n kube-system exec ds/cilium -- cilium-dbg encrypt statusTo make sure that the OTLP token no longer crosses the wire in cleartext, capture on a node that
carries an OTLP sender. Capture the physical path, not cilium_wg0, which carries the plaintext
before encryption:
sudo tcpdump -ni any 'udp port 51871' -c 5 # WireGuard packetssudo tcpdump -nAi any 'tcp port 4317' -c 20 | grep -i authorization # expect no matchThe grep authorization line is a secondary signal. Over h2c the bearer token rides in
HPACK-compressed HTTP/2 HEADERS frames, so a literal authorization string does not appear even
on a cleartext capture. The proof is that inter-node pod-to-collector traffic travels inside the
UDP 51871 tunnel and no cleartext TCP 4317 flow crosses the wire between nodes.
To turn encryption off, set cilium_encryption_enabled: false in the role defaults and re-run
the role (--tags cilium). Cilium tears down cilium_wg0 and returns to the cleartext overlay,
with a short pod-to-pod connectivity blip per node.
Version upgrade recovery
Section titled “Version upgrade recovery”A Cilium version downgrade is not supported, and no symmetric rollback exists. Three things move
forward with cilium_version and none of them comes back on its own:
-
the
cilium.ioCRD schemas the cluster validates every CiliumNetworkPolicy against. -
the vendored Gateway API bundle, which the chart requires at a matching minimum.
-
the
safe-upgrades.gateway.networking.k8s.ioValidatingAdmissionPolicy, which ships inside that bundle and reads its ownbundle-versionannotation to set its deny floor. Re-applying an older CRD set is refused by the cluster’s own admission chain. Read the floor it enforces now:Terminal window kubectl get validatingadmissionpolicy safe-upgrades.gateway.networking.k8s.io -o yaml | grep -n 'bundle-version'
Do not run helm rollback. The recovery is a forward re-run at the prior chart pin, with the CRDs
left forward:
-
Set
cilium_versionback to the prior patch or minor inansible/roles/cilium/defaults/main.yml. Leavecilium_gateway_api_crd_versionand the vendored CRDs alone. They stay forward. -
Re-run the role:
Terminal window cd ansibleuv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium-installDo not reorder the role’s tasks. It installs the Gateway API CRDs and then the admission policy before the Helm release. That order is what admits a bundle at all. The reverse order has the policy adjudicate the CRDs it shipped with.
-
Watch the roll. The DaemonSet rolls under the chart’s rolling update strategy, so the agents restart in batches, not all at once. Read the strategy with
kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy}'. The role’s ownrollout statustask blocks untilobservedGenerationcatches up, so a hung roll fails the play rather than passing silently. -
Make sure that the release and the agents agree:
Terminal window helm -n kube-system list --filter cilium # chart version and revision; STATUS deployedkubectl -n kube-system get ds cilium -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose | head -20 -
If one agent does not come back, delete that node’s agent pod and let the DaemonSet recreate it, then re-read
cilium-dbg status --verboseon the replacement:Terminal window kubectl -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName=<node>Do not cordon or drain as a first move. On this cluster the CNI is what a drain needs in order to work.
Before you run it
Section titled “Before you run it”The local Helm cache can block the re-run. kubernetes.core.helm_repository strips a trailing /
from the configured repo_url and compares the stripped form against the raw cached URL, so a
cache entry that holds https://helm.cilium.io/ fails the task with Repository already have a repository named cilium while helm repo list reads correct. The role carries
force_update: true, which makes the task authoritative over the cache. From an older checkout
without it, normalise the cache first:
helm repo add cilium https://helm.cilium.io --force-updateTwo of the role’s tasks are unconditional by design: the repository add (force_update: true)
and the rollout restart, because helm upgrade alone does not roll the DaemonSet when only
ConfigMap values change. The signal to read is the Helm task. A configuration that changes nothing
reports the Helm task ok with only those two changed. A Helm task that reports changed on a
run you believed was a no-op means the rendered values moved.
Open simplifications
Section titled “Open simplifications”- Cilium’s gateway controller is on and serves nothing, because the repository declares no
GatewayorHTTPRoute. Whether to keepcilium_gateway_api_enabledon is open. - Native routing (
cilium_auto_direct_node_routes: true) is viable, because all nodes share one L2 segment, and it would remove the VXLAN encapsulation overhead. It is a separate change.
References
Section titled “References”- Cilium 1.20 docs
- Cilium 1.20 Helm values reference
- Hubble docs
ansible/roles/cilium/, the role sourcedocs/engineering/specs/2026-05-10-calico-to-cilium-design.md, the design