Skip to content

Cilium Operations Runbook

Cilium is the cluster’s CNI. This page states what exists and where it is set, the procedures that touch it, and a check for each. The Lab VLAN leg that Cilium manages as a second device is in Lab service VLAN. kube-proxy replacement has its own page.

ansible/roles/cilium installs the chart cilium/cilium from https://helm.cilium.io, from the play Install Cilium CNI in ansible/k3s-playbook.yml (tags cilium, cilium-install, k3s-cilium). The role keeps every feature flag in defaults/main.yml, and there is no override layer. templates/values.yaml.j2 renders the chart values.

Setting Value Variable
Version 1.20.1 cilium_version
Namespace kube-system cilium_namespace
Pod CIDR, IPv4 10.245.0.0/16, one /24 per node cilium_cluster_pool_ipv4_cidr, cilium_cluster_pool_ipv4_mask_size
Pod CIDR, IPv6 fd00:10:245::/48, one /64 per node cilium_cluster_pool_ipv6_cidr, cilium_cluster_pool_ipv6_mask_size
IPAM cluster-pool, both families the template
Routing tunnel mode, VXLAN, UDP port 8473 cilium_routing_mode, cilium_tunnel_protocol, cilium_tunnel_port
Devices end0, end0.1000 and end0.3000, no wildcard cilium_devices
kube-proxy replacement "true", API endpoint k8sServiceHost = 127.0.0.1 on port 6444, kube-apiserver on a server and the k3s client load balancer on an agent cilium_kube_proxy_replacement, cilium_k8s_service_host, cilium_k8s_service_port
BPF host routing on (hostLegacyRouting: false), BPF masquerade on cilium_bpf_host_legacy_routing, cilium_bpf_masquerade
Policy enforcement default cilium_policy_enforcement_mode
L7 proxy on, DNS proxy transparent mode off cilium_l7_proxy_enabled, cilium_dns_proxy_transparent_mode
Hubble agent, relay and UI on, metrics dns drop tcp flow icmp http cilium_hubble_*
Gateway API controller on, seven Standard CRD kinds vendored cilium_gateway_api_enabled, files/gateway-api-crds/
Encryption WireGuard, pod to pod, node encryption off cilium_encryption_*

The device pin keeps tailscale0 out of the NodePort and masquerade datapath, and listing end0.1000 and end0.3000 lets tagged Lab and Main frames through Cilium’s VLAN filter on end0. Cilium drops a tagged frame at end0 for a VLAN device that the list does not name. A restart of the agent never adopts an unlisted device, so the list changes before a leg moves onto a tag. Each node’s InternalIP is its Main address, which sits on end0.3000.

The Service CIDR is 10.43.0.0/16 and fd00:10:43::/112, set as k8s_service_cidr in ansible/inventory/group_vars/tp_cluster_nodes.yml and rendered into the k3s configuration.

k8s_cluster_cidr in the same group_vars file reads 10.42.0.0/16,fd00:10:245::/48, and k3s writes it as cluster-cidr. That value populates Node.spec.podCIDR. Cilium in cluster-pool mode ignores Node.spec.podCIDR and writes pod ranges from its own pools to CiliumNode.spec.ipam.podCIDRs. The two coexist. A change to the k3s value restarts k3s on every server, so it stays as it is.

The role installs the seven Standard CRDs from files/gateway-api-crds/ (gatewayclasses, gateways, httproutes, grpcroutes, referencegrants, tlsroutes, backendtlspolicies) and then the safe-upgrades.gateway.networking.k8s.io ValidatingAdmissionPolicy from the same directory, before the Helm release. The kinds are listed in cilium_gateway_api_crd_files. Every vendored file carries gateway.networking.k8s.io/bundle-version: v1.6.2, and the role variable cilium_gateway_api_crd_version reads the same value. Before the role applies the files, an assert task makes sure that every file carries the version the variable names. If Renovate bumps the variable, the role fails until the files are re-vendored from the matching upstream release. The repository declares no Gateway, GatewayClass or HTTPRoute, so Cilium’s gateway controller serves nothing.

To re-vendor the bundle, run this from the repository root with the new tag in GAPI_VERSION:

Terminal window
GAPI_VERSION=v1.6.2
BASE="https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/${GAPI_VERSION}/config/crd/standard"
DEST=ansible/roles/cilium/files/gateway-api-crds
for crd in gatewayclasses gateways httproutes grpcroutes referencegrants tlsroutes backendtlspolicies; do
curl -fsSL "${BASE}/gateway.networking.k8s.io_${crd}.yaml" -o "${DEST}/${crd}.yaml"
done
curl -fsSL "${BASE}/gateway.networking.k8s.io_vap_safeupgrades.yaml" -o "${DEST}/safe-upgrades-policy.yaml"
rg -o -N --no-filename 'bundle-version: v[0-9.]+' "${DEST}" | sort -u # one line, equal to GAPI_VERSION
Terminal window
# Cluster health
cilium status --wait
cilium connectivity test # full e2e suite, about 5 minutes
cilium connectivity test --hubble=false # quick smoke
# Hubble flows
cilium hubble observe --last 100
cilium hubble observe --since 5m --verdict DROPPED # policy drops
cilium hubble observe --pod NAMESPACE/POD-NAME
cilium hubble ui # opens the UI through a port-forward
Terminal window
# Per-node CNI state
kubectl get ciliumnodes
kubectl get ciliumendpoints -A | head -20
# Policy resources
kubectl get ciliumnetworkpolicies -A
kubectl get ciliumclusterwidenetworkpolicies
# Agent diagnostics
kubectl exec -n kube-system ds/cilium -- cilium-dbg status
kubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint list
Terminal window
kubectl get ciliumnodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}'

The Hubble UI is on https://hubble.fzymgc.house behind the shared oauth2-proxy forward auth. Hubble UI operations covers access and the group that grants it. A port-forward works without the ingress:

Terminal window
cilium hubble ui
# or
kubectl port-forward -n kube-system svc/hubble-ui 12000:80

The cluster-wide policyEnforcementMode is default. A pod that no CiliumNetworkPolicy selects is allowed everything. A pod that a policy selects gets only what the policy permits, and each direction-specific policy engages default deny for its direction on its own, so no baseline policy is needed.

The repository declares fourteen CiliumNetworkPolicies in nine namespaces and no CiliumClusterwideNetworkPolicy:

Namespace Policies Manifest
clickhouse-mcp 1 argocd/app-configs/clickhouse-mcp/networkpolicy.yaml
clickstack 1 argocd/app-configs/clickstack/networkpolicy.yaml
dolt 5 argocd/app-configs/dolt/network-policies.yaml
firewalla-mcp 1 argocd/app-configs/firewalla-mcp/networkpolicy.yaml
headroom-agents 1 argocd/app-configs/headroom-agents/networkpolicy.yaml
hl-assets 1 argocd/app-configs/hl-assets/networkpolicy.yaml
kubernetes-mcp 1 argocd/app-configs/kubernetes-mcp/networkpolicy.yaml
litellm 2 argocd/app-configs/litellm/networkpolicy.yaml

Read what is in force with kubectl get ciliumnetworkpolicies -A. To make sure that enforcement works, run a probe pod in a namespace without a policy and connect to a workload in a policed namespace. cilium hubble observe --verdict DROPPED shows Policy denied for the attempt.

hubble.metrics.enabled includes dns, so DNS flow events appear in hubble observe --type l7 and in the Hubble UI service map. Make sure that it works:

Terminal window
cilium hubble observe --type l7 --protocol dns --last 20

No policy in the repository carries an L7 rules.dns block or a toFQDNs rule. On this cluster a policy with an L7 DNS rule leaves the pods it selects unable to resolve names on most of the nodes, and the agents on those nodes log nothing about it. toFQDNs depends on that rule, because Cilium fills its FQDN cache only from DNS proxy observation. Turning dnsProxy.enableTransparentMode on does not change which nodes fail, so it stays off. Namespaces that need egress to named hosts use the coarse toEntities: world allow instead, and argocd/app-configs/dolt/network-policies.yaml stays L4-only for the same reason. DNS visibility through Hubble metrics is not affected.

The standalone DNS proxy is not a workaround. The published 1.20.1 chart ships no image for it, so its DaemonSet cannot start.

A CiliumClusterwideNetworkPolicy with enableDefaultDeny: {egress: false, ingress: false} does not engage L7 enforcement, so it cannot provide cluster-wide DNS visibility either.

Pod-to-pod traffic is encrypted cluster-wide with Cilium’s WireGuard transparent encryption. That keeps in-cluster credentials off the cleartext overlay, for example the HyperDX OTLP ingest token that senders push to cs-otel-collector:4317 over h2c.

The role sets cilium_encryption_enabled: true, cilium_encryption_type: wireguard and cilium_encryption_node_encryption: false. Node-to-node encryption is beta, and pod-to-pod covers the credentials, because senders and the collector are all pods. The chart renders enable-wireguard: "true" into cilium-config. Each agent brings up cilium_wg0 on UDP 51871, distributes peer public keys through the CiliumNode objects, and lowers the MTU for the WireGuard and VXLAN overhead.

Make sure that encryption is active:

Terminal window
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep Encryption
# Encryption: Wireguard [cilium_wg0 (Pubkey: ..., Port: 51871, Peers: 7)]
kubectl -n kube-system exec ds/cilium -- cilium-dbg encrypt status

To make sure that the OTLP token no longer crosses the wire in cleartext, capture on a node that carries an OTLP sender. Capture the physical path, not cilium_wg0, which carries the plaintext before encryption:

Terminal window
sudo tcpdump -ni any 'udp port 51871' -c 5 # WireGuard packets
sudo tcpdump -nAi any 'tcp port 4317' -c 20 | grep -i authorization # expect no match

The grep authorization line is a secondary signal. Over h2c the bearer token rides in HPACK-compressed HTTP/2 HEADERS frames, so a literal authorization string does not appear even on a cleartext capture. The proof is that inter-node pod-to-collector traffic travels inside the UDP 51871 tunnel and no cleartext TCP 4317 flow crosses the wire between nodes.

To turn encryption off, set cilium_encryption_enabled: false in the role defaults and re-run the role (--tags cilium). Cilium tears down cilium_wg0 and returns to the cleartext overlay, with a short pod-to-pod connectivity blip per node.

A Cilium version downgrade is not supported, and no symmetric rollback exists. Three things move forward with cilium_version and none of them comes back on its own:

  • the cilium.io CRD schemas the cluster validates every CiliumNetworkPolicy against.

  • the vendored Gateway API bundle, which the chart requires at a matching minimum.

  • the safe-upgrades.gateway.networking.k8s.io ValidatingAdmissionPolicy, which ships inside that bundle and reads its own bundle-version annotation to set its deny floor. Re-applying an older CRD set is refused by the cluster’s own admission chain. Read the floor it enforces now:

    Terminal window
    kubectl get validatingadmissionpolicy safe-upgrades.gateway.networking.k8s.io -o yaml | grep -n 'bundle-version'

Do not run helm rollback. The recovery is a forward re-run at the prior chart pin, with the CRDs left forward:

  1. Set cilium_version back to the prior patch or minor in ansible/roles/cilium/defaults/main.yml. Leave cilium_gateway_api_crd_version and the vendored CRDs alone. They stay forward.

  2. Re-run the role:

    Terminal window
    cd ansible
    uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium-install

    Do not reorder the role’s tasks. It installs the Gateway API CRDs and then the admission policy before the Helm release. That order is what admits a bundle at all. The reverse order has the policy adjudicate the CRDs it shipped with.

  3. Watch the roll. The DaemonSet rolls under the chart’s rolling update strategy, so the agents restart in batches, not all at once. Read the strategy with kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy}'. The role’s own rollout status task blocks until observedGeneration catches up, so a hung roll fails the play rather than passing silently.

  4. Make sure that the release and the agents agree:

    Terminal window
    helm -n kube-system list --filter cilium # chart version and revision; STATUS deployed
    kubectl -n kube-system get ds cilium -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
    kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose | head -20
  5. If one agent does not come back, delete that node’s agent pod and let the DaemonSet recreate it, then re-read cilium-dbg status --verbose on the replacement:

    Terminal window
    kubectl -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName=<node>

    Do not cordon or drain as a first move. On this cluster the CNI is what a drain needs in order to work.

The local Helm cache can block the re-run. kubernetes.core.helm_repository strips a trailing / from the configured repo_url and compares the stripped form against the raw cached URL, so a cache entry that holds https://helm.cilium.io/ fails the task with Repository already have a repository named cilium while helm repo list reads correct. The role carries force_update: true, which makes the task authoritative over the cache. From an older checkout without it, normalise the cache first:

Terminal window
helm repo add cilium https://helm.cilium.io --force-update

Two of the role’s tasks are unconditional by design: the repository add (force_update: true) and the rollout restart, because helm upgrade alone does not roll the DaemonSet when only ConfigMap values change. The signal to read is the Helm task. A configuration that changes nothing reports the Helm task ok with only those two changed. A Helm task that reports changed on a run you believed was a no-op means the rendered values moved.

  • Cilium’s gateway controller is on and serves nothing, because the repository declares no Gateway or HTTPRoute. Whether to keep cilium_gateway_api_enabled on is open.
  • Native routing (cilium_auto_direct_node_routes: true) is viable, because all nodes share one L2 segment, and it would remove the VXLAN encapsulation overhead. It is a separate change.