Cilium Operations Runbook

Evergreen operational guide for the cluster’s Cilium 1.20.1 CNI deployment. For the one-time Calico→Cilium migration log, see cilium-migration.md.

Deployment

Installed by: ansible/roles/cilium (Phase 5 of ansible/k3s-playbook.yml)
Chart: cilium/cilium from https://helm.cilium.io (no trailing slash — see Version upgrade recovery)
Version: 1.20.1 (pinned as cilium_version in roles/cilium/defaults/main.yml; moved off the previous minor 2026-09-08)
Namespace: kube-system
Pod CIDR: 10.245.0.0/16 (IPv4) + fd00:10:245::/48 (IPv6)
Service CIDR: 10.43.0.0/16 (k3s default; unchanged from Calico era)
Routing: tunnel mode, VXLAN protocol, port 8473
Devices: explicit devices list, end0 and end0.1000 (cilium_devices), with no wildcard. The pin keeps tailscale0 out of the NodePort/masquerade datapath, and listing end0.1000 lets tagged Lab frames through Cilium’s VLAN filter on end0. Direct routing stays on end0 because each node’s InternalIP is its Main address. The leg itself is described in Lab service VLAN.
kube-proxy: replaced (kubeProxyReplacement: "true", flipped per hl-3ed)
Gateway API: v1.6.1 Standard CRDs, seven kinds, vendored under roles/cilium/files/gateway-api-crds/ and installed by this role (gatewayclasses, gateways, httproutes, grpcroutes, referencegrants, tlsroutes, backendtlspolicies — tlsroutes and backendtlspolicies were added for 1.20, which requires them). Measured 2026-09-08: the CRDs this role installs are consumed by the other controller. Two GatewayClasses are Accepted=True — cilium (io.cilium/gateway-controller) and agentgateway (agentgateway.dev/agentgateway) — and the one Gateway that exists cluster-wide, agentgateway/agentgateway, is on the agentgateway class with 22 HTTPRoutes attached. Cilium’s own GatewayClass serves zero Gateways. A regression from a CRD bundle move therefore surfaces in agentgateway, not in Cilium’s status.

k3s –cluster-cidr note

k3s-common/defaults/main.yml still references 10.42.0.0/16 as the k3s cluster CIDR. This is intentionally retained post-Calico-removal:

Cilium owns IPAM via ipam.mode: cluster-pool and writes pod IPs from 10.245.0.0/16 to CiliumNode.spec.ipam.podCIDRs.
k3s passes --cluster-cidr=10.42.0.0/16 to kube-controller-manager-equivalent, which populates Node.spec.podCIDR from that range.
Cilium IGNORES Node.spec.podCIDR when in cluster-pool mode; the two coexist harmlessly.
Changing the k3s value would require a server restart on every control-plane node — out of scope for any non-emergency operation.

Common commands

cilium-cli (workstation)

# Cluster health
cilium status --wait
cilium connectivity test     # full e2e suite — takes ~5 minutes
cilium connectivity test --hubble=false  # quick smoke

# Hubble flows
cilium hubble observe --last 100
cilium hubble observe --since 5m --verdict DROPPED  # see policy drops
cilium hubble observe --pod NAMESPACE/POD-NAME
cilium hubble ui            # opens browser to internal UI via port-forward

kubectl

# Per-node CNI state
kubectl get ciliumnodes
kubectl get ciliumendpoints -A | head -20

# Policy resources
kubectl get ciliumnetworkpolicies -A
kubectl get ciliumclusterwidenetworkpolicies

# Agent diagnostics
kubectl exec -n kube-system ds/cilium -- cilium-dbg status
kubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint list

Hubble UI access

Hubble UI is internal-only (no IngressRoute by design).

cilium hubble ui  # auto-port-forwards and opens browser
# Or manually:
kubectl port-forward -n kube-system svc/hubble-ui 12000:80

Pod CIDR allocation per node

Each node gets a /24 from the cluster pool. View allocation:

kubectl get ciliumnodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}'

NetworkPolicy enforcement

Cluster-wide policyEnforcementMode: default — pods without matching CiliumNetworkPolicy are allowed by default; pods with matching policies are restricted to what the policies permit.

Crown-jewel namespace coverage is tracked under the post-migration hardening initiative (see docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md).

Currently enforced — 13 CiliumNetworkPolicies across 9 namespaces, measured 2026-09-08. Those are two different numbers and both are stated here deliberately: dolt alone carries 5 of the 13, so reading the namespace count as a policy count is wrong.

clickhouse-mcp — 1 CNP: ingress on 8000 from the fronting gateways; egress to ClickHouse (8123) and kube-dns (53). See argocd/app-configs/clickhouse-mcp/networkpolicy.yaml.
clickstack — 1 CNP: OTLP ingest (4317/4318), ClickHouse (8123), the collector’s own 8888, kube-dns, and toEntities: world on 443. See argocd/app-configs/clickstack/networkpolicy.yaml.
dolt — 5 CNPs covering SQL (3306), Remote API (50051), Prometheus metrics (9418), L4-only DNS egress to kube-dns, and kube-apiserver egress. See argocd/app-configs/dolt/network-policies.yaml.
firewalla-mcp — 1 CNP: ingress on 8080 from the fronting gateways; egress to kube-dns and toEntities: world on 443. See argocd/app-configs/firewalla-mcp/networkpolicy.yaml.
fovea — 1 CNP scoped to the qdrant-fovea endpoint: gRPC 6334 from the fovea client only, and 6333 from the host entity for the kubelet readiness probe. See argocd/app-configs/fovea/qdrant-netpol.yaml.
headroom-agents — 1 CNP: ingress on 8787; egress to kube-dns, OTLP 4318, and toEntities: world on 443. See argocd/app-configs/headroom-agents/networkpolicy.yaml.
headroom-apps — 1 CNP: ingress on 8787; egress to kube-dns, OTLP 4318, Traefik 8443, and toEntities: world on 443. See argocd/app-configs/headroom-apps/networkpolicy.yaml.
hl-assets — 1 CNP (hl-assets-egress; the name is historical, it now carries an ingress half too): ingress on 8080, egress to kube-dns, OTLP 4318, and toEntities: world on 443. See argocd/app-configs/hl-assets/networkpolicy.yaml.
kubernetes-mcp — 1 CNP: ingress on 8080 from agentgateway and litellm only; egress to kube-dns and toEntities: kube-apiserver. See argocd/app-configs/kubernetes-mcp/networkpolicy.yaml.

Zero of the 13 carry an L7 rules.dns block. The namespaces that need egress to named hosts are all on the coarse toEntities: world fallback rather than toFQDNs — see DNS L7 visibility for why that is a measured decision and not an oversight.

Each direction-specific CNP engages default-deny for its direction automatically — no separate baseline CNP needed. To verify enforcement empirically: run a probe pod in an unauthorized namespace and try to reach a workload in a policy-covered namespace; expect Policy denied DROPPED in hubble observe.

Crown-jewel coverage for cert-manager / external-secrets / vault was reverted (PRs #1044+#1045+#1048) after the cluster’s ESO ClusterSecretStore failed to recover post-merge. A more careful redesign is tracked for future work.

DNS L7 visibility (cluster-wide)

One chart-level setting surfaces cluster-wide DNS visibility (see ansible/roles/cilium/templates/values.yaml.j2):

hubble.metrics.enabled includes dns — surfaces DNS flow events visible via hubble observe --type l7 and in the Hubble UI (hubble.fzymgc.house) service map.

dnsProxy.enableTransparentMode is false — and on this cluster that choice is immaterial to whether an L7 DNS rule works. Measured 2026-09-08 and 2026-09-09 on Cilium 1.20.1, one probe pod per node across all eight nodes, one identical configuration per sweep:

Transparent mode off — a policy carrying rules.dns with matchPattern resolves on tpi-alpha-1 and tpi-alpha-3 and fails on the other six nodes, with the DNS redirect installed and the FQDN cache empty (connection timed out; no servers could be reached). The failing agents emit no log line at all — no error, no warning — which is why this went four months without being localised.
Transparent mode on — the split is identical. The same two nodes work and the same six fail. Turning it on broke DNS for nothing (every unpolicied control passed on all eight nodes, and all nine policy-bound namespaces kept resolving), and it repaired nothing either.
The standalone DNS proxy is not an option and not a mitigation — the published 1.20.1 chart ships no image for it (repository, tag and digest all empty with useDigest: true), so its DaemonSet renders image: "@" and reached 0 of 8 ready with InvalidImageName. Running it would mean building, signing and hosting a custom ARM64 image: a project, not a configuration value.

So transparent mode is left off because nothing measured argues for moving it, not because moving it would break DNS. What decides whether an L7 DNS rule works here is the node, and that is still open: the next step is a node-level diff of the two working nodes against the six failing ones (kernel and Armbian build, NIC and driver, the TPROXY/socket-lookup sysctls, the nft rules Cilium installs for the proxy redirect, conntrack, the cilium_host/cilium_net route pair). Two things a follow-up must not repeat: do not measure this with a single probe pod — n=1 against a two-of-eight population produced four confidently wrong readings — and do not reach for a version bump or an interception-mode change as the remedy, because both are now measured not to touch it.

What this leaves unusable: L7 DNS rules — a rules.dns block with matchPattern, and the toFQDNs egress that depends on one — cannot be relied on, on six of the eight nodes, in either interception mode. A toFQDNs tightening would fail silently on those six with no log line, which is the worst available failure shape for a policy change, so the coarse toEntities: world fallback in the nine policed namespaces is vindicated rather than merely unreplaced. It is also why argocd/app-configs/dolt/network-policies.yaml is deliberately L4-only. DNS visibility via Hubble metrics is unaffected and still functions.

Policy-based engagement is NOT used for cluster-wide visibility — earlier attempts with a CiliumClusterwideNetworkPolicy carrying enableDefaultDeny: {egress: false, ingress: false} were inert because enableDefaultDeny: false skips L7 enforcement engagement.

Mechanically, toFQDNs in a per-namespace CNP still requires that namespace’s own L7 DNS rule: Cilium populates the FQDN-to-IP cache exclusively from DNS-proxy observation, so transparent mode supplies visibility while toFQDNs needs opt-in policy. On this cluster that pairing is what fails on six of the eight nodes, per the readings above.

Transparent encryption (WireGuard)

Pod-to-pod traffic is encrypted cluster-wide with Cilium’s WireGuard transparent encryption (hl-kt1r). This keeps in-cluster credentials — notably the HyperDX OTLP ingest token pushed to cs-otel-collector:4317 over h2c — off the cleartext overlay.

Configuration (Ansible, ansible/roles/cilium):

cilium_encryption_enabled: true
cilium_encryption_type: wireguard
cilium_encryption_node_encryption: false — node-to-node is beta; pod-to-pod covers the credential-exposure concern (senders and collector are all pods).

Renders enable-wireguard: "true" into cilium-config. Each agent brings up a cilium_wg0 interface on UDP/51871 and auto-distributes peer public keys via CiliumNode CRDs. MTU is auto-reduced by Cilium for the WireGuard + VXLAN overhead.

Verify encryption is active:

# Per-agent status (run for one or all agents)
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep Encryption
# Expect: Encryption: Wireguard [cilium_wg0 (Pubkey: ..., Port: 51871, Peers: 7)]

# Detailed peer/handshake state
kubectl -n kube-system exec ds/cilium -- cilium-dbg encrypt status

Confirm the OTLP token is no longer cleartext (run on a node carrying an OTLP sender; requires node shell access):

# Encrypted tunnel on the wire (capture the node's physical path, NOT cilium_wg0 —
# cilium_wg0 carries the pre-encryption plaintext). Expect WireGuard packets:
sudo tcpdump -ni any 'udp port 51871' -c 5
# Inter-node pod→collector traffic should no longer appear as cleartext TCP/4317:
sudo tcpdump -nAi any 'tcp port 4317' -c 20 | grep -i authorization   # expect: no match

The grep authorization line is a secondary signal, not definitive proof: over h2c the bearer token rides in HPACK-compressed HTTP/2 HEADERS frames, so a literal authorization string may not appear even on a cleartext capture. The authoritative proof is that, with WireGuard active, inter-node pod→collector traffic is carried inside the encrypted UDP/51871 tunnel — you should see UDP/51871 packets and no cleartext TCP/4317 flow crossing the wire between nodes.

Rollback: set cilium_encryption_enabled: false in ansible/roles/cilium/defaults/main.yml (the same toggle used to enable it — this role keeps cilium feature flags in defaults/main.yml, there is no override layer) and re-run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium. Cilium tears down cilium_wg0 and returns to the cleartext overlay; expect a brief pod-to-pod connectivity blip per node during teardown.

Version upgrade recovery

A Cilium version downgrade is not supported, and no symmetric rollback exists. Three things move forward with cilium_version and none of them comes back on its own:

the cilium.io CRD schemas the cluster validates every CiliumNetworkPolicy against;
the vendored Gateway API bundle, which the chart requires at a matching minimum;
the safe-upgrades.gateway.networking.k8s.io ValidatingAdmissionPolicy, which ships inside that same bundle and reads its own bundle-version annotation to set its deny floor. It is live now annotating bundle-version: v1.6.1, and that floor has risen from v1.[0-3].x to v1.[0-5].x — so re-applying the previous CRD set would be refused by the cluster’s own admission chain. Restoring the old bundle is not merely awkward; it is denied.

Do not run helm rollback. The recovery is a forward re-run at the prior chart pin, with the CRDs left forward, which is what the steps below do.

Set cilium_version back to the prior patch or minor in ansible/roles/cilium/defaults/main.yml. That file is the only toggle — this role keeps its feature flags in defaults/main.yml and there is no override layer. Leave cilium_gateway_api_crd_version and the vendored CRDs alone; they stay forward.
Re-run the role: ansible-playbook -i ansible/inventory/hosts.yml ansible/k3s-playbook.yml --tags cilium-install. Do not reorder the role’s tasks. It installs the Gateway API CRDs and then the admission policy before the Helm release, and that CRDs-first / policy-second order is what admits a bundle at all — inverting it has the policy adjudicate the CRDs it was shipped with.
Watch the roll. The DaemonSet is RollingUpdate with maxUnavailable: 2, maxSurge: 0 across 8 nodes, so rollout restart proceeds in four batches of two, not all at once. Across seven rolls observed 2026-09-08/09 it took 42–44 s end to end, never dropped below 6/8 ready, and left no node Ready=False. The role’s own rollout status task blocks until observedGeneration catches up, so a hung roll fails the play rather than passing silently.

Confirm the release and the agents agree:

helm -n kube-system list --filter cilium   # chart version + revision; STATUS should be deployed
kubectl -n kube-system get ds cilium -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose | head -20

If one agent will not come back, delete that node’s agent pod and let the DaemonSet recreate it — kubectl -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName=<node> — then re-read cilium-dbg status --verbose on the replacement. Do not cordon or drain as a first move: on this cluster the CNI is what a drain needs in order to work.

Before you run it

The local Helm cache can block the re-run. kubernetes.core.helm_repository rstrips / from the configured repo_url and then compares the stripped form against the RAW url in the workstation’s Helm cache, so a cache entry holding https://helm.cilium.io/ fails the task with Repository already have a repository named cilium while helm repo list reads correct character-for-character (measured 2026-09-08). The role carries force_update: true for exactly this, which makes it authoritative over the cache. From an older checkout that lacks it, normalise the cache first: helm repo add cilium https://helm.cilium.io --force-update.

What a genuine no-op looks like. Two of the role’s tasks are unconditional by design — the repo add (force_update: true) and the rollout restart, because helm upgrade alone does not roll the DaemonSet when only ConfigMap values change. The signal to read is therefore the Helm task: a configuration that changes nothing reports the Helm task ok with only those two changed. A Helm task reporting changed on a run you believed was a no-op means the rendered values moved.

The standalone DNS proxy is not a recovery option and not a mitigation. It cannot be run from the published chart at all — see DNS L7 visibility. Do not reach for it during an incident.

Known follow-ups

~~bpf.hostLegacyRouting: true → false~~ — done 2026-05-13 at the production-mode cutover. cilium_bpf_host_legacy_routing reads false and enable-host-legacy-routing: false is live in cilium-config; the entry stood here claiming it was still true and needed a maintenance window until 2026-09-09. Kept as a note rather than deleted, because an item that vanishes reads as one somebody forgot to write down.
Gateway API route migration — Cilium’s gateway controller is installed and serves zero Gateways; the cluster’s Gateway API traffic runs through agentgateway. Migrating the remaining Traefik IngressRoutes to HTTPRoutes is a much bigger separate project, and whether Cilium’s own Gateway API support should stay enabled at all is an open simplification.
Native routing (autoDirectNodeRoutes: true) — viable since all nodes are on the same L2 segment; would eliminate VXLAN encapsulation overhead. Future single-PR optimization.

References

Cilium 1.20 docs
Cilium 1.20 Helm values reference
Hubble docs
Repo: ansible/roles/cilium/ (role source)
Repo: docs/engineering/specs/2026-05-10-calico-to-cilium-design.md (original design)
Repo: docs/operations/cilium-migration.md (one-time migration log)