Cilium Operations Runbook

Evergreen operational guide for the cluster’s Cilium 1.19.3 CNI deployment. For the one-time Calico→Cilium migration log, see cilium-migration.md.

Deployment

Installed by: ansible/roles/cilium (Phase 5 of ansible/k3s-playbook.yml)
Chart: cilium/cilium from https://helm.cilium.io/
Version: 1.19.3 (pinned in roles/cilium/defaults/main.yml)
Namespace: kube-system
Pod CIDR: 10.245.0.0/16 (IPv4) + fd00:10:245::/48 (IPv6)
Service CIDR: 10.43.0.0/16 (k3s default; unchanged from Calico era)
Routing: tunnel mode, VXLAN protocol, port 8473
kube-proxy: replaced (kubeProxyReplacement: "true", flipped per hl-3ed)
Gateway API: v1.4.1 Standard CRDs installed; Cilium gateway controller enabled; GatewayClass is cilium. No HTTPRoutes/Gateways exist yet.

k3s —cluster-cidr note

k3s-common/defaults/main.yml still references 10.42.0.0/16 as the k3s cluster CIDR. This is intentionally retained post-Calico-removal:

Cilium owns IPAM via ipam.mode: cluster-pool and writes pod IPs from 10.245.0.0/16 to CiliumNode.spec.ipam.podCIDRs.
k3s passes --cluster-cidr=10.42.0.0/16 to kube-controller-manager-equivalent, which populates Node.spec.podCIDR from that range.
Cilium IGNORES Node.spec.podCIDR when in cluster-pool mode; the two coexist harmlessly.
Changing the k3s value would require a server restart on every control-plane node — out of scope for any non-emergency operation.

Common commands

cilium-cli (workstation)

# Cluster health
cilium status --wait
cilium connectivity test     # full e2e suite — takes ~5 minutes
cilium connectivity test --hubble=false  # quick smoke

# Hubble flows
cilium hubble observe --last 100
cilium hubble observe --since 5m --verdict DROPPED  # see policy drops
cilium hubble observe --pod NAMESPACE/POD-NAME
cilium hubble ui            # opens browser to internal UI via port-forward

kubectl

# Per-node CNI state
kubectl get ciliumnodes
kubectl get ciliumendpoints -A | head -20

# Policy resources
kubectl get ciliumnetworkpolicies -A
kubectl get ciliumclusterwidenetworkpolicies

# Agent diagnostics
kubectl exec -n kube-system ds/cilium -- cilium-dbg status
kubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint list

Hubble UI access

Hubble UI is internal-only (no IngressRoute by design).

cilium hubble ui  # auto-port-forwards and opens browser
# Or manually:
kubectl port-forward -n kube-system svc/hubble-ui 12000:80

Pod CIDR allocation per node

Each node gets a /24 from the cluster pool. View allocation:

kubectl get ciliumnodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}'

NetworkPolicy enforcement

Cluster-wide policyEnforcementMode: default — pods without matching CiliumNetworkPolicy are allowed by default; pods with matching policies are restricted to what the policies permit.

Crown-jewel namespace coverage is tracked under the post-migration hardening initiative (see docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md).

Currently enforced (namespaces with CiliumNetworkPolicy opt-in):

dolt — 5 CNPs covering SQL (3306), Remote API (50051), Prometheus (9418), DNS L7 to kube-dns, kube-apiserver egress. See argocd/app-configs/dolt/network-policies.yaml.

Each direction-specific CNP engages default-deny for its direction automatically — no separate baseline CNP needed. To verify enforcement empirically: run a probe pod in an unauthorized namespace and try to reach a workload in a policy-covered namespace; expect Policy denied DROPPED in hubble observe.

Crown-jewel coverage for cert-manager / external-secrets / vault was reverted (PRs #1044+#1045+#1048) after the cluster’s ESO ClusterSecretStore failed to recover post-merge. A more careful redesign is tracked for future work.

DNS L7 visibility (cluster-wide)

One chart-level setting surfaces cluster-wide DNS visibility (see ansible/roles/cilium/templates/values.yaml.j2):

hubble.metrics.enabled includes dns — surfaces DNS flow events visible via hubble observe --type l7 and in the Hubble UI (hubble.fzymgc.house) service map.

dnsProxy.enableTransparentMode is currently false (disabled per hl-bdr). The transparent tproxy path caused silent DNS drops for certain pods; the workaround is to leave transparent mode off. DNS visibility via Hubble metrics still functions without it.

Policy-based engagement is NOT used for cluster-wide visibility — earlier attempts with a CiliumClusterwideNetworkPolicy carrying enableDefaultDeny: {egress: false, ingress: false} were inert because enableDefaultDeny: false skips L7 enforcement engagement.

toFQDNs rules in subsequent per-namespace CNPs (e.g. cert-manager’s ACME/Cloudflare egress in PR4) still need per-namespace L7 DNS rules to learn IPs from DNS responses — transparent mode provides visibility but toFQDNs requires opt-in policy.

Transparent encryption (WireGuard)

Pod-to-pod traffic is encrypted cluster-wide with Cilium’s WireGuard transparent encryption (hl-kt1r). This keeps in-cluster credentials — notably the HyperDX OTLP ingest token pushed to cs-otel-collector:4317 over h2c — off the cleartext overlay.

Configuration (Ansible, ansible/roles/cilium):

cilium_encryption_enabled: true
cilium_encryption_type: wireguard
cilium_encryption_node_encryption: false — node-to-node is beta; pod-to-pod covers the credential-exposure concern (senders and collector are all pods).

Renders enable-wireguard: "true" into cilium-config. Each agent brings up a cilium_wg0 interface on UDP/51871 and auto-distributes peer public keys via CiliumNode CRDs. MTU is auto-reduced by Cilium for the WireGuard + VXLAN overhead.

Verify encryption is active:

# Per-agent status (run for one or all agents)
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep Encryption
# Expect: Encryption: Wireguard [cilium_wg0 (Pubkey: ..., Port: 51871, Peers: 7)]

# Detailed peer/handshake state
kubectl -n kube-system exec ds/cilium -- cilium-dbg encrypt status

Confirm the OTLP token is no longer cleartext (run on a node carrying an OTLP sender; requires node shell access):

# Encrypted tunnel on the wire (capture the node's physical path, NOT cilium_wg0 —
# cilium_wg0 carries the pre-encryption plaintext). Expect WireGuard packets:
sudo tcpdump -ni any 'udp port 51871' -c 5
# Inter-node pod→collector traffic should no longer appear as cleartext TCP/4317:
sudo tcpdump -nAi any 'tcp port 4317' -c 20 | grep -i authorization   # expect: no match

The grep authorization line is a secondary signal, not definitive proof: over h2c the bearer token rides in HPACK-compressed HTTP/2 HEADERS frames, so a literal authorization string may not appear even on a cleartext capture. The authoritative proof is that, with WireGuard active, inter-node pod→collector traffic is carried inside the encrypted UDP/51871 tunnel — you should see UDP/51871 packets and no cleartext TCP/4317 flow crossing the wire between nodes.

Rollback: set cilium_encryption_enabled: false in ansible/roles/cilium/defaults/main.yml (the same toggle used to enable it — this role keeps cilium feature flags in defaults/main.yml, there is no override layer) and re-run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium. Cilium tears down cilium_wg0 and returns to the cleartext overlay; expect a brief pod-to-pod connectivity blip per node during teardown.

Known follow-ups

bpf.hostLegacyRouting: true → false — currently true because flipping requires a scheduled low-traffic window (upstream warns of brief packet loss). Plan a future maintenance window.
Gateway API route migration — Cilium gateway controller is installed but unused; future project to migrate Traefik IngressRoutes to HTTPRoutes (much bigger scope).
Native routing (autoDirectNodeRoutes: true) — viable since all nodes are on the same L2 segment; would eliminate VXLAN encapsulation overhead. Future single-PR optimization.

References

Cilium 1.19 docs
Cilium 1.19 Helm values reference
Hubble docs
Repo: ansible/roles/cilium/ (role source)
Repo: docs/engineering/specs/2026-05-10-calico-to-cilium-design.md (original design)
Repo: docs/operations/cilium-migration.md (one-time migration log)