Cilium WireGuard Transparent Encryption

Design: Cilium WireGuard Pod-to-Pod Transparent Encryption

Bead: hl-kt1r Date: 2026-06-13 Status: Design (pre-implementation)

Problem

Cilium transparent encryption is off — cilium-config has no enable-wireguard/enable-ipsec key (verified against the live cluster on 2026-06-13). As a result, all pod-to-pod traffic crosses the overlay in cleartext.

The concrete trigger, surfaced during the hl-pwwf Traefik OTLP work: in-cluster OTLP senders — Traefik, engram, fovea, and the otel-scraper collectors — push the HyperDX ingest token to cs-otel-collector:4317 over plaintext h2c. The collector serves no TLS (otel-gateway itself uses scheme: h2c), so a long-lived bearer credential is exposed on the wire to anything able to observe pod traffic.

Desired outcome: the OTLP ingest token — and, by the same mechanism, every other in-cluster credential and data flow — must not traverse the pod overlay in cleartext.

The token exposure is the authoritative problem. “Enable WireGuard” was a candidate fix recorded in the bead; this spec validates it against the live cluster and current Cilium docs before adopting it.

Decision

Enable Cilium WireGuard transparent encryption, pod-to-pod scope (GA), via the existing ansible/roles/cilium Helm role. This encrypts all pod-to-pod overlay traffic cluster-wide, fixing the cleartext-token concern (senders and collector are all pods) plus every other in-cluster pod flow, with zero per-application or per-collector changes.

Node-to-node encryption (encryption.nodeEncryption) is out of scope: it is beta, carries extra caveats around health-checks and kube-proxy replacement, and is not needed — the OTLP senders and collector are all pods, so pod-to-pod scope fully covers the problem (YAGNI).

Approaches considered

	Approach	Verdict
A	Cilium WireGuard pod-to-pod — one Helm-values change in the Ansible role; encrypts all pod overlay traffic, no per-app changes	Chosen. Kernel-native, GA, single config surface, solves cluster-wide
B	Per-link OTLP TLS — collector server cert (cert-manager) + CA distribution to every sender, OpAMP-managed	Rejected: per-sender config churn (Traefik, engram, fovea, otel-scraper), CA distribution, fixes only the one link, higher long-term maintenance
C	Accept cleartext	Rejected: a real, long-lived credential exposure

Grounding (Rule 7)

• Codebase / live state. Cilium is deployed by ansible/roles/cilium from templates/values.yaml.j2 (chart v1.19.3); no encryption keys are present in the template or in the live cilium-config. Confirmed live: routing-mode: tunnel, tunnel-protocol: vxlan, tunnel-port: 8473, kube-proxy-replacement: true, enable-l7-proxy: true, dual-stack IPv4+IPv6, packetization-layer-pmtud-mode: blackhole.
• Cluster. 8 nodes (3 control-plane tpi-alpha-1..3 + 5 workers), all arm64 Armbian noble on kernel 6.18.10-current-rockchip64. WireGuard has been in-kernel since 5.6, so the module is guaranteed present — no DKMS/out-of-tree build.
• Cilium docs (context7 /websites/cilium_io_en_stable). Enable via Helm encryption.enabled=true + encryption.type=wireguard (pod-to-pod, GA); encryption.nodeEncryption=true adds node-to-node (beta) and is only effective with type: wireguard. Verify with cilium-dbg status | grep Encryption (interface cilium_wg0, port 51871, Peers: N). WireGuard is compatible with kube-proxy replacement and the L7 proxy; the eBPF KPR + WireGuard path is currently slightly slower than KPR alone (being improved upstream). MTU is auto-reduced for the encryption overhead.

Implementation

Role defaults — ansible/roles/cilium/defaults/main.yml

# Transparent encryption (hl-kt1r). WireGuard pod-to-pod, GA. Encrypts all
# pod-to-pod overlay traffic cluster-wide so credentials (e.g. the HyperDX OTLP
# ingest token pushed to cs-otel-collector:4317 over h2c) never cross the overlay
# in cleartext. Kernel-native on the 6.18 rockchip64 nodes (WireGuard in-kernel
# since 5.6).
cilium_encryption_enabled: true
cilium_encryption_type: wireguard
# Node-to-node encryption is beta and unneeded (OTLP senders + collector are pods).
cilium_encryption_node_encryption: false

Helm values — ansible/roles/cilium/templates/values.yaml.j2

# Transparent encryption (hl-kt1r). Renders enable-wireguard into cilium-config;
# Cilium creates the cilium_wg0 interface (UDP/51871) on every agent and
# auto-distributes peer public keys via CiliumNode CRDs. nodeEncryption is only
# effective when type is wireguard.
encryption:
  enabled: {{ cilium_encryption_enabled | lower }}
  type: {{ cilium_encryption_type }}
  nodeEncryption: {{ cilium_encryption_node_encryption | lower }}

No cilium-node-config.yaml.j2 change is required — encryption is a global agent setting, not a per-node migration toggle.

MTU

WireGuard adds ~60 B of overhead; the cluster already runs VXLAN tunnelling (~50 B). Cilium auto-detects and reduces the route/pod MTU to account for both — this design does not set a manual MTU value. The existing packetization-layer-pmtud-mode: blackhole is the correct PLPMTUD posture for this stacking and needs no change.

This is the bead’s explicit “verify perf impact + MTU” concern. The implementation plan must verify post-deploy that the pod-veth MTU dropped appropriately and that Hubble shows no new drops or PMTU blackholes. The exact post-encryption pod MTU is whatever Cilium computes — the verification step reads the actual value rather than asserting a predicted target (a base-1500 link drops by roughly the VXLAN ~50 B + WireGuard ~60 B overhead, landing in the ~1390 range, but this is illustrative, not an acceptance number). Getting MTU wrong is the primary silent-failure mode on this cluster (cf. the hl-bdr L7 DNS-proxy silent-drop history).

Rollout (live CNI, 8 nodes)

0. Pre-flight: UDP/51871. WireGuard peers handshake over UDP port 51871; if a host firewall blocks it, peers never complete the handshake — traffic still passes but is never encrypted (a silent failure that the acceptance test would catch only via the tcpdump check). The Armbian/k3s nodes run no host firewall by default; confirm this (or open 51871/udp node-to-node) before applying.
1. Backup. velero backup create pre-wireguard-$(date +%Y%m%d-%H%M%S) --wait (per the Ansible pre-deploy checklist).
2. Apply off-hours. Run the k3s playbook with --tags cilium (Helm upgrade → cilium-agent DaemonSet rolling restart). Syntax-check / --check --diff first.
3. Transient disruption window. During the roll, an already-encrypting node and a not-yet-restarted node briefly cannot exchange pod traffic until both carry keys. Expected duration is on the order of minutes (8 agents). Monitor Hubble and application health throughout.
4. Rollback. Revert the three cilium_encryption_* defaults (to absent/false) and re-run --tags cilium; Cilium tears down cilium_wg0 and returns to cleartext overlay.

Verification

• cilium-dbg status | grep Encryption on each agent → Wireguard [cilium_wg0 (Pubkey: …, Port: 51871, Peers: 7)] (7 peers = the other 7 nodes).
• kubectl -n kube-system exec ds/cilium -- cilium-dbg encrypt status (note the cilium-dbg binary, not cilium).
• Token check (the core acceptance criterion). Node-level tcpdump on the OTLP path shows traffic as UDP/51871 WireGuard, with no cleartext h2c bearer token visible to cs-otel-collector:4317.
• No regression. HyperDX continues ingesting telemetry from Traefik, engram, fovea, and the otel-scraper collectors; Hubble DNS/L7 flows remain healthy (guards against the hl-bdr-class L7 interaction); no MTU/PMTU drops.

Risks

Risk	Mitigation
L7-proxy interaction (this cluster’s hl-bdr DNS-proxy silent-drop history)	Verify DNS/L7 flows in Hubble immediately post-deploy; rollback ready
Performance cost (KPR + WireGuard slightly slower per Cilium benchmark)	Acceptable for a homelab; document the trade-off
MTU / PMTU blackhole	Rely on Cilium auto-MTU + existing PLPMTUD blackhole mode; explicit MTU verification step in the plan
Rollout connectivity gap	Off-hours apply, Velero backup, active Hubble/app monitoring, one-step rollback

Documentation

Update docs/operations/cilium.md with a transparent-encryption section covering the configuration, the verify commands, and the rollback runbook.

Out of scope

• Node-to-node encryption (encryption.nodeEncryption) — beta; revisit only if host-network credential exposure is later identified.
• IPsec as an alternative transport — WireGuard is the chosen, kernel-native mechanism.
• Per-link OTLP TLS (Approach B) — superseded by transparent encryption.