Use Cilium WireGuard pod-to-pod encryption cluster-wide
Date: 2026-06-14 Status: Accepted Decision: hl-qovi Deciders: Sean
Context
Section titled “Context”Cilium transparent encryption was off, leaving all pod-to-pod overlay traffic in cleartext. The concrete trigger (surfaced during hl-pwwf Traefik OTLP work): in-cluster OTLP senders (Traefik, engram, fovea, otel-scraper) push a long-lived HyperDX ingest token to cs-otel-collector:4317 over plaintext h2c (the collector serves no TLS). The cluster runs Cilium 1.19.3 with kube-proxy replacement and the L7 proxy, tunnel/VXLAN routing, dual-stack, on 8 arm64 Armbian nodes (kernel 6.18, WireGuard in-kernel since 5.6).
Decision
Section titled “Decision”Enable Cilium WireGuard transparent encryption at pod-to-pod scope (GA) via two Helm-values additions in the Ansible cilium role (encryption.enabled=true, encryption.type=wireguard), encrypting all pod overlay traffic cluster-wide with zero per-application changes. Node-to-node encryption (encryption.nodeEncryption) is explicitly deferred as beta and out of scope.
Rationale
Section titled “Rationale”- The OTLP ingest token and any other in-cluster credential are fully covered because both senders and the collector are pods — pod-to-pod scope is sufficient (YAGNI for node-to-node).
- WireGuard is GA in Cilium and in-kernel on the 6.18 rockchip64 nodes, requiring no DKMS or out-of-tree build.
- A single config surface (two Helm values) vs. per-link TLS avoids per-sender churn and covers every future in-cluster credential by default.
- The homelab context makes the modest KPR+WireGuard throughput penalty acceptable, and it is documented as a trade-off.
Alternatives Considered
Section titled “Alternatives Considered”- Cilium WireGuard pod-to-pod (GA) — CHOSEN. Single Helm-values change; encrypts all pod overlay traffic cluster-wide; no per-app/collector changes; kernel-native; GA; compatible with KPR + L7 proxy. Cost: KPR+WireGuard slightly slower; ~60B overhead atop VXLAN ~50B; brief per-node connectivity gap during the rolling restart.
- Per-link OTLP TLS (cert-manager server cert + CA distribution) — REJECTED. Encrypts only the at-risk link with no CNI change, but requires per-sender config churn across Traefik/engram/fovea/otel-scraper, CA distribution to every sender, fixes only the one OTLP link, and carries higher long-term maintenance.
- Accept cleartext — REJECTED. No change, but a real long-lived bearer credential stays exposed on the wire.
Consequences
Section titled “Consequences”Positive:
- All pod-to-pod overlay traffic is encrypted cluster-wide, covering current and future in-cluster credentials without application changes.
- Cilium auto-reduces pod MTU for WireGuard + VXLAN overhead; existing PLPMTUD blackhole mode handles path-MTU correctly.
- Rollback is a single Ansible re-run reverting the encryption defaults.
Negative:
- KPR + WireGuard is slightly slower than KPR alone (Cilium upstream benchmark); accepted for homelab.
- Rolling restart of the cilium-agent DaemonSet causes a per-node transient connectivity gap (~minutes) during upgrade.
Neutral:
- Node-to-node encryption remains off (beta); must be revisited explicitly if host-network credential exposure is later identified.
- IPsec remains available in Cilium but is not the chosen mechanism; that path is closed for this cluster unless a future ADR supersedes this one.