Cilium WireGuard transparent encryption — implementation plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Enable Cilium WireGuard pod-to-pod transparent encryption cluster-wide via the Ansible Helm role, so the cleartext HyperDX OTLP ingest token (and all other pod-overlay traffic) is encrypted on the wire.

Architecture: Add encryption.{enabled,type,nodeEncryption} to the Cilium Helm values rendered by ansible/roles/cilium. The role’s existing machinery (helm upgrade → kubectl rollout restart daemonset/cilium → rollout status) applies the change and rolls all 8 agents deterministically. Cilium then creates the cilium_wg0 interface (UDP/51871) per node and auto-distributes peer keys via CiliumNode CRDs. No per-application or collector changes.

Tech Stack: Cilium 1.19.3, Ansible (kubernetes.core.helm), k3s, WireGuard (in-kernel, Linux 6.18 rockchip64), Velero, Hubble.

Spec: docs/engineering/specs/2026-06-13-cilium-wireguard-encryption-design.md

Scope note: Pod-to-pod encryption only (GA). nodeEncryption stays false (beta, out of scope per spec). Tasks 1–3 are repo changes shipped in one PR; Tasks 4–5 are operator-gated live-cluster operations performed after that PR merges to main.

---

Task 1: Enable WireGuard in the Cilium Helm role

Files:

• Modify: ansible/roles/cilium/defaults/main.yml (append after the Hubble/Gateway feature-scope block, near the existing cilium_gateway_api_enabled group)
• Modify: ansible/roles/cilium/templates/values.yaml.j2 (append a new top-level encryption: block after the gatewayAPI: block at the end of the file)
• Step 1: Add the encryption variables to role defaults

Edit ansible/roles/cilium/defaults/main.yml. Add this block immediately after the cilium_gateway_api_enabled: true line:

# Transparent encryption (hl-kt1r). WireGuard pod-to-pod, GA. Encrypts all
# pod-to-pod overlay traffic cluster-wide so credentials (e.g. the HyperDX OTLP
# ingest token pushed to cs-otel-collector:4317 over h2c) never cross the overlay
# in cleartext. Kernel-native on the 6.18 rockchip64 nodes (WireGuard in-kernel
# since Linux 5.6).
cilium_encryption_enabled: true
cilium_encryption_type: wireguard
# Node-to-node encryption is beta and unneeded (OTLP senders + collector are pods).
cilium_encryption_node_encryption: false

• Step 2: Add the encryption block to the Helm values template

Edit ansible/roles/cilium/templates/values.yaml.j2. Append at the end of the file (after the closing of the gatewayAPI: block):

# Transparent encryption (hl-kt1r). Renders enable-wireguard into cilium-config;
# Cilium creates the cilium_wg0 interface (UDP/51871) on every agent and
# auto-distributes peer public keys via CiliumNode CRDs. nodeEncryption is only
# effective when type is wireguard.
encryption:
  enabled: {{ cilium_encryption_enabled | lower }}
  type: {{ cilium_encryption_type }}
  nodeEncryption: {{ cilium_encryption_node_encryption | lower }}

• Step 3: Lint the YAML

Run: cd ansible && yamllint roles/cilium/defaults/main.yml Expected: no errors (exit 0). (The repo .yamllint.yaml governs rules.)

• Step 4: Syntax-check the playbook

Run: cd ansible && ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --syntax-check Expected: playbook: k3s-playbook.yml with no syntax errors (exit 0). This validates YAML/Jinja syntax (it does not fully evaluate the template against inventory — the Task 2 --check --diff dry-run is the real render proof).

• Step 5: Commit

jj commit -m "feat(cilium): enable WireGuard pod-to-pod transparent encryption [hl-kt1r]

Adds encryption.{enabled,type,nodeEncryption} to the Cilium Helm values.
Encrypts all pod-to-pod overlay traffic cluster-wide so the HyperDX OTLP
ingest token (pushed to cs-otel-collector:4317 over h2c) and every other
in-cluster pod flow no longer cross the overlay in cleartext. Pod-to-pod
scope only (GA); nodeEncryption left false (beta, out of scope).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"

---

Task 2: Validate the rendered change against the live cluster (dry run)

Files: none modified (read-only verification against the cluster).

This task proves the change renders to the intended cilium-config mutation before any live apply. It produces no commit.

• Step 1: Render-and-diff with Ansible check mode

Run:

cd ansible && ansible-playbook -i inventory/hosts.yml k3s-playbook.yml \
  --tags cilium --check --diff

Expected: the Install or upgrade Cilium … via Helm task reports a changed result, and the diff includes the new enable-wireguard: "true" key being added to the cilium-config ConfigMap. No other unexpected config keys should change.

If --check cannot fully resolve the Helm diff (the kubernetes.core.helm module’s check-mode diff can be partial), fall back to rendering the values locally: cd ansible && ansible -i inventory/hosts.yml tp_cluster_controlplane[0] -m template -a "src=roles/cilium/templates/values.yaml.j2 dest=/tmp/cilium-values.yaml" --check is not reliable for lookups; instead confirm the template output by eye — the rendered values must contain encryption:\n enabled: true\n type: wireguard\n nodeEncryption: false.

• Step 2: Confirm current live state is unencrypted (baseline)

Run:

kubectl -n kube-system get configmap cilium-config -o jsonpath='{.data.enable-wireguard}'; echo

Expected: empty output (the key does not yet exist) — confirms encryption is currently OFF and the change is additive.

---

Task 3: Document transparent encryption in the operations runbook

Files:

• Modify: docs/operations/cilium.md (insert a new ## Transparent encryption (WireGuard) section after the ## NetworkPolicy enforcement section’s ### DNS L7 visibility (cluster-wide) subsection, i.e. before ## Known follow-ups)

• Step 1: Add the encryption runbook section

Insert this section into docs/operations/cilium.md immediately before the ## Known follow-ups heading:

## Transparent encryption (WireGuard)

Pod-to-pod traffic is encrypted cluster-wide with Cilium's WireGuard transparent
encryption (`hl-kt1r`). This keeps in-cluster credentials — notably the HyperDX OTLP
ingest token pushed to `cs-otel-collector:4317` over h2c — off the cleartext overlay.

**Configuration** (Ansible, `ansible/roles/cilium`):

- `cilium_encryption_enabled: true`
- `cilium_encryption_type: wireguard`
- `cilium_encryption_node_encryption: false` — node-to-node is beta; pod-to-pod
  covers the credential-exposure concern (senders and collector are all pods).

Renders `enable-wireguard: "true"` into `cilium-config`. Each agent brings up a
`cilium_wg0` interface on UDP/51871 and auto-distributes peer public keys via
CiliumNode CRDs. MTU is auto-reduced by Cilium for the WireGuard + VXLAN overhead.

**Verify encryption is active:**

```bash
# Per-agent status (run for one or all agents)
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep Encryption
# Expect: Encryption: Wireguard [cilium_wg0 (Pubkey: ..., Port: 51871, Peers: 7)]

# Detailed peer/handshake state
kubectl -n kube-system exec ds/cilium -- cilium-dbg encrypt status

Confirm the OTLP token is no longer cleartext (run on a node carrying an OTLP sender; requires node shell access):

# WireGuard traffic appears as UDP/51871; the bearer token must NOT appear in cleartext
sudo tcpdump -ni any 'udp port 51871' -c 5          # expect WireGuard packets
sudo tcpdump -nAi any 'tcp port 4317' -c 20 | grep -i authorization   # expect: no match

Rollback: set cilium_encryption_enabled: false in ansible/roles/cilium/defaults/main.yml and re-run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium. Cilium tears down cilium_wg0 and returns to cleartext overlay.

- [ ] **Step 2: Correct two stale facts in the same runbook (drive-by doc-rot)**

While editing `docs/operations/cilium.md`, fix two factual claims that contradict the
current role defaults (both predate this change; correct them since we're in the file):

In the `## Deployment` bullet list, change:
```markdown
- **kube-proxy:** still running (`kubeProxyReplacement: "false"`); replacement is a follow-up project

to:

- **kube-proxy:** replaced (`kubeProxyReplacement: "true"`, flipped per hl-3ed)

In the ### DNS L7 visibility (cluster-wide) section, the text states dnsProxy.enableTransparentMode: true, but the role default cilium_dns_proxy_transparent_mode is false (disabled per hl-bdr). Update that subsection to reflect that transparent mode is currently disabled (the hl-bdr silent-drop workaround), so the runbook matches defaults/main.yml.

If either line has already been corrected by an intervening change when you reach this step, skip it — the goal is consistency with defaults/main.yml, not a mechanical edit.

• Step 3: Lint the markdown

Run: rumdl check docs/operations/cilium.md Expected: no errors (exit 0). Fix any reported issues (the repo uses .rumdl.toml).

• Step 4: Commit

jj commit -m "docs(cilium): document WireGuard transparent encryption runbook [hl-kt1r]

Verify (cilium-dbg status / encrypt status), token-cleartext check via tcpdump,
and rollback procedure for the pod-to-pod WireGuard encryption enabled in the
Cilium Helm role.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"

• Step 5: Open the PR (repo changes complete)

Push the bookmark and open a PR containing Tasks 1 + 3 (the spec/plan/ADRs land here too if not already merged). Branch protection requires a PR — do not push to main.

jj git push -b worktree-cilium-wireguard
gh pr create --head worktree-cilium-wireguard \
  --title "feat(cilium): WireGuard pod-to-pod transparent encryption [hl-kt1r]" \
  --body "Enables Cilium WireGuard pod-to-pod transparent encryption to remove the cleartext HyperDX OTLP token from the pod overlay. Spec + plan + runbook included. Deploy + live verification (Tasks 4-5) happen post-merge per the plan."

Expected: PR created. STOP — Tasks 4–5 run only after this PR is reviewed and merged to main.

---

Task 4: Pre-flight, backup, and deploy to the live cluster

Files: none (operational; runs after the Task 3 PR merges).

Operator gate: this is a live production CNI change. Confirm the PR is merged and you are applying off-hours with monitoring available before proceeding.

• Step 1: Pre-flight — confirm UDP/51871 is not firewalled between nodes

WireGuard peers handshake on UDP/51871; if a host firewall blocks it, peers never handshake and traffic passes unencrypted (silent failure). The Armbian/k3s nodes run no host firewall by default. Confirm:

Run (per node, or spot-check one control-plane + one worker):

ssh tpi-alpha-1 'sudo nft list ruleset 2>/dev/null | grep -i 51871; sudo iptables -S 2>/dev/null | grep -i 51871; echo "exit:$?"'

Expected: no DROP/REJECT rule for 51871; an empty/no-match result means no host firewall is blocking it. If a firewall exists, open 51871/udp node-to-node first.

• Step 2: Velero backup (per the Ansible pre-deploy checklist)

Run: velero backup create pre-wireguard-$(date +%Y%m%d-%H%M%S) --wait Expected: backup completes with Phase: Completed.

• Step 3: Sync the merged main into the deploy checkout

Run: jj git fetch && jj rebase -o main --skip-emptied (or deploy from a fresh checkout of main). Confirm ansible/roles/cilium/defaults/main.yml contains cilium_encryption_enabled: true.

• Step 4: Apply

Run:

cd ansible && ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium

Expected: the Helm upgrade task is changed; the role’s kubectl rollout restart daemonset/cilium + rollout status --timeout=10m tasks complete successfully (all 8 agents roll to the new generation). A brief pod-to-pod disruption window during the roll is expected. Watch Hubble / app health in parallel.

---

Task 5: Verify encryption and confirm the OTLP token is no longer cleartext

Files: none (operational acceptance test — this is the bead’s acceptance criterion).

• Step 1: Confirm enable-wireguard is now in cilium-config

Run:

kubectl -n kube-system get configmap cilium-config -o jsonpath='{.data.enable-wireguard}'; echo

Expected: true.

• Step 2: Confirm every agent reports WireGuard active with all peers

Run:

for p in $(kubectl -n kube-system get pods -l k8s-app=cilium -o name); do
  echo "== $p =="
  kubectl -n kube-system exec "$p" -- cilium-dbg status | grep Encryption
done

Expected: each agent prints Encryption: Wireguard [cilium_wg0 (Pubkey: ..., Port: 51871, Peers: 7)] (Peers: 7 = the other 7 nodes). Any agent showing Disabled or fewer peers means an incomplete roll or a blocked 51871 — investigate before declaring success.

• Step 3: Verify the OTLP token is encrypted on the wire

On a node running an OTLP sender (e.g. a Traefik or engram pod), with node shell access:

sudo tcpdump -ni any 'udp port 51871' -c 5
sudo tcpdump -nAi any 'tcp port 4317' -c 40 | grep -i 'authorization\|bearer'

Expected: the first command shows WireGuard packets; the second shows no cleartext authorization/bearer token to cs-otel-collector:4317. This is the acceptance criterion — the cleartext-token exposure is closed.

• Step 4: Verify MTU and no new drops

Run:

# Actual computed pod MTU (read the value; do not assert a predicted number)
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose | grep -i mtu
# No new MTU/PMTU or policy drops
kubectl -n kube-system exec ds/cilium -- hubble observe --type drop --last 200

Expected: pod MTU reflects the WireGuard+VXLAN reduction (a base-1500 link lands in the ~1390 range — illustrative, not an assertion); no spike of MTU/PMTU/encryption-related drops. DNS/L7 flows remain healthy (guards against the hl-bdr-class L7 interaction).

• Step 5: Confirm telemetry still flows into HyperDX

Verify in HyperDX that Traefik, engram, fovea, and the otel-scraper collectors are still ingesting (recent data present). Encryption must not have broken OTLP delivery.

• Step 6: Record the outcome on the bead

Run:

bd note hl-kt1r "Deployed + verified: enable-wireguard=true in cilium-config; all 8 agents report Wireguard cilium_wg0 Peers:7; tcpdump confirms OTLP token no longer cleartext (UDP/51871); MTU auto-reduced, no new drops; HyperDX ingest healthy."

---

Verification summary

Gate	Command	Pass condition
YAML lint	yamllint roles/cilium/defaults/main.yml	exit 0
Playbook syntax	ansible-playbook … --syntax-check	exit 0
Dry-run diff	ansible-playbook … --tags cilium --check --diff	adds enable-wireguard: "true"
Markdown lint	rumdl check docs/operations/cilium.md	exit 0
Live config	kubectl get cm cilium-config -o jsonpath='{.data.enable-wireguard}'	true
Agent status	`cilium-dbg status	grep Encryption`
Token check	`tcpdump ‘tcp port 4317’	grep -i authorization`
No regression	hubble observe --type drop + HyperDX ingest	no new drops; telemetry flowing