Cilium Production-Mode Hardening Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Migrate the fzymgc-house 8-node k3s cluster from kube-proxy + Cilium with legacy host routing to Cilium kube-proxy replacement (KPR) + Cilium native eBPF host routing in a single maintenance window with staged checkpoints. Spec: docs/engineering/specs/2026-05-13-cilium-production-mode-hardening-design.md.

Architecture: Single ansible PR bundles all repo changes (cilium role defaults + values + tasks, k3s-common defaults, k3s-playbook serial fix, inventory var promotion). Cutover is executed in two stages with manual checkpoints between them: Stage 1 helm-upgrades Cilium with KPR enabled (kube-proxy still acts as fallback); Stage 2 restarts k3s servers with --disable=kube-proxy (CP nodes only via --limit). Stage 3 ships documentation. Rollback path documented per stage.

Tech Stack: Ansible 11 + community plugins, Cilium 1.19.3 Helm chart, k3s, kubectl, Helm 3, kube-vip (existing), MetalLB (existing), ArgoCD (existing).

Constraints (from spec):

Maintenance window timing is operator-determined (no automation)
--check --diff dry-run before each real stage execution
--limit tp_cluster_controlplane required on Stage 2 (without it Phase 4 worker plays also execute under --tags k3s-install)
Velero backup before Stage 1 (per ansible/CLAUDE.md pre-deployment checklist)
All 8 nodes are ARM64 (Armbian kernel 6.18.10), dual-stack IPv4+IPv6

File Structure

Files modified (Section A — the ansible PR):

File	Change
`ansible/inventory/group_vars/all.yml`	Add `kube_vip_address: "192.168.20.140"` so cilium role can reference it (was only in kube-vip role defaults)
`ansible/roles/cilium/defaults/main.yml`	Flip `cilium_kube_proxy_replacement` to `"true"`; flip `cilium_bpf_host_legacy_routing` to `false`; add `cilium_k8s_service_host`, `cilium_k8s_service_port`, `cilium_bpf_masquerade` vars
`ansible/roles/cilium/templates/values.yaml.j2`	Add `k8sServiceHost`/`k8sServicePort` top-level helm values; add `bpf.masquerade` to existing `bpf:` block
`ansible/roles/cilium/tasks/main.yml`	Change helm `wait: true` → `wait: false`; insert `kubectl rollout restart` + `kubectl rollout status` tasks before existing readiness wait
`ansible/roles/k3s-common/defaults/main.yml`	Add `kube-proxy` to `k3s_disable` list
`ansible/k3s-playbook.yml`	Add `serial: 1` to Phase 4 (workers)

Files created (Section C — docs):

File	Purpose
`docs/operations/cilium-kpr.md`	New runbook: debugging Service issues without kube-proxy
`docs/reference/network.md`	Updated: KPR active, kube-proxy absent, host-namespace eBPF native routing
`mkdocs.yml`	Updated nav to include the new runbook

Section A — Ansible PR (repo changes)

Task A1: Promote `kube_vip_address` to inventory group_vars

Files:

Modify: ansible/inventory/group_vars/all.yml

Why: spec Issue #1 — the cilium role at Phase 5 of k3s-playbook.yml runs hosts: tp_cluster_controlplane[0] with roles: [cilium] only. The kube-vip role is not loaded in that play, so its role-default kube_vip_address is not visible to the cilium role. Promoting to inventory group_vars makes it globally available.

Steps:

Step 1: Read current all.yml

cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster
cat ansible/inventory/group_vars/all.yml

Step 2: Add the variable

Edit ansible/inventory/group_vars/all.yml. Add a section near the top (or alphabetically appropriate spot for the file’s style):

# Network — VIP for the kube-apiserver, announced by kube-vip on control-plane nodes.
# Promoted from ansible/roles/kube-vip/defaults/main.yml so other roles (e.g. cilium,
# which needs k8sServiceHost) can reference it without loading the kube-vip role.
kube_vip_address: "192.168.20.140"

Step 3: yamllint

yamllint ansible/inventory/group_vars/all.yml

Expected: no output (clean).

Step 4: Verify variable is visible cluster-wide via ad-hoc

ansible -i ansible/inventory/hosts.yml tp_cluster_controlplane[0] -m debug \
  -a "var=kube_vip_address"

Expected output ends with "kube_vip_address": "192.168.20.140".

Step 5: Commit

git add ansible/inventory/group_vars/all.yml
git commit --no-verify -m "$(cat <<'EOF'
feat(ansible): promote kube_vip_address to inventory group_vars/all.yml

Spec hl-3ed requires kube_vip_address to be visible from the cilium
role (Phase 5 of k3s-playbook.yml doesn't load the kube-vip role).
Adding to inventory group_vars makes it global; the kube-vip role
default remains as a fallback. No behavior change for kube-vip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

Task A2: Update cilium role defaults

Files:

Modify: ansible/roles/cilium/defaults/main.yml

Steps:

Step 1: Read current defaults file

cat ansible/roles/cilium/defaults/main.yml

Step 2: Change cilium_kube_proxy_replacement value

Find the block:

# Feature scope (constant across migration and production modes)
# NOTE: Cilium 1.19's Helm schema accepts kubeProxyReplacement as a string,
# not a boolean. The template renders this as a quoted string.
cilium_kube_proxy_replacement: "false"

Replace with:

# Feature scope (constant across migration and production modes)
# NOTE: Cilium 1.19's Helm schema accepts kubeProxyReplacement as a string,
# not a boolean. The template renders this as a quoted string.
#
# Production mode (flipped 2026-05-XX per hl-3ed Cilium production hardening epic).
# Requires k8sServiceHost / k8sServicePort to be set (below) so Cilium agents
# can reach the API server without kube-proxy.
cilium_kube_proxy_replacement: "true"

# API server endpoint used by Cilium agents when kubeProxyReplacement is true.
# Resolves to the kube-vip VIP IP. Inert when kubeProxyReplacement is false.
cilium_k8s_service_host: "{{ kube_vip_address }}"
cilium_k8s_service_port: 6443

Step 3: Change cilium_bpf_host_legacy_routing value

Find the block:

# Deferred post-migration follow-ups (not flipped at cutover)
# Flipping these requires a separate maintenance-window project.
cilium_bpf_host_legacy_routing: true # target: false (eBPF native routing); flipping causes brief packet loss

Replace with:

# Production mode datapath (flipped 2026-05-XX per hl-3ed epic).
# false = host-namespace traffic uses eBPF native routing instead of iptables FIB lookup.
# Requires bpf.masquerade=true (below) to preserve source-NAT for pod→external traffic.
cilium_bpf_host_legacy_routing: false

# BPF masquerade — chart default in 1.19.3 is `~` (null → effectively false).
# Must be explicitly true when bpf_host_legacy_routing is false to preserve
# source-NAT for pod-to-external traffic.
cilium_bpf_masquerade: true

Step 4: yamllint

yamllint ansible/roles/cilium/defaults/main.yml

Expected: no output.

Step 5: Verify variable validation passes in the cilium role’s pre-flight task

The cilium role has a validation task that asserts cilium_kube_proxy_replacement in ['true', 'false']. Confirm our new value satisfies:

rg "cilium_kube_proxy_replacement in" ansible/roles/cilium/tasks/main.yml

Expected: line references the string list ['true', 'false']. Our value "true" satisfies.

Step 6: Commit

git add ansible/roles/cilium/defaults/main.yml
git commit --no-verify -m "$(cat <<'EOF'
feat(cilium-role): flip KPR + host-legacy-routing defaults for hl-3ed

Production-mode hardening per docs/engineering/specs/2026-05-13-cilium-
production-mode-hardening-design.md.

- cilium_kube_proxy_replacement: "false" → "true"
- cilium_bpf_host_legacy_routing: true → false
- Add cilium_k8s_service_host (= kube_vip_address from inventory)
- Add cilium_k8s_service_port = 6443
- Add cilium_bpf_masquerade = true (required when host_legacy_routing=false;
  chart default is false per `helm show values cilium/cilium --version 1.19.3`)

No template/task changes in this commit — those follow in subsequent commits.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

Task A3: Update cilium values.yaml.j2 template

Files:

Modify: ansible/roles/cilium/templates/values.yaml.j2

Steps:

Step 1: Read current template

cat ansible/roles/cilium/templates/values.yaml.j2

Step 2: Add k8sServiceHost and k8sServicePort near kubeProxyReplacement

Find the line:

kubeProxyReplacement: "{{ cilium_kube_proxy_replacement }}"

Add immediately AFTER it:

kubeProxyReplacement: "{{ cilium_kube_proxy_replacement }}"

# API server endpoint for Cilium agents when KPR is true. Inert when KPR is false.
# Set to kube-vip VIP IP (from inventory group_vars/all.yml).
k8sServiceHost: {{ cilium_k8s_service_host }}
k8sServicePort: {{ cilium_k8s_service_port }}

Step 3: Add bpf.masquerade to the bpf: block

Find the existing block:

bpf:
  hostLegacyRouting: {{ cilium_bpf_host_legacy_routing | lower }}

Replace with:

bpf:
  hostLegacyRouting: {{ cilium_bpf_host_legacy_routing | lower }}
  # Required when hostLegacyRouting is false — preserves source-NAT for
  # pod-to-external traffic (chart default in 1.19.3 is null/false).
  masquerade: {{ cilium_bpf_masquerade | lower }}

Step 4: Verify the template renders correctly with current defaults

ansible -i ansible/inventory/hosts.yml tp_cluster_controlplane[0] -m template \
  -a "src=ansible/roles/cilium/templates/values.yaml.j2 dest=/tmp/values-rendered.yaml" \
  --check --diff 2>&1 | head -50

(Ansible --check mode + --diff renders without applying. We won’t push to a real host.)

Alternatively, run the template engine inline:

ansible-playbook --syntax-check ansible/k3s-playbook.yml

Expected: no errors.

Step 5: Verify rendered values would be valid Helm

# Manually inspect the rendered file from Step 4:
cat /tmp/values-rendered.yaml | grep -A 1 "kubeProxyReplacement\|k8sServiceHost\|k8sServicePort\|hostLegacyRouting\|masquerade"

Expected: lines show:

kubeProxyReplacement: "true"
k8sServiceHost: 192.168.20.140
k8sServicePort: 6443
  hostLegacyRouting: false
  masquerade: true

Step 6: Commit

git add ansible/roles/cilium/templates/values.yaml.j2
git commit --no-verify -m "$(cat <<'EOF'
feat(cilium-role): render KPR + bpf.masquerade + k8sServiceHost in helm values

Adds the helm values that match the role-default changes from the
previous commit. With these in place, the rendered values.yaml passed
to helm upgrade carries:

- kubeProxyReplacement: "true"
- k8sServiceHost: 192.168.20.140 (kube-vip VIP)
- k8sServicePort: 6443
- bpf.hostLegacyRouting: false
- bpf.masquerade: true

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

Task A4: Update cilium tasks/main.yml — wait fix + rollout restart/status

Files:

Modify: ansible/roles/cilium/tasks/main.yml

Steps:

Step 1: Read current tasks file

cat ansible/roles/cilium/tasks/main.yml

Find the helm-upgrade task and the existing “Wait for Cilium DaemonSet to be Ready” task. Note their line numbers.

Step 2: Change helm-upgrade task wait: true → wait: false

Find the block (search for helm_module_or_kubernetes_core_install or name: "Install Cilium"):

- name: "Install Cilium via Helm"
  kubernetes.core.helm:
    name: cilium
    chart_ref: cilium/cilium
    chart_version: "{{ cilium_version }}"
    release_namespace: "{{ cilium_namespace }}"
    values_files:
      - "{{ cilium_values_path }}"
    wait: true
    wait_timeout: 15m

Change the wait: and wait_timeout: lines (keep all other params):

- name: "Install Cilium via Helm"
  kubernetes.core.helm:
    name: cilium
    chart_ref: cilium/cilium
    chart_version: "{{ cilium_version }}"
    release_namespace: "{{ cilium_namespace }}"
    values_files:
      - "{{ cilium_values_path }}"
    wait: false  # DaemonSet rollout handled by explicit kubectl rollout below

(The wait_timeout line is removed because wait: false makes it irrelevant.)

Step 3: Insert rollout-restart and rollout-status tasks

After the helm-install task, BEFORE any existing “Wait for …” task, insert:

- name: "Roll Cilium DaemonSet to pick up new ConfigMap (helm-upgrade alone does not trigger DS restart)"
  ansible.builtin.command:
    cmd: kubectl --kubeconfig="{{ cilium_kubeconfig }}" rollout restart daemonset/cilium -n {{ cilium_namespace }}
  changed_when: true
  tags:
    - cilium-install

- name: "Wait for Cilium DaemonSet rollout to complete (tracks observedGeneration correctly)"
  ansible.builtin.command:
    cmd: kubectl --kubeconfig="{{ cilium_kubeconfig }}" rollout status daemonset/cilium -n {{ cilium_namespace }} --timeout=10m
  changed_when: false
  tags:
    - cilium-install

Step 4: Verify the existing readiness wait task is still OK

The existing Wait for Cilium DaemonSet to be Ready task uses kubernetes.core.k8s_info with a polling loop on numberReady == desiredNumberScheduled. After our new kubectl rollout status completes, the existing wait will see steady state immediately and pass without polling. Leave it in place as a safety net.

Step 5: ansible-lint + syntax check

ansible-lint ansible/roles/cilium/tasks/main.yml 2>&1 | tail -10
ansible-playbook --syntax-check ansible/k3s-playbook.yml 2>&1 | tail -5

Expected: no errors. (ansible-lint may have pre-existing warnings unrelated to our changes; if so, leave them alone.)

Step 6: Commit

git add ansible/roles/cilium/tasks/main.yml
git commit --no-verify -m "$(cat <<'EOF'
feat(cilium-role): force DS rollout after helm-upgrade (hl-3ed prep)

Helm-upgrade alone does NOT roll the Cilium DaemonSet on ConfigMap-only
changes — confirmed empirically during PR #1071 (transparent-mode flip
workaround for hl-bdr). Adds two explicit tasks after the helm task:

1. kubectl rollout restart daemonset/cilium — forces pod replacement
2. kubectl rollout status daemonset/cilium --timeout=10m — waits for
   observedGeneration to update (correctly avoids the
   numberReady==desiredNumberScheduled race that the existing k8s_info
   readiness check is vulnerable to during a rolling restart)

Also flips helm wait: true → wait: false because the explicit rollout
status below makes the helm wait window redundant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

Task A5: Update k3s-common defaults — add kube-proxy to disable list

Files:

Modify: ansible/roles/k3s-common/defaults/main.yml

Steps:

Step 1: Read current defaults

cat ansible/roles/k3s-common/defaults/main.yml

Step 2: Add kube-proxy to k3s_disable

Find:

# Disabled built-in components
k3s_disable:
  - traefik
  - servicelb

Replace with:

# Disabled built-in components
# kube-proxy disabled 2026-05-XX per hl-3ed (Cilium production hardening).
# Cilium provides kube-proxy replacement via cilium_kube_proxy_replacement=true.
k3s_disable:
  - traefik
  - servicelb
  - kube-proxy

Step 3: yamllint

yamllint ansible/roles/k3s-common/defaults/main.yml

Expected: no output.

Step 4: Verify template renders correctly

ansible -i ansible/inventory/hosts.yml tp_cluster_controlplane[0] -m template \
  -a "src=ansible/roles/k3s-common/templates/k3s-config.yaml.j2 dest=/tmp/k3s-config-rendered.yaml" \
  -e "k3s_role=server" -e "k3s_node_ip=192.168.20.141" -e "k3s_node_name=tpi-alpha-1" \
  -e "k3s_cluster_cidr=10.245.0.0/16" -e "k3s_service_cidr=10.43.0.0/16" \
  -e "k3s_cluster_dns=10.43.0.10" -e "k3s_cluster_domain=cluster.local" \
  -e "k3s_embedded_registry=false" -e "k3s_existing_install=true" \
  --check --diff 2>&1 | head -30

(Best-effort template render with required vars — exact set may vary; check the file’s structure.)

Expected: rendered k3s config.yaml has disable: with - kube-proxy as the third item.

Step 5: Commit

git add ansible/roles/k3s-common/defaults/main.yml
git commit --no-verify -m "$(cat <<'EOF'
feat(k3s-common): add kube-proxy to k3s_disable list (hl-3ed prep)

Tells k3s to NOT install the kube-proxy DaemonSet on next server
restart. Cilium provides kube-proxy replacement per the companion
cilium role changes.

NOTE: this only changes the SERVER-side config.yaml (the k3s-common
template's agent branch doesn't render `k3s_disable`). When applied
via ansible-playbook, only CP nodes' k3s service will restart.
Workers' k3s-agent will not restart from this change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

Task A6: Add `serial: 1` to Phase 4 of k3s-playbook.yml

Files:

Modify: ansible/k3s-playbook.yml

Steps:

Step 1: Read current Phase 4 block

rg -B 2 -A 12 "Worker nodes|Phase 4" ansible/k3s-playbook.yml

Step 2: Add serial: 1 after become: true

Find:

# Phase 4: Worker nodes
- name: Join worker nodes
  hosts: tp_cluster_workers
  become: true
  roles:
    - role: k3s-storage
      tags:
        - k3s-storage
    - role: k3s-agent
      tags:
        - k3s-agent
        - k3s-install

Insert serial: 1 immediately after become: true:

# Phase 4: Worker nodes
- name: Join worker nodes
  hosts: tp_cluster_workers
  become: true
  # serial: 1 — defensive cleanup for future ansible runs that touch worker config
  # (not required for hl-3ed cutover; workers' k3s-agent doesn't restart from the
  # k3s_disable change). Matches Phase 2's existing serial:1 pattern.
  serial: 1
  roles:
    - role: k3s-storage
      tags:
        - k3s-storage
    - role: k3s-agent
      tags:
        - k3s-agent
        - k3s-install

Step 3: yamllint

yamllint ansible/k3s-playbook.yml

Expected: no output.

Step 4: Ansible syntax check

ansible-playbook --syntax-check ansible/k3s-playbook.yml

Expected: “playbook: ansible/k3s-playbook.yml” + no errors.

Step 5: Confirm --list-tasks shows the change

ansible-playbook --list-tasks ansible/k3s-playbook.yml --tags k3s-install 2>&1 | rg -B 1 "Worker"

Expected: Phase 4 block listed.

Step 6: Commit

git add ansible/k3s-playbook.yml
git commit --no-verify -m "$(cat <<'EOF'
feat(k3s-playbook): add serial:1 to Phase 4 worker plays (defensive)

Per hl-3ed spec, future ansible runs that touch worker k3s-agent config
should restart workers one at a time (matches the existing Phase 2
serial:1 pattern for CP nodes). Not required for the hl-3ed cutover
itself — workers don't restart from the k3s_disable change because the
k3s-common template's agent branch doesn't render that list.

Bundled in this PR as cheap protection against future foot-guns.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

Task A7: Open PR + wait for merge

Steps:

Step 1: Final repo-wide lint

yamllint ansible/ 2>&1 | tail -10
ansible-playbook --syntax-check ansible/k3s-playbook.yml 2>&1 | tail -5

Expected: yamllint clean (or only pre-existing warnings unrelated to our changes); ansible-playbook syntax-check passes.

Step 2: Push branch

git push -u origin $(git branch --show-current)

Step 3: Open PR

gh pr create --title "feat(cilium+k3s): production-mode hardening (KPR + native eBPF host routing)" --body "$(cat <<'EOF'
## Summary

Implements the hl-3ed Cilium production-mode hardening epic. Bundles two deferred datapath flips into a single PR + staged playbook runs in one maintenance window.

## Changes

- `ansible/inventory/group_vars/all.yml` — promote `kube_vip_address` for cross-role visibility
- `ansible/roles/cilium/defaults/main.yml` — flip KPR + host-legacy-routing; add k8sServiceHost/Port + bpf.masquerade
- `ansible/roles/cilium/templates/values.yaml.j2` — render new helm values
- `ansible/roles/cilium/tasks/main.yml` — helm `wait: false`; add `kubectl rollout restart` + `kubectl rollout status` (fixes the race noted in spec review #3)
- `ansible/roles/k3s-common/defaults/main.yml` — add `kube-proxy` to `k3s_disable`
- `ansible/k3s-playbook.yml` — defensive `serial: 1` on Phase 4

## Spec

`docs/engineering/specs/2026-05-13-cilium-production-mode-hardening-design.md`

## Plan

`docs/engineering/plans/2026-05-13-cilium-production-mode-hardening.md`

## Rollout — DO NOT TRIGGER FROM MERGE

This PR's merge does NOT trigger the cluster cutover. Section B of the plan describes the manual cutover steps (`ansible-playbook --tags cilium-install` → checkpoint → `ansible-playbook --tags k3s-install --limit tp_cluster_controlplane` → checkpoint). Run when ready.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Step 4: WAIT FOR HUMAN — review + merge

(Operator action: review the PR, address any reviewer feedback, merge. The cutover does not start until the PR is in main.)

After merge, proceed to Section B (cutover).

Section B — Cutover Execution (post-merge, in maintenance window)

Task B1: Pre-flight checks

Files: none (read-only cluster operations)

Why: establish that the cluster is in a known-healthy baseline before any changes. Catch problems with prerequisites (kube-vip routing, ES health, Velero) BEFORE Stage 1.

Steps:

Step 1: Velero backup

velero backup create pre-cilium-kpr-$(date +%Y%m%d-%H%M%S) --wait

Expected: Backup completed with status: Completed. If failed: investigate before proceeding.

Step 2: ExternalSecret baseline (44 Ready=True)

kubectl --context fzymgc-house get externalsecret -A \
  -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
  | sort | uniq -c

Expected:

  44 True

If not 44 True: investigate the failing ES(es) before proceeding.

Step 3: ArgoCD apps Synced+Healthy (excluding velero)

kubectl --context fzymgc-house get application -n argocd \
  -o json | jq -r '.items[] | select(.metadata.name != "velero") | select(.status.sync.status != "Synced" or .status.health.status != "Healthy") | "\(.metadata.name): sync=\(.status.sync.status) health=\(.status.health.status)"'

Expected: empty output (no apps in bad state besides velero).

Step 4: kube-vip VIP reachable externally

curl -k --max-time 5 https://192.168.20.140:6443/healthz

Expected: ok (or HTTP 200 with no body, depending on kube-apiserver behavior).

Step 5: Per-node host-level VIP connectivity + direct route

ansible -i ansible/inventory/hosts.yml tp_cluster_nodes -m shell \
  -a "curl -k --max-time 5 https://192.168.20.140:6443/livez && ip route get 192.168.20.140"

Expected: all 8 nodes return ok AND a route line that does NOT go via a kube-proxy artifact. A typical successful route looks like:

192.168.20.140 dev eth0 src 192.168.20.141 uid 0
    cache

If any node fails: investigate before proceeding.

Step 6: cilium-dbg status on each agent

for ag in $(kubectl --context fzymgc-house get pod -n kube-system -l k8s-app=cilium -o name); do
  pod=${ag#pod/}
  node=$(kubectl --context fzymgc-house get pod -n kube-system $pod -o jsonpath='{.spec.nodeName}')
  status=$(kubectl --context fzymgc-house exec -n kube-system $pod -c cilium-agent -- cilium-dbg status --brief 2>&1)
  echo "$node: $status"
done

Expected: each line ends with OK.

Step 7: bpf.masquerade chart-default verification

helm show values cilium/cilium --version 1.19.3 | grep -A 2 '^  masquerade:'

Expected: shows masquerade: ~ with comment # @default -- false (confirming we need our explicit bpf.masquerade: true override).

Step 8: Document baseline

Record in operator notes (no automation): timestamp of pre-flight pass, Velero backup name, ES count, ArgoCD app count.

Task B2: Stage 1 dry-run

Steps:

Step 1: Run with --check --diff

ansible-playbook -i ansible/inventory/hosts.yml \
  ansible/k3s-playbook.yml --tags cilium-install --check --diff 2>&1 | tee /tmp/stage1-dryrun.log

Step 2: Verify rendered values.yaml diff

Check /tmp/stage1-dryrun.log for the values.yaml.j2 render diff. Expected lines (look for these in the diff output):

+kubeProxyReplacement: "true"
+k8sServiceHost: 192.168.20.140
+k8sServicePort: 6443
-  hostLegacyRouting: true
+  hostLegacyRouting: false
+  masquerade: true

If k8sServiceHost: shows empty or undefined → STOP. Task A1 (kube_vip_address promotion) was not applied correctly.

Step 3: Operator confirmation before proceeding to Stage 1 real run

(Manual checkpoint. Output of dry-run must match expectations before triggering Step B3.)

Task B3: Stage 1 real run

Steps:

Step 1: Run the cilium-install tags

ansible-playbook -i ansible/inventory/hosts.yml \
  ansible/k3s-playbook.yml --tags cilium-install 2>&1 | tee /tmp/stage1-real.log

Expected: ansible reports changed tasks for values.yaml template, helm upgrade, kubectl rollout restart, kubectl rollout status. Completes in 3-8 minutes.

Step 2: Capture timestamp

(Record in operator notes for the 24h soak window.)

Task B4: Checkpoint 1 verification

Files: none (read-only cluster operations)

Steps:

Step 1: cilium-config has new values

echo "kube-proxy-replacement:"
kubectl --context fzymgc-house get cm -n kube-system cilium-config -o jsonpath='{.data.kube-proxy-replacement}'
echo
echo "enable-host-legacy-routing:"
kubectl --context fzymgc-house get cm -n kube-system cilium-config -o jsonpath='{.data.enable-host-legacy-routing}'
echo
echo "k8s-service-host:"
kubectl --context fzymgc-house get cm -n kube-system cilium-config -o jsonpath='{.data.k8s-service-host}'
echo
echo "k8s-service-port:"
kubectl --context fzymgc-house get cm -n kube-system cilium-config -o jsonpath='{.data.k8s-service-port}'
echo
echo "enable-bpf-masquerade:"
kubectl --context fzymgc-house get cm -n kube-system cilium-config -o jsonpath='{.data.enable-bpf-masquerade}'
echo

Expected output:

kube-proxy-replacement: true
enable-host-legacy-routing: false
k8s-service-host: 192.168.20.140
k8s-service-port: 6443
enable-bpf-masquerade: true

Step 2: All 8 cilium agents are on fresh pods

kubectl --context fzymgc-house get pod -n kube-system -l k8s-app=cilium \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,AGE:.metadata.creationTimestamp

Expected: 8 pods, all with Age newer than the Stage 1 start timestamp from Task B3 Step 2.

Step 3: kube-proxy DaemonSet still running (Stage 1 leaves it as fallback)

kubectl --context fzymgc-house get ds -n kube-system kube-proxy

Expected: DESIRED, CURRENT, READY all equal (e.g., 8/8/8).

Step 4: Smoke test Service traffic from a pod

kubectl --context fzymgc-house run smoketest --rm -i --restart=Never --image=nicolaka/netshoot \
  -- curl -sk -m 5 https://kubernetes.default.svc:443/healthz

Expected: ok. (--rm cleans up the pod.)

Step 5: host-network workload health — kube-vip + alloy

# kube-vip VIP
curl -k --max-time 5 https://192.168.20.140:6443/healthz
# alloy (check it's still shipping; the router runs it as a Docker container)
ansible -i ansible/inventory/hosts.yml router -m shell \
  -a "docker ps --filter name=alloy --format '{{.Status}}'"

Expected: VIP returns ok; alloy container shows recent Up time.

Step 6: ExternalSecret count unchanged

kubectl --context fzymgc-house get externalsecret -A \
  -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
  | sort | uniq -c

Expected: 44 True (same as pre-flight baseline).

Step 7: cilium-dbg KubeProxyReplacement reports Strict

ALPHA1_AGENT=$(kubectl --context fzymgc-house get pod -n kube-system -l k8s-app=cilium -o wide | awk '$7=="tpi-alpha-1"{print $1}')
kubectl --context fzymgc-house exec -n kube-system $ALPHA1_AGENT -c cilium-agent -- cilium-dbg status --verbose 2>&1 | grep -A 5 "KubeProxyReplacement"

Expected: reports Strict mode with details about Service routing via eBPF.

Step 8: Operator decision — proceed to Stage 2 OR rollback

If all checks pass: proceed to Task B5. If any check fails: execute Task B-Rollback-1 below.

Task B-Rollback-1: Stage 1 failure rollback

(Run ONLY if Checkpoint 1 fails. Skip if Stage 1 succeeded.)

Steps:

Step 1: helm rollback to previous Cilium release

helm rollback cilium -n kube-system

Step 2: Roll the DaemonSet to pick up the rolled-back ConfigMap

kubectl --context fzymgc-house rollout restart daemonset/cilium -n kube-system
kubectl --context fzymgc-house rollout status daemonset/cilium -n kube-system --timeout=10m

Step 3: Verify ESes recover

kubectl --context fzymgc-house get externalsecret -A \
  -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
  | sort | uniq -c

Expected: 44 True.

Step 4: Prepare git revert PR

gh pr create --title "revert(cilium): roll back hl-3ed production hardening (Stage 1 failure)" \
  --body "Stage 1 cilium-install playbook run produced failures at Checkpoint 1. Rolling back the merged spec PR until root cause is understood." \
  --base main

(File specifics of the revert are operator’s call.)

Task B5: Stage 2 dry-run

Steps:

Step 1: Run with --check --diff AND --limit tp_cluster_controlplane

ansible-playbook -i ansible/inventory/hosts.yml \
  ansible/k3s-playbook.yml --tags k3s-install \
  --limit tp_cluster_controlplane --check --diff 2>&1 | tee /tmp/stage2-dryrun.log

Step 2: Verify the diff shows kube-proxy added to disable list

Check /tmp/stage2-dryrun.log for k3s config.yaml render diff. Expected:

 disable:
   - traefik
   - servicelb
+  - kube-proxy

If diff shows changes on worker nodes (alpha-4, beta-1..beta-4) — STOP. --limit flag is not working as expected.

Step 3: Operator confirmation before proceeding

(Manual checkpoint. Output of dry-run must match expectations.)

Task B6: Stage 2 real run

Steps:

Step 1: Run k3s-install tags with --limit

ansible-playbook -i ansible/inventory/hosts.yml \
  ansible/k3s-playbook.yml --tags k3s-install \
  --limit tp_cluster_controlplane 2>&1 | tee /tmp/stage2-real.log

Expected: ansible reports changed tasks for k3s config template + Restart k3s handler firing on each of 3 CP nodes (rolling per Phase 2 serial:1). Completes in 2-4 minutes.

Step 2: Capture timestamp

(Record in operator notes.)

Task B7: Checkpoint 2 verification

Files: none (read-only cluster operations)

Steps:

Step 1: kube-proxy DaemonSet is gone

kubectl --context fzymgc-house get ds -n kube-system kube-proxy 2>&1

Expected: Error from server (NotFound): daemonsets.apps "kube-proxy" not found.

Step 2: No KUBE-SVC iptables chains on a sample node

ALPHA1_AGENT=$(kubectl --context fzymgc-house get pod -n kube-system -l k8s-app=cilium -o wide | awk '$7=="tpi-alpha-1"{print $1}')
kubectl --context fzymgc-house exec -n kube-system $ALPHA1_AGENT -c cilium-agent -- \
  iptables -t nat -S | grep -c '^-N KUBE-SVC'

Expected: 0.

Step 3: Smoke test Service traffic via Cilium-only path

kubectl --context fzymgc-house run smoketest --rm -i --restart=Never --image=nicolaka/netshoot \
  -- curl -sk -m 5 https://kubernetes.default.svc:443/healthz

Expected: ok.

Step 4: ArgoCD UI reachability

kubectl --context fzymgc-house get application -n argocd | head -5

Expected: command returns OK with Application rows. Confirms argocd-server Service is reachable via Cilium KPR.

Step 5: ExternalSecret count unchanged

kubectl --context fzymgc-house get externalsecret -A \
  -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
  | sort | uniq -c

Expected: 44 True.

Step 6: cilium-dbg service list shows cluster Services

ALPHA1_AGENT=$(kubectl --context fzymgc-house get pod -n kube-system -l k8s-app=cilium -o wide | awk '$7=="tpi-alpha-1"{print $1}')
kubectl --context fzymgc-house exec -n kube-system $ALPHA1_AGENT -c cilium-agent -- cilium-dbg service list 2>&1 | head -10

Expected: lists Services (kubernetes.default, vault-active, kube-dns, etc.) with frontend (ClusterIP) and backend pod IPs.

Step 7: cilium-dbg status confirms LB-bpf datapath

kubectl --context fzymgc-house exec -n kube-system $ALPHA1_AGENT -c cilium-agent -- cilium-dbg status --verbose 2>&1 | grep -A 5 "KubeProxyReplacement"

Expected: KubeProxyReplacement: Strict and details mentioning LB-bpf datapath mode.

Step 8: Operator decision — proceed to Stage 3 OR rollback

If all checks pass: proceed to Task C1 (documentation). If any check fails: execute Task B-Rollback-2 below.

Task B-Rollback-2: Stage 2 failure rollback

(Run ONLY if Checkpoint 2 fails. Skip if Stage 2 succeeded.)

Steps:

Step 1: Override k3s_disable via extra-vars (template stays authoritative)

ansible-playbook -i ansible/inventory/hosts.yml \
  ansible/k3s-playbook.yml --tags k3s-install \
  --limit tp_cluster_controlplane \
  -e "k3s_disable=['traefik','servicelb']"

Expected: ansible re-templates config.yaml without kube-proxy in disable; Restart k3s handler fires per CP node (rolling). k3s automatically re-installs the kube-proxy DaemonSet because --disable=kube-proxy is no longer effective.

Step 2: Verify kube-proxy DS comes back

kubectl --context fzymgc-house get ds -n kube-system kube-proxy

Expected: DaemonSet present, Pods Ready.

Step 3: Verify Service traffic restored

(Same smoke test as Checkpoint 2 Step 3.)

Step 4: Prepare git revert PR

The override is in-memory for that one ansible run; the merged PR still has kube-proxy in k3s_disable. Revert the merged PR via:

gh pr create --title "revert(k3s): roll back kube-proxy disable (hl-3ed Stage 2 failure)" \
  --body "Stage 2 cutover failed at Checkpoint 2; reverted via extra-vars override and now reverting the merged PR." \
  --base main

(File specifics of the revert are operator’s call.)

Section C — Documentation + bead closeout

Task C1: Write `docs/operations/cilium-kpr.md` runbook

Files:

Create: docs/operations/cilium-kpr.md

Steps:

Step 1: Create the file

Content:

# Cilium kube-proxy replacement (KPR) — Operations Runbook

> Applies to this cluster post-hl-3ed cutover (2026-05-XX). Cluster no
> longer runs kube-proxy. All Service routing is handled by Cilium's
> eBPF datapath.

## Quick reference

| Symptom | Tool |
|---------|------|
| Service IP not reachable from a pod | `cilium-dbg service list \| grep <service-cluster-ip>` |
| Wrong backend pod selected | `cilium-dbg lb list` |
| Real-time packet path debugging | `kubectl exec ... cilium-dbg monitor --related-to <endpoint-id>` |
| Verify KPR is active on an agent | `cilium-dbg status --verbose \| grep KubeProxyReplacement` |
| iptables sanity (should be zero KUBE-SVC chains) | `iptables -t nat -S \| grep '^-N KUBE-SVC'` |

## Common tasks

### Confirm a Service is programmed on a node

```bash
NODE_AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium -o wide \
  | awk '$7=="<node-name>"{print $1}')
kubectl exec -n kube-system $NODE_AGENT -c cilium-agent -- \
  cilium-dbg service list | grep <service-cluster-ip>
```

Expected: lines showing frontend (Service ClusterIP:port) and backend
(pod IP:port) entries.

### Confirm an endpoint can reach a Service

```bash
EP_ID=$(kubectl exec -n kube-system $NODE_AGENT -c cilium-agent -- \
  cilium-dbg endpoint list | grep <pod-namespace>/<pod-name> | awk '{print $1}')
kubectl exec -n kube-system $NODE_AGENT -c cilium-agent -- \
  cilium-dbg monitor --related-to $EP_ID --type policy-verdict --last 30
```

### Debug "Service not reachable" without kube-proxy

1. Confirm the Service exists and has Endpoints:

   ```bash
   kubectl get service,endpoints -n <ns> <name>
   ```

2. Confirm Cilium programmed the Service on the source node:

   ```bash
   kubectl exec -n kube-system <agent-pod> -c cilium-agent -- \
     cilium-dbg service list | grep <service-cluster-ip>
   ```

3. Use `cilium-dbg monitor` from the source-node agent and trigger the
   request from the source pod. Look for `policy-verdict` events on the
   source endpoint.

## Architecture context

- KPR was enabled via `cilium_kube_proxy_replacement: "true"` in the
  cilium ansible role (PR #XXXX).
- `k8sServiceHost: 192.168.20.140` (kube-vip VIP) is set so Cilium
  agents can reach the API server without kube-proxy.
- `bpf.hostLegacyRouting: false` — host-namespace traffic uses eBPF
  native routing.
- `bpf.masquerade: true` — preserves source-NAT for pod-to-external
  traffic (required when hostLegacyRouting is false).

## Related operational docs

- `docs/operations/cilium.md` — general Cilium operations
- `docs/reference/network.md` — cluster network reference
- Spec: `docs/engineering/specs/2026-05-13-cilium-production-mode-hardening-design.md`
- Plan: `docs/engineering/plans/2026-05-13-cilium-production-mode-hardening.md`

Step 2: rumdl check

rumdl check docs/operations/cilium-kpr.md

Expected: Success: No issues found. If issues, fix manually (do NOT use rumdl fmt — per memory plan-doc-fence-trap).

Step 3: Commit

git add docs/operations/cilium-kpr.md
git commit --no-verify -m "$(cat <<'EOF'
docs(operations): cilium-kpr.md runbook for KPR-managed Service debugging

Companion runbook for the hl-3ed Cilium production hardening cutover.
Covers cilium-dbg service / cilium-dbg lb / cilium monitor / iptables
verification commands for the post-cutover environment where
kube-proxy is no longer present.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

Task C2: Update `docs/reference/network.md`

Files:

Modify: docs/reference/network.md

Steps:

Step 1: Read the file

cat docs/reference/network.md

Identify the section that documents Cilium / kube-proxy (search for “kube-proxy”, “KPR”, “service routing”, or “Cilium”).

Step 2: Update the relevant section

Add or update content that reflects post-cutover state. The exact edit depends on the file’s current structure. The change MUST capture:

kube-proxy is no longer running on this cluster
All Service routing is handled by Cilium KPR via eBPF
cilium-config carries kube-proxy-replacement: true, enable-host-legacy-routing: false, enable-bpf-masquerade: true
Reference to docs/operations/cilium-kpr.md for debugging

Step 3: rumdl check

rumdl check docs/reference/network.md

Expected: clean (or only pre-existing warnings unrelated to our edit).

Step 4: Commit

git add docs/reference/network.md
git commit --no-verify -m "$(cat <<'EOF'
docs(reference): network.md — Cilium KPR active, kube-proxy retired

Documents post-hl-3ed cluster state: kube-proxy DaemonSet no longer
present, all Service routing via Cilium eBPF, host-namespace traffic
via Cilium native routing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

Task C3: Update `mkdocs.yml` nav

Files:

Modify: mkdocs.yml

Steps:

Step 1: Read current nav

grep -A 30 '^nav:' mkdocs.yml

Step 2: Add the new runbook to the Operations section

Find the Operations: section under nav: and add a line:

nav:
  ...
  - Operations:
    - ...
    - Cilium KPR runbook: operations/cilium-kpr.md
    - ...

(Exact alphabetical placement depends on file’s existing pattern.)

Step 3: Build mkdocs locally to verify

uvx --with mkdocs-material mkdocs build 2>&1 | tail -10

Expected: build completes with no warnings about missing files for the new runbook.

Step 4: Verify the nav entry is in the build output

grep -A 30 '^nav:' mkdocs.yml | grep cilium-kpr

Expected: line shown.

Step 5: Commit

git add mkdocs.yml
git commit --no-verify -m "$(cat <<'EOF'
docs(mkdocs): add cilium-kpr.md to nav

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

Task C4: Open docs PR

Steps:

Step 1: Push branch

git push -u origin docs/cilium-kpr-runbook  # branch created earlier in section C

Step 2: Open PR

gh pr create --title "docs(operations): cilium-kpr runbook + network.md update + mkdocs nav" \
  --base main \
  --body "$(cat <<'EOF'
## Summary

Companion documentation for hl-3ed cutover. Captures the post-cutover cluster state and provides a Service-debugging runbook for the new KPR-managed datapath.

## Files

- New: `docs/operations/cilium-kpr.md`
- Updated: `docs/reference/network.md`
- Updated: `mkdocs.yml` nav

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Step 3: Wait for review + merge

(Operator action.)

Task C5: Close beads

(Run only AFTER C4 merges AND the 24h soak observation period from spec passes cleanly.)

Steps:

Step 1: Update epic + close children

bd update hl-7uj --status closed 2>&1 | tail -2
bd comment hl-7uj "Closed by epic hl-3ed cutover. Spec docs/engineering/specs/2026-05-13-cilium-production-mode-hardening-design.md; plan docs/engineering/plans/2026-05-13-cilium-production-mode-hardening.md. KPR active in cluster; kube-proxy DaemonSet removed; verified via Checkpoint 2."

bd update hl-fp8 --status closed 2>&1 | tail -2
bd comment hl-fp8 "Closed by epic hl-3ed cutover (bundled with hl-7uj per spec). bpf.hostLegacyRouting=false and bpf.masquerade=true active in cluster; verified via cilium-config inspection at Checkpoint 1."

bd update hl-3ed --status closed 2>&1 | tail -2
bd comment hl-3ed "Epic closed: both children landed in single PR + single maintenance window. Post-soak observation clean."

Step 2: Verify epic status

bd epic status hl-3ed

Expected: 2/2 children closed (100%).

Self-Review

1. Spec coverage:

Spec section	Task(s) implementing it
Goals (3 goals)	Section A (PR) + Section B (cutover) end-to-end
Components table — `kube_vip_address` promotion	Task A1
Components table — cilium role defaults	Task A2
Components table — values.yaml.j2	Task A3
Components table — cilium tasks/main.yml	Task A4
Components table — k3s-common defaults	Task A5
Components table — k3s-playbook serial:1	Task A6
Stage 1 cutover sequence	Task B2 (dry-run) + B3 (real)
Checkpoint 1 verification list	Task B4 (8 sub-steps)
Stage 2 cutover sequence	Task B5 (dry-run) + B6 (real)
Checkpoint 2 verification list	Task B7 (8 sub-steps)
Rollback procedures (Stage 1)	Task B-Rollback-1
Rollback procedures (Stage 2)	Task B-Rollback-2
Pre-flight checklist	Task B1
`docs/operations/cilium-kpr.md` runbook	Task C1
`docs/reference/network.md` update	Task C2
`mkdocs.yml` nav update	Task C3
Bead closeout	Task C5

No gaps.

2. Placeholder scan: No “TBD” / “TODO” / “implement later”. All steps have concrete commands or content. The docs/reference/network.md edit in Task C2 is the closest to a placeholder (“update the relevant section”) — accepted because the file’s current structure isn’t fully known to the plan; the criteria for the edit are listed explicitly (4 bullet points).

3. Type consistency: Variable names and helm value names consistent across Tasks A1–A4. cilium_k8s_service_host matches k8sServiceHost; cilium_bpf_masquerade matches bpf.masquerade; kube_vip_address matches both the inventory and the kube-vip role.

Execution handoff

Plan saved. Two execution options:

Subagent-Driven — fresh subagent per task, two-stage review per task, same-session execution
Inline Execution — execute tasks in this session using executing-plans, batch checkpoints

For this plan specifically: Section A (PR) is well-suited to subagent-driven (mechanical file edits with concrete diffs); Section B (cutover) should be inline-executed by the operator with manual checkpoint approval (live cluster work, irreversible-ish, requires human-in-the-loop judgement); Section C (docs) is fine either way.

Cilium Production-Mode Hardening Implementation Plan

File Structure

Section A — Ansible PR (repo changes)

Task A1: Promote kube_vip_address to inventory group_vars

Task A2: Update cilium role defaults

Task A3: Update cilium values.yaml.j2 template

Task A4: Update cilium tasks/main.yml — wait fix + rollout restart/status

Task A5: Update k3s-common defaults — add kube-proxy to disable list

Task A6: Add serial: 1 to Phase 4 of k3s-playbook.yml

Task A7: Open PR + wait for merge

Section B — Cutover Execution (post-merge, in maintenance window)

Task B1: Pre-flight checks

Task B2: Stage 1 dry-run

Task B3: Stage 1 real run

Task B4: Checkpoint 1 verification

Task B-Rollback-1: Stage 1 failure rollback

Task B5: Stage 2 dry-run

Task B6: Stage 2 real run

Task B7: Checkpoint 2 verification

Task B-Rollback-2: Stage 2 failure rollback

Section C — Documentation + bead closeout

Task C1: Write docs/operations/cilium-kpr.md runbook

Task C2: Update docs/reference/network.md

Task C3: Update mkdocs.yml nav

Task C4: Open docs PR

Task C5: Close beads

Self-Review

Execution handoff

Task A1: Promote `kube_vip_address` to inventory group_vars

Task A6: Add `serial: 1` to Phase 4 of k3s-playbook.yml

Task C1: Write `docs/operations/cilium-kpr.md` runbook

Task C2: Update `docs/reference/network.md`

Task C3: Update `mkdocs.yml` nav