Lab Service VLAN: node legs and rollout

Every k3s node carries a tagged leg on the Lab VLAN so that Lab-hosted LoadBalancer VIPs have L2 adjacency and a symmetric return path. This page describes what the change puts in place and the ordered live rollout that brings it up. kubectl commands assume --context fzymgc-house. Run the Ansible commands from ansible/.

What exists after this change

Every k3s node carries a tagged Lab leg, end0.1000, on the same uplink as Main (the ports are already trunked: native 3000, tagged 1000). The leg’s static address mirrors the node’s Main host number (192.168.200.141-.144, .151-.154, /22). A routing-policy rule (from 192.168.200.0/22 lookup 1000, priority 6000) sends Lab-sourced traffic through table 1000. That table holds the on-link 192.168.200.0/22 route plus a default via 192.168.200.1, so a reply from a Lab address leaves by the Lab leg. Cilium manages end0.1000 as a second device. MetalLB has a lab pool (192.168.200.145-149, .155-159) that is never auto-assigned. The Lab DHCP pool starts at .200.160. Nothing clients dial is on Lab yet.

There is no switch step: the node uplinks are already trunked. There is no Firewalla step: Main to Lab is already permitted.

Names that must match

• end0.1000 appears in three places: the netplan VLAN (tp2-bootstrap-node, <default iface>.<tp2_node_lab_vlan_id>), cilium_devices, and the lab L2Advertisement’s interfaces.
• .200.140-.200.159 is static. The Lab kea_pool must start above it, in both resolver host_vars.

Step 0: merge; ArgoCD syncs MetalLB

Safe before any leg exists. The lab pool (autoAssign: false) has no requester. The lab advertisement names a device no node has yet. The default advertisement is pinned to end0, the interface its VIPs are answered on today. That pin must be live before step 3: once a node has end0.1000, an unpinned advertisement would also answer ARP for Main VIPs on the Lab segment.

Verify:

kubectl -n metallb get ipaddresspools.metallb.io,l2advertisements.metallb.io \
  -o custom-columns=KIND:.kind,NAME:.metadata.name,AUTO:.spec.autoAssign,IFACES:.spec.interfaces
curl -s --max-time 5 -o /dev/null -w '%{http_code}\n' http://192.168.20.145/   # any HTTP status: Traefik still answers

Rollback: revert the MetalLB commit. ArgoCD prunes lab and drops the pin.

Step 1: shrink the Kea Lab pool, one resolver per window

Do the node that does NOT hold the Lab VIP 192.168.200.5 first. Find it:

scripts/dns-adhoc.sh dns_hosts -m ansible.builtin.command -a 'ip -o -4 addr show' | rg '\| (CHANGED|SUCCESS)|192\.168\.200\.5/'

For that node, then for the other (the example uses portland first):

export VAULT_TOKEN="$(cat ~/.vault-token)"   # the kea tag reads the token from the environment, not the CLI's file
scripts/dns-playbook.sh --limit portland --tags kea --check --diff   # the Lab pool line changes (see below)
scripts/dns-playbook.sh --limit portland --tags kea
scripts/dns-playbook.sh --limit portland --tags kea-verify
scripts/dns-adhoc.sh portland -b -m ansible.builtin.shell \
  -a 'kea-shell --host 127.0.0.1 --port 8000 --auth-user kea-ctrl --auth-password-file /etc/kea/creds/ctrl-basic-auth.pw config-get </dev/null' \
  | rg -o '192\.168\.200\.160-192\.168\.201\.19'   # Kea reports pools without spaces
scripts/dns-adhoc.sh portland -m ansible.builtin.command \
  -a 'systemctl is-active isc-kea-dhcp4-server isc-kea-dhcp-ddns-server'   # active, active
# repeat with --limit sandiego / sandiego

The dry run can show more than the pool line. When dns_tsig_key_generations_granted has narrowed since the last kea converge, the role also removes the retired generation’s D2 TSIG secret files (kea-d2-<leg>.tsig beside the current kea-d2-<leg>-<n>.tsig). That is the role’s own cleanup: confirm D2 references only the current generation before applying:

scripts/dns-adhoc.sh portland -b -m ansible.builtin.command -a 'cat /etc/kea/kea-dhcp-ddns.conf' \
  | rg -o 'kea-d2-[a-z]+(-[0-9]+)?\.tsig' | sort -u   # every name carries the current -<n>

Gate before step 3. No lease may sit in the static space:

scripts/dns-adhoc.sh dns_hosts -b -m ansible.builtin.shell \
  -a 'kea-shell --host 127.0.0.1 --port 8000 --auth-user kea-ctrl --auth-password-file /etc/kea/creds/ctrl-basic-auth.pw lease4-get-all </dev/null' \
  | rg '192\.168\.200\.1[45][0-9]"'   # expect no output

If a lease appears there, wait out its renewal (T1 is at most 6 h) and re-check. Do not start step 3 while one remains.

Expected side effect: the live Lab clients below .160 (master, main, rec and nas on lab.lan) are refused at their next renew (at most 6 h). They re-DISCOVER into .160+, and DDNS moves their names. Nothing in the repo pins their addresses.

Rollback: restore the previous kea_pool value (revert the Kea commit) and re-run the same converge, one node per window. Renumbered clients stay renumbered, which is harmless.

Step 2: Cilium device list (listed before the device exists)

velero backup create pre-deploy-$(date +%Y%m%d-%H%M%S) --wait
helm repo update cilium   # check mode does not write the local chart index the dry run reads
uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium --check --diff   # devices: "end0" -> "end0 end0.1000"
uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium

The backup waits for Longhorn to copy every CSI-snapshotted volume to the backup target. The ClickHouse volumes dominate that wait.

Verify:

kubectl -n kube-system get cm cilium-config -o yaml | rg -n 'end0'     # both names present
kubectl -n kube-system rollout status ds/cilium
cilium status --wait
curl -s --max-time 5 -o /dev/null -w '%{http_code}\n' http://192.168.20.145/

end0.1000 is not selected yet because it does not exist. That is expected.

Rollback: revert the cilium_devices change and re-run the same playbook command.

Step 3: node legs, one node at a time

Order: tpi-beta-1, tpi-beta-2, tpi-beta-3, then the control plane (tpi-alpha-1..3, with the current kube-vip holder last), then the tailnet routers tpi-alpha-4 and tpi-beta-4 last. Until this step is complete, never run bootstrap-nodes-playbook.yml without --limit <one node>.

Find the kube-vip holder:

for n in tpi-alpha-1 tpi-alpha-2 tpi-alpha-3; do ssh fzymgc@$n ip -br addr show end0 | rg -q '192\.168\.20\.140/' && echo "holder: $n"; done

Per node:

R=$(mktemp -d)   # once per session
N=tpi-beta-1
ssh fzymgc@$N ip rule > "$R/$N.rules.before"
uv run ansible-playbook -i inventory/hosts.yml bootstrap-nodes-playbook.yml --tags network --limit $N --check --diff   # only the vlans block is added
uv run ansible-playbook -i inventory/hosts.yml bootstrap-nodes-playbook.yml --tags network --limit $N
ssh fzymgc@$N ip rule | diff "$R/$N.rules.before" -     # exactly one added line: 6000: from 192.168.200.0/22 lookup 1000
ssh fzymgc@$N ip -br addr show end0.1000                 # UP, 192.168.200.<host>/22
ssh fzymgc@$N ip route show table 1000                   # 192.168.200.0/22 scope link, default via 192.168.200.1
ssh fzymgc@$N ip -br addr show end0                      # Main address unchanged; .20.140/32 still on the kube-vip holder
kubectl get --raw /readyz && ping -c2 192.168.20.140
kubectl get node $N                                      # Ready

• The ip rule diff also proves Cilium’s rules (9, 100) and, on tpi-alpha-4/tpi-beta-4, tailscale’s rules (5210-5270) survived. If tailscale’s rules are gone, run ssh fzymgc@$N sudo systemctl restart tailscaled and re-diff.
• If .20.140 vanished and /readyz fails, restart kube-vip on the node that held it: ssh fzymgc@$N 'sudo k3s crictl ps --name kube-vip -q | xargs -r sudo k3s crictl stop'. The kubelet restarts the static pod, and leader election re-announces the VIP.
• Cilium selects end0.1000 as soon as netplan creates it and rebuilds the VLAN allow set on end0. Within seconds, cilium-dbg status on that node lists end0.1000 192.168.200.<host>.
• A ping from the node’s Lab address to the gateway is not a gate. The Firewalla starts answering ARP from a new address on Lab after anything from seconds to several minutes, while other Lab hosts (the resolvers, other nodes) answer at once. ping -I needs sudo on the nodes: ssh fzymgc@$N sudo ping -c2 -I 192.168.200.<host> 192.168.200.1.

Rollback for one node: from a worktree at the pre-merge commit (git worktree add ../pre-lab-legs <pre-merge-sha>), run the same --tags network --limit $N command. netplan removes the VLAN, the rule and the table. Break-glass (non-persistent): ssh fzymgc@$N 'sudo ip rule del priority 6000; sudo ip link del end0.1000'. If the node drops off the network, use its board’s Turing Pi BMC console (tpi uart -n <slot>, see docs/operations/evidence/2026-08-23-tpi-beta-4-console-ring.md).

Step 4: confirm every agent adopted its leg

Each agent adopts end0.1000 at runtime (step 3), so no restart is needed when every node passes this check:

for n in tpi-alpha-1 tpi-alpha-2 tpi-alpha-3 tpi-alpha-4 tpi-beta-1 tpi-beta-2 tpi-beta-3 tpi-beta-4; do
  p=$(kubectl -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=$n -o name)
  echo "== $n"
  kubectl -n kube-system exec $p -c cilium-agent -- cilium-dbg status --verbose | rg 'KubeProxyReplacement:|Masquerading:'
  kubectl -n kube-system exec $p -c cilium-agent -- cilium-dbg metrics list \
    | rg 'cilium_drop_count_total.*direction=INGRESS reason=VLAN'   # sample twice, 60 s apart: unchanged
done
# expect end0.1000 on both lines. (Direct Routing) stays on end0; the device line lists the
# IPv6 link-local before the IPv4 address, so match on the device name, not the address order.
ssh fzymgc@tpi-beta-1 sudo ping -c2 -I 192.168.200.151 192.168.200.1   # repeat per node with its Lab address

If a node fails the check, restart the agents so each re-reads its device set and the VLAN allow set on end0:

uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium

Rollback: roll the legs back first (step 3 rollback, reverse node order), then revert cilium_devices and re-run. A leg without its Cilium device is dead on arrival, and pod-to-Lab traffic leaves un-masqueraded.

Step 5: scratch-Service proof

1. Read Traefik’s selector: kubectl -n traefik get svc traefik -o jsonpath='{.spec.selector}'.
2. Open a short-lived PR. It adds argocd/app-configs/traefik/lab-proof-service.yaml, a Service lab-proof of type LoadBalancer with annotation metallb.io/loadBalancerIPs: 192.168.200.159, externalTrafficPolicy: Local (the same as the traefik Service, so the reply takes the rev-NAT path the cutover depends on), the selector from step 1, and port 80 → targetPort web. The PR also lists the file in argocd/app-configs/traefik/kustomization.yaml. Use .159: the cutover assigns .145 and .149.
3. After sync, prove the return path with packet counters. The nodes carry no tcpdump, and end0.1000 is nearly silent, so a burst of requests stands out on it. The Firewalla flow log does not record these inter-VLAN flows, so it cannot serve as a second reading.

kubectl -n traefik get svc lab-proof -o jsonpath='{.status.loadBalancer.ingress[0].ip}'   # 192.168.200.159
kubectl -n metallb get servicel2statuses.metallb.io -o json \
  | jq -r '.items[] | select(.status.serviceName=="lab-proof") | "\(.status.node) \(.status.interfaces)"'   # announcing node, end0.1000
N=<announcing node>
c() { ssh fzymgc@$N ip -s -j link show end0.1000 | jq -r '.[0].stats64 | "\(.rx.packets) \(.tx.packets)"'; }
c; sleep 10; c                                                  # baseline: near zero change
for i in $(seq 200); do curl -s -o /dev/null --max-time 3 http://192.168.200.159/; done   # from a Main client
c                                                               # rx and tx both up by several per request
arp -an | rg '192\.168\.200\.'                                  # on the client: expect no entry (routed, not on-link)

Pass: every request answers, and end0.1000 shows both rx and tx rising by several packets per request (about 6 in and 4 out) over a flat baseline. If tx stays at baseline, the replies are leaving on end0, which means Cilium’s reply path ignored the source rule. Stop, revert the proof PR, and do not start the cutover. That becomes a Cilium-side follow-up. A successful curl on its own proves nothing: the client accepts a SYN-ACK from .159 whichever path it took.

Cleanup and rollback: revert the proof PR. The traefik app prunes.

Known limitations

• Residual asymmetry: a Lab host that dials a node’s Main address or a Main VIP arrives through the gateway on end0. The reply is sourced from 192.168.20.x, so the Lab rule does not match, and main’s connected Lab route sends it out end0.1000. No mirror rule has been added. Decide at cutover.
• NodePort frontends also answer on the node Lab addresses. Lab is a trusted segment.
• Lab is not advertised on the tailnet. Off-LAN tailnet clients cannot reach Lab VIPs until the cutover adds the route and the ACL.
• The leg is IPv4-only (accept-ra: false). ULA is on-link only because nothing routes it between VLANs (see IPv6 ULA Addressing, #2113), so a ULA VIP on Lab would be unreachable from Main. Accepting RAs would add SLAAC addresses that cannot mirror host numbers, a GUA from a prefix that changes on ISP re-delegation, and a second RA-learned route.

Whole-slice rollback

In reverse order: step 5 cleanup, then step 3 rollback on every node (reverse order), then step 2 rollback, then step 1 rollback, then step 0 revert.