Skip to content

Lab Service VLAN: node legs and LoadBalancer VIPs

Every k3s node carries a tagged leg on the Lab VLAN (VLAN 1000). The routed LoadBalancer VIPs reach the nodes over that leg in both address families. A client on Main reaches a VIP through the UCG-Fiber, so the path is routed and not on-link. This page states what exists, where each part is defined, and the procedures that touch it. kubectl commands assume --context fzymgc-house. Run the Ansible commands from ansible/.

Each node has the VLAN interface end0.1000 on the same uplink as Main, which is the VLAN interface end0.3000. The uplink of the board is a trunk with native Core and tagged VLAN 3000 and 1000 (Turing Pi boards). The leg holds two static addresses:

Family Address Source
IPv4 192.168.200.<host>/22 host_lab_address in the node’s host_vars
IPv6 fddb:f665:73f7:4:aaaa::<host>/64 tp2_node_lab_ula_prefix plus the last octet of host_lab_address

<host> is the last octet of the node’s Main address: 141 to 144 for tpi-alpha-1 to tpi-alpha-4, and 151 to 154 for tpi-beta-1 to tpi-beta-4.

The leg keeps accept-ra: false, so it takes no SLAAC address and no delegated GUA. A routing-policy rule at priority 6000 in each family sends traffic sourced from the Lab network to table 1000. That table holds the on-link Lab route and a default route through the Lab gateway (192.168.200.1 and fddb:f665:73f7:4::1). A reply from a Lab address to a Lab destination leaves on-link, and a reply to any other destination leaves through the gateway. The netplan template is ansible/roles/tp2-bootstrap-node/templates/01-netcfg.yaml.j2, and the values are in that role’s defaults/main.yml.

Cilium manages end0.1000 as a second device (cilium_devices in ansible/roles/cilium/defaults/main.yml).

The Lab leg also carries BGP. Each node peers with the UCG-Fiber from its Lab addresses, one session per family, to 192.168.200.1 and fddb:f665:73f7:4::1. The gateway routes each routed VIP (192.168.250.0/24 and fddb:f665:73f7:250::/64) to the Lab address of a node that holds an endpoint of the Service. A request from Main, MainWireless, Core or IoT crosses the gateway into Lab, and the reply crosses back. See BGP in the UniFi runbook.

A second rule at priority 6100 in each family sends traffic sourced from a routed VIP to table 1001. That table holds only the default route through the Lab gateway, so every reply from a routed VIP crosses the gateway, a reply to a Lab client too. Table 1000 does not serve here, because its on-link Lab route sends a reply to a Lab client past the gateway. The values are tp2_node_vip_route_table and tp2_node_vip_rule_priority in the tp2-bootstrap-node defaults, and the prefixes are lb_vip_prefix_ipv4 and lb_vip_prefix_ipv6 in ansible/inventory/group_vars/all.yml.

192.168.200.140 to .159 is static space. The node legs hold .141 to .144 and .151 to .154. .140, .145 to .149 and .155 to .159 are free. The Lab Kea pool starts at .160 (kea_pool on the lab entry in both resolver host_vars). The IPv6 suffixes mirror the IPv4 host octets under fddb:f665:73f7:4:aaaa::. The network reference lists every address.

Each LoadBalancer Service is PreferDualStack with IPv4 first. It runs externalTrafficPolicy: Local, so that the backend sees the client address. It sets loadBalancerClass: io.cilium/bgp-control-plane and pins both routed VIPs with one host number through lbipam.cilium.io/ips. Cilium LB-IPAM assigns them from the pool routed-vips (argocd/app-configs/cilium-lb/ip-pool.yaml). The pool serves only the namespaces that its serviceSelector lists, and Cilium assigns no address to a Service without the class.

Kubernetes refuses a change of loadBalancerClass on an existing LoadBalancer Service. To move a Service to another class, add a second Service with the same selector and ports, move every DNS writer to it, and then delete the first Service.

Service IPv4 VIP IPv6 VIP DNS writer
traefik/traefik-vip 192.168.250.145 fddb:f665:73f7:250::145 the target annotation on every IngressRoute
temporal/temporal-node-gateway-vip 192.168.250.146 fddb:f665:73f7:250::146 none. The resolver Pis dial the IPv4 literal
dolt/dolt-vip 192.168.250.147 fddb:f665:73f7:250::147 the external-dns service source, doltdb.fzymgc.house
mosquitto/mosquitto-vip 192.168.250.148 fddb:f665:73f7:250::148 the curated entry mqtt in dns-control
clickstack/otel-collector-vip 192.168.250.149 fddb:f665:73f7:250::149 none

dolt/dolt is a ClusterIP Service for the in-cluster scraper. It holds no VIP.

Three writers publish the names of the LoadBalancers, each with an A record and an AAAA record:

  • An IngressRoute carries external-dns.alpha.kubernetes.io/target: "192.168.250.145,fddb:f665:73f7:250::145". The traefik-proxy source of external-dns reads targets from this annotation only. An IngressRoute without it publishes nothing. argocd/app-configs/external-dns/values.yaml explains the trap.
  • The external-dns service source reads both addresses from the status of dolt/dolt-vip. The ingress source does the same for Ingress/argocd-server. Traefik copies that status from traefik/traefik-vip, the Service that publishedService.pathOverride names in argocd/cluster-app/templates/traefik.yaml.
  • The curated entry mqtt in dns-control carries address, address6 and ptr: false. A plan and an apply change it, as Entry changes describes.

macOS ranks IPv4 above a ULA AAAA (RFC 6724), so a Mac uses IPv4 unless a tool forces IPv6, for example with curl -6. A glibc client uses the ULA as soon as an AAAA exists.

Tailnet clients resolve the same records. Tailscale split DNS sends fzymgc.house to the resolver pair (tf/tailscale/dns.tf). The two subnet routers and the NAS advertise the routed VIP prefixes (192.168.250.0/24 and fddb:f665:73f7:250::/64). The lists are k8s_tailscale_advertise_routes and nas_tailscale_advertise_routes, which must stay equal. The tailnet policy grants the same access to both VIP prefixes. No part of Lab is on the tailnet.

Guest and Telework are untrusted legs. The UCG-Fiber puts each of them in its own firewall zone, apart from the Internal zone that holds Lab (tf/unifi/firewall-zones.tf). The resolver pair marks both legs trusted: false, so the public recursor process answers them from public DNS only, and the internal records of the VIPs do not reach them. See the client view in the DNS runbook.

  • end0.1000 appears in two places: the netplan VLAN (<default iface>.<tp2_node_lab_vlan_id> in tp2-bootstrap-node) and cilium_devices.
  • The Lab /64 is written once, as index 4 in ula_leg_index in ansible/inventory/group_vars/all.yml. The tp2-bootstrap-node defaults derive from ula_leg_prefix.lab.
  • The static block .140 to .159 sits below the Lab kea_pool in both resolver host_vars.
  • k8s_tailscale_advertise_routes equals nas_tailscale_advertise_routes.
  • The namespace of each LoadBalancer Service appears in the serviceSelector of the pool routed-vips. A Service in another namespace gets no address.

Use this for a new node, or for a node rebuilt from scratch.

  1. Set host_lab_address in the node’s host_vars to 192.168.200.<host>, where <host> is the last octet of the node’s Main address.

  2. Make sure that the uplink of the node’s board holds the port profile TpiUplink in tf/unifi, which tags VLAN 1000.

  3. Run the network tag for that node only:

    Terminal window
    N=tpi-beta-1
    uv run ansible-playbook -i inventory/hosts.yml bootstrap-nodes-playbook.yml --tags network --limit $N --check --diff
    uv run ansible-playbook -i inventory/hosts.yml bootstrap-nodes-playbook.yml --tags network --limit $N
  4. Make sure that the leg is up:

    Terminal window
    ssh fzymgc@$N ip -br addr show end0.1000 # UP, 192.168.200.<host>/22, fddb:f665:73f7:4:aaaa::<host>/64
    ssh fzymgc@$N ip -6 addr show dev end0.1000 # scope global, no tentative, no dadfailed
    ssh fzymgc@$N ip rule # 6000: from 192.168.200.0/22 lookup 1000, 6100: from 192.168.250.0/24 lookup 1001
    ssh fzymgc@$N ip -6 rule # 6000: from fddb:f665:73f7:4::/64 lookup 1000, 6100: from fddb:f665:73f7:250::/64 lookup 1001
    ssh fzymgc@$N ip route show table 1000 # 192.168.200.0/22 scope link, default via 192.168.200.1
    ssh fzymgc@$N ip -6 route show table 1000 # fddb:f665:73f7:4::/64, default via fddb:f665:73f7:4::1
    ssh fzymgc@$N ip route show table 1001 # default via 192.168.200.1 dev end0.1000, nothing else
    ssh fzymgc@$N ip -6 route show table 1001 # default via fddb:f665:73f7:4::1 dev end0.1000, nothing else
    kubectl get node $N # Ready

    tentative is normal for a few seconds after the apply, while duplicate address detection runs. Read again until it clears. dadfailed means that another host holds the address. Stop and find that host.

    The ip rule output also lists Cilium’s rules (9 and 100) and, on the tailnet subnet routers, the tailscale rules (5210 to 5270). If the tailscale rules are missing, run ssh fzymgc@$N sudo systemctl restart tailscaled.

  5. Make sure that Cilium adopted the device. It selects end0.1000 as soon as netplan creates it:

    Terminal window
    p=$(kubectl -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=$N -o name)
    kubectl -n kube-system exec $p -c cilium-agent -- cilium-dbg status --verbose | rg 'Masquerading:' # lists end0.1000

    If the line lacks end0.1000, restart the agents so that each one re-reads its device set:

    Terminal window
    uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium

A new VIP goes in the routed VIP prefixes.

  1. Pick a free host number <n> in the pool routed-vips, from 1 to 253. The IPv4 VIP and the IPv6 VIP share it:

    Terminal window
    kubectl get svc -A -o jsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.metadata.namespace}/{.metadata.name} {.status.loadBalancer.ingress[*].ip}{"\n"}{end}'
  2. If the namespace of the Service is not in the serviceSelector of argocd/app-configs/cilium-lb/ip-pool.yaml, add it.

  3. In the Service, set type: LoadBalancer, loadBalancerClass: io.cilium/bgp-control-plane, ipFamilyPolicy: PreferDualStack, ipFamilies to IPv4 then IPv6, externalTrafficPolicy: Local, and the annotation lbipam.cilium.io/ips: "192.168.250.<n>,fddb:f665:73f7:250::<n>".

  4. Give each name a writer. For a name behind Traefik, put the target annotation on its IngressRoute. For a name that points at the Service itself, put the annotation external-dns.alpha.kubernetes.io/hostname on the Service. Otherwise, create a curated entry in dns-control with address, address6 and ptr: false, as mqtt has.

  5. Add the VIP to the table on this page and to the network reference.

  6. After sync, make sure that Cilium assigned both VIPs, that the gateway routes them, and that the name resolves. The ssh command needs SSH to the console of the UCG-Fiber:

    Terminal window
    kubectl -n <namespace> get svc <name> -o jsonpath='{.status.loadBalancer.ingress[*].ip}{"\n"}' # both VIPs
    ssh root@192.168.40.1 "vtysh -c 'show ip route 192.168.250.<n>/32' -c 'show ipv6 route fddb:f665:73f7:250::<n>/128'"
    dig +short A <name>.fzymgc.house; dig +short AAAA <name>.fzymgc.house

    Each route lists one next hop for each node that holds a ready endpoint of the Service, at the Lab address of that node. A Service with no ready endpoint gets no route.

Run these commands on a Mac on wired Main. The Keycloak discovery document answers without a key. If the Mac accepts tailnet routes, run tailscale set --accept-routes=false first, because the tailnet also carries the VIP prefixes. Set it back after the check.

Terminal window
dig +short AAAA argocd.fzymgc.house # fddb:f665:73f7:250::145
curl -6 -s -o /dev/null -w '%{http_code} %{local_ip} %{remote_ip}\n' \
https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration # 200, a Main ULA, fddb:f665:73f7:250::145
curl -4 -s -o /dev/null -w '%{http_code} %{local_ip} %{remote_ip}\n' \
https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration # 200, a Main address, 192.168.250.145
route -n get 192.168.250.145 # gateway: 192.168.20.1, the UCG-Fiber
route -n get -inet6 fddb:f665:73f7:250::145 # gateway: fe80::76f9:2cff:fed8:fe0c, the UCG-Fiber
arp -an | rg '192\.168\.250\.' # expect no output: the path is routed, not on-link
ndp -an | rg 'fddb:f665:73f7:250:' # expect no output, for the same reason

To list every VIP name with its AAAA record, take the hosts of the target-annotated IngressRoutes and add the names that the other writers publish. Run it from the repository root:

Terminal window
{ rg -l -F 'target: "192.168.250.145,fddb:f665:73f7:250::145"' argocd \
| xargs rg -o --no-filename 'Host(SNI)?\(`[a-z0-9.-]+\.fzymgc\.house`' \
| rg -o '[a-z0-9.-]+\.fzymgc\.house'
echo argocd.fzymgc.house; echo doltdb.fzymgc.house; echo mqtt.fzymgc.house; } \
| sort -u | while read -r n; do printf '%s %s\n' "$n" "$(dig +short AAAA "$n" | tr '\n' ' ')"; done

An IngressRoute host and argocd answer fddb:f665:73f7:250::145. doltdb answers ::147, and mqtt answers ::148. If a new VIP name comes from a writer other than an IngressRoute, add it to the echo list.

A reply from a VIP must leave on end0.1000. The nodes carry no tcpdump, and the leg is nearly silent, so a burst of requests stands out in its packet counters. Read the counters on each node that the gateway lists as a next hop (vtysh -c 'show ip route 192.168.250.145/32' on the UCG-Fiber). The gateway hashes each client to one next hop, so the counters rise on one node only:

Terminal window
N=<a next-hop node of 192.168.250.145/32>
c() { ssh fzymgc@$N ip -s -j link show end0.1000 | jq -r '.[0].stats64 | "\(.rx.packets) \(.tx.packets)"'; }
c; sleep 10; c # baseline: near zero change
for i in $(seq 200); do curl -s -o /dev/null --max-time 3 http://192.168.250.145/; done # from a Main client
c # rx and tx both rise by several packets per request

If tx stays at the baseline, the replies leave on end0, and the gateway holds half-flows. A successful curl on its own proves nothing, because the client accepts a SYN-ACK from the VIP whichever path it took.

These commands do not survive a reboot or a netplan apply. To restore the leg, run the network tag again (Add a node leg, step 3).

This one removes only the IPv6 half and keeps the IPv4 leg. The IPv6 BGP session of the node runs from the removed address, so it goes down, and the gateway stops routing the IPv6 VIPs to the node:

Terminal window
ssh fzymgc@$N 'sudo ip -6 rule del priority 6000; sudo ip -6 route flush table 1000; sudo ip -6 addr del fddb:f665:73f7:4:aaaa::<host>/64 dev end0.1000'

This one deletes the whole leg. It also takes down the IPv4 Lab leg and both BGP sessions of the node, so the gateway stops routing any VIP to the node:

Terminal window
ssh fzymgc@$N 'sudo ip rule del priority 6000; sudo ip -6 rule del priority 6000; sudo ip link del end0.1000'

If the node drops off the network, use its board’s Turing Pi BMC console (tpi uart -n <slot>, see docs/operations/evidence/2026-08-23-tpi-beta-4-console-ring.md).

  • Half-flows in both families: a Lab host that dials a node’s Main address, IPv4 or the Main ULA fddb:f665:73f7:1:aaaa::<host>, arrives through the gateway on end0. The reply is sourced from the Main address, so the Lab rule does not match, and the connected Lab route in main sends it out end0.1000, past the gateway. There is no mirror rule.
  • NodePort frontends also answer on the node Lab addresses. Lab is a trusted segment.
  • The leg ignores router advertisements. An RA adds a SLAAC address that cannot mirror the host number, a GUA from a prefix that changes on ISP re-delegation, and a second default route.