Lab Service VLAN: node legs and LoadBalancer VIPs
Every k3s node carries a tagged leg on the Lab VLAN (VLAN 1000). The routed LoadBalancer VIPs
reach the nodes over that leg in both address families. A client on Main reaches a VIP through the
UCG-Fiber, so the path is routed and not on-link. This page states what exists, where each part is
defined, and the procedures that touch it. kubectl commands assume --context fzymgc-house. Run the Ansible
commands from ansible/.
What exists
Section titled “What exists”The node leg
Section titled “The node leg”Each node has the VLAN interface end0.1000 on the same uplink as Main, which is the VLAN
interface end0.3000. The uplink of the board is a trunk with native Core and tagged VLAN 3000
and 1000 (Turing Pi boards). The leg holds two static addresses:
| Family | Address | Source |
|---|---|---|
| IPv4 | 192.168.200.<host>/22 |
host_lab_address in the node’s host_vars |
| IPv6 | fddb:f665:73f7:4:aaaa::<host>/64 |
tp2_node_lab_ula_prefix plus the last octet of host_lab_address |
<host> is the last octet of the node’s Main address: 141 to 144 for tpi-alpha-1 to
tpi-alpha-4, and 151 to 154 for tpi-beta-1 to tpi-beta-4.
The leg keeps accept-ra: false, so it takes no SLAAC address and no delegated GUA. A
routing-policy rule at priority 6000 in each family sends traffic sourced from the Lab network to
table 1000. That table holds the on-link Lab route and a default route through the Lab gateway
(192.168.200.1 and fddb:f665:73f7:4::1). A reply from a Lab address to a Lab destination
leaves on-link, and a reply to any other destination leaves through the gateway. The netplan
template is ansible/roles/tp2-bootstrap-node/templates/01-netcfg.yaml.j2, and the values are in
that role’s defaults/main.yml.
Cilium manages end0.1000 as a second device (cilium_devices in
ansible/roles/cilium/defaults/main.yml).
The transit leg for BGP
Section titled “The transit leg for BGP”The Lab leg also carries BGP. Each node peers with the UCG-Fiber from its Lab addresses, one
session per family, to 192.168.200.1 and fddb:f665:73f7:4::1. The gateway routes each routed
VIP (192.168.250.0/24 and fddb:f665:73f7:250::/64) to the Lab address of a node that holds an
endpoint of the Service. A request from Main, MainWireless, Core or IoT crosses the gateway into
Lab, and the reply crosses back. See BGP in the UniFi runbook.
A second rule at priority 6100 in each family sends traffic sourced from a routed VIP to table
1001. That table holds only the default route through the Lab gateway, so every reply from a
routed VIP crosses the gateway, a reply to a Lab client too. Table 1000 does not serve here,
because its on-link Lab route sends a reply to a Lab client past the gateway. The values are
tp2_node_vip_route_table and tp2_node_vip_rule_priority in the tp2-bootstrap-node defaults,
and the prefixes are lb_vip_prefix_ipv4 and lb_vip_prefix_ipv6 in
ansible/inventory/group_vars/all.yml.
The address plan
Section titled “The address plan”192.168.200.140 to .159 is static space. The node legs hold
.141 to .144 and .151 to .154. .140, .145 to .149 and .155 to .159 are free. The Lab Kea
pool starts at .160 (kea_pool on the lab entry in both resolver host_vars). The IPv6 suffixes mirror the
IPv4 host octets under fddb:f665:73f7:4:aaaa::. The network reference
lists every address.
The LoadBalancers
Section titled “The LoadBalancers”Each LoadBalancer Service is PreferDualStack with IPv4 first. It runs
externalTrafficPolicy: Local, so that the backend sees the client address. It sets
loadBalancerClass: io.cilium/bgp-control-plane and pins both routed VIPs with one host number
through lbipam.cilium.io/ips. Cilium LB-IPAM assigns them from the pool routed-vips
(argocd/app-configs/cilium-lb/ip-pool.yaml). The pool serves only the namespaces that its
serviceSelector lists, and Cilium assigns no address to a Service without the class.
Kubernetes refuses a change of loadBalancerClass on an existing LoadBalancer Service. To move a
Service to another class, add a second Service with the same selector and ports, move every DNS
writer to it, and then delete the first Service.
| Service | IPv4 VIP | IPv6 VIP | DNS writer |
|---|---|---|---|
traefik/traefik-vip |
192.168.250.145 |
fddb:f665:73f7:250::145 |
the target annotation on every IngressRoute |
temporal/temporal-node-gateway-vip |
192.168.250.146 |
fddb:f665:73f7:250::146 |
none. The resolver Pis dial the IPv4 literal |
dolt/dolt-vip |
192.168.250.147 |
fddb:f665:73f7:250::147 |
the external-dns service source, doltdb.fzymgc.house |
mosquitto/mosquitto-vip |
192.168.250.148 |
fddb:f665:73f7:250::148 |
the curated entry mqtt in dns-control |
clickstack/otel-collector-vip |
192.168.250.149 |
fddb:f665:73f7:250::149 |
none |
dolt/dolt is a ClusterIP Service for the in-cluster scraper. It holds no VIP.
DNS records
Section titled “DNS records”Three writers publish the names of the LoadBalancers, each with an A record and an AAAA record:
- An IngressRoute carries
external-dns.alpha.kubernetes.io/target: "192.168.250.145,fddb:f665:73f7:250::145". Thetraefik-proxysource of external-dns reads targets from this annotation only. An IngressRoute without it publishes nothing.argocd/app-configs/external-dns/values.yamlexplains the trap. - The external-dns
servicesource reads both addresses from the status ofdolt/dolt-vip. Theingresssource does the same forIngress/argocd-server. Traefik copies that status fromtraefik/traefik-vip, the Service thatpublishedService.pathOverridenames inargocd/cluster-app/templates/traefik.yaml. - The curated entry
mqttin dns-control carriesaddress,address6andptr: false. A plan and an apply change it, as Entry changes describes.
macOS ranks IPv4 above a ULA AAAA (RFC 6724), so a Mac uses IPv4 unless a tool forces IPv6, for
example with curl -6. A glibc client uses the ULA as soon as an AAAA exists.
Tailnet clients resolve the same records. Tailscale split DNS sends fzymgc.house to the resolver
pair (tf/tailscale/dns.tf). The two subnet routers and the NAS advertise the routed VIP
prefixes (192.168.250.0/24 and fddb:f665:73f7:250::/64). The lists are
k8s_tailscale_advertise_routes and nas_tailscale_advertise_routes, which must stay equal. The
tailnet policy grants the same access to both VIP prefixes. No part of Lab is on the tailnet.
Guest and Telework are untrusted legs. The UCG-Fiber puts each of them in its own firewall zone,
apart from the Internal zone that holds Lab (tf/unifi/firewall-zones.tf). The resolver pair
marks both legs trusted: false, so the public recursor process answers them from public DNS
only, and
the internal records of the VIPs do not reach them. See
the client view
in the DNS runbook.
Names that must match
Section titled “Names that must match”end0.1000appears in two places: the netplan VLAN (<default iface>.<tp2_node_lab_vlan_id>intp2-bootstrap-node) andcilium_devices.- The Lab
/64is written once, as index4inula_leg_indexinansible/inventory/group_vars/all.yml. Thetp2-bootstrap-nodedefaults derive fromula_leg_prefix.lab. - The static block
.140to.159sits below the Labkea_poolin both resolver host_vars. k8s_tailscale_advertise_routesequalsnas_tailscale_advertise_routes.- The namespace of each LoadBalancer Service appears in the
serviceSelectorof the poolrouted-vips. A Service in another namespace gets no address.
Procedures
Section titled “Procedures”Add a node leg
Section titled “Add a node leg”Use this for a new node, or for a node rebuilt from scratch.
-
Set
host_lab_addressin the node’s host_vars to192.168.200.<host>, where<host>is the last octet of the node’s Main address. -
Make sure that the uplink of the node’s board holds the port profile
TpiUplinkintf/unifi, which tags VLAN 1000. -
Run the network tag for that node only:
Terminal window N=tpi-beta-1uv run ansible-playbook -i inventory/hosts.yml bootstrap-nodes-playbook.yml --tags network --limit $N --check --diffuv run ansible-playbook -i inventory/hosts.yml bootstrap-nodes-playbook.yml --tags network --limit $N -
Make sure that the leg is up:
Terminal window ssh fzymgc@$N ip -br addr show end0.1000 # UP, 192.168.200.<host>/22, fddb:f665:73f7:4:aaaa::<host>/64ssh fzymgc@$N ip -6 addr show dev end0.1000 # scope global, no tentative, no dadfailedssh fzymgc@$N ip rule # 6000: from 192.168.200.0/22 lookup 1000, 6100: from 192.168.250.0/24 lookup 1001ssh fzymgc@$N ip -6 rule # 6000: from fddb:f665:73f7:4::/64 lookup 1000, 6100: from fddb:f665:73f7:250::/64 lookup 1001ssh fzymgc@$N ip route show table 1000 # 192.168.200.0/22 scope link, default via 192.168.200.1ssh fzymgc@$N ip -6 route show table 1000 # fddb:f665:73f7:4::/64, default via fddb:f665:73f7:4::1ssh fzymgc@$N ip route show table 1001 # default via 192.168.200.1 dev end0.1000, nothing elsessh fzymgc@$N ip -6 route show table 1001 # default via fddb:f665:73f7:4::1 dev end0.1000, nothing elsekubectl get node $N # Readytentativeis normal for a few seconds after the apply, while duplicate address detection runs. Read again until it clears.dadfailedmeans that another host holds the address. Stop and find that host.The
ip ruleoutput also lists Cilium’s rules (9 and 100) and, on the tailnet subnet routers, the tailscale rules (5210 to 5270). If the tailscale rules are missing, runssh fzymgc@$N sudo systemctl restart tailscaled. -
Make sure that Cilium adopted the device. It selects
end0.1000as soon as netplan creates it:Terminal window p=$(kubectl -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=$N -o name)kubectl -n kube-system exec $p -c cilium-agent -- cilium-dbg status --verbose | rg 'Masquerading:' # lists end0.1000If the line lacks
end0.1000, restart the agents so that each one re-reads its device set:Terminal window uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium
Add a dual-stack LoadBalancer VIP
Section titled “Add a dual-stack LoadBalancer VIP”A new VIP goes in the routed VIP prefixes.
-
Pick a free host number
<n>in the poolrouted-vips, from1to253. The IPv4 VIP and the IPv6 VIP share it:Terminal window kubectl get svc -A -o jsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.metadata.namespace}/{.metadata.name} {.status.loadBalancer.ingress[*].ip}{"\n"}{end}' -
If the namespace of the Service is not in the
serviceSelectorofargocd/app-configs/cilium-lb/ip-pool.yaml, add it. -
In the Service, set
type: LoadBalancer,loadBalancerClass: io.cilium/bgp-control-plane,ipFamilyPolicy: PreferDualStack,ipFamiliestoIPv4thenIPv6,externalTrafficPolicy: Local, and the annotationlbipam.cilium.io/ips: "192.168.250.<n>,fddb:f665:73f7:250::<n>". -
Give each name a writer. For a name behind Traefik, put the target annotation on its IngressRoute. For a name that points at the Service itself, put the annotation
external-dns.alpha.kubernetes.io/hostnameon the Service. Otherwise, create a curated entry in dns-control withaddress,address6andptr: false, asmqtthas. -
Add the VIP to the table on this page and to the network reference.
-
After sync, make sure that Cilium assigned both VIPs, that the gateway routes them, and that the name resolves. The
sshcommand needs SSH to the console of the UCG-Fiber:Terminal window kubectl -n <namespace> get svc <name> -o jsonpath='{.status.loadBalancer.ingress[*].ip}{"\n"}' # both VIPsssh root@192.168.40.1 "vtysh -c 'show ip route 192.168.250.<n>/32' -c 'show ipv6 route fddb:f665:73f7:250::<n>/128'"dig +short A <name>.fzymgc.house; dig +short AAAA <name>.fzymgc.houseEach route lists one next hop for each node that holds a ready endpoint of the Service, at the Lab address of that node. A Service with no ready endpoint gets no route.
Check the path from Main
Section titled “Check the path from Main”Run these commands on a Mac on wired Main. The Keycloak discovery document answers without a key.
If the Mac accepts tailnet routes, run tailscale set --accept-routes=false first, because the
tailnet also carries the VIP prefixes. Set it back after the check.
dig +short AAAA argocd.fzymgc.house # fddb:f665:73f7:250::145curl -6 -s -o /dev/null -w '%{http_code} %{local_ip} %{remote_ip}\n' \ https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration # 200, a Main ULA, fddb:f665:73f7:250::145curl -4 -s -o /dev/null -w '%{http_code} %{local_ip} %{remote_ip}\n' \ https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration # 200, a Main address, 192.168.250.145route -n get 192.168.250.145 # gateway: 192.168.20.1, the UCG-Fiberroute -n get -inet6 fddb:f665:73f7:250::145 # gateway: fe80::76f9:2cff:fed8:fe0c, the UCG-Fiberarp -an | rg '192\.168\.250\.' # expect no output: the path is routed, not on-linkndp -an | rg 'fddb:f665:73f7:250:' # expect no output, for the same reasonTo list every VIP name with its AAAA record, take the hosts of the target-annotated IngressRoutes and add the names that the other writers publish. Run it from the repository root:
{ rg -l -F 'target: "192.168.250.145,fddb:f665:73f7:250::145"' argocd \ | xargs rg -o --no-filename 'Host(SNI)?\(`[a-z0-9.-]+\.fzymgc\.house`' \ | rg -o '[a-z0-9.-]+\.fzymgc\.house' echo argocd.fzymgc.house; echo doltdb.fzymgc.house; echo mqtt.fzymgc.house; } \ | sort -u | while read -r n; do printf '%s %s\n' "$n" "$(dig +short AAAA "$n" | tr '\n' ' ')"; doneAn IngressRoute host and argocd answer fddb:f665:73f7:250::145. doltdb answers ::147,
and mqtt answers ::148. If a new VIP name comes from a writer other than an IngressRoute, add
it to the echo list.
Check the reply path on a node
Section titled “Check the reply path on a node”A reply from a VIP must leave on end0.1000. The nodes carry no tcpdump, and the leg is nearly
silent, so a burst of requests stands out in its packet counters. Read the counters on each node
that the gateway lists as a next hop (vtysh -c 'show ip route 192.168.250.145/32' on the
UCG-Fiber). The gateway hashes each client to one next hop, so the counters rise on one node only:
N=<a next-hop node of 192.168.250.145/32>c() { ssh fzymgc@$N ip -s -j link show end0.1000 | jq -r '.[0].stats64 | "\(.rx.packets) \(.tx.packets)"'; }c; sleep 10; c # baseline: near zero changefor i in $(seq 200); do curl -s -o /dev/null --max-time 3 http://192.168.250.145/; done # from a Main clientc # rx and tx both rise by several packets per requestIf tx stays at the baseline, the replies leave on end0, and the gateway holds half-flows. A
successful curl on its own proves nothing, because the client accepts a SYN-ACK from the VIP
whichever path it took.
Take a leg down without netplan
Section titled “Take a leg down without netplan”These commands do not survive a reboot or a netplan apply. To restore the leg, run the network tag again (Add a node leg, step 3).
This one removes only the IPv6 half and keeps the IPv4 leg. The IPv6 BGP session of the node runs from the removed address, so it goes down, and the gateway stops routing the IPv6 VIPs to the node:
ssh fzymgc@$N 'sudo ip -6 rule del priority 6000; sudo ip -6 route flush table 1000; sudo ip -6 addr del fddb:f665:73f7:4:aaaa::<host>/64 dev end0.1000'This one deletes the whole leg. It also takes down the IPv4 Lab leg and both BGP sessions of the node, so the gateway stops routing any VIP to the node:
ssh fzymgc@$N 'sudo ip rule del priority 6000; sudo ip -6 rule del priority 6000; sudo ip link del end0.1000'If the node drops off the network, use its board’s Turing Pi BMC console (tpi uart -n <slot>,
see docs/operations/evidence/2026-08-23-tpi-beta-4-console-ring.md).
Known limitations
Section titled “Known limitations”- Half-flows in both families: a Lab host that dials a node’s Main address, IPv4 or the Main ULA
fddb:f665:73f7:1:aaaa::<host>, arrives through the gateway onend0. The reply is sourced from the Main address, so the Lab rule does not match, and the connected Lab route inmainsends it outend0.1000, past the gateway. There is no mirror rule. - NodePort frontends also answer on the node Lab addresses. Lab is a trusted segment.
- The leg ignores router advertisements. An RA adds a SLAAC address that cannot mirror the host number, a GUA from a prefix that changes on ISP re-delegation, and a second default route.