Design: LoadBalancer VIPs over BGP
Context
Section titled “Context”See openspec/changes/lb-vips-over-bgp/proposal.md, section Why, for the motivation. The current state that shapes the approach:
docs/reference/network.mdis the addressing authority. Five LoadBalancer Services hold Lab VIPs from the MetalLBvlan-1000pool:traefik/traefik-lab(.145),temporal/temporal-node-gateway(.146),dolt/dolt(.147),mosquitto/mosquitto-lab(.148) andclickstack/otel-collector-lb(.149), each with the matchingfddb:f665:73f7:4:aaaa::address. Each one isPreferDualStack,externalTrafficPolicy: Local, and pins its addresses withmetallb.io/loadBalancerIPs.- MetalLB v0.16.1 runs in L2 mode from
argocd/cluster-app/templates/metallb.yamlandargocd/app-configs/metallb/. Itsvlan-1000L2Advertisementis pinned toend0.1000and excludes nodes labelledfzymgc.house/tailnet-subnet-router. - Each node has a Main address on
end0(192.168.20.<host>) and a Lab leg onend0.1000(192.168.200.<host>andfddb:f665:73f7:4:aaaa::<host>). The rule at priority 6000 sends traffic from the Lab/22and the Lab/64to table 1000. That table holds the on-link Lab route and a default route through the Lab gateway in each family. The template isansible/roles/tp2-bootstrap-node/templates/01-netcfg.yaml.j2. - Cilium 1.20.1 runs from
ansible/roles/ciliumwith kube-proxy replacement, eBPF host routing anddevices: [end0, end0.1000]. The BGP control plane is off. LB-IPAM is on by default, and noCiliumLoadBalancerIPPoolexists, so it assigns nothing today. - The UCG-Fiber holds
192.168.200.1andfddb:f665:73f7:4::1on Lab. Its controller collectionbgp-configholds no object (tf/unifi/inventory-summary.json). The fork release0.56.0-fzymgc.11hasunifi_bgp. - The resolver Pis have one default route, on Core. They dial the Temporal node gateway at the IPv4
literal
192.168.200.146, and verify its certificate by the nametemporal-nodes.fzymgc.house. The network policy of the gateway admits192.168.200.6/32and192.168.200.7/32only. - Twenty Traefik routes carry
external-dns.alpha.kubernetes.io/targetwith the Traefik Lab VIPs. external-dns readsdoltdbfrom the status ofdolt/dolt, andargocdfrom the Ingress status that Traefik copies fromtraefik/traefik-lab. The curatedmqttentry lives in the dns-control database. - No repository and no runbook names a consumer of
otel-collector-lb.docs/operations/clickstack.mdsends external OTLP throughotel-gateway.fzymgc.houseinstead.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- Move every LoadBalancer VIP to a routed-only prefix that the UCG-Fiber learns over BGP, with no outage of any name and no outage of the Pi workers.
- Keep the reply of every node on the path through the gateway, so the gateway sees both directions of each flow.
- Remove MetalLB and every workaround that exists only for L2 announcement.
Non-Goals:
- The API VIP
192.168.20.140. kube-vip keeps it, with ARP. D13 reserves its routed address for a follow-on change. - The removal of the Lab leg. It stays, as the transit leg for BGP.
- BFD, TCP-MD5 on the sessions, and the announcement of pod CIDRs or ClusterIPs.
- A test of a tailnet path. Sean ruled on 2026-09-23 that the tailnet proof is the configuration of the advertisers and of the policy.
- The retirement of the OTLP LoadBalancer. Its twin is the canary, and it stays.
Decisions
Section titled “Decisions”D1. Cilium BGP control plane and LB-IPAM, with the default IPAM off
Section titled “D1. Cilium BGP control plane and LB-IPAM, with the default IPAM off”The cilium role gains two variables, cilium_bgp_control_plane_enabled: true and
cilium_default_lb_service_ipam: none. They render as bgpControlPlane.enabled and
defaultLBServiceIPAM in templates/values.yaml.j2. The role already restarts the agents after
a values change.
With defaultLBServiceIPAM: none, LB-IPAM assigns an address only to a Service with a Cilium
class. MetalLB assigns only to a Service without a class. So the two never assign the same
Service, and the converge must run before the pool exists.
Alternatives: MetalLB in BGP mode keeps two components, and Sean rejected it on 2026-09-28. Cilium
L2 announcements keep the VIPs on-link, are Beta, and do not support externalTrafficPolicy: Local.
D2. One session per family from the Lab leg of each node
Section titled “D2. One session per family from the Lab leg of each node”Sean chose the Lab leg on 2026-10-04. Each node peers with 192.168.200.1 for IPv4 and with
fddb:f665:73f7:4::1 for IPv6. The kernel picks the source of each session from the connected Lab
route, so each session runs from the Lab address of the node in its family, and BGP sets that
address as the next hop. A separate session per family gives each family a native next hop.
The gateway forwards a request from Main, MainWireless, Core or IoT out of its Lab interface, and the reply comes back in on Lab. No interface sees a request leave the way it came in. Peering over Main was rejected because wired Main clients, the workstation and the NAS among them, are on-link with the Main addresses of the nodes.
D3. A reply table for the VIP prefixes, separate from table 1000
Section titled “D3. A reply table for the VIP prefixes, separate from table 1000”tp2-bootstrap-node adds table 1001 (tp2_node_vip_route_table) with one route per family: the
default route through the Lab gateway on end0.1000. Two rules at priority 6100
(tp2_node_vip_rule_priority) send traffic from 192.168.250.0/24 and from
fddb:f665:73f7:250::/64 to that table. The existing Lab rules at 6000 cover the Lab /22 and
/64, which do not overlap the VIP prefixes.
Table 1000 was rejected for these rules. It holds the on-link Lab route, so a reply to a Lab client leaves on-link, and the gateway sees only the request. With table 1001, the reply to a Lab client also goes through the gateway. The Lab client proof in PR 1 measures this path, because the gateway can send an ICMP redirect for the request of a Lab client.
The existing Lab VIPs prove the mechanism. A reply from 192.168.200.145 follows the rule at 6000
today, and docs/operations/lab-service-vlan.md measures it on the counters of end0.1000.
D4. The VIP prefixes and the pool
Section titled “D4. The VIP prefixes and the pool”Sean chose 192.168.250.0/24 and fddb:f665:73f7:250::/64 on 2026-10-04. PR 1 allocates both
in docs/reference/network.md before any consumer uses them. The /64 is a routed prefix, not
a leg, so it takes no index in ula_leg_index. A new leg still takes index 8.
ansible/inventory/group_vars/all.yml gains lb_vip_prefix_ipv4: 192.168.250.0/24 and
lb_vip_prefix_ipv6: "{{ ula_site_prefix }}:250::/64". tp2-bootstrap-node, k8s-tailscale and
nas-tailscale read them.
Each Service keeps its host number in both families: Traefik .145, the Temporal node gateway
.146, Dolt .147, Mosquitto .148, the OTLP collector .149. One
CiliumLoadBalancerIPPool, routed-vips, holds one range per family: 192.168.250.1 to
.253, and fddb:f665:73f7:250::1 to ::253. The IPv6 range holds every suffix that mirrors an
IPv4 host number. Each range stops below .254, which D13 reserves. Each twin pins its two
addresses with lbipam.cilium.io/ips.
The pool serves only Services in the namespaces that hold a twin: clickstack, dolt,
mosquitto, temporal and traefik. Its serviceSelector matches
io.kubernetes.service.namespace. Sean chose the list on 2026-10-04, after a security review
found that an open pool gives a routed VIP to any Service that sets the class. A new routed Service
in another namespace needs one line in the pool.
D5. The FRR file on the gateway
Section titled “D5. The FRR file on the gateway”tf/unifi/bgp.tf declares one unifi_bgp in raw mode (config, enabled = true,
upload_file_name, description). Structured mode was rejected: its template
redistributes the connected routes of the gateway and permits every route in both directions.
The file has this shape:
frr defaults traditional!ip prefix-list LB-VIPS seq 10 permit 192.168.250.0/24 ge 32ipv6 prefix-list LB-VIPS6 seq 10 permit fddb:f665:73f7:250::/64 ge 128route-map CILIUM-IN permit 10 match ip address prefix-list LB-VIPSroute-map CILIUM-IN6 permit 10 match ipv6 address prefix-list LB-VIPS6route-map DENY-ALL deny 10!router bgp 65000 bgp router-id 192.168.200.1 bgp log-neighbor-changes bgp graceful-restart no bgp default ipv4-unicast neighbor CILIUM4 peer-group neighbor CILIUM4 remote-as 65001 neighbor CILIUM4 timers 3 9 neighbor CILIUM4 passive neighbor CILIUM6 peer-group neighbor CILIUM6 remote-as 65001 neighbor CILIUM6 timers 3 9 neighbor CILIUM6 passive neighbor 192.168.200.141 peer-group CILIUM4 neighbor fddb:f665:73f7:4:aaaa::141 peer-group CILIUM6 ! the same pair for each node, .141 to .144 and .151 to .154 address-family ipv4 unicast maximum-paths 8 neighbor CILIUM4 activate neighbor CILIUM4 route-map CILIUM-IN in neighbor CILIUM4 route-map DENY-ALL out neighbor CILIUM4 soft-reconfiguration inbound neighbor CILIUM4 maximum-prefix 256 exit-address-family address-family ipv6 unicast maximum-paths 8 neighbor CILIUM6 activate neighbor CILIUM6 route-map CILIUM-IN6 in neighbor CILIUM6 route-map DENY-ALL out neighbor CILIUM6 soft-reconfiguration inbound neighbor CILIUM6 maximum-prefix 256 exit-address-family- The gateway is AS 65000 and the cluster is AS 65001, both private.
- The gateway peers with the Lab addresses of the nodes and with no other host. Each node is a
passive neighbor in each family, so the gateway waits for the node to connect.
bgp.tfbuilds the lines from one list of host octets,bgp_node_hosts. Sean chose exact neighbors over listen ranges on 2026-10-04. With a listen range, a Lab host can set a free static address in the range, peer, and announce a VIP.show bgp summaryalso lists a node that is down. - The import route-maps accept host routes inside the VIP prefixes and nothing else. The export route-map sends nothing to the nodes.
maximum-paths 8installs one next hop per announcing node.
The wire struct is unifi.BGPConfig in unifi/bgp_config.generated.go of the go-unifi fork. It
models frr_bgpd_config, description, enabled and uploaded_file_name, and the controller
keys _id, site_id, attr_hidden, attr_hidden_id, attr_no_delete and attr_no_edit. The
provider sets all four writable fields in modelToBGP (unifi/bgp_resource.go). An update posts
the whole object to v2/api/site/<site>/bgp/config, the same call as a create, so a key the
struct does not model is lost on the first update. The collection holds no object today, so the
field audit runs after the first apply, and it must print no gap for bgp-config before any
later change to bgp.tf.
D6. The Cilium objects live in a new ArgoCD app, cilium-lb
Section titled “D6. The Cilium objects live in a new ArgoCD app, cilium-lb”argocd/app-configs/cilium-lb/ holds the pool of D4 and four BGP objects, and
argocd/cluster-app/templates/cilium-lb.yaml syncs it:
CiliumBGPClusterConfigucg-fiber: no node selector, so every node runs the instance. One instance withlocalASN: 65001and two peers,ucg-fiber-v4at192.168.200.1anducg-fiber-v6atfddb:f665:73f7:4::1, both withpeerASN: 65000.- Two
CiliumBGPPeerConfigobjects, one per family. Each setsholdTimeSeconds: 9,keepAliveTimeSeconds: 3, graceful restart withrestartTimeSeconds: 60, its one family, and the advertisement label. CiliumBGPAdvertisementlb-vips:advertisementType: Servicewithaddresses: [LoadBalancerIP], and a selector that matches every Service.
The selector matches every Service, so while MetalLB exists the nodes also announce the Lab VIPs of the MetalLB Services. The import route-map of D5 drops them. A label on each Service was rejected, because every future LoadBalancer Service has the class anyway.
The CRDs exist only after the converge of D1. The operator runs that converge from the branch of PR 1 before the merge, so the first sync of the app succeeds.
D7. Each Service moves as a twin
Section titled “D7. Each Service moves as a twin”Sean chose twins on 2026-10-04. Kubernetes rejects a change of spec.loadBalancerClass on an
existing LoadBalancer Service, and the Traefik values already warn against a change of address in
place. Each twin copies the selector, the ports, externalTrafficPolicy: Local and the dual-stack
fields of the old Service. It adds loadBalancerClass: io.cilium/bgp-control-plane and
lbipam.cilium.io/ips.
| Old Service | Twin | VIPs |
|---|---|---|
traefik/traefik-lab |
traefik/traefik-vip (chart key additionalServices.vip) |
192.168.250.145, fddb:f665:73f7:250::145 |
temporal/temporal-node-gateway |
temporal/temporal-node-gateway-vip |
.146, ::146 |
dolt/dolt |
dolt/dolt-vip |
.147, ::147 |
mosquitto/mosquitto-lab |
mosquitto/mosquitto-vip |
.148, ::148 |
clickstack/otel-collector-lb |
clickstack/otel-collector-vip |
.149, ::149 |
PR 3 deletes the old Services, except dolt. dolt.dolt.svc is a scrape target in
argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml, so dolt becomes a
ClusterIP Service and loses externalTrafficPolicy and the MetalLB annotation.
D8. The DNS writers move in PR 2
Section titled “D8. The DNS writers move in PR 2”- The 20 target annotations change to
"192.168.250.145,fddb:f665:73f7:250::145". providers.kubernetesIngress.publishedService.pathOverrideinargocd/cluster-app/templates/traefik.yamlchanges totraefik/traefik-vip, soargocdfollows.- The
external-dns.alpha.kubernetes.io/hostnameannotation moves fromdolttodolt-vip. - The operator changes the curated
mqttentry to192.168.250.148andfddb:f665:73f7:250::148in dns-control. The entry keepsptr: false.
D9. The Pi workers and the Temporal gateway policy
Section titled “D9. The Pi workers and the Temporal gateway policy”dns_node_worker_gateway_addr changes to "192.168.250.146:7233". A Pi sends to the routed
prefix through its default route on Core, so the gateway sees 192.168.40.6 and 192.168.40.7.
In PR 2 the policy admits the Core addresses beside the Lab addresses, because both Services
select the same pods. PR 3 drops the Lab addresses. The certificate needs no change, because the
worker verifies temporal-nodes.fzymgc.house and not the address.
D10. The tailnet follows the prefixes in three steps
Section titled “D10. The tailnet follows the prefixes in three steps”- PR 1 adds
lb_vip_prefix_ipv4andlb_vip_prefix_ipv6tok8s_tailscale_advertise_routesandnas_tailscale_advertise_routes. It adds both toautoApprovers.routesintailscale/policy.hujson, with the host aliaseslb-vipsandlb-vips-v6and the matching entries in the admin grant. The operator converges all three advertisers in one session, so every advertiser announces the same set. - PR 2 points
auth-fzymgcandauth-fzymgc-v6at the Traefik twin. - PR 3 removes
192.168.200.128/27,fddb:f665:73f7:4:aaaa::/80,network-labandnetwork-lab-v6, and converges the advertisers again.
A subnet router sends VIP traffic out of end0 through its default route, and Tailscale rewrites
the source to the Main address of the router. That address is inside 192.168.16.0/20, and its
Main ULA is inside fddb:f665:73f7:1::/64. Both lists in
argocd/app-configs/keycloak/ingress-route.yaml already trust those ranges, so PR 3 removes the
four Lab entries of the routers from both lists together.
D11. The fence of the untrusted legs is measured before the move
Section titled “D11. The fence of the untrusted legs is measured before the move”The UniFi documentation does not state which zone a BGP-learned destination falls in. The
canary of PR 1 answers it: a throwaway client on Guest and one on Telework (the
nas-throwaway-lxc skill) try the canary VIP in both families. Both attempts must fail before PR
1 merges. If one connects, PR 1 gains a unifi_firewall_policy in
tf/unifi/firewall-zones.tf that blocks the two VIP prefixes from that zone, and the probe runs
again.
D12. MetalLB goes in PR 3, after the old VIPs are idle
Section titled “D12. MetalLB goes in PR 3, after the old VIPs are idle”PR 3 deletes argocd/cluster-app/templates/metallb.yaml and argocd/app-configs/metallb/.
ArgoCD prunes the chart objects. The CRDs carry helm.sh/resource-policy: keep, and the namespace
came from CreateNamespace=true, so the operator deletes both. PR 3 starts only after no node
holds a Cilium connection-tracking entry for an old VIP (task 3.1). That gate finds a device that
uses an old address as a literal, an MQTT client on IoT for example.
D13. The routed API VIP is reserved, and a follow-on change moves it
Section titled “D13. The routed API VIP is reserved, and a follow-on change moves it”192.168.250.254 and fddb:f665:73f7:250::254 are reserved for the API VIP in
docs/reference/network.md. No Service pins them. A follow-on change, api-vip-over-bgp, moves
the API VIP off Main after a study of the cold start, and the reserved address is one of its
candidates. This change already gives the address a route: the import filter of D5 accepts it,
the reply rule of D3 covers it, and the tailnet advertises its prefix.
The follow-on is a separate change for three reasons:
- Cilium reaches the API server through
cilium_k8s_service_host, the kube-vip address. If Cilium announces the API VIP, a cold start of the cluster deadlocks. So Cilium’s own path to the API server must move to a node-local endpoint first. - The control-plane nodes carry no taint, so they hold LoadBalancer endpoints, and Cilium peers from them. A kube-vip BGP speaker on the same node opens a second session from the same Lab address to the same gateway address. Peering kube-vip over Main brings back the hairpin of D2.
- The new address needs a
tls-sanentry in k3s, and every consumer of192.168.20.140must move: the agent join URL,scripts/k8s-login.sh, the kubeconfigs outside the cluster and the Kubernetes auth of Vault.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [The gateway sends an ICMP redirect for the request of a Lab client, and the client then bypasses the gateway in one direction] → The Lab client proof of PR 1 measures the path. No repository consumer of a VIP sits on Lab.
- [A BGP-learned destination falls in a zone that the fence does not cover] → D11. The probe gates PR 1.
- [A Lab host takes the address of a node that is down, peers, and announces a VIP] → The gateway peers only with the exact node addresses, so the host must take a node address while that node is off. The import filter still accepts only VIP host routes. Lab is a trusted leg. TCP-MD5 on the sessions closes the rest, at the cost of a password in the FRR text and in the workspace state, and it is not part of this change.
- [A Service in another namespace sets the class and takes a routed VIP] → The pool serves only the five listed namespaces (D4).
- [A node crashes, and graceful restart keeps its routes for up to 60 seconds] → Accepted. Without graceful restart, a restart of a Cilium agent withdraws every route of the node, and a Service with one replica goes dark for the length of the restart.
- [A later update of
bgp.tfdeletes a key that the controller adds] → The field audit after the first apply (task 1.13). - [A client keeps an old VIP as a literal] → The twins keep the old VIPs answering until PR 3, and the connection-tracking gate (task 3.1) finds the client first.
- [A change of next hops rehashes ECMP and resets a long-lived connection] → Accepted. Next hops change only when an endpoint moves.
- [The agent restart in PR 1 interrupts the datapath on each node in turn] → It is the same rolling restart as every Cilium converge.
Migration Plan
Section titled “Migration Plan”Each step names its rollback. A revert pull request merges like any other, and HCP Terraform
applies the tf/ part of it on merge.
PR 1, converge and canary:
- The operator converges the node rules one node at a time:
uv run ansible-playbook -i inventory/hosts.yml bootstrap-nodes-playbook.yml --tags network --limit $N. Rollback on a node:ssh fzymgc@$N 'sudo ip rule del priority 6100; sudo ip -6 rule del priority 6100'. - The operator converges Cilium from the branch:
uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium. Rollback: check outorigin/mainand run the same command. - The merge syncs
cilium-lband the canary, and HCP Terraform appliesunifi_bgp. Rollback:git revert -m 1 <merge commit of PR 1>, merged as a pull request. ArgoCD prunes the objects, the routes withdraw, and the apply deletes the gateway configuration. Then run step 2 again fromorigin/main. - The operator converges the advertisers:
uv run ansible-playbook -i inventory/hosts.yml tailscale-subnet-routers-playbook.ymlandscripts/nas-playbook.sh --tags nas-tailscale. Rollback: the same two commands after the revert.
PR 2, the move:
- The merge syncs the twins and the DNS writers. The operator changes the
mqttentry and converges the Pi workers one at a time:uv run ansible-playbook -i inventory/hosts.yml dns-playbook.yml --tags dns-node-worker --limit sandiego, then--limit portland. - Rollback:
git revert -m 1 <merge commit of PR 2>, merged as a pull request. The DNS writers return to the Lab VIPs, which still answer. The operator setsmqttback and runs the dns-node-worker command again on both Pis.
PR 3, teardown:
- The merge prunes MetalLB and the old Services. The operator deletes the MetalLB CRDs and namespace and converges the advertisers.
- Rollback:
git revert -m 1 <merge commit of PR 3>, merged as a pull request. The chart installs the CRDs again with server-side apply, and the old Services and their L2 announcements return. The operator converges the advertisers again. A rollback of BGP itself after PR 3 needs the revert of PR 2 first.
Open Questions
Section titled “Open Questions”- The restart time of 60 seconds for graceful restart is a first value. The operator can tune it after the restart of one agent is timed. The value changes no spec and no task.