Skip to content

Design: LoadBalancer VIPs over BGP

See openspec/changes/lb-vips-over-bgp/proposal.md, section Why, for the motivation. The current state that shapes the approach:

  • docs/reference/network.md is the addressing authority. Five LoadBalancer Services hold Lab VIPs from the MetalLB vlan-1000 pool: traefik/traefik-lab (.145), temporal/temporal-node-gateway (.146), dolt/dolt (.147), mosquitto/mosquitto-lab (.148) and clickstack/otel-collector-lb (.149), each with the matching fddb:f665:73f7:4:aaaa:: address. Each one is PreferDualStack, externalTrafficPolicy: Local, and pins its addresses with metallb.io/loadBalancerIPs.
  • MetalLB v0.16.1 runs in L2 mode from argocd/cluster-app/templates/metallb.yaml and argocd/app-configs/metallb/. Its vlan-1000 L2Advertisement is pinned to end0.1000 and excludes nodes labelled fzymgc.house/tailnet-subnet-router.
  • Each node has a Main address on end0 (192.168.20.<host>) and a Lab leg on end0.1000 (192.168.200.<host> and fddb:f665:73f7:4:aaaa::<host>). The rule at priority 6000 sends traffic from the Lab /22 and the Lab /64 to table 1000. That table holds the on-link Lab route and a default route through the Lab gateway in each family. The template is ansible/roles/tp2-bootstrap-node/templates/01-netcfg.yaml.j2.
  • Cilium 1.20.1 runs from ansible/roles/cilium with kube-proxy replacement, eBPF host routing and devices: [end0, end0.1000]. The BGP control plane is off. LB-IPAM is on by default, and no CiliumLoadBalancerIPPool exists, so it assigns nothing today.
  • The UCG-Fiber holds 192.168.200.1 and fddb:f665:73f7:4::1 on Lab. Its controller collection bgp-config holds no object (tf/unifi/inventory-summary.json). The fork release 0.56.0-fzymgc.11 has unifi_bgp.
  • The resolver Pis have one default route, on Core. They dial the Temporal node gateway at the IPv4 literal 192.168.200.146, and verify its certificate by the name temporal-nodes.fzymgc.house. The network policy of the gateway admits 192.168.200.6/32 and 192.168.200.7/32 only.
  • Twenty Traefik routes carry external-dns.alpha.kubernetes.io/target with the Traefik Lab VIPs. external-dns reads doltdb from the status of dolt/dolt, and argocd from the Ingress status that Traefik copies from traefik/traefik-lab. The curated mqtt entry lives in the dns-control database.
  • No repository and no runbook names a consumer of otel-collector-lb. docs/operations/clickstack.md sends external OTLP through otel-gateway.fzymgc.house instead.

Goals:

  • Move every LoadBalancer VIP to a routed-only prefix that the UCG-Fiber learns over BGP, with no outage of any name and no outage of the Pi workers.
  • Keep the reply of every node on the path through the gateway, so the gateway sees both directions of each flow.
  • Remove MetalLB and every workaround that exists only for L2 announcement.

Non-Goals:

  • The API VIP 192.168.20.140. kube-vip keeps it, with ARP. D13 reserves its routed address for a follow-on change.
  • The removal of the Lab leg. It stays, as the transit leg for BGP.
  • BFD, TCP-MD5 on the sessions, and the announcement of pod CIDRs or ClusterIPs.
  • A test of a tailnet path. Sean ruled on 2026-09-23 that the tailnet proof is the configuration of the advertisers and of the policy.
  • The retirement of the OTLP LoadBalancer. Its twin is the canary, and it stays.

D1. Cilium BGP control plane and LB-IPAM, with the default IPAM off

Section titled “D1. Cilium BGP control plane and LB-IPAM, with the default IPAM off”

The cilium role gains two variables, cilium_bgp_control_plane_enabled: true and cilium_default_lb_service_ipam: none. They render as bgpControlPlane.enabled and defaultLBServiceIPAM in templates/values.yaml.j2. The role already restarts the agents after a values change.

With defaultLBServiceIPAM: none, LB-IPAM assigns an address only to a Service with a Cilium class. MetalLB assigns only to a Service without a class. So the two never assign the same Service, and the converge must run before the pool exists.

Alternatives: MetalLB in BGP mode keeps two components, and Sean rejected it on 2026-09-28. Cilium L2 announcements keep the VIPs on-link, are Beta, and do not support externalTrafficPolicy: Local.

D2. One session per family from the Lab leg of each node

Section titled “D2. One session per family from the Lab leg of each node”

Sean chose the Lab leg on 2026-10-04. Each node peers with 192.168.200.1 for IPv4 and with fddb:f665:73f7:4::1 for IPv6. The kernel picks the source of each session from the connected Lab route, so each session runs from the Lab address of the node in its family, and BGP sets that address as the next hop. A separate session per family gives each family a native next hop.

The gateway forwards a request from Main, MainWireless, Core or IoT out of its Lab interface, and the reply comes back in on Lab. No interface sees a request leave the way it came in. Peering over Main was rejected because wired Main clients, the workstation and the NAS among them, are on-link with the Main addresses of the nodes.

D3. A reply table for the VIP prefixes, separate from table 1000

Section titled “D3. A reply table for the VIP prefixes, separate from table 1000”

tp2-bootstrap-node adds table 1001 (tp2_node_vip_route_table) with one route per family: the default route through the Lab gateway on end0.1000. Two rules at priority 6100 (tp2_node_vip_rule_priority) send traffic from 192.168.250.0/24 and from fddb:f665:73f7:250::/64 to that table. The existing Lab rules at 6000 cover the Lab /22 and /64, which do not overlap the VIP prefixes.

Table 1000 was rejected for these rules. It holds the on-link Lab route, so a reply to a Lab client leaves on-link, and the gateway sees only the request. With table 1001, the reply to a Lab client also goes through the gateway. The Lab client proof in PR 1 measures this path, because the gateway can send an ICMP redirect for the request of a Lab client.

The existing Lab VIPs prove the mechanism. A reply from 192.168.200.145 follows the rule at 6000 today, and docs/operations/lab-service-vlan.md measures it on the counters of end0.1000.

Sean chose 192.168.250.0/24 and fddb:f665:73f7:250::/64 on 2026-10-04. PR 1 allocates both in docs/reference/network.md before any consumer uses them. The /64 is a routed prefix, not a leg, so it takes no index in ula_leg_index. A new leg still takes index 8.

ansible/inventory/group_vars/all.yml gains lb_vip_prefix_ipv4: 192.168.250.0/24 and lb_vip_prefix_ipv6: "{{ ula_site_prefix }}:250::/64". tp2-bootstrap-node, k8s-tailscale and nas-tailscale read them.

Each Service keeps its host number in both families: Traefik .145, the Temporal node gateway .146, Dolt .147, Mosquitto .148, the OTLP collector .149. One CiliumLoadBalancerIPPool, routed-vips, holds one range per family: 192.168.250.1 to .253, and fddb:f665:73f7:250::1 to ::253. The IPv6 range holds every suffix that mirrors an IPv4 host number. Each range stops below .254, which D13 reserves. Each twin pins its two addresses with lbipam.cilium.io/ips.

The pool serves only Services in the namespaces that hold a twin: clickstack, dolt, mosquitto, temporal and traefik. Its serviceSelector matches io.kubernetes.service.namespace. Sean chose the list on 2026-10-04, after a security review found that an open pool gives a routed VIP to any Service that sets the class. A new routed Service in another namespace needs one line in the pool.

tf/unifi/bgp.tf declares one unifi_bgp in raw mode (config, enabled = true, upload_file_name, description). Structured mode was rejected: its template redistributes the connected routes of the gateway and permits every route in both directions. The file has this shape:

frr defaults traditional
!
ip prefix-list LB-VIPS seq 10 permit 192.168.250.0/24 ge 32
ipv6 prefix-list LB-VIPS6 seq 10 permit fddb:f665:73f7:250::/64 ge 128
route-map CILIUM-IN permit 10
match ip address prefix-list LB-VIPS
route-map CILIUM-IN6 permit 10
match ipv6 address prefix-list LB-VIPS6
route-map DENY-ALL deny 10
!
router bgp 65000
bgp router-id 192.168.200.1
bgp log-neighbor-changes
bgp graceful-restart
no bgp default ipv4-unicast
neighbor CILIUM4 peer-group
neighbor CILIUM4 remote-as 65001
neighbor CILIUM4 timers 3 9
neighbor CILIUM4 passive
neighbor CILIUM6 peer-group
neighbor CILIUM6 remote-as 65001
neighbor CILIUM6 timers 3 9
neighbor CILIUM6 passive
neighbor 192.168.200.141 peer-group CILIUM4
neighbor fddb:f665:73f7:4:aaaa::141 peer-group CILIUM6
! the same pair for each node, .141 to .144 and .151 to .154
address-family ipv4 unicast
maximum-paths 8
neighbor CILIUM4 activate
neighbor CILIUM4 route-map CILIUM-IN in
neighbor CILIUM4 route-map DENY-ALL out
neighbor CILIUM4 soft-reconfiguration inbound
neighbor CILIUM4 maximum-prefix 256
exit-address-family
address-family ipv6 unicast
maximum-paths 8
neighbor CILIUM6 activate
neighbor CILIUM6 route-map CILIUM-IN6 in
neighbor CILIUM6 route-map DENY-ALL out
neighbor CILIUM6 soft-reconfiguration inbound
neighbor CILIUM6 maximum-prefix 256
exit-address-family
  • The gateway is AS 65000 and the cluster is AS 65001, both private.
  • The gateway peers with the Lab addresses of the nodes and with no other host. Each node is a passive neighbor in each family, so the gateway waits for the node to connect. bgp.tf builds the lines from one list of host octets, bgp_node_hosts. Sean chose exact neighbors over listen ranges on 2026-10-04. With a listen range, a Lab host can set a free static address in the range, peer, and announce a VIP. show bgp summary also lists a node that is down.
  • The import route-maps accept host routes inside the VIP prefixes and nothing else. The export route-map sends nothing to the nodes.
  • maximum-paths 8 installs one next hop per announcing node.

The wire struct is unifi.BGPConfig in unifi/bgp_config.generated.go of the go-unifi fork. It models frr_bgpd_config, description, enabled and uploaded_file_name, and the controller keys _id, site_id, attr_hidden, attr_hidden_id, attr_no_delete and attr_no_edit. The provider sets all four writable fields in modelToBGP (unifi/bgp_resource.go). An update posts the whole object to v2/api/site/<site>/bgp/config, the same call as a create, so a key the struct does not model is lost on the first update. The collection holds no object today, so the field audit runs after the first apply, and it must print no gap for bgp-config before any later change to bgp.tf.

D6. The Cilium objects live in a new ArgoCD app, cilium-lb

Section titled “D6. The Cilium objects live in a new ArgoCD app, cilium-lb”

argocd/app-configs/cilium-lb/ holds the pool of D4 and four BGP objects, and argocd/cluster-app/templates/cilium-lb.yaml syncs it:

  • CiliumBGPClusterConfig ucg-fiber: no node selector, so every node runs the instance. One instance with localASN: 65001 and two peers, ucg-fiber-v4 at 192.168.200.1 and ucg-fiber-v6 at fddb:f665:73f7:4::1, both with peerASN: 65000.
  • Two CiliumBGPPeerConfig objects, one per family. Each sets holdTimeSeconds: 9, keepAliveTimeSeconds: 3, graceful restart with restartTimeSeconds: 60, its one family, and the advertisement label.
  • CiliumBGPAdvertisement lb-vips: advertisementType: Service with addresses: [LoadBalancerIP], and a selector that matches every Service.

The selector matches every Service, so while MetalLB exists the nodes also announce the Lab VIPs of the MetalLB Services. The import route-map of D5 drops them. A label on each Service was rejected, because every future LoadBalancer Service has the class anyway.

The CRDs exist only after the converge of D1. The operator runs that converge from the branch of PR 1 before the merge, so the first sync of the app succeeds.

Sean chose twins on 2026-10-04. Kubernetes rejects a change of spec.loadBalancerClass on an existing LoadBalancer Service, and the Traefik values already warn against a change of address in place. Each twin copies the selector, the ports, externalTrafficPolicy: Local and the dual-stack fields of the old Service. It adds loadBalancerClass: io.cilium/bgp-control-plane and lbipam.cilium.io/ips.

Old Service Twin VIPs
traefik/traefik-lab traefik/traefik-vip (chart key additionalServices.vip) 192.168.250.145, fddb:f665:73f7:250::145
temporal/temporal-node-gateway temporal/temporal-node-gateway-vip .146, ::146
dolt/dolt dolt/dolt-vip .147, ::147
mosquitto/mosquitto-lab mosquitto/mosquitto-vip .148, ::148
clickstack/otel-collector-lb clickstack/otel-collector-vip .149, ::149

PR 3 deletes the old Services, except dolt. dolt.dolt.svc is a scrape target in argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml, so dolt becomes a ClusterIP Service and loses externalTrafficPolicy and the MetalLB annotation.

  • The 20 target annotations change to "192.168.250.145,fddb:f665:73f7:250::145".
  • providers.kubernetesIngress.publishedService.pathOverride in argocd/cluster-app/templates/traefik.yaml changes to traefik/traefik-vip, so argocd follows.
  • The external-dns.alpha.kubernetes.io/hostname annotation moves from dolt to dolt-vip.
  • The operator changes the curated mqtt entry to 192.168.250.148 and fddb:f665:73f7:250::148 in dns-control. The entry keeps ptr: false.

D9. The Pi workers and the Temporal gateway policy

Section titled “D9. The Pi workers and the Temporal gateway policy”

dns_node_worker_gateway_addr changes to "192.168.250.146:7233". A Pi sends to the routed prefix through its default route on Core, so the gateway sees 192.168.40.6 and 192.168.40.7. In PR 2 the policy admits the Core addresses beside the Lab addresses, because both Services select the same pods. PR 3 drops the Lab addresses. The certificate needs no change, because the worker verifies temporal-nodes.fzymgc.house and not the address.

D10. The tailnet follows the prefixes in three steps

Section titled “D10. The tailnet follows the prefixes in three steps”
  • PR 1 adds lb_vip_prefix_ipv4 and lb_vip_prefix_ipv6 to k8s_tailscale_advertise_routes and nas_tailscale_advertise_routes. It adds both to autoApprovers.routes in tailscale/policy.hujson, with the host aliases lb-vips and lb-vips-v6 and the matching entries in the admin grant. The operator converges all three advertisers in one session, so every advertiser announces the same set.
  • PR 2 points auth-fzymgc and auth-fzymgc-v6 at the Traefik twin.
  • PR 3 removes 192.168.200.128/27, fddb:f665:73f7:4:aaaa::/80, network-lab and network-lab-v6, and converges the advertisers again.

A subnet router sends VIP traffic out of end0 through its default route, and Tailscale rewrites the source to the Main address of the router. That address is inside 192.168.16.0/20, and its Main ULA is inside fddb:f665:73f7:1::/64. Both lists in argocd/app-configs/keycloak/ingress-route.yaml already trust those ranges, so PR 3 removes the four Lab entries of the routers from both lists together.

D11. The fence of the untrusted legs is measured before the move

Section titled “D11. The fence of the untrusted legs is measured before the move”

The UniFi documentation does not state which zone a BGP-learned destination falls in. The canary of PR 1 answers it: a throwaway client on Guest and one on Telework (the nas-throwaway-lxc skill) try the canary VIP in both families. Both attempts must fail before PR 1 merges. If one connects, PR 1 gains a unifi_firewall_policy in tf/unifi/firewall-zones.tf that blocks the two VIP prefixes from that zone, and the probe runs again.

D12. MetalLB goes in PR 3, after the old VIPs are idle

Section titled “D12. MetalLB goes in PR 3, after the old VIPs are idle”

PR 3 deletes argocd/cluster-app/templates/metallb.yaml and argocd/app-configs/metallb/. ArgoCD prunes the chart objects. The CRDs carry helm.sh/resource-policy: keep, and the namespace came from CreateNamespace=true, so the operator deletes both. PR 3 starts only after no node holds a Cilium connection-tracking entry for an old VIP (task 3.1). That gate finds a device that uses an old address as a literal, an MQTT client on IoT for example.

D13. The routed API VIP is reserved, and a follow-on change moves it

Section titled “D13. The routed API VIP is reserved, and a follow-on change moves it”

192.168.250.254 and fddb:f665:73f7:250::254 are reserved for the API VIP in docs/reference/network.md. No Service pins them. A follow-on change, api-vip-over-bgp, moves the API VIP off Main after a study of the cold start, and the reserved address is one of its candidates. This change already gives the address a route: the import filter of D5 accepts it, the reply rule of D3 covers it, and the tailnet advertises its prefix.

The follow-on is a separate change for three reasons:

  • Cilium reaches the API server through cilium_k8s_service_host, the kube-vip address. If Cilium announces the API VIP, a cold start of the cluster deadlocks. So Cilium’s own path to the API server must move to a node-local endpoint first.
  • The control-plane nodes carry no taint, so they hold LoadBalancer endpoints, and Cilium peers from them. A kube-vip BGP speaker on the same node opens a second session from the same Lab address to the same gateway address. Peering kube-vip over Main brings back the hairpin of D2.
  • The new address needs a tls-san entry in k3s, and every consumer of 192.168.20.140 must move: the agent join URL, scripts/k8s-login.sh, the kubeconfigs outside the cluster and the Kubernetes auth of Vault.
  • [The gateway sends an ICMP redirect for the request of a Lab client, and the client then bypasses the gateway in one direction] → The Lab client proof of PR 1 measures the path. No repository consumer of a VIP sits on Lab.
  • [A BGP-learned destination falls in a zone that the fence does not cover] → D11. The probe gates PR 1.
  • [A Lab host takes the address of a node that is down, peers, and announces a VIP] → The gateway peers only with the exact node addresses, so the host must take a node address while that node is off. The import filter still accepts only VIP host routes. Lab is a trusted leg. TCP-MD5 on the sessions closes the rest, at the cost of a password in the FRR text and in the workspace state, and it is not part of this change.
  • [A Service in another namespace sets the class and takes a routed VIP] → The pool serves only the five listed namespaces (D4).
  • [A node crashes, and graceful restart keeps its routes for up to 60 seconds] → Accepted. Without graceful restart, a restart of a Cilium agent withdraws every route of the node, and a Service with one replica goes dark for the length of the restart.
  • [A later update of bgp.tf deletes a key that the controller adds] → The field audit after the first apply (task 1.13).
  • [A client keeps an old VIP as a literal] → The twins keep the old VIPs answering until PR 3, and the connection-tracking gate (task 3.1) finds the client first.
  • [A change of next hops rehashes ECMP and resets a long-lived connection] → Accepted. Next hops change only when an endpoint moves.
  • [The agent restart in PR 1 interrupts the datapath on each node in turn] → It is the same rolling restart as every Cilium converge.

Each step names its rollback. A revert pull request merges like any other, and HCP Terraform applies the tf/ part of it on merge.

PR 1, converge and canary:

  1. The operator converges the node rules one node at a time: uv run ansible-playbook -i inventory/hosts.yml bootstrap-nodes-playbook.yml --tags network --limit $N. Rollback on a node: ssh fzymgc@$N 'sudo ip rule del priority 6100; sudo ip -6 rule del priority 6100'.
  2. The operator converges Cilium from the branch: uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium. Rollback: check out origin/main and run the same command.
  3. The merge syncs cilium-lb and the canary, and HCP Terraform applies unifi_bgp. Rollback: git revert -m 1 <merge commit of PR 1>, merged as a pull request. ArgoCD prunes the objects, the routes withdraw, and the apply deletes the gateway configuration. Then run step 2 again from origin/main.
  4. The operator converges the advertisers: uv run ansible-playbook -i inventory/hosts.yml tailscale-subnet-routers-playbook.yml and scripts/nas-playbook.sh --tags nas-tailscale. Rollback: the same two commands after the revert.

PR 2, the move:

  1. The merge syncs the twins and the DNS writers. The operator changes the mqtt entry and converges the Pi workers one at a time: uv run ansible-playbook -i inventory/hosts.yml dns-playbook.yml --tags dns-node-worker --limit sandiego, then --limit portland.
  2. Rollback: git revert -m 1 <merge commit of PR 2>, merged as a pull request. The DNS writers return to the Lab VIPs, which still answer. The operator sets mqtt back and runs the dns-node-worker command again on both Pis.

PR 3, teardown:

  1. The merge prunes MetalLB and the old Services. The operator deletes the MetalLB CRDs and namespace and converges the advertisers.
  2. Rollback: git revert -m 1 <merge commit of PR 3>, merged as a pull request. The chart installs the CRDs again with server-side apply, and the old Services and their L2 announcements return. The operator converges the advertisers again. A rollback of BGP itself after PR 3 needs the revert of PR 2 first.
  • The restart time of 60 seconds for graceful restart is a first value. The operator can tune it after the restart of one agent is timed. The value changes no spec and no task.