Design: The API VIP over BGP
Context
Section titled “Context”See openspec/changes/api-vip-over-bgp/proposal.md, section Why, for the motivation. This section holds the study that the proposal
asked for. Every fact below was read on 2026-10-16 from the live estate, from tpi-alpha-1 (a
server), from tpi-beta-1 (an agent) and from the UCG-Fiber, unless a sentence names the file it
comes from.
What listens for the API on a node
Section titled “What listens for the API on a node”k3s runs one process per node. On a server the process is k3s-server, and it opens two listeners:
| Listener | What it is | Certificate SANs include |
|---|---|---|
*:6443 |
The k3s supervisor. It proxies the API and serves the join protocol | 127.0.0.1, localhost, every node address, 192.168.20.140, k8s-cluster.fzymgc.house |
127.0.0.1:6444 |
kube-apiserver itself | 127.0.0.1, every node address, 192.168.20.140, k8s-cluster.fzymgc.house |
On an agent the process is k3s-agent, and it opens a client load balancer on 127.0.0.1:6444 and
[::1]:6444. The load balancer forwards each connection to the 6443 port of one server. It
learned the three server addresses from the default/kubernetes EndpointSlice and persisted them in
/var/lib/rancher/k3s/agent/etc/k3s-agent-load-balancer.json. The file on tpi-beta-1 holds
192.168.20.141, .142 and .143, and ServerURL: https://10.255.254.6:6443.
https://127.0.0.1:6444/readyz answered 200 on both node types with the cluster CA as the trust
anchor. The kubelet of every agent already uses that endpoint. The kubelet of every server uses
https://127.0.0.1:6443.
Who dials the VIP today
Section titled “Who dials the VIP today”| Consumer | Address it dials | Source |
|---|---|---|
The Cilium agent on every node, the Cilium operator and cilium-envoy |
192.168.20.140:6443, as KUBERNETES_SERVICE_HOST in the pod environment |
cilium_k8s_service_host in ansible/roles/cilium/defaults/main.yml, rendered as k8sServiceHost |
| The operator kubeconfigs | https://k8s-cluster.fzymgc.house:6443, a CNAME to k8s-api.fzymgc.house, an A record 192.168.20.140 in dns-control |
the k3s-kubeconfig tasks of ansible/k3s-playbook.yml write the name |
scripts/k8s-login.sh |
https://192.168.20.140:6443 as a literal |
API_SERVER in the script |
| A tailnet admin | 192.168.20.140 through the Main route, under the grant network-main:* |
tailscale/policy.hujson |
| The k3s certificate | 192.168.20.140 as a SAN |
k8s_cluster_sans in tp_cluster_controlplane.yml and tp_cluster_nodes.yml, two copies of one list |
| The kube-vip static pod | It binds the VIP. For its own API calls it uses /etc/rancher/k3s/k3s.yaml, which points at 127.0.0.1:6443 |
ansible/roles/kube-vip |
These do not dial the VIP:
- The kubelet of every node. It uses loopback, as above.
- The k3s agents.
K3S_URLin/etc/systemd/system/k3s-agent.service.envishttps://10.255.254.6:6443. No route to that address exists anywhere in the estate, and the agents run. So the agents use the persisted server list and never the registration address after the first join. The inventory valuek8s_cluster_endpoint_ip: 10.255.254.6is a dead address. A new agent cannot join with it. - ArgoCD. It holds no cluster Secret, and all 58 Application destinations are
https://kubernetes.default.svc. - The Vault Kubernetes auth roles. Each names the audience
https://kubernetes.default.svc.cluster.local. - The hosted Kubernetes MCP server. Its errors name
https://kubernetes.default.svc. - CoreDNS, Hubble, metrics-server and every other pod. They use the ClusterIP
10.43.0.1.
Cold start of the estate
Section titled “Cold start of the estate”The parts come up in this order from power-off. Each line names what the part dials and what it does while that address does not answer.
- The UCG-Fiber boots. Its FRR starts with the eight nodes as passive neighbors and waits. It dials nothing in the cluster.
- Each node boots, and netplan sets the Main leg, the Lab leg, the rules at priority 6000 and 6100, and the tables 1000 and 1001. Nothing waits on the gateway here.
- Each server starts
k3s-server. etcd needs two of the three servers for quorum, over the Main addresses, on-link. Until quorum exists, kube-apiserver answers nothing and kubelet retries127.0.0.1:6443. Nothing on a server waits for the VIP. - Each agent starts
k3s-agent. Its load balancer dials the three persisted server addresses on Main, on-link, and retries until one answers. The gateway is not in this path. - kubelet on every node reads its pods from the API over loopback and starts the Cilium agent.
Today the agent dials
192.168.20.140, which exists only after kube-vip on a server has won the lease. kube-vip wins the lease through127.0.0.1:6443, so the chain closes, but the agent of a server depends on another server’s Main leg by ARP. After this change the agent dials127.0.0.1:6444and depends on its own node only. - Each Cilium agent opens its two BGP sessions to
192.168.200.1andfddb:f665:73f7:4::1from the Lab leg. If the gateway is not up, the agent retries. Every other function of the agent runs without the sessions. - The gateway installs a host route for each VIP whose announcing node has a session. The API VIP is up when one server has a Cilium agent with an established session.
A reboot of one control-plane node takes its sessions down within the hold time of 9 seconds. The
gateway keeps the routes of that node for restartTimeSeconds, 60 seconds, because graceful
restart is on. For an API VIP with three next hops, up to one third of new connections go to the
rebooting node in that window and fail. After the window the route has two next hops. Today a
reboot of the kube-vip leader moves the VIP within the 15 second lease, and a reboot of a
non-leader costs nothing. The drill in PR 1 measures the window.
A reboot of the gateway takes every session down. No node dials the VIP, so no node notices. The kubelets, the Cilium agents and ArgoCD run on. The operator loses the API VIP, every routed VIP and every inter-leg path for the reboot time. When the gateway returns, each agent reconnects, and the route returns within the connect retry of Cilium. The operator can dial a server address on Main from a Main client meanwhile, because every node address is a SAN of the certificate.
The three speakers
Section titled “The three speakers”| Speaker | Path of a Main client | What it needs | Why not |
|---|---|---|---|
| Cilium BGP announces a LoadBalancer Service for the API server | Gateway, Lab leg of a server | Cilium’s own path moves to loopback first. One Service, two EndpointSlices, one pool. No FRR change: the import filter already permits 192.168.250.0/24 ge 32, the reply rule at 6100 covers the prefix, and the tailnet advertises it |
It is the pick |
| kube-vip in BGP mode | Gateway, then a server | A second BGP speaker on each server. From the Lab address it collides with the Cilium session, because FRR holds one neighbor per address. From the Main address it needs three new neighbors in tf/unifi/bgp.tf, and bgp-config still has the two unmodelled keys error and redistribute_to_sd_wan, so the file changes only through the UI with an adopt |
Two BGP stacks and a UI step for every FRR change |
kube-vip keeps ARP on the Lab leg at 192.168.200.140 |
Gateway, then on-link Lab | A tailnet route for 192.168.200.140/32, against the rule that Lab is not on the tailnet. A Lab client stays on-link with the VIP |
It keeps a VIP on a leg, which lb-vips-over-bgp removed everywhere else |
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- The Kubernetes API answers at
192.168.250.254andfddb:f665:73f7:250::254, announced over BGP by the three servers, with no new FRR text on the gateway. - No node needs the API VIP. Every path from a node to the API is loopback.
- The move is reversible at each pull request, and the old VIP answers until the last one.
Non-Goals:
- A health check of kube-apiserver behind the VIP. See D3.
- A change to the graceful restart timer. It is shared by every VIP. The drill measures it first.
- The retirement of the
10.255.254.xconvention beyond the one value this change touches. - A test of a tailnet path. Sean ruled on 2026-09-23 that the tailnet proof is the configuration.
Decisions
Section titled “Decisions”D1. Cilium BGP is the speaker
Section titled “D1. Cilium BGP is the speaker”The API VIP is a LoadBalancer Service that Cilium announces, like every other VIP. kube-vip goes. The table above holds the alternatives and their costs. The cold start closes because no node dials the VIP after D2: the servers announce the VIP, and only clients outside the nodes consume it.
D2. Cilium reaches the API server on loopback
Section titled “D2. Cilium reaches the API server on loopback”cilium_k8s_service_host becomes 127.0.0.1 and cilium_k8s_service_port becomes 6444. The
chart puts both into KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT of the Cilium agent,
the operator and cilium-envoy. All three run with hostNetwork: true, so loopback is the node.
No other kube-system workload carries the variable.
On a server, 127.0.0.1:6444 is kube-apiserver. On an agent it is the k3s load balancer, which
fails over between the servers and already carries the kubelet. The certificate behind both carries
127.0.0.1, and the probe with the cluster CA returned 200 on each. The Cilium documentation
prescribes the same loopback path for RKE2, where the agent proxy listens on 6443.
Alternative considered: k8s.apiServerURLs with the three server addresses. Cilium then fails
over by itself. Rejected, because the k3s load balancer already does that on the agents with its
own health checks, and the list is a second place that holds the server addresses.
Trade-off: on a server whose kube-apiserver is down, the Cilium agent of that server has no API.
The kubelet of that server is in the same state, because k3s is one process, and systemd restarts
it after 5 seconds (Restart=always).
D3. One selector-less Service with hand-written EndpointSlices
Section titled “D3. One selector-less Service with hand-written EndpointSlices”kube-apiserver in k3s is not a pod, so no selector can find it. argocd/app-configs/cilium-lb/
gains api-vip.yaml with the pool and the Service, and the cilium role applies the slices:
- A
CiliumLoadBalancerIPPoolapi-vipwith two blocks,startandstopboth192.168.250.254, and bothfddb:f665:73f7:250::254. ItsserviceSelectormatchesio.kubernetes.service.namespace: kube-system. The poolrouted-vipskeeps its namespace list and its ranges, so the runbook text about it stays true. - A Service
kube-system/kubernetes-apiof typeLoadBalancer, classio.cilium/bgp-control-plane,externalTrafficPolicy: Local,ipFamilyPolicy: RequireDualStack, port6443to target port6443, andlbipam.cilium.io/ipspinned to the two addresses. It has no selector. - Two EndpointSlices, one per address type, labelled
kubernetes.io/service-name: kubernetes-apiandendpointslice.kubernetes.io/managed-by: fzymgc.house. Each endpoint holds the Main address of one server in that family, itsnodeName, andconditions.ready: true. The IPv6 addresses are the Main ULAsfddb:f665:73f7:1:aaaa::141to::143, which the certificate carries and on whichk3s-serverlistens. TheciliumAnsible role applies the slices, from the templateapi-vip-endpointslices.yaml.j2and the host vars oftp_cluster_controlplane, the way it applies the Gateway API CRDs. ArgoCD 3.x excludes the EndpointSlice kind by default inargocd-cm, and an excluded kind is neither synced nor reported, so a slice inapi-vip.yamlnever reached the cluster while the app showed Synced. Sean chose the role over an override ofresource.exclusionson 2026-10-17: the slices change only with the control-plane set, the server list then has one source, and ArgoCD does not start to watch every EndpointSlice of the cluster.
With externalTrafficPolicy: Local, Cilium announces the VIP only from a node that holds a ready
endpoint, so only the three servers announce. On a server, the datapath sends a packet for the VIP
to the server’s own Main address on port 6443, and the reply leaves from the VIP through table
1001 to the Lab gateway, as every other routed VIP does.
The default/kubernetes Service was rejected. The API server reconciles it, and its type stays
ClusterIP.
Trade-off: the EndpointSlices are static, so a server whose kube-apiserver is down but whose Cilium agent is up keeps announcing. The window is the systemd restart of D2. The operator fallback is a server address on Main.
D4. The consumers move in three pull requests
Section titled “D4. The consumers move in three pull requests”PR 1 makes the new address answer beside the old one:
k8s_cluster_sansgains192.168.250.254andfddb:f665:73f7:250::254in both files that hold the list. Thek3s-configtag rolls the servers one at a time with the readiness gate.- D2 and D3. The Cilium converge runs before the merge, so the agents already use loopback when the Service appears.
docs/reference/network.mdchanges the row of.254from reserved to the Service, and drops.200.140 reserved for kube-vipfrom the Lab row.
PR 2 moves every client:
- The operator changes the dns-control entry
k8s-apito192.168.250.254and adds the AAAAfddb:f665:73f7:250::254.k8s-cluster.fzymgc.houseis a CNAME to it, so the operator kubeconfigs follow. scripts/k8s-login.shdialshttps://k8s-cluster.fzymgc.house:6443, the same name as the admin kubeconfig.k8s_cluster_endpoint_ipbecomes192.168.250.254in both group_vars files. A new agent can join again. The existing agents keep the deadK3S_URLin their unit environment, because the install script writes it once and the persisted server list serves them. The runbook names the one-line fix for an agent that must be re-pointed.docs/getting-started/cluster-access.md,docs/operations/cilium-kpr.md,README.mdanddocs/reference/index.mdname the new address.
PR 3 tears down kube-vip:
- The
kube-viprole gainskube_vip_state. Withabsent, it removes the static pod manifest, waits for the pod to leave, removes192.168.20.140/32fromend0if it is still bound, and deletes the Leasekube-system/plndr-cp-lock. kube_vip_addressleavesansible/inventory/group_vars/all.yml, and192.168.20.140and10.255.254.6leavek8s_cluster_sans.- A fourth pull request deletes the role, its play and
kube_vip_stateafter the converge, the same two-step shape as the MetalLB removal.
The tailnet needs no change. 192.168.250.0/24 and fddb:f665:73f7:250::/64 are advertised, and
the admin grant covers lb-vips:* and lb-vips-v6:*. No other grant reaches 6443 today.
The fence needs no change. The VIP prefix sits behind the Lab leg, in the zone Internal with Main,
MainWireless, Core, IoT and Lab. Guest and Telework are blocked from Internal already, and the
lb-vips-over-bgp probes measured that for the prefix. PR 1 repeats the probe on port 6443.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [The gateway reboots, and the operator loses the API VIP for that time] → Every node runs on. The operator dials a server address on Main, which is a SAN. This is the new dependency that the proposal names, and Sean accepts it with D1.
- [A control-plane node reboots, and graceful restart keeps its next hop for 60 seconds] → Up to a
third of new connections fail in the window. The drill in PR 1 times the window, and a change to
restartTimeSecondsis a follow-on, because it is shared by every VIP. - [kube-apiserver dies on a server whose Cilium agent lives] → The static endpoint keeps the route. systemd restarts k3s in 5 seconds. The operator fallback is the same as above.
- [The agents keep a dead
K3S_URL] → They never use it after the first join. The runbook holds the fix for an agent that must be re-pointed. - [A client keeps
192.168.20.140as a literal] → PR 3 waits until no server sees a connection to the old VIP from outside the nodes. The gate reads the Cilium connection-tracking table on the kube-vip leader, as task 3.1 oflb-vips-over-bgpdid. - [The pool
api-vipserves every LoadBalancer Service inkube-system] → It holds two addresses, and the Service pins both, so a second Service gets none. - [The Cilium converge of D2 restarts every agent] → It is the same rolling restart as every Cilium converge, and the agents reconnect to the API over loopback.
Migration Plan
Section titled “Migration Plan”Each step names its rollback. A revert pull request merges like any other.
PR 1, the new address answers:
- The operator rolls the SAN list onto the servers from the branch:
uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags k3s-config --limit tp_cluster_controlplane. Rollback: check outorigin/mainand run the same command. - The operator converges Cilium from the branch:
uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium-install. Rollback: check outorigin/mainand run the same command. The agents dial the kube-vip VIP again. - The merge syncs
api-vip.yaml. The three servers announce the VIP. Rollback:git revert -m 1 <merge commit of PR 1>, merged as a pull request. ArgoCD prunes the objects and the routes withdraw. - The proofs:
/readyzthrough the VIP from a Mac on wired Main, in both families, the route on the gateway with three next hops, the fence probe on 6443 from Guest and Telework, and the two drills that Sean approves, a reboot of one control-plane node and a reboot of the gateway.
PR 2, the move:
- The operator changes the
k8s-apientry in dns-control and waits for the TTL of 300 seconds. Rollback: set the entry back to192.168.20.140and remove the AAAA. - The merge changes
scripts/k8s-login.sh, the inventory and the docs. Rollback:git revert -m 1 <merge commit of PR 2>, merged as a pull request.
PR 3, teardown:
- The gate: no connection to
192.168.20.140:6443from outside the nodes in the connection tracking table of the kube-vip leader. - The merge sets
kube_vip_state: absentand trims the SAN list. The operator runsuv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags kube-vip,k3s-config --limit tp_cluster_controlplane. Rollback:git revert -m 1 <merge commit of PR 3>, merged as a pull request, then the same command. kube-vip returns and binds the old VIP within its lease. - The cold-start drill, after Sean approves it. It needs no operator action between power-on and
ok.
Open Questions
Section titled “Open Questions”- The length of the window after a control-plane reboot. The drill of 2026-10-17 measured it: the gateway kept the next hop of the rebooting node for about 60 seconds, and no probe through the VIP failed in that window at a 5 second cadence. The timer stays at 60 seconds.
- Whether kube-vip removes the VIP from
end0on SIGTERM. The teardown task removes the address itself if it remains, so the answer changes no task.