Skip to content

Design: The API VIP over BGP

See openspec/changes/api-vip-over-bgp/proposal.md, section Why, for the motivation. This section holds the study that the proposal asked for. Every fact below was read on 2026-10-16 from the live estate, from tpi-alpha-1 (a server), from tpi-beta-1 (an agent) and from the UCG-Fiber, unless a sentence names the file it comes from.

k3s runs one process per node. On a server the process is k3s-server, and it opens two listeners:

Listener What it is Certificate SANs include
*:6443 The k3s supervisor. It proxies the API and serves the join protocol 127.0.0.1, localhost, every node address, 192.168.20.140, k8s-cluster.fzymgc.house
127.0.0.1:6444 kube-apiserver itself 127.0.0.1, every node address, 192.168.20.140, k8s-cluster.fzymgc.house

On an agent the process is k3s-agent, and it opens a client load balancer on 127.0.0.1:6444 and [::1]:6444. The load balancer forwards each connection to the 6443 port of one server. It learned the three server addresses from the default/kubernetes EndpointSlice and persisted them in /var/lib/rancher/k3s/agent/etc/k3s-agent-load-balancer.json. The file on tpi-beta-1 holds 192.168.20.141, .142 and .143, and ServerURL: https://10.255.254.6:6443.

https://127.0.0.1:6444/readyz answered 200 on both node types with the cluster CA as the trust anchor. The kubelet of every agent already uses that endpoint. The kubelet of every server uses https://127.0.0.1:6443.

Consumer Address it dials Source
The Cilium agent on every node, the Cilium operator and cilium-envoy 192.168.20.140:6443, as KUBERNETES_SERVICE_HOST in the pod environment cilium_k8s_service_host in ansible/roles/cilium/defaults/main.yml, rendered as k8sServiceHost
The operator kubeconfigs https://k8s-cluster.fzymgc.house:6443, a CNAME to k8s-api.fzymgc.house, an A record 192.168.20.140 in dns-control the k3s-kubeconfig tasks of ansible/k3s-playbook.yml write the name
scripts/k8s-login.sh https://192.168.20.140:6443 as a literal API_SERVER in the script
A tailnet admin 192.168.20.140 through the Main route, under the grant network-main:* tailscale/policy.hujson
The k3s certificate 192.168.20.140 as a SAN k8s_cluster_sans in tp_cluster_controlplane.yml and tp_cluster_nodes.yml, two copies of one list
The kube-vip static pod It binds the VIP. For its own API calls it uses /etc/rancher/k3s/k3s.yaml, which points at 127.0.0.1:6443 ansible/roles/kube-vip

These do not dial the VIP:

  • The kubelet of every node. It uses loopback, as above.
  • The k3s agents. K3S_URL in /etc/systemd/system/k3s-agent.service.env is https://10.255.254.6:6443. No route to that address exists anywhere in the estate, and the agents run. So the agents use the persisted server list and never the registration address after the first join. The inventory value k8s_cluster_endpoint_ip: 10.255.254.6 is a dead address. A new agent cannot join with it.
  • ArgoCD. It holds no cluster Secret, and all 58 Application destinations are https://kubernetes.default.svc.
  • The Vault Kubernetes auth roles. Each names the audience https://kubernetes.default.svc.cluster.local.
  • The hosted Kubernetes MCP server. Its errors name https://kubernetes.default.svc.
  • CoreDNS, Hubble, metrics-server and every other pod. They use the ClusterIP 10.43.0.1.

The parts come up in this order from power-off. Each line names what the part dials and what it does while that address does not answer.

  1. The UCG-Fiber boots. Its FRR starts with the eight nodes as passive neighbors and waits. It dials nothing in the cluster.
  2. Each node boots, and netplan sets the Main leg, the Lab leg, the rules at priority 6000 and 6100, and the tables 1000 and 1001. Nothing waits on the gateway here.
  3. Each server starts k3s-server. etcd needs two of the three servers for quorum, over the Main addresses, on-link. Until quorum exists, kube-apiserver answers nothing and kubelet retries 127.0.0.1:6443. Nothing on a server waits for the VIP.
  4. Each agent starts k3s-agent. Its load balancer dials the three persisted server addresses on Main, on-link, and retries until one answers. The gateway is not in this path.
  5. kubelet on every node reads its pods from the API over loopback and starts the Cilium agent. Today the agent dials 192.168.20.140, which exists only after kube-vip on a server has won the lease. kube-vip wins the lease through 127.0.0.1:6443, so the chain closes, but the agent of a server depends on another server’s Main leg by ARP. After this change the agent dials 127.0.0.1:6444 and depends on its own node only.
  6. Each Cilium agent opens its two BGP sessions to 192.168.200.1 and fddb:f665:73f7:4::1 from the Lab leg. If the gateway is not up, the agent retries. Every other function of the agent runs without the sessions.
  7. The gateway installs a host route for each VIP whose announcing node has a session. The API VIP is up when one server has a Cilium agent with an established session.

A reboot of one control-plane node takes its sessions down within the hold time of 9 seconds. The gateway keeps the routes of that node for restartTimeSeconds, 60 seconds, because graceful restart is on. For an API VIP with three next hops, up to one third of new connections go to the rebooting node in that window and fail. After the window the route has two next hops. Today a reboot of the kube-vip leader moves the VIP within the 15 second lease, and a reboot of a non-leader costs nothing. The drill in PR 1 measures the window.

A reboot of the gateway takes every session down. No node dials the VIP, so no node notices. The kubelets, the Cilium agents and ArgoCD run on. The operator loses the API VIP, every routed VIP and every inter-leg path for the reboot time. When the gateway returns, each agent reconnects, and the route returns within the connect retry of Cilium. The operator can dial a server address on Main from a Main client meanwhile, because every node address is a SAN of the certificate.

Speaker Path of a Main client What it needs Why not
Cilium BGP announces a LoadBalancer Service for the API server Gateway, Lab leg of a server Cilium’s own path moves to loopback first. One Service, two EndpointSlices, one pool. No FRR change: the import filter already permits 192.168.250.0/24 ge 32, the reply rule at 6100 covers the prefix, and the tailnet advertises it It is the pick
kube-vip in BGP mode Gateway, then a server A second BGP speaker on each server. From the Lab address it collides with the Cilium session, because FRR holds one neighbor per address. From the Main address it needs three new neighbors in tf/unifi/bgp.tf, and bgp-config still has the two unmodelled keys error and redistribute_to_sd_wan, so the file changes only through the UI with an adopt Two BGP stacks and a UI step for every FRR change
kube-vip keeps ARP on the Lab leg at 192.168.200.140 Gateway, then on-link Lab A tailnet route for 192.168.200.140/32, against the rule that Lab is not on the tailnet. A Lab client stays on-link with the VIP It keeps a VIP on a leg, which lb-vips-over-bgp removed everywhere else

Goals:

  • The Kubernetes API answers at 192.168.250.254 and fddb:f665:73f7:250::254, announced over BGP by the three servers, with no new FRR text on the gateway.
  • No node needs the API VIP. Every path from a node to the API is loopback.
  • The move is reversible at each pull request, and the old VIP answers until the last one.

Non-Goals:

  • A health check of kube-apiserver behind the VIP. See D3.
  • A change to the graceful restart timer. It is shared by every VIP. The drill measures it first.
  • The retirement of the 10.255.254.x convention beyond the one value this change touches.
  • A test of a tailnet path. Sean ruled on 2026-09-23 that the tailnet proof is the configuration.

The API VIP is a LoadBalancer Service that Cilium announces, like every other VIP. kube-vip goes. The table above holds the alternatives and their costs. The cold start closes because no node dials the VIP after D2: the servers announce the VIP, and only clients outside the nodes consume it.

D2. Cilium reaches the API server on loopback

Section titled “D2. Cilium reaches the API server on loopback”

cilium_k8s_service_host becomes 127.0.0.1 and cilium_k8s_service_port becomes 6444. The chart puts both into KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT of the Cilium agent, the operator and cilium-envoy. All three run with hostNetwork: true, so loopback is the node. No other kube-system workload carries the variable.

On a server, 127.0.0.1:6444 is kube-apiserver. On an agent it is the k3s load balancer, which fails over between the servers and already carries the kubelet. The certificate behind both carries 127.0.0.1, and the probe with the cluster CA returned 200 on each. The Cilium documentation prescribes the same loopback path for RKE2, where the agent proxy listens on 6443.

Alternative considered: k8s.apiServerURLs with the three server addresses. Cilium then fails over by itself. Rejected, because the k3s load balancer already does that on the agents with its own health checks, and the list is a second place that holds the server addresses.

Trade-off: on a server whose kube-apiserver is down, the Cilium agent of that server has no API. The kubelet of that server is in the same state, because k3s is one process, and systemd restarts it after 5 seconds (Restart=always).

D3. One selector-less Service with hand-written EndpointSlices

Section titled “D3. One selector-less Service with hand-written EndpointSlices”

kube-apiserver in k3s is not a pod, so no selector can find it. argocd/app-configs/cilium-lb/ gains api-vip.yaml with the pool and the Service, and the cilium role applies the slices:

  • A CiliumLoadBalancerIPPool api-vip with two blocks, start and stop both 192.168.250.254, and both fddb:f665:73f7:250::254. Its serviceSelector matches io.kubernetes.service.namespace: kube-system. The pool routed-vips keeps its namespace list and its ranges, so the runbook text about it stays true.
  • A Service kube-system/kubernetes-api of type LoadBalancer, class io.cilium/bgp-control-plane, externalTrafficPolicy: Local, ipFamilyPolicy: RequireDualStack, port 6443 to target port 6443, and lbipam.cilium.io/ips pinned to the two addresses. It has no selector.
  • Two EndpointSlices, one per address type, labelled kubernetes.io/service-name: kubernetes-api and endpointslice.kubernetes.io/managed-by: fzymgc.house. Each endpoint holds the Main address of one server in that family, its nodeName, and conditions.ready: true. The IPv6 addresses are the Main ULAs fddb:f665:73f7:1:aaaa::141 to ::143, which the certificate carries and on which k3s-server listens. The cilium Ansible role applies the slices, from the template api-vip-endpointslices.yaml.j2 and the host vars of tp_cluster_controlplane, the way it applies the Gateway API CRDs. ArgoCD 3.x excludes the EndpointSlice kind by default in argocd-cm, and an excluded kind is neither synced nor reported, so a slice in api-vip.yaml never reached the cluster while the app showed Synced. Sean chose the role over an override of resource.exclusions on 2026-10-17: the slices change only with the control-plane set, the server list then has one source, and ArgoCD does not start to watch every EndpointSlice of the cluster.

With externalTrafficPolicy: Local, Cilium announces the VIP only from a node that holds a ready endpoint, so only the three servers announce. On a server, the datapath sends a packet for the VIP to the server’s own Main address on port 6443, and the reply leaves from the VIP through table 1001 to the Lab gateway, as every other routed VIP does.

The default/kubernetes Service was rejected. The API server reconciles it, and its type stays ClusterIP.

Trade-off: the EndpointSlices are static, so a server whose kube-apiserver is down but whose Cilium agent is up keeps announcing. The window is the systemd restart of D2. The operator fallback is a server address on Main.

D4. The consumers move in three pull requests

Section titled “D4. The consumers move in three pull requests”

PR 1 makes the new address answer beside the old one:

  • k8s_cluster_sans gains 192.168.250.254 and fddb:f665:73f7:250::254 in both files that hold the list. The k3s-config tag rolls the servers one at a time with the readiness gate.
  • D2 and D3. The Cilium converge runs before the merge, so the agents already use loopback when the Service appears.
  • docs/reference/network.md changes the row of .254 from reserved to the Service, and drops .200.140 reserved for kube-vip from the Lab row.

PR 2 moves every client:

  • The operator changes the dns-control entry k8s-api to 192.168.250.254 and adds the AAAA fddb:f665:73f7:250::254. k8s-cluster.fzymgc.house is a CNAME to it, so the operator kubeconfigs follow.
  • scripts/k8s-login.sh dials https://k8s-cluster.fzymgc.house:6443, the same name as the admin kubeconfig.
  • k8s_cluster_endpoint_ip becomes 192.168.250.254 in both group_vars files. A new agent can join again. The existing agents keep the dead K3S_URL in their unit environment, because the install script writes it once and the persisted server list serves them. The runbook names the one-line fix for an agent that must be re-pointed.
  • docs/getting-started/cluster-access.md, docs/operations/cilium-kpr.md, README.md and docs/reference/index.md name the new address.

PR 3 tears down kube-vip:

  • The kube-vip role gains kube_vip_state. With absent, it removes the static pod manifest, waits for the pod to leave, removes 192.168.20.140/32 from end0 if it is still bound, and deletes the Lease kube-system/plndr-cp-lock.
  • kube_vip_address leaves ansible/inventory/group_vars/all.yml, and 192.168.20.140 and 10.255.254.6 leave k8s_cluster_sans.
  • A fourth pull request deletes the role, its play and kube_vip_state after the converge, the same two-step shape as the MetalLB removal.

The tailnet needs no change. 192.168.250.0/24 and fddb:f665:73f7:250::/64 are advertised, and the admin grant covers lb-vips:* and lb-vips-v6:*. No other grant reaches 6443 today.

The fence needs no change. The VIP prefix sits behind the Lab leg, in the zone Internal with Main, MainWireless, Core, IoT and Lab. Guest and Telework are blocked from Internal already, and the lb-vips-over-bgp probes measured that for the prefix. PR 1 repeats the probe on port 6443.

  • [The gateway reboots, and the operator loses the API VIP for that time] → Every node runs on. The operator dials a server address on Main, which is a SAN. This is the new dependency that the proposal names, and Sean accepts it with D1.
  • [A control-plane node reboots, and graceful restart keeps its next hop for 60 seconds] → Up to a third of new connections fail in the window. The drill in PR 1 times the window, and a change to restartTimeSeconds is a follow-on, because it is shared by every VIP.
  • [kube-apiserver dies on a server whose Cilium agent lives] → The static endpoint keeps the route. systemd restarts k3s in 5 seconds. The operator fallback is the same as above.
  • [The agents keep a dead K3S_URL] → They never use it after the first join. The runbook holds the fix for an agent that must be re-pointed.
  • [A client keeps 192.168.20.140 as a literal] → PR 3 waits until no server sees a connection to the old VIP from outside the nodes. The gate reads the Cilium connection-tracking table on the kube-vip leader, as task 3.1 of lb-vips-over-bgp did.
  • [The pool api-vip serves every LoadBalancer Service in kube-system] → It holds two addresses, and the Service pins both, so a second Service gets none.
  • [The Cilium converge of D2 restarts every agent] → It is the same rolling restart as every Cilium converge, and the agents reconnect to the API over loopback.

Each step names its rollback. A revert pull request merges like any other.

PR 1, the new address answers:

  1. The operator rolls the SAN list onto the servers from the branch: uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags k3s-config --limit tp_cluster_controlplane. Rollback: check out origin/main and run the same command.
  2. The operator converges Cilium from the branch: uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium-install. Rollback: check out origin/main and run the same command. The agents dial the kube-vip VIP again.
  3. The merge syncs api-vip.yaml. The three servers announce the VIP. Rollback: git revert -m 1 <merge commit of PR 1>, merged as a pull request. ArgoCD prunes the objects and the routes withdraw.
  4. The proofs: /readyz through the VIP from a Mac on wired Main, in both families, the route on the gateway with three next hops, the fence probe on 6443 from Guest and Telework, and the two drills that Sean approves, a reboot of one control-plane node and a reboot of the gateway.

PR 2, the move:

  1. The operator changes the k8s-api entry in dns-control and waits for the TTL of 300 seconds. Rollback: set the entry back to 192.168.20.140 and remove the AAAA.
  2. The merge changes scripts/k8s-login.sh, the inventory and the docs. Rollback: git revert -m 1 <merge commit of PR 2>, merged as a pull request.

PR 3, teardown:

  1. The gate: no connection to 192.168.20.140:6443 from outside the nodes in the connection tracking table of the kube-vip leader.
  2. The merge sets kube_vip_state: absent and trims the SAN list. The operator runs uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags kube-vip,k3s-config --limit tp_cluster_controlplane. Rollback: git revert -m 1 <merge commit of PR 3>, merged as a pull request, then the same command. kube-vip returns and binds the old VIP within its lease.
  3. The cold-start drill, after Sean approves it. It needs no operator action between power-on and ok.
  • The length of the window after a control-plane reboot. The drill of 2026-10-17 measured it: the gateway kept the next hop of the rebooting node for about 60 seconds, and no probe through the VIP failed in that window at a 5 second cadence. The timer stays at 60 seconds.
  • Whether kube-vip removes the VIP from end0 on SIGTERM. The teardown task removes the address itself if it remains, so the answer changes no task.