Skip to content

Design: Main leaves the tailnet

See proposal.md, section Why, for the motivation. Every fact below was read on 2026-10-18 from the live controller, the repository and the tailnet policy, unless a sentence names another source.

The advertisers and what the Main route serves

Section titled “The advertisers and what the Main route serves”

truenas-scale, tpi-alpha-4 and tpi-beta-4 advertise 192.168.20.0/22, 192.168.40.0/22, 192.168.250.0/24 and fddb:f665:73f7:250::/64. The lists are nas_tailscale_advertise_routes and k8s_tailscale_advertise_routes, and tailscale/policy.hujson auto-approves the same four for tag:nas and tag:subnet-router. The runbook docs/operations/tailscale-subnet-routers.md describes the failover group.

Target on Main Path today Path after this change
Home Assistant VM ha, 192.168.20.220 The Main route. ha.fzymgc.net is not on the Cloudflare tunnel by decision of 2026-10-05 The VM is a tailnet node (D1)
nas-support, 192.168.20.202, vault-unseal-nas, 192.168.20.204 The Main route, only while a worker is primary (truenas-scale cannot reach its own macvlan children) Core addresses on a Core VLAN interface of the NAS (D2)
tpi-alpha-bmc, 192.168.22.140, tpi-beta-bmc, 192.168.23.229 The Main route Core, after the uplinks go native Core and the nodes tag Main (D3)
The node Main legs, 192.168.20.141 to .154 The Main route, grant network-main:* heimdall. tpi-alpha-4 and tpi-beta-4 are tailnet nodes
The NAS host, 192.168.20.200, alias nas-main The Main route truenas-scale is a tailnet node
The gateway, alias router-main, port 53 Stale. Split DNS points at the tailnet addresses of the resolvers (tf/tailscale/dns.tf) The grant goes

The aliases vault1 to vault3 name 192.168.20.6 to .8, which are the Main legs of the resolvers. Vault runs in the cluster. tp-alpha-1 to tp-beta-4 name the node Main legs and no grant uses them.

nas1 is a link aggregation on switch-agg-main ports 3 and 4, Main native, excluding only WAN2 (tf/unifi/ports-agg-main.tf). Core is tagged on it. The NAS already runs a VLAN interface on bond0 for Lab: the MAC 3c:ec:ef:fa:66:ae holds the Lab lease 192.168.200.162. nas-support, vault-unseal-nas and precipice are macvlan children of bond0 (nas_sandbox_nic_parent, nas_game_nic_parent in ansible/inventory/host_vars/nas.yml). Macvlan over bond0 blocks traffic between the host and its children, which is why the NAS as primary advertiser cannot forward to them.

Both boards hang on Main-native trunks of switch-office: port 12 for tpi-alpha and port 11 for tpi-beta, both port 11 on the profile all_main_default. Each profile excludes only WAN2 and MainWireless, so Core is tagged on both. Each board carries an onboard switch, a Realtek RTL8365MB, that joins the BMC and the four nodes to the uplink. The nodes take Main untagged on end0 and Lab tagged on end0.1000. The BMC takes the untagged VLAN.

Stock BMC firmware cannot put the BMC on its own VLAN. The switch driver has no VLAN filtering, and turing-machines/BMC-Firmware#196 is open on it. Upstream last released v2.1.0 in February 2025 and last pushed in September 2025. The fork excavador-turing/BMC-Firmware (turingpi.xyz) adds a switch page with per-port VLAN configuration, boots a new image on trial and rolls back on its own, leaves the node rails alone during a BMC update, and tracks kernel 6.12. On 2026-10-18 it had 28 stars, one fork, one open issue, one maintainer with 145 commits over upstream, and 39 releases since 2026-09-08. tpi-alpha-bmc runs an upstream development build, commit d9c55b7a of 2025-09-02 built on 2025-10-04, with the daemon at 2.3.4. tpi-beta-bmc could not be read, because Vault holds no bmc/tpi-beta password. The API login is a JSON POST to /api/bmc/authenticate, and the certificate is self-signed, so a read pins the public key.

Sean’s Mac accepts tailnet routes. Tailscale documents that macOS picks the longest prefix among the accepted routes and the connected networks, and that the fix for an overlap is a route shorter than the LAN prefix. Two equal /22 prefixes have no shorter one. tailscale/tailscale#14995 records the failure mode: the accepted route wins and a reply to a LAN connection leaves through the tunnel.

Goals:

  • No prefix inside 192.168.16.0/20 on any advertiser or in autoApprovers.
  • Every host that a remote operator needs is a tailnet node or holds a Core address.
  • Every alias in the policy names an address that exists.

Non-Goals:

  • The collapse of MainWireless into Main. That is the change mainwireless-collapse, which requires this one.
  • A tailnet path to Lab.
  • Moving precipice off Main. The tailnet reaches it by its own tag.
  • Pinning the NAS Lab lease as a static. See Open Questions.

D1: Home Assistant runs the Tailscale add-on and publishes the Tailscale Service svc:ha

Section titled “D1: Home Assistant runs the Tailscale add-on and publishes the Tailscale Service svc:ha”

Home Assistant Core serves HTTPS on port 443 itself, with the Let’s Encrypt certificate for ha.fzymgc.house, and the home-assistant-host spec requires that. The add-on’s share_homeassistant option is a Tailscale Serve proxy to Home Assistant over plain HTTP, and its documentation requires SSL/TLS off in Home Assistant. So the node publishes a Tailscale Service instead. A Service is a named resource of the tailnet with its own virtual addresses, a stable MagicDNS name and a certificate from Tailscale, served by one or more tagged nodes.

The HAOS VM runs the add-on hassio-addons/addon-tailscale with this configuration:

Option Value Reason
advertise_tags ["tag:homeassistant"] One tag, owned by autogroup:admin in tagOwners. A Service host must be tagged
advertise_routes [], stated The VM is never a router. An empty list is stated for the same reason the NAS and worker roles always emit --advertise-routes=
advertise_exit_node false truenas-scale is the only exit node
userspace_networking false The add-on turns its Service proxy off in userspace mode. The VM gets a tailscale0 interface and can reach other tailnet nodes, which nothing on it needs
accept_routes false The VM is on Main and IoT and needs no tailnet route
share_homeassistant disabled Serve needs Home Assistant on HTTP
services one entry: name: svc:ha, target: https+insecure://127.0.0.1:443, protocol: https, port: 443 The proxy terminates TLS for https://ha.hound-skate.ts.net with a Tailscale certificate and dials Home Assistant over its own HTTPS. +insecure skips the certificate check on the loopback hop, because the Let’s Encrypt certificate names ha.fzymgc.house and not 127.0.0.1

tf/tailscale defines the Service with the resource tailscale_service: name = "svc:ha", ports = ["tcp:443"], tags = ["tag:homeassistant"]. The policy gains "tag:homeassistant": ["autogroup:admin"] in tagOwners, "svc:ha": ["tag:homeassistant"] under autoApprovers.services, and a grants list with one entry from group:admin and autogroup:admin to svc:ha on port 443. A Service is a grants destination and not an acls destination, so the policy carries both lists from here on. The add-on logs in once through the link it prints, as the admin user. No OAuth client is created: the NAS and worker roles need one because Ansible re-registers them, and the add-on keeps its state in the VM. The add-on registers the node under the name homeassistant, and no option of the add-on changes that. The Service proxy is a reverse proxy: it adds an X-Forwarded-For header to each request it dials to the loopback. Home Assistant answers 400: Bad Request to a request that carries that header from an address it does not trust. So the HTTP server settings of Home Assistant, under Settings, System, Network, trust X-Forwarded-For and list 127.0.0.1 as a trusted proxy. This Home Assistant version keeps those settings in the UI and ignores an http: block in configuration.yaml. The trust covers the loopback only. Home Assistant reads the tailnet address of the client from the header on a request from the add-on, and it answers 400 to a request from any other address that carries the header. A request without the header is not affected. The companion app keeps Home Assistant Cloud as its Internet URL, as the runbook already states. The add-on configuration and the HTTP server settings are entered in the Home Assistant UI, because the HAOS VM has no Ansible-managed interior, and docs/operations/home-assistant.md records them.

Alternatives considered. Serve with share_homeassistant: ssl: rejected, because it needs Home Assistant on HTTP and a second proxy to keep https://ha.fzymgc.house on the LAN. A node with no Service: works, but the operator reaches the VM by its tailnet address with a certificate that names another host. A /32 route for 192.168.20.220 on the three advertisers: rejected, because a /32 through the tunnel beats the connected /22 on every client that accepts routes, so the hairpin becomes certain, and a Main prefix stays on the advertisers. Nabu Casa remote UI: it stays the Internet URL of the companion app and is not a path for an operator.

Rollback: disable the add-on in the Home Assistant UI, and revert the pull request, which removes the Service, the tag, the auto-approver and the grant. Nothing else depends on the node.

D2: The NAS gets a Core VLAN interface with no address, and the two containers move to it

Section titled “D2: The NAS gets a Core VLAN interface with no address, and the two containers move to it”

A VLAN interface vlan1 for Core on bond0, created through the middleware (interface.create, type VLAN, parent bond0, tag 1), with no address, DHCP off and IPv6 autoconfiguration off. Core is the untagged default network of the controller (unifi_network.core states no VLAN id), and a trunk whose native network is another one carries the default network as VLAN 1. The port nas1 is native Main, so Core reaches the NAS as VLAN 1, the same way Lab reaches it as vlan1000. docs/reference/network.md allocates the container addresses first: 192.168.40.32 for nas-support and 192.168.40.34 for vault-unseal-nas. The Core statics in use are .40.1, .40.2, .40.5 to .40.7, .40.20, .40.21, .40.25 and .40.199, and the Kea pool starts at 192.168.41.10.

The host holds no Core address. A macvlan parent never passes a frame from the host’s own stack to a child on it, which is why truenas-scale cannot reach nas-support on Main today. A host address on vlan1 routes Core on-link and recreates that block on Core. Without one, the host routes Core through the gateway, the gateway delivers to Core, and the frame arrives on vlan1 from the wire, where macvlan hands it to the child. So the tailnet path through truenas-scale works, and the host still cannot reach its own containers, as the NAS runbook already states. The interface exists only as the macvlan parent of the two containers.

nas_sandbox_nic_parent and the vault-unseal parent become vlan1. The container roles add a NIC device only when the container has none, so each gains a task that updates the device when its nic_attach differs and restarts the container. The two Kea reservations and the two dns-control entries move to the Core addresses. ansible_host of nas-support and vault-unseal-nas in ansible/inventory/hosts.yml follow, and so does the Uptime Kuma monitor of nas-support in tf/uptime-kuma. The NFS export of the Home Assistant backups and the Kopia timers on nas-support are unaffected: the export admits the VM, not the container, and the timers dial the NAS by name.

Why not Lab: Lab is not on the tailnet, so a Lab address gives the containers no remote path.

Why not a bridge: a bridge br-core over vlan1 with a host address lets the host reach the containers. The estate has no need for that path, and a bridge is a second interface to keep. A bridge stays the option if a host-to-container path is needed later.

Rollback: a converge of nas-lxc-sandbox and vault-unseal with the parents back on bond0, the entries back on Main, then midclt call interface.delete vlan1 and interface.commit.

Section titled “D3: The uplinks go native Core, and the nodes tag Main themselves”

The onboard switch is VLAN-unaware. It passes a tagged frame untouched, which end0.1000 on every node proves today, and it puts every untagged frame on the native VLAN of the uplink. So the native VLAN of the uplink decides where the BMC lands, and each node decides its own VLAN by tagging. The BMC cannot tag. The nodes can.

The change has two halves. The netplan template of tp2-bootstrap-node renders Main as the VLAN sub-interface end0.3000, with the same addresses, the same on-link default route, the same RA and DNS settings that end0 holds today, and it leaves end0 bare with DHCP and RA off. The template names the physical interface from a variable, tp2_node_interface, instead of ansible_default_ipv4.interface, because the default interface becomes end0.3000 after the change. The two uplink ports, switch-office port 12 for alpha and port 11 for beta, take the port profile TpiUplink in tf/unifi: Core native, Main and Lab tagged, every other network excluded. A profile, not the port’s own keys, because the provider never clears the profile of a port that moves to its own keys, and the controller then keeps the old profile and ignores the keys. The BMC then lands on Core untagged. docs/reference/network.md allocates 192.168.40.40 for tpi-alpha-bmc and 192.168.40.41 for tpi-beta-bmc. The dns-control entry tpi_alpha_bmc moves. tpi_beta_bmc does not exist and is created, for the MAC c4:ff:84:10:0b:e7, which holds a Main pool lease today. The tailnet tag tag:tp-bmc keeps its grant, and docs/operations/vault.md keeps the password path.

The template renders the tagged shape only when tp2_node_main_tagged is true, which the group_vars of each board set in the same pull request as its port change. The default is false, so a converge of the bootstrap playbook changes nothing on a board whose port has not flipped.

The order on one board. The pull request of the board sets the variable and the port. The port flips when a person confirms the main-cluster-unifi run. At that moment the untagged Main leg of each node stops and the BMC appears on Core. The Lab leg of each node is tagged and keeps working, so Ansible reaches each node at its Lab address, renders the tagged Main leg and applies it, one node at a time. If a node is lost, the BMC serial console (tpi uart -n <slot>) is the recovery path, and the BMC is on Core by then. Before the first flip, cilium_devices of the cilium role lists end0.3000 and the role converges, because Cilium’s VLAN filter on end0 drops a tagged frame for a VLAN device it does not own, and a restart of the agent never adopts an unlisted device. A listed device that does not exist yet is selected when it appears. After each node, kubectl get nodes -o wide shows the same INTERNAL-IP. tpi-beta goes first, because it holds workers only. tpi-alpha follows after beta runs for a day, because it holds the control plane.

A rebuilt node boots with the untagged DHCP configuration of the image, so it takes a Core lease until the role renders the tagged configuration. Core has Kea, and the role connects by the address the operator passes it, so the bootstrap path still works. The runbook states it.

Rollback of one board: the port back to native Main, and netplan back through the Lab leg.

Why not the fork firmware: excavador-turing/BMC-Firmware can put the BMC port on its own VLAN without touching the nodes, and the security review at docs/engineering/reviews/2026-10-18-bmc-firmware-fork-security-review.md found it usable with mitigations. It also found that the fork is one GitHub account with unsigned releases, and that adopting it adds a trust and a flash risk on two boards. The native flip is a change to gear this repository already manages, and it adds no trust. The fork stays the fallback if the flip fails for a reason the Lab leg does not cover.

D4: The route leaves after the hosts have their new paths

Section titled “D4: The route leaves after the hosts have their new paths”

Pull request 4 of the proposal removes 192.168.20.0/22 from the two role defaults and from autoApprovers.routes, deletes the grants and aliases of the proposal, and the operator converges ansible/tailscale-subnet-routers-playbook.yml and scripts/nas-playbook.sh --tags nas-tailscale. The three lists change in one pull request so that the failover group never disagrees. The main-cluster-tailscale workspace auto-applies the policy on merge.

Rollback: revert the pull request and converge both roles again. The route returns on the next tailscale up of each advertiser.

  • [Home Assistant remote access depends on the add-on] → heimdall on Core is the fallback for an operator. Local control and Apple Home do not depend on the tailnet.
  • [The add-on advertises Main and IoT by mistake] → advertise_routes: [] is stated, and the scenario “The node advertises nothing” reads PrimaryRoutes as null.
  • [Tailscale Services are a beta feature] → The Service is one resource and one add-on entry. If the feature changes, the fallback is the node without a Service, reached by its tailnet address.
  • [The port flip cuts the four nodes of a board off Main at once] → The Lab leg of each node is tagged and survives the flip, so Ansible applies the tagged Main leg through it, one node at a time. The BMC serial console is the last resort, and the BMC is on Core by then. Beta goes first, because it holds workers only.
  • [A node with a bad netplan loses both legs] → The netplan is rendered and read before the flip, and the first node of beta proves it. The uplink switch accepts tagged frames for Main today, because it carries Lab tagged through the same onboard switch.
  • [Cilium drops the tagged Main leg] → end0.3000 is in cilium_devices and the role converges before the first flip, the same way end0.1000 was listed for Lab. kubectl get nodes -o wide reads the same INTERNAL-IP after each node.
  • [The failover group disagrees on routes between converges] → All three lists change in one pull request, and the operator converges the workers and the NAS in the same window.
  • [A grant that names a Main address survives] → The acceptance command of pull request 4 greps the policy and both role defaults for any 192.168.16.0/20 address.
  • [A tailnet client keeps a cached Main route] → A client reads routes from the control plane on each change. The scenario reads PrimaryRoutes from a client, not from an advertiser.

The proposal’s pull request split is the plan. The order is Home Assistant, the NAS, the BMC gates, then the route. The route leaves last so that every host has its new path before the old one goes. Each pull request names its rollback in the decision above.

  • Does any consumer depend on the NAS Lab lease 192.168.200.162? If the k3s nodes mount storage over Lab, the address becomes a static in 192.168.200.145 to .149 in pull request 2. If nothing names it, the lease stays as it is. The answer does not change the specs.
  • Does the uplink switch accept a frame tagged with its own native VLAN? If it does, a node can take the tagged Main leg before the port flips, and the flip costs no reach at all. The first node of beta answers it. If it does not, the Lab leg carries the apply, as D3 states.