Design: Main leaves the tailnet
Context
Section titled “Context”See proposal.md, section Why, for the motivation. Every fact below was read on 2026-10-18 from the
live controller, the repository and the tailnet policy, unless a sentence names another source.
The advertisers and what the Main route serves
Section titled “The advertisers and what the Main route serves”truenas-scale, tpi-alpha-4 and tpi-beta-4 advertise 192.168.20.0/22, 192.168.40.0/22,
192.168.250.0/24 and fddb:f665:73f7:250::/64. The lists are nas_tailscale_advertise_routes
and k8s_tailscale_advertise_routes, and tailscale/policy.hujson auto-approves the same four
for tag:nas and tag:subnet-router. The runbook docs/operations/tailscale-subnet-routers.md
describes the failover group.
| Target on Main | Path today | Path after this change |
|---|---|---|
Home Assistant VM ha, 192.168.20.220 |
The Main route. ha.fzymgc.net is not on the Cloudflare tunnel by decision of 2026-10-05 |
The VM is a tailnet node (D1) |
nas-support, 192.168.20.202, vault-unseal-nas, 192.168.20.204 |
The Main route, only while a worker is primary (truenas-scale cannot reach its own macvlan children) |
Core addresses on a Core VLAN interface of the NAS (D2) |
tpi-alpha-bmc, 192.168.22.140, tpi-beta-bmc, 192.168.23.229 |
The Main route | Core, after the uplinks go native Core and the nodes tag Main (D3) |
The node Main legs, 192.168.20.141 to .154 |
The Main route, grant network-main:* |
heimdall. tpi-alpha-4 and tpi-beta-4 are tailnet nodes |
The NAS host, 192.168.20.200, alias nas-main |
The Main route | truenas-scale is a tailnet node |
The gateway, alias router-main, port 53 |
Stale. Split DNS points at the tailnet addresses of the resolvers (tf/tailscale/dns.tf) |
The grant goes |
The aliases vault1 to vault3 name 192.168.20.6 to .8, which are the Main legs of the
resolvers. Vault runs in the cluster. tp-alpha-1 to tp-beta-4 name the node Main legs and no
grant uses them.
The NAS attachment
Section titled “The NAS attachment”nas1 is a link aggregation on switch-agg-main ports 3 and 4, Main native, excluding only WAN2
(tf/unifi/ports-agg-main.tf). Core is tagged on it. The NAS already runs a VLAN interface on
bond0 for Lab: the MAC 3c:ec:ef:fa:66:ae holds the Lab lease 192.168.200.162. nas-support,
vault-unseal-nas and precipice are macvlan children of bond0
(nas_sandbox_nic_parent, nas_game_nic_parent in ansible/inventory/host_vars/nas.yml).
Macvlan over bond0 blocks traffic between the host and its children, which is why the NAS as
primary advertiser cannot forward to them.
The Turing Pi uplinks and the BMC
Section titled “The Turing Pi uplinks and the BMC”Both boards hang on Main-native trunks of switch-office: port 12 for tpi-alpha and port 11
for tpi-beta, both
port 11 on the profile all_main_default. Each profile excludes only WAN2 and MainWireless, so Core
is tagged on both. Each board carries an onboard switch, a Realtek RTL8365MB, that joins the BMC
and the four nodes to the uplink. The nodes take Main untagged on end0 and Lab tagged on
end0.1000. The BMC takes the untagged VLAN.
Stock BMC firmware cannot put the BMC on its own VLAN. The switch driver has no VLAN filtering, and
turing-machines/BMC-Firmware#196 is open on it. Upstream last released v2.1.0 in February 2025 and
last pushed in September 2025. The fork excavador-turing/BMC-Firmware (turingpi.xyz) adds a
switch page with per-port VLAN configuration, boots a new image on trial and rolls back on its own,
leaves the node rails alone during a BMC update, and tracks kernel 6.12. On 2026-10-18 it had 28
stars, one fork, one open issue, one maintainer with 145 commits over upstream, and 39 releases
since 2026-09-08. tpi-alpha-bmc runs an upstream development build, commit d9c55b7a of
2025-09-02 built on 2025-10-04, with the daemon at 2.3.4. tpi-beta-bmc could not be read,
because Vault holds no bmc/tpi-beta password. The API login is a JSON POST to
/api/bmc/authenticate, and the certificate is self-signed, so a read pins the public key.
The macOS client
Section titled “The macOS client”Sean’s Mac accepts tailnet routes. Tailscale documents that macOS picks the longest prefix among
the accepted routes and the connected networks, and that the fix for an overlap is a route shorter
than the LAN prefix. Two equal /22 prefixes have no shorter one. tailscale/tailscale#14995
records the failure mode: the accepted route wins and a reply to a LAN connection leaves through
the tunnel.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- No prefix inside
192.168.16.0/20on any advertiser or inautoApprovers. - Every host that a remote operator needs is a tailnet node or holds a Core address.
- Every alias in the policy names an address that exists.
Non-Goals:
- The collapse of MainWireless into Main. That is the change
mainwireless-collapse, which requires this one. - A tailnet path to Lab.
- Moving
precipiceoff Main. The tailnet reaches it by its own tag. - Pinning the NAS Lab lease as a static. See Open Questions.
Decisions
Section titled “Decisions”D1: Home Assistant runs the Tailscale add-on and publishes the Tailscale Service svc:ha
Section titled “D1: Home Assistant runs the Tailscale add-on and publishes the Tailscale Service svc:ha”Home Assistant Core serves HTTPS on port 443 itself, with the Let’s Encrypt certificate for
ha.fzymgc.house, and the home-assistant-host spec requires that. The add-on’s
share_homeassistant option is a Tailscale Serve proxy to Home Assistant over plain HTTP, and its
documentation requires SSL/TLS off in Home Assistant. So the node publishes a Tailscale Service
instead. A Service is a named resource of the tailnet with its own virtual addresses, a stable
MagicDNS name and a certificate from Tailscale, served by one or more tagged nodes.
The HAOS VM runs the add-on hassio-addons/addon-tailscale with this configuration:
| Option | Value | Reason |
|---|---|---|
advertise_tags |
["tag:homeassistant"] |
One tag, owned by autogroup:admin in tagOwners. A Service host must be tagged |
advertise_routes |
[], stated |
The VM is never a router. An empty list is stated for the same reason the NAS and worker roles always emit --advertise-routes= |
advertise_exit_node |
false |
truenas-scale is the only exit node |
userspace_networking |
false |
The add-on turns its Service proxy off in userspace mode. The VM gets a tailscale0 interface and can reach other tailnet nodes, which nothing on it needs |
accept_routes |
false |
The VM is on Main and IoT and needs no tailnet route |
share_homeassistant |
disabled |
Serve needs Home Assistant on HTTP |
services |
one entry: name: svc:ha, target: https+insecure://127.0.0.1:443, protocol: https, port: 443 |
The proxy terminates TLS for https://ha.hound-skate.ts.net with a Tailscale certificate and dials Home Assistant over its own HTTPS. +insecure skips the certificate check on the loopback hop, because the Let’s Encrypt certificate names ha.fzymgc.house and not 127.0.0.1 |
tf/tailscale defines the Service with the resource tailscale_service: name = "svc:ha",
ports = ["tcp:443"], tags = ["tag:homeassistant"]. The policy gains
"tag:homeassistant": ["autogroup:admin"] in tagOwners,
"svc:ha": ["tag:homeassistant"] under autoApprovers.services, and a grants list with one
entry from group:admin and autogroup:admin to svc:ha on port 443. A Service is a grants
destination and not an acls destination, so the policy carries both lists from here on. The
add-on logs in once through the link it prints, as the admin user. No OAuth client is created: the
NAS and worker roles need one because Ansible re-registers them, and the add-on keeps its state in
the VM. The add-on registers the node under the name homeassistant, and no option of the add-on
changes that. The Service proxy is a reverse proxy: it adds an X-Forwarded-For header to each
request it dials to the loopback. Home Assistant answers 400: Bad Request to a request that
carries that header from an address it does not trust. So the HTTP server settings of Home
Assistant, under Settings, System, Network, trust X-Forwarded-For and list 127.0.0.1 as a
trusted proxy. This Home Assistant version keeps those settings in the UI and ignores an http:
block in configuration.yaml. The trust
covers the loopback only. Home Assistant reads the tailnet address of the client from the header
on a request from the add-on, and it answers 400 to a request from any other address that carries
the header. A request without the header is not affected. The companion app keeps Home Assistant
Cloud as its Internet URL, as the runbook already states. The add-on configuration and the
HTTP server settings are entered in the Home Assistant UI, because the HAOS VM has no
Ansible-managed interior, and docs/operations/home-assistant.md records them.
Alternatives considered. Serve with share_homeassistant: ssl: rejected, because it needs Home
Assistant on HTTP and a second proxy to keep https://ha.fzymgc.house on the LAN. A node with no
Service: works, but the operator reaches the VM by its tailnet address with a certificate that
names another host. A /32 route for 192.168.20.220 on the three advertisers: rejected, because
a /32 through the tunnel beats the connected /22 on every client that accepts routes, so the
hairpin becomes certain, and a Main prefix stays on the advertisers. Nabu Casa remote UI: it stays
the Internet URL of the companion app and is not a path for an operator.
Rollback: disable the add-on in the Home Assistant UI, and revert the pull request, which removes the Service, the tag, the auto-approver and the grant. Nothing else depends on the node.
D2: The NAS gets a Core VLAN interface with no address, and the two containers move to it
Section titled “D2: The NAS gets a Core VLAN interface with no address, and the two containers move to it”A VLAN interface vlan1 for Core on bond0, created through the middleware (interface.create,
type VLAN, parent bond0, tag 1), with no address, DHCP off and IPv6 autoconfiguration off.
Core is the untagged default network of the controller (unifi_network.core states no VLAN id),
and a trunk whose native network is another one carries the default network as VLAN 1. The port
nas1 is native Main, so Core reaches the NAS as VLAN 1, the same way Lab reaches it as
vlan1000. docs/reference/network.md allocates the container addresses first:
192.168.40.32 for nas-support and 192.168.40.34 for vault-unseal-nas. The Core statics in
use are .40.1, .40.2, .40.5 to .40.7, .40.20, .40.21, .40.25 and .40.199, and the
Kea pool starts at 192.168.41.10.
The host holds no Core address. A macvlan parent never passes a frame from the host’s own stack
to a child on it, which is why truenas-scale cannot reach nas-support on Main today. A host
address on vlan1 routes Core on-link and recreates that block on Core. Without one, the host
routes Core through the gateway, the gateway delivers to Core, and the frame arrives on vlan1
from the wire, where macvlan hands it to the child. So the tailnet path through truenas-scale
works, and the host still cannot reach its own containers, as the NAS runbook already states.
The interface exists only as the macvlan parent of the two containers.
nas_sandbox_nic_parent and the vault-unseal parent become vlan1. The container roles add a
NIC device only when the container has none, so each gains a task that updates the device when
its nic_attach differs and restarts the container. The two Kea reservations and the two
dns-control entries move to the Core addresses. ansible_host of nas-support and
vault-unseal-nas in ansible/inventory/hosts.yml follow, and so does the Uptime Kuma monitor
of nas-support in tf/uptime-kuma. The NFS export of the Home Assistant backups and the Kopia
timers on nas-support are unaffected: the export admits the VM, not the container, and the
timers dial the NAS by name.
Why not Lab: Lab is not on the tailnet, so a Lab address gives the containers no remote path.
Why not a bridge: a bridge br-core over vlan1 with a host address lets the host reach the
containers. The estate has no need for that path, and a bridge is a second interface to keep. A
bridge stays the option if a host-to-container path is needed later.
Rollback: a converge of nas-lxc-sandbox and vault-unseal with the parents back on bond0,
the entries back on Main, then midclt call interface.delete vlan1 and interface.commit.
D3: The uplinks go native Core, and the nodes tag Main themselves
Section titled “D3: The uplinks go native Core, and the nodes tag Main themselves”The onboard switch is VLAN-unaware. It passes a tagged frame untouched, which end0.1000 on every
node proves today, and it puts every untagged frame on the native VLAN of the uplink. So the
native VLAN of the uplink decides where the BMC lands, and each node decides its own VLAN by
tagging. The BMC cannot tag. The nodes can.
The change has two halves. The netplan template of tp2-bootstrap-node renders Main as the VLAN
sub-interface end0.3000, with the same addresses, the same on-link default route, the same RA
and DNS settings that end0 holds today, and it leaves end0 bare with DHCP and RA off. The
template names the physical interface from a variable, tp2_node_interface, instead of
ansible_default_ipv4.interface, because the default interface becomes end0.3000 after the
change. The two uplink ports, switch-office port 12 for alpha and port 11 for beta, take the port
profile TpiUplink in tf/unifi: Core native, Main and Lab tagged, every other network
excluded. A profile, not the port’s own keys, because the provider never clears the profile of a
port that moves to its own keys, and the controller then keeps the old profile and ignores the
keys. The BMC then lands on Core untagged.
docs/reference/network.md allocates 192.168.40.40 for tpi-alpha-bmc and 192.168.40.41
for tpi-beta-bmc. The dns-control entry tpi_alpha_bmc moves. tpi_beta_bmc does not exist
and is created, for the MAC c4:ff:84:10:0b:e7, which holds a Main pool lease today. The tailnet
tag tag:tp-bmc keeps its grant, and docs/operations/vault.md keeps the password path.
The template renders the tagged shape only when tp2_node_main_tagged is true, which the
group_vars of each board set in the same pull request as its port change. The default is false,
so a converge of the bootstrap playbook changes nothing on a board whose port has not flipped.
The order on one board. The pull request of the board sets the variable and the port. The port
flips when a person confirms the main-cluster-unifi run. At that moment the untagged Main leg
of each node stops and the BMC appears on Core. The Lab leg of each node is tagged and keeps
working, so Ansible reaches each node at its Lab address, renders the tagged Main leg and
applies it, one node at a time. If a node is lost, the BMC serial console (tpi uart -n <slot>)
is the recovery path, and the BMC is on Core by then.
Before the first flip, cilium_devices of the cilium role lists end0.3000 and the role
converges, because Cilium’s VLAN filter on end0 drops a tagged frame for a VLAN device it does
not own, and a restart of the agent never adopts an unlisted device. A listed device that does
not exist yet is selected when it appears. After each node, kubectl get nodes -o wide shows the
same INTERNAL-IP. tpi-beta goes first, because it holds
workers only. tpi-alpha follows after beta runs for a day, because it holds the control plane.
A rebuilt node boots with the untagged DHCP configuration of the image, so it takes a Core lease until the role renders the tagged configuration. Core has Kea, and the role connects by the address the operator passes it, so the bootstrap path still works. The runbook states it.
Rollback of one board: the port back to native Main, and netplan back through the Lab leg.
Why not the fork firmware: excavador-turing/BMC-Firmware can put the BMC port on its own VLAN
without touching the nodes, and the security review at
docs/engineering/reviews/2026-10-18-bmc-firmware-fork-security-review.md found it usable with
mitigations. It also found that the fork is one GitHub account with unsigned releases, and that
adopting it adds a trust and a flash risk on two boards. The native flip is a change to gear this
repository already manages, and it adds no trust. The fork stays the fallback if the flip fails
for a reason the Lab leg does not cover.
D4: The route leaves after the hosts have their new paths
Section titled “D4: The route leaves after the hosts have their new paths”Pull request 4 of the proposal removes 192.168.20.0/22 from the two role defaults and from
autoApprovers.routes, deletes the grants and aliases of the proposal, and the operator converges
ansible/tailscale-subnet-routers-playbook.yml and scripts/nas-playbook.sh --tags nas-tailscale.
The three lists change in one pull request so that the failover group never disagrees. The
main-cluster-tailscale workspace auto-applies the policy on merge.
Rollback: revert the pull request and converge both roles again. The route returns on the next
tailscale up of each advertiser.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [Home Assistant remote access depends on the add-on] → heimdall on Core is the fallback for an operator. Local control and Apple Home do not depend on the tailnet.
- [The add-on advertises Main and IoT by mistake] →
advertise_routes: []is stated, and the scenario “The node advertises nothing” readsPrimaryRoutesasnull. - [Tailscale Services are a beta feature] → The Service is one resource and one add-on entry. If the feature changes, the fallback is the node without a Service, reached by its tailnet address.
- [The port flip cuts the four nodes of a board off Main at once] → The Lab leg of each node is tagged and survives the flip, so Ansible applies the tagged Main leg through it, one node at a time. The BMC serial console is the last resort, and the BMC is on Core by then. Beta goes first, because it holds workers only.
- [A node with a bad netplan loses both legs] → The netplan is rendered and read before the flip, and the first node of beta proves it. The uplink switch accepts tagged frames for Main today, because it carries Lab tagged through the same onboard switch.
- [Cilium drops the tagged Main leg] →
end0.3000is incilium_devicesand the role converges before the first flip, the same wayend0.1000was listed for Lab.kubectl get nodes -o widereads the sameINTERNAL-IPafter each node. - [The failover group disagrees on routes between converges] → All three lists change in one pull request, and the operator converges the workers and the NAS in the same window.
- [A grant that names a Main address survives] → The acceptance command of pull request 4 greps
the policy and both role defaults for any
192.168.16.0/20address. - [A tailnet client keeps a cached Main route] → A client reads routes from the control plane on
each change. The scenario reads
PrimaryRoutesfrom a client, not from an advertiser.
Migration Plan
Section titled “Migration Plan”The proposal’s pull request split is the plan. The order is Home Assistant, the NAS, the BMC gates, then the route. The route leaves last so that every host has its new path before the old one goes. Each pull request names its rollback in the decision above.
Open Questions
Section titled “Open Questions”- Does any consumer depend on the NAS Lab lease
192.168.200.162? If the k3s nodes mount storage over Lab, the address becomes a static in192.168.200.145to.149in pull request 2. If nothing names it, the lease stays as it is. The answer does not change the specs. - Does the uplink switch accept a frame tagged with its own native VLAN? If it does, a node can take the tagged Main leg before the port flips, and the flip costs no reach at all. The first node of beta answers it. If it does not, the Lab leg carries the apply, as D3 states.