Tailscale Subnet Routers
Three tailnet nodes advertise the same three routes, so a tailnet client keeps a path to the LAN
when one advertiser goes down. The routes are 192.168.40.0/22 (Core) and the routed
LoadBalancer VIP prefixes 192.168.250.0/24 and fddb:f665:73f7:250::/64. No prefix inside the
trusted-Main supernet 192.168.16.0/20 is advertised: every host that a remote operator needs
has a Core address, a routed VIP or a tailnet node of its own, and a Main route on a client that
sits on Main duplicates an on-link prefix. This page states what exists, the procedures that
touch it, and a check for each.
What exists
Section titled “What exists”The advertisers
Section titled “The advertisers”| Node | Managed by | Route list | Tags |
|---|---|---|---|
truenas-scale |
ansible/roles/nas-tailscale, a TrueNAS custom app |
nas_tailscale_advertise_routes |
tag:nas, tag:core |
tpi-alpha-4 |
ansible/roles/k8s-tailscale, native tailscaled |
k8s_tailscale_advertise_routes |
tag:tp-host, tag:subnet-router |
tpi-beta-4 |
ansible/roles/k8s-tailscale, native tailscaled |
k8s_tailscale_advertise_routes |
tag:tp-host, tag:subnet-router |
The two route lists must stay equal. The VIP prefixes in each list come from lb_vip_prefix_ipv4
and lb_vip_prefix_ipv6 in ansible/inventory/group_vars/all.yml. The inventory group tailscale_subnet_routers in
ansible/inventory/hosts.yml names the two workers.
The two workers advertise subnet routes only. They advertise no exit node and run no Tailscale
SSH. heimdall is the SSH bastion. The role labels each worker
fzymgc.house/tailnet-subnet-router=true.
truenas-scale also advertises an exit node, and it is the only exit node on the tailnet.
Tailscale picks one primary per route among the advertisers and moves the route to another advertiser when the primary goes offline. It does not move the route back when the old primary returns, and when a worker and the NAS both advertise, it picks the worker. Read the current primary from any tailnet client:
tailscale status --json | jq '.Peer[] | select(.PrimaryRoutes != null) | {HostName, PrimaryRoutes}'Route approval
Section titled “Route approval”The tailnet policy is tailscale/policy.hujson, and tf/tailscale applies it. Its
autoApprovers.routes block approves the three routes for tag:nas and tag:subnet-router, and
autoApprovers.exitNode approves an exit node for tag:nas and tag:subnet-router.
An operator does not approve these routes in the admin console. A route that an operator disables
by hand comes back when the node next advertises it.
The VIP prefixes hold the LoadBalancer VIPs, and no leg holds an address in them. An advertiser
forwards VIP traffic through the gateway, and Tailscale rewrites the source to the Main address of
the advertiser. Lab is not on the tailnet. The policy aliases lb-vips-v6 and auth-fzymgc-v6 sit
next to lb-vips and auth-fzymgc in every grant, so the IPv6 prefix gets the same access as the
IPv4 prefix.
The OAuth client’s tag set, the node’s live tag set and the role’s tag list must be identical. A
re-registration assigns only the client’s tags, so a tag missing from the client disappears from
the node. The three places are tf/tailscale/oauth_clients.tf, the role default
(k8s_tailscale_advertise_tags or nas_tailscale_advertise_tags) and tagOwners in the policy.
Change them together.
Tailnet split DNS sends fzymgc.house to the two resolvers’ own tailnet addresses
(tf/tailscale/dns.tf). The workers run with --accept-dns=false
(k8s_tailscale_accept_dns), and the NAS app runs with TS_ACCEPT_DNS=true
(nas_tailscale_accept_dns).
Known limitations
Section titled “Known limitations”- An advertiser that is online on the tailnet but has a dead LAN forwarding path stays primary. Tailscale does not fail over on that condition (tailscale#16357). Keep advertisers simple. The Firewalla stopped being an advertiser for this reason, and the subnet route breakage report records the failure that removed it.
- No Main prefix is advertised, so a tailnet client reaches no macvlan child of the NAS on Main,
such as
precipiceat192.168.22.109. The Core containersnas-supportandvault-unseal-nasare reachable through the Core route even whiletruenas-scaleis primary, because the NAS holds no Core address and forwards to them through the gateway.
Procedures
Section titled “Procedures”Change the advertised routes
Section titled “Change the advertised routes”-
Edit
k8s_tailscale_advertise_routesandnas_tailscale_advertise_routesidentically. -
If the change adds a route, add it to
autoApprovers.routesintailscale/policy.hujsonand merge that change first. A route that a node advertises before its approver exists stays unapproved until the node advertises it again. -
Converge the workers. The role runs
tailscale seton a registered node, so no re-authentication happens and no tag drops. The--advertise-routesflag is always emitted, and an empty list clears the persisted routes.Terminal window cd ansibleuv run ansible-playbook -i inventory/hosts.yml tailscale-subnet-routers-playbook.yml --tags tailscale [--limit <node>] -
Converge the NAS. The wrapper needs an authenticated Vault session. The marker task is silent, so the dry run reports a change without a diff. containerboot applies
TS_ROUTESon every container start.Terminal window scripts/nas-playbook.sh --tags nas-tailscale --check --diffscripts/nas-playbook.sh --tags nas-tailscale -
Make sure that each advertiser lists the routes and that a tailnet client sees a primary:
Terminal window ssh fzymgc@tpi-alpha-4 sudo tailscale status # lists the routestailscale status --json | jq '.Peer[] | select(.PrimaryRoutes != null) | {HostName, PrimaryRoutes}' # on a client
Fail the routes back to truenas-scale
Section titled “Fail the routes back to truenas-scale”When the NAS’s Tailscale restarts, for example after a Docker address-pool change, a worker becomes primary and stays primary. If the worker path is healthy, nothing needs doing. To hand the routes back:
-
Withdraw the routes on both workers. Control then picks the NAS.
Terminal window ssh fzymgc@tpi-alpha-4 sudo tailscale set --advertise-routes=ssh fzymgc@tpi-beta-4 sudo tailscale set --advertise-routes=tailscale status --json | jq '.Peer[] | select(.PrimaryRoutes != null) | {HostName, PrimaryRoutes}' -
Restore the workers as standbys with the playbook from Change the advertised routes, or with
tailscale set --advertise-routes=and the three routes.
Register a worker
Section titled “Register a worker”A fresh install, or a node after tailscale logout, reports BackendState as NeedsLogin or
NoState. The role gates on that login state, not on the state file, because native tailscaled
writes tailscaled.state on first start before any login. For such a node the role reads the OAuth
client from Vault (fzymgc-house/infrastructure/k8s/tailscale-auth, minted by tf/tailscale) and
runs tailscale up with the full flag set. Log in to Vault first with
scripts/vault-helper.sh login, then run the playbook command from Change the advertised
routes. After a re-image the node re-registers and the policy approves its routes.
Make sure that the machine shows the tags tp-host and subnet-router and offers no exit node.
Check a worker after any tailscaled change
Section titled “Check a worker after any tailscaled change”Cilium runs kube-proxy replacement with BPF host routing and an explicit device pin
(cilium_devices: end0, end0.1000 and end0.3000). The check makes sure that tailscale0
never enters the datapath.
-
Devices. On the node’s Cilium agent pod,
cilium-dbg status --verboselistsend0,end0.1000andend0.3000and nevertailscale0, and reports kube-proxy replacement healthy:Terminal window p=$(kubectl -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=<node> -o name)kubectl -n kube-system exec $p -c cilium-agent -- cilium-dbg status --verbose | rg 'Devices|KubeProxyReplacement' -
Pod networking. Run a short-lived pod pinned to the node with a
kubernetes.io/hostnamenodeSelector. From it, ping a pod on another node, resolvekubernetes.default, and reach a ClusterIP and a NodePort Service. -
Node.
sudo tailscale statuslists the routes,ip ruleandip route show table 52exist, the node pings192.168.20.1,nc -zw3 192.168.250.254 6443connects (a routed VIP answers no ping), andkubectl get node <node>reportsReady. -
Datapath as primary. Read the primary from an off-LAN tailnet client. A worker that advertises next to the NAS becomes primary on its own. From the client, make sure that it reaches
192.168.20.1, the NAS,https://192.168.250.254:6443and a VIP such ashttps://vault.fzymgc.house/v1/sys/health. The VIP proves the route of the VIP prefixes. If no off-LAN client is at hand,heimdallserves:sudo tailscale set --accept-routesfor the test andsudo tailscale set --accept-routes=falseafter it.
Recover a worker whose networking degraded after tailscaled came up
Section titled “Recover a worker whose networking degraded after tailscaled came up”-
Take the node off the mesh with
ssh fzymgc@<node> sudo tailscale down. Cilium re-reads its devices. Make sure that pod networking recovers and the node isReady. -
Make sure that the device pin is in place. Without it, Cilium’s device auto-detection picks up
tailscale0and blackholes tailnet traffic that the node forwards to LoadBalancer VIPs, while plain forwarded traffic keeps working. The pin iscilium_devicesinansible/roles/cilium/defaults/main.yml, rendered asdevices:in the chart values. To re-apply it:Terminal window cd ansibleuv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium -
Re-join. The state persists, so no auth key is needed, and every flag is re-specified because
tailscale upresets flags it does not receive:Terminal window sudo tailscale up --advertise-tags=tag:tp-host,tag:subnet-router \--advertise-routes=192.168.40.0/22,192.168.250.0/24,fddb:f665:73f7:250::/64 \--accept-dns=false --hostname=<node> -
Run Check a worker after any tailscaled change.
References
Section titled “References”- Subnet route breakage report, the failure that removed the Firewalla from the advertiser set
- ADR hl-imbj, Terraform-minted OAuth clients for tailscale node auth
- Tailscale high availability