Skip to content

Tailscale Subnet Routers

Three tailnet nodes advertise the same three routes, so a tailnet client keeps a path to the LAN when one advertiser goes down. The routes are 192.168.40.0/22 (Core) and the routed LoadBalancer VIP prefixes 192.168.250.0/24 and fddb:f665:73f7:250::/64. No prefix inside the trusted-Main supernet 192.168.16.0/20 is advertised: every host that a remote operator needs has a Core address, a routed VIP or a tailnet node of its own, and a Main route on a client that sits on Main duplicates an on-link prefix. This page states what exists, the procedures that touch it, and a check for each.

Node Managed by Route list Tags
truenas-scale ansible/roles/nas-tailscale, a TrueNAS custom app nas_tailscale_advertise_routes tag:nas, tag:core
tpi-alpha-4 ansible/roles/k8s-tailscale, native tailscaled k8s_tailscale_advertise_routes tag:tp-host, tag:subnet-router
tpi-beta-4 ansible/roles/k8s-tailscale, native tailscaled k8s_tailscale_advertise_routes tag:tp-host, tag:subnet-router

The two route lists must stay equal. The VIP prefixes in each list come from lb_vip_prefix_ipv4 and lb_vip_prefix_ipv6 in ansible/inventory/group_vars/all.yml. The inventory group tailscale_subnet_routers in ansible/inventory/hosts.yml names the two workers.

The two workers advertise subnet routes only. They advertise no exit node and run no Tailscale SSH. heimdall is the SSH bastion. The role labels each worker fzymgc.house/tailnet-subnet-router=true. truenas-scale also advertises an exit node, and it is the only exit node on the tailnet.

Tailscale picks one primary per route among the advertisers and moves the route to another advertiser when the primary goes offline. It does not move the route back when the old primary returns, and when a worker and the NAS both advertise, it picks the worker. Read the current primary from any tailnet client:

Terminal window
tailscale status --json | jq '.Peer[] | select(.PrimaryRoutes != null) | {HostName, PrimaryRoutes}'

The tailnet policy is tailscale/policy.hujson, and tf/tailscale applies it. Its autoApprovers.routes block approves the three routes for tag:nas and tag:subnet-router, and autoApprovers.exitNode approves an exit node for tag:nas and tag:subnet-router. An operator does not approve these routes in the admin console. A route that an operator disables by hand comes back when the node next advertises it.

The VIP prefixes hold the LoadBalancer VIPs, and no leg holds an address in them. An advertiser forwards VIP traffic through the gateway, and Tailscale rewrites the source to the Main address of the advertiser. Lab is not on the tailnet. The policy aliases lb-vips-v6 and auth-fzymgc-v6 sit next to lb-vips and auth-fzymgc in every grant, so the IPv6 prefix gets the same access as the IPv4 prefix.

The OAuth client’s tag set, the node’s live tag set and the role’s tag list must be identical. A re-registration assigns only the client’s tags, so a tag missing from the client disappears from the node. The three places are tf/tailscale/oauth_clients.tf, the role default (k8s_tailscale_advertise_tags or nas_tailscale_advertise_tags) and tagOwners in the policy. Change them together.

Tailnet split DNS sends fzymgc.house to the two resolvers’ own tailnet addresses (tf/tailscale/dns.tf). The workers run with --accept-dns=false (k8s_tailscale_accept_dns), and the NAS app runs with TS_ACCEPT_DNS=true (nas_tailscale_accept_dns).

  • An advertiser that is online on the tailnet but has a dead LAN forwarding path stays primary. Tailscale does not fail over on that condition (tailscale#16357). Keep advertisers simple. The Firewalla stopped being an advertiser for this reason, and the subnet route breakage report records the failure that removed it.
  • No Main prefix is advertised, so a tailnet client reaches no macvlan child of the NAS on Main, such as precipice at 192.168.22.109. The Core containers nas-support and vault-unseal-nas are reachable through the Core route even while truenas-scale is primary, because the NAS holds no Core address and forwards to them through the gateway.
  1. Edit k8s_tailscale_advertise_routes and nas_tailscale_advertise_routes identically.

  2. If the change adds a route, add it to autoApprovers.routes in tailscale/policy.hujson and merge that change first. A route that a node advertises before its approver exists stays unapproved until the node advertises it again.

  3. Converge the workers. The role runs tailscale set on a registered node, so no re-authentication happens and no tag drops. The --advertise-routes flag is always emitted, and an empty list clears the persisted routes.

    Terminal window
    cd ansible
    uv run ansible-playbook -i inventory/hosts.yml tailscale-subnet-routers-playbook.yml --tags tailscale [--limit <node>]
  4. Converge the NAS. The wrapper needs an authenticated Vault session. The marker task is silent, so the dry run reports a change without a diff. containerboot applies TS_ROUTES on every container start.

    Terminal window
    scripts/nas-playbook.sh --tags nas-tailscale --check --diff
    scripts/nas-playbook.sh --tags nas-tailscale
  5. Make sure that each advertiser lists the routes and that a tailnet client sees a primary:

    Terminal window
    ssh fzymgc@tpi-alpha-4 sudo tailscale status # lists the routes
    tailscale status --json | jq '.Peer[] | select(.PrimaryRoutes != null) | {HostName, PrimaryRoutes}' # on a client

When the NAS’s Tailscale restarts, for example after a Docker address-pool change, a worker becomes primary and stays primary. If the worker path is healthy, nothing needs doing. To hand the routes back:

  1. Withdraw the routes on both workers. Control then picks the NAS.

    Terminal window
    ssh fzymgc@tpi-alpha-4 sudo tailscale set --advertise-routes=
    ssh fzymgc@tpi-beta-4 sudo tailscale set --advertise-routes=
    tailscale status --json | jq '.Peer[] | select(.PrimaryRoutes != null) | {HostName, PrimaryRoutes}'
  2. Restore the workers as standbys with the playbook from Change the advertised routes, or with tailscale set --advertise-routes= and the three routes.

A fresh install, or a node after tailscale logout, reports BackendState as NeedsLogin or NoState. The role gates on that login state, not on the state file, because native tailscaled writes tailscaled.state on first start before any login. For such a node the role reads the OAuth client from Vault (fzymgc-house/infrastructure/k8s/tailscale-auth, minted by tf/tailscale) and runs tailscale up with the full flag set. Log in to Vault first with scripts/vault-helper.sh login, then run the playbook command from Change the advertised routes. After a re-image the node re-registers and the policy approves its routes.

Make sure that the machine shows the tags tp-host and subnet-router and offers no exit node.

Check a worker after any tailscaled change

Section titled “Check a worker after any tailscaled change”

Cilium runs kube-proxy replacement with BPF host routing and an explicit device pin (cilium_devices: end0, end0.1000 and end0.3000). The check makes sure that tailscale0 never enters the datapath.

  1. Devices. On the node’s Cilium agent pod, cilium-dbg status --verbose lists end0, end0.1000 and end0.3000 and never tailscale0, and reports kube-proxy replacement healthy:

    Terminal window
    p=$(kubectl -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=<node> -o name)
    kubectl -n kube-system exec $p -c cilium-agent -- cilium-dbg status --verbose | rg 'Devices|KubeProxyReplacement'
  2. Pod networking. Run a short-lived pod pinned to the node with a kubernetes.io/hostname nodeSelector. From it, ping a pod on another node, resolve kubernetes.default, and reach a ClusterIP and a NodePort Service.

  3. Node. sudo tailscale status lists the routes, ip rule and ip route show table 52 exist, the node pings 192.168.20.1, nc -zw3 192.168.250.254 6443 connects (a routed VIP answers no ping), and kubectl get node <node> reports Ready.

  4. Datapath as primary. Read the primary from an off-LAN tailnet client. A worker that advertises next to the NAS becomes primary on its own. From the client, make sure that it reaches 192.168.20.1, the NAS, https://192.168.250.254:6443 and a VIP such as https://vault.fzymgc.house/v1/sys/health. The VIP proves the route of the VIP prefixes. If no off-LAN client is at hand, heimdall serves: sudo tailscale set --accept-routes for the test and sudo tailscale set --accept-routes=false after it.

Recover a worker whose networking degraded after tailscaled came up

Section titled “Recover a worker whose networking degraded after tailscaled came up”
  1. Take the node off the mesh with ssh fzymgc@<node> sudo tailscale down. Cilium re-reads its devices. Make sure that pod networking recovers and the node is Ready.

  2. Make sure that the device pin is in place. Without it, Cilium’s device auto-detection picks up tailscale0 and blackholes tailnet traffic that the node forwards to LoadBalancer VIPs, while plain forwarded traffic keeps working. The pin is cilium_devices in ansible/roles/cilium/defaults/main.yml, rendered as devices: in the chart values. To re-apply it:

    Terminal window
    cd ansible
    uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium
  3. Re-join. The state persists, so no auth key is needed, and every flag is re-specified because tailscale up resets flags it does not receive:

    Terminal window
    sudo tailscale up --advertise-tags=tag:tp-host,tag:subnet-router \
    --advertise-routes=192.168.40.0/22,192.168.250.0/24,fddb:f665:73f7:250::/64 \
    --accept-dns=false --hostname=<node>
  4. Run Check a worker after any tailscaled change.