Skip to content
fzymgc-house Cluster Docs

Tailscale Subnet Routers

HA failover subnet routing for 192.168.20.0/22 on the tailnet (#1711, Option B).

Advertiser set

Section titled “Advertiser set”
Node Role Managed by
truenas-scale Primary (oldest-registered) ansible/roles/nas-tailscale (custom app)
tpi-alpha-4 Failover standby ansible/roles/k8s-tailscale (native systemd)
tpi-beta-4 Failover standby ansible/roles/k8s-tailscale (native systemd)

Subnet routes only — no exit node (the router keeps exit duty) and no tailscale SSH (the Firewalla stays the SSH bastion).

Failover semantics

Section titled “Failover semantics”
  • Failover only, no active-active on this plan tier: overlapping advertisers form a failover group and ONE serves all clients at a time.
  • Selection is oldest-registered first, so truenas-scale stays primary; the two k8s workers are standbys in registration order.
  • Graceful failover (tailscale down on the primary) takes ~15s; longer on a network partition.
  • An online-but-broken primary does NOT fail over (tailscale#16357): if the advertiser is reachable on the tailnet but its LAN forwarding path is dead, clients stay pinned to it. Keep advertisers structurally simple — this is exactly why the Firewalla was demoted (see the 2026-07-21 CoE).

Deploy / converge

Section titled “Deploy / converge”

From ansible/ (requires an active Vault session on fresh deploys — scripts/vault-helper.sh login):

Terminal window
uv run ansible-playbook -i inventory/hosts.yml \
  tailscale-subnet-routers-playbook.yml --tags tailscale [--limit <node>]
  • Unregistered node (tailscale status --json BackendState is NeedsLogin/NoState — fresh install or after tailscale logout): the role reads the OAuth client from Vault (secret/fzymgc-house/infrastructure/k8s/tailscale-auth, minted by tf/tailscale) and runs tailscale up with the full flag set. The guard is login state, NOT the state file — native tailscaled writes tailscaled.state on first daemon start, before any login.
  • Registered node: the Vault lookup is skipped (login-state guard); routes and DNS preference converge via tailscale set — no re-auth, no route/tag drop.
  • --advertise-routes= is always emitted: an empty k8s_tailscale_advertise_routes list explicitly clears persisted routes (PR #1714 pattern); never conditionally omit the flag.

Tag completeness and re-register behavior

Section titled “Tag completeness and re-register behavior”

The OAuth client’s tag set, the node’s live tag set, and k8s_tailscale_advertise_tags must all be exactly [tag:tp-host, tag:subnet-router]. A re-register (e.g. after node re-image — state loss) assigns only the client’s tags: any tag missing from the client is silently dropped from the node. If you add a tag to these nodes, update the OAuth client (tf/tailscale/oauth_clients.tf), the role default, and the live ACL’s tagOwners together. After a re-image, the node re-registers and the tailnet policy approves its /22 routes automatically.

Route approval

Section titled “Route approval”

The tailnet policy approves these routes automatically. Its autoApprovers.routes block grants tag:nas and tag:subnet-router approval for 192.168.20.0/22 and for 192.168.40.0/22. An operator does not approve these routes in the admin console. A route an operator disables by hand comes back when the node next advertises it.

After a node advertises its routes, run this check:

  1. Confirm the machine shows tags tp-host + subnet-router and offers no exit node.

Cilium gate checklist (run after any tailscaled change on a node)

Section titled “Cilium gate checklist (run after any tailscaled change on a node)”

Cilium 1.20 runs KPR + BPF host routing with an explicit devices pin (cilium_devices: end0, end0.1000); the check confirms tailscale0 never appears:

  1. Devices: exec into the cilium agent pod on the node (kubectl -n kube-system get pod -l k8s-app=cilium --field-selector spec.nodeName=<node>), run cilium status --verbose: Devices list shows end0, plus end0.1000 once that node’s Lab leg is up, and never tailscale0; KPR healthy.
  2. Pod networking: short-lived pod pinned to the node (nodeSelector kubernetes.io/hostname): cross-node pod↔pod ping, DNS (nslookup kubernetes.default), and a ClusterIP/NodePort service hit.
  3. Node sanity: sudo tailscale status shows routes; ip rule and ip route show table 52 present; node pings 192.168.20.1 and the cluster VIP 192.168.20.140; node Ready in kubectl get node.
  4. Datapath as primary: the NAS stays primary unless forced — temporarily disable truenas-scale’s /22 route in the console, verify from an off-LAN tailnet client that 192.168.20.1, the NAS, and https://192.168.20.140:6443 are reachable, then re-enable the NAS route.

Recovery

Section titled “Recovery”

Node networking degraded after tailscaled came up:

  1. Immediate: ssh fzymgc@<node> sudo tailscale down — node leaves the mesh, Cilium re-detects devices; confirm pod networking recovers and the node is Ready.
  2. Devices pin — ACTIVE since the 2026-07-24 node-1 gate: Cilium’s runtime device auto-detection DID pick up tailscale0 (and the leftover vxlan.calico) on tpi-alpha-4, which blackholed tailnet-ingress traffic to Cilium-managed LoadBalancer VIPs (e.g. traefik at 192.168.20.145) forwarded through the node, while plain forwarded traffic (gateway, node IPs, kube-vip :6443) still worked. The pin is now permanent: cilium_devices: ["end0", "end0.1000"] in ansible/roles/cilium/defaults/main.yml, rendered as devices: in templates/values.yaml.j2. If it must be re-applied: re-run the cilium role (uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags cilium). end0 is the only physical NIC on the RK1 nodes, and end0.1000 is its Lab VLAN subinterface, so scoping NodePort/masquerade to the two is safe.
  3. Re-join: sudo tailscale up --advertise-tags=tag:tp-host,tag:subnet-router --advertise-routes=192.168.20.0/22,192.168.40.0/22 --accept-dns=false --hostname=<node> (state persists — no auth key needed; all flags re-specified to satisfy up’s reset protection). Re-run the gate checklist.

References

Section titled “References”