CoE: Tailscale-on-Router Subnet Route Breakage

Field	Value
Date	2026-07-21
Severity	Medium (remote LAN/cluster access broken for tailnet clients; non-Tailscale LAN devices unaffected)
Duration	Morning of 2026-07-21 ET (recurring incident class since ~June 2026)
Impact	Tailnet clients routing 192.168.20.0/22 through the router’s tailscale lost access to the LAN, cluster VIPs, and *.fzymgc.house DNS (split-DNS targets 192.168.20.1)
Resolution	Disabled the router’s 192.168.20.0/22 route in the Tailscale admin console; truenas-scale became sole primary subnet router
Follow-ups	#1711, #1712, #1713

Summary

The Firewalla runs Tailscale as a host-network docker container (ansible/roles/router-tailscale) advertising 192.168.20.0/22 and exit node. When that node is the primary subnet router for the /22, every tailnet client with accept-routes — remote clients especially, but also on-LAN clients using an exit node without “Allow LAN access” — sends all LAN-destined traffic through the Firewalla’s tailscale forwarding path. That path is structurally unsound (details below) and fails intermittently, which presents as “the network is broken” on Tailscale clients while everything else is healthy.

This is at least the third incident in this class (see #1660 — on-LAN raw-socket breakage; the 2026-07-16 NAS-shutdown DNS outage was the mirror image, when the router’s tailscale was found stopped and the NAS was the accidental sole subnet router).

Timeline (UTC)

Time	Event
2026-07-20 17:14	fzymgc-house/tailscale-config auto-applies ACL (Renovate digest bump; content unchanged — noted, ruled out as trigger)
2026-07-21 ~00:30	Operator debugging DNS on the router: containers could not resolve internal fzymgc.house names (host resolv.conf points at WAN resolvers). Hand-added dns: [192.168.20.1] to the tailscale and vault-unseal compose files
2026-07-21 00:46	docker compose up -d --force-recreate — fresh container, fresh tailscale up re-applying --advertise-routes=192.168.20.0/22 --advertise-exit-node (old container’s logs destroyed)
2026-07-21 09:00	Brief control-plane map-poll blip in tailscaled logs (recovered)
2026-07-21 morning ET	Routing breakage observed on tailnet clients; router was primary for the /22
2026-07-21 ~14:00	Operator disabled the router’s 192.168.20.0/22 route in the admin console → truenas-scale took over as primary → resolved

The tailscale container logged zero errors during the breakage window — the failure is silent at the tailscale layer, consistent with packets dying in the Firewalla’s own netfilter/policy-routing stack, not in tailscaled.

Root Cause

Not one bug — a structurally conflicted datapath. Four layers, all verified on the live router:

1. Two policy-routing engines on one host

Firewalla policy-routes per interface (eth1/eth2 dual-WAN, bond0.* VLANs, wg0/wg_ap) across custom tables at ip rule priorities 499–8001. Tailscale (kernel mode, network_mode: host, NET_ADMIN) inserts its own rules at 5210–5270, including 5270: from all lookup 52 — wedged between Firewalla’s LAN-routing rules (5002) and its default-route rules (5999–8001). Every forwarded packet on the router traverses both engines’ decisions.

2. Netfilter split-brain: iptables-legacy vs nft

Firewalla’s firewall (FW_FORWARD, FW_POSTROUTING, FR_POSTROUTING) lives in iptables-nft. Tailscaled’s in-container backend heuristic saw no rules and chose iptables-legacy — its ts-input/ts-forward/ts-postrouting chains live in a completely separate rule set. The kernel evaluates both stacks on every packet, including two independent NAT engines hooked at POSTROUTING (ts-postrouting MASQUERADE in legacy; FW_POSTROUTING in nft) competing over the same conntrack entries. Firewalla rewrites its nft rules dynamically (policies, device discovery, ACLs), so forwarded tailscale flows break intermittently with no tailscale-side change — matching the recurrence pattern.

3. Degraded container environment

From the container startup log:

• failed to enable src_valid_mark: ... read-only file system → tailscaled falls back to a connmark-based rp_filter workaround. Firewalla’s routing is itself built on fwmark/connmark conventions (0x6/0x3ff, 0x18/0x3ff, …); tailscale uses 0x40000/0xff0000 + connmark save/restore on shared conntrack state.
• ip6tables filtering is not supported on this host → v6filter=false; IPv6 tailnet traffic is unfiltered.

4. Client topology amplifies any failure into “everything is down”

Verified on a client: RouteAll=true (accept-routes) + Mullvad exit node + ExitNodeAllowLANAccess=false. The accepted /22 is more specific than the exit-node default route, so all LAN-destined traffic — including DNS to 192.168.20.1, the split-DNS target for fzymgc.house — rides the tunnel to whichever node is primary for the /22 (netstat confirms 192.168.20/22 → utun10). When the primary was the router and its forwarding path failed, clients lost the LAN, the cluster, and all internal DNS at once. Remote clients have no fallback path at all.

What Was Ruled Out

• MagicDNS resolv.conf hijack on the router — CorpDNS=false in the container prefs; host resolv.conf untouched.
• Container crash/OOM — container healthy, RestartCount=0, no errors logged.
• ACL/autoApprovers approval flapping — the live ACL (fzymgc-house/tailscale-config) has no autoApprovers block; approval state only changes by hand. (But see latent finding below — the monorepo copy would have introduced exactly this failure mode.)

Latent Findings

Finding	Risk
docker-compose@tailscale systemd unit is disabled + dead; container survives on docker restart: unless-stopped only	No self-healing; silent runtime drift (same theme as 2026-07-16)
Live compose file hand-edited (dns: [192.168.20.1]) — Ansible would clobber it on next run	IaC drift; codified into the role by this PR
tailscale/policy.hujson in this repo is not live — the hl-swfp ACL relocation stalled midway; fzymgc-house/tailscale-config still authoritative and diverged	The monorepo copy’s autoApprovers (tag:router for the /22) would silently revert this incident’s fix on any container restart the moment relocation completes. Defused in this PR; tracked in #1712
Router container stdout not shipped to ClickStack; --force-recreate erased pre-incident logs	Blind spot; tracked in #1713
truenas-scale is now the sole primary for the /22 and the tailnet’s only home exit node besides the router	SPOF — the exact inverse of the 2026-07-16 incident; tracked in #1711
Tailscale state lives on /extdata, whose boot mount has silently failed before (#1659)	A boot with /extdata unmounted starts tailscale with empty state → node re-registration

Resolution and Durable Fix

Operator action (immediate): disabled the router’s 192.168.20.0/22 route in the admin console.

This PR makes it durable:

1. ansible/roles/router-tailscale: tailscale_advertise_routes is now empty, and the compose template always emits --advertise-routes= (explicitly clearing routes rather than omitting the flag, so a redeploy converges instead of tripping tailscale up’s unspecified-flag reset protection). The hand-added dns: fix is codified as tailscale_dns_server.
2. tailscale/policy.hujson (monorepo copy): tag:router removed from the autoApprovers /22 entry so completing the ACL relocation can never silently re-approve the router.
3. Exit-node advertisement from the router is kept for now — the NAS is the only other home exit node, and removing both concerns at once would couple this fix to the replacement decision (#1711).

The role change takes effect on the next router-tailscale Ansible deploy; until then the console-disable holds (nothing auto-approves).

Replacing the Router as Subnet Router / Exit Node

The requirement: at least two always-on advertisers for 192.168.20.0/22 (HA failover), neither of them the Firewalla, so remote access survives NAS maintenance and router complexity. Options considered:

	A: seattle (Mac mini)	B: native tailscaled on k8s nodes	C: status quo (NAS only)
Network stack	Simple (single NIC, no policy routing)	Simple host stack, but Cilium eBPF datapath on-node	Simple; proven today
Advertising support	✅ GUI client + CLI integration now supports --advertise-routes/--advertise-exit-node on macOS (IP forwarding auto-enabled)	✅ First-class Linux	✅ Running
Ansible manageability	Weak — GUI app login, CLI-integration toggle, and tagging of a currently-untagged personal device are manual steps	Strong — apt install + OAuth client + tags, same pattern as the router role; in-house precedent (artis3n.tailscale in tp2-cluster)	n/a
HA	One more single box; also carries ollama duties (launchd fragility, brew-update reboots)	Two nodes (one per chassis: alpha + beta) → N+1 across power domains; node reboots during upgrades covered by the pair	❌ SPOF (2026-07-16 proved it)
Tagging/ACL	Needs a new tag + autoApprovers entry; device is seanb4t@ today	tag:tp-host exists in ACL already; add to autoApprovers	tag:nas already approved
Key risk	macOS as network infra; sleep/updates	Cilium 1.19 KPR + BPF host routing + auto device detection may interact with tailscale0; must validate on one worker first (pin cilium devices to the physical NIC if needed)	Outage window = every NAS maintenance

Recommendation (B): native tailscaled on two worker nodes (one tpi-alpha, one tpi-beta), OAuth-tagged, advertising the /22 (+ exit node), validated on a single worker before the second. The NAS stays as third advertiser. The router keeps its tailscale as a plain node — no routes, no special role beyond tailscale SSH bastion access to the Firewalla itself — and its exit-node advertisement is retired once the node pair offers exit. seattle remains a fine tertiary/exit-only addition, but building the primary HA pair on ansible-managed Linux beats building it on a GUI-managed Mac.

This supersedes the June 2026 decision that made the Firewalla the sole approved primary /22 router after the in-cluster (operator-based) subnet router was retired — note the k8s option here is native on the node OS, not the in-cluster operator pattern that was retired.

Client-Side Hygiene

Clients that sit on the home LAN while using an exit node should enable “Allow LAN access” — without it, even local traffic hairpins through the current primary subnet router, adding a remote dependency (and a failure mode) to same-room communication. Remote clients are unaffected by this setting.

Lessons Learned

• Don’t run a policy-routing VPN daemon on a policy-routing router. Two rule engines + two netfilter backends on one forwarding host fail intermittently and log nothing.
• A silent tailscale layer means look at the host firewall. Zero errors in tailscaled logs during a forwarding outage points at the surrounding netfilter, not the daemon.
• --force-recreate is evidence destruction when container stdout isn’t shipped anywhere (#1713).
• Half-finished migrations are landmines. The stalled ACL relocation left a diverged policy copy whose autoApprovers would have re-armed this exact incident (#1712).
• Console toggles are not fixes. The durable fix lives in the IaC that regenerates the setting (TS_EXTRA_ARGS here), not in the admin UI state it produces.