Network Reference

Network configuration and addressing for the fzymgc-house cluster.

The addressing below is the Firewalla’s and the resolver pair’s. The UniFi switching fabric sits under it and can veto it: DHCP Guard drops DHCP offers from any server not on a network’s trusted list. See UniFi and tf/unifi.

IP Addressing Summary

Network	CIDR	Purpose
Trusted-Main Supernet	192.168.16.0/20	Single covering CIDR for “Main, wired or wireless” in policies (e.g. the Keycloak admin allowlist). Covers Main + MainWireless; the unassigned 16-19/28-31 blocks are reserved for trusted-Main use — numbering an untrusted network inside this /20 silently grants it trusted-policy access (decision 2026-08-04)
Main (wired)	192.168.20.0/22	Wired trusted LAN, VLAN 3000 — one L2 segment (.20.0-.23.255); wired client devices are DHCP-assigned anywhere in this range, not just .20.0/24
MainWireless	192.168.24.0/22	Wireless trusted LAN, VLAN 3001, same trust policy as Main. Gateway 192.168.24.1; DHCP pool 24.200-27.254 (statics live in 24.1-24.199, keeping .5/.6/.7 free per the resolver-address convention); search domain wireless.lan.fzymgc.house; mDNS + SSDP relay on. Still not advertised on the tailnet (deliberate — wireless clients need no inbound)
Core	192.168.40.0/22	Untagged/native infrastructure LAN — gateway 192.168.40.1, DHCP pool 41.10-43.249. Holds the two authoritative resolvers (sandiego .40.6, portland .40.7), their Core VIP .40.5, and heimdall .40.20. Declared as network-core in the tailnet ACL. Advertised on the tailnet since 2026-08-10 — see the note below
Lab	192.168.200.0/22	Trusted lab LAN, VLAN 1000. Gateway 192.168.200.1; resolver VIP .200.5, sandiego .200.6, portland .200.7. Static space .200.140-.200.159: the k3s node Lab legs .141-.144 and .151-.154 on each node’s end0.1000, the MetalLB lab pool .145-.149 and .155-.159, and .140 reserved for kube-vip. DHCP: Kea 192.168.200.160-192.168.201.19, appliance 192.168.201.20-192.168.203.249. Not advertised on the tailnet
Node Network	192.168.20.0/24	Physical node IPs (a subset of Main above)
Pod Network	10.42.0.0/16	Kubernetes pods
Service Network	10.43.0.0/16	Kubernetes services
MetalLB Pool 1	192.168.20.145-149	LoadBalancer VIPs
MetalLB Pool 2	192.168.20.155-159	LoadBalancer VIPs
MetalLB Lab Pool 1	192.168.200.145-149	LoadBalancer VIPs by explicit request only (autoAssign: false)
MetalLB Lab Pool 2	192.168.200.155-159	LoadBalancer VIPs by explicit request only (autoAssign: false)
Guest	192.168.79.0/24	Untrusted segment, VLAN 3100. Gateway 192.168.79.1; the appliance serves its DHCPv4 — unchanged through this milestone (Phase 02 D-04) — but is no longer its DNS server: 03-01 moved the appliance’s DHCPv4 Primary DNS Server from 192.168.79.1 to the pair’s VIP 192.168.79.5 on 2026-08-30. Over IPv6 this leg exposes no appliance DHCPv6 Primary DNS Server field, so that half is recorded NOT ATTEMPTED (D-07), which D-07 records as leaving the pair’s RA/RDNSS (D-10) as this segment’s only v6 announcement; fddb:f665:73f7:2::5 is the leg’s UNUSED v6 VIP, typed nowhere. CORRECTED 2026-08-31 (03-04) — this read “the appliance serves its DHCPv4 and remains its DNS server in both families”; the DHCPv4 clause stands, the DNS clause is false from 2026-08-30. The resolver pair holds a leg and a VIP here from 02-04 (2026-08-29) — .5 VIP, .6 sandiego, .7 portland — with trusted: false, so an internal name asked of the pair from this segment returns NXDOMAIN by design — and that sentence is now more load-bearing, not less: it describes what a Guest client actually gets, rather than a property of the render. Not inside the trusted-Main supernet, deliberately
Telework	192.168.217.0/24	Untrusted segment, VLAN 3200. Gateway 192.168.217.1; same arrangement as Guest — the appliance serves its DHCPv4 (D-04, unchanged) and is no longer its DNS server: 03-02 moved the appliance’s DHCPv4 Primary DNS Server from 192.168.217.1 to the pair’s VIP 192.168.217.5 on 2026-08-31, a day after Guest. Its DHCPv6 Primary DNS Server field is likewise ABSENT — determined from Telework’s own Edit view rather than inherited from Guest’s — so that half is NOT ATTEMPTED (D-07) and the pair’s RA/RDNSS is this segment’s only v6 announcement; fddb:f665:73f7:5::5 is the leg’s UNUSED v6 VIP. CORRECTED 2026-08-31 (03-04) — this read “same arrangement as Guest — appliance-served DHCPv4 and DNS”; the DHCPv4 half stands, the DNS half is false from 2026-08-31. Resolver leg and VIP .5/.6/.7 from 02-04, trusted: false. Not inside the trusted-Main supernet, deliberately

Adding a leg to dns_vlan_interfaces is not the same as serving it. Five separate consumers read that structure — the systemd-networkd legs, the nftables accept sets, Blocky’s listen addresses, keepalived’s vrrp_instance blocks and the RA announcer’s config — and each needs its own converge and its own adoption step. A leg declared but not converged everywhere presents as a total DNS failure on that segment. See docs/operations/dns.md for the per-consumer checks.

The Guest and Telework rows were added 2026-08-29 (02-05), and no gate demanded them. This table is pinned by nothing — rg -n 'IP Addressing Summary' tools/ docs/ matches only the heading itself. They were added because the dns_vlan_interfaces header block in ansible/inventory/host_vars/{sandiego,portland}.yml names this file as the authority a reader consults before allocating a leg, and a reader who consulted it for IPv4 found two of the estate’s seven segments missing from the view. The IPv6 half of the same estate has carried both rows in VLAN Subnets below for as long as that table has existed.

What the tailnet advertises, and what it deliberately does not

Advertised: 192.168.20.0/22 (Main) and 192.168.40.0/22 (Core). Nothing else.

Core was added on 2026-08-10 so an off-LAN operator can reach the resolvers, heimdall, and the Core VIP for administration. Before it, a remote session could reach 192.168.20.1 and nothing on 192.168.40.0/22 — which read as “the resolver is down” when the truth was “this vantage cannot see it”.

Three constraints hold, and each has a reason that is not style:

• The Firewalla is never a subnet router. tag:router is deliberately absent from the ACL’s autoApprovers.routes. Its forwarding path runs two policy-routing engines and two NAT engines over one conntrack table, and it fails silently and intermittently — three incidents, see the 2026-07-21 CoE. It keeps exit-node duty and the SSH bastion role, nothing more. No advertiser sits on Core, so Core traffic is forwarded advertiser → Firewalla → Core by ordinary L3 routing, which does not re-enter that datapath.
• Every advertiser announces the same set. truenas-scale (primary) and the tpi-*-4 standbys form a failover group and Tailscale serves ONE at a time, so a route the primary announces and the standbys do not disappears on failover, silently. Adding a route means adding it in every advertiser’s configuration — the NAS included, which is not yet codified in this repo.
• No ULA route is advertised. fddb:f665:73f7::/48 stays off the tailnet. The IPv6 success criteria are source-bound (dig -b from a host holding a real address in the leg’s own /64), which a tailnet-routed client cannot satisfy; and subnet-route SNAT rewrites the source, so such a probe would read as evidence it is not. A ULA route would buy diagnostic convenience at the cost of a vantage that looks on-link and is not — see D-DEF-11.

IPv6 ULA Addressing

Unique Local Addresses (ULA) provide stable internal IPv6 addressing that works without internet connectivity.

Prefix	CIDR	Purpose
ULA Base	fddb:f665:73f7::/48	Site-local IPv6 prefix

VLAN Subnets

Each VLAN receives a /64 subnet from the ULA prefix:

VLAN	Interface	IPv6 Subnet	Appliance ULA address
Main (3000)	bond0.3000	fddb:f665:73f7:1::/64	fddb:f665:73f7:1::1 (held by router-ula-reconciler)
Guest (3100)	bond0.3100	fddb:f665:73f7:2::/64	none
IoT (3020)	bond0.3020	fddb:f665:73f7:3::/64	fddb:f665:73f7:3::1 (held by router-ula-reconciler)
Lab (1000)	bond0.1000	fddb:f665:73f7:4::/64	fddb:f665:73f7:4::1 (held by router-ula-reconciler)
Telework (3200)	bond0.3200	fddb:f665:73f7:5::/64	none
Core (untagged)	bond0	fddb:f665:73f7:6::/64	fddb:f665:73f7:6::1 (held by router-ula-reconciler)
MainWireless (3001)	bond0.3001	fddb:f665:73f7:7::/64	fddb:f665:73f7:7::1 (held by router-ula-reconciler)

ULA is routed between the trusted legs. The Firewalla holds ::1 in the ULA /64 of Main, IoT, Lab, Core and MainWireless, so it has a connected route to each of those prefixes and forwards ULA traffic between them under the same FireMain rules that govern IPv4. Guest and Telework hold no appliance ULA address: traffic between their ULA prefixes and any other leg is not routed, and the app rules block those two networks from the local networks in any case. The addresses are kept on the legs by the router-ula-reconciler role, because the Firewalla app drops them on every save of a network (docs/operations/firewalla-config.md, IPv6 specifics). Tracked in #2113.

This table is the allocating authority for ULA indices. Allocate a new leg here, in this table, and nowhere else.

It was not always. Until phase 04.3 the authority was an Ansible map on the appliance role, and this table was a reading of it. Phase 04.3 (decision D-12) retired that role entirely — see Configuration below for why — so the map is gone and this table inherits the job.

CORRECTED 2026-08-29 (02-05). This paragraph read, verbatim:

That inheritance is load-bearing for indices 2 and 5. The resolver pair’s own dns_vlan_interfaces carries only the five in-scope legs — 1 (main), 3 (iot), 4 (lab), 6 (core), 7 (mainwireless). Guest (index 2) and Telework (index 5) are deliberately out of scope for the pair and appear in no inventory structure anywhere in this repository. With the retired map deleted, the two rows above are the only record that those /64s are taken. Allocating a new leg without reading them is a /64 collision with a live broadcast domain, and nothing in the estate would report it.

Two of its claims are false from 02-04 (2026-08-29). dns_vlan_interfaces now carries all seven legs on both resolvers, so it no longer “carries only the five in-scope legs”; and Guest and Telework now appear in an inventory structure, so they no longer “appear in no inventory structure anywhere in this repository”. Their /64s are fddb:f665:73f7:2::/64 and fddb:f665:73f7:5::/64 respectively, exactly as the table above allocates them, and the inventory’s address6 and vip6 values sit inside those prefixes.

The paragraph’s actual point survives the sentence that carried it, and it matters more. This table is the allocating authority for ULA indices. Allocating a new leg without reading it is a /64 collision with a live broadcast domain, and nothing in the estate would report it. What changed is only that the inventory is now a second place those two /64s appear — and it is a CONSUMER of this allocation, not a second authority for it. The direction of the dependency is unchanged: allocate here, then write the addresses into dns_vlan_interfaces, never the reverse.

Indices 1–5 are frozen: they are already live in client addresses and in dns_vlan_interfaces on both resolvers, so renumbering any of them invalidates addresses that exist today. Indices 6 and 7 were allocated by phase 04.1 and continue that sequence. A new leg takes 8.

Resolver addresses on each ULA leg

The resolver pair holds a floating address and one address per node on every in-scope leg. The last hextet mirrors the IPv4 last octet, so an address reads the same in both stacks: ::5 VIP, ::6 sandiego, ::7 portland. ::1 is the gateway: the appliance holds it on every trusted leg (see VLAN Subnets above).

VLAN	ULA /64	VIP (::5)	sandiego (::6)	portland (::7)
Main (3000)	fddb:f665:73f7:1::/64	fddb:f665:73f7:1::5	fddb:f665:73f7:1::6	fddb:f665:73f7:1::7
IoT (3020)	fddb:f665:73f7:3::/64	fddb:f665:73f7:3::5	fddb:f665:73f7:3::6	fddb:f665:73f7:3::7
Lab (1000)	fddb:f665:73f7:4::/64	fddb:f665:73f7:4::5	fddb:f665:73f7:4::6	fddb:f665:73f7:4::7
Core (untagged)	fddb:f665:73f7:6::/64	fddb:f665:73f7:6::5	fddb:f665:73f7:6::6	fddb:f665:73f7:6::7
MainWireless (3001)	fddb:f665:73f7:7::/64	fddb:f665:73f7:7::5	fddb:f665:73f7:7::6	fddb:f665:73f7:7::7
Guest (3100)	fddb:f665:73f7:2::/64	fddb:f665:73f7:2::5	fddb:f665:73f7:2::6	fddb:f665:73f7:2::7
Telework (3200)	fddb:f665:73f7:5::/64	fddb:f665:73f7:5::5	fddb:f665:73f7:5::6	fddb:f665:73f7:5::7

CORRECTED 2026-08-29 (02-05). This read, verbatim: “Guest (index 2) and Telework (index 5) carry no resolver addresses. They are deliberately out of scope and still resolve through the appliance in both families.”

The two sentences did not move together:

• “carry no resolver addresses” — FALSE from 02-04 (2026-08-29), and the two rows above are what replaced it. Both legs now carry the pair’s .5 / .6 / .7 in IPv4 (192.168.79.0/24 and 192.168.217.0/24) and ::5 / ::6 / ::7 in IPv6.

• “still resolve through the appliance in both families” — was STILL TRUE when 02-05 wrote it, and is FALSE from 2026-08-30. CORRECTED 2026-08-31 (03-04). This bullet’s verdict read, verbatim: “STILL TRUE. Phase 03 is the client cut; nothing about which box a Guest or Telework client asks has changed.” Phase 03 then made the cut. 03-01 moved Guest’s appliance DHCPv4 Primary DNS Server to 192.168.79.5 on 2026-08-30; 03-02 moved Telework’s to 192.168.217.5 on 2026-08-31. Neither leg exposed a DHCPv6 Primary DNS Server field, so both v6 halves are recorded NOT ATTEMPTED (D-07) rather than cut. What is still true is narrower and is stated positively so it is not lost with the sentence that carried it: the appliance still serves DHCPv4 on both segments (D-04) and still answers any client that has not renewed its lease.

  RE-CORRECTED 2026-08-31, later the same day, after a Telework DNS outage. The correction directly above is itself now partly wrong, and is kept rather than rewritten because the sequence is the lesson. It reads the cut as effective in v4 on both legs. The cut moved the appliance field on both legs, but the pair was never reachable on either segment — eth0.3100 and eth0.3200 do not exist on portland or sandiego — so no client ever resolved through the pair on Guest or Telework. Telework was rolled back to the appliance on 2026-08-31 and does resolve through it in both families again; Guest points at a dead address and resolves nothing. The original “still resolve through the appliance in both families” is therefore TRUE again for Telework and false for Guest only because Guest resolves nothing at all, which is not what that sentence meant. See docs/engineering/plans/2026-08-31-serve-the-untrusted-legs.md.

Both entries declare trusted: false and neither carries a kea_subnet_id or a kea_pool, and the pair announces RA with RDNSS on both legs by decision (D-10, LEG-06). The operator-facing consequences of that — what a dual-stack client on those legs now learns, and what it gets when it asks — are in the DNS runbook rather than restated here.

The source of truth for every address above is dns_vlan_interfaces in ansible/inventory/host_vars/sandiego.yml and portland.yml.

The digit repetition is a readability hazard, and the eye is not the control

The /64 index and the node suffix sit in different hextets and use the same small integers, so several addresses repeat a digit for two entirely unrelated reasons:

• fddb:f665:73f7:6::6 — index 6 is Core, suffix 6 is sandiego. The two sixes mean different things.
• fddb:f665:73f7:7::7 — index 7 is MainWireless, suffix 7 is portland. Likewise.

Worse, and the case to actually watch for: index 5 is Telework. fddb:f665:73f7:5::7 looks like a portland address and is a Telework address.

CORRECTED 2026-08-29 (02-05), and the hazard got sharper rather than going away. This read: “Worse, and the case to actually watch for: index 5 is Telework, which is out of scope. fddb:f665:73f7:5::7 looks like a portland address and is a Telework address. Nothing in this estate should ever carry it.” The first clause is still true and is now worse. The last sentence is FALSE from 02-04: fddb:f665:73f7:5::7 is portland’s own address on the Telework leg, so it appears exactly once in the estate — in ansible/inventory/host_vars/portland.yml, on the telework entry — and a reviewer who deletes it as an obvious typo deletes a live address.

The paragraph’s conclusion is unchanged: the eye is not the control. What the control IS has moved. The range-based EXCLUDED_NETWORKS6 containment gate that used to catch this was retired in place by 02-03, because a range exclusion cannot say anything about a range that is now legitimately occupied. What catches it now is stronger: the exact-value per-leg pins in tools/dns-role-gates/tests/test_inventory_shape.py and test_dns_common.py — EXPECTED_LEG_ADDRESSES6 and EXPECTED_LEG_VIPS6 — which do not merely say an address is inside an allowed prefix, they say what every address on every leg on every node IS. Those still parse each value with ip_network and test containment as well; a string-prefix check would do neither.

Configuration

The resolver pair announces the ULA prefixes. The Firewalla does not.

dns-common-ra-announce.service on sandiego and portland runs radvd against a config rendered from dns_vlan_interfaces, one interface stanza per in-scope leg. Each stanza advertises that leg’s ULA /64 with:

Directive	Value	Why
AdvPreferredLifetime	3600	Finite, so the prefix can be deprecated. Under RFC 4862 §5.5.3(e) a host retires a prefix only by watching these count down — with infinite there is no withdrawal path at all
AdvValidLifetime	7200	Finite, and at the two-hour clamp of §5.5.3(e) so a reduction is honoured rather than discarded
AdvDefaultLifetime	0	The resolvers are not routers. Zero router lifetime is what lets a non-router announce a prefix while the Firewalla remains sole default gateway

Clients auto-configure via SLAAC as before; what changed is who transmits, and that the lifetimes are now finite.

Render: ansible/roles/dns-common/templates/dns-common-ra-announce.conf.j2.

Why the appliance stopped being the announcer. On 2026-08-10 the ISP re-delegated the GUA prefix, FireRouter rebuilt its interface addressing, and every ULA address on all seven appliance legs disappeared — they had been added out-of-band by a post_main.d hook that FireRouter knows nothing about and that only runs at boot. The resolvers’ static ULA VIPs were untouched. Anything this estate depends on should not sit on a device whose vendor daemon rebuilds its address set on an event we neither control nor observe. Phase 04.3 (decision D-12) therefore moved RA duty to the pair and retired the appliance-side ULA Ansible role in full — role, playbook entry and gate module. Its name and full history are in git log -- ansible/roles/.

RDNSS now has two sources, and they advertise the same address. The appliance side is unchanged and this repository still never writes it: the recursive DNS server each network advertises over DHCPv6 is set in the Firewalla app’s per-network DHCPv6 → Primary DNS Server field, which FireMain writes into bond0.*_v6.conf. Phase 02 added the second source — the announcer above now carries an RFC 8106 RDNSS option on every in-scope leg, addressed to that leg’s vip6, which is the same floating address the appliance’s DHCPv6 field already points at on the four non-Lab legs. This paragraph previously stated that the announcer declared no recursive-DNS directive whatsoever; that was true until phase 02 and is not true now.

The addresses have to agree rather than merely coexist: RFC 8106 §5.3.1 leaves multi-source ordering to the host, and a conforming host puts the DHCP-sourced value first, so two sources disagreeing would be a coin flip rather than a fallback. Lab is the exception in the useful direction — it has no appliance DHCPv6 at all, so there the pair is the only source, which is why Lab is where the phase takes its proof.

Appliance-side removal — discharged 2026-08-17 under V6-05

Retiring the Ansible role stopped this repository placing those files. It did not remove the copies already sitting on the Firewalla.

Phase 02 removed them, as a one-shot in-phase named precondition of moving RDNSS to the resolver pair (requirement V6-05). It is not a standing chore and not a recurring check: it was performed once, its pre- and post-removal readbacks were captured in a single dispatch each, and the record lives in docs/operations/dns.md under V6-05 — reading the appliance-side ULA artifacts absent.

That framing does not license a recurring appliance check, and the reason it does not is the standing rule the removal was always subject to. Repo rule afja1qt48h — test what we own — puts appliance state outside a standing green bar: FireRouter and FireMain rebuild that filesystem on events this repository neither controls nor observes, so a required check over it would go red for reasons nobody here can fix. Draft criterion 7 for phase 04.3 was dropped for exactly this reason, and V6-05 does not reinstate it. A precondition is discharged once and recorded; a bar is asserted forever. This is the first, and adding the second would reintroduce the failure mode 04.3 avoided.

Paths removed — all five are required, per phase 02’s D-12. The run markers are not optional cleanup: leaving the script or its markers behind means the hook can be recreated by hand or by a restore and immediately works again, which is the re-arming path this removal exists to close.

Path	What it is
/home/pi/.router/config/dhcp/conf/zzzz-ula-dns.conf	The re-arming landmine — seven dhcp-range declarations at lifetime infinite
/home/pi/.firewalla/config/post_main.d/0001-ipv6-ula.sh	The boot hook
/extdata/scripts/configure-ipv6-ula.py	The script the hook runs
/var/log/ipv6-ula-failure.log, /extdata/ipv6-ula-boot.log	Run markers — required, so a restore cannot silently re-arm the path

Read-back, run on the Firewalla:

ls -la /home/pi/.router/config/dhcp/conf/ | grep -i ula        # expect no output
ls -la /home/pi/.firewalla/config/post_main.d/ | grep -i ipv6  # expect no output

A third line used to sit here — grep -rl 'fddb:f665:73f7' /home/pi/.router/config/dhcp/conf/ — and it is removed because reading it as a removal check is actively dangerous. It returns four hits in the fully-correct post-removal state. Those four are FireRouter-written bond0*_v6.conf files carrying the app’s per-network DHCPv6 → Primary DNS Server value, which points at the resolver pair’s own ULA VIPs — the same-address invariant V6-02 depends on. Deleting them to make that command return nothing would take IPv6 DNS off four VLANs. The discriminating replacement, which excludes them by construction and carries its own positive control, is in docs/operations/dns.md.

These use grep, and that is not an oversight. The Firewalla has no ripgrep — only /usr/bin/grep, awk and sed. This repository’s rg-only rule is scoped to repo-side searches; an unscoped reading of it once rewrote a working on-box grep in docs/operations/dns.md into a command that cannot execute there, silently breaking a documented runbook. Anything written to run on that appliance stays in the tools it actually has.

Three facts that went with the removal, and still hold after it:

1. The per-leg ULA addresses on bond0.* need no action — measured gone on 2026-08-10, when the ISP re-delegated the GUA prefix and FireRouter rebuilt interface addressing. ip -6 -br addr show returns zero fddb: addresses. Nothing to remove.
2. The removal is inert until dnsmasq re-reads the conf directory, and since the declarations are already inert (no ULA address on any leg for them to bind to), it changes nothing on the wire today. It removes the re-arming path, not a live advertisement.
3. The appliance’s own RDNSS is untouched by the removal, and phase 02 depends on that. It comes from the app’s per-network DHCPv6 Primary DNS Server field, which FireMain writes into bond0.*_v6.conf — never from the file that was removed, and those four _v6.conf files were verified intact after the removal. Phase 02 then made the resolver pair a second RDNSS source on every in-scope leg, advertising the same vip6 those four files already point at, so what those files hold is now V6-02’s same-address invariant rather than merely adjacent state. Deleting them would take IPv6 DNS off four VLANs and break that invariant; they are out of this repository’s control and out of scope (threat T-02-05).

Node Addresses

Control Plane Nodes

Node	IP Address	Lab leg (end0.1000)	Role
tpi-alpha-1	192.168.20.141	192.168.200.141	Control plane (etcd)
tpi-alpha-2	192.168.20.142	192.168.200.142	Control plane (etcd)
tpi-alpha-3	192.168.20.143	192.168.200.143	Control plane (etcd)

Worker Nodes

Node	IP Address	Lab leg (end0.1000)	Board
tpi-alpha-4	192.168.20.144	192.168.200.144	alpha
tpi-beta-1	192.168.20.151	192.168.200.151	beta
tpi-beta-2	192.168.20.152	192.168.200.152	beta
tpi-beta-3	192.168.20.153	192.168.200.153	beta
tpi-beta-4	192.168.20.154	192.168.200.154	beta

Each node’s Lab leg routes Lab-sourced traffic through table 1000 (rule priority 6000), and the Kubernetes InternalIP stays on the Main address. See Lab service VLAN.

Virtual IPs

Address	Purpose	Provider
192.168.20.140	Kubernetes API VIP	kube-vip
192.168.20.145-149	Service LoadBalancers	MetalLB
192.168.20.155-159	Service LoadBalancers	MetalLB
192.168.200.145-149	Service LoadBalancers, explicit request only	MetalLB lab pool
192.168.200.155-159	Service LoadBalancers, explicit request only	MetalLB lab pool

DNS Configuration

External DNS (Cloudflare)

Domain: fzymgc.house

Record	Type	Target	Proxy
vault	CNAME	Cloudflare Tunnel	No (TCP passthrough)
auth	CNAME	Cloudflare Tunnel	Yes
grafana	CNAME	Cloudflare Tunnel	Yes
argocd	CNAME	Cloudflare Tunnel	Yes
mealie	CNAME	Cloudflare Tunnel	Yes
longhorn	CNAME	Cloudflare Tunnel	Yes
status	CNAME	Cloudflare Tunnel	Yes

Internal DNS Subdomain

k8s.fzymgc.house - Direct cluster access (not via Cloudflare)

Record	Type	Target
*.k8s	A	MetalLB Traefik VIP
doltdb	A	MetalLB Dolt VIP

Kubernetes DNS (CoreDNS)

Pattern	Resolution
<svc>.<ns>.svc.cluster.local	Service ClusterIP
<pod-ip-dashed>.<ns>.pod.cluster.local	Pod IP
<svc>.<ns>.svc	Short form (within cluster)

Internal .house Resolver (Authoritative)

CUT OVER in milestone v1.4. The chain below is the sandiego/portland resolver pair, not the appliance. All seven VLANs advertise the pair over IPv4 — the five trusted legs on 2026-08-07, then Guest on 2026-08-30 (03-01) and Telework on 2026-08-31 (03-02) — and four legs advertise it over IPv6 (2026-08-08), with Lab, Guest and Telework each recorded NOT ATTEMPTED because none of the three exposes a DHCPv6 Primary DNS Server field to set. The appliance chain has not gone away — it still serves DHCPv4 on Guest and Telework (D-04, unchanged by this milestone), still answers un-renewed clients that keep asking it, and still produces the curated zone the pair serves. Both are described below, in that order.

CORRECTED 2026-08-31 (03-04). Two clauses were superseded by the Guest and Telework cut and are named rather than deleted. (1) “The five trusted VLANs advertise the pair over IPv4 (2026-08-07) and four of the five over IPv6 (2026-08-08; Lab has no DHCPv6 field to set)” — the counts were right when written and understate the estate from 2026-08-30; the IPv6 exception set is now Lab, Guest and Telework rather than Lab alone. (2) “it still resolves for Guest and Telework” — FALSE from those cuts. The un-renewed-clients and curated-zone clauses beside it were true and remain true, and are kept rather than swept away with the clause they sat next to. The counts above are transcribed from the record section in the DNS runbook.

The authoritative status statement is the STATUS banner in the DNS runbook — this section is the address-and-topology reference, and where the two disagree, the runbook wins.

Client DNS on all seven VLANs follows a fixed chain, not a flat forward:

client :53
  -> the advertised resolver address on the client's VLAN   (VIP .5, or .6/.7 per node)
  -> Blocky on the resolver pair            <-- THE FRONT TIER (client-facing, per-VLAN listeners)
       |- conditional mapping: fzymgc.house -> unbound's loopback socket
       '- everything else -> Control D over DoT
  -> unbound (authoritative for .house)

What the chain RETURNS differs by trust; the chain itself does not. On the five trusted legs it resolves fzymgc.house internally. On Guest and Telework — trusted: false since 02-04, and cut onto the pair on 2026-08-30 (03-01) and 2026-08-31 (03-02) — the same chain runs, and Blocky’s per-client view withholds the whole apex, its real public names included, so a client on those legs gets NXDOMAIN for a .house name rather than an internal answer. See the ADR and the client-view chapter of the DNS runbook. (CORRECTED 2026-08-31, 03-04. The chain sentence above read: “Client DNS for fzymgc.house on the five trusted VLANs follows a fixed chain, not a flat forward.” It was true when written and became INCOMPLETE rather than wrong on 2026-08-30 — seven legs now reach the pair. It is widened, with this paragraph attached, rather than deleted, so that a reader does not conclude either that Guest is still on the appliance or that Guest resolves .house internally.)

Addresses are in Resolver addresses on each ULA leg for IPv6; the IPv4 last octet mirrors it (.5 VIP, .6 sandiego, .7 portland). The pair holds those floating addresses under keepalived, so the VIP answers from whichever node currently owns it.

• unbound is authoritative for every .house name and answers its configured RR types; any unlisted type (including type 65/HTTPS) gets NODATA, closing the split-horizon ECH leak that previously let clients see a public HTTPS record for an internal-only name.
• dig SOA fzymgc.house returns REFUSED, and that is by design — the zone is a set of per-name local-zone … static entries with no apex SOA, under a local-zone: "." refuse default. Verify with a curated name (dig +short nas.fzymgc.house), never with the apex. The textbook SOA probe manufactures a false incident here; see the runbook for the full note.
• A systemd .path unit watches the published zone directory on each resolver and reloads unbound whenever the file changes, so zone updates take effect without manual intervention.

The appliance chain, still live for un-renewed clients and the curated zone

RENAMED 2026-08-31 (03-04), and the rename is about the subject rather than the wording. This heading read “#### The appliance chain, still live for Guest, Telework and un-renewed clients”. 03-01 and 03-02 cut Guest and Telework onto the pair on 2026-08-30 and 2026-08-31, so the two segments the heading named as this chain’s population are no longer its population. What survives is what the heading now names: clients that have not renewed a lease and still ask the appliance, and the appliance’s continuing role as the producer of the curated .house zone. Note the boundary — the appliance also still serves DHCPv4 on both untrusted segments (D-04), which is a different statement from serving their DNS and is unchanged by this milestone. Before the rename, docs/ and tools/ were searched for anchor links to the old heading’s slug: zero hits, so nothing pointed at it. The search literal is deliberately not reproduced here — writing it into the page would make the same search match this note forever and turn a live check into a permanently red one.

client :53
  -> FW_PREROUTING_DNS_DEFAULT REDIRECT      (Firewalla nat, health-gated by FireMain)
  -> firerouter_dns (dnsmasq) on <vlan-gateway>:53
       |- addn-hosts=/home/pi/.firewalla/run/hosts   (per-MAC device names, LocalDomainSensor)
       |- address=/name/ip                          (Firewalla Custom DNS Rules)
       '- server=127.0.0.1#8953$unbound$*wan        (upstream entry: hands the query to unbound)
  -> unbound 127.0.0.1:8953   (the appliance's own unbound)
  -> Control D over DoT       (everything else)

• Queries land on firerouter_dns (Firewalla’s dnsmasq), bound to the VLAN gateway address the client’s traffic arrives on. It answers from addn-hosts and address= first, and forwards the rest to the appliance’s unbound at 127.0.0.1:8953. FireMain health-gates the :53 redirect in front of it on WAN-resolver reachability; see the 2026-07-23 CoE and the failure-mode runbook in the DNS runbook.
  • firemasq reporting inactive is expected on a Firewalla Gold and is not evidence about the listener. platform/gold/GoldPlatform.js overrides the DNS/DHCP service names to firerouter_dns/firerouter_dhcp, so the legacy unit name is simply wrong on this hardware and its state carries no diagnostic information. Check the listeners — sudo ss -tlnup | grep :53 — never the legacy unit.
  • The per-name address=/name/ip custom-rule path is not retired: it is one of the three live writers into fzymgc.house, alongside the Terraform-declared unbound zone and the Firewalla LocalDomainSensor per-MAC files. All three are documented at Why does this name resolve when there is no Terraform entry for it?.

See the DNS runbook for operational commands and the verification runbook, and docs/engineering/specs/2026-07-07-authoritative-house-resolver-design.md for the full design.

Ingress Architecture

Traffic Flow - External

Internet
    |
    v
Cloudflare Edge (WAF, DDoS protection)
    |
    v
Cloudflare Tunnel (encrypted)
    |
    v
cloudflared pod (namespace: cloudflared)
    |
    v
Traefik (namespace: traefik)
    |
    v
Backend Services

Traffic Flow - Internal

Internal Client
    |
    v
DNS: *.k8s.fzymgc.house -> MetalLB VIP
    |
    v
Traefik (namespace: traefik)
    |
    v
Backend Services

Traefik Configuration

Entrypoints

Name	Port	Protocol	Purpose
web	80	HTTP	Redirect to HTTPS
websecure	443	HTTPS	TLS termination

Middleware

Name	Namespace	Type	Purpose
oauth-auth-<app> (one per dashboard: temporal, hubble, traefik-dashboard, longhorn)	oauth2-proxy	ForwardAuth	Per-route Keycloak-backed SSO, gated by ?allowed_groups=<app>-users (ADR hl-1fia)
oauth-errors	oauth2-proxy	Errors	Converts oauth2-proxy’s 401/403 into a redirect to /oauth2/sign_in
redirect-https	traefik	RedirectScheme	HTTP to HTTPS
admin-allowlist	keycloak	IPAllowList	Restricts id.fzymgc.house /admin + /realms/master to 192.168.16.0/20 (the trusted-Main supernet: Main wired + MainWireless) + 100.64.0.0/10 (Tailscale CGNAT) for traffic that reaches Traefik directly. Uses default RemoteAddrStrategy; Traefik’s Service runs externalTrafficPolicy: Local so this Middleware sees real client IPs, not a SNAT’d address (hl-ba3a). The Cloudflare-tunnel path never reaches this Middleware at all — it’s gated separately by a Cloudflare Access Application (tf/cloudflare/access.tf, keycloak_admin) — see argocd/app-configs/keycloak/ingress-route.yaml
offnet-marker	keycloak	Headers	Sets X-Fzymgc-Offnet: 1 on every id.fzymgc.house request that reaches Traefik from outside the trusted set; Keycloak’s passkey path forces OTP when the header is present (tf/keycloak/authentication.tf). A priority-2 route in the keycloak IngressRoute bypasses it for ClientIP matches: the same IPv4 CIDRs as admin-allowlist (the two lists change together), plus the Main and MainWireless ULA /64s, which match nothing while Traefik’s LoadBalancer is IPv4-only. The Cloudflare-tunnel path bypasses Traefik; Keycloak reads Cloudflare’s Cf-Connecting-IP / Cf-Ray headers there — see argocd/app-configs/keycloak/ingress-route.yaml

TLS Configuration

Certificate	Issuer	Domains
Wildcard	Let’s Encrypt	*.fzymgc.house
Per-service	Let’s Encrypt	Individual hostnames

MetalLB Configuration

MetalLB runs in the metallb namespace (argocd/app-configs/metallb/kustomization.yaml).

Address Pools

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
spec:
  addresses:
    - "192.168.20.145-192.168.20.149"
    - "192.168.20.155-192.168.20.159"
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab
spec:
  autoAssign: false
  addresses:
    - "192.168.200.145-192.168.200.149"
    - "192.168.200.155-192.168.200.159"

L2 Advertisement

All pools use L2 mode (ARP). Each advertisement is pinned to one interface (default → end0, lab → end0.1000), and the lab pool is never auto-assigned.

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
spec:
  ipAddressPools:
    - default
  interfaces:
    - end0
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lab
spec:
  ipAddressPools:
    - lab
  interfaces:
    - end0.1000

Network Policies (Cilium)

Policy Types

The cluster uses Cilium for policy enforcement (policyEnforcementMode: default):

• Standard Kubernetes NetworkPolicy (networking.k8s.io/v1) — Cilium honors these as L3/L4 rules
• CiliumNetworkPolicy (cilium.io/v2) — used when identity-aware rules (FQDN, ServiceAccount, etc.) or L7 awareness are needed
• CiliumClusterwideNetworkPolicy (cilium.io/v2) — cluster-scoped policies (not currently in use)

Cilium’s enforcer evaluates policies per pod identity. Pods with NO selecting policy default to allow-all (standard K8s NetworkPolicy semantics).

Active policies

13 CiliumNetworkPolicies are in force, across 9 namespaces (measured 2026-09-08). Both numbers are stated deliberately: dolt alone carries 5 of the 13, so reading the namespace count as a policy count is wrong.

Namespace	CNPs	Coverage	Manifest
clickhouse-mcp	1	ingress 8000 from the fronting gateways; egress ClickHouse 8123, kube-dns	argocd/app-configs/clickhouse-mcp/networkpolicy.yaml
clickstack	1	OTLP ingest 4317/4318, ClickHouse 8123, collector 8888, kube-dns, world 443	argocd/app-configs/clickstack/networkpolicy.yaml
dolt	5	SQL 3306, Remote API 50051, Prometheus 9418, L4-only DNS egress, kube-apiserver egress	argocd/app-configs/dolt/network-policies.yaml
firewalla-mcp	1	ingress 8080; egress kube-dns, world 443	argocd/app-configs/firewalla-mcp/networkpolicy.yaml
fovea	1	qdrant-fovea only: gRPC 6334 from the fovea client, 6333 from host for the kubelet probe	argocd/app-configs/fovea/qdrant-netpol.yaml
headroom-agents	1	ingress 8787; egress kube-dns, OTLP 4318, world 443	argocd/app-configs/headroom-agents/networkpolicy.yaml
headroom-apps	1	ingress 8787; egress kube-dns, OTLP 4318, Traefik 8443, world 443	argocd/app-configs/headroom-apps/networkpolicy.yaml
hl-assets	1	ingress 8080; egress kube-dns, OTLP 4318, world 443	argocd/app-configs/hl-assets/networkpolicy.yaml
kubernetes-mcp	1	ingress 8080 from litellm only; egress kube-dns, kube-apiserver, keycloak 8443 (JWKS + issuer discovery for local token validation, D-68 — without this rule the provider never builds and every call answers 401 temporarily_unavailable)	argocd/app-configs/kubernetes-mcp/networkpolicy.yaml

Zero of the 13 carry an L7 rules.dns block, and every namespace needing egress to named hosts is on the coarse toEntities: world fallback rather than toFQDNs. That is a measured decision, not an oversight — toFQDNs fails on six of the eight nodes in either DNS-interception mode. See docs/operations/cilium.md § DNS L7 visibility.

Crown-jewel namespace coverage is tracked under the post-migration hardening initiative (see docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md).

History note

The cluster previously used Calico for both CNI and policy enforcement. Migration to Cilium completed 2026-05-12 (PR3 #1017 + PR4). See docs/operations/cilium-migration.md for the full migration log.

Service Routing (Cilium KPR)

Production-mode posture (since 2026-05-13)

The cluster runs Cilium kube-proxy replacement in Strict mode. kube-proxy is no longer running on any node.

Setting	Value	Source
kube-proxy-replacement	true (Strict mode + Socket LB)	cilium-config ConfigMap
enable-host-legacy-routing	false (eBPF native routing for host net ns)	cilium-config
enable-bpf-masquerade	true (preserves source-NAT for pod-to-external)	cilium-config
KUBERNETES_SERVICE_HOST / _PORT	192.168.20.140 (kube-vip VIP) / 6443	Cilium agent pod env
disable-kube-proxy: true	on all 3 control-plane nodes	/etc/rancher/k3s/config.yaml

Service traffic path

1. Pod opens a connection to a Service ClusterIP
2. Cilium socketLB intercepts at the socket layer (BEFORE iptables) and rewrites the destination to a backend pod IP via the eBPF Service map
3. Packet flows directly to the backend pod (across nodes via the existing tunnel/VXLAN datapath if needed)

There are no KUBE-SVC-* iptables chains. The only KUBE-* chain on each node is KUBE-KUBELET-CANARY, which kubelet manages for its own iptables health-check.

Debugging

• Service routing issues: see docs/operations/cilium-kpr.md
• Service map state: cilium-dbg service list from any agent
• Real-time policy verdicts: cilium-dbg monitor filtered by endpoint ID

Migration history

Pre-cutover state had kube-proxy-replacement: false with k3s embedded kube-proxy programming iptables KUBE-SVC-* chains for Service routing. Cutover landed via PR #1076 (KPR enable + bpf flips) and PR #1078 (correct k3s disable-kube-proxy flag). Spec at docs/engineering/specs/2026-05-13-cilium-production-mode-hardening-design.md; plan at docs/engineering/plans/2026-05-13-cilium-production-mode-hardening.md.

Cloudflare Tunnel

Tunnel Configuration

Setting	Value
Tunnel Name	fzymgc-house-main
Namespace	cloudflared
Replicas	2
Protocol	QUIC (primary), HTTP/2 (fallback)

Ingress Rules

Configured in Cloudflare Zero Trust dashboard, routes to:

• http://traefik.traefik.svc:80 (most services)
• tcp://vault-active.vault.svc:8200 (Vault - TCP passthrough)

Port Reference

External Ports

Port	Protocol	Service
443	HTTPS	All web services (via Cloudflare)
6443	HTTPS	Kubernetes API (internal only)

Internal Service Ports

Service	Port	Protocol
Traefik	80/443	HTTP/HTTPS
Vault	8200	HTTPS
Keycloak	443	HTTPS
Grafana	3000	HTTP
ArgoCD	80/443	HTTP/HTTPS
VictoriaMetrics	8428	HTTP
Loki	3100	HTTP
Dolt	3306	MySQL

Troubleshooting

Verify MetalLB

kubectl get ipaddresspools -n metallb
kubectl get l2advertisements -n metallb
kubectl get svc -A | grep LoadBalancer

Check Traefik Routes

kubectl get ingressroutes -A
kubectl get ingressroutetcps -A
kubectl get middlewares -A

DNS Resolution

# From within cluster
kubectl run -it --rm debug --image=busybox -- nslookup vault.vault.svc

# External resolution
dig vault.fzymgc.house

Network Policy Debugging

# Standard Kubernetes NetworkPolicies
kubectl get networkpolicies -A
kubectl describe networkpolicy <name> -n <namespace>

# CiliumNetworkPolicies (cilium.io/v2)
kubectl get ciliumnetworkpolicies -A
kubectl describe ciliumnetworkpolicy <name> -n <namespace>

# Check policy verdicts via Hubble Relay (cilium-cli)
hubble observe --verdict DROPPED
hubble observe --pod <namespace>/<pod>

# Check enforcement state on a specific node's cilium agent
kubectl exec -n kube-system <cilium-pod> -c cilium-agent -- \
  cilium-dbg endpoint list