Network Reference
Network configuration and addressing for the fzymgc-house cluster.
Three places define what this page describes. tf/unifi declares the UCG-Fiber gateway and the
UniFi switching fabric under it: the subnet and the gateway address of every VLAN leg, the firewall
zone of each leg, and the switch ports. The gateway routes every leg and serves no DHCP. The fabric
can veto a DHCP server: DHCP Guard drops offers from any server that is not on a network’s trusted
list. See UniFi. The resolver pair sandiego and portland owns its leg
addresses, the floating resolver addresses and the Kea DHCP pools, all declared in
dns_vlan_interfaces in ansible/inventory/host_vars/sandiego.yml and portland.yml. The
cluster’s own addressing (Cilium, Traefik) is in argocd/.
IP Addressing Summary
Section titled “IP Addressing Summary”| Network | CIDR | Purpose |
|---|---|---|
| Trusted-Main Supernet | 192.168.16.0/20 |
One covering CIDR for “Main, wired or wireless” in policies, for example the Keycloak admin allowlist. It covers Main and MainWireless. The unassigned 16-19 and 28-31 blocks are reserved for trusted-Main use. A network numbered inside this /20 gains trusted-policy access, so never number an untrusted network here |
| Main | 192.168.20.0/22 |
Trusted LAN, VLAN 3000, one L2 segment (.20.0 to .23.255) for the wired clients and the SSID kessel-run. Gateway 192.168.20.1. Resolver VIP .20.5, sandiego .20.6, portland .20.7. The Home Assistant VM ha .20.220. The printer office-printer .21.81, a Kea reservation below the pool. Kea pool 192.168.22.20 to 192.168.23.249, so a client can hold any address in the /22, not only .20.0/24 |
| MainWireless | 192.168.24.0/22 |
Trusted LAN, VLAN 3001, the same trust policy as Main. No SSID puts a client on it. Gateway 192.168.24.1. Resolver VIP .24.5, sandiego .24.6, portland .24.7. Kea pool 192.168.24.200 to 192.168.27.254. Static addresses sit below the pool. Not advertised on the tailnet, because wireless clients need no inbound path |
| Core | 192.168.40.0/22 |
Untagged native infrastructure LAN. Gateway 192.168.40.1. Holds the two resolvers (sandiego .40.6, portland .40.7), their Core VIP .40.5, heimdall .40.20, calgary .40.21, the NAS containers nas-support .40.32 and vault-unseal-nas .40.34, macvlan children of the NAS interface vlan1, which holds no address, and the Turing Pi BMCs tpi-alpha-bmc .40.40 and tpi-beta-bmc .40.41, untagged on uplinks whose native network is Core. Kea pool 192.168.41.10 to 192.168.43.249. Declared as network-core in the tailnet policy and advertised on the tailnet |
| IoT | 192.168.218.0/24 |
Trusted IoT LAN, VLAN 3020. Gateway 192.168.218.1. Resolver VIP .218.5, sandiego .218.6, portland .218.7. Kea pool 192.168.218.101 to 192.168.218.223 |
| Lab | 192.168.200.0/22 |
Trusted lab LAN, VLAN 1000. Gateway 192.168.200.1. Resolver VIP .200.5, sandiego .200.6, portland .200.7. Static space .200.140 to .200.159: the k3s node Lab legs .141 to .144 and .151 to .154, and free addresses .140, .145 to .149 and .155 to .159. Kea pool 192.168.200.160 to 192.168.201.19. Lab is not on the tailnet |
| Guest | 192.168.79.0/24 |
Untrusted segment, VLAN 3100. Gateway 192.168.79.1. Kea on the pair serves its DHCPv4 and advertises the pair’s VIP 192.168.79.5 as the only DNS server, with no search domain. The pair holds .79.6 (sandiego) and .79.7 (portland) with trusted: false, so the public recursor process pdns-recursor@public answers this segment from public DNS only. Kea pool 192.168.79.20 to 192.168.79.249 (subnet id 6), with a 600-second lease and no DNS update. Not inside the trusted-Main supernet |
| Telework | 192.168.217.0/24 |
Untrusted segment, VLAN 3200. Gateway 192.168.217.1. The same arrangement as Guest: Kea DHCPv4, VIP 192.168.217.5, sandiego .217.6, portland .217.7, trusted: false. Kea pool 192.168.217.20 to 192.168.217.249 (subnet id 7), with a 600-second lease and no DNS update. Not inside the trusted-Main supernet |
| Node Network | 192.168.20.0/24 |
Physical node addresses, a subset of Main |
| Pod Network | 10.245.0.0/16, fd00:10:245::/48 |
Kubernetes pods, from Cilium cluster-pool IPAM. Cilium does not use the k3s cluster-cidr IPv4 value 10.42.0.0/16 |
| Service Network | 10.43.0.0/16, fd00:10:43::/112 |
Kubernetes Services |
| Routed VIPs | 192.168.250.0/24 |
LoadBalancer VIPs from Cilium LB-IPAM. The nodes announce each VIP over BGP to the UCG-Fiber. No leg holds an address in it, so no client is on-link with a VIP. Not inside the trusted-Main supernet. 192.168.250.N mirrors fddb:f665:73f7:250::N |
| Routed VIPs, IPv6 | fddb:f665:73f7:250::/64 |
The IPv6 match of the routed VIPs. A routed prefix, not a leg, so it takes no ULA leg index |
Kea on the pair serves DHCPv4 on all seven legs. No device serves DHCPv6. A client forms its IPv6 addresses by SLAAC from the router advertisements of the gateway and of the pair. See IPv6 ULA Addressing.
Adding a leg to dns_vlan_interfaces is not the same as serving it. Five consumers read that
structure: the systemd-networkd legs, the nftables accept sets, the listen addresses of the two
recursor processes,
keepalived’s vrrp_instance blocks and the RA announcer’s configuration. Each one needs its own
converge and its own adoption step. A leg declared but not converged everywhere presents as a total
DNS failure on that segment. The DNS runbook has the per-consumer checks.
WiFi SSIDs
Section titled “WiFi SSIDs”One SSID per wireless VLAN. The UniFi access points broadcast them, and tf/unifi/wlans.tf
declares them. A client joins a VLAN by SSID and by nothing else.
| SSID | VLAN | Security |
|---|---|---|
kessel-run |
Main (3000) | WPA2/WPA3 |
kessel-run-iot |
IoT (3020) | WPA2 only, PMF off |
kessel-run-g |
Guest (3100) | Enhanced Open (OWE) with a captive portal, wireless clients isolated from each other |
kessel-run-tw |
Telework (3200) | WPA3 only |
2.4 GHz channels
Section titled “2.4 GHz channels”The UniFi controller chooses the Wi-Fi channels. On 2.4 GHz it uses only channels 1, 6 and 11.
ap-office serves no 2.4 GHz. Three 802.15.4 networks share the band with Wi-Fi. A Zigbee
network keeps the channel that it forms on. So the ZHA network forms last, after this table holds
the Thread and Hue channels and after a ZHA energy scan from the location of the coordinator. When
you read or set a channel, write it in its row.
| Network | Radio | 802.15.4 channel |
|---|---|---|
Thread (MyHome16) |
The Apple TV, the HomePod, the Aqara Hub M3 and the ZBT-2 | 25 (2475 MHz) |
| Hue Zigbee | The Hue Bridge Pro | 15 (2425 MHz) |
| ZHA Zigbee | The SLZB-06MG24 | Not formed. Thread holds 25 and Hue holds 15. Channel 20 (2450 MHz) is the free channel between Wi-Fi channels 6 and 11 |
What the tailnet advertises, and what it deliberately does not
Section titled “What the tailnet advertises, and what it deliberately does not”Advertised: 192.168.40.0/22 (Core), and the routed VIP prefixes 192.168.250.0/24 and
fddb:f665:73f7:250::/64. Nothing else, and no prefix inside the trusted-Main supernet
192.168.16.0/20. The list is k8s_tailscale_advertise_routes in
ansible/roles/k8s-tailscale/defaults/main.yml, and the tailnet policy tailscale/policy.hujson
auto-approves the same three routes for tag:nas and tag:subnet-router. Main is not advertised,
because every host a remote operator needs has a Core address, a routed VIP or a tailnet node of
its own, and a Main route on a client that sits on Main duplicates an on-link prefix.
Core is advertised so that an off-LAN operator can reach the resolvers, heimdall and the Core
VIP. Without it a remote session reaches 192.168.20.1 and nothing on 192.168.40.0/22, which
reads as “the resolver is down” when the truth is “this vantage cannot see it”.
Four constraints hold, and each has a reason that is not style:
- The gateway is never a subnet router. Only
tag:nasandtag:subnet-routerappear in the policy’sautoApprovers.routes. The Firewalla, the gateway before 2026-10-01, failed silently and intermittently as a subnet router. See the tailscale router subnet route breakage report. No advertiser sits on Core, so Core traffic is forwarded from the advertiser through the UCG-Fiber to Core by ordinary L3 routing. - Every advertiser announces the same set.
truenas-scale(primary) and thetpi-*-4standbys form a failover group, and Tailscale serves one at a time. A route the primary announces and the standbys do not disappears on failover, silently. To add a route, add it tok8s_tailscale_advertise_routesandnas_tailscale_advertise_routes, and converge both roles. No Main prefix is advertised, so a tailnet client reaches no macvlan child of the NAS on Main, such asprecipiceat192.168.22.109. The Core containersnas-supportandvault-unseal-nasare reachable through the Core route even whiletruenas-scaleis primary, because the NAS holds no Core address and forwards to them through the gateway. A host never reaches its own macvlan children. - Only the routed VIP prefix is advertised in IPv6. The VIPs publish AAAA records, and a tailnet
client can try the ULA before IPv4.
fddb:f665:73f7:250::/64holds the VIPs and no leg. It is the only ULA route on the tailnet. The rest offddb:f665:73f7::/48stays off, because each ULA route widens what the tailnet reaches and needs its own decision. Lab is not on the tailnet. - No source-bound IPv6 proof uses a tailnet vantage. A source-bound proof runs
dig -bfrom a host that holds a real address in the leg’s own/64. Subnet-route SNAT replaces the source address of a tailnet client, so a probe from a tailnet client looks on-link when it is not.
The /80 boundary is weaker than the /27 boundary. The Lab Kea pool starts at .160, so no
DHCP client gets an address inside the /27. The Lab RA gives clients SLAAC addresses in
fddb:f665:73f7:4::/64, and nothing keeps them out of the /80. A SLAAC address lands inside the
/80 only if its interface identifier starts with aaaa, a chance of 1 in 65,536 for each
address. Privacy addresses rotate, so over time a Lab client can land inside the /80.
IPv6 ULA Addressing
Section titled “IPv6 ULA Addressing”Unique Local Addresses (ULA) give the estate stable internal IPv6 addressing that does not depend on the ISP prefix.
| Prefix | CIDR | Purpose |
|---|---|---|
| ULA Base | fddb:f665:73f7::/48 |
Site-local IPv6 prefix |
VLAN Subnets
Section titled “VLAN Subnets”Each VLAN has a /64 from the ULA prefix:
| VLAN | IPv6 Subnet | Gateway ULA address |
|---|---|---|
| Main (3000) | fddb:f665:73f7:1::/64 |
fddb:f665:73f7:1::1 |
| Guest (3100) | fddb:f665:73f7:2::/64 |
fddb:f665:73f7:2::1 |
| IoT (3020) | fddb:f665:73f7:3::/64 |
fddb:f665:73f7:3::1 |
| Lab (1000) | fddb:f665:73f7:4::/64 |
fddb:f665:73f7:4::1 |
| Telework (3200) | fddb:f665:73f7:5::/64 |
fddb:f665:73f7:5::1 |
| Core (untagged) | fddb:f665:73f7:6::/64 |
fddb:f665:73f7:6::1 |
| MainWireless (3001) | fddb:f665:73f7:7::/64 |
fddb:f665:73f7:7::1 |
ULA is routed between all seven legs. The UCG-Fiber holds ::1 in the ULA /64 of each leg, so
it has a connected route to each prefix and forwards ULA traffic between them under its zone
policies. Each address is the ipv6.aliases value of the network in tf/unifi/networks.tf. The
gateway holds it only while the network takes a delegated prefix (ipv6.interface_type = "pd").
With the type none, no leg has a ULA gateway address.
Each leg also takes a global /64 from the /56 that FiOS delegates. The gateway picks the /64
of each leg itself, so the global prefix of a leg does not hold its ULA index. Guest takes a global
/64 too, but a Guest client has no IPv6 path to the internet, because the captive portal
authorizes a client over IPv4 only.
Guest and Telework are untrusted. The zone firewall of the gateway fences their traffic to the
other legs in both families: Guest is in the zone Hotspot and Telework is in the zone Telework.
The gateway redirects no DNS query on any leg. See
Zone firewall in the UniFi runbook.
This table is the allocating authority for ULA indices. Allocate a new leg here, in this table, and
nowhere else. Indices 1 to 7 are in use, in client addresses and in dns_vlan_interfaces on both
resolvers, so renumbering one invalidates live addresses. A new leg takes 8. Allocate here first,
then add the leg to ula_leg_index in ansible/inventory/group_vars/all.yml, never the reverse.
The inventory is a consumer of this table, not a second authority. Every ULA address in Ansible
derives from ula_leg_prefix.<leg>, which that file builds from ula_site_prefix and the index. A leg allocated without reading this table is a /64
collision with a live broadcast domain, and nothing in the estate reports it.
fddb:f665:73f7:250::/64 is the routed VIP prefix, not a leg, and it takes no index. No leg holds
an address in it. The UCG-Fiber learns a host route for each VIP in it over BGP from the nodes.
Its fourth hextet reads 250 to match the IPv4 block 192.168.250.0/24. A new leg still takes
index 8. The variables are lb_vip_prefix_ipv4 and lb_vip_prefix_ipv6 in
ansible/inventory/group_vars/all.yml.
Resolver addresses on each ULA leg
Section titled “Resolver addresses on each ULA leg”The resolver pair holds a floating address and one address per node on every leg. The last hextet
mirrors the IPv4 last octet, so an address reads the same in both stacks: ::5 VIP, ::6
sandiego, ::7 portland. ::1 is the gateway, held by the UCG-Fiber on every leg (see VLAN
Subnets above).
| VLAN | ULA /64 |
VIP (::5) |
sandiego (::6) |
portland (::7) |
|---|---|---|---|---|
| Main (3000) | fddb:f665:73f7:1::/64 |
fddb:f665:73f7:1::5 |
fddb:f665:73f7:1::6 |
fddb:f665:73f7:1::7 |
| IoT (3020) | fddb:f665:73f7:3::/64 |
fddb:f665:73f7:3::5 |
fddb:f665:73f7:3::6 |
fddb:f665:73f7:3::7 |
| Lab (1000) | fddb:f665:73f7:4::/64 |
fddb:f665:73f7:4::5 |
fddb:f665:73f7:4::6 |
fddb:f665:73f7:4::7 |
| Core (untagged) | fddb:f665:73f7:6::/64 |
fddb:f665:73f7:6::5 |
fddb:f665:73f7:6::6 |
fddb:f665:73f7:6::7 |
| MainWireless (3001) | fddb:f665:73f7:7::/64 |
fddb:f665:73f7:7::5 |
fddb:f665:73f7:7::6 |
fddb:f665:73f7:7::7 |
| Guest (3100) | fddb:f665:73f7:2::/64 |
fddb:f665:73f7:2::5 |
fddb:f665:73f7:2::6 |
fddb:f665:73f7:2::7 |
| Telework (3200) | fddb:f665:73f7:5::/64 |
fddb:f665:73f7:5::5 |
fddb:f665:73f7:5::6 |
fddb:f665:73f7:5::7 |
Guest and Telework carry trusted: false, a kea_subnet_id and a kea_pool. Kea sends no DNS
update and no search domain on either leg. The pair announces
RA with RDNSS on both legs. What a dual-stack client on those legs learns, and what it gets when it
asks, is in the DNS runbook.
The suffix of every address above is in dns_vlan_interfaces in
ansible/inventory/host_vars/sandiego.yml and portland.yml. The /64 of each leg comes from
ula_leg_prefix in ansible/inventory/group_vars/all.yml.
The digit repetition is a readability hazard, and the eye is not the control
Section titled “The digit repetition is a readability hazard, and the eye is not the control”The /64 index and the node suffix sit in different hextets and use the same small integers, so
several addresses repeat a digit for two unrelated reasons:
fddb:f665:73f7:6::6is Core (index 6) on sandiego (suffix 6). The two sixes mean different things.fddb:f665:73f7:7::7is MainWireless (index 7) on portland (suffix 7).fddb:f665:73f7:5::7looks like a portland address and is portland’s own address on the Telework leg (index 5). It appears once in the estate, on theteleworkentry inansible/inventory/host_vars/portland.yml. A reviewer who deletes it as a typo deletes a live address.
Do not check an address by eye. Parse it with ip_network, test containment in the leg’s /64,
and compare the exact value against the tables above.
Lab ULA addresses of the nodes
Section titled “Lab ULA addresses of the nodes”Each k3s node holds one static Lab ULA on end0.1000, under the prefix fddb:f665:73f7:4:aaaa::.
The suffix is the last octet of the IPv4 Lab address of the node. netplan sets the address
statically, and the leg keeps accept-ra: false, so it takes no SLAAC address. The rule at
priority 6000 sends replies from the Lab /64 to table 1000. That table holds the on-link Lab
/64 route and a default route via fddb:f665:73f7:4::1 on end0.1000. A reply to a Lab
destination leaves on-link. A reply to any other destination leaves through ::1. The prefix is
tp2_node_lab_ula_prefix in ansible/roles/tp2-bootstrap-node/defaults/main.yml, derived from
ula_leg_prefix.lab in ansible/inventory/group_vars/all.yml.
| Suffixes | First address | Last address | Holder |
|---|---|---|---|
::141-::144 |
fddb:f665:73f7:4:aaaa::141 |
fddb:f665:73f7:4:aaaa::144 |
tpi-alpha-1 to tpi-alpha-4 |
::151-::154 |
fddb:f665:73f7:4:aaaa::151 |
fddb:f665:73f7:4:aaaa::154 |
tpi-beta-1 to tpi-beta-4 |
Configuration
Section titled “Configuration”The resolver pair announces the ULA prefixes. The gateway announces them too, because its router
advertisement carries every prefix on the bridge of a network, and the ULA ::1 is on that bridge.
dns-common-ra-announce.service on sandiego and portland runs radvd against a configuration
rendered from dns_vlan_interfaces, one interface stanza per leg. Each stanza advertises that
leg’s ULA /64 with:
| Directive | Value | Why |
|---|---|---|
AdvPreferredLifetime |
3600 |
Finite, so the prefix can be deprecated. Under RFC 4862 section 5.5.3(e) a host retires a prefix only by watching these count down. With infinite there is no withdrawal path |
AdvValidLifetime |
7200 |
Finite, and at the two-hour clamp of section 5.5.3(e), so a reduction is honored rather than discarded |
AdvDefaultLifetime |
0 |
The resolvers are not routers. A zero router lifetime lets a non-router announce a prefix while the UCG-Fiber stays the sole default gateway |
AdvRDNSSLifetime |
1800 |
The RFC 8106 RDNSS option that names the leg’s vip6 as the recursive resolver |
The values are dns_common_ra_announce_*_seconds in ansible/roles/dns-common/defaults/main.yml.
The template is ansible/roles/dns-common/templates/dns-common-ra-announce.conf.j2. Clients
configure themselves through SLAAC.
The announcement of the pair does not depend on the gateway. The gateway holds the ULA ::1 of a
leg only while the network takes a delegated prefix. With ipv6_interface_type = "none", the
bridge of the network holds a link-local address only.
RDNSS has two sources on every leg, and they must agree. The pair’s RA carries the leg’s vip6.
The gateway’s RA carries the servers of dhcp_v6_server.dns_servers on the network in
tf/unifi/networks.tf, and each leg states its own vip6 there and nothing else. RFC 8106
section 5.3.1 leaves multi-source ordering to the host, so two sources that disagree are a coin
flip rather than a fallback. The gateway runs no DHCPv6 service, so no leg has a DHCP-sourced DNS
server. See Network fields in the UniFi runbook.
Node Addresses
Section titled “Node Addresses”Control Plane Nodes
Section titled “Control Plane Nodes”| Node | IP Address | Lab leg (end0.1000) | Role |
|---|---|---|---|
| tpi-alpha-1 | 192.168.20.141 |
192.168.200.141 |
Control plane (etcd) |
| tpi-alpha-2 | 192.168.20.142 |
192.168.200.142 |
Control plane (etcd) |
| tpi-alpha-3 | 192.168.20.143 |
192.168.200.143 |
Control plane (etcd) |
Worker Nodes
Section titled “Worker Nodes”| Node | IP Address | Lab leg (end0.1000) | Board |
|---|---|---|---|
| tpi-alpha-4 | 192.168.20.144 |
192.168.200.144 |
alpha |
| tpi-beta-1 | 192.168.20.151 |
192.168.200.151 |
beta |
| tpi-beta-2 | 192.168.20.152 |
192.168.200.152 |
beta |
| tpi-beta-3 | 192.168.20.153 |
192.168.200.153 |
beta |
| tpi-beta-4 | 192.168.20.154 |
192.168.200.154 |
beta |
Each node’s Lab leg routes Lab-sourced traffic through table 1000 (rule priority 6000), and the Kubernetes InternalIP stays on the Main address. See Lab service VLAN.
The k3s node-ip of each node lists the IPv4 address above first and then the Main ULA
fddb:f665:73f7:1:aaaa::<host octet> (host_address and host_address_ipv6 in the node’s
host_vars).
Virtual IPs
Section titled “Virtual IPs”| Address | Purpose | Provider |
|---|---|---|
192.168.250.145, fddb:f665:73f7:250::145 |
Traefik, Service traefik/traefik-vip |
Cilium LB-IPAM pool routed-vips, announced over BGP |
192.168.250.146, fddb:f665:73f7:250::146 |
Temporal gateway for the node workers on the resolver Pis, Service temporal/temporal-node-gateway-vip on TCP 7233. The workers dial the IPv4 literal. The network policy admits the Pis only, at 192.168.40.6 and 192.168.40.7 on Core |
Cilium LB-IPAM pool routed-vips, announced over BGP |
192.168.250.147, fddb:f665:73f7:250::147 |
Dolt SQL on TCP 3306, Service dolt/dolt-vip |
Cilium LB-IPAM pool routed-vips, announced over BGP |
192.168.250.148, fddb:f665:73f7:250::148 |
MQTT over TLS on TCP 8883, Service mosquitto/mosquitto-vip |
Cilium LB-IPAM pool routed-vips, announced over BGP |
192.168.250.149, fddb:f665:73f7:250::149 |
OTLP on TCP 4317 and 4318, Service clickstack/otel-collector-vip |
Cilium LB-IPAM pool routed-vips, announced over BGP |
192.168.250.150, fddb:f665:73f7:250::150 |
Syslog on UDP 514 from the UniFi gateway, switches and APs, Service otel-scraper/network-collector-vip. The devices send to the IPv4 address. The network policy admits the Lab address of the gateway, 192.168.200.1, and Core, 192.168.40.0/22, only |
Cilium LB-IPAM pool routed-vips, announced over BGP |
192.168.250.254, fddb:f665:73f7:250::254 |
Kubernetes API on TCP 6443, Service kube-system/kubernetes-api. The three servers announce it |
Cilium LB-IPAM pool api-vip, announced over BGP |
The gateway holds a blackhole route for 192.168.250.0/24 and for fddb:f665:73f7:250::/64
(tf/unifi/static-routes.tf). It drops the traffic for a VIP that no node announces, so that
traffic does not leave through the default route to the WAN.
DNS Configuration
Section titled “DNS Configuration”External DNS (Cloudflare)
Section titled “External DNS (Cloudflare)”Terraform in tf/cloudflare is the source of truth for every public record in every zone of the
account. See Cloudflare. Three classes of name reach the cluster from
the internet, all through the Cloudflare Tunnel:
| Name | What it reaches |
|---|---|
id.fzymgc.house |
Keycloak, for the Cloudflare Access OIDC back-channel |
<service>-wh.fzymgc.net |
Webhook receivers (webhook_services in tf/cloudflare/variables.tf) |
<service>.fzymgc.net |
Internal services behind Cloudflare Access (internal_services) |
The webhook and Access names live under fzymgc.net so that they never collide with the internal
fzymgc.house view. The rest of the fzymgc.house zone is mail, GitHub Pages and off-site hosts,
declared in tf/cloudflare/dns-fzymgc_house.tf.
Internal DNS names
Section titled “Internal DNS names”The internal resolvers answer these names under fzymgc.house with the routed VIPs. Each name
has an A record and an AAAA record. See
DNS records for the writer of each record.
| Record | Type | Target |
|---|---|---|
Traefik-routed hosts, for example id, argocd and *.k8s |
A, AAAA | Traefik VIPs (192.168.250.145, fddb:f665:73f7:250::145) |
doltdb |
A, AAAA | Dolt VIPs (192.168.250.147, fddb:f665:73f7:250::147) |
mqtt |
A, AAAA | Mosquitto VIPs (192.168.250.148, fddb:f665:73f7:250::148) |
Names under k8s.fzymgc.house (for example vault.k8s, karakeep.k8s, traefik.k8s) are
Traefik-routed hosts like any other. They do not pass through Cloudflare.
Kubernetes DNS (CoreDNS)
Section titled “Kubernetes DNS (CoreDNS)”| Pattern | Resolution |
|---|---|
<svc>.<ns>.svc.cluster.local |
Service ClusterIP |
<pod-ip-dashed>.<ns>.pod.cluster.local |
Pod IP |
<svc>.<ns>.svc |
Short form (within cluster) |
Internal .house resolver
Section titled “Internal .house resolver”The resolver pair sandiego and portland answers client DNS on every VLAN. Client DNS follows a
fixed chain, not a flat forward:
client :53 -> the advertised resolver address on the client's VLAN (VIP .5, or .6/.7 per node) -> pdns-recursor@main on the resolver pair <-- the client-facing tier, per-VLAN listeners |- the policy zones (RPZ): block lists, allow list, the entries of dns-control |- forward zone: fzymgc.house -> PowerDNS on the loopback listener '- everything else -> dnscrypt-proxy on 127.0.0.1:5300 -> a relay -> a public resolver -> PowerDNS (authoritative for fzymgc.house and the reverse zones)
client :53 on Guest or Telework -> the leg VIP or node address -> pdns-recursor@public on the resolver pair <-- no path to PowerDNS |- the withheld zone: fzymgc.house and the reverse zones -> NXDOMAIN '- every other name -> dnscrypt-proxy on 127.0.0.1:5300 -> a relay -> a public resolverThe client-facing tier is PowerDNS Recursor, in two processes on each node. Every block rule and
allow rule of it is a response policy zone (RPZ), a DNS zone whose records are rules. What the
chain returns differs by trust. On the five trusted legs it resolves fzymgc.house internally.
Guest and Telework (trusted: false) use the second chain, so a client on those legs gets
NXDOMAIN for a .house name. See
the ADR and
the client view
in the DNS runbook.
The addresses are in Resolver addresses on each ULA leg above for IPv6. The IPv4 last octet
mirrors it (.5 VIP, .6 sandiego, .7 portland). keepalived holds the floating addresses, so
the VIP answers from whichever node owns it.
PowerDNS holds the curated zone. dns-control and external-dns write records through its API, and lightningstream reconciles the two nodes. The loopback listener is
dns_authoritative_listen (127.0.0.1:1053) in ansible/inventory/group_vars/dns_hosts.yml. See
the PowerDNS authoritative tier.
The UCG-Fiber keeps a small DNS role beside the pair. Its dnsmasq answers on the .1 address of
every leg and forwards each query to the Core VIP 192.168.40.5, which is the IPv4 DNS server of
both WANs. So a client on a trusted leg that carries a hand-set .1 nameserver gets the internal
view. The zone firewall blocks DNS to the gateway from Guest and Telework. The gateway serves no
DHCP, produces no curated zone and redirects no DNS query. See
Zone firewall in the UniFi runbook. The design of the pair is
in docs/engineering/specs/2026-07-07-authoritative-house-resolver-design.md.
Ingress Architecture
Section titled “Ingress Architecture”Traffic Flow - External
Section titled “Traffic Flow - External”Internet | vCloudflare Edge (WAF, DDoS protection) | vCloudflare Tunnel (encrypted) | vcloudflared pod (namespace: cloudflared) | vTraefik (namespace: traefik), or Keycloak directly for id.fzymgc.house | vBackend ServicesTraffic Flow - Internal
Section titled “Traffic Flow - Internal”Internal Client | vDNS: <name>.fzymgc.house -> Traefik VIP (192.168.250.145 or fddb:f665:73f7:250::145) | vUCG-Fiber: BGP route to the Lab address of a node with a ready Traefik endpoint | vTraefik (namespace: traefik) | vBackend ServicesTraefik Configuration
Section titled “Traefik Configuration”Traefik is deployed by argocd/cluster-app/templates/traefik.yaml. Its Service is the dual-stack
traefik-vip LoadBalancer on the routed VIP prefixes with externalTrafficPolicy: Local, so every
Middleware below sees the real client address.
Entrypoints
Section titled “Entrypoints”| Name | Port | Protocol | Purpose |
|---|---|---|---|
| web | 80 | HTTP | Redirects to websecure |
| websecure | 443 | HTTPS | TLS termination |
| traefik | 8080 | HTTP | Dashboard, reached only through the traefik-internal Service and its IngressRoute |
Middleware
Section titled “Middleware”| Name | Namespace | Type | Purpose |
|---|---|---|---|
oauth-auth-<app> (one per dashboard: temporal, hubble, traefik-dashboard, longhorn) |
oauth2-proxy | ForwardAuth | Per-route Keycloak-backed SSO, gated by ?allowed_groups=<app>-users |
oauth-errors |
oauth2-proxy | Errors | Converts oauth2-proxy’s 401 and 403 into a redirect to /oauth2/sign_in |
admin-allowlist |
keycloak | IPAllowList | Restricts id.fzymgc.house /admin and /realms/master to traffic that reaches Traefik directly from the trusted set: 192.168.16.0/20 (the trusted-Main supernet), 100.64.0.0/10 (Tailscale CGNAT), and fddb:f665:73f7:1::/64 and fddb:f665:73f7:7::/64 (the Main and MainWireless ULA /64s). The Middleware uses the default RemoteAddrStrategy. The Cloudflare tunnel path never reaches this Middleware. A Cloudflare Access Application gates that path (tf/cloudflare/access.tf). See argocd/app-configs/keycloak/ingress-route.yaml |
offnet-marker |
keycloak | Headers | Sets X-Fzymgc-Offnet: 1 on every id.fzymgc.house request that reaches Traefik from outside the trusted set. If the header is present, Keycloak’s passkey path forces OTP (tf/keycloak/authentication.tf). A priority-2 route in the keycloak IngressRoute bypasses this Middleware for ClientIP matches on the same CIDRs as admin-allowlist. The two lists change together. On the Cloudflare tunnel path Keycloak reads the Cf-Connecting-IP and Cf-Ray headers instead |
llm-hsts |
litellm | Headers | HSTS on llm.fzymgc.house |
miniflux-greader-stripprefix |
miniflux | StripPrefix | Strips /api/greader.php for the Google Reader API of Miniflux |
redirect-to-cloudflare |
uptime-kuma | RedirectRegex | Sends status.fzymgc.house to status.fzymgc.net |
To list what is in force:
kubectl get middlewares -ATLS Configuration
Section titled “TLS Configuration”| Item | Value |
|---|---|
| Default certificate | The secret wildcard-fzymgc-house-tls in the traefik namespace, served for any host without its own certificate |
cloudflare-acme-issuer |
ClusterIssuer, Let’s Encrypt through the Cloudflare DNS-01 solver (argocd/app-configs/cert-manager-config/cloudflare-acme-issuer.yaml), for public names |
vault-issuer |
ClusterIssuer, the internal Vault CA (vault-certmanager-issuer.yaml), for in-cluster hops |
LoadBalancer IPAM and BGP
Section titled “LoadBalancer IPAM and BGP”Cilium LB-IPAM assigns each LoadBalancer VIP from the pool routed-vips, and the Cilium BGP
control plane announces the VIP to the UCG-Fiber. The objects are in
argocd/app-configs/cilium-lb/:
| Object | File | Role |
|---|---|---|
CiliumLoadBalancerIPPool routed-vips |
ip-pool.yaml |
The ranges of the routed VIP prefixes, below the reserved .254 and ::254. It serves only the namespaces in its serviceSelector |
CiliumBGPClusterConfig ucg-fiber |
bgp.yaml |
One BGP instance on every node, with one peer per family at the Lab gateway |
CiliumBGPPeerConfig ucg-fiber-v4 and ucg-fiber-v6 |
bgp.yaml |
The timers, graceful restart, the family and the advertisement label of each peer |
CiliumBGPAdvertisement lb-vips |
bgp.yaml |
Announces the LoadBalancer addresses of every Service |
A Service gets a VIP only when it sets loadBalancerClass: io.cilium/bgp-control-plane, because
the cilium role sets defaultLBServiceIPAM to none. The gateway side is the FRR file in
tf/unifi/bgp.tf. See BGP in the UniFi runbook and
The LoadBalancers in the Lab runbook.
Network Policies (Cilium)
Section titled “Network Policies (Cilium)”Policy Types
Section titled “Policy Types”The cluster uses Cilium for policy enforcement (policyEnforcementMode: default,
cilium_policy_enforcement_mode in ansible/roles/cilium/defaults/main.yml):
- Standard Kubernetes NetworkPolicy (
networking.k8s.io/v1). Cilium honors these as L3 and L4 rules. - CiliumNetworkPolicy (
cilium.io/v2). Used when a rule needs identity awareness (FQDN, ServiceAccount) or L7 awareness. - CiliumClusterwideNetworkPolicy (
cilium.io/v2). Cluster-scoped. The repository declares none.
Cilium evaluates policies per pod identity. A pod that no policy selects defaults to allow-all, which is the standard Kubernetes NetworkPolicy semantics.
Active policies
Section titled “Active policies”The repository declares CiliumNetworkPolicies in nine namespaces. dolt alone carries five and
litellm two, so the namespace count is not the policy count.
| Namespace | CNPs | Coverage | Manifest |
|---|---|---|---|
clickhouse-mcp |
1 | ingress 8000 from the fronting gateways; egress ClickHouse 8123, kube-dns | argocd/app-configs/clickhouse-mcp/networkpolicy.yaml |
clickstack |
1 | OTLP ingest 4317 and 4318, ClickHouse 8123, collector 8888, kube-dns, world 443 |
argocd/app-configs/clickstack/networkpolicy.yaml |
dolt |
5 | SQL 3306, Remote API 50051, Prometheus 9418, L4-only DNS egress, kube-apiserver egress | argocd/app-configs/dolt/network-policies.yaml |
firewalla-mcp |
1 | ingress 8080; egress kube-dns, world 443 |
argocd/app-configs/firewalla-mcp/networkpolicy.yaml |
headroom-agents |
1 | ingress 8787; egress kube-dns, OTLP 4318, world 443 |
argocd/app-configs/headroom-agents/networkpolicy.yaml |
hl-assets |
1 | ingress 8080; egress kube-dns, OTLP 4318, world 443 |
argocd/app-configs/hl-assets/networkpolicy.yaml |
kubernetes-mcp |
1 | ingress 8080 from litellm only; egress kube-dns, kube-apiserver, keycloak 8443 for JWKS and issuer discovery. Without that egress the provider never builds and every call answers 401 temporarily_unavailable |
argocd/app-configs/kubernetes-mcp/networkpolicy.yaml |
litellm |
2 | proxy ingress 4000 from Traefik, the HCP Terraform agents, life-agent, octopus, karakeep and mealie; valkey ingress from the proxy | argocd/app-configs/litellm/networkpolicy.yaml |
To compare the declared set with what is in force:
kubectl get ciliumnetworkpolicies -ANo policy carries an L7 rules.dns block, and every namespace that needs egress to named hosts
uses the coarse toEntities: world rule rather than toFQDNs. That is a decision, not an
oversight: toFQDNs fails on most of the nodes in either DNS-interception mode. See
DNS L7 visibility in the Cilium runbook.
Crown-jewel namespace coverage is tracked under the post-migration hardening initiative
(docs/engineering/specs/2026-05-12-cilium-post-migration-hardening-design.md).
Service Routing (Cilium KPR)
Section titled “Service Routing (Cilium KPR)”Posture
Section titled “Posture”The cluster runs Cilium kube-proxy replacement. kube-proxy runs on no node. The values are in
ansible/roles/cilium/defaults/main.yml and ansible/roles/k3s-common/defaults/main.yml.
| Setting | Value | Source |
|---|---|---|
kube-proxy-replacement |
true (Socket LB) |
cilium_kube_proxy_replacement |
enable-host-legacy-routing |
false (eBPF native routing for the host network namespace) |
cilium_bpf_host_legacy_routing |
enable-bpf-masquerade |
true (preserves source NAT for pod-to-external) |
cilium_bpf_masquerade |
routing-mode |
tunnel, VXLAN on port 8473 |
cilium_routing_mode, cilium_tunnel_protocol, cilium_tunnel_port |
KUBERNETES_SERVICE_HOST / _PORT |
127.0.0.1 / 6444, kube-apiserver on a server and the k3s client load balancer on an agent, so no node dials the API VIP |
cilium_k8s_service_host, cilium_k8s_service_port |
disable-kube-proxy: true |
on all 3 control-plane nodes | k3s_disable_kube_proxy, rendered into /etc/rancher/k3s/config.yaml |
| devices | end0, end0.1000 and end0.3000 |
cilium_devices |
Service traffic path
Section titled “Service traffic path”- A pod opens a connection to a Service ClusterIP.
- Cilium socket LB intercepts at the socket layer, before iptables, and rewrites the destination to a backend pod IP through the eBPF Service map.
- The packet flows directly to the backend pod, across nodes through the VXLAN tunnel when needed.
There are no KUBE-SVC-* iptables chains. The only KUBE-* chain on each node is
KUBE-KUBELET-CANARY, which kubelet manages for its own iptables health check.
Debugging
Section titled “Debugging”- Service routing issues: see Cilium KPR.
- Service map state:
cilium-dbg service listfrom any agent. - Real-time policy verdicts:
cilium-dbg monitorfiltered by endpoint ID.
Cloudflare Tunnel
Section titled “Cloudflare Tunnel”Tunnel Configuration
Section titled “Tunnel Configuration”| Setting | Value |
|---|---|
| Tunnel name | fzymgc-house-main (tunnel_name in tf/cloudflare/variables.tf) |
| Namespace | cloudflared (argocd/app-configs/cloudflared-main/) |
| Replicas | 2 |
Ingress Rules
Section titled “Ingress Rules”tf/cloudflare/tunnel.tf declares the ingress rules and the DNS records that point at the tunnel:
id.fzymgc.housetohttps://keycloak-service.keycloak.svc.cluster.local:8443.<service>-wh.fzymgc.netto each webhook receiver inwebhook_services.<service>.fzymgc.netto each service ininternal_services, behind Cloudflare Access.<service>.fzymgc.netto each SSH bastion inssh_services, through the Access browser terminal.- Anything else answers
http_status:404.
Port Reference
Section titled “Port Reference”External Ports
Section titled “External Ports”| Port | Protocol | Service |
|---|---|---|
| 443 | HTTPS | All web services (through Cloudflare) |
| 6443 | HTTPS | Kubernetes API (internal only) |
Internal Service Ports
Section titled “Internal Service Ports”| Service | Port | Protocol |
|---|---|---|
| Traefik | 80/443 | HTTP/HTTPS |
| Vault | 8200 | HTTPS |
| Keycloak | 8443 | HTTPS |
| Grafana | 3000 | HTTP |
| ArgoCD | 80/443 | HTTP/HTTPS |
| VictoriaMetrics | 8428 | HTTP |
| Loki | 3100 | HTTP |
| Dolt | 3306 | MySQL |
Troubleshooting
Section titled “Troubleshooting”Verify the LoadBalancers
Section titled “Verify the LoadBalancers”Run the ssh command with SSH to the console of the UCG-Fiber:
kubectl get ciliumloadbalancerippools routed-vipskubectl get svc -A | rg LoadBalancerssh root@192.168.40.1 "vtysh -c 'show ip route bgp' -c 'show ipv6 route bgp'"Check Traefik Routes
Section titled “Check Traefik Routes”kubectl get ingressroutes -Akubectl get ingressroutetcps -Akubectl get middlewares -ADNS Resolution
Section titled “DNS Resolution”# From within clusterkubectl run -it --rm debug --image=busybox -- nslookup vault.vault.svc
# From a LAN clientdig vault.fzymgc.houseNetwork Policy Debugging
Section titled “Network Policy Debugging”# Standard Kubernetes NetworkPolicieskubectl get networkpolicies -Akubectl describe networkpolicy <name> -n <namespace>
# CiliumNetworkPolicies (cilium.io/v2)kubectl get ciliumnetworkpolicies -Akubectl describe ciliumnetworkpolicy <name> -n <namespace>
# Check policy verdicts via Hubble Relay (cilium-cli)hubble observe --verdict DROPPEDhubble observe --pod <namespace>/<pod>
# Check enforcement state on a specific node's cilium agentkubectl exec -n kube-system <cilium-pod> -c cilium-agent -- \ cilium-dbg endpoint list