Design: calgary as the Home Assistant host
Context
Section titled “Context”See openspec/changes/calgary-home-assistant-host/proposal.md for the motivation and for the decisions Sean made on 2026-10-04.
An SSH reading of calgary on 2026-10-04 found this state:
- Ubuntu 24.04.4 with kernel 6.8 and a ZFS root on
rpool. - Incus 6.0.0 from the Ubuntu archive. Its storage pool
defaultis onrpool/incus, and its NAT bridge isincusbr0. One leftover VM,test-vm, runs. - libvirt 10.0.0 and QEMU 8.2.2 are installed too. A purge simulation of the libvirt packages
removes only
libvirt*, not Incus and not QEMU. - systemd-networkd runs the network from one file,
/etc/systemd/network/10-enp1s0.network, withDHCP=ipv4andIPv6AcceptRA=yes. The host has no netplan configuration. - unattended-upgrades is not installed.
- The user
fzymgchas passwordless sudo, and its login shell is fish. The host has norg. - The MAC of
enp1s0is78:55:36:01:3b:b8.
The estate sets these constraints:
docs/reference/network.mdis the addressing authority. On 2026-10-04 dns-control held.200,.201,.202,.204and.210in192.168.20.0/24, and.20and.25in192.168.40.0/24. The Kea pools start at192.168.22.20on Main and at192.168.41.10on Core.- dns-control is the only writer of curated names and DHCP reservations (
curated-entry-store,kea-reservation-fragment).tf/dns-recordsno longer exists. tf/unifideclares switch ports throughunifi_device_ports. The resource patches theport_overridesarray of the device record as raw JSON (docs/engineering/specs/2026-09-19-unifi-device-ports-design.md, decision D7).- The tier table in
docs/operations/unifi.mdkeeps the radios of the access points and the channel plan controller-owned. - A
tf/unifipull request merges only when itsmain-cluster-unifiplan changes nothing that the pull request does not change. The workspace applies only after a person confirms the run (“Adopt a UI change” indocs/operations/unifi.md). - heimdall, the resolver pair and the NAS each run Ansible through a wrapper in
scripts/. The wrapper reads an SSH key from Vault and uses no SSH agent.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- One playbook converges calgary from a workstation, and a second run in check mode changes nothing.
- Ansible owns the definition of the VM. HAOS owns everything inside the VM.
- The live network change happens once, and its rollback works when the host is unreachable.
Non-Goals:
- A shared OpenTelemetry Collector role for all hosts.
dns-otel-collectorstays as it is. - Removing
incusbr0or the default profile of Incus. The VM uses neither. - An Incus agent inside the VM. HAOS does not ship one.
Decisions
Section titled “Decisions”D1. Connection model: a Vault-held key behind scripts/calgary-playbook.sh
Section titled “D1. Connection model: a Vault-held key behind scripts/calgary-playbook.sh”The wrapper copies scripts/heimdall-playbook.sh. It makes sure that the project venv and the Vault session exist.
It reads the private key from secret/fzymgc-house/infrastructure/calgary/automation-ssh, field
private_key, and runs ansible-playbook calgary-playbook.yml with IdentityAgent=none. The
public key goes into ~fzymgc/.ssh/authorized_keys on calgary. The infrastructure-developer
policy already grants secret/fzymgc-house/infrastructure/*, so the new path needs no policy
change.
Alternative: the SSH agent. Rejected for the reason that docs/operations/heimdall.md gives. An
agent with many keys trips the sshd MaxAuthTries limit, and a working agent hides a broken Vault
path.
D2. The fish login shell: the default sh shell plugin
Section titled “D2. The fish login shell: the default sh shell plugin”Ansible sends its commands through the login shell of the remote user, and on calgary that shell is
fish. Ansible wraps every command in /bin/sh -c, so fish parses only one quoted string. The
default sh shell plugin works, and the inventory sets no ansible_shell_type.
An ad-hoc test against calgary on 2026-10-04 measured both plugins with pipelining turned off. With
pipelining on, no module needs a remote temporary directory, so a ping passes with either plugin.
The sh plugin passed with and without become. The fish plugin failed to create the remote
temporary directory, because it writes fish syntax that /bin/sh then runs.
Alternatives:
ansible_shell_type: fish. Rejected: the test above measured it to fail.- A bash login shell for
fzymgc. Rejected, because it changes the interactive shell of Sean on the host to suit a tool.
D3. The network legs: systemd-networkd, and bridges with no address
Section titled “D3. The network legs: systemd-networkd, and bridges with no address”The untagged interface enp1s0 keeps DHCPv4 and router advertisements. After the port change, it
takes the Core reservation. Two VLAN netdevs, enp1s0.3000 and enp1s0.3020, join the bridges
br-main and br-iot. The VLANs and the bridges set DHCP=no, IPv6AcceptRA=no and
LinkLocalAddressing=no, so the host holds no address on Main or IoT. dns-common builds the VLAN
legs of the resolver pair with systemd-networkd in the same way.
Alternatives:
- netplan. Rejected: calgary has no netplan configuration, and
dns-commonuses networkd. - Incus-managed bridges. Rejected: Incus runs dnsmasq and NAT on the bridges it manages.
- macvlan NICs. Rejected: through macvlan, the host cannot reach the VM.
D4. The VM definition: community.general.lxd_container on the Incus socket
Section titled “D4. The VM definition: community.general.lxd_container on the Incus socket”community.general 13.4.0 has no Incus instance module. lxd_container with
url: unix:/var/lib/incus/unix.socket creates and reconciles Incus instances, and a second run
recreates nothing. The role declares the VM with type: virtual-machine and profiles: [], so
each device is on the instance:
| Item | Value |
|---|---|
| Name | haos |
| Configuration | security.secureboot=false, limits.cpu=4, limits.memory=8GiB, boot.autostart=true, snapshots.schedule=@daily, snapshots.expiry=14d |
root |
disk on the pool default, size 64GiB |
eth0 |
nic, nictype=bridged, parent=br-main, fixed hwaddr |
eth1 |
nic, nictype=bridged, parent=br-iot, fixed hwaddr |
thread-radio |
usb, the vendor and product IDs of the ZBT-2, required=false |
The two MACs come from the Incus range 00:16:3e:xx:xx:xx and live in host_vars/calgary.yml. The
dns-control entry for ha names the MAC of eth0 before the VM exists, so the MAC cannot come
from Incus.
Alternatives:
- The Terraform provider
lxc/incus. Rejected: HCP Terraform runs on an agent inside the cluster, and that makes the definition of Home Assistant depend on the cluster. incusCLI tasks. Rejected: each task needs idempotence written by hand.
D5. The HAOS image is a first-boot seed
Section titled “D5. The HAOS image is a first-boot seed”The role downloads the pinned haos_ova-<version>.qcow2.xz from the HAOS GitHub release. It
compares the SHA-256 of the download with the pinned value. It decompresses the image and imports it with a generated metadata.yaml as the split
image alias haos-seed-<version>. The Incus image format documents this form for a VM: a metadata
tarball and a qcow2 disk.
The role runs these tasks only when the instance does not exist. After the first boot, HAOS updates
itself on its A/B slots. The role never compares or replaces the root disk. A new seed version in
host_vars changes nothing on a running VM.
The first pin is HAOS 18.3, the latest stable release when pull request 3 started. Its SHA-256 is
the digest that GitHub reports for the release asset. Incus imported the split image as type
VIRTUAL-MACHINE from the qcow2 alone, so the metadata needs only architecture, creation_date
and descriptive properties.
Alternatives:
incus-migrate. Rejected: the tool is interactive.ddinto the zvol. Rejected: it bypasses the image and storage layers of Incus.
D6. The Thread radio: passthrough by vendor and product ID
Section titled “D6. The Thread radio: passthrough by vendor and product ID”One ZBT-2 attaches to calgary, so its vendor and product IDs select it. The Home Assistant
integration homeassistant_connect_zbt2 matches the radio under vendor 303a with either product
4001 or 831a. The VM has one USB device for each product ID, so the radio stays with the VM
under either ID. The pull request declared both IDs before the radio arrived. Task 3.1 compares them
with the radio. required=false lets the VM start without the radio. If a second identical radio is attached later, the device must also
match on serial. The Incus documentation lists that option, but nobody tested it on 6.0.0. The
onboard MediaTek Bluetooth stays with the host and is not used.
On 2026-10-10 the radio arrived and showed 303a:831a, which matches one of the two devices. Incus
6.0.0 did not attach it to the running VM. The host kernel bound it to cdc_acm and logged no
Incus event. A restart of the VM attached it, and the host driver changed to usbfs. So a radio
that is plugged in while the VM runs needs one restart of the VM.
D7. The host baseline
Section titled “D7. The host baseline”- Install
unattended-upgradeswith the security and updates origins andUnattended-Upgrade::Automatic-Reboot "true"at04:00. The VM starts again by itself. Lighting does not depend on Home Assistant, so the host needs no gated reboot like heimdall has. - Purge the libvirt packages.
- Delete
test-vm. - Assert that the storage pool
defaultexists onrpool/incus. The role creates and changes no pool.
D8. Addresses and names
Section titled “D8. Addresses and names”Pull request 2 allocates the two addresses in docs/reference/network.md before the dns-control
entries exist:
| Name | Address | Leg | MAC |
|---|---|---|---|
calgary |
192.168.40.21 |
Core | 78:55:36:01:3b:b8 (enp1s0) |
ha |
192.168.20.220 |
Main | the eth0 MAC from host_vars/calgary.yml |
During pull request 1, on 2026-10-04, the operator created the curated entry calgary on Main at
192.168.22.35. calgary already held that address as a dynamic lease, so the host did not change.
The address is inside the Main pool, so the entry sets ptr to false, as every in-pool entry does.
Kea D2 owns the PTR of a pool address. The reservation keeps the Ansible target fixed until pull
request 2.
Pull request 2 changes the address of calgary to 192.168.40.21 and sets ptr. It creates the
entry ha with ptr set. The operator makes both changes through a plan and an apply. The IoT leg
of the VM takes a lease from the pool. Kea D2 publishes
homeassistant.iot.lan.fzymgc.house from the hostname that HAOS sends.
The same pull request adds the 2.4 GHz table to docs/reference/network.md. The table states that
the UniFi controller chooses the Wi-Fi channels from 1, 6 and 11, and that ap-office serves no
2.4 GHz. It holds one row each for the Thread, Hue and ZHA channels. Sean fills a row when he reads
or sets the value.
D9. Switch ports
Section titled “D9. Switch ports”| Port | Switch | Native | Tagged | Other keys |
|---|---|---|---|---|
| calgary | switch-main port 5, in the closet |
Core | Main and IoT: tagged_vlan_mgmt = "custom", excluded_networkconf_ids holds wan2, guest, lab, mainwireless and telework |
op_mode = "switch", poe_mode = "off" |
| Hue Bridge Pro | switch-kitchen port 3 |
Main | none: tagged_vlan_mgmt = "block_all" |
Declared by pull request #2722 before pull request 2 of this change |
| SLZB-06MG24 | switch-kitchen port 1 |
IoT | none: tagged_vlan_mgmt = "block_all" |
op_mode = "switch". Declared in its own pull request |
On 2026-10-05, port 5 of switch-main carried the stale name apc-networkcloset. Its link was
down, no client used it, and it was already Core native with every VLAN tagged. Before the move,
calgary was on switch-office port 5, a Main native trunk. That port keeps its generic entry.
The calgary port has the shape of the heimdall port, switch_main port 11. go-unifi
DevicePortOverrides models every key in this table, and the heimdall and abode-gateway ports
already use each one. unifi_device_ports writes only the declared keys into the raw
port_overrides entry, so keys that the provider does not model survive (rule qa8kh6gtea).
D10. Telemetry
Section titled “D10. Telemetry”A new role, calgary-otel-collector, copies the shape of dns-otel-collector. It installs the same
pinned otelcol-contrib release. The control node issues the client certificate from Vault PKI. The
exporter sends to otel-gateway.fzymgc.house:443 with the ClickStack ingest token. The receivers
are journald for the whole journal and hostmetrics. The resource processor sets
host.name=calgary.
tf/vault gains pki-otel-collector-calgary.tf, with the role otel-collector-calgary-client, the
common name otelcol-calgary.fzymgc.house and a TTL of 90 days. The role issues a new certificate
when fewer than 30 days remain, so a monthly converge keeps it valid. A 30-day certificate with the
7-day window of dns-otel-collector needs a converge inside the last 7 days of each certificate,
which a monthly cadence misses. It also gains
policy-otel-collector-calgary.tf. The pair mirrors pki-otel-collector-dns.tf and
policy-otel-collector-dns.tf.
Alternative: one generic collector role for all hosts. Rejected for this change, because the refactor changes the live telemetry of the resolver pair.
D11. Backups on the NAS
Section titled “D11. Backups on the NAS”A new role, nas-ha-backups, creates the dataset main/home-assistant-backups and an NFS export of
it through midclt, as nas-forgejo and nas-rustfs create their datasets. The export allows
192.168.20.220 only and maps every client user to nobody, which owns the dataset with mode
0770. The dataset joins nas_kopia_leaves in group_vars/all.yml and nas_sandbox_mounts in
host_vars/nas.yml, because Kopia runs in nas-support and reads each leaf through a bind mount.
A new leaf takes four steps: the converge of nas-ha-backups and nas-sandbox-instance, a restart
of nas-support, nas-kopia-acls, and nas-kopia-backup on nas-support. Without the last step,
the nightly backup skips the leaf and still reports success. Inside Home Assistant, Sean adds the
share as network storage and sets the backup schedule. That is Home Assistant content.
On 2026-10-05 the NFS service was enabled and running, with no export, NFSv4 only, bound to
192.168.20.200. The role adds the first export and changes no service state.
Alternative: an SMB share. Rejected: SMB needs a password stored in Home Assistant content. NFS needs none, because the export allows one address.
D12. The monitor
Section titled “D12. The monitor”tf/uptime-kuma gains uptimekuma_monitor_http.home_assistant on
https://ha.fzymgc.house/manifest.json, which answers 200 without a login. The monitor also alerts
before the certificate of D14 expires. Since Home Assistant
2026.8, a new HAOS install serves on port 80. Port 8123 only redirects to port 80 until onboarding
ends, and then it stops answering. D14 then moves Home Assistant to HTTPS on port 443, so every
check of Home Assistant uses https://ha.fzymgc.house/. It notifies
uptimekuma_notification_pushover.cluster_edge, like the monitors in tf/uptime-kuma/main.tf.
D13. The address change: a cable move
Section titled “D13. The address change: a cable move”switch-main port 5 was already Core native. So calgary changes its address when its cable moves
from switch-office port 5, a Main native port, to switch-main port 5. When the link comes back,
systemd-networkd starts a new DHCP exchange on Core. If the entry calgary already holds
192.168.40.21, Kea answers with that reservation. If calgary takes a pool address on Core first,
change the entry and run sudo networkctl reconfigure enp1s0 on calgary. The reconfigure drops the
lease and starts a new DHCP exchange, and Kea answers with the reservation. A networkctl renew is
not enough. On 2026-10-05 Kea extended the pool lease that the renew asked for.
The Terraform apply after the merge changes no native VLAN. It names the port, limits the tagged VLANs to Main and IoT, and turns PoE off.
The plan first held a loop on the host that renewed the lease every minute. The cable move makes the
loop unnecessary, because the new link starts the DHCP exchange. The command of the loop was also
wrong. ssh joins its arguments without their quotes, so the fish login shell split the loop at its
first ;.
D14. HTTPS with a Let’s Encrypt certificate
Section titled “D14. HTTPS with a Let’s Encrypt certificate”Home Assistant serves HTTPS on port 443 with a Let’s Encrypt certificate for ha.fzymgc.house, as
the UniFi console does for unifi.fzymgc.house. Every client verifies it with its system trust
store, so no phone, app or browser needs the internal CA.
The certificate uses a DNS-01 challenge through Cloudflare, which holds the public zone
fzymgc.house. tf/cloudflare/ha_acme.tf creates the token ha-acme-dns01 with DNS Write and Zone
Read on that zone, and writes it to Vault at secret/fzymgc-house/infrastructure/home-assistant/acme.
The Vault grant for the cloudflare workspace applies first, in its own pull request. Inside Home
Assistant, the Let’s Encrypt app issues the certificate into /ssl, and the web server settings
set port 443 and the two files. The app does not renew by itself. It runs once and exits, and it
renews only when started within 30 days of expiry. Home Assistant also keeps the certificate that it
loaded at start. So two automations renew it: one starts the app every day, and one restarts Home
Assistant Core when the certificate that it serves has fewer than 25 days left. The URLs of Home
Assistant are https://ha.fzymgc.house for the local network and Home Assistant Cloud for the
Internet. Sean decided on 2026-10-05 not to publish ha.fzymgc.net through the Cloudflare tunnel.
All of this is Home Assistant content.
The token can change any record of the public zone fzymgc.house, because a Cloudflare token
cannot be limited to one name. Home Assistant holds it, so the Home Assistant backups hold it too.
Alternatives:
- The internal Vault CA. Rejected: every phone, the Companion app and every browser then needs the internal CA.
- A reverse proxy in the cluster. Rejected: Home Assistant must not depend on the cluster.
- A CNAME delegation of
_acme-challenge.hato a separate zone. Not chosen: it limits the token to one name, but it adds a zone for one host.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [The cable move strands calgary] → The new link starts a DHCP exchange on Core (D13). If the host
does not answer on Core within 10 minutes, the rollback of pull request 2 applies. calgary stands
in the closet beside
switch-main, so a console is a walk away. - [A later Ansible release stops wrapping commands in
/bin/sh -c] → The fish login shell then parses the commands of Ansible. The fallback is a dedicated automation user with a bash shell, likenas-automationon the NAS. - [HAOS has no Incus agent] →
incus execand the address column ofincus listdo not work for the VM. The acceptance reads DNS and HTTP instead, and the runbook says so. - [Snapshots are crash-consistent] → Without an agent, Incus cannot quiesce HAOS. The Home Assistant backups on the NAS are the consistent copy. The snapshots roll back the whole VM after a bad update.
- [The collector certificate expires after 90 days] → The role issues a new certificate at converge time when fewer than 30 days remain. The runbook names a monthly converge. Without it, telemetry stops and Home Assistant keeps running.
- [The Let’s Encrypt certificate is not renewed] → The two automations of D14 renew it. The Uptime Kuma monitor of D12 alerts before it expires.
- [The VM depends on the radio] →
required=falseon the USB device. - [Remote access depends on the cluster] → The tailnet routes for Main and Core come from subnet routers on two k3s workers and on the NAS. Only remote access depends on them. Local control and Apple Home do not.
- [The NFS service is NAS-wide] → Before the converge, the role reads
service.queryandsharing.nfs.query, and the pull request body records both readings.
Migration Plan
Section titled “Migration Plan”- Host baseline. Run
scripts/calgary-playbook.sh --check --diff, then without--check. Rollback: revert the merge commit through a pull request. If libvirt is needed again,sudo apt-get install libvirt-daemon-systemon calgary restores it. The deletion oftest-vmis final, and the VM holds nothing. - Network. Apply the change of the entry
calgaryand the new entryha, and move the cable of calgary toswitch-mainport 5 (D13). Make sure thatssh fzymgc@192.168.40.21 trueexits 0. Then merge the allocation indocs/reference/network.mdwith thetf/unifichange, and a person confirms themain-cluster-unifirun. Rollback: change the entrycalgaryback to192.168.22.35withptrfalse, and move the cable back toswitch-officeport 5. If the merge happened, revert it through a pull request. - The VM. Run the playbook. Rollback:
ssh fzymgc@calgary.fzymgc.house incus delete --force haos. Home Assistant holds no content yet. Then delete the networkd files of D3 that the runbook lists, and runsudo networkctl reloadon calgary. systemd-networkd does not delete a link whose file goes away, so also delete the two bridges and the two VLAN links withsudo ip link delete. - Telemetry and monitor. HCP Terraform applies
tf/vaultandtf/uptime-kumaon merge, then run the playbook. Rollback:ssh fzymgc@calgary.fzymgc.house sudo systemctl disable --now otelcol-calgary, and revert the merge commit through a pull request. - Backups. Run
scripts/nas-playbook.sh --tags nas-ha-backups. Rollback:scripts/nas-adhoc.sh nas -m ansible.builtin.command -a 'midclt call sharing.nfs.delete <id>'with the export id fromsharing.nfs.query. The dataset stays until Sean deletes it.
Open Questions
Section titled “Open Questions”- Whether the ZBT-2 shows one of the two product IDs that design D6 declares. Task 3.1 reads them from the radio when it arrives.