Skip to content

Design: calgary as the Home Assistant host

See openspec/changes/calgary-home-assistant-host/proposal.md for the motivation and for the decisions Sean made on 2026-10-04.

An SSH reading of calgary on 2026-10-04 found this state:

  • Ubuntu 24.04.4 with kernel 6.8 and a ZFS root on rpool.
  • Incus 6.0.0 from the Ubuntu archive. Its storage pool default is on rpool/incus, and its NAT bridge is incusbr0. One leftover VM, test-vm, runs.
  • libvirt 10.0.0 and QEMU 8.2.2 are installed too. A purge simulation of the libvirt packages removes only libvirt*, not Incus and not QEMU.
  • systemd-networkd runs the network from one file, /etc/systemd/network/10-enp1s0.network, with DHCP=ipv4 and IPv6AcceptRA=yes. The host has no netplan configuration.
  • unattended-upgrades is not installed.
  • The user fzymgc has passwordless sudo, and its login shell is fish. The host has no rg.
  • The MAC of enp1s0 is 78:55:36:01:3b:b8.

The estate sets these constraints:

  • docs/reference/network.md is the addressing authority. On 2026-10-04 dns-control held .200, .201, .202, .204 and .210 in 192.168.20.0/24, and .20 and .25 in 192.168.40.0/24. The Kea pools start at 192.168.22.20 on Main and at 192.168.41.10 on Core.
  • dns-control is the only writer of curated names and DHCP reservations (curated-entry-store, kea-reservation-fragment). tf/dns-records no longer exists.
  • tf/unifi declares switch ports through unifi_device_ports. The resource patches the port_overrides array of the device record as raw JSON (docs/engineering/specs/2026-09-19-unifi-device-ports-design.md, decision D7).
  • The tier table in docs/operations/unifi.md keeps the radios of the access points and the channel plan controller-owned.
  • A tf/unifi pull request merges only when its main-cluster-unifi plan changes nothing that the pull request does not change. The workspace applies only after a person confirms the run (“Adopt a UI change” in docs/operations/unifi.md).
  • heimdall, the resolver pair and the NAS each run Ansible through a wrapper in scripts/. The wrapper reads an SSH key from Vault and uses no SSH agent.

Goals:

  • One playbook converges calgary from a workstation, and a second run in check mode changes nothing.
  • Ansible owns the definition of the VM. HAOS owns everything inside the VM.
  • The live network change happens once, and its rollback works when the host is unreachable.

Non-Goals:

  • A shared OpenTelemetry Collector role for all hosts. dns-otel-collector stays as it is.
  • Removing incusbr0 or the default profile of Incus. The VM uses neither.
  • An Incus agent inside the VM. HAOS does not ship one.

D1. Connection model: a Vault-held key behind scripts/calgary-playbook.sh

Section titled “D1. Connection model: a Vault-held key behind scripts/calgary-playbook.sh”

The wrapper copies scripts/heimdall-playbook.sh. It makes sure that the project venv and the Vault session exist. It reads the private key from secret/fzymgc-house/infrastructure/calgary/automation-ssh, field private_key, and runs ansible-playbook calgary-playbook.yml with IdentityAgent=none. The public key goes into ~fzymgc/.ssh/authorized_keys on calgary. The infrastructure-developer policy already grants secret/fzymgc-house/infrastructure/*, so the new path needs no policy change.

Alternative: the SSH agent. Rejected for the reason that docs/operations/heimdall.md gives. An agent with many keys trips the sshd MaxAuthTries limit, and a working agent hides a broken Vault path.

D2. The fish login shell: the default sh shell plugin

Section titled “D2. The fish login shell: the default sh shell plugin”

Ansible sends its commands through the login shell of the remote user, and on calgary that shell is fish. Ansible wraps every command in /bin/sh -c, so fish parses only one quoted string. The default sh shell plugin works, and the inventory sets no ansible_shell_type.

An ad-hoc test against calgary on 2026-10-04 measured both plugins with pipelining turned off. With pipelining on, no module needs a remote temporary directory, so a ping passes with either plugin. The sh plugin passed with and without become. The fish plugin failed to create the remote temporary directory, because it writes fish syntax that /bin/sh then runs.

Alternatives:

  • ansible_shell_type: fish. Rejected: the test above measured it to fail.
  • A bash login shell for fzymgc. Rejected, because it changes the interactive shell of Sean on the host to suit a tool.

D3. The network legs: systemd-networkd, and bridges with no address

Section titled “D3. The network legs: systemd-networkd, and bridges with no address”

The untagged interface enp1s0 keeps DHCPv4 and router advertisements. After the port change, it takes the Core reservation. Two VLAN netdevs, enp1s0.3000 and enp1s0.3020, join the bridges br-main and br-iot. The VLANs and the bridges set DHCP=no, IPv6AcceptRA=no and LinkLocalAddressing=no, so the host holds no address on Main or IoT. dns-common builds the VLAN legs of the resolver pair with systemd-networkd in the same way.

Alternatives:

  • netplan. Rejected: calgary has no netplan configuration, and dns-common uses networkd.
  • Incus-managed bridges. Rejected: Incus runs dnsmasq and NAT on the bridges it manages.
  • macvlan NICs. Rejected: through macvlan, the host cannot reach the VM.

D4. The VM definition: community.general.lxd_container on the Incus socket

Section titled “D4. The VM definition: community.general.lxd_container on the Incus socket”

community.general 13.4.0 has no Incus instance module. lxd_container with url: unix:/var/lib/incus/unix.socket creates and reconciles Incus instances, and a second run recreates nothing. The role declares the VM with type: virtual-machine and profiles: [], so each device is on the instance:

Item Value
Name haos
Configuration security.secureboot=false, limits.cpu=4, limits.memory=8GiB, boot.autostart=true, snapshots.schedule=@daily, snapshots.expiry=14d
root disk on the pool default, size 64GiB
eth0 nic, nictype=bridged, parent=br-main, fixed hwaddr
eth1 nic, nictype=bridged, parent=br-iot, fixed hwaddr
thread-radio usb, the vendor and product IDs of the ZBT-2, required=false

The two MACs come from the Incus range 00:16:3e:xx:xx:xx and live in host_vars/calgary.yml. The dns-control entry for ha names the MAC of eth0 before the VM exists, so the MAC cannot come from Incus.

Alternatives:

  • The Terraform provider lxc/incus. Rejected: HCP Terraform runs on an agent inside the cluster, and that makes the definition of Home Assistant depend on the cluster.
  • incus CLI tasks. Rejected: each task needs idempotence written by hand.

The role downloads the pinned haos_ova-<version>.qcow2.xz from the HAOS GitHub release. It compares the SHA-256 of the download with the pinned value. It decompresses the image and imports it with a generated metadata.yaml as the split image alias haos-seed-<version>. The Incus image format documents this form for a VM: a metadata tarball and a qcow2 disk.

The role runs these tasks only when the instance does not exist. After the first boot, HAOS updates itself on its A/B slots. The role never compares or replaces the root disk. A new seed version in host_vars changes nothing on a running VM.

The first pin is HAOS 18.3, the latest stable release when pull request 3 started. Its SHA-256 is the digest that GitHub reports for the release asset. Incus imported the split image as type VIRTUAL-MACHINE from the qcow2 alone, so the metadata needs only architecture, creation_date and descriptive properties.

Alternatives:

  • incus-migrate. Rejected: the tool is interactive.
  • dd into the zvol. Rejected: it bypasses the image and storage layers of Incus.

D6. The Thread radio: passthrough by vendor and product ID

Section titled “D6. The Thread radio: passthrough by vendor and product ID”

One ZBT-2 attaches to calgary, so its vendor and product IDs select it. The Home Assistant integration homeassistant_connect_zbt2 matches the radio under vendor 303a with either product 4001 or 831a. The VM has one USB device for each product ID, so the radio stays with the VM under either ID. The pull request declared both IDs before the radio arrived. Task 3.1 compares them with the radio. required=false lets the VM start without the radio. If a second identical radio is attached later, the device must also match on serial. The Incus documentation lists that option, but nobody tested it on 6.0.0. The onboard MediaTek Bluetooth stays with the host and is not used.

On 2026-10-10 the radio arrived and showed 303a:831a, which matches one of the two devices. Incus 6.0.0 did not attach it to the running VM. The host kernel bound it to cdc_acm and logged no Incus event. A restart of the VM attached it, and the host driver changed to usbfs. So a radio that is plugged in while the VM runs needs one restart of the VM.

  • Install unattended-upgrades with the security and updates origins and Unattended-Upgrade::Automatic-Reboot "true" at 04:00. The VM starts again by itself. Lighting does not depend on Home Assistant, so the host needs no gated reboot like heimdall has.
  • Purge the libvirt packages.
  • Delete test-vm.
  • Assert that the storage pool default exists on rpool/incus. The role creates and changes no pool.

Pull request 2 allocates the two addresses in docs/reference/network.md before the dns-control entries exist:

Name Address Leg MAC
calgary 192.168.40.21 Core 78:55:36:01:3b:b8 (enp1s0)
ha 192.168.20.220 Main the eth0 MAC from host_vars/calgary.yml

During pull request 1, on 2026-10-04, the operator created the curated entry calgary on Main at 192.168.22.35. calgary already held that address as a dynamic lease, so the host did not change. The address is inside the Main pool, so the entry sets ptr to false, as every in-pool entry does. Kea D2 owns the PTR of a pool address. The reservation keeps the Ansible target fixed until pull request 2.

Pull request 2 changes the address of calgary to 192.168.40.21 and sets ptr. It creates the entry ha with ptr set. The operator makes both changes through a plan and an apply. The IoT leg of the VM takes a lease from the pool. Kea D2 publishes homeassistant.iot.lan.fzymgc.house from the hostname that HAOS sends.

The same pull request adds the 2.4 GHz table to docs/reference/network.md. The table states that the UniFi controller chooses the Wi-Fi channels from 1, 6 and 11, and that ap-office serves no 2.4 GHz. It holds one row each for the Thread, Hue and ZHA channels. Sean fills a row when he reads or sets the value.

Port Switch Native Tagged Other keys
calgary switch-main port 5, in the closet Core Main and IoT: tagged_vlan_mgmt = "custom", excluded_networkconf_ids holds wan2, guest, lab, mainwireless and telework op_mode = "switch", poe_mode = "off"
Hue Bridge Pro switch-kitchen port 3 Main none: tagged_vlan_mgmt = "block_all" Declared by pull request #2722 before pull request 2 of this change
SLZB-06MG24 switch-kitchen port 1 IoT none: tagged_vlan_mgmt = "block_all" op_mode = "switch". Declared in its own pull request

On 2026-10-05, port 5 of switch-main carried the stale name apc-networkcloset. Its link was down, no client used it, and it was already Core native with every VLAN tagged. Before the move, calgary was on switch-office port 5, a Main native trunk. That port keeps its generic entry.

The calgary port has the shape of the heimdall port, switch_main port 11. go-unifi DevicePortOverrides models every key in this table, and the heimdall and abode-gateway ports already use each one. unifi_device_ports writes only the declared keys into the raw port_overrides entry, so keys that the provider does not model survive (rule qa8kh6gtea).

A new role, calgary-otel-collector, copies the shape of dns-otel-collector. It installs the same pinned otelcol-contrib release. The control node issues the client certificate from Vault PKI. The exporter sends to otel-gateway.fzymgc.house:443 with the ClickStack ingest token. The receivers are journald for the whole journal and hostmetrics. The resource processor sets host.name=calgary.

tf/vault gains pki-otel-collector-calgary.tf, with the role otel-collector-calgary-client, the common name otelcol-calgary.fzymgc.house and a TTL of 90 days. The role issues a new certificate when fewer than 30 days remain, so a monthly converge keeps it valid. A 30-day certificate with the 7-day window of dns-otel-collector needs a converge inside the last 7 days of each certificate, which a monthly cadence misses. It also gains policy-otel-collector-calgary.tf. The pair mirrors pki-otel-collector-dns.tf and policy-otel-collector-dns.tf.

Alternative: one generic collector role for all hosts. Rejected for this change, because the refactor changes the live telemetry of the resolver pair.

A new role, nas-ha-backups, creates the dataset main/home-assistant-backups and an NFS export of it through midclt, as nas-forgejo and nas-rustfs create their datasets. The export allows 192.168.20.220 only and maps every client user to nobody, which owns the dataset with mode 0770. The dataset joins nas_kopia_leaves in group_vars/all.yml and nas_sandbox_mounts in host_vars/nas.yml, because Kopia runs in nas-support and reads each leaf through a bind mount. A new leaf takes four steps: the converge of nas-ha-backups and nas-sandbox-instance, a restart of nas-support, nas-kopia-acls, and nas-kopia-backup on nas-support. Without the last step, the nightly backup skips the leaf and still reports success. Inside Home Assistant, Sean adds the share as network storage and sets the backup schedule. That is Home Assistant content.

On 2026-10-05 the NFS service was enabled and running, with no export, NFSv4 only, bound to 192.168.20.200. The role adds the first export and changes no service state.

Alternative: an SMB share. Rejected: SMB needs a password stored in Home Assistant content. NFS needs none, because the export allows one address.

tf/uptime-kuma gains uptimekuma_monitor_http.home_assistant on https://ha.fzymgc.house/manifest.json, which answers 200 without a login. The monitor also alerts before the certificate of D14 expires. Since Home Assistant 2026.8, a new HAOS install serves on port 80. Port 8123 only redirects to port 80 until onboarding ends, and then it stops answering. D14 then moves Home Assistant to HTTPS on port 443, so every check of Home Assistant uses https://ha.fzymgc.house/. It notifies uptimekuma_notification_pushover.cluster_edge, like the monitors in tf/uptime-kuma/main.tf.

switch-main port 5 was already Core native. So calgary changes its address when its cable moves from switch-office port 5, a Main native port, to switch-main port 5. When the link comes back, systemd-networkd starts a new DHCP exchange on Core. If the entry calgary already holds 192.168.40.21, Kea answers with that reservation. If calgary takes a pool address on Core first, change the entry and run sudo networkctl reconfigure enp1s0 on calgary. The reconfigure drops the lease and starts a new DHCP exchange, and Kea answers with the reservation. A networkctl renew is not enough. On 2026-10-05 Kea extended the pool lease that the renew asked for.

The Terraform apply after the merge changes no native VLAN. It names the port, limits the tagged VLANs to Main and IoT, and turns PoE off.

The plan first held a loop on the host that renewed the lease every minute. The cable move makes the loop unnecessary, because the new link starts the DHCP exchange. The command of the loop was also wrong. ssh joins its arguments without their quotes, so the fish login shell split the loop at its first ;.

D14. HTTPS with a Let’s Encrypt certificate

Section titled “D14. HTTPS with a Let’s Encrypt certificate”

Home Assistant serves HTTPS on port 443 with a Let’s Encrypt certificate for ha.fzymgc.house, as the UniFi console does for unifi.fzymgc.house. Every client verifies it with its system trust store, so no phone, app or browser needs the internal CA.

The certificate uses a DNS-01 challenge through Cloudflare, which holds the public zone fzymgc.house. tf/cloudflare/ha_acme.tf creates the token ha-acme-dns01 with DNS Write and Zone Read on that zone, and writes it to Vault at secret/fzymgc-house/infrastructure/home-assistant/acme. The Vault grant for the cloudflare workspace applies first, in its own pull request. Inside Home Assistant, the Let’s Encrypt app issues the certificate into /ssl, and the web server settings set port 443 and the two files. The app does not renew by itself. It runs once and exits, and it renews only when started within 30 days of expiry. Home Assistant also keeps the certificate that it loaded at start. So two automations renew it: one starts the app every day, and one restarts Home Assistant Core when the certificate that it serves has fewer than 25 days left. The URLs of Home Assistant are https://ha.fzymgc.house for the local network and Home Assistant Cloud for the Internet. Sean decided on 2026-10-05 not to publish ha.fzymgc.net through the Cloudflare tunnel. All of this is Home Assistant content.

The token can change any record of the public zone fzymgc.house, because a Cloudflare token cannot be limited to one name. Home Assistant holds it, so the Home Assistant backups hold it too.

Alternatives:

  • The internal Vault CA. Rejected: every phone, the Companion app and every browser then needs the internal CA.
  • A reverse proxy in the cluster. Rejected: Home Assistant must not depend on the cluster.
  • A CNAME delegation of _acme-challenge.ha to a separate zone. Not chosen: it limits the token to one name, but it adds a zone for one host.
  • [The cable move strands calgary] → The new link starts a DHCP exchange on Core (D13). If the host does not answer on Core within 10 minutes, the rollback of pull request 2 applies. calgary stands in the closet beside switch-main, so a console is a walk away.
  • [A later Ansible release stops wrapping commands in /bin/sh -c] → The fish login shell then parses the commands of Ansible. The fallback is a dedicated automation user with a bash shell, like nas-automation on the NAS.
  • [HAOS has no Incus agent] → incus exec and the address column of incus list do not work for the VM. The acceptance reads DNS and HTTP instead, and the runbook says so.
  • [Snapshots are crash-consistent] → Without an agent, Incus cannot quiesce HAOS. The Home Assistant backups on the NAS are the consistent copy. The snapshots roll back the whole VM after a bad update.
  • [The collector certificate expires after 90 days] → The role issues a new certificate at converge time when fewer than 30 days remain. The runbook names a monthly converge. Without it, telemetry stops and Home Assistant keeps running.
  • [The Let’s Encrypt certificate is not renewed] → The two automations of D14 renew it. The Uptime Kuma monitor of D12 alerts before it expires.
  • [The VM depends on the radio] → required=false on the USB device.
  • [Remote access depends on the cluster] → The tailnet routes for Main and Core come from subnet routers on two k3s workers and on the NAS. Only remote access depends on them. Local control and Apple Home do not.
  • [The NFS service is NAS-wide] → Before the converge, the role reads service.query and sharing.nfs.query, and the pull request body records both readings.
  1. Host baseline. Run scripts/calgary-playbook.sh --check --diff, then without --check. Rollback: revert the merge commit through a pull request. If libvirt is needed again, sudo apt-get install libvirt-daemon-system on calgary restores it. The deletion of test-vm is final, and the VM holds nothing.
  2. Network. Apply the change of the entry calgary and the new entry ha, and move the cable of calgary to switch-main port 5 (D13). Make sure that ssh fzymgc@192.168.40.21 true exits 0. Then merge the allocation in docs/reference/network.md with the tf/unifi change, and a person confirms the main-cluster-unifi run. Rollback: change the entry calgary back to 192.168.22.35 with ptr false, and move the cable back to switch-office port 5. If the merge happened, revert it through a pull request.
  3. The VM. Run the playbook. Rollback: ssh fzymgc@calgary.fzymgc.house incus delete --force haos. Home Assistant holds no content yet. Then delete the networkd files of D3 that the runbook lists, and run sudo networkctl reload on calgary. systemd-networkd does not delete a link whose file goes away, so also delete the two bridges and the two VLAN links with sudo ip link delete.
  4. Telemetry and monitor. HCP Terraform applies tf/vault and tf/uptime-kuma on merge, then run the playbook. Rollback: ssh fzymgc@calgary.fzymgc.house sudo systemctl disable --now otelcol-calgary, and revert the merge commit through a pull request.
  5. Backups. Run scripts/nas-playbook.sh --tags nas-ha-backups. Rollback: scripts/nas-adhoc.sh nas -m ansible.builtin.command -a 'midclt call sharing.nfs.delete <id>' with the export id from sharing.nfs.query. The dataset stays until Sean deletes it.
  • Whether the ZBT-2 shows one of the two product IDs that design D6 declares. Task 3.1 reads them from the radio when it arrives.