Skip to content

Vault unsealer on two hosts: design

This page is a copy of the OpenSpec change vault-unseal-two-hosts, taken when the first pull request of the change landed. The change directory under openspec/changes/ stays the working copy until the change is archived.

The only Vault unsealer runs on the Firewalla, on a fire-once boot hook, and the Firewalla is being retired for a UniFi gateway. If Vault seals while that box is off, ExternalSecrets, cert-manager and every consumer of a Vault secret stop. Sean decided on 2026-09-28 that the unsealer moves to two hosts that are not routers: heimdall and a new container on the NAS.

  • A new host-agnostic Ansible role installs the vault-unseal binary as a systemd service. It reads the key shares from ansible/secrets/vault-unseal.sops.yaml at converge and writes a mode 0600 configuration file. The role carries no Firewalla, Docker or post_main.d logic.
  • The role is applied to heimdall (192.168.40.20, arm64).
  • A new libvirt-LXC container on the NAS, vault-unseal, runs a second instance. The nas-game-lxc role creates it a second time with role parameters (macvlan on bond0, fixed MAC, bootstrap through nsenter). Its fixed address comes from a Kea reservation in tf/dns-records/hosts.yml.
  • Each instance holds a full quorum of shares, so either host alone restores a sealed member. The split layout from the tool’s README (two shares per instance, three instances) is a later upgrade when a third host exists. It is not part of this change.
  • BREAKING for the router: ansible/roles/router-vault-unseal and its post_main.d hook are removed, and a standalone play, ansible/router-vault-unseal-retire.yml, removes the container, its files and the share file from the Firewalla. The removal and the teardown land in one pull request.
  • docs/operations/vault.md describes the two hosts and the new proof procedure.
  • vault-unseal: an out-of-cluster unsealer on two independent hosts restores any sealed Vault member without an operator, with the shares stored only in SOPS and in 0600 files on those hosts.

None. The project has no existing specs.

  • Ansible: new role vault-unseal, heimdall-playbook.yml, two new plays in nas-playbook.yml, a new inventory group, the retire play, and the removal of router-vault-unseal from router-playbook.yml.
  • tf/dns-records: one host entry with a mac for the NAS container, which is also its Kea reservation.
  • Secrets: the SOPS file is unchanged. No share leaves the repository except into the two 0600 files.
  • Docs: docs/operations/vault.md, docs/operations/heimdall.md, docs/operations/nas.md, docs/operations/router-boot-ordering.md.
  • Depends on nothing else. This change can ship before the UniFi hardware arrives.
  • Acceptance is a shell command: stop every other unsealer, delete one standby Vault pod, and read sealed: false from that member’s /v1/sys/health within 60 seconds of the pod reaching Running. A restarted member comes up sealed, which is the failure the unsealer exists for.
  • Pull requests, in order: (1) the role plus heimdall, (2) the NAS container, (3) the router teardown. Each is one converge and one proof.

See proposal.md for the motivation. The current unsealer is ansible/roles/router-vault-unseal. It runs ghcr.io/lrstanley/vault-unseal:0.7.0 under Docker Compose on the Firewalla, with its configuration in /extdata/vault-unseal/vault-unseal.yaml (mode 0600) and a boot hook, post_main.d/0200-start-vault-unseal.sh. It reads the three key shares, vault_unseal_tokens, from ansible/secrets/vault-unseal.sops.yaml at converge. It watches https://vault-{0,1,2}.fzymgc.house every 15 seconds, backs off to 30 minutes, and verifies TLS against ansible/files/fzymgc-ca-bundle.pem. The member names resolve only at the resolver pair.

The two new hosts:

  • heimdall, 192.168.40.20 on Core, a Raspberry Pi 4 on Debian bookworm arm64, converged by ansible/heimdall-playbook.yml through scripts/heimdall-playbook.sh. Its only management path is SSH, so a converge that breaks it needs a walk to the device.
  • A new libvirt-LXC container on the NAS (amd64). ansible/roles/nas-game-lxc creates the precipice container today from nas_game_* values in ansible/inventory/host_vars/nas.yml. A container’s fixed address is a Kea reservation. An entry with a mac in tf/dns-records/hosts.yml is both the A record and the reservation, and a Kea converge publishes it.

Upstream ships the unsealer as release binaries for linux_arm64 and linux_amd64, with vault-unseal_checksums.txt. The latest upstream release is v1.0.1.

Goals:

  • Vault recovers from a sealed member with the Firewalla off, and with either new host off.
  • The key shares exist only in SOPS and in one 0600 file on each host.
  • One role serves both hosts with no host-specific branches.

Non-Goals:

  • A share split across three hosts. That needs a third host.
  • Moving to vault-unseal 1.x. This change keeps the version and the configuration grammar that run on the router today.
  • Vault auto-unseal through a KMS or Transit seal. That replaces the unsealer and is a separate decision.

D1: A release binary under a systemd unit that the role owns

Section titled “D1: A release binary under a systemd unit that the role owns”

The role downloads vault-unseal_linux_<arch> for the pinned version with get_url, checks it against vault-unseal_checksums.txt, and installs it as /usr/local/bin/vault-unseal. The role writes its own unit, vault-unseal.service, with User=vault-unseal, Restart=always and Environment=SSL_CERT_FILE=/etc/vault-unseal/fzymgc-ca-bundle.pem. The version sits in the role defaults with a Renovate comment for github-releases and lrstanley/vault-unseal.

Alternatives: the upstream .deb, and the container image. Rejected. The .deb brings a unit the role does not own. A container needs Docker on both hosts, and the router outage of 2026-08-10 came from Docker creating a 0755 stub over a bind-mounted configuration file. A plain unit has no bind mount and no runtime in the start path.

D2: The mode is asserted at the write and checked as the service user

Section titled “D2: The mode is asserted at the write and checked as the service user”

The configuration file is /etc/vault-unseal/vault-unseal.yaml, mode 0600, owner vault-unseal, written with no_log: true. The unsealer refuses a configuration file whose mode is wider than 0600. So the role checks the file after the write with sudo -u vault-unseal test -r, not as root. A root check passes on a file the service cannot read. The directory /etc/vault-unseal is 0750, owner vault-unseal.

D3: The host resolver answers the member names

Section titled “D3: The host resolver answers the member names”

Both hosts get their resolvers from Kea, which hands out the resolver pair’s VIP. The unit sets no resolver of its own. The role checks getent hosts vault-0.fzymgc.house on the host during the converge, so a host that cannot resolve the names fails the converge, not a later unseal.

Each instance holds all three shares, as Sean decided on 2026-09-28. Either host alone restores a member. The split layout waits for a third host.

D5: The NAS container reuses nas-game-lxc with role parameters

Section titled “D5: The NAS container reuses nas-game-lxc with role parameters”

A new play in ansible/nas-playbook.yml applies nas-game-lxc with role parameters: nas_game_name: vault-unseal, the image of precipice, a new fixed MAC in the 00:16:3e range, and no mounts. Role parameters outrank the nas_game_* values in host_vars, so precipice does not change. A new inventory group, nas_vault_unseal_hosts, holds the container at its reserved address. A second new play applies the vault-unseal role to that group. The container’s root filesystem lives on the apps pool, which is not in nas_kopia_leaves, so the shares never reach the B2 backup.

Alternative: a new role for the container. Rejected. nas-game-lxc already holds the create and bootstrap steps that TrueNAS 26 needs, and a copy is a second place to fix them.

A member that restarts comes up sealed. That is the failure the unsealer exists for. The proof stops every other unsealer, deletes one standby pod with kubectl -n vault delete pod, and reads sealed from that member’s /v1/sys/health. The runbook already uses pod deletion to restart Vault, one standby at a time.

Alternative: vault operator seal. Rejected. It needs a token with sudo on sys/seal, and in an HA cluster it seals the active node, which forces a leader election.

D7: A standalone play retires the router unsealer

Section titled “D7: A standalone play retires the router unsealer”

ansible/router-vault-unseal-retire.yml follows ansible/unbound-retire.yml: it stays in the tree and no playbook wires it in. It stops and removes the container, and deletes the compose directory, the boot hook and /extdata/vault-unseal, which holds the share file. The same pull request removes the role and its entries from ansible/router-playbook.yml.

  • [Two more hosts hold the shares] → Each copy is one 0600 file readable only by the service user. Neither host backs the file up off-site.
  • [heimdall reboots in its patch window] → The NAS instance covers the reboot. The unit is enabled, so heimdall’s instance returns with the boot.
  • [The NAS reboots or the container stops] → heimdall covers it. The container is created with autostart: true.
  • [The new MAC or address collides with an existing reservation] → The task checks tf/dns-records/hosts.yml with rg for the MAC and the address before the entry lands.
  • [A converge on heimdall breaks SSH] → The role touches no SSH, network or firewall setting. heimdall-common still asserts the sshd checksum on every run.

Pull request 1 adds the role and heimdall. The router instance keeps running. Use the literal address, because four identities answer to the name heimdall. Rollback on heimdall:

Terminal window
ssh fzymgc@192.168.40.20 'sudo systemctl disable --now vault-unseal'

Then revert the pull request.

Pull request 2 adds the NAS container. Rollback:

Terminal window
ssh nas-automation@<container address> 'sudo systemctl disable --now vault-unseal'

Then revert the pull request. The container stays and holds no running unsealer.

Pull request 3 retires the router instance. Rollback is a revert of the pull request and a converge that puts the container back:

Terminal window
cd ansible && ansible-playbook -i inventory/hosts.yml router-playbook.yml --tags vault-unseal

If every unsealer fails, unseal by hand with the shares from SOPS:

Terminal window
sops -d ansible/secrets/vault-unseal.sops.yaml
VAULT_ADDR=https://vault-<n>.fzymgc.house vault operator unseal

Run vault operator unseal once for each share until sealed reads false.

  • Move to vault-unseal 1.x after this change lands. The upgrade checks the configuration grammar of 1.x against the template. It does not change the hosts or the specs.
  • The Main address of the NAS container. Any free address below the Kea pool (192.168.22.20) works. Task 2.1 picks it.