Vault unsealer on two hosts: design
This page is a copy of the OpenSpec change vault-unseal-two-hosts, taken when the first pull
request of the change landed. The change directory under openspec/changes/ stays the working
copy until the change is archived.
Proposal
Section titled “Proposal”The only Vault unsealer runs on the Firewalla, on a fire-once boot hook, and the Firewalla is being retired for a UniFi gateway. If Vault seals while that box is off, ExternalSecrets, cert-manager and every consumer of a Vault secret stop. Sean decided on 2026-09-28 that the unsealer moves to two hosts that are not routers: heimdall and a new container on the NAS.
What Changes
Section titled “What Changes”- A new host-agnostic Ansible role installs the
vault-unsealbinary as a systemd service. It reads the key shares fromansible/secrets/vault-unseal.sops.yamlat converge and writes a mode0600configuration file. The role carries no Firewalla, Docker orpost_main.dlogic. - The role is applied to heimdall (
192.168.40.20, arm64). - A new libvirt-LXC container on the NAS,
vault-unseal, runs a second instance. Thenas-game-lxcrole creates it a second time with role parameters (macvlan onbond0, fixed MAC, bootstrap throughnsenter). Its fixed address comes from a Kea reservation intf/dns-records/hosts.yml. - Each instance holds a full quorum of shares, so either host alone restores a sealed member. The split layout from the tool’s README (two shares per instance, three instances) is a later upgrade when a third host exists. It is not part of this change.
- BREAKING for the router:
ansible/roles/router-vault-unsealand itspost_main.dhook are removed, and a standalone play,ansible/router-vault-unseal-retire.yml, removes the container, its files and the share file from the Firewalla. The removal and the teardown land in one pull request. docs/operations/vault.mddescribes the two hosts and the new proof procedure.
Capabilities
Section titled “Capabilities”New Capabilities
Section titled “New Capabilities”vault-unseal: an out-of-cluster unsealer on two independent hosts restores any sealed Vault member without an operator, with the shares stored only in SOPS and in0600files on those hosts.
Modified Capabilities
Section titled “Modified Capabilities”None. The project has no existing specs.
Impact
Section titled “Impact”- Ansible: new role
vault-unseal,heimdall-playbook.yml, two new plays innas-playbook.yml, a new inventory group, the retire play, and the removal ofrouter-vault-unsealfromrouter-playbook.yml. - tf/dns-records: one host entry with a
macfor the NAS container, which is also its Kea reservation. - Secrets: the SOPS file is unchanged. No share leaves the repository except into the two
0600files. - Docs:
docs/operations/vault.md,docs/operations/heimdall.md,docs/operations/nas.md,docs/operations/router-boot-ordering.md. - Depends on nothing else. This change can ship before the UniFi hardware arrives.
- Acceptance is a shell command: stop every other unsealer, delete one standby Vault pod, and
read
sealed: falsefrom that member’s/v1/sys/healthwithin 60 seconds of the pod reaching Running. A restarted member comes up sealed, which is the failure the unsealer exists for. - Pull requests, in order: (1) the role plus heimdall, (2) the NAS container, (3) the router teardown. Each is one converge and one proof.
Design
Section titled “Design”Context
Section titled “Context”See proposal.md for the motivation. The current unsealer is ansible/roles/router-vault-unseal.
It runs ghcr.io/lrstanley/vault-unseal:0.7.0 under Docker Compose on the Firewalla, with its
configuration in /extdata/vault-unseal/vault-unseal.yaml (mode 0600) and a boot hook,
post_main.d/0200-start-vault-unseal.sh. It reads the three key shares, vault_unseal_tokens,
from ansible/secrets/vault-unseal.sops.yaml at converge. It watches
https://vault-{0,1,2}.fzymgc.house every 15 seconds, backs off to 30 minutes, and verifies TLS
against ansible/files/fzymgc-ca-bundle.pem. The member names resolve only at the resolver pair.
The two new hosts:
- heimdall,
192.168.40.20on Core, a Raspberry Pi 4 on Debian bookworm arm64, converged byansible/heimdall-playbook.ymlthroughscripts/heimdall-playbook.sh. Its only management path is SSH, so a converge that breaks it needs a walk to the device. - A new libvirt-LXC container on the NAS (amd64).
ansible/roles/nas-game-lxccreates theprecipicecontainer today fromnas_game_*values inansible/inventory/host_vars/nas.yml. A container’s fixed address is a Kea reservation. An entry with amacintf/dns-records/hosts.ymlis both the A record and the reservation, and a Kea converge publishes it.
Upstream ships the unsealer as release binaries for linux_arm64 and linux_amd64, with
vault-unseal_checksums.txt. The latest upstream release is v1.0.1.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- Vault recovers from a sealed member with the Firewalla off, and with either new host off.
- The key shares exist only in SOPS and in one
0600file on each host. - One role serves both hosts with no host-specific branches.
Non-Goals:
- A share split across three hosts. That needs a third host.
- Moving to vault-unseal
1.x. This change keeps the version and the configuration grammar that run on the router today. - Vault auto-unseal through a KMS or Transit seal. That replaces the unsealer and is a separate decision.
Decisions
Section titled “Decisions”D1: A release binary under a systemd unit that the role owns
Section titled “D1: A release binary under a systemd unit that the role owns”The role downloads vault-unseal_linux_<arch> for the pinned version with get_url, checks it
against vault-unseal_checksums.txt, and installs it as /usr/local/bin/vault-unseal. The role
writes its own unit, vault-unseal.service, with User=vault-unseal, Restart=always and
Environment=SSL_CERT_FILE=/etc/vault-unseal/fzymgc-ca-bundle.pem. The version sits in the role
defaults with a Renovate comment for github-releases and lrstanley/vault-unseal.
Alternatives: the upstream .deb, and the container image. Rejected. The .deb brings a unit
the role does not own. A container needs Docker on both hosts, and the router outage of
2026-08-10 came from Docker creating a 0755 stub over a bind-mounted configuration file. A
plain unit has no bind mount and no runtime in the start path.
D2: The mode is asserted at the write and checked as the service user
Section titled “D2: The mode is asserted at the write and checked as the service user”The configuration file is /etc/vault-unseal/vault-unseal.yaml, mode 0600, owner
vault-unseal, written with no_log: true. The unsealer refuses a configuration file whose mode
is wider than 0600. So the role checks the file after the write with
sudo -u vault-unseal test -r, not as root. A root check passes on a file the service cannot
read. The directory /etc/vault-unseal is 0750, owner vault-unseal.
D3: The host resolver answers the member names
Section titled “D3: The host resolver answers the member names”Both hosts get their resolvers from Kea, which hands out the resolver pair’s VIP. The unit sets
no resolver of its own. The role checks getent hosts vault-0.fzymgc.house on the host during
the converge, so a host that cannot resolve the names fails the converge, not a later unseal.
D4: Every host holds the full quorum
Section titled “D4: Every host holds the full quorum”Each instance holds all three shares, as Sean decided on 2026-09-28. Either host alone restores a member. The split layout waits for a third host.
D5: The NAS container reuses nas-game-lxc with role parameters
Section titled “D5: The NAS container reuses nas-game-lxc with role parameters”A new play in ansible/nas-playbook.yml applies nas-game-lxc with role parameters:
nas_game_name: vault-unseal, the image of precipice, a new fixed MAC in the 00:16:3e
range, and no mounts. Role parameters outrank the nas_game_* values in host_vars, so
precipice does not change. A new inventory group, nas_vault_unseal_hosts, holds the container
at its reserved address. A second new play applies the vault-unseal role to that group. The
container’s root filesystem lives on the apps pool, which is not in nas_kopia_leaves, so the
shares never reach the B2 backup.
Alternative: a new role for the container. Rejected. nas-game-lxc already holds the create and
bootstrap steps that TrueNAS 26 needs, and a copy is a second place to fix them.
D6: The proof restarts a standby pod
Section titled “D6: The proof restarts a standby pod”A member that restarts comes up sealed. That is the failure the unsealer exists for. The proof
stops every other unsealer, deletes one standby pod with kubectl -n vault delete pod, and reads
sealed from that member’s /v1/sys/health. The runbook already uses pod deletion to restart
Vault, one standby at a time.
Alternative: vault operator seal. Rejected. It needs a token with sudo on sys/seal, and in
an HA cluster it seals the active node, which forces a leader election.
D7: A standalone play retires the router unsealer
Section titled “D7: A standalone play retires the router unsealer”ansible/router-vault-unseal-retire.yml follows ansible/unbound-retire.yml: it stays in the
tree and no playbook wires it in. It stops and removes the container, and deletes the compose
directory, the boot hook and /extdata/vault-unseal, which holds the share file. The same pull
request removes the role and its entries from ansible/router-playbook.yml.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [Two more hosts hold the shares] → Each copy is one
0600file readable only by the service user. Neither host backs the file up off-site. - [heimdall reboots in its patch window] → The NAS instance covers the reboot. The unit is enabled, so heimdall’s instance returns with the boot.
- [The NAS reboots or the container stops] → heimdall covers it. The container is created with
autostart: true. - [The new MAC or address collides with an existing reservation] → The task checks
tf/dns-records/hosts.ymlwithrgfor the MAC and the address before the entry lands. - [A converge on heimdall breaks SSH] → The role touches no SSH, network or firewall setting.
heimdall-commonstill asserts the sshd checksum on every run.
Migration Plan
Section titled “Migration Plan”Pull request 1 adds the role and heimdall. The router instance keeps running. Use the literal address, because four identities answer to the name heimdall. Rollback on heimdall:
ssh fzymgc@192.168.40.20 'sudo systemctl disable --now vault-unseal'Then revert the pull request.
Pull request 2 adds the NAS container. Rollback:
ssh nas-automation@<container address> 'sudo systemctl disable --now vault-unseal'Then revert the pull request. The container stays and holds no running unsealer.
Pull request 3 retires the router instance. Rollback is a revert of the pull request and a converge that puts the container back:
cd ansible && ansible-playbook -i inventory/hosts.yml router-playbook.yml --tags vault-unsealIf every unsealer fails, unseal by hand with the shares from SOPS:
sops -d ansible/secrets/vault-unseal.sops.yamlVAULT_ADDR=https://vault-<n>.fzymgc.house vault operator unsealRun vault operator unseal once for each share until sealed reads false.
Open Questions
Section titled “Open Questions”- Move to vault-unseal
1.xafter this change lands. The upgrade checks the configuration grammar of1.xagainst the template. It does not change the hosts or the specs. - The Main address of the NAS container. Any free address below the Kea pool
(
192.168.22.20) works. Task 2.1 picks it.