heimdall (out-of-band recovery host) Ansible Operations
heimdall (192.168.40.20) is a Raspberry Pi 4 Model B on a USB-flash root running Debian
bookworm. It has no console at hand. If a change leaves it wrong, the fix is a walk to the device.
It is the cluster’s out-of-band recovery host: the way in when the network that carries every
other path is the thing being changed. Everything on this page follows from that fact.
heimdall does not have rg. Every command that runs on the box uses grep, awk or sed. The
repository rule that requires rg governs searches of this repository, not commands on a remote
host that does not ship it.
The baseline role is ansible/roles/heimdall-common, run by ansible/heimdall-playbook.yml
against the heimdall_hosts group. The same playbook applies ansible/roles/vault-unseal (see
The Vault unsealer). Host values live in
ansible/inventory/host_vars/heimdall.yml.
Connection model
Section titled “Connection model”Ansible connects as the existing fzymgc login at 192.168.40.20 over the LAN. fzymgc has
passwordless sudo on the host. The private key lives in Vault at
secret/fzymgc-house/infrastructure/heimdall/automation-ssh, field private_key, and nowhere
else. It is an ed25519 key with the comment heimdall-automation.
The SSH agent is not used. An agent that holds many keys offers them all, and sshd refuses the
connection after its attempt limit before the right key is reached. On a host whose only
management path is SSH, Too many authentication failures is a lockout of the recovery host. Read
the limit with sudo sshd -T | grep -i maxauthtries. The wrappers write the key to a 0600 temp
file and force ssh to offer only it:
IdentitiesOnly=yesoffers no identity other than the one passed with-i.IdentityAgent=noneconsults no agent at all. Without it a working agent can satisfy the connection on its own, and the wrapper reports success while the Vault path is broken.
The repository modifies no sshd configuration on this host. The one change it made is one line
appended to ~fzymgc/.ssh/authorized_keys. The verify block asserts both halves on every run: the
checksum of /etc/ssh/sshd_config equals heimdall_sshd_config_sha256 in host_vars, and
dpkg --verify openssh-server reports no changed file.
Why the tailnet path cannot carry automation
Section titled “Why the tailnet path cannot carry automation”heimdall is Tailscale-tagged tag:bastion and tag:security. The matching SSH rule in
tailscale/policy.hujson is action: "check". When Tailscale SSH handles a connection,
tailscaled terminates it and authenticates by tailnet identity. authorized_keys is never
read, so a Vault-held key offered over heimdall.hound-skate.ts.net does nothing. check mode
also demands interactive re-authentication, which no unattended run can give.
Two error strings tell you which stack answered:
| What you see | Who emitted it | What it means |
|---|---|---|
tailscale: tailnet policy does not permit you to SSH as user "…" |
tailscaled |
you are on the tailnet path. Keys do not matter here |
Permission denied (publickey) |
OpenSSH | you reached the real sshd. Your key was wrong or absent |
The second message is progress. The first means the transport is wrong, and no key fixes it.
Preferred: the wrapper script
Section titled “Preferred: the wrapper script”scripts/heimdall-playbook.sh checks the Vault session, reads the key, writes it to a 0600 temp
file that an EXIT trap removes, and runs ansible-playbook with the connection flags above. Pass
through any ansible-playbook arguments:
# dry-run first — read-only verify block, no changesscripts/heimdall-playbook.sh --tags heimdall-verify --check --diff# the verify run itself (still read-only; a verify run reporting changed>0 is a defect)scripts/heimdall-playbook.sh --tags heimdall-verify# full convergescripts/heimdall-playbook.sh --tags heimdall-commonThe wrapper needs an authenticated Vault session. Run vault login or
scripts/vault-helper.sh login first. vault status alone is not proof of a session, because it
reads the unauthenticated seal-status endpoint and succeeds with an expired token. The wrapper
names vault login in its error when the read fails.
Both wrappers prefix their run with ANSIBLE_CACHE_PLUGIN_TIMEOUT=0. See
Known limits for why.
Ad-hoc commands (one-off diagnostics)
Section titled “Ad-hoc commands (one-off diagnostics)”scripts/heimdall-adhoc.sh uses the same key handling and wraps ansible <pattern> … instead of
ansible-playbook:
# reachability, no agent involvedscripts/heimdall-adhoc.sh heimdall -m ansible.builtin.ping# a read as rootscripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command -a 'ss -lntp'Ansible renders Jinja in ad-hoc module arguments, so do not put a literal {{ }} in the command.
-m shell runs through the box’s /bin/sh. Prefer -m command unless you need a shell. Add -b
when the target needs root. ansible.builtin.shell has no changed_when, so an ad-hoc run reports
CHANGED for a pure read. That word means nothing here.
Verifying the credential (and the negative control that makes it mean something)
Section titled “Verifying the credential (and the negative control that makes it mean something)”Run the positive check and the negative control together, in this order. Without the negative control an ambient SSH agent can satisfy the connection on its own, and the green proves nothing.
# POSITIVE — the Vault-held key alone must authenticate.vault kv get -mount=secret -field=private_key \ fzymgc-house/infrastructure/heimdall/automation-ssh > /tmp/heimdall.key && chmod 600 /tmp/heimdall.keyssh -o IdentitiesOnly=yes -o IdentityAgent=none -o BatchMode=yes \ -i /tmp/heimdall.key fzymgc@192.168.40.20 'echo authenticated-with-vault-key'
# NEGATIVE CONTROL — same invocation, no valid key. This MUST fail.ssh -o IdentitiesOnly=yes -o IdentityAgent=none -o BatchMode=yes \ -o PasswordAuthentication=no -i /nonexistent fzymgc@192.168.40.20 true# expected: Permission denied (publickey).
rm -f /tmp/heimdall.keyIf the negative control succeeds, something other than the Vault key produced the positive result.
Stop and find out what. If either command returns tailscale: tailnet policy does not permit…,
you are on the tailnet path and neither result is about keys.
Manual fallback (if you need to run a different playbook)
Section titled “Manual fallback (if you need to run a different playbook)”vault kv get -mount=secret -field=private_key \ fzymgc-house/infrastructure/heimdall/automation-ssh > /tmp/heimdall.key && chmod 600 /tmp/heimdall.key
cd ansibleansible-playbook -i inventory heimdall-playbook.yml --tags <tag> \ -e ansible_ssh_private_key_file=/tmp/heimdall.key \ -e "ansible_ssh_common_args='-o IdentitiesOnly=yes -o IdentityAgent=none'"The inner single quotes around the -o arguments are required. They keep the arguments together
as one value for Ansible’s k=v parser. Without them Ansible splits on the spaces and fails with
argument -o: expected one argument.
The four heimdalls
Section titled “The four heimdalls”Four distinct things answer to the name heimdall. Treat a bare heimdall in any networking
context as a defect. Fully qualify it.
| Identity | Address | Where it is defined |
|---|---|---|
| The Pi, over the tailnet | heimdall.hound-skate.ts.net |
tailnet MagicDNS. Unusable for automation, see above |
| The same Pi, on the LAN | heimdall.fzymgc.house → 192.168.40.20 |
the heimdall entry in tf/dns-records/hosts.yml, which also carries the MAC that the Kea reservation on Core reads |
| A tailscale ACL alias, not DNS | 192.168.40.20 |
the hosts map in tailscale/policy.hujson |
| An unrelated public machine | heimdall.fzymgc.net |
tf/cloudflare/dns-fzymgc_net.tf |
The Pi has a second interface, wlan0, on the IoT leg at 192.168.218.224. That address has no
record and no reservation.
The ACL alias and the DNS record agree on 192.168.40.20 because both are edited by hand. Nothing
keeps them in step. If the address changes, change the reservation entry, the alias and
ansible_host together.
ansible/inventory/hosts.yml pins ansible_host to the literal address 192.168.40.20, not to a
name. A literal address cannot be retargeted by a search-domain change, a MagicDNS change or an
edit to a Cloudflare zone.
How eth0 gets its address
Section titled “How eth0 gets its address”The Kea reservation for 192.168.40.20 comes from the mac on the heimdall entry in
tf/dns-records/hosts.yml. The Kea role reads that file to render the Core reservations, and Core
is in kea_served_legs in ansible/inventory/group_vars/dns_hosts.yml.
The role does not manage networking on heimdall. The NetworkManager profiles for eth0, wlan0
and the cell bearer are host state with no git representation. Read them on the box:
nmcli device statusnmcli -f IP4 device show eth0 # APPLIED statenmcli con show "Wired connection 1" # STORED profileip -4 route show defaultnmcli con show reports the stored profile and nmcli device show reports the applied state.
They can differ. Read both before you write either one down. The Sixfab core_manager daemon
rewrites the routing table with ip route and outranks NetworkManager, which sets routes only at
activation. A default-route metric chosen with nmcli does not hold against it. Do not reconfigure
the host’s networking to find out what wins.
If Ansible stops reaching the host at 192.168.40.20, use the Sixfab CORE Remote Terminal over the
cell bearer, read the current address with ip -4 -o addr show dev eth0, and then fix the
reservation entry or ansible_host.
Vault path
Section titled “Vault path”| Field | Value |
|---|---|
| Path | secret/fzymgc-house/infrastructure/heimdall/automation-ssh |
| Fields | private_key, public_key |
| Key type | ed25519, comment heimdall-automation |
The path is operator-seeded, not Terraform-declared. No machine identity reads it: the wrappers run
on an operator’s workstation with that operator’s own Vault token, and
tf/vault/policy-infrastructure-developer.hcl already grants read on
secret/data/fzymgc-house/infrastructure/*. The NAS automation key at
…/infrastructure/nas/automation-ssh is managed the same way. If a machine identity ever needs
this path, add a policy at that point.
Seeding and rotation
Section titled “Seeding and rotation”This is an operator action. The private half never lands on the control node except as the wrapper’s temp file.
ssh-keygen -t ed25519 -C heimdall-automation -f ./heimdall-automation -N ''vault kv put -mount=secret fzymgc-house/infrastructure/heimdall/automation-ssh \ private_key=@./heimdall-automation public_key=@./heimdall-automation.pubshred -u ./heimdall-automation ./heimdall-automation.pubThen append the public half to ~fzymgc/.ssh/authorized_keys on heimdall over the tailnet, which
is the only path that authenticates before the key is in place. Append, never rewrite. A botched
authorized_keys on this host is unrecoverable without a walk to the device.
Patching policy
Section titled “Patching policy”Continuous patching on heimdall is the distribution’s mechanism, not Ansible’s: the apt-daily and
apt-daily-upgrade timers plus unattended-upgrades, configured by a drop-in this repository
renders at /etc/apt/apt.conf.d/52unattended-upgrades-heimdall. It sorts at 52 because apt reads
apt.conf.d in lexical order, and a drop-in overrides the distribution’s 50unattended-upgrades
only if it sorts after it. Named 50 or lower it becomes the file being overridden, and that
inversion fails silently because both files parse.
The declared origins cover the Debian archive (bookworm, bookworm-security,
bookworm-updates) and the Raspberry Pi archives. That is all updates, not security only.
Tailscale is not an allowed origin. It is the transport this repository reaches heimdall over, and
an unattended upgrade of it on the recovery host is an outage risk, not a patching win. The two
periodic switches live in /etc/apt/apt.conf.d/20auto-upgrades
(APT::Periodic::Update-Package-Lists "1" and APT::Periodic::Unattended-Upgrade "1"). With
either set to "0" the whole policy is inert while every surface an operator glances at still
looks healthy.
The role does not upgrade packages. It installs a small pinned set, unattended-upgrades,
needrestart and jq, at state: present, plus whatever heimdall_packages in host_vars adds,
and asserts everything else. A role that upgrades does something different on every run, so its
success carries no information. Package currency is the job of unattended-upgrades.
needrestart is pinned to list-only, and must stay that way
Section titled “needrestart is pinned to list-only, and must stay that way”/etc/needrestart/conf.d/50-heimdall.conf sets $nrconf{restart} = 'l';, which detects and
reports and restarts nothing. Installing needrestart also installs
/etc/apt/apt.conf.d/99needrestart, an apt hook that runs after every apt invocation, including
every unattended-upgrades run. The shipped configuration leaves the restart mode unset, so the
behaviour is otherwise decided at runtime by frontend detection. The service needrestart would
most want to restart after a security upgrade is sshd, the only management path into this host.
The pin also protects the reboot gate, which reads needrestart -b -k -r l as its second
trigger. A needrestart that had already restarted things would have destroyed that signal.
How to check patching health
Section titled “How to check patching health”Use the read-only verify block, never a hand-run apt command:
scripts/heimdall-playbook.sh --tags heimdall-verifyIt reports SC#2 in three tiers:
| Tier | Severity | Condition |
|---|---|---|
| 2a | HARD | unattended-upgrades installed, 20auto-upgrades present, and apt’s own parse (apt-config dump APT::Periodic) reports both periodic switches as "1" |
| 2b | WARN | /var/lib/apt/lists is fresher than heimdall_apt_lists_max_age_days (3 days) |
| 2c | WARN | nothing has been pending past one upgrade window |
2a asserts the mechanism, not timer liveness. apt-daily.timer and apt-daily-upgrade.timer
ship with apt itself and are active on a host with no unattended-upgrades installed, so timer
liveness passes on a host with no patching mechanism. The role keeps it in the assert only as an
extra conjunct.
The 3-day freshness threshold sits against the timers’ 1-day period. It allows two whole missed windows before it warns, so the warning means something when it fires.
apt list --upgradable is not expected to be empty
Section titled “apt list --upgradable is not expected to be empty”The verify block does not assert an empty upgradable list. There is always a window between an
archive publish and the next upgrade run, widened by RandomizedDelaySec, and tailscale is
pending whenever upstream ships a release, because the origins policy excludes it on purpose. A
non-empty list is reported at WARN, not asserted. An assert that goes red for a normal condition
trains the operator to ignore it.
Reboot authority
Section titled “Reboot authority”Four guards default to false in ansible/roles/heimdall-common/defaults/main.yml and are
restated at host level in ansible/inventory/host_vars/heimdall.yml:
| Variable | Default | What flipping it grants |
|---|---|---|
heimdall_allow_reboot |
false |
an Ansible-initiated reboot, during a run someone is watching |
heimdall_allow_eeprom_flash |
false |
rpi-eeprom-update -a, a bootloader write. On a Pi 4 the flash needs a reboot to apply, which is why the two are coupled |
heimdall_gated_reboot_enabled |
false |
installs and enables the self-reboot timer, which is permission for the host to reboot itself with nobody watching |
heimdall_allow_networkd_migration |
false |
the NetworkManager to systemd-networkd migration. It belongs to the network stack, not to reboot authority |
They are separate flags because “I am watching this run right now” and “the host can reboot itself unattended” are different consents. One flag would let the second ride in on the first.
The gate, and why it is a script rather than a setting
Section titled “The gate, and why it is a script rather than a setting”unattended-upgrades is configured Automatic-Reboot "false" and has no reboot authority. It
has no pre-reboot veto hook: its reboot decision reads the marker and that setting and then
schedules the shutdown, with no consultable hook between. So the decision is removed from it and
re-implemented where a live check can run.
That home is /usr/local/sbin/heimdall-gated-reboot.sh, run by heimdall-gated-reboot.timer on
heimdall_gated_reboot_schedule (*-*-* 04:15) with RandomizedDelaySec of
heimdall_gated_reboot_jitter (1800 s, so this host does not reboot in lockstep with anything
else on a round hour) and Persistent=true (a missed window replays at next boot). It reboots only
if both gates pass, read at the reboot moment against live state:
- Is a reboot pending?
/run/reboot-requiredexists, or needrestart reportsNEEDRESTART-KSTAof 2 or 3 (a kernel upgrade is owed). - Is the out-of-band rescue path up right now? The bearer interface
heimdall_bearer_interface(usb0) carries a global, non-link-local IPv4 address, andheimdall_bearer_probe_target(1.1.1.1) answers three pings bound to that interface withinheimdall_bearer_probe_timeout_seconds. Registration with the carrier is not reachability. The probe target is off-modem because the modem’s own next hop answers whether or not the carrier path works.
If either gate fails, the script logs the decision and returns, leaving the update staged for the next window. Staging is free: the marker lives on tmpfs and survives until the next boot.
On Debian /run/reboot-required is a policy convention written by individual postinst scripts,
not a guarantee. A library can be upgraded with nothing touching the marker. That is why the gate
reads needrestart as a second trigger and logs which trigger fired. Read the decisions with
journalctl -t heimdall-gated-reboot.
The timer is installed and stays disabled until heimdall_gated_reboot_enabled is true. A host
with no proven second path does not get permission to reboot itself unattended. The gate’s inputs
are asserted shell-safe before the script is rendered, because they are interpolated into a script
that runs as root.
Exercising the gate: no-act mode only
Section titled “Exercising the gate: no-act mode only”The script has a no-act mode, reachable two ways:
--dry-runor-nas an argument, orHEIMDALL_GATED_REBOOT_DRY_RUN=1in the environment.
It evaluates every gate as a real run would, logs the decision it would take, and returns without invoking the reboot command.
Any manual exercise of the gate, by an operator or by an agent, uses no-act mode. A check must never perform the action it is checking. Acting mode belongs to the timer.
Three more guards bound a manual run:
- Every argument other than
--dry-runand-nis refused with exit64before any gate is evaluated, and the journal names the argument. A mistyped flag cannot select acting mode. - A non-default value for any gate input forces no-act mode:
HEIMDALL_GATED_REBOOT_MARKER,HEIMDALL_GATED_REBOOT_BEARER_IFACE,HEIMDALL_GATED_REBOOT_PROBE_TARGETorHEIMDALL_GATED_REBOOT_PROBE_TIMEOUT. OnlyHEIMDALL_GATED_REBOOT_ALLOW_REAL_REBOOT=1lifts that. The script logsexercise override in effect: <input>=<value> (default <default>). The interlock covers those four inputs and not process-environment inputs such asPATH. HEIMDALL_GATED_REBOOT_REBOOT_CMDoverrides the reboot command, so a caller can substitute an inert sentinel and the script cannot reboot the host however its internal guards are arranged.
Two internal exit codes exist. A completed decision (no-reboot-pending, rescue-path-down,
DRY-RUN-WOULD-REBOOT) returns 0.
| Code | Meaning |
|---|---|
64 |
usage error: an unrecognised argument, refused before any gate is evaluated |
78 |
the reboot command is empty or whitespace-only, refused rather than claimed |
Out-of-band rescue path
Section titled “Out-of-band rescue path”The design is a vendor cloud broker (Sixfab CORE) with an on-host agent that dials out and exposes a remote terminal in the vendor dashboard. No port-forward and no inbound rule. It is independent of this network’s tailscale and this network’s DNS, which are the things most likely to be collateral damage in the changes this host exists to survive.
Do not re-run the vendor installer to fix a disconnected agent. It is piped from
install.connect.sixfab.com to a shell, it purges modemmanager, and it runs
git reset --hard HEAD && git pull against an unpinned upstream into /home/sixfab/. Re-running
it is a supply-chain action dressed up as a repair. The modem is no longer owned by ModemManager:
it presents as a plain ECM ethernet device (usb0) whose address comes from NetworkManager’s
internal DHCP client.
If the agent shows Disconnected, in this order:
- Read the agent log first:
sudo tail -n 200 /home/sixfab/.core/logs/cm/agent-log. - Check what exists:
systemctl is-active core_agent core_managerandsystemctl is-enabled core_agent core_manager. - Only if the units exist, restart the existing unit:
sudo systemctl restart core_agent. - Confirm registration in the vendor dashboard, then re-read the log.
The bearer’s SIM sits on a 1 GB per month pool. heimdall_bearer_route_metric (4096) keeps the
bearer up without letting it win route selection. If the bearer ever becomes the default route,
the pool is gone in hours and the rescue path with it. A remote terminal session costs kilobytes.
This configuration is not Ansible-managed. It lives in the vendor dashboard and in on-host files with no git representation.
wlan0 is a second, independent path
Section titled “wlan0 is a second, independent path”wlan0 carries 192.168.218.224/24 on the IoT leg with its own default route through
192.168.218.1. It is reachable from the control node, and it rides neither the modem nor
tailscale. It comes up unattended at boot from the persisted profile
/etc/NetworkManager/system-connections/preconfigured.nmconnection.
It is not a rescue path for a NetworkManager removal. NetworkManager is the sole owner of its WPA
credentials, its supplicant and its DHCP lease. Take NetworkManager down and wlan0 goes down with
eth0. It also rides the house AP infrastructure.
Network stack
Section titled “Network stack”heimdall runs NetworkManager. systemd-networkd is not enabled, and the migration to it is behind
heimdall_allow_networkd_migration: false. The role performs no network change.
The rescue path is carried by NetworkManager. usb0 gets its address, gateway and default route
from NetworkManager’s internal DHCP client, driven by an in-memory profile (Wired connection 2)
that is not on disk. dhcpcd does not cover it: its configuration is an allow-list,
allowinterfaces wwan*, and the modem enumerates as usb0. Remove NetworkManager and the bearer
loses its address and its route.
A migration must therefore carry three interfaces together: eth0, wlan0 (its PSK relocated out
of the NetworkManager keyfile and wpa_supplicant@wlan0 enabled) and usb0 (a DHCP client and a
high-metric default route). Bring the rescue interfaces onto networkd first, in their own change
window, and make sure that they survive a reboot. Then migrate eth0. Doing all three at once puts
every rescue path in flight during the change it is meant to insure.
The Vault unsealer
Section titled “The Vault unsealer”heimdall runs one of the Vault unsealers. The role ansible/roles/vault-unseal installs the
pinned vault-unseal release binary as /usr/local/bin/vault-unseal and runs it under the systemd
unit vault-unseal as the system user vault-unseal. The unit is enabled, so the unsealer
returns with every boot. The unsealer holds all three key shares, so it alone restores a sealed
member. Vault Operations lists the other unsealers.
| Path | Mode | Owner | Content |
|---|---|---|---|
/etc/vault-unseal/ |
0750 |
vault-unseal |
Directory |
/etc/vault-unseal/vault-unseal.yaml |
0600 |
vault-unseal |
Member URLs, intervals and the key shares |
/etc/vault-unseal/fzymgc-ca-bundle.pem |
0644 |
root |
CA bundle, set as SSL_CERT_FILE in the unit |
The role touches no SSH, network or firewall setting. Converge it alone:
scripts/heimdall-playbook.sh --tags vault-unsealThe converge ends with a verify block that makes sure of three facts. The host resolves every
member name, the user vault-unseal reads the configuration file, and the user nobody cannot
read it. Run the verify block alone with --tags vault-unseal-verify. A --check run on a host
without the unit fails at the start task, because check mode does not write the unit file.
Read the state of the unsealer:
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \ -a 'systemctl is-active vault-unseal; journalctl -u vault-unseal --since -15min --no-pager | tail -20'Stop it for a single-host proof, and start it again after the proof:
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command -a 'systemctl stop vault-unseal'scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command -a 'systemctl start vault-unseal'To roll back, disable the unit and revert the pull request that added the role:
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command -a 'systemctl disable --now vault-unseal'Verify
Section titled “Verify”Every step is read-only and safe to repeat.
-
Dry-run the verify block:
scripts/heimdall-playbook.sh --tags heimdall-verify --check --diff. Expect exit 0 withchanged=0 unreachable=0 failed=0and a terminal roll-up that names every criterion with its observed value. Theok=count changes whenever the role gains tasks and is not the thing to read. Two things are defects: a run reportingchanged>0, and a run that stops before the roll-up. -
Run it for real:
scripts/heimdall-playbook.sh --tags heimdall-verify. Same expectation. HARD criteria (SC#1b, SC#1c, SC#2a, SC#3a, SC#5) fail the run. WARN criteria (SC#2b, SC#2c, SC#3b) print aWARNING:line and the run still exits 0. A pending-package count at WARN is normal.SC#3a is the bootloader release-channel pin (
heimdall_eeprom_release_channel), which either matches or does not. SC#3b is bootloader image drift, reported at WARN because flashing is guarded byheimdall_allow_eeprom_flashand a drifted image is a deliberate condition. SC#5 asserts that the modem and the serial UART survived the last boot by reading the USB bus: the Quectel pair2c7c:0125, the FT232 id0403:6001, and a non-zero/dev/ttyUSB*node count.When a HARD criterion is red, the roll-up still prints. Criteria below the failure read
(not reached), and the roll-up carries aVERIFY BLOCK INCOMPLETEline. The run exits non-zero. The roll-up printing is not the run passing. Readfailed=and the exit status. -
Confirm reachability without the agent:
scripts/heimdall-adhoc.sh heimdall -m ansible.builtin.ping. ExpectSUCCESSwith"ping": "pong". Because the wrapper forcesIdentitiesOnly=yesandIdentityAgent=none, this result can only come from the Vault-held key. -
Confirm the reboot timer is still disabled:
Terminal window scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \-a 'systemctl is-enabled heimdall-gated-reboot.timer; systemctl is-active heimdall-gated-reboot.timer'Expect
disabledtheninactive. Anything else means someone granted the host unattended reboot authority.If you script this check, do not pipe it under
set -o pipefail.systemctl is-enabledexits1fordisabledandis-activeexits3forinactive, so the expected answer makes the task report FAILED. Capture the output with|| trueand assert on the anchored literals^disabled$and^inactive$. An unanchoredenabledmatchesdisabled. -
Exercise the gate in no-act mode. The flag is not optional:
Terminal window scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command \-a '/usr/local/sbin/heimdall-gated-reboot.sh --dry-run'On a healthy host with nothing pending, expect
DECISION=SKIP reason=no-reboot-pending. The same decisions land injournalctl -t heimdall-gated-reboot.That first gate short-circuits, so the run above never reaches the later gates. To exercise the rescue-path gate and the no-act guard without touching real reboot-pending state, override the marker path and substitute an inert reboot command:
Terminal window scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \-a 'M=$(mktemp); HEIMDALL_GATED_REBOOT_MARKER=$M \HEIMDALL_GATED_REBOOT_REBOOT_CMD="/usr/bin/touch /run/gsd-would-have-rebooted" \/usr/local/sbin/heimdall-gated-reboot.sh --dry-run; RC=$?; rm -f $M; echo "SCRIPT-RC=$RC"'Expect, in order: the interlock line
exercise override in effect: marker=… (default /run/reboot-required); forcing no-act mode …, thenreboot is pending, trigger=marker:…, thenrescue path is up, detail=bearer interface usb0 carries …, thenDECISION=DRY-RUN-WOULD-REBOOT all gates passed; no-act mode, not invoking the reboot command, andSCRIPT-RC=0. Then make sure that the guard held. The sentinel must be absent and the boot time unchanged:Terminal window scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \-a 'test ! -e /run/gsd-would-have-rebooted && uptime -s'--dry-runand the inertHEIMDALL_GATED_REBOOT_REBOOT_CMDare two independent belts. Never exercise the gate without both.
Known limits
Section titled “Known limits”Each one is a place where something looks wired up and is not.
- The gated-reboot unit has no
OnFailure=notifier. heimdall has no notify unit and no template in this repository renders one. PointingOnFailure=at a unit that does not exist would make the failure path itself fail silently. The journal is the record until a notifier exists. - A permanently closed rescue gate looks the same as a working one on every operator-visible
surface. Both report
DECISION=SKIPand exit 0. A wrongheimdall_bearer_interfacefails closed, so it is safe and silent. The journal is the only detection surface. An alert on consecutiverescue-path-downdecisions does not exist. Skip-Updates-On-Metered-Connections "true"reads NetworkManager’s metered property. It works while heimdall runs NetworkManager. A migration tosystemd-networkdmakes it inert, because networkd exposes no such property, and the SIM’s 1 GB pool then depends on the bearer staying off the default route.heimdall-gated-reboot.servicehasAfter=network-online.target, and that does not establish that the bearer is settled.usb0’s profile is created in memory after the device appears, andNetworkManager-wait-onlinewaits only on the profiles it knows at startup. The unit can run before the bearer has an address and read the rescue path as down. The gate then skips, which is the safe direction, and the timer fires again the next day. Do not read a singlerescue-path-downat boot as a modem fault.- Two daemons write
/etc/resolv.conf.core_managerappends to it, and tailscale rewrites it with its own banner. While tailscale keeps rewriting, the file stays clean. If tailscale stops, the file grows without bound. Read the writers withsudo journalctl -u core_manager | grep resolv. - The reboot marker path is overridable through
HEIMDALL_GATED_REBOOT_MARKER. That is what makes the gate exercisable without touching real state, and it means a root caller can open the first gate on purpose. The service unit carries noEnvironment=orEnvironmentFile=directive, so the scheduled path always reads the real/run/reboot-required. On the supervised path, any non-default gate input forces no-act mode unlessHEIMDALL_GATED_REBOOT_ALLOW_REAL_REBOOT=1is also set. - Cross-run detection depends on an Ansible fact cache that is control-node-local and gitignored.
Two verify readings, the boot epoch behind “no reboot since the last recorded reading” and the
unattended-upgrades effective-origins outcome behind “unverified twice running”, are
cacheablefacts. They compare this run against the previous run from this checkout on this machine.ansible/ansible.cfgsetsfact_caching=jsonfilewith a relative, gitignoredfact_caching_connection, so another operator, another clone or a CI runner starts empty and reports no prior reading. Treat a first-run “no prior reading” as absence of evidence. The cache entries expire after 86400 seconds by default, which is shorter than this host’s converge cadence, so both wrappers prefix their run withANSIBLE_CACHE_PLUGIN_TIMEOUT=0. The prefix is not repo-wide: other plays rungather_facts: falseand a never-expiring cache would feed them stale facts. The ad-hoc wrapper carries the prefix too, because an unprefixed ad-hoc run after the cache expires replaces the stored facts instead of merging them and destroys both readings.