Skip to content

heimdall (out-of-band recovery host) Ansible Operations

heimdall (192.168.40.20) is a Raspberry Pi 4 Model B on a USB-flash root running Debian bookworm. It has no console at hand. If a change leaves it wrong, the fix is a walk to the device. It is the cluster’s out-of-band recovery host: the way in when the network that carries every other path is the thing being changed. Everything on this page follows from that fact.

heimdall does not have rg. Every command that runs on the box uses grep, awk or sed. The repository rule that requires rg governs searches of this repository, not commands on a remote host that does not ship it.

The baseline role is ansible/roles/heimdall-common, run by ansible/heimdall-playbook.yml against the heimdall_hosts group. The same playbook applies ansible/roles/vault-unseal (see The Vault unsealer). Host values live in ansible/inventory/host_vars/heimdall.yml.

Ansible connects as the existing fzymgc login at 192.168.40.20 over the LAN. fzymgc has passwordless sudo on the host. The private key lives in Vault at secret/fzymgc-house/infrastructure/heimdall/automation-ssh, field private_key, and nowhere else. It is an ed25519 key with the comment heimdall-automation.

The SSH agent is not used. An agent that holds many keys offers them all, and sshd refuses the connection after its attempt limit before the right key is reached. On a host whose only management path is SSH, Too many authentication failures is a lockout of the recovery host. Read the limit with sudo sshd -T | grep -i maxauthtries. The wrappers write the key to a 0600 temp file and force ssh to offer only it:

  • IdentitiesOnly=yes offers no identity other than the one passed with -i.
  • IdentityAgent=none consults no agent at all. Without it a working agent can satisfy the connection on its own, and the wrapper reports success while the Vault path is broken.

The repository modifies no sshd configuration on this host. The one change it made is one line appended to ~fzymgc/.ssh/authorized_keys. The verify block asserts both halves on every run: the checksum of /etc/ssh/sshd_config equals heimdall_sshd_config_sha256 in host_vars, and dpkg --verify openssh-server reports no changed file.

Why the tailnet path cannot carry automation

Section titled “Why the tailnet path cannot carry automation”

heimdall is Tailscale-tagged tag:bastion and tag:security. The matching SSH rule in tailscale/policy.hujson is action: "check". When Tailscale SSH handles a connection, tailscaled terminates it and authenticates by tailnet identity. authorized_keys is never read, so a Vault-held key offered over heimdall.hound-skate.ts.net does nothing. check mode also demands interactive re-authentication, which no unattended run can give.

Two error strings tell you which stack answered:

What you see Who emitted it What it means
tailscale: tailnet policy does not permit you to SSH as user "…" tailscaled you are on the tailnet path. Keys do not matter here
Permission denied (publickey) OpenSSH you reached the real sshd. Your key was wrong or absent

The second message is progress. The first means the transport is wrong, and no key fixes it.

scripts/heimdall-playbook.sh checks the Vault session, reads the key, writes it to a 0600 temp file that an EXIT trap removes, and runs ansible-playbook with the connection flags above. Pass through any ansible-playbook arguments:

Terminal window
# dry-run first — read-only verify block, no changes
scripts/heimdall-playbook.sh --tags heimdall-verify --check --diff
# the verify run itself (still read-only; a verify run reporting changed>0 is a defect)
scripts/heimdall-playbook.sh --tags heimdall-verify
# full converge
scripts/heimdall-playbook.sh --tags heimdall-common

The wrapper needs an authenticated Vault session. Run vault login or scripts/vault-helper.sh login first. vault status alone is not proof of a session, because it reads the unauthenticated seal-status endpoint and succeeds with an expired token. The wrapper names vault login in its error when the read fails.

Both wrappers prefix their run with ANSIBLE_CACHE_PLUGIN_TIMEOUT=0. See Known limits for why.

scripts/heimdall-adhoc.sh uses the same key handling and wraps ansible <pattern> … instead of ansible-playbook:

Terminal window
# reachability, no agent involved
scripts/heimdall-adhoc.sh heimdall -m ansible.builtin.ping
# a read as root
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command -a 'ss -lntp'

Ansible renders Jinja in ad-hoc module arguments, so do not put a literal {{ }} in the command. -m shell runs through the box’s /bin/sh. Prefer -m command unless you need a shell. Add -b when the target needs root. ansible.builtin.shell has no changed_when, so an ad-hoc run reports CHANGED for a pure read. That word means nothing here.

Verifying the credential (and the negative control that makes it mean something)

Section titled “Verifying the credential (and the negative control that makes it mean something)”

Run the positive check and the negative control together, in this order. Without the negative control an ambient SSH agent can satisfy the connection on its own, and the green proves nothing.

Terminal window
# POSITIVE — the Vault-held key alone must authenticate.
vault kv get -mount=secret -field=private_key \
fzymgc-house/infrastructure/heimdall/automation-ssh > /tmp/heimdall.key && chmod 600 /tmp/heimdall.key
ssh -o IdentitiesOnly=yes -o IdentityAgent=none -o BatchMode=yes \
-i /tmp/heimdall.key fzymgc@192.168.40.20 'echo authenticated-with-vault-key'
# NEGATIVE CONTROL — same invocation, no valid key. This MUST fail.
ssh -o IdentitiesOnly=yes -o IdentityAgent=none -o BatchMode=yes \
-o PasswordAuthentication=no -i /nonexistent fzymgc@192.168.40.20 true
# expected: Permission denied (publickey).
rm -f /tmp/heimdall.key

If the negative control succeeds, something other than the Vault key produced the positive result. Stop and find out what. If either command returns tailscale: tailnet policy does not permit…, you are on the tailnet path and neither result is about keys.

Manual fallback (if you need to run a different playbook)

Section titled “Manual fallback (if you need to run a different playbook)”
Terminal window
vault kv get -mount=secret -field=private_key \
fzymgc-house/infrastructure/heimdall/automation-ssh > /tmp/heimdall.key && chmod 600 /tmp/heimdall.key
cd ansible
ansible-playbook -i inventory heimdall-playbook.yml --tags <tag> \
-e ansible_ssh_private_key_file=/tmp/heimdall.key \
-e "ansible_ssh_common_args='-o IdentitiesOnly=yes -o IdentityAgent=none'"

The inner single quotes around the -o arguments are required. They keep the arguments together as one value for Ansible’s k=v parser. Without them Ansible splits on the spaces and fails with argument -o: expected one argument.

Four distinct things answer to the name heimdall. Treat a bare heimdall in any networking context as a defect. Fully qualify it.

Identity Address Where it is defined
The Pi, over the tailnet heimdall.hound-skate.ts.net tailnet MagicDNS. Unusable for automation, see above
The same Pi, on the LAN heimdall.fzymgc.house → 192.168.40.20 the heimdall entry in tf/dns-records/hosts.yml, which also carries the MAC that the Kea reservation on Core reads
A tailscale ACL alias, not DNS 192.168.40.20 the hosts map in tailscale/policy.hujson
An unrelated public machine heimdall.fzymgc.net tf/cloudflare/dns-fzymgc_net.tf

The Pi has a second interface, wlan0, on the IoT leg at 192.168.218.224. That address has no record and no reservation.

The ACL alias and the DNS record agree on 192.168.40.20 because both are edited by hand. Nothing keeps them in step. If the address changes, change the reservation entry, the alias and ansible_host together.

ansible/inventory/hosts.yml pins ansible_host to the literal address 192.168.40.20, not to a name. A literal address cannot be retargeted by a search-domain change, a MagicDNS change or an edit to a Cloudflare zone.

The Kea reservation for 192.168.40.20 comes from the mac on the heimdall entry in tf/dns-records/hosts.yml. The Kea role reads that file to render the Core reservations, and Core is in kea_served_legs in ansible/inventory/group_vars/dns_hosts.yml.

The role does not manage networking on heimdall. The NetworkManager profiles for eth0, wlan0 and the cell bearer are host state with no git representation. Read them on the box:

Terminal window
nmcli device status
nmcli -f IP4 device show eth0 # APPLIED state
nmcli con show "Wired connection 1" # STORED profile
ip -4 route show default

nmcli con show reports the stored profile and nmcli device show reports the applied state. They can differ. Read both before you write either one down. The Sixfab core_manager daemon rewrites the routing table with ip route and outranks NetworkManager, which sets routes only at activation. A default-route metric chosen with nmcli does not hold against it. Do not reconfigure the host’s networking to find out what wins.

If Ansible stops reaching the host at 192.168.40.20, use the Sixfab CORE Remote Terminal over the cell bearer, read the current address with ip -4 -o addr show dev eth0, and then fix the reservation entry or ansible_host.

Field Value
Path secret/fzymgc-house/infrastructure/heimdall/automation-ssh
Fields private_key, public_key
Key type ed25519, comment heimdall-automation

The path is operator-seeded, not Terraform-declared. No machine identity reads it: the wrappers run on an operator’s workstation with that operator’s own Vault token, and tf/vault/policy-infrastructure-developer.hcl already grants read on secret/data/fzymgc-house/infrastructure/*. The NAS automation key at …/infrastructure/nas/automation-ssh is managed the same way. If a machine identity ever needs this path, add a policy at that point.

This is an operator action. The private half never lands on the control node except as the wrapper’s temp file.

Terminal window
ssh-keygen -t ed25519 -C heimdall-automation -f ./heimdall-automation -N ''
vault kv put -mount=secret fzymgc-house/infrastructure/heimdall/automation-ssh \
private_key=@./heimdall-automation public_key=@./heimdall-automation.pub
shred -u ./heimdall-automation ./heimdall-automation.pub

Then append the public half to ~fzymgc/.ssh/authorized_keys on heimdall over the tailnet, which is the only path that authenticates before the key is in place. Append, never rewrite. A botched authorized_keys on this host is unrecoverable without a walk to the device.

Continuous patching on heimdall is the distribution’s mechanism, not Ansible’s: the apt-daily and apt-daily-upgrade timers plus unattended-upgrades, configured by a drop-in this repository renders at /etc/apt/apt.conf.d/52unattended-upgrades-heimdall. It sorts at 52 because apt reads apt.conf.d in lexical order, and a drop-in overrides the distribution’s 50unattended-upgrades only if it sorts after it. Named 50 or lower it becomes the file being overridden, and that inversion fails silently because both files parse.

The declared origins cover the Debian archive (bookworm, bookworm-security, bookworm-updates) and the Raspberry Pi archives. That is all updates, not security only. Tailscale is not an allowed origin. It is the transport this repository reaches heimdall over, and an unattended upgrade of it on the recovery host is an outage risk, not a patching win. The two periodic switches live in /etc/apt/apt.conf.d/20auto-upgrades (APT::Periodic::Update-Package-Lists "1" and APT::Periodic::Unattended-Upgrade "1"). With either set to "0" the whole policy is inert while every surface an operator glances at still looks healthy.

The role does not upgrade packages. It installs a small pinned set, unattended-upgrades, needrestart and jq, at state: present, plus whatever heimdall_packages in host_vars adds, and asserts everything else. A role that upgrades does something different on every run, so its success carries no information. Package currency is the job of unattended-upgrades.

needrestart is pinned to list-only, and must stay that way

Section titled “needrestart is pinned to list-only, and must stay that way”

/etc/needrestart/conf.d/50-heimdall.conf sets $nrconf{restart} = 'l';, which detects and reports and restarts nothing. Installing needrestart also installs /etc/apt/apt.conf.d/99needrestart, an apt hook that runs after every apt invocation, including every unattended-upgrades run. The shipped configuration leaves the restart mode unset, so the behaviour is otherwise decided at runtime by frontend detection. The service needrestart would most want to restart after a security upgrade is sshd, the only management path into this host.

The pin also protects the reboot gate, which reads needrestart -b -k -r l as its second trigger. A needrestart that had already restarted things would have destroyed that signal.

Use the read-only verify block, never a hand-run apt command:

Terminal window
scripts/heimdall-playbook.sh --tags heimdall-verify

It reports SC#2 in three tiers:

Tier Severity Condition
2a HARD unattended-upgrades installed, 20auto-upgrades present, and apt’s own parse (apt-config dump APT::Periodic) reports both periodic switches as "1"
2b WARN /var/lib/apt/lists is fresher than heimdall_apt_lists_max_age_days (3 days)
2c WARN nothing has been pending past one upgrade window

2a asserts the mechanism, not timer liveness. apt-daily.timer and apt-daily-upgrade.timer ship with apt itself and are active on a host with no unattended-upgrades installed, so timer liveness passes on a host with no patching mechanism. The role keeps it in the assert only as an extra conjunct.

The 3-day freshness threshold sits against the timers’ 1-day period. It allows two whole missed windows before it warns, so the warning means something when it fires.

apt list --upgradable is not expected to be empty

Section titled “apt list --upgradable is not expected to be empty”

The verify block does not assert an empty upgradable list. There is always a window between an archive publish and the next upgrade run, widened by RandomizedDelaySec, and tailscale is pending whenever upstream ships a release, because the origins policy excludes it on purpose. A non-empty list is reported at WARN, not asserted. An assert that goes red for a normal condition trains the operator to ignore it.

Four guards default to false in ansible/roles/heimdall-common/defaults/main.yml and are restated at host level in ansible/inventory/host_vars/heimdall.yml:

Variable Default What flipping it grants
heimdall_allow_reboot false an Ansible-initiated reboot, during a run someone is watching
heimdall_allow_eeprom_flash false rpi-eeprom-update -a, a bootloader write. On a Pi 4 the flash needs a reboot to apply, which is why the two are coupled
heimdall_gated_reboot_enabled false installs and enables the self-reboot timer, which is permission for the host to reboot itself with nobody watching
heimdall_allow_networkd_migration false the NetworkManager to systemd-networkd migration. It belongs to the network stack, not to reboot authority

They are separate flags because “I am watching this run right now” and “the host can reboot itself unattended” are different consents. One flag would let the second ride in on the first.

The gate, and why it is a script rather than a setting

Section titled “The gate, and why it is a script rather than a setting”

unattended-upgrades is configured Automatic-Reboot "false" and has no reboot authority. It has no pre-reboot veto hook: its reboot decision reads the marker and that setting and then schedules the shutdown, with no consultable hook between. So the decision is removed from it and re-implemented where a live check can run.

That home is /usr/local/sbin/heimdall-gated-reboot.sh, run by heimdall-gated-reboot.timer on heimdall_gated_reboot_schedule (*-*-* 04:15) with RandomizedDelaySec of heimdall_gated_reboot_jitter (1800 s, so this host does not reboot in lockstep with anything else on a round hour) and Persistent=true (a missed window replays at next boot). It reboots only if both gates pass, read at the reboot moment against live state:

  1. Is a reboot pending? /run/reboot-required exists, or needrestart reports NEEDRESTART-KSTA of 2 or 3 (a kernel upgrade is owed).
  2. Is the out-of-band rescue path up right now? The bearer interface heimdall_bearer_interface (usb0) carries a global, non-link-local IPv4 address, and heimdall_bearer_probe_target (1.1.1.1) answers three pings bound to that interface within heimdall_bearer_probe_timeout_seconds. Registration with the carrier is not reachability. The probe target is off-modem because the modem’s own next hop answers whether or not the carrier path works.

If either gate fails, the script logs the decision and returns, leaving the update staged for the next window. Staging is free: the marker lives on tmpfs and survives until the next boot.

On Debian /run/reboot-required is a policy convention written by individual postinst scripts, not a guarantee. A library can be upgraded with nothing touching the marker. That is why the gate reads needrestart as a second trigger and logs which trigger fired. Read the decisions with journalctl -t heimdall-gated-reboot.

The timer is installed and stays disabled until heimdall_gated_reboot_enabled is true. A host with no proven second path does not get permission to reboot itself unattended. The gate’s inputs are asserted shell-safe before the script is rendered, because they are interpolated into a script that runs as root.

The script has a no-act mode, reachable two ways:

  • --dry-run or -n as an argument, or
  • HEIMDALL_GATED_REBOOT_DRY_RUN=1 in the environment.

It evaluates every gate as a real run would, logs the decision it would take, and returns without invoking the reboot command.

Any manual exercise of the gate, by an operator or by an agent, uses no-act mode. A check must never perform the action it is checking. Acting mode belongs to the timer.

Three more guards bound a manual run:

  • Every argument other than --dry-run and -n is refused with exit 64 before any gate is evaluated, and the journal names the argument. A mistyped flag cannot select acting mode.
  • A non-default value for any gate input forces no-act mode: HEIMDALL_GATED_REBOOT_MARKER, HEIMDALL_GATED_REBOOT_BEARER_IFACE, HEIMDALL_GATED_REBOOT_PROBE_TARGET or HEIMDALL_GATED_REBOOT_PROBE_TIMEOUT. Only HEIMDALL_GATED_REBOOT_ALLOW_REAL_REBOOT=1 lifts that. The script logs exercise override in effect: <input>=<value> (default <default>). The interlock covers those four inputs and not process-environment inputs such as PATH.
  • HEIMDALL_GATED_REBOOT_REBOOT_CMD overrides the reboot command, so a caller can substitute an inert sentinel and the script cannot reboot the host however its internal guards are arranged.

Two internal exit codes exist. A completed decision (no-reboot-pending, rescue-path-down, DRY-RUN-WOULD-REBOOT) returns 0.

Code Meaning
64 usage error: an unrecognised argument, refused before any gate is evaluated
78 the reboot command is empty or whitespace-only, refused rather than claimed

The design is a vendor cloud broker (Sixfab CORE) with an on-host agent that dials out and exposes a remote terminal in the vendor dashboard. No port-forward and no inbound rule. It is independent of this network’s tailscale and this network’s DNS, which are the things most likely to be collateral damage in the changes this host exists to survive.

Do not re-run the vendor installer to fix a disconnected agent. It is piped from install.connect.sixfab.com to a shell, it purges modemmanager, and it runs git reset --hard HEAD && git pull against an unpinned upstream into /home/sixfab/. Re-running it is a supply-chain action dressed up as a repair. The modem is no longer owned by ModemManager: it presents as a plain ECM ethernet device (usb0) whose address comes from NetworkManager’s internal DHCP client.

If the agent shows Disconnected, in this order:

  1. Read the agent log first: sudo tail -n 200 /home/sixfab/.core/logs/cm/agent-log.
  2. Check what exists: systemctl is-active core_agent core_manager and systemctl is-enabled core_agent core_manager.
  3. Only if the units exist, restart the existing unit: sudo systemctl restart core_agent.
  4. Confirm registration in the vendor dashboard, then re-read the log.

The bearer’s SIM sits on a 1 GB per month pool. heimdall_bearer_route_metric (4096) keeps the bearer up without letting it win route selection. If the bearer ever becomes the default route, the pool is gone in hours and the rescue path with it. A remote terminal session costs kilobytes.

This configuration is not Ansible-managed. It lives in the vendor dashboard and in on-host files with no git representation.

wlan0 carries 192.168.218.224/24 on the IoT leg with its own default route through 192.168.218.1. It is reachable from the control node, and it rides neither the modem nor tailscale. It comes up unattended at boot from the persisted profile /etc/NetworkManager/system-connections/preconfigured.nmconnection.

It is not a rescue path for a NetworkManager removal. NetworkManager is the sole owner of its WPA credentials, its supplicant and its DHCP lease. Take NetworkManager down and wlan0 goes down with eth0. It also rides the house AP infrastructure.

heimdall runs NetworkManager. systemd-networkd is not enabled, and the migration to it is behind heimdall_allow_networkd_migration: false. The role performs no network change.

The rescue path is carried by NetworkManager. usb0 gets its address, gateway and default route from NetworkManager’s internal DHCP client, driven by an in-memory profile (Wired connection 2) that is not on disk. dhcpcd does not cover it: its configuration is an allow-list, allowinterfaces wwan*, and the modem enumerates as usb0. Remove NetworkManager and the bearer loses its address and its route.

A migration must therefore carry three interfaces together: eth0, wlan0 (its PSK relocated out of the NetworkManager keyfile and wpa_supplicant@wlan0 enabled) and usb0 (a DHCP client and a high-metric default route). Bring the rescue interfaces onto networkd first, in their own change window, and make sure that they survive a reboot. Then migrate eth0. Doing all three at once puts every rescue path in flight during the change it is meant to insure.

heimdall runs one of the Vault unsealers. The role ansible/roles/vault-unseal installs the pinned vault-unseal release binary as /usr/local/bin/vault-unseal and runs it under the systemd unit vault-unseal as the system user vault-unseal. The unit is enabled, so the unsealer returns with every boot. The unsealer holds all three key shares, so it alone restores a sealed member. Vault Operations lists the other unsealers.

Path Mode Owner Content
/etc/vault-unseal/ 0750 vault-unseal Directory
/etc/vault-unseal/vault-unseal.yaml 0600 vault-unseal Member URLs, intervals and the key shares
/etc/vault-unseal/fzymgc-ca-bundle.pem 0644 root CA bundle, set as SSL_CERT_FILE in the unit

The role touches no SSH, network or firewall setting. Converge it alone:

Terminal window
scripts/heimdall-playbook.sh --tags vault-unseal

The converge ends with a verify block that makes sure of three facts. The host resolves every member name, the user vault-unseal reads the configuration file, and the user nobody cannot read it. Run the verify block alone with --tags vault-unseal-verify. A --check run on a host without the unit fails at the start task, because check mode does not write the unit file.

Read the state of the unsealer:

Terminal window
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \
-a 'systemctl is-active vault-unseal; journalctl -u vault-unseal --since -15min --no-pager | tail -20'

Stop it for a single-host proof, and start it again after the proof:

Terminal window
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command -a 'systemctl stop vault-unseal'
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command -a 'systemctl start vault-unseal'

To roll back, disable the unit and revert the pull request that added the role:

Terminal window
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command -a 'systemctl disable --now vault-unseal'

Every step is read-only and safe to repeat.

  1. Dry-run the verify block: scripts/heimdall-playbook.sh --tags heimdall-verify --check --diff. Expect exit 0 with changed=0 unreachable=0 failed=0 and a terminal roll-up that names every criterion with its observed value. The ok= count changes whenever the role gains tasks and is not the thing to read. Two things are defects: a run reporting changed>0, and a run that stops before the roll-up.

  2. Run it for real: scripts/heimdall-playbook.sh --tags heimdall-verify. Same expectation. HARD criteria (SC#1b, SC#1c, SC#2a, SC#3a, SC#5) fail the run. WARN criteria (SC#2b, SC#2c, SC#3b) print a WARNING: line and the run still exits 0. A pending-package count at WARN is normal.

    SC#3a is the bootloader release-channel pin (heimdall_eeprom_release_channel), which either matches or does not. SC#3b is bootloader image drift, reported at WARN because flashing is guarded by heimdall_allow_eeprom_flash and a drifted image is a deliberate condition. SC#5 asserts that the modem and the serial UART survived the last boot by reading the USB bus: the Quectel pair 2c7c:0125, the FT232 id 0403:6001, and a non-zero /dev/ttyUSB* node count.

    When a HARD criterion is red, the roll-up still prints. Criteria below the failure read (not reached), and the roll-up carries a VERIFY BLOCK INCOMPLETE line. The run exits non-zero. The roll-up printing is not the run passing. Read failed= and the exit status.

  3. Confirm reachability without the agent: scripts/heimdall-adhoc.sh heimdall -m ansible.builtin.ping. Expect SUCCESS with "ping": "pong". Because the wrapper forces IdentitiesOnly=yes and IdentityAgent=none, this result can only come from the Vault-held key.

  4. Confirm the reboot timer is still disabled:

    Terminal window
    scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \
    -a 'systemctl is-enabled heimdall-gated-reboot.timer; systemctl is-active heimdall-gated-reboot.timer'

    Expect disabled then inactive. Anything else means someone granted the host unattended reboot authority.

    If you script this check, do not pipe it under set -o pipefail. systemctl is-enabled exits 1 for disabled and is-active exits 3 for inactive, so the expected answer makes the task report FAILED. Capture the output with || true and assert on the anchored literals ^disabled$ and ^inactive$. An unanchored enabled matches disabled.

  5. Exercise the gate in no-act mode. The flag is not optional:

    Terminal window
    scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.command \
    -a '/usr/local/sbin/heimdall-gated-reboot.sh --dry-run'

    On a healthy host with nothing pending, expect DECISION=SKIP reason=no-reboot-pending. The same decisions land in journalctl -t heimdall-gated-reboot.

    That first gate short-circuits, so the run above never reaches the later gates. To exercise the rescue-path gate and the no-act guard without touching real reboot-pending state, override the marker path and substitute an inert reboot command:

    Terminal window
    scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \
    -a 'M=$(mktemp); HEIMDALL_GATED_REBOOT_MARKER=$M \
    HEIMDALL_GATED_REBOOT_REBOOT_CMD="/usr/bin/touch /run/gsd-would-have-rebooted" \
    /usr/local/sbin/heimdall-gated-reboot.sh --dry-run; RC=$?; rm -f $M; echo "SCRIPT-RC=$RC"'

    Expect, in order: the interlock line exercise override in effect: marker=… (default /run/reboot-required); forcing no-act mode …, then reboot is pending, trigger=marker:…, then rescue path is up, detail=bearer interface usb0 carries …, then DECISION=DRY-RUN-WOULD-REBOOT all gates passed; no-act mode, not invoking the reboot command, and SCRIPT-RC=0. Then make sure that the guard held. The sentinel must be absent and the boot time unchanged:

    Terminal window
    scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \
    -a 'test ! -e /run/gsd-would-have-rebooted && uptime -s'

    --dry-run and the inert HEIMDALL_GATED_REBOOT_REBOOT_CMD are two independent belts. Never exercise the gate without both.

Each one is a place where something looks wired up and is not.

  • The gated-reboot unit has no OnFailure= notifier. heimdall has no notify unit and no template in this repository renders one. Pointing OnFailure= at a unit that does not exist would make the failure path itself fail silently. The journal is the record until a notifier exists.
  • A permanently closed rescue gate looks the same as a working one on every operator-visible surface. Both report DECISION=SKIP and exit 0. A wrong heimdall_bearer_interface fails closed, so it is safe and silent. The journal is the only detection surface. An alert on consecutive rescue-path-down decisions does not exist.
  • Skip-Updates-On-Metered-Connections "true" reads NetworkManager’s metered property. It works while heimdall runs NetworkManager. A migration to systemd-networkd makes it inert, because networkd exposes no such property, and the SIM’s 1 GB pool then depends on the bearer staying off the default route.
  • heimdall-gated-reboot.service has After=network-online.target, and that does not establish that the bearer is settled. usb0’s profile is created in memory after the device appears, and NetworkManager-wait-online waits only on the profiles it knows at startup. The unit can run before the bearer has an address and read the rescue path as down. The gate then skips, which is the safe direction, and the timer fires again the next day. Do not read a single rescue-path-down at boot as a modem fault.
  • Two daemons write /etc/resolv.conf. core_manager appends to it, and tailscale rewrites it with its own banner. While tailscale keeps rewriting, the file stays clean. If tailscale stops, the file grows without bound. Read the writers with sudo journalctl -u core_manager | grep resolv.
  • The reboot marker path is overridable through HEIMDALL_GATED_REBOOT_MARKER. That is what makes the gate exercisable without touching real state, and it means a root caller can open the first gate on purpose. The service unit carries no Environment= or EnvironmentFile= directive, so the scheduled path always reads the real /run/reboot-required. On the supervised path, any non-default gate input forces no-act mode unless HEIMDALL_GATED_REBOOT_ALLOW_REAL_REBOOT=1 is also set.
  • Cross-run detection depends on an Ansible fact cache that is control-node-local and gitignored. Two verify readings, the boot epoch behind “no reboot since the last recorded reading” and the unattended-upgrades effective-origins outcome behind “unverified twice running”, are cacheable facts. They compare this run against the previous run from this checkout on this machine. ansible/ansible.cfg sets fact_caching=jsonfile with a relative, gitignored fact_caching_connection, so another operator, another clone or a CI runner starts empty and reports no prior reading. Treat a first-run “no prior reading” as absence of evidence. The cache entries expire after 86400 seconds by default, which is shorter than this host’s converge cadence, so both wrappers prefix their run with ANSIBLE_CACHE_PLUGIN_TIMEOUT=0. The prefix is not repo-wide: other plays run gather_facts: false and a never-expiring cache would feed them stale facts. The ad-hoc wrapper carries the prefix too, because an unprefixed ad-hoc run after the cache expires replaces the stored facts instead of merging them and destroys both readings.