Operational guide for heimdall (192.168.40.20) under Ansible management.
heimdall is a Raspberry Pi 4 Model B Rev 1.1 (BCM2711) on a USB-flash root running Debian
bookworm, with no handy console access — if a change leaves it wrong, the fix is a walk to the
device, not a command. It is the cluster’s out-of-band recovery host: the escape hatch the v1.4
DNS cutover phases depend on, precisely because the network those phases change is the network you
would otherwise use to log in. Everything on this page is shaped by that one fact.
heimdall does not have rg. Every on-box command on this page uses grep/awk/sed.
The repo’s MUST-use-rg rule governs searches of this repository, not commands executed on a
remote box that does not ship it. jq was also absent until the role began converging it — a
jq-dependent probe run before that reported “no modem enumerated” on a host where the modem is
plainly present, which is why the gated-reboot script fails closed on a missing parser rather
than answering confidently.
the only path that authenticates before a key exists in authorized_keys; used once, to install the credential
Everything after
fzymgc at 192.168.40.20over the LAN (inventory default)
key is in Vault, not on disk
The two paths are complementary, not alternatives: the tailnet path authenticates by identity
and is how the credential got installed; the LAN path authenticates by key and is the only one an
unattended run can use. See Why the tailnet path cannot carry automation.
Ansible connects as the existing fzymgc login — no new user was created, and fzymgc already
has passwordless sudo on this host. The private key lives in Vault
(secret/fzymgc-house/infrastructure/heimdall/automation-ssh, field private_key), not on the
control node. It is an ed25519 key with comment heimdall-automation and fingerprint
SHA256:gBIDCva2ut+5UrDOpDEcfZfDHMcjkdLg3w4mgI6jw2E; Vault holds the only copy of the private half.
The SSH agent is deliberately not used. A typical agent holds many unrelated keys, and offering
them all trips sshd’s authentication-attempt limit before the right key is reached. The number is
not hypothetical here — sshd -T on heimdall reports maxauthtries 6, so an agent carrying
seven or more keys exhausts the budget and never gets to the right one. On a host whose only
management path is SSH, “Too many authentication failures” is not a nuisance, it is a lockout of
the recovery host itself. So the key is materialized to a 0600 temp file and two flags force ssh
to offer only it:
IdentitiesOnly=yes — do not offer any identity other than the one passed with -i.
IdentityAgent=none — do not consult an agent at all. This is not redundant with the first:
without it a working agent can satisfy authentication on its own, and the wrapper would report
success while the Vault path was broken — a green that proves nothing.
No sshd configuration was modified on this host, and none will be./etc/ssh/sshd_config has
mtime 2025-05-12 20:14:39 and is byte-identical to the packaged version (dpkg --verify openssh-server exits 0 with empty output); onboarding appended exactly one line to
~fzymgc/.ssh/authorized_keys and touched nothing else. The verify block asserts both halves of
that claim on every run.
This is the failure mode most likely to waste an hour, so it is stated first and plainly.
heimdall is Tailscale-tagged ["tag:bastion","tag:security"], and the matching rule in
tailscale/policy.hujson is action: "check" over users: ["root","pi","ubuntu","fzymgc"].
When Tailscale SSH handles a connection, tailscaled terminates it and authenticates by tailnet
identity — authorized_keys is never consulted. A Vault-held keypair used over
heimdall.hound-skate.ts.net therefore does nothing at all, silently; and check mode
additionally demands interactive re-authentication, which no unattended timer or CI run can
satisfy.
Two error strings tell you which stack answered, and this is the single most reusable diagnostic on
this page:
What you see
Who emitted it
What it means
tailscale: tailnet policy does not permit you to SSH as user "…"
tailscaled
you are on the intercepted tailnet path; keys are irrelevant here
Permission denied (publickey)
OpenSSH
you reached the real sshd; authorized_keys is live and your key was wrong or absent
Reaching the second message is progress, not failure. Reaching the first means the transport is
wrong, and no amount of key fixing will help.
scripts/heimdall-playbook.sh does all of the above in one command — it checks the Vault session,
reads the key, materializes it to a 0600 temp file (scrubbed by an EXIT trap), and runs
ansible-playbook with the correct connection flags. Pass through any ansible-playbook args:
Terminal window
# dry-run first — read-only verify block, no changes
# the verify run itself (still read-only; a verify run reporting changed>0 is a defect)
scripts/heimdall-playbook.sh--tagsheimdall-verify
# full converge
scripts/heimdall-playbook.sh--tagsheimdall-common
Prereq: an authenticated Vault session (run vault login, or scripts/vault-helper.sh login,
first). The wrapper checks it and fails before touching the host, naming vault login in the
error — vault status alone is not sufficient, because it reads the unauthenticated seal-status
endpoint and succeeds happily with an expired token.
scripts/heimdall-adhoc.sh is the ad-hoc sibling — same Vault-key handling, but it wraps
ansible <pattern> … instead of ansible-playbook, for one-off read-only checks without writing a
throwaway playbook:
Gotchas: ad-hoc module args are Jinja-templated by Ansible, so avoid a literal {{ }} in the
command; -m shell runs via the box’s /bin/sh (prefer -m command when you do not need a
shell); add -b when the target needs root. ansible.builtin.shell has no changed_when, so
ad-hoc runs report CHANGED even for a pure read — that word means nothing here.
Verifying the credential (and the negative control that makes it mean something)
Auth verification must be agent-free and paired with a negative control. Without the negative
control an ambient SSH agent can satisfy the connection on its own and the green proves nothing at
all. Run both, in this order:
Terminal window
# POSITIVE — the Vault-held key alone must authenticate.
If the negative control succeeds, your positive result was produced by something other than the
Vault key — stop and find out what. If either command returns
tailscale: tailnet policy does not permit…, you are on the tailnet path and neither result is
about keys at all.
Manual fallback (if you need to run a different playbook)
The inner single quotes around the -o args are required: they keep the args together as one
value for Ansible’s k=v parser. Without them, ansible’s ssh TTY-parser mis-splits on the spaces
and fails with argument -o: expected one argument. This is why the wrapper hard-codes the
quoted form — don’t “simplify” the quoting.
the Firewalla’s per-device publisher. Appears in no git-tracked file — searching tf/ for either address will always come up empty, and that is expected
A tailscale ACL alias — not DNS at all
192.168.40.20
tailscale/policy.hujson
An unrelated public machine
heimdall.fzymgc.net → 159.203.126.35
tf/cloudflare/dns-fzymgc_net.tf
Row 2 returns two addresses for one name because the Pi has two interfaces on two different
subnets (eth0 on Core, wlan0 on IoT) and the publisher emits one file per device MAC with no
notion that they are the same host. A client gets whichever the resolver hands back.
Rows 2 and 3 currently agree on 192.168.40.20, and that agreement is a coincidence of
maintenance rather than a mechanism. The ACL alias was stale and pointing at nothing until a
Firewalla DHCP reservation pinned eth0 to that address on 2026-07-30; before then the two
disagreed (192.168.41.103 versus 192.168.40.20), which is the ordinary state of affairs here.
Nothing keeps them in step — the alias is hand-maintained and will drift again the moment the lease
changes without someone editing policy.hujson.
192.168.41.103 is dead. It was heimdall’s pre-reservation lease and stopped being an
address for this host on 2026-07-30. It appears here only so that a reader who finds it in an old
note recognises it as history. It must not appear in inventory or in any runbook.
ansible/inventory/hosts.yml therefore pins ansible_host to the literal IP192.168.40.20,
not to any name: a literal address cannot be retargeted by a search-domain change, a MagicDNS
change, or an edit to a Cloudflare zone.
For whyheimdall.fzymgc.house resolves at all when no declared source publishes it, see
Internal DNS
— the local:domain:suffix section of the three-writer model. That explanation is not repeated
here.
The 192.168.40.20 reservation has no declared source
Added 2026-07-31 (01-09, per D-25). If you are here because you are wondering why heimdall has
that address, this is the answer.
A Firewalla DHCP reservation, created 2026-07-30, pins heimdall’s eth0 to
192.168.40.20. That reservation is what makes ansible_host: 192.168.40.20 in
ansible/inventory/hosts.yml correct.
Property
Value
Owner
The Firewalla app. That is its only representation
Declared source
None. No git representation, no Terraform resource, no Ansible task declares it
Closure
Deferred to Phase 5
Say the consequence plainly: the recovery host’s management path depends on undeclared state
living on the very appliance the host exists to survive. That is the same undeclared-writer class
this milestone exists to close, arriving on the one host whose whole purpose is staying reachable
when the Firewalla is what broke.
Nothing here says the literal IP was the wrong call — it is genuinely stronger than a name, since
no search-domain or MagicDNS change can retarget a literal address. The undeclared reservation
underneath it is a separate weakness that the literal IP neither creates nor fixes.
Nothing is built here to fix it, deliberately. There is no Firewalla Terraform provider
anywhere in this repo — Firewalla appears only as a Vault PKI/AppRole consumer, an ArgoCD MCP
deployment, and ansible/router-playbook.yml — so building one would be scope expansion rather than
gap closure. Closure belongs to Phase 5, which owns namespace closure and drift detection: its
SC#1 already names heimdall as a resolution with no declared source and requires it resolved one
way or the other, and execution gate G3 already observes that heimdall resolved for months with no
declared source.
What to do if the reservation is lost. heimdall takes a different lease, ansible_host stops
pointing at the host, and Ansible stops reaching it at that address. The recovery path in that
situation is the Sixfab CORE Remote Terminal over the cell bearer — not the LAN. Get a shell
that way, read the current address with ip -4 -o addr show dev eth0, and either re-create the
reservation in the Firewalla app or update ansible_host.
Updated 2026-07-31 (01-14) — the reservation is no longer what pins eth0, and its
replacement is undeclared in the same way. The failure mode above stopped being hypothetical:
heimdall’s LAN path went down on 2026-07-31 and was repaired by hand. The reservation sat inside
the operator’s 192.168.40.0/24 STATIC block, and a renewal stopped sending DHCP option 3 — so
eth0 kept an address and lost its gateway. eth0 is now a static NetworkManager
profile (192.168.40.20/22, gateway 192.168.40.1, DNS 192.168.40.1, NM route-metric 100),
and usb0’s route-metric is pinned to 500 so the cell bearer is a BACKUP default and never
primary. Route ordering as it now stands: eth0 100 < wlan0 300 < usb0 500.
This repo does not declare any of it.heimdall-common does not manage networking — 01-07
deferred that to Phase 6 / DNS-14 — so this is nmcli-level host state with no git representation,
exactly like the reservation it replaced. Recorded here as a known-undeclared item alongside
the reservation above, not as something this phase closed. 01-14 deliberately did not expand
scope to declare it, and a reader planning Phase 6 should treat these three metrics as measured
host state to be ported, not as configuration to be discovered.
Superseded 2026-07-31 (01-17). The route ordering stated above is not what the host is
running. It was written from nmcli con show — the STORED profile — a few hours after the repair,
and it records the intent of an nmcli edit as though it were applied state. Measured since:
Terminal window
$ ip -4 route show default
default via 192.168.218.1 dev wlan0 proto dhcp src 192.168.218.224 metric 300
default via 192.168.225.1 dev usb0 proto dhcp src 192.168.225.59 metric 500
nmcli con show reports stored configuration; nmcli device show reports applied state. On
this host they diverge silently, and the stored value is the one that reads like an answer. Check
both before writing either down.
Why it will not stay put. Sixfab CORE’s core_manager rewrites the routing table directly
with sudo ip route del/add and outranks NetworkManager — NM sets routes only at activation,
core_manager rewrites them continuously, and its metric policy is not a constant. The
metric-100 default held ~50 minutes (60/60 pings, ansible.builtin.ping → pong, no churn in the
window), then at 12:09 core_manager re-added it at metric 10000 — behind wlan0 300 and
usb0 500 — and it later vanished. 60 route operations since boot. No externally-chosen
metric is durable against it, so do not conclude “just reactivate the profile”.
heimdall is still reachable, and the reason matters: replies now leave via wlan0, and
192.168.218.0/24 is a real routable VLAN. During the earlier outage usb0 was primary and
sourced replies from 192.168.225.59, a modem-private subnet with no return path, so they were
dropped as martians. It is the reply source address, not the metric, that decides whether
off-subnet traffic works.
A reboot is pending (/var/run/reboot-required exists) and the gated-reboot timer is
disabled/inactive, so nothing fires unattended. A reboot would re-apply the stored profile —
whether core_manager then re-demotes it is an open question, not a prediction. Settling it
is Phase 6’s, in a change window; do not reconfigure this host’s networking to find out.
The path is operator-seeded, not Terraform-declared, and that is a decision rather than an
omission.
No Terraform policy resource is created for this path.tf/CLAUDE.md requires a matching Vault
policy update alongside a new secret path, and the reason that rule does not bite here is that
no machine identity reads this path: the wrapper scripts run on an operator’s workstation and
read it over that operator’s own Vault login token. The grant already exists —
tf/vault/policy-infrastructure-developer.hcl wildcards
secret/{data,metadata}/fzymgc-house/infrastructure/* (lines 12 and 17), so this path is covered
the moment it exists. The closest analogous path in this repo, the NAS automation SSH key
(…/infrastructure/nas/automation-ssh), is managed exactly the same way: a search of tf/ finds
no policy and no secret resource for it, and docs/operations/nas.md records it as an operator
prerequisite.
If a machine identity ever needs to read this path, a policy must be added at that point. The
condition that would reverse this decision is that specific one, and nothing else.
Recorded so the credential can be rotated without re-deriving the commands. This is an operator
action; the private half never lands on the control node except as the wrapper’s scrubbed temp file.
The public half must then be appended to ~fzymgc/.ssh/authorized_keys on heimdall — over the
tailnet, which is the only path that authenticates before the key is in place. Append; never
rewrite. The file carries a pre-existing duplication (5 lines, 3 unique keys) that is deliberately
left alone: de-duplicating it would be churn on the one host where a botched authorized_keys is
unrecoverable without a walk to the device.
Continuous patching on heimdall is the distro’s mechanism, not Ansible’s: the apt-daily and
apt-daily-upgrade timers plus unattended-upgrades, configured by a policy drop-in this repo
owns at /etc/apt/apt.conf.d/52unattended-upgrades-heimdall. It sorts at 52 deliberately — apt
reads apt.conf.d in lexical order, so a drop-in overrides the distro-shipped
50unattended-upgrades only if it sorts after it. Named 50 or lower it would be the file being
overridden, and that inversion fails silently because both files parse fine either way.
The declared origins cover the Debian archive (bookworm, bookworm-security, bookworm-updates)
and the Raspberry Pi archives — all updates, not security only, which stock Debian policy does
not give you. Tailscale is deliberately excluded even though it is an available origin: it is the
transport this repo reaches heimdall over, and upgrading it unattended on the recovery host is a
self-inflicted-outage risk, not a patching win. The two periodic switches live in
/etc/apt/apt.conf.d/20auto-upgrades (APT::Periodic::Update-Package-Lists "1" and
APT::Periodic::Unattended-Upgrade "1"); with either set to "0" the entire policy above is inert
while every surface an operator glances at still looks healthy.
The role does not upgrade packages. It converges a small pinned set — unattended-upgrades,
needrestart, jq — at state: present, and asserts everything else. The reason is a durable
lesson rather than a style preference: a role that upgrades does something different on every run,
so its “success” carries no information and a second run can never be a no-op. Package currency
on this host is unattended-upgrades’ job; it is never performed imperatively by a converge task.
needrestart is pinned to list-only, and must stay that way
/etc/needrestart/conf.d/50-heimdall.conf sets $nrconf{restart} = 'l'; — detect and report,
restart nothing. This is not tidiness, and removing it is not a cleanup.
Installing needrestart also installs /etc/apt/apt.conf.d/99needrestart, an apt hook that runs
after every apt invocation, including every unattended-upgrades run. The shipped
/etc/needrestart/needrestart.conf leaves the restart mode commented out, so the behaviour would
otherwise be decided at runtime by frontend/TTY detection — and the service needrestart would most
want to restart after a security upgrade is sshd, the only management path into this host.
“It probably falls back to list mode under DEBIAN_FRONTEND=noninteractive” is a guess, and this
host is not a place to build a reboot-adjacent invariant on a guess.
The pin also protects the reboot gate, which consults needrestart -b -k -r l as its second
trigger. A needrestart that had already silently restarted things would have destroyed the very
signal the gate is asking it for.
Use the read-only verify block — never a hand-run apt command:
Terminal window
scripts/heimdall-playbook.sh--tagsheimdall-verify
It reports SC#2 in three tiers, and the tiering is the point:
Tier
Severity
Condition
2a
HARD
unattended-upgrades installed, 20auto-upgrades present, and apt’s own parse (apt-config dump APT::Periodic) reports both periodic switches as "1"
2b
WARN
/var/lib/apt/lists is fresher than heimdall_apt_lists_max_age_days (3 days)
2c
WARN
nothing has been pending past one upgrade window
2a asserts the mechanism, deliberately, not timer liveness. apt-daily.timer and
apt-daily-upgrade.timer ship with apt itself and were both active on heimdall on 2026-07-30
while unattended-upgrades was not installed at all and 20auto-upgrades did not exist. Timer
liveness therefore passes on a host with no patching mechanism whatsoever and must never be read
as evidence. It is carried in the assert only as an extra conjunct.
The 3-day freshness threshold is chosen against the timers’ 1-day period: it allows two entirely
missed windows before warning, so the warning means something when it fires.
An empty upgradable list is deliberately not asserted, because it is unsatisfiable by
construction on a healthy host: there is always a window between an archive publish and the next
upgrade run, widened further by RandomizedDelaySec. This is not theory. Measured on heimdall with
the mechanism fully installed and unattended-upgrades having succeeded fourteen minutes earlier:
Six packages pending on a correct, healthy, fully-patched host — and tailscale is pending
precisely because the origins policy excludes it on purpose. A non-empty list is normal mid-window
and is reported at WARN, not asserted. An assert that goes red for a normal condition trains the
operator to ignore it, which costs more than it ever catches.
Three separate guards, all defaulting to false, declared in
ansible/roles/heimdall-common/defaults/main.yml and restated at host level in
ansible/inventory/host_vars/heimdall.yml:
Variable
Default
What flipping it grants
heimdall_allow_reboot
false
an Ansible-initiated reboot, during a run someone is watching
heimdall_allow_eeprom_flash
false
rpi-eeprom-update -a — a bootloader write. This is a BCM2711 (Pi 4), so the flash needs a reboot to apply, which is why the two are coupled
heimdall_gated_reboot_enabled
false
installs and enables the self-reboot timer — permission for the host to reboot itself at 04:15 with nobody watching
They are separate flags rather than one because “I am watching this run right now” and “the host may
reboot itself unattended” are materially different consents, and collapsing them would let the
second ride in on the first. (A fourth guard, heimdall_allow_networkd_migration, also defaults to
false; it belongs to the network migration, not to reboot authority.)
The gate, and why it is a script rather than a setting
unattended-upgrades is configured Automatic-Reboot "false" and has no reboot authority at
all. That is a structural necessity, not a preference: u-u has no pre-reboot veto hook. Upstream’s
main() calls reboot_if_requested_and_needed(), which consults the reboot marker and that setting
and then calls schedule_shutdown(); there is no consultable hook in between, and the plugin API’s
postrun runs earlier with its return value discarded. So the reboot decision is removed from u-u
entirely and re-implemented where the live check can actually happen.
That home is /usr/local/sbin/heimdall-gated-reboot.sh, run by heimdall-gated-reboot.timer at
*-*-* 04:15 with RandomizedDelaySec=1800 (30 minutes of jitter, so this host does not reboot in
lockstep with anything else that picked a round hour) and Persistent=true (a missed window replays
at next boot rather than being skipped silently). It reboots only if both gates pass, evaluated
at the reboot moment against live state:
Is a reboot genuinely pending?/run/reboot-required exists, or needrestart reports
NEEDRESTART-KSTA of 2 or 3 (an ABI-compatible or version kernel upgrade is owed).
Is the out-of-band rescue path up right now? A connected bearer carrying an IP — and
nothing less. Registration is not reachability; see below.
If either fails, it logs the decision and returns, leaving the update staged for the next window.
Staging is free: the reboot marker lives on tmpfs and survives until the next boot, so a skipped
window is simply retried with no state to manage and nothing to clean up.
Marker caveat: on Debian, /run/reboot-required is a Policy convention written by
individual postinst scripts, not a guarantee — Ubuntu’s broad update-notifier-common hook has
no Debian equivalent. A security library can therefore be upgraded with nothing touching the
marker, and a gate keyed on it alone would never fire while the box quietly ran old code. That is
why the gate consults needrestart as a second, independent trigger, and why it logs which trigger
fired: journalctl -t heimdall-gated-reboot.
The timer ships installed and deliberately disabled, and if you run the gate right now it logs
DECISION=SKIP. Both are by design.
The rescue-path gate requires a bearer carrying an IP. Measured on heimdall: the modem is
registered, packet service is attached, operator AT&T Twilio — and .modem.generic.bearers is
the empty array. Registered and attached describe the radio’s relationship with the carrier;
they do not mean a usable network path exists. There is no bearer, no interface, no address, no
route. A gate keyed on registration would have opened wide on a host with no reachable rescue
channel at all, which is precisely the failure this gate exists to prevent.
Corrected 2026-07-31 (01-09) — the measurement in the paragraph above predates the CORE
install and is no longer true. There is a bearer now, and it does carry an address: usb0
holds 192.168.225.59/24 and 1.1.1.1 answers through it. The empty-bearers-array reading was
taken before the operator installed Sixfab CORE on 2026-07-30. It is kept rather than deleted
because it is the reason the gate was designed to key on a carried address instead of on
registration state — and that design decision is still the right one, which is exactly why the
gate survived the bearer appearing underneath it.
Enablement is coupled to the rescue path being proven, not assumed: plan 01-07 flips
heimdall_gated_reboot_enabled to true only after its rescue-path gate passes. If that gate
fails, the timer correctly stays off — a host with no proven second path does not get permission to
reboot itself unattended. A gate that opened today would be the bug.
Updated 2026-07-30 (01-07). The rescue path is now proven, but the timer still stays
disabled — for a different reason than the one above, and the difference matters. The gate probes
the bearer through mmcli, and the Sixfab CORE installer purged ModemManager. The probe now
fails closed on a missing tool and reports DECISION=SKIP reason=rescue-path-downeven though
the bearer is up and carrying traffic. The gate is not merely closed; it is blind — it measures
ModemManager, which no longer owns this modem. Enabling the timer today would install a timer that
can never fire. Re-pointing the probe at the live stack is follow-up work; see
the D-07 decision.
Updated 2026-07-31 (01-08). The gate is no longer blind. The block above is the record of
what was believed on 2026-07-30 and is what makes this correction legible; it is superseded, not
wrong-at-the-time.
The probe no longer asks any daemon anything. It reads the bearer interface’s own address
plus a reachability check bound to that interface — a global, non-link-local IPv4 on the
named bearer AND an off-modem target answering through it. That is a condition, not a package,
so it can see the live path.
Demonstrated RED by overriding the bearer interface name to one that does not exist
(HEIMDALL_GATED_REBOOT_BEARER_IFACE=gsd-absent0), not by taking the rescue path down.
Taking the out-of-band path down on the recovery host to manufacture evidence is forbidden —
the host’s whole purpose is remaining reachable when the Firewalla is what broke. Observed:
DECISION=SKIP reason=rescue-path-down detail=bearer interface gsd-absent0 is absent or its address could not be read.
Demonstrated GREEN against the real bearer: rescue path is up, detail=bearer interface usb0 carries 192.168.225.59 and the probe target 1.1.1.1 answered through it.
The timer is still disabled and is-active still inactive — now for a different and
weaker reason than before, and the difference is the whole point. Previously the gate could
not see. Now it can, and the remaining question is narrower: does its GREEN keep corresponding
to a working Remote Terminal over time? That correspondence has been observed once
(operator-attested at 01-08’s checkpoint, 2026-07-31) rather than over a period. One
observation is enough to prove the probe is not blind; it is not enough to hand an unattended
reboot to a host with no console. Phase 6 owns that decision.
HEIMDALL_GATED_REBOOT_DRY_RUN=1 in the environment.
It evaluates every gate exactly as a real run would, logs the decision it would take, and returns
without invoking the reboot command.
Any manual exercise of the gate — by an operator or by an agent — uses no-act mode. Running it
for real can reboot the host, and a check must never perform the action it is checking. Acting mode
belongs to the timer, and to one deliberate supervised run after the rescue path is proven.
A third safeguard exists for anyone scripting around it: HEIMDALL_GATED_REBOOT_REBOOT_CMD
overrides the reboot command itself, so a caller can substitute an inert sentinel and the script
cannot reboot the host regardless of how its internal guards are arranged. That is a property a
caller can rely on without reading the file.
Updated 2026-07-31 (01-11) — this section described a parser that no longer exists. Until
today the script examined only its FIRST argument and discarded anything it did not recognise
in silence, which meant a mistyped no-act flag selected ACTING mode on a host nobody can see.
The code review measured four plausible mistypings — --dryrun, --dry_run, --dry-run=1 and
--verbose — all four selecting acting mode. Three things changed:
-n is accepted as an alias for --dry-run. The most plausible short-form typo now lands
on the SAFE mode rather than the acting one.
Every other argument is refused with exit 64, before any gate is evaluated. The refusal
names the offending argument verbatim in the journal. All four mistypings above were
re-measured on the rendered script and now exit 64.
A non-default value for ANY input to either gate FORCES no-act mode — the reboot marker
HEIMDALL_GATED_REBOOT_MARKER, the bearer interface HEIMDALL_GATED_REBOOT_BEARER_IFACE, the
probe target HEIMDALL_GATED_REBOOT_PROBE_TARGET or the probe timeout
HEIMDALL_GATED_REBOOT_PROBE_TIMEOUT — unless a second, explicit consent variable
HEIMDALL_GATED_REBOOT_ALLOW_REAL_REBOOT=1 is also present. Corrected 2026-07-31 by
01-16: this bullet stated the interlock as a list of two of those inputs until today, which
is the same enumeration whose incompleteness was the round-2 Critical — and the pair it left out
included the probe target, the knob the review reproduced reaching a real reboot on a dead
carrier path. It is stated as a rule over every input rather than as a list deliberately,
because a list is how the probe target was missed once already; the script calls one interlock
helper once per input so a fifth input cannot quietly escape. The rule is bounded to the
script’s own four gate inputs above, and this bound is stated rather than left to inference:
it does not reach process-environment inputs such as PATH and IFS, which the script does not
pin and which reach both gates through the unqualified needrestart, ip and ping calls.
That is an OPEN gap (WR-13 in this phase’s deferrals ledger), it needs root, and the scheduled
path is unaffected because the service unit carries no environment directives. The script logs
exercise override in effect: <input>=<value> (default <default>) naming which one it saw.
“I am running a supervised exercise” and “this host may reboot right now” are materially
different consents, and this stops the second riding in on the first.
The interlock is a second belt, not a licence. The standing instruction above is unchanged:
any manual exercise of the gate — by an operator or by an agent — uses no-act mode. Nothing
here authorises dropping the flag; it bounds what happens when someone does.
Status of the interlock, stated honestly: source-verified and runtime-unproven. Its code path
has been observed executing and logging in no-act mode, and observed NOT logging on the default
path — but proving it CHANGES the outcome requires entering acting mode, which the instruction
above forbids. That was put to the operator at 01-11’s checkpoint on 2026-07-31 and declined
(no-act-only). The parser change, which closes the mistyped-flag scenario, was measured on the
live host and is not affected by that. Recorded as INCONCLUSIVE in 01-VALIDATION.md rather than
written up as a pass.
Two new internal exit codes, so a reader who finds one in a journal knows what it means. Both
are internal errors; a COMPLETED decision — no-reboot-pending, rescue-path-down,
DRY-RUN-WOULD-REBOOT — still returns 0.
Code
Meaning
64
usage error: an unrecognised argument. Refused before any gate is evaluated
78
the reboot command is empty or whitespace-only. Refused rather than claimed, because an empty argv makes the shell return 0 and the journal would record a reboot that did not happen
The design is a vendor cloud broker (Sixfab CORE) with an on-host agent that dials out and
exposes a remote terminal in the vendor dashboard. No port-forward, no inbound rule — and,
critically, independent of both this network’s tailscale and this network’s DNS. Those two are
exactly the things most likely to be collateral damage in the network changes this host exists to
survive.
Do not re-run the vendor installer to “fix” a disconnected agent. It is piped from
install.connect.sixfab.com to a shell; its remove_conflicting_packages() purges
modemmanager — which currently owns this host’s modem end to end — and its install_agent()
does git reset --hard HEAD && git pull against an unpinned upstream into /home/sixfab/. Read
the agent’s own log first, then restart the existing unit. Re-running the installer is a
supply-chain action dressed up as a repair.
Updated 2026-07-31 (01-11) — the ownership clause above is stale; the warning around it is
not.modemmanager has not owned this host’s modem since 2026-07-30, when the installer was
run by hand and remove_conflicting_packages() did exactly what this warning says it does. Read
“which currently owns this host’s modem end to end” as the pre-2026-07-30 state, kept because it
is what made the cost of that run concrete.
The warning itself is still correct and still binding. Re-running the installer is still a
supply-chain action dressed up as a repair; install_agent() still does
git reset --hard HEAD && git pull against an unpinned upstream into /home/sixfab/. What
changed is only that the purge it warns about has already happened here — which removes none of
the reasons not to do it again.
Operator steps if the agent shows Disconnected, in this order:
Read the agent log first:sudo tail -n 200 /home/sixfab/.core/logs/cm/agent-log
(heimdall has no rg — use grep/awk/sed on the box).
Check what actually exists: systemctl is-active core_agent core_manager and
systemctl is-enabled core_agent core_manager.
Only if the units exist, restart the existing unit: sudo systemctl restart core_agent.
Confirm registration in the vendor dashboard, then re-read the log.
Measured state, stated plainly because it changes what step 3 means today: on 2026-07-30
core_agent and core_manager were both inactivewith no unit files at all
(Failed to get unit file state … No such file or directory), and /home/sixfab/.core/monitor.yaml
was absent. So CORE is not the live rescue path on this host right now, and step 3 has nothing to
restart. Nothing here purged ModemManager, which is why the modem is still ModemManager’s.
Superseded 2026-07-30, later the same day (01-07). The paragraph above records the state
before the operator installed CORE by hand. Both units are now active and enabled,
sixfab-terminal v2 is running, and an interactive Remote Terminal was proven end to end through
connect.sixfab.com (prompt sixfab@heimdall:/opt/sixfab/core/agent $). Step 3 now has a real
unit to restart. The bearer is up: usb0 holds 192.168.225.59/24, ping -I usb0 1.1.1.1
returns 2/2 at ~179–293 ms, and the terminal session cost 4.28 kB against the 1 GB/month pool
(usb0 total since boot: 0.09 MB). The route ordering is correct — eth0 100, wlan0 300,
usb0 500 — so the bearer is present but never wins route selection, which is what D-11 asks for.
Superseded 2026-07-31 (01-17) — the conclusion survives, the reason does not. D-11 still
holds: usb0 is not primary. But it holds via wlan0 at metric 300, not via the
eth0 100 cited above — eth0 currently has no default route at all (IP4.GATEWAY: --,
mt = 10000), because core_manager re-added it at metric 10000 and it later vanished.
Read the ordering above as the state on 2026-07-30, not as the grounds for believing D-11
today. The full measurement and why no chosen metric survives core_manager are in the
01-17 correction under “The 192.168.40.20 reservation”.
The installer’s documented cost was paid.remove_conflicting_packages() purged
modemmanager; dpkg now reports un modemmanager <none> and mmcli is gone. The warning above
is therefore still correct and still binding — it describes what happens, and it happened. One
consequence is live right now: verify.yml’s SC#5 assert shells out to mmcli -L and fails. See
the D-07 decision.
Corrected 2026-07-31 (01-11) — the “live right now” clause above expired on 2026-07-31, and
this is the section you are reading DURING a rescue-path incident.01-08 closed it.
verify.yml contains zero occurrences of the purged modem CLI. SC#5’s assert reads the USB
bus directly: the Quectel VENDOR:PRODUCT pair 2c7c:0125 (tightened from the bare vendor
prefix by 01-10), the FT232 identifier 0403:6001, and a non-zero /dev/ttyUSB* node
count. The gated-reboot rescue probe reads the bearer interface’s own global IPv4 address plus
an off-modem target answering through that interface. Neither asks a daemon anything.
A non-zero failed= on the verify block is now a REAL FINDING, ALWAYS. That is the opposite
of what the sentence above tells you to expect, and carrying the old expectation into an incident
is precisely how a broken instrument gets read as normal.
What survives, and it is the more important half: the installer’s documented cost was
paid, and the warning about re-running it is still correct and still binding. Only the “one
consequence is live right now” clause expired. The superseded text is kept rather than deleted
because it is the reason verify.yml and the rescue probe have the shape they have.
This configuration is deliberately not Ansible-managed in this phase. It lives in a vendor
dashboard and in on-host files with no git representation, so codifying it is a later phase’s work,
not something to improvise during an incident.
Worth knowing before you conclude there is no way in: wlan0 carries 192.168.218.224/24 on a
subnet independent of eth0, with its own default route (192.168.218.1, metric 600), and it is
reachable from the control node. That is a genuinely independent path involving neither the modem
nor tailscale.
Two caveats before treating it as the rescue path: it rides the house AP infrastructure, and
“reachable today” is not the same as “reachable while eth0 is being reconfigured”. Evaluating it
as a rescue path is 01-07’s decision, not an assumption to build on here.
Superseded 2026-07-31 (01-17). Two corrections. The metric is 300, not 600 — measured
default via 192.168.218.1 dev wlan0 proto dhcp src 192.168.218.224 metric 300. And wlan0 is
no longer a second, independent path held in reserve: with eth0 carrying no default route it
is the host’s primary default, and every off-subnet reply leaves through it. That is why
heimdall stays reachable at 192.168.40.20 despite eth0 having no gateway — 192.168.218.0/24
is a real routable VLAN, so replies get home. The paragraph’s caveats stand and now bite harder:
the path currently in use is the one that rides the house AP infrastructure.
Evaluated 2026-07-30 (01-07); both caveats settled by measurement. The first is affirmative:
wlan0 comes up unattended at boot — connection.autoconnect is yes, the profile is persisted at
/etc/NetworkManager/system-connections/preconfigured.nmconnection, and the journal shows it
associating inside the boot sequence rather than by hand afterwards. The second is negative, and
it is decisive: NetworkManager is the sole owner of wlan0’s WPA credentials (the PSK exists only
in that keyfile), of its supplicant configuration (wpa_supplicant runs in D-Bus mode —
-u -s -O … with no -c config and no -i interface — and wpa_supplicant@wlan0.service is
disabled/inactive, with no wpa_supplicant*.conf anywhere), and of its DHCP lease (no dhcpcd,
dhclient or udhcpc covers it). Take NetworkManager down and wlan0 goes down with eth0.
An insurance policy that lapses at the moment of the claim is not insurance. So wlan0 is a
genuine second path today and a genuine one after a reboot — but it is not a rescue path for a
NetworkManager removal, which is the only event it was being considered for.
Outcome: the NetworkManager → systemd-networkd migration is DEFERRED to Phase 6 / DNS-14.
heimdall still runs NetworkManager; systemd-networkd remains disabled and inactive, and
/etc/systemd/network/ still holds only 73-usb-net-by-mac.link and 99-default.link. Nothing on
the host’s network stack was changed by 01-07.
This is D-07 working as designed, not a failure. D-06 pulled the migration forward into Phase 1 on
the condition (D-07) that the out-of-band rescue path be proven first; D-07 states that if it cannot
be, the migration defers and Phase 1 still closes green on onboarding, verification and the doc
correction. What is unusual is why it deferred: the gate passed and the migration was still the
wrong move.
The rescue path that was proven is carried by the thing the migration removes. usb0 gets its
address, its gateway and its default route from NetworkManager’s internal DHCP client, driven by
an auto-generated in-memory profile:
Terminal window
$ nmcli device status
usb0 ethernet connected Wired connection 2
$ ls /etc/NetworkManager/system-connections/
preconfigured.nmconnection # <- "Wired connection 2" is NOT here; it is in-memory only
$ ls /var/lib/NetworkManager/ |grepusb0
internal-056db167-…-usb0.lease # <- NM's internal DHCP client holds the lease
$ nmcli -t -f IP4.ADDRESS,IP4.GATEWAY,IP4.ROUTE device show usb0
dhcpcd does not cover it and would not take over: its configuration is an allow-list,
allowinterfaces wwan*, and the modem enumerates as usb0. dhcpcd -4 -U usb0 returns nothing and
/var/lib/dhcpcd/ is empty. Remove NetworkManager and the bearer loses its address and its route —
the same trap that ruled wlan0 out, reproduced on the path that had just been proven.
D-10 says the modem is deliberately absent from systemd-networkd’s .network files, declared
unmanaged, so a broken eth0 configuration cannot take the rescue path down with it. That reasoning
was sound when ModemManager owned the modem and did its own bearer IP configuration. ModemManager
is now purged. Under the CORE stack the modem presents as an ordinary ECM ethernet device that needs
an external DHCP client, so applying D-10’s mechanism literally today — Unmanaged=yes, no
DHCP=, no [Route] — would leave the bearer with no DHCP client at all. The instruction that
exists to protect the rescue path would have killed it.
D-10’s rationale (failure-domain isolation) survives; its mechanism no longer delivers that
rationale and must be re-decided against the CORE stack. That is Phase 6’s work, and it is the
concrete thing Phase 6 now inherits instead of a ported configuration.
A proven, independent rescue path — CORE Remote Terminal over the cell bearer, which did not
exist when this phase began. That is the precondition D-08/D-09 wanted, and it is now real.
A measured statement of what the migration must carry, not a guess: three interfaces come off
NetworkManager together — eth0 (the thing being changed), wlan0 (needs its PSK relocated out
of the NM keyfile and wpa_supplicant@wlan0 enabled) and usb0 (needs a DHCP client and a
metric-500 default route). Migrating eth0 alone strands the other two.
A named ordering constraint: bring the rescue interfaces onto networkd first, in their own
change window, and prove they survive a reboot — then migrate eth0. Doing all three at once
puts every rescue path in flight during the change they are supposed to insure.
Two host-side reconciliations that the CORE install created and that must land before a
cutover can be certified: verify.yml’s mmcli-based SC#5 assert, and the gate script’s
mmcli-based rescue probe. Both LANDED 2026-07-31 (01-08) — pointer added 2026-07-31
(01-11) so a reader who stops at this bullet is not left with the outstanding reading.
Neither call site queries a modem daemon any more; the correction three subsections below
(Why the timing argued for deferral independently)
records what each was re-pointed at.
Even setting the NetworkManager coupling aside, the verification contract is currently red:
Corrected 2026-07-31 (01-08) — “currently red” was true on 2026-07-30 and is not true now.
Both of the Two host-side reconciliations listed just above have landed: verify.yml’s
SC#5 assert now reads the USB bus directly (2c7c:0125, 0403:6001, a non-zero ttyUSB* node
count) and the gate script’s rescue probe now reads the bearer’s own address plus off-modem
reachability. Neither calls a modem daemon. The instrument is green — the verify block
completes to its terminal roll-up with failed=0, and the full-role --check --diff run reports
ok=49 changed=0 failed=0 skipped=9. The cutover bar stated below — you cannot certify a
cutover with a red instrument — still stands and is now met. The migration itself remains
deferred to Phase 6 for the NetworkManager-coupling reason, which is untouched by any of this.
Identifier corrected 2026-07-31 (01-14). The modem conjunct in the paragraph above named
only the bare Quectel vendor prefix until 01-10 tightened SC#5 to the full VENDOR:PRODUCT pair,
in that same round. The pair is what the assert enforces today, and the rescue-path correction
earlier on this page — the one headed Corrected 2026-07-31 (01-11) — already states it. The
identifier is corrected in place rather than annotated-and-left, because it is a present-tense
claim about code that runs right now: the dated-block convention protects recorded readings,
and a wrong identifier on a page consulted mid-incident is precisely what that convention exists
to prevent. The superseded value is described rather than quoted, deliberately — the gate that
keeps this page honest asserts that exactly one distinct vendor-id token appears anywhere on it,
so re-quoting the old one here would turn a correct page red.
The ok=49 … skipped=9 reading above is a superseded READING, not a current expectation
(01-14, 2026-07-31). It was true when 01-08 measured it and it stays here unaltered for that
reason. The page’s current source for these numbers is the Expected counts table under
Verify below, re-measured after round 3’s last change. Read the table, not this line, when
deciding whether a run looks normal.
The transcript below is the superseded 2026-07-30 reading, kept verbatim. It is the measured
evidence that the instrument was red, and rewriting a recorded PLAY RECAP to match today
would be inventing evidence in the opposite direction.
TASK [heimdall-common : Query ModemManager for enumerated modems]
[ERROR]: … No such file or directory: b'mmcli' verify.yml:725
PLAY RECAP: ok=27 changed=0 unreachable=0 failed=1 skipped=2
One assert, for a known and accepted reason — but the plan’s own bar for a cutover is that the full
01-05 verification block is still green after it. You cannot certify a cutover with a red
instrument. Restoring the instrument comes first.
Every step below is read-only, safe to run repeatedly, and changes nothing.
Expected counts, RE-MEASURED 2026-07-31 (01-16, after gap-closure round 4’s last change).
These numbers are copied from a real run, not predicted. That distinction is the reason this note
exists: an earlier version of this page carried its counts as a prediction, the prediction was
wrong for a whole day, and a reader following it would have accepted a broken instrument as
normal. They moved from 01-14’s readings (46 / 4 and 61 / 13 — themselves moved from 01-11’s
36 / 4 and 50 / 11, and before that 01-08’s 35 / 3 and 49 / 9) because 01-15 added roughly
twenty tasks to the role: a per-criterion ENTER fact set immediately before each assert, three
named read asserts with their own pass keys (SC#1b-read, SC#2a-read, SC#5-read), and drift
rows for the capture tasks that previously carried failed_when: false with no failure handling.
Both readings below were taken here, on the live host, read-only, rather than carried forward —
01-15 measured them and 01-16 confirmed them with one further --check --diff run before
writing them into this table.
The ok= count moving is a READING, not a finding. What the phase actually constrains is
changed=0 failed=0; a count that changes when the role gains tasks is arithmetic. A count that
changes when nothing was added is the thing to look at.
Reading
Measured value
--tags heimdall-verify, ok= count
64
same run — changed= / failed= / unreachable= / skipped=
0 / 0 / 0 / 8
full-role --check --diff, ok= count
79
same run — changed= / failed= / skipped=
0 / 0 / 17
A non-zero failed= is now a real finding, always. From 2026-07-30 until 01-08 landed on
2026-07-31 this note said the opposite — it told you to expect one failing task, because
verify.yml:725 called the purged mmcli. That instruction was actively harmful once it outlived
its cause: it trained the reader to accept the exact signal that means the instrument is broken.
Worse, Ansible halts the play on task failure, so that single task was not “one red assert” — it
blinded four instruments downstream of it, including the /proc/stat btime reboot-detection
primitive and this block’s own terminal roll-up. The block now runs to its roll-up.
Step 4’s “until 01-07 proves the rescue path” also needs qualifying: 01-07did prove it,
and the timer is still disabled on purpose. See
DECISION=SKIP today is correct.
Dry-run the verify block:scripts/heimdall-playbook.sh --tags heimdall-verify --check --diff.
Expect exit 0 in about 20 seconds, with the ok= count at 64 and
changed=0 unreachable=0 failed=0 skipped=8, and a terminal roll-up naming every criterion with
its observed value. Two things are defects, not variations: a run reporting changed>0, and a
run that stops before the roll-up.
Run it for real:scripts/heimdall-playbook.sh --tags heimdall-verify.
Same counts. HARD criteria (SC#1b, SC#1c, SC#2a, SC#3a, SC#5) fail the run; WARN criteria
(SC#2b, SC#2c, SC#3b) print a WARNING:-prefixed line and the run still exits 0. Seeing
6 package(s) pending at WARN is normal — see above.
SC#3 became two criteria on 2026-07-31 (01-10), and the split is why the lists above
changed.SC#3a is HARD — the bootloader release-channel pin, which either exists and
matches the repo’s declared channel or does not; there is no honest middle reading of it.
SC#3b is WARN — bootloader image drift against the latest available image. It is WARN
because flashing is guarded by heimdall_allow_eeprom_flash, which is false in the role
defaults and restated false in host_vars, so a drifted image is a deliberate condition
and a HARD red for it would train you to ignore the criterion. The drift is still reported on
every run rather than suppressed, which is what makes the demotion honest. ROADMAP’s
Restated criteria block carries the contract half of this, dated SC#3 (D-30, 2026-07-31).
What a red HARD criterion looks like now — the output shape changed, and an operator who has
not seen it will misread it. The roll-up still prints. Criteria below the failure read
(not reached), and the roll-up carries a VERIFY BLOCK INCOMPLETE line naming what was not
evaluated. The run still exits non-zero. Say the trap plainly: the roll-up printing is not
the run passing. A roll-up carrying either (not reached) or VERIFY BLOCK INCOMPLETE is a
run that FAILED — read the failed= count and the exit status, not the presence of the block.
Confirm reachability without the agent:scripts/heimdall-adhoc.sh heimdall -m ansible.builtin.ping.
Expect SUCCESS with "ping": "pong". Because the wrapper forces IdentitiesOnly=yes and
IdentityAgent=none, this result can only have come from the Vault-held key.
Expect disabled then inactive until 01-07 proves the rescue path. Anything else means
someone granted the host unattended reboot authority.
Qualified 2026-07-31 (01-11): 01-07 DID prove the rescue path, and the timer is still
disabled — for a different and weaker reason. Read the expectation as unconditional today,
not as pending on 01-07. See
DECISION=SKIP today is correct.
If you SCRIPT this check, do not pipe it under set -o pipefail (added 2026-07-31 by 01-14;
measured by 01-12 against a host that was correct).systemctl is-enabled exits 1 for
disabled and is-active exits 3 for inactive — so for the expected answer the wrapper
reports the Ansible task FAILED and exits non-zero. Under pipefail the pipeline is then
non-zero even though the match succeeded, and the check reads RED on a host that is exactly
right. Capture the output with || true and assert on the anchored literals ^disabled$
and ^inactive$. The anchoring is load-bearing in the other direction: an unanchored enabled
matches disabled, so it would pass on a host that had just been granted reboot authority —
the same defect class one level down.
Exercise the gate in no-act mode — note the flag; it is not optional:
On a healthy host with nothing pending, expect exactly
DECISION=SKIP reason=no-reboot-pending. reason=rescue-path-down with modem has no bearer
is no longer a reason this script can produce — that wording described the ModemManager probe
01-08 replaced, and it is recorded here only so a reader who finds it in an old note knows it
is dead. The same decisions land in the journal under journalctl -t heimdall-gated-reboot.
That first gate short-circuits, so the run above never reaches the later gates. To exercise
the rescue-path gate and the no-act guard without mutating real reboot-pending state, override
the marker path and substitute an inert reboot command:
Expect reboot is pending, trigger=marker:…, then rescue path is up, detail=bearer interface usb0 carries …, then DECISION=DRY-RUN-WOULD-REBOOT all gates passed; no-act mode, not invoking the reboot command, and SCRIPT-RC=0 — preceded by the interlock line, added to this list
2026-07-31 by 01-16 because the script has always emitted it here and the list did not say
so. The command above sets a non-default marker, so the interlock speaks before gate 1 is
even evaluated and its line is the FIRST thing logged:
exercise override in effect: marker=/tmp/tmp.XXXXXXXX (default /run/reboot-required); forcing no-act mode (set HEIMDALL_GATED_REBOOT_ALLOW_REAL_REBOOT=1 to consent to a real run). That is
correct output rather than an anomaly, and an expected-output list that omits a line the script
always prints either trains a reader to treat a correct line as a fault or reads red the first
time somebody scripts the check as an exact match — the same shape as the pipefail trap
documented three steps earlier. Then confirm the guard held — the sentinel must be
absent and the boot time unchanged:
--dry-run is not optional in any of this, and the inert HEIMDALL_GATED_REBOOT_REBOOT_CMD
is a second, independent belt: with it set, the script cannot reboot the host however its
internal guards are arranged. Never exercise the gate without both.
Recorded rather than hidden, because each one is a place where something looks wired up and is not:
The gated-reboot unit has no OnFailure= notifier. It is unwired deliberately — heimdall
has no notify unit and no template in this repo renders one, and pointing OnFailure= at a
nonexistent unit would make the failure-notification path itself fail silently while looking
correct in review. The journal is the record until a real notifier exists.
Skip-Updates-On-Metered-Connections "true" is NetworkManager-based. It works today because
heimdall still runs NetworkManager. If 01-07 migrates the host to systemd-networkd this
protection goes silently inert — networkd exposes no metered property for u-u to read — while
the SIM sits on a 1 GB/month pool. That migration must re-establish the protection by other means
(keeping the bearer off the default route is the primary control); it must not read that line as
continuing cover.
mmcli is gone, and two repo-owned checks called it — RESOLVED 2026-07-31 by 01-08. The
original observation, kept because it is why this code has the shape it has: the CORE installer
purged modemmanager; verify.yml’s SC#5 assert (mmcli -L) failed the verify run, and the
gated-reboot script’s rescue probe (for tool in mmcli jq) failed closed and reported
rescue-path-downwhile the rescue path was up. Both looked like host faults and were not.
Both call sites are now re-pointed. SC#5 reads the USB bus directly — the Quectel
VENDOR:PRODUCT pair 2c7c:0125, the FT232 id 0403:6001, and a non-zero /dev/ttyUSB* node
count — with the daemon query deleted rather than guarded, because the inference it supported
required the daemon to own the modem, which it no longer does. The rescue probe reads the
bearer interface’s own global IPv4 address plus an off-modem target answering through that
interface. Numeric USB identifiers carry the bar rather than a model string, because the two
sources disagreed — mmcli said EG25, lsusb says EC25, same physical device. Added 2026-07-30
by 01-07; resolved by 01-08. Identifier corrected 2026-07-31 (01-14): this entry named
the bare Quectel vendor prefix until 01-10 tightened SC#5 to the full pair in the same round.
Corrected in place because it describes the assert that runs today; the old value is described
rather than re-quoted, so the page keeps exactly one distinct vendor-id token.
Two daemons write /etc/resolv.conf.core_manager invokes sudo tee -a /etc/resolv.conf
roughly every 33 seconds (57 journal entries in 20 minutes), while the file is tailscale-generated
and carries tailscale’s CHANGES WILL BE OVERWRITTEN banner. Tailscale is winning, so the file is
clean today — but one side is appending, so if tailscale ever stops rewriting it the file grows
without bound. On the DNS milestone’s own rescue host this is worth watching. Added 2026-07-30 by
01-07; not fixed here.
Skip-Updates-On-Metered-Connections is still live — recorded because the bullet above
predicts its loss. The migration deferred, NetworkManager stays, so the metered check keeps
working. It becomes inert only when Phase 6 performs the migration, and that is now Phase 6’s
problem to solve rather than an accepted regression.
lefthook.yaml’s lint-ansible hook lints nothing. It passes repo-root-relative
{staged_files} after a cd ansible, so every path resolves as ansible/ansible/… and fails to
load. It fails loudly rather than silently, but it provides zero coverage; run ansible-lint
directly from ansible/ instead. Pre-existing and not fixed by this phase.
The reboot marker path is overridable through an environment variable.HEIMDALL_GATED_REBOOT_MARKER is what makes the gate exercisable without mutating real
reboot-pending state on the host — that is the point, and it is what produced the first runtime
observation that the no-act guard actually holds. It cuts the other way too: a root caller can
open the first gate deliberately, so the script’s first decision is only as trustworthy as the
environment it was invoked in. The bound on that is the service unit, which carries no
Environment= or EnvironmentFile= directives — so the scheduled path always reads the real
/run/reboot-required and cannot be steered this way. Added 2026-07-31 by 01-09.
Qualified 2026-07-31 by 01-11: that bound covered the SCHEDULED path only — the timer — and
said nothing about the supervised-exercise path, which is the only path anyone actually invokes.
That second half is now bounded too. Corrected 2026-07-31 by 01-16: this sentence named two
inputs until today, which is the enumeration whose incompleteness was the round-2 Critical. The
rule the code implements is that a non-default value for ANY input to either gate forces no-act
mode — HEIMDALL_GATED_REBOOT_MARKER, HEIMDALL_GATED_REBOOT_BEARER_IFACE,
HEIMDALL_GATED_REBOOT_PROBE_TARGET or HEIMDALL_GATED_REBOOT_PROBE_TIMEOUT — unless
HEIMDALL_GATED_REBOOT_ALLOW_REAL_REBOOT=1 is also set. Stated as a rule rather than as a list,
because a list is how the probe target was missed once already — and bounded to those four gate
inputs, because process-environment inputs such as PATH and IFS are not interlocked (WR-13,
open, root-only, scheduled path unaffected). The original observation
stands; what changed is that opening the first gate deliberately no longer also opens the last
one.
A permanently-closed rescue gate is indistinguishable in every operator-visible surface from
one that works. Both report DECISION=SKIP and both exit 0; a gate that will never fire again
looks exactly like a gate on a healthy host with nothing pending. Two reasons compound it.
ansible/inventory/host_vars/heimdall.yml states that a wrong heimdall_bearer_interface
declaration fails closed — safe, and therefore silent. And the service unit deliberately
wires no OnFailure= notifier, because heimdall has no notify unit and pointing at a
nonexistent one would make the notification path itself fail silently while looking correct in
review. What this round changed and did not change, stated exactly: ping -c 3 -i 0.3 replaces
ping -c 1, so one dropped packet over a bearer measured at 167–292 ms RTT no longer closes the
window. That lowers the false-negative RATE. It creates no alarm. The journal
(journalctl -t heimdall-gated-reboot) remains the only detection surface, and a real one — an
alert on consecutive rescue-path-down decisions, or on the timer never reaching a reboot — is
Phase 6’s natural home. Added 2026-07-31 by 01-11.
heimdall-gated-reboot.service’s After=network-online.target does NOT establish that the
bearer interface is settled.usb0’s address comes from an auto-generated in-memory
NetworkManager profile (“Wired connection 2”) created after the device appears, and
NetworkManager-wait-online does not wait on it — it waits on the profiles it knows about at
startup. So the unit can run before the bearer has an address and the rescue-path gate can read
it as down for a boot-ordering reason rather than a rescue-path one. The consequence is bounded
rather than open-ended: the gate SKIPs, which is the safe direction, and the timer is
Persistent=true and fires daily, so the cost is one window. Worth knowing before reading a
single rescue-path-down at boot as a modem fault. Added 2026-07-31 by 01-11.
yamllint cannot fail for any ansible/** path..yamllint.yaml ignores that tree, and
yamllint exits 0 on an explicitly-passed ignored file regardless of its contents. Intentional
repo configuration — but a green yamllint on those paths is not evidence of clean YAML.
ansible-lint and yamlfmt -lint are the load-bearing gates there.
Measured 2026-07-31 (01-13), recorded here by 01-14: it is not merely that yamllint
tolerates those files — it considers zero of them. yamllint -c .yamllint.yaml --list-files ansible/roles/heimdall-common/ prints nothing at all, and the ordinary run exits 0 in silence.
The cause is one ignore: entry whose own comment says “Skip Ansible files (use ansible-lint
instead — different indentation conventions)”. The config is correct by design; a gate that
runs yamllint over an ansible/ path is the thing that is wrong. The remedy is to drop that
step, not to unignore the path — unignoring would invert the dependency and start enforcing
the wrong indentation convention on the whole tree. Nothing is lost by dropping it:
ansible-lint ansible/roles/heimdall-common/ exits 0 with no findings and is what actually
reads those files. Say the consequence plainly, because it reaches backwards: every gate in
this phase that carried a yamllint step over an ansible/ path has been vacuously green since
the config was written, so a prior green reading on one of those gates carried less evidence
than it appeared to. The other half of each of those gates — ansible-lint, --syntax-check,
the content assertions — did the work.
Cross-run detection depends on an ansible fact cache that is control-node-local and
gitignored, and no fix removes that. Two verify readings — the boot epoch behind “no reboot
since the last recorded reading” and the unattended-upgrades effective-origins outcome behind
“this control has now gone unverified twice running” — are cacheable facts, so they compare
this run against the previous converge from this checkout on this machine, not against any
shared history. ansible/ansible.cfg sets fact_caching=jsonfile with
fact_caching_connection=.ansible_facts_cache, a relative, gitignored directory: a second
operator, a different clone and any CI runner all start empty and honestly report that they have
no prior reading. The wrappers cd into ansible/ before invoking ansible-playbook, which is
what keeps the ordinary path pointed at one store rather than at a new one per working directory.
Cross-run detection is therefore best-effort and machine-local — treat a first-run “no prior
reading” as absence of evidence, not as evidence of absence.What 01-15 fixed, at its actual scope (2026-07-31, recorded here by 01-16): the cache
entries expire after 86400 seconds by default, which is shorter than this host’s real converge
cadence, so both detectors were unreachable — the second-consecutive-miss branch could never
compare against anything. The lifetime is now pinned with an ANSIBLE_CACHE_PLUGIN_TIMEOUT=0
prefix on the two heimdall wrappers, scripts/heimdall-playbook.sh and
scripts/heimdall-adhoc.sh — not repo-wide: ansible/ansible.cfg is untouched and every
other host in the inventory keeps the 86400 default, because five plays here run
gather_facts: false and a never-expiring repo-wide cache would feed them stale facts. So:
within a wrapper-driven run the expiry no longer drops the cross-run facts and the detectors
can fire; outside one, nothing changed. That is sufficient rather than lucky because
ansible/CLAUDE.md mandates the wrappers for heimdall and the mandate has teeth — the automation
key lives in Vault and not on disk, so a bare ansible-playbook against heimdall fails SSH auth
outright.
And the surprising half, which is why the ad-hoc wrapper carries the prefix too.01-15
measured, on a backdated copy of the real cache, that a bare
scripts/heimdall-adhoc.sh heimdall -m ansible.builtin.ping — a read-only diagnostic that
changes nothing on the host — destroys both cross-run facts if it runs unprefixed more than a
day after the last converge. An expired read raises inside the cache plugin, so ansible’s fact
write becomes a replace instead of a merge and the stored payload collapsed from 114 keys to
1 (just the newly-discovered interpreter). With the prefix, both facts survived. Nothing in any
procedure on this page makes that visible, and the failure it prevents is silent — which is
exactly why it is recorded here.