Skip to content

Router Boot Ordering

How Docker and its services come up on the Firewalla router, why they are gated on /extdata, and how to recover when they don’t start.

Docker must not start unless /extdata is mounted, and nothing may block the router from booting. Docker’s data-root is /extdata/dockerdata; a dockerd started before the USB disk is mounted creates that directory on the root filesystem, containers come up against empty state, and the later mount shadows it — a degraded outage that looks like every containerized service losing its data at once.

Docker can be asked to start two ways at boot:

  1. Socket activationdocker.service is disabled, but docker.socket is enabled, and docker.service carries TriggeredBy=docker.socket. The socket brings dockerd up early in the boot transaction — before FireMain has run any post_main.d hook, so before /extdata can be mounted.
  2. The boot hookpost_main.d/0050-start-docker.sh, after 0000-mount-extdata.sh has mounted the disk.

Corrected 2026-08-10. This section previously named the first path as a systemd pull-in — the enabled docker-compose@* units declaring Requires=docker.service and dragging Docker into the boot transaction at multi-user.target. That is not what happens. systemctl show docker -p TriggeredBy returns docker.socket, and on the measured boot dockerd went active at 13:46:19 while all four compose units activated after it (13:46:52 / 13:47:19 / 13:47:51 / 13:48:54). The compose units still carry Requires=docker.service, so they would pull Docker in were it not already up; they are simply not what starts it. The distinction matters because it changes when the start happens — the socket fires far earlier than multi-user.target, which is why the gate loses the race described below.

Both paths pass through one gate: a systemd drop-in at /etc/systemd/system/docker.service.d/10-extdata-gate.conf sets

[Unit]
ConditionPathIsMountPoint=/extdata

A failed condition skips the unit rather than failing it, so boot is never blocked. The condition is re-evaluated on every start attempt; because the compose units retry on a Restart=always loop and each retry re-queues Docker’s start job, the entire stack converges on its own within seconds of the mount appearing — no hook re-kick required.

🚨 The gate does not protect the socket-activation path on an ordinary boot, and this page used to claim it closed “that path and every other start path”. It does not. The drop-in lives in /etc, which is an overlay on a 384 M tmpfs upperdir and is wiped every boot (see the measured note below). On a cold boot the file therefore does not exist at the moment socket activation starts dockerd; 0050-start-docker.sh writes it later. A condition that is not on disk cannot refuse anything.

So the gate protects the manual and post-hook start paths, and nothing else. The mitigation that is robust to boot ordering is create_host_path: false on every /extdata-sourced bind — it lives on /extdata itself and does not care what started dockerd or when.

ConditionPathIsMountPoint is used instead of RequiresMountsFor because /extdata has no fstab entry: a post_main.d hook mounts it, so there is no mount unit systemd could start, and a requirement on an unstartable mount unit would fail the boot transaction.

Measured, 2026-08-05 — /etc does not persist, and that includes enable

Section titled “Measured, 2026-08-05 — /etc does not persist, and that includes enable”

This was previously an assumption. A deliberate reboot (plan 01-08) measured it: / is an overlay whose upperdir is tmpfs (tmpfs-root on /media/root-rw, 384 M), while /home’s overlay upperdir is a real ext4 partition (/dev/mmcblk0p6). That asymmetry is exactly why post_main.d hooks survive a reboot and everything they write under /etc does not.

The consequence that surprises people: systemctl enable does not persist either. The enablement symlink lives in /etc/systemd/system/multi-user.target.wants/, so it is volatile too, and each compose instance is re-enabled on every boot by its own hook. Disabling a hook therefore does not merely skip a start — the unit is not in the boot transaction at all. Measured directly: with z0225-start-otel-collector.sh renamed aside, docker-compose@otel-collector came back is-enabled: disabled, inactive/dead, NRestarts=0, with zero journal entries for the boot.

Anything Ansible deploys into /etc that is not re-asserted by a hook is lost on every boot. Known instance: docker-compose@.service.d/pull-policy.conf and its start-with-pull-policy.sh (role router-common), which no hook re-asserts — see the phase-01 deferred-items.md D6.

Those two facts — a socket that brings dockerd up early, and an /etc that does not survive the boot — met on 2026-08-10.

Measured, 2026-08-10 — the gate lost the race, and Vault stayed sealed for twelve days

Section titled “Measured, 2026-08-10 — the gate lost the race, and Vault stayed sealed for twelve days”

The boot trace that turned the section above from a claim into a measurement. Postmortem: engram w36znjkyp4.

13:46:17 docker.service condition evaluated
13:46:19 dockerd ACTIVE, NRestarts=0 <- 27 s before the mount
13:46:45 post_main.d begins
13:46:46 /extdata mount successful
13:46:47 0050 writes the gate — and logs "Docker is already running", exits
13:47:19 0200-start-vault-unseal.sh: "SKIP: docker not running"

Read it in order and every step is individually reasonable:

  1. Socket activation started dockerd before /extdata was mounted and before the gate drop-in existed. With /etc wiped it had read neither daemon.json nor the drop-in.
  2. Docker fabricated the missing bind sources as root-owned 0755 stubs beneath the empty mountpoint — short-syntax binds create their source unconditionally, by documented design.
  3. vault-unseal 0.7.0 fataled on permissions of "/etc/vault-unseal.yaml" are insecure: -rwxr-xr-x — against the stub. The real file, on the unmounted device, was 0600 the whole time. dockerd gave up at RestartCount=9.
  4. 0050 wrote the gate at 13:46:47 and then early-returned on “Docker is already running”, so the gate it had just written was never evaluated.
  5. The one path that could have self-healed this — 0200-start-vault-unseal.sh at 13:47:19 — skipped on a false-negative daemon probe (docker info returning non-zero while docker.service was active). The same skip line appears at the 2026-08-05 09:58, 10:28 and 14:05 boots, so it is reproducible, not a one-off.

Consequence: Vault stayed sealed from 2026-08-04 to 2026-08-16. Every ExternalSecret stalled, and nothing alerted.

What shipped in response (all repo-side; converge is a separate operator action):

  • create_host_path: false on every /extdata-sourced bind in all four router compose roles — the only mitigation robust to boot ordering, because it does not have to exist in /etc at dockerd’s start. Enforced as a standing invariant by tools/dns-role-gates/tests/test_router_extdata_binds.py.
  • 0050-start-docker.sh now verifies the running daemon’s effective data-root instead of accepting “already running”, and restarts a pre-gate dockerd under the gate. An unreadable data-root logs WARN and leaves the daemon alone — this is the shared start path for four compose instances, so restarting on ambiguous evidence would cost router DNS and all telemetry at once.
  • The false-negative probe is gone from every hook, replaced by systemctl is-active --quiet docker.

router-hosts is the highest-consequence bind in the sweep (engram nkn5760c06). It serves the estate’s authoritative internal DNS, and a pre-mount start hands it a fabricated empty data/ directory and an absent config/server.toml. It then answers authoritatively with nothing while every surface reports a running container — a silent wrong answer rather than a visible failure. With the flag it refuses to start instead.

Nothing in /etc is assumed to persist across a reboot. The Firewalla root is an overlay (0000-a-remount-root.sh resizes /media/root-rw), so the boot path never relies on files Ansible wrote there. The 0050 hook is self-sufficient: on every boot, before starting anything, it re-writes /etc/docker/daemon.json and the mount-gate drop-in and runs systemctl daemon-reload — both files are Jinja-included from the same role templates Ansible deploys, so the two paths cannot drift. Ansible’s direct deploy exists only for immediate effect at converge time. If /etc happens to survive a given reboot, the gate is additionally present during the early systemd transaction, before hooks run; if it does not, nothing is enabled in that window either, so Docker cannot start before the hook re-asserts the gate.

Hooks in /home/pi/.firewalla/config/post_main.d/ run in filename order. Exactly one hook waits; everything downstream skips fast so the chain always completes:

Hook Behavior when /extdata is absent
0000-mount-extdata.sh Waits up to 60s (bounded) for the USB disk to enumerate, then exits 2. The only hook that waits.
0050-start-docker.sh Re-asserts daemon.json + the mount gate + daemon-reload unconditionally (self-sufficient against a wiped /etc), then skips (exit 0) immediately, logging SKIP. When the mount is present it clears a latched docker.service (reset-failed) before starting. Since 2026-08-16 it no longer accepts an already-running dockerd: it reads the running daemon’s effective data-root (docker info --format '{{.DockerRootDir}}', three attempts 2 s apart) and restarts docker under the gate if that root is not /extdata/dockerdata. An unreadable data-root logs WARN and leaves the daemon running — never a restart on ambiguous evidence, because this is the shared start path for all four compose instances.
0150-start-tailscale.sh, z0100-start-router-hosts.sh, 0200-start-vault-unseal.sh, z0225-start-otel-collector.sh Skip (exit 0) when /extdata is unmounted or docker is not active, logging SKIP to /var/log/postmain.log. All four changed together on 2026-08-16: the readiness test was docker info, which returned non-zero at 13:47:19 on 2026-08-10 while docker.service was active and skipped the outage’s one self-healing path. It is now systemctl is-active --quiet docker, the same signal 0050 uses. No copy of the old probe survives in any role template.
z0225-start-otel-collector.sh (additional behaviour) Re-asserts the collector’s two per-instance drop-ins (10-ghcr-auth.conf, 20-start-limit.conf) + daemon-reload unconditionally, before the skip branches — /etc does not persist and both are load-bearing. When prerequisites are met it clears a latched docker-compose@otel-collector (reset-failed) before enabling it.

✅ RESOLVED — 0050-start-docker.sh heredocs were unterminated

Section titled “✅ RESOLVED — 0050-start-docker.sh heredocs were unterminated”

Found 2026-08-04 (plan 01-08). Fixed the same day in commit 3623e6823, outside that plan’s scope — the file belongs to PR #1787 and 01-08 lists it as read-do-not-edit. Kept here because the failure mode is silent and the fix is one blank line that looks like whitespace noise.

Both self-assertion heredocs rendered with the delimiter glued to the last content line, so it never matched at the start of a line:

Terminal window
$ sudo grep -n 'DAEMON_JSON_EOF\|GATE_EOF' 0050-start-docker.sh | cat -A
54:}DAEMON_JSON_EOF$ # before
85:ConditionPathIsMountPoint=/extdataGATE_EOF$ # before
67:DAEMON_JSON_EOF$ # after — alone on its line
99:GATE_EOF$ # after — alone on its line

Cause: Jinja strips an included template’s trailing newline (keep_trailing_newline defaults false) and Ansible’s trim_blocks defaults true, eating the newline after %}. Two independent strippings compose, so {% include %} immediately followed by the delimiter emits them on one line. bash then reads the heredoc to end of file.

Consequence had it run: /etc/docker/daemon.json would receive the entire remainder of the script — invalid JSON — and every command below it would never run: no mount-gate drop-in, no ipsets, no reset-failed docker.service, no systemctl start docker. A dockerd started against that file fails on its config, taking down every container on the router, including docker-compose@router-hosts (authoritative internal DNS) and vault-unseal.

Why nothing broke: the #1787 version of this hook never ran — the last docker-start entries in /var/log/postmain.log were from 2026-08-02 in the pre-#1787 wording (ERROR: /extdata is not mounted, not SKIP:). The corruption would have fired the first time the hook executed, which would have been the next reboot.

Why no gate caught it: an unterminated heredoc makes bash exit 0. The only signal is a stderr warning, here-document delimited by end-of-file. There is no failure status for a hook wrapper, a postmain.log outcome line, or a systemd unit to branch on. bash -n against the deployed file was the check that discriminated — it now returns clean.

The fix and how to keep it: a blank line before each delimiter, restoring exactly the newline Jinja removed — the same idiom z0225-start-otel-collector.sh already carries. That blank line is load-bearing; removing it as whitespace cleanup re-arms the outage. After any change to a heredoc whose body is a Jinja include, verify with cat -A that each delimiter sits alone on its own line, and with bash -n that no here-document warning is emitted. A sweep of all 76 tracked .j2 templates on 2026-08-04 found no other instance of this shape.

Skipping (exit 0) rather than failing is deliberate: the hard gate lives in systemd, hook exit codes therefore carry no safety load, and a non-zero exit must never risk aborting whatever runs the hook chain.

The create_host_path sweep — and the two binds it deliberately does not protect

Section titled “The create_host_path sweep — and the two binds it deliberately does not protect”

Every /extdata-sourced bind in the three router compose roles (router-vault-unseal, router-tailscale, otel-collector) carries bind: {create_host_path: false}, enforced by set equality in tools/dns-role-gates/tests/test_router_extdata_binds.py. Short syntax cannot express the flag — Compose creates a missing short-syntax bind source unconditionally — so each such bind is long syntax.

Binds not under /extdata (/proc, /sys, /, /dev/net/tun, /etc/machine-id, the container-runtime socket, /run/log/journal, and the unbound include dir under /home/pi/.firewalla) stay short syntax on purpose. They exist before the mount and before dockerd, so flagging them protects nothing and would teach the next reader that the flag is a house style rather than a mount-race mitigation.

Two exceptions are recorded here so each reads as a ruling with a reason, not as something that fell off.

Residual 1 — /host/extdata carries the flag as uniformity, not protection

Section titled “Residual 1 — /host/extdata carries the flag as uniformity, not protection”

The collector’s filesystem-scraper mount (/extdata:/host/extdata:ro) is flagged, but the flag cannot refuse anything there: /extdata is the mountpoint directory, and it exists on the root overlay whether or not the device is mounted (verified against the live router). There is nothing for Docker to fabricate.

It is set anyway so the file carries one rule rather than one rule and an exception. Do not read it as a guarantee that the scraper sees real /extdata — when the device is unmounted the scraper sees an empty directory and reports the root filesystem’s numbers under that mount point.

Residual 2 — the diag emitter’s certs/client.crt is deliberately unflagged

Section titled “Residual 2 — the diag emitter’s certs/client.crt is deliberately unflagged”

This is the one /extdata bind in the repository without the flag. It is named as a closed one-element list (EXEMPT_SOURCES) in the gate, so any other unflagged /extdata bind still turns the gate red, and so does this one if it is ever deleted or ever gains the flag.

certs/client.crt is written at runtime by vault-agent’s split-certs.sh; no Ansible task creates it. The natural reading is that the sidecar’s depends_on: otel-collector: {condition: service_healthy} protects the bind. It does notdepends_on gates start, not create, and bind-source validation is daemon-side at container create. Measured on a fixture of exactly this shape:

Terminal window
Container writer-1 Created <- dependency created, NOT started, NOT healthy
Container sidecar-1 Creating
Error response from daemon: invalid mount config for type "bind":
bind source path does not exist: .../certs/client.crt

The whole compose up aborts, so the writer never starts, so client.crt is never written, so the next attempt fails identically. Flagging this bind would deadlock a first converge and take vault-agent and the collector down with it — strictly worse than the defect it prevents. Measured on Compose v5.3.1; this router runs v5.0.1, the same major line.

Accepted cost, stated plainly: on an unmounted /extdata, Docker fabricates an empty root-owned stub at that path and the cert-expiry gauge goes wrong or silent. One metric, bounded.

Rejected alternative — this is the fix a future reader will reach for first. Binding the certs/ directory instead of the single file would make the flag safe, because the directory is Ansible-created. It is rejected because it violates T-QT-03: that directory also holds client.key, the estate’s edge-device mTLS private key, and the emitter reads one file and only its not-after field. Do not “simplify” the bind to the directory.

Scenario Result
Disk present, enumerates within 60s Everything starts normally, at most a few seconds later than before.
Disk absent/dead Boot completes; router routes traffic. Docker stays down (condition skip). Compose units churn-retry harmlessly. Telemetry liveness alerts fire on collector absence.
Disk appears late (after the 60s window) Mount it (run the 0000 hook or sudo mount /dev/disk/by-label/firewalla_extdat /extdata); the compose-unit retry loop pulls Docker up on its own, or run the 0050 hook to start it immediately.

vault-unseal has no systemd unit — only its hook and a restart: unless-stopped container policy. That still recovers automatically: the container is registered in the persisted data-root, so dockerd restarts it whenever Docker itself comes up. Run the vault-unseal hook manually only if the container was explicitly stopped or removed.

Terminal window
# Mount manually (or re-run the hook)
sudo /home/pi/.firewalla/config/post_main.d/0000-mount-extdata.sh
# Start docker after a late mount (also clears latched state)
sudo /home/pi/.firewalla/config/post_main.d/0050-start-docker.sh
# If a compose unit latched its start limit.
# SHIPPED TODAY for docker-compose@otel-collector (900s window / burst 5, plan
# 01-07). The other instances still carry the vendor 10s/5, which cannot engage
# at RestartSec=5 and so never latches.
# Record the state BEFORE clearing — reset-failed destroys the only local
# evidence of what failed:
sudo systemctl show -p ActiveState -p NRestarts -p Result docker-compose@<name>
sudo journalctl -u docker-compose@<name> | grep 'Start request repeated too quickly'
sudo systemctl reset-failed docker-compose@<name>
sudo systemctl start docker-compose@<name>
# For otel-collector specifically, its own hook now does all of the above
# (re-assert drop-ins -> reset-failed -> compose up -> enable):
sudo /home/pi/.firewalla/config/post_main.d/z0225-start-otel-collector.sh

Recognising a latch on this box. systemd 249.11 does not set Result=start-limit-hit — measured, 0 occurrences. Result stays exit-code. The latch is ActiveState=failed + NRestarts pinned at StartLimitBurst and frozen across further explicit start requests + the journal line Start request repeated too quickly. Do not branch on Result=.

Check /var/log/postmain.log first — every hook logs its decision (SKIP, ERROR, or success) with a timestamp.

Every managed post_main.d hook writes an unconditional timestamped begin line to /var/log/postmain.log and an outcome line for both success and failure (hooks running under set -e also carry an ERR trap that logs the aborting line number). The log therefore doubles as a boot-order trace: the absence of the next hook’s begin line tells you exactly where the chain stopped, which FireMain’s fast-rotating journal cannot.

Log tags, in boot order: remount-root, dhcpcd6-duid, extdata-mount, ipv6-ula (wraps the python script’s output), dhcp6-vzw-fix-boot, docker-start, docker-compose-install, tailscale-boot, vault-unseal-boot, otel-collector-boot. The cron-driven dhcp6-vzw-fix payload logs only when it acts (it runs every minute; silent no-op is deliberate).

One VLAN’s dnsmasq can stop resolving .house — reboot clears it

Section titled “One VLAN’s dnsmasq can stop resolving .house — reboot clears it”

Seen 2026-08-05, resolved. The serving chain is client → dnsmasq → unbound, and it broke at the handoff for one VLAN only:

  • Queries arriving on bond0.3000 (LAN VLAN 3000, 192.168.20.1) were forwarded straight out eth1 to the WAN upstream and never reached unbound.
  • Queries arriving on bond0.3001 (WiFi, 192.168.24.1) were answered authoritatively from unbound’s local zone throughout.

The broken instance had also stopped applying dnsmasq_local — the statically pinned vault.fzymgc.house stopped answering on that VLAN. That is the sharpest single indicator, because a static address= never forwards anywhere: dnsmasq answers it locally or not at all.

It was runtime state, not configuration. Everything comparable between the two instances was identical — conf-dir set, resolv-file upstreams, per-network policy, per-device policy, device-group membership, per-interface dir structure. Same binary, same simultaneous restart, divergent behaviour.

systemctl restart firerouter_dns did NOT clear it. A full router reboot did. Reach for the reboot early if the symptom matches; repeated DNS-service restarts only waste a maintenance window.

Ruled out by direct test — don’t re-test these: Tailscale, the plan 01-08 reboots as a cause, the Ansible converge, unbound itself, the group-level Unbound setting, a 2021 group-scoped allow rule, and a LAN search domain that collided with the zone name.

Workaround while it is happening: another VLAN (WiFi) resolves .house correctly. Tracked as WINDOWS #44/#45, both now closed.

Never probe with vault.fzymgc.house. It is statically pinned in dnsmasq_local/00-bootstrap-vault.conf (address=/vault.fzymgc.house/…, plus vault-0/1/2), so it answers on every path whether or not unbound is reachable. Probing with it manufactured a false timeline and cost this investigation hours. Use llm.fzymgc.house — it exists only in unbound’s zone (measured 2026-09-09: absent from dnsmasq_local, answers without aa and with a counting TTL — the shape vault never has).

Read the flags before anything else:

Response Meaning
aa, ANSWER: 1 authoritative — the query reached unbound
ad, ANSWER: 0 + Cloudflare SOA a validating public resolver answered; the query went to the WAN
SERVFAIL / timeout something else entirely — not this defect

dig -b <src> cannot separate “source” from “instance”, because binding the source also changes the egress interface. A query addressed to 192.168.20.1 from the wireless interface still arrives on bond0.3001 and is handled by the working instance. Only a capture on the router discriminates:

Terminal window
sudo tcpdump -ni any "port 53 or port 8953" | grep llm.fzymgc

Look for the ingress interface, then whether the next packet goes to 127.0.0.1:8953 (unbound) or out eth1 (WAN).

Trap — the two dnsmasq config directories are not interchangeable. The generated config loads both, but only one is yours:

conf-dir=/home/pi/.firewalla/config/dnsmasq/ # global — FIREWALLA-MANAGED, curates files away
conf-dir=/home/pi/.firewalla/config/dnsmasq_local # user local — PUT YOUR CONFIG HERE

A file written to the managed directory disappears silently and never loads — no error, no warning. Confirm a rule actually loaded before believing it:

Terminal window
RESTART=$(systemctl show -p ActiveEnterTimestamp --value firerouter_dns)
sudo journalctl -u firerouter_dns --since "$RESTART" | grep "for domain fzymgc.house"

Scope the window with --since: the ACL log is high-volume enough that -n 400 scrolls past the startup lines and reads as absence.

Firewalla’s policy state lives in the box’s redis, not in this repo and not in any file — redis-cli hgetall policy:tag:<id>, policy:mac:<MAC>, policy:network:<uuid>, and tag:uid:<id> for group names. The files under config/dnsmasq/ are rendered from that state at each boot and policy push, so read redis for intent and the files only for what dnsmasq actually sees.

This gate protects the Docker layer. The per-unit boot robustness of the otel-collector composes with it rather than replacing it. As of plan 01-08 that per-unit work has landed, with one question still open:

Item Status
Bounded start limit (900 s / 5) Shipped (plan 01-07)
Latch cleared on boot by the unit’s own hook Shipped (plan 01-08) — but see the narrowing below; a latch does not survive a reboot
Per-instance drop-ins re-asserted on every boot Shipped (plan 01-08), verified through a real boot 2026-08-05
pull-policy.conf re-asserted on every boot OPEN — no hook re-asserts it; lost at every reboot, deferred-items.md D6
Readiness ExecStartPre= gate Decided against — see firewalla-otel-collector.md
Mount requirement at instance level Decided against, all three forms
Wants= dependency form on docker.service OPEN — not implementable as a drop-in, see below

What this discharges, stated narrowly. It does not show the stack boots without the hook mechanism — the mount, docker and compose-plugin hooks all still run and are all still load-bearing — and plan 01-08 deliberately adds a second dependence on the collector’s own hook, for re-asserting its drop-ins onto a non-persistent /etc.

Correction, measured 2026-08-05. An earlier revision of this note claimed the collector “no longer depends on its own hook for start ordering”. That is false and was refuted by the hook-aside reboot: with z0225 renamed away the unit did not start at all, because its enablement symlink is volatile (see the /etc note near the top of this page). The collector’s boot start is wholly produced by its own hook, not merely assisted by it.

Two further results from the same plan’s second reboot narrow what the per-unit work above actually buys.

The latch clear: what it does and does not buy you (measured 2026-08-05)

Section titled “The latch clear: what it does and does not buy you (measured 2026-08-05)”

The second reboot of plan 01-08 drove docker-compose@otel-collector into a genuine start-limit latch, held it latched, and rebooted with the hook enabled. Two findings, both counter to the intuition the step was written on:

  • A latch does not survive a reboot on this box. systemd’s failure state — ActiveState=failed, NRestarts, Result — is in-memory state held by PID 1 and a reboot discards it. Combined with the volatile enablement symlink, the unit crosses a reboot carrying no state at all. The hook’s reset-failed accordingly took its non-zero branch and logged reset-failed … returned non-zero (tolerated); reset-failed returns 1 against a unit that is not loaded, which is the normal case at boot here.
  • The step’s non-zero tolerance is the part that is load-bearing every boot. Had that line been strict (or the hook run under set -e), the hook would have exited before compose up and before enable --now, and the collector would not have come up on an ordinary boot. Do not “tidy” the tolerated-failure branch away.

The clear step’s real value stays where it was actually proven: recovering a unit that latched within a boot — after the hook has already run, or after any later failure — because Restart=always does not resurrect a unit past its start limit. That was proven by direct hook invocation against a real latch (43 s recovery) before either reboot was spent.

The same reboot settled the other hazard the start limit created.

The boot race: measured, and structurally unreachable for this unit

Section titled “The boot race: measured, and structurally unreachable for this unit”

The hazard modelled was a compose instance started at multi-user.target racing an unmounted /extdata, consuming its whole burst in ~30 s and latching for the rest of the boot. Measured at the 2026-08-05 hook-enabled reboot with the limiter genuinely installed (15min/5, re-asserted by the hook at 10:28:38 — before the unit’s only start at 10:29:18): NRestarts=0, no latch.

It cannot occur, and the reason is structural rather than lucky: the unit is never in the boot transaction. Its enablement symlink is created by its own hook one second before that hook’s enable --now, by which point the hook has already passed its mountpoint -q /extdata and docker-readiness gates (that second gate was docker info when this was measured; it is systemctl is-active --quiet docker since 2026-08-16 — the change makes the gate strictly harder to pass spuriously, so the closure below is unaffected). The unit issues zero start requests during the window the hazard describes.

This closure rests on /etc being volatile, not on design. If the enablement symlink were ever made persistent — a durable /etc, a systemd-preset, an image change — the hazard becomes reachable again and this note must be re-measured rather than carried forward.

Mechanism drift: the mount gate’s header no longer enumerates every instance

Section titled “Mechanism drift: the mount gate’s header no longer enumerates every instance”

extdata-gate.conf.j2 carries a shipped comment stating that the enabled docker-compose@* units pull docker into the boot transaction via Requires=docker.service. Plan 01-08 intended to move the collector instance to the optional Wants= form, which would have made that enumeration incomplete.

That change did not ship, because it is not implementable: systemd’s dependency parser has no empty-assignment reset, so a Requires= inherited from the shared base unit cannot be removed by a drop-in on any systemd version (measured on 249.11 and confirmed against upstream config_parse_unit_deps). The collector therefore still carries Requires=docker.service.

Superseded 2026-08-16, and by something larger than drift. This section used to end “and the gate’s header remains accurate as written”. The enumeration was fine; the premise underneath it was wrong. The 2026-08-10 boot trace showed that what starts dockerd at boot is socket activation (TriggeredBy=docker.socket), not the compose units’ Requires= — dockerd was active at 13:46:19 and all four compose units activated afterwards. The header has been corrected accordingly, so the question this section was tracking — whether the Requires= enumeration stays complete — is no longer the thing that determines whether docker starts early. It is retained because the dependency form still matters for the conclusion below, and because the supersession is itself the lesson: the enumeration was audited for two plans while the mechanism it enumerated was not what fired.

Recorded here rather than in #1787’s own file, which plan 01-08 must not touch. If the dependency form is ever changed — via a full per-instance unit file, the only remaining route — the gate’s enumeration becomes incomplete while its conclusion still holds, because either form pulls docker into the transaction and the two sibling instances would still carry the mandatory one. What would break the conclusion is all three instances moving off Requires= and nothing else pulling docker in — and note that even then the hook is not the only starter: docker.socket is enabled and starts dockerd on its own.