Firewalla OTel collector

Runbook for the OpenTelemetry collector that ships the Firewalla router’s host metrics and journal logs to ClickStack. This is the target every host-collector liveness alert message points at.

Between 2026-07-31 and 2026-08-02 this collector was dark for roughly 48 hours and nothing paged. The alerts described below exist so that cannot recur; this page exists so the page you receive is actionable at 03:00.

Overview

Property	Value
Host	Firewalla (capital F — the metric label is case-sensitive)
systemd unit	docker-compose@otel-collector.service
Compose instance	otel-collector (templated unit; instance name is the compose project)
Compose directory	/home/pi/.firewalla/run/docker/otel-collector
Persistent data	/extdata/otel-collector (see the mount warning below)
Container image	ghcr.io/fzymgc-house/otel-collector-firewalla, private, pinned to a short-commit tag
Ansible role	ansible/roles/otel-collector
Gateway endpoint	otel-gateway.fzymgc.house:443
Alert source of truth	argocd/app-configs/clickstack-alerts/bootstrap-script.yaml

The unit is a systemd template instance: docker-compose@.service takes the instance name after the @ and runs docker compose in that instance’s directory. Two sibling instances (docker-compose@vector and docker-compose@tailscale) run from the same template, so any change scoped to this collector must be made in a per-instance drop-in directory (/etc/systemd/system/docker-compose@otel-collector.service.d/) and never in the shared template.

Topology

hostmetrics receiver ─┐
                      ├─→ otel-collector (container) ─→ otel-gateway.fzymgc.house:443
journald receiver ────┘                                        │
                                                               ↓
                                                   ClickHouse otel_metrics_gauge
                                                               │
                                                               ↓
                                            HyperDX tile alert → Pushover

The collector authenticates to the gateway with a Vault-issued client certificate (renewed by a sidecar Vault agent) plus an ingest token. Both are provisioned by the Ansible role; neither is fetched at unit start.

The image and its pull credential

The collector image is not built on the appliance. It is built, provenance- attested and cosign-signed by .github/workflows/build-otel-collector-firewalla.yml, published to GHCR, and pulled by the unit. The on-box build path was removed in full rather than kept as a fallback: a second image source is precisely how the image went missing for 48 hours with nothing noticing.

Property	Value
Package	ghcr.io/fzymgc-house/otel-collector-firewalla
Visibility	private — confirmed by anonymous token probe (unauthenticated manifest fetch returns HTTP 403), not assumed from the platform default
Deployed tag	an immutable short-commit tag, pinned in ansible/roles/otel-collector/defaults/main.yml as otel_collector_firewalla_image_tag
Other tags	the version tag and floating latest are also published, reserved for local testing — never deploy a floating tag
Compose pull policy	pull_policy: missing
Credential	/extdata/otel-collector/docker/config.json, root:root, 0600, inside a 0700 root:root directory
Credential reaches Docker via	DOCKER_CONFIG, set by /etc/systemd/system/docker-compose@otel-collector.service.d/10-ghcr-auth.conf
Credential source	Vault secret/fzymgc-house/cluster/ghcr/pull-secret, fields username + password — the shared estate pull secret, also used by the NAS role and the cluster’s imagePullSecret

Three properties are load-bearing and easy to undo by accident:

The router never reads Vault for this. The Ansible task is delegate_to: localhost — the control node reads the secret and writes a plain file to the router. The router participates in the Vault unseal chain, so a design that fetched the credential at unit start could deadlock recovery against the very service the router helps bring up.

The drop-in is per-instance, not template-level. vector and tailscale read only the shared docker-compose@.service.d/ directory, have no DOCKER_CONFIG, and resolve registry credentials exactly as they always have, which is nowhere. Putting this credential in the shared template directory would hand a registry PAT to two unrelated services.

The compose service must never gain a build: key. Not even as an on-box fallback. docker compose returns a pull error only when the service has no build key; with one present, the same registry 401 is downgraded to a logged warning and docker compose pull exits 0 — silently recreating the exact absorbing failure this collector’s whole remediation exists to eliminate, while reading in review as a safety improvement. If a fallback is ever wanted, it is a shell branch that runs after a failed pull, never a compose key.

Rotating the pull credential

The credential is shared across the estate, so one Vault write plus three converges rotates it everywhere. Rotate in this order — write first, so no consumer is ever converged against a secret that no longer exists.

1. Mint a new GitHub PAT with read:packages scope only.

2. Write it to the shared path (both fields, even if only one changed):

   vault kv put -mount=secret fzymgc-house/cluster/ghcr/pull-secret \
     username='<github-user>' password='<new-PAT>'

3. Converge each consumer:

   # router (this collector)
   ansible-playbook -i inventory/hosts.yml router-playbook.yml --tags otel-collector
   # NAS collector
   scripts/nas-playbook.sh --tags nas-otel-collector
   # cluster imagePullSecret is ESO-driven; force a refresh if it has not rolled

4. Prove the router picked it up — a pull that actually contacts the registry, not a cached-image no-op:

   ssh pi@192.168.20.1
   sudo systemctl restart docker-compose@otel-collector
   sudo journalctl -u docker-compose@otel-collector -n 50 --no-pager

5. Revoke the old PAT only after step 4 succeeds.

Recorded expiry of the credential in use: 2027-04-01 04:00:00 UTC. Read on 2026-08-04 from the github-authentication-token-expiration response header returned by an authenticated call to https://api.github.com/ using the PAT stored at secret/fzymgc-house/cluster/ghcr/pull-secret (field password, user seanb4t). Nothing enforces this date — the date lives here and only here.

Expiry produces an authorization failure indistinguishable from an absent credential (see the failure table below) and is the likeliest future recurrence of the 48-hour outage. Rotate before 2027-04-01. To re-read the date after a rotation:

PAT=$(vault kv get -mount=secret -field=password fzymgc-house/cluster/ghcr/pull-secret)
curl -s -D - -o /dev/null -H "Authorization: Bearer ${PAT}" https://api.github.com/ \
  | rg -i 'github-authentication-token-expiration'

Because this PAT is shared with the k8s imagePullSecret and the nas-otel role, its expiry is not a Firewalla-local concern: the same date strands three consumers at once.

/extdata is mounted by a boot hook, not by fstab

The credential, the collector config and the mTLS certificates all live under /extdata, which is not in /etc/fstab. It is mounted by /home/pi/.firewalla/config/post_main.d/0000-mount-extdata.sh, a Firewalla boot hook. If that hook does not run or exits early, /extdata silently resolves to a directory on the small root overlay instead: every path above appears to exist, is empty, and every compose instance on the box fails to start with missing bind-mount sources. df -h /extdata showing overlayroot rather than a ~235 G /dev/sda1 is the tell. Confirm with:

findmnt /extdata            # must show /dev/sda1, ext4
sudo grep extdata-mount /var/log/postmain.log | tail -5

A boot where the hook logged nothing at all means it never reached its first log line — treat that as the fault, not as a collector problem. This coupling is why the boot-ordering work is tracked separately from the image work.

Query form

The host identity for this collector’s metrics is ResourceAttributes['host.name'], and only that:

SELECT count() FROM default.otel_metrics_gauge
WHERE ResourceAttributes['host.name'] = 'Firewalla'
  AND MetricName = 'system.memory.utilization'
  AND Attributes['state'] = 'used'
  AND TimeUnix >= now() - INTERVAL 1 HOUR

A healthy router returns about 120 rows per hour (a ~30 s emit cadence).

Trap — do not filter on ServiceName. Measured against a healthy router in an hour where the query above returned 120 rows, WHERE ServiceName = 'firewalla' returned 0: this carrier shipped an empty ServiceName for the whole of the retained history up to 2026-08-06, when a resource processor began setting service.name. Both facts are why the rule stands. Rows written before that change keep the empty value and rows written after carry firewalla, so a ServiceName filter silently splits this host’s history at the identity change — the same window answers a question differently depending on which side of it you land. An alert built on ServiceName against the older history reads a permanently-empty series on a live host, fires continuously, and gets muted, which is indistinguishable from having no alert at all. The trap is carrier-specific, not global: check it per carrier rather than assuming it either way.

host.name is also case-sensitive: the value is Firewalla, not firewalla.

Memory model — zram, zsmalloc and the real ceiling

Read this before reacting to a swap number on this box. On this appliance a rising swap figure means close to the opposite of what it means on an ordinary server, and the reflex reading will send you the wrong way.

Swap here is compressed RAM, not disk

All four swap devices are zram. A page swapped out on this box is not written to storage and has not left memory — it is compressed in place and held by zsmalloc, the kernel’s compressed-page allocator, in the very RAM the swap was supposed to relieve. Paging out buys you the compression ratio and nothing else.

That inverts the signal:

	disk-backed swap	zram swap (this box)
A page swapped out	leaves RAM	stays in RAM, compressed
Swap-in-use rises	after RAM is exhausted	before available RAM collapses
Therefore the metric is	a lagging indicator	a leading indicator

So on a normal host you look at swap to confirm a memory problem you already have. Here you look at swap to see one arriving. Non-zero and climbing swap usage is an early warning, not a post-mortem — and because zsmalloc’s arena grows inside the same 7.6 GiB, heavy swapping reduces the memory available to everything else rather than freeing it.

Implied capacity vs the effective ceiling

Measured on the appliance 2026-08-06T19:24:10Z (free -m):

	MiB	GiB
RAM total	7803	7.62
Swap total (4 × zram)	3899	3.81
Implied capacity (RAM + swap)	11702	11.4
Effective ceiling	7803	7.6

7.6 GiB is the real number. The implied 11.4 GiB double-counts, because the 3.81 GiB of “swap” is storage carved out of the same 7.62 GiB of RAM it claims to extend. Anything that sizes a workload against 11.4 GiB is sizing against memory that does not exist.

The four devices, and why they should agree

There are four devices — /dev/zram0 through /dev/zram3 — each 975 MiB, all at swap priority 5. Equal priority means the kernel stripes across them round-robin rather than filling one and moving on, so in normal operation the four should track each other closely.

Divergence between the devices is itself a signal. If one device carries materially more than its siblings, the round-robin assumption has broken — suspect a device that failed to initialise at boot, or one that was reset while the others stayed up. Check the spread before concluding anything about total swap pressure, because a single loaded device and four evenly loaded devices mean very different things at the same total.

At the time of writing all four report Used=0 in /proc/swaps, and per-device mm_stat reads 4096 74 12288 0 12288 0 0 0 0 on every one: 4 KiB of original data compressed to 74 bytes, while zsmalloc holds 12 KiB for it. At idle the allocator’s own bookkeeping dominates, which is why a small non-zero figure here is normal and only a trend is interesting.

Where the numbers live

Swap is carried by system.paging.usage (a non-monotonic Sum, so it lands in otel_metrics_sum) and system.paging.utilization (a Gauge, in otel_metrics_gauge) — both broken out per device, with state in {used, free}. Querying the wrong one of those two tables returns no rows against a perfectly healthy scraper. For the host identity to filter on, see Query form above; it is the same rule as everywhere else in this document and is not repeated here.

system.paging.operations and system.paging.faults are deliberately disabled — they are host aggregates rather than per-device, and nothing here asks a question they answer.

No alert is attached to any of this. These are diagnostic numbers, and whether any of them deserves a threshold is a decision to take on observed data rather than in advance.

Dashboards

Everything above is on one page: the HyperDX dashboard Firewalla memory diagnostics (tools/hyperdx/dashboards/firewalla-memory-diagnostics.json). Open it first during a memory incident on this box — it is built to answer two questions together: what is consuming memory and is the box stalling.

Tiles are listed by name. A rename breaks the reference here rather than silently pointing at nothing, which is the point of naming them:

Tile	The question it answers
zram compressed store per device	How much compressed data each of the four devices actually holds — and whether they have diverged from each other.
zram effective ceiling	The real ceiling, as one number. It is a constant by design; a flat line is correct, and a change means kernel or hardware, never operations.
Memory pressure stall (PSI some/full avg60)	Whether the box is stalling rather than merely full. some is the early warning; full is the livelock signature.
Swap in use	How many bytes are actually in compressed swap, per device.
Swap utilization	The same as a fraction, per device and per used/free state, so one saturating device is visible instead of averaged away.
Per-process memory (RSS)	Which named process is consuming the memory — the attribution question.
Host memory utilization (baseline)	The aggregate the estate already alerts on, carried as the contrast so the new picture is read against the familiar number.
Diag emitter source health	Whether the diagnostic emitter’s own sources (zram, psi, cert) are still parsing. Deliberately unalerted — this tile is where a degraded source is meant to be seen.

The dashboard is applied through an authenticated browser session, not by the deployment pipeline, so the repository JSON is an export of what was applied rather than a source of truth. Re-export after any change, or the two diverge silently. The query forms behind the tiles are owned by Query form and Where the numbers live above and are not restated here.

No alert reads any tile on this dashboard, and in particular none reads a per-process series. That is checked by resolving every live alert document to the dashboard and tile it points at, not by recalling that none was added.

Triage capture (Step 0)

Capture before you touch anything. Everything in Recovery below destroys evidence: restarting the unit rotates the container, clearing the failed state erases the only local record of how it failed, and the journal that would name the onset rolls over within hours under an error loop. One prior investigation on this box arrived to find the journal had already discarded the morning it needed.

Why this step is written the way it is. During the 2026-07-31 livelock the scarce resource was the process-creation call itself. sshd could not accept new sessions, D-Bus died with ENOTCONN, and systemctl — and even plain reboot — hung. Anything that spawns a process fails at exactly the moment you need it. So the block below runs entirely on shell builtins: no pipeline, no command substitution around an external reader, no awk, no journal tool. It is designed to be pasted into a shell you already have open, because during that failure you will not get a new one.

The diagnostic emitter cannot cover you here, and a gap in its series is a finding rather than an all-clear. The emitter forks to send: wget is a fork and an exec, once per datapoint. Every avoidable fork has been removed from it — timestamps come from $EPOCHSECONDS and printf %(…)T, both builtins — but the transport fork is irreducible, because bash has no builtin HTTP client. So during a fork-starved livelock the emitter goes dark, and a gap in firewalla.psi.* is evidence of the condition, not evidence against it. firewalla.psi.memory.full.avg60 is the metric that literally describes a livelock and it is precisely the metric that cannot be emitted during one. Do not read the gap as “no pressure”: read the Step-0 block below instead.

Paste this whole block. It needs bash (read -d '' and $EPOCHREALTIME are bash builtins); measured on this router, it forks nothing.

printf 'capture at epoch %s on host %s\n' "$EPOCHREALTIME" "$HOSTNAME"
IFS= read -r -d '' mi < /proc/meminfo; printf 'BEGIN meminfo\n%sEND meminfo\n' "$mi"
IFS= read -r -d '' pr < /proc/pressure/memory; printf 'BEGIN pressure\n%sEND pressure\n' "$pr"
for d in /sys/block/zram*; do IFS= read -r -d '' mm < "$d/mm_stat"; printf 'BEGIN zram %s\n%sEND zram\n' "$d" "$mm"; done
IFS= read -r -d '' sw < /proc/swaps; printf 'BEGIN swaps\n%sEND swaps\n' "$sw"

Do not “simplify” the reads to $(<file). That form is widely documented as a fork-free builtin read and it is not one here. Measured on this router on 2026-08-07 under strace -f -e trace=clone,clone3,fork,vfork,execve, an earlier revision of this block whose only substitutions were seven $(<file) reads produced seven clone() calls — one per occurrence. The block above, byte-for-byte, produces zero. read/mapfile with an input redirect is the form that actually survives a box that cannot fork.

Every reading is labelled, so the output can be pasted straight into an incident record and still be legible a week later.

Reading	What it tells you	Where to go next
meminfo	MemAvailable is the number that matters; MemFree is not. SwapCached, Shmem and the Slab lines say where the memory went	Memory model
pressure	some avg10 climbing means tasks are stalling on memory now; full non-zero means everything is stalled. total is monotonic microseconds since boot, so two captures minutes apart give you a rate	Memory model
zram per device	mm_stat is orig_data_size compr_data_size mem_used_total mem_limit mem_used_max same_pages pages_compacted huge_pages …. mem_used_total is RAM zsmalloc is holding — it is not freed memory	Memory model
swaps	Used per device, and whether all four are present at equal priority. One device carrying materially more than its siblings is itself a finding	Memory model

Do not do the arithmetic from memory: a rising swap figure on this appliance means close to the opposite of what it means on an ordinary server, and Memory model owns that reasoning, including why the effective ceiling is 7.6 GiB and not 11.4 GiB.

Then read the host identity rule in Query form before you query anything — it is the difference between a dark host and a wrong filter — and only then work through Recovery, which owns the verb order and the reason the two readouts there come first.

If the box will still let you in

The block above pays a real cost — builtins only, no filtering, no journal — to survive a box that cannot fork. On a healthy or merely degraded router, do not pay it. Run scripts/firewalla-capture-snapshot.sh from this repository instead: it captures the same four readings plus container states, unit state and a journal tail into one timestamped file, and it is free to fork because nothing is stopping it. Each artefact is explicit about which failure mode it serves; using the wrong one is not dangerous, only wasteful in one direction and useless in the other.

Reading the onset from telemetry — and its control

If the question is when did this host go quiet, the earliest retained point is not the answer until you have proved it is not the retention floor. Run both, in the same breath:

-- FILTERED: earliest retained point for this host
SELECT min(TimeUnix) FROM default.otel_metrics_gauge
WHERE ResourceAttributes['host.name'] = 'Firewalla';

-- CONTROL, unfiltered: earliest retained point for ANY host
SELECT min(TimeUnix) FROM default.otel_metrics_gauge;

If the two are equal, you are reading the store’s retention floor and have learned nothing about onset — the data simply does not go back further. Only a filtered minimum that is later than the global minimum is evidence about this host. The same pairing applies to the logs table, whose timestamp column is Timestamp (there is no TimestampTime).

No dead-man’s-switch belongs on this box

Do not add an on-host dead-man’s-switch, heartbeat timer or watchdog process to this router — not a systemd timer, not a cron entry, not a supervisor loop, and not as a “small improvement” alongside something else.

The reason is the failure mode above: the process-creation call was the scarce resource, so an on-box watcher dies in the same instant as the thing it is watching, and its silence is indistinguishable from a healthy quiet box. The timer-based freshness pattern used on the support host is correct there — that host was not the one failing — and wrong here.

The substitute already exists and is off-box: the presence alerts described in Alert lifecycle, evaluated in the cluster, which observe this router’s absence from outside it. If you are here because you want faster detection, change the alert interval; do not put a watcher on the patient.

Recovery

When Firewalla host collector MISSING (host scraper dark) fires, SSH to the router and run:

systemctl status docker-compose@otel-collector          # why is it not running?
journalctl -u docker-compose@otel-collector -n 200      # the actual cause
systemctl reset-failed docker-compose@otel-collector
systemctl restart docker-compose@otel-collector

reset-failed comes first, and skipping it is why a plain restart can look like it did nothing. systemd rate-limits restarts of a unit that has failed repeatedly. Once the unit trips its start limit it enters ActiveState=failed with NRestarts pinned at StartLimitBurst, and every subsequent systemctl restart is refused immediately — the unit never even attempts to start, and journalctl shows Start request repeated too quickly. rather than the original fault. (On this router’s systemd 249.11 the stored Result stays exit-code; it does not become start-limit-hit. See the latch.) reset-failed clears the accumulated failure counter so the following restart is actually attempted. Always pair the two verbs.

The two readouts above are not optional preamble — they are the reason the order is what it is. reset-failed erases the restart counter and the failed state, and those are the only local record of how the unit failed. Run systemctl status and journalctl and keep their output before you clear anything. See the bounded start limit and the latch for what a latched unit looks like and why its restart policy will not recover it on its own.

Confirm recovery by re-running the query form above and expecting a non-zero count, then watching the alert return to OK on its next evaluation. Do not re-sync the clickstack-alerts Application to “check” the alert — see Alert lifecycle below for why that destroys the evidence you are looking at.

Troubleshooting

The container is not running: image pull failures

The collector image is pulled from a private registry. Four failure modes present almost identically at the unit level and are told apart by the journal message, not by the exit code — the first two in particular are indistinguishable from exit status alone, so never branch on it.

Failure	Journal signature	Fix
Credential absent	pull access denied / authentication required with no prior auth attempt; DOCKER_CONFIG points at a directory with no config.json	Re-run the Ansible role so the per-instance DOCKER_CONFIG drop-in and config.json are written; confirm the drop-in is present under docker-compose@otel-collector.service.d/
Credential expired or revoked	401 Unauthorized returned after an auth request, or denied: denied on a token exchange	Rotate the registry token in Vault, re-run the role, then reset-failed + restart
Registry unreachable	dial tcp … i/o timeout, no such host, connection refused, or a TLS handshake error — no HTTP status at all	Network/DNS problem on the router, not an auth problem. Check the router’s resolver and WAN before touching credentials
Package flipped public	The pull succeeds with no credential present	Not an outage. Verify intent: a package that became public unexpectedly is a supply-chain signal, not a convenience

The distinction matters because an absent credential and an expired credential both leave the unit in exactly the same failed state with the same non-zero exit status. Reading the journal is the only way to know whether to re-run the role or rotate the token.

The unit refuses to start at all: the bounded start limit and the latch

Start request repeated too quickly. in the journal means systemd is rate-limiting, not that the collector failed this time. Run reset-failed first (see Recovery), then read the journal from before the limit was hit to find the original fault.

Do not go looking for Result=start-limit-hit on this router — it never appears. systemd documents that value, but the Firewalla runs systemd 249.11, and there the limiter refusing a start does not overwrite the stored result: Result keeps reporting the last execution outcome, exit-code. Measured during the 01-07 credential-missing test: the unit latched, three subsequent explicit systemctl start/restart attempts were all refused, and Result=exit-code throughout while the string start-limit-hit appeared zero times in the journal. Branching on Result to detect a latch would therefore never fire. Use the three signals in the next section instead.

Rebooting is the other reflex worth heading off here.

A reboot clears a latch, but not for the reason you might assume — and it is the slower fix. Measured 2026-08-05 (plan 01-08): the unit was driven to a genuine latch, held latched, and the router rebooted. The latch was gone afterwards because systemd’s failure state is in-memory and a reboot discards it, not because the boot hook cleared it. The hook’s reset-failed in fact logged returned non-zero (tolerated) — at that point in the boot the unit is not even loaded, and reset-failed returns 1 against a unit that is not loaded. What brings the collector back at boot is the hook enabling it, after re-asserting the drop-in that carries DOCKER_CONFIG so the private image can be pulled.

Practical consequence: reset-failed + restart is the correct fix and takes seconds; rebooting costs roughly three minutes of telemetry (86 s of host downtime plus the boot chain) and takes down internal DNS while it happens. Reboot only if you have a separate reason to.

What the bound is, and why the default was decorative

docker-compose@otel-collector carries a per-instance start-limit drop-in at /etc/systemd/system/docker-compose@otel-collector.service.d/20-start-limit.conf, rendered by the otel-collector Ansible role:

[Unit]
StartLimitIntervalSec=900
StartLimitBurst=5

The vendor default — a 10 second window with a burst of 5 — could never fire on this unit. The shared template sets RestartSec=5, so five starts are at least 25 seconds apart before a single second of actual start work is counted. No failure of any duration fits five starts into ten seconds. That is why the unit accumulated hundreds of automatic restarts in silence rather than latching, and it is the reason the window was widened rather than the burst lowered.

Sizing invariant, if you ever change these numbers:

StartLimitIntervalSec  >  StartLimitBurst x (RestartSec + failure_duration)

A wider window makes latching more likely, because more starts fit inside it — so size against the slowest failure, not the fastest:

Failure	One failing start	Minimum window
Credential absent / expired (fast 401)	~5 s	5 x (5 + 5) = 50 s
Registry unreachable (dial + TLS timeouts)	~120 s	5 x (5 + 120) = 625 s

900 s clears both, with roughly 44% headroom over the binding case.

What a latched unit looks like

Verbatim from the 01-07 test:

$ systemctl show -p ActiveState -p SubState -p Result -p NRestarts \
    docker-compose@otel-collector
ActiveState=failed
SubState=failed
Result=exit-code          # NOT start-limit-hit — see the note above
NRestarts=5               # == StartLimitBurst, and frozen there

$ journalctl -u docker-compose@otel-collector | tail -3
systemd[1]: docker-compose@otel-collector.service: Scheduled restart job, restart counter is at 5.
systemd[1]: docker-compose@otel-collector.service: Start request repeated too quickly.
systemd[1]: Failed to start otel-collector service with docker compose.

Three signals together are the latch, and all three are needed:

1. ActiveState=failed — the unit is down, not retrying.
2. NRestarts has reached StartLimitBurst and stopped climbing. A unit still in the loop has a counter that increments; a latched one does not move no matter how many starts you request.
3. The journal carries Start request repeated too quickly. — this is the limiter naming itself, and it is the only string that does so.

A unit in this state is not recovered by its restart policy, despite Restart=always:

“units which are configured for Restart=, and which reach the start limit are not attempted to be restarted anymore” — systemd.unit(5)

So the collector stays down until a human or the boot hook intervenes. A plain systemctl restart is refused outright — the unit never attempts a start and the journal reports start request repeated too quickly instead of the original fault.

This is intended. A transient network outage lasting longer than the window’s slow-case span will also latch the unit, and that trade was accepted on purpose: failing loud is the objective, the Firewalla host collector MISSING (host scraper dark) alert reports the resulting silence, and the boot hook clears the latch unattended on reboot.

Clearing the latch — record first, then clear

# 1. RECORD. reset-failed destroys this; there is no second chance.
systemctl show -p ActiveState -p SubState -p Result -p NRestarts \
    docker-compose@otel-collector
journalctl -u docker-compose@otel-collector -n 200 --no-pager

# 2. Only then clear and restart.
systemctl reset-failed docker-compose@otel-collector
systemctl restart docker-compose@otel-collector

MUST record the failure result, the restart counter and the journal message before running reset-failed. The latch is the diagnostic — it is the whole reason the bound exists. Clearing it blind converts a loud failure back into a silent one and destroys the only local evidence of what failed, which is precisely the behaviour this bound replaced. If the collector comes back and you never wrote down why it went away, you have restored the outage conditions, not fixed them.

Use the journal message, not the exit status, to classify the fault — see the pull-failure table above.

The alert fires but the router is healthy

Check the tile’s where clause against the query form. A presence alert that filters on ServiceName, or on a lowercase firewalla, matches zero rows on a live host and fires forever. Fix the clause in bootstrap-script.yaml; do not mute the alert.

Never mute, disable or widen a liveness alert in response to it firing without first recording the measured population of false positives that justifies the change. A muted alert returns the estate to silence while leaving a false belief of coverage — the precise failure this alerting exists to prevent.

Alert lifecycle

Alert definitions live in argocd/app-configs/clickstack-alerts/bootstrap-script.yaml and are applied by an ArgoCD PostSync hook Job. See Alerting operations for the general workflow.

The provisioner upserts by name: re-runs converge state rather than duplicating records, and a rename is a create plus an orphan. Changes reach the cluster by merge and ArgoCD sync only — never by editing HyperDX’s MongoDB by hand, and never by kubectl apply. An out-of-band write diverges from the source of truth invisibly and is silently reverted by the next sync.

Two consequences of that design are load-bearing, and both are the kind you otherwise meet for the first time in the middle of an incident.

Retiring an alert needs a tombstone, not a deletion

Deleting an alert’s upsert call from the bootstrap script does not delete the alert. upsert() matches by name and never reconciles deletions, so the document stays live in HyperDX’s MongoDB, stays bound to its tile, and keeps paging — with no line left in the script to trace it back to.

Retirement is therefore two edits, both required:

1. Remove the upsert or *TileAlert invocation, so the alert is not recreated.
2. Add an explicit retireAlert("…", "…") call, so the document already in MongoDB is removed. (The second argument is only the wording of the log receipt.)

The retireAlert call is a tombstone: it stays in the script permanently, long after the alert it retires is gone. Do not tidy it away — on a fresh cluster it is a harmless no-op, and on this one it is the only thing standing between a retired alert and a resurrection. bootstrap-script.yaml carries several, both as one-liners and as a list; follow the existing shape rather than inventing a new one.

retireAlert() queues the deletion rather than performing it. Nothing is written to db.alerts — neither upserts nor deletions — until the two-pass gate at the end of the script has validated every intended document. See Alert intervals below.

Confirm a retirement landed by querying db.alerts for the name after sync and expecting zero documents.

Re-syncing the alerts Application resets every alert it touches to OK

Every alert closure in bootstrap-script.yaml includes state: "OK" in the document it upserts, and upsert() $sets the whole document on every run. So each time the clickstack-alerts PostSync Job runs, every alert it touches is written back to state: "OK" — regardless of what the evaluator computed a second earlier.

“Every alert it touches” is the precise scope, and the distinction is observable: an alert this script does not define is left alone. Across the three syncs run while gathering evidence/criterion-2-observed-firing.md, 18 of the 19 documents in db.alerts were rewritten with the sync’s timestamp, while Traefik TLS cert expiring (<14d) — which has no closure in this script — kept its original updatedAt throughout. Do not assume an alert is safe from a sync just because this script does not mention it, though: check, rather than infer.

Coverage is not lost: the alert re-fires on its next evaluation. What is lost is the page and the state history across that window. Two rules follow:

• Do not re-sync the clickstack-alerts Application to “refresh” or “re-check” an alert while you are investigating one that is firing. You will clear the very state you are looking at, and suppress the page until the next evaluation.
• Never read an ALERT → OK transition that follows a sync as a recovery. That OK was written by the provisioner, not computed by the evaluator, and it says nothing at all about whether the underlying condition cleared. Confirm recovery from the data — re-run the query form and expect a non-zero count — not from the alert’s stored state.

Alert intervals

Every alert document carries an interval — the evaluation window. HyperDX validates it against a fixed enum on its own write path (AlertIntervalSchema). The permitted values, and the only ones that may appear in bootstrap-script.yaml, are:

1m   5m   15m   30m   1h   6h   12h   1d

This script bypasses that validation. The PostSync Job writes straight to HyperDX’s MongoDB rather than going through the product’s API, so an out-of-enum value can be persisted and nothing rejects it. Two of them lived in this file for months.

Be precise about what the defect actually is. Such an alert does still evaluate — the evaluator parses the interval string (ms(alert.interval)) instead of looking it up in the enum, and completed evaluations on 10-minute windows were observed live before the values were repaired. The problem is that the value is illegal by the product’s own schema, survives only through a hole in the write path, and is one upstream refactor away from breaking silently.

Do not write — in a comment, a plan, or a PR body — that an out-of-enum alert is dormant or has failed to run. That claim is false: three completed aligned 10-minute evaluations were observed in the running application’s own log before the repair. Writing it teaches the estate something untrue in a durable artifact and undermines the real justification.

Two independent instruments enforce the enum

Neither is the sole line of defence, deliberately.

#	Instrument	Where it lives	When it fires
1	Runtime two-pass gate	bootstrap-script.yaml, the ALERT_INTERVALS constant plus the validation pass at the end	at sync time, inside the PostSync Job
2	Blocking pre-merge check	CI	before the change can merge

The runtime gate throws. bootstrap-job.yaml sets restartPolicy: OnFailure with backoffLimit: 3, so a throw is a visible Job failure, not a silent partial apply. That is intended.

The gate is two-pass, and that is load-bearing

The script validates every alert document it intends to write before it performs a single write. Alert closures call queueAlert() and tombstones call retireAlert(); both only append to an in-memory list. Nothing touches db.alerts until the validation pass has run over the whole list.

This is not tidiness. upsert() $sets the whole document as each one is encountered, and every alert closure puts state: "OK" into that document. A gate that validated and wrote in the same iteration would therefore have already reset the stored state of every alert it had processed by the time it threw on a late invalid one — the fail-safe destroying the live firing state it exists to protect, which is the same damage described in Re-syncing the alerts Application resets every alert it touches to OK.

Two passes is what makes a rejected run leave db.alerts exactly as it found it — updatedAt included. A single-pass validate-and-write is a defect here, not an acceptable simplification.

Intervals are literals at every call site

An interval must be written as a quoted literal in each *TileAlert call. Never a shared constant, a variable, or a concatenation.

This is deliberate, and it is worth stating because the uniformity of the presence detectors actively invites a const PRESENCE_INTERVAL = "15m" refactor. Don’t. The pre-merge instrument is a text-level reader, and a value it cannot resolve is a value it cannot vouch for. The audit’s entire worth is that it is checkable by reading the call sites; hoisting the value into a constant trades a real guarantee for cosmetic tidiness, and the check will reject it.

Quoting an old or invalid interval inside a comment is fine — the gate strips comment lines before parsing, so a comment can neither trigger nor suppress a finding.

Per-alert interval map

A non-15m interval is not automatically a defect. The window has to suit the carrier’s emit cadence.

Alert	Interval	Why
Firewalla host collector MISSING (host scraper dark)	15m	presence detector; carrier emits ~120 points/hour
nas host collector MISSING (host scraper dark)	15m	same
nas-support host collector MISSING (host scraper dark)	15m	same
ClickStack gateway log intake MISSING	15m	presence detector; carrier emits ~240 points/hour
ClickStack OTel collector ClickHouse exporter missing	15m	presence detector (repaired from 10m)
Firewalla memory utilization HIGH (backstop)	15m	threshold alert on the same ~30 s carrier
Firewalla ULA reconcile listener DOWN	5m	a zero value fires on the next alert evaluation; a missing series needs a full window
NAS backup telemetry MISSING (collector dark)	1h	see below — this is the worked example
NAS Kopia snapshot FAILED	1d	the gauge is emitted only on a failed source at end-of-run

15m for the presence detectors is itself a measured choice, not a preference: across 30 days of the host carrier the complete population of gaps above 45 seconds was three multi-hour outages and one 49-second gap, so a 15-minute window would have produced zero false positives that month while still catching every real outage.

The worked example — why NAS backup telemetry MISSING (collector dark) keeps 1h. Its carrier is the backup pipeline’s file.mtime heartbeat, which is written on a ~25-hour backup cadence, not on a 30-second scrape cadence. Narrowing it to a presence-detector window would make it fire on very nearly every window on a perfectly healthy box. That manufactures exactly the false-positive population that the measurement above was gathered to avoid — and the first response to a noisy alert is a mute, which returns the estate to the silence this alerting exists to end. 1h is inside the permitted enum, so the audit does not flag it and no repair is owed. It is a decision, not an oversight.

Host collector coverage map

Three hosts run their own OTel collector and each carries the same exposure: if the collector or the box goes quiet, everything downstream of it goes quiet with it. Each host therefore has its own statically-filtered presence tile and its own alert.

Host	Tile	Alert	Carrier	Recovery
Firewalla	fw-host-present	Firewalla host collector MISSING (host scraper dark)	system.memory.utilization	systemctl reset-failed/restart docker-compose@otel-collector
nas	nas-host-present	nas host collector MISSING (host scraper dark)	system.memory.utilization	reprovision the otelcol-nas TrueNAS app: scripts/nas-playbook.sh --tags nas-otel
nas-support	nas-support-host-present	nas-support host collector MISSING (host scraper dark)	system.memory.utilization	systemctl reset-failed/restart otelcol-nas-support

nas-support additionally carries NAS backup telemetry MISSING (collector dark) on a second, different carrier (the backup pipeline’s file.mtime heartbeat). The two are complementary and their messages say so explicitly: both firing means the whole collector or the box is gone; only the backup one firing means the collector is alive and the fault is in the backup pipeline or its mount.

Two rules govern this map, and both exist because breaking either produces silent non-coverage:

• Every tile carries its own host.name literal. The metric name is shared: measured 2026-08-04, ten distinct host.name values emit system.memory.utilization in any given hour — these three hosts plus seven otel-node-*-agent-* Kubernetes node collectors. An unfiltered clause matches all ten, so its count can never reach zero and the alert is silent forever.
• Never express the three hosts as one alert with a groupBy. The empty-bucket branch fires only when the whole bucket returns no rows, so a grouped presence alert goes quiet precisely when one host vanishes and the others keep reporting — the case it would exist to catch. Replication is by additional alerts on additional statically-filtered tiles, never by grouping.

One alert on the same dashboard is not a per-host detector: ClickStack gateway log intake MISSING watches the cluster-side gateway’s OTLP receiver. It proves the gateway is accepting logs from someone; it has no per-source breakdown and cannot tell you that any particular host has gone dark.

Boot model, and the two design decisions behind it

Settled in plan 01-08. This section records what the boot path does today, what was decided not to build and why, and — explicitly — the limits of what any of it proves.

What is still load-bearing

The stack does not boot without the post_main.d hook mechanism, and no document in this repo should say otherwise. Four hooks remain load-bearing:

Hook	Still required for
0000-mount-extdata.sh	Mounting /extdata — everything below depends on it
0050-start-docker.sh	Re-asserting daemon.json + the mount gate, and starting dockerd
0100-install-docker-compose.sh	The compose CLI plugin
z0225-start-otel-collector.sh	Re-asserting this unit’s drop-ins, clearing its latch, enabling it

What plan 01-08 narrows is only this: the collector no longer depends on its own hook for start ordering. That is strictly narrower than HEAL-03 as written, and the plan deepens the reliance on that same hook in a second way — drop-in re-assertion (below). Both facts belong in any claim made about this work.

The hook re-asserts this unit’s drop-ins on every boot

/etc on the Firewalla is overlay-backed and is not assumed to survive a reboot. Ansible writes the per-instance drop-ins for immediate effect at converge time; the boot path never depends on that write having survived. z0225 therefore re-writes both drop-ins from the same role templates Ansible deploys, then runs daemon-reload — placed after the unconditional begin line and before the prerequisite skip branches, so they are in force even on a boot that then skips.

Both are load-bearing:

Drop-in	Absent after a reboot means
10-ghcr-auth.conf	No DOCKER_CONFIG → the private collector image cannot be pulled. This is the original outage class.
20-start-limit.conf	The unit reverts to the vendor 10s/5 window, which plan 01-07 measured as structurally incapable of engaging at RestartSec=5 — back to retrying forever in silence.

The blank line before each heredoc delimiter in z0225 is load-bearing. Jinja strips an included template’s trailing newline, so {% include %} followed immediately by the delimiter renders the delimiter glued to the last content line, where it never matches. bash then reads the heredoc to end of file: the drop-in receives the whole rest of the script and every command below it silently never runs. Verify any change with cat -A on the rendered file and confirm each delimiter sits alone on its own line.

Decision one — no readiness script, and no ExecStartPre= gate (option (b))

#1787 performs both prerequisite checks (mount present, daemon responding) inline in the boot hook. No separately-installed, environment-overridable readiness script is built. Three reasons, in order of weight:

1. Structural (decisive). A drop-in ExecStartPre= appends after the base unit’s. The base unit already declares ExecStartPre=/bin/mkdir -p $TMPDIR and ExecStartPre=/usr/bin/docker compose rm -fv, and the second fails first against an unresponsive daemon — so on the one boot a readiness gate exists for, the start dies before a gate placed there is ever reached.

   It is not true that no pre-start gate could ever be made to run first. One can: reset the inherited list with ExecStartPre=, the same empty-reset idiom that does work for ExecStart=. That is precisely why it is rejected — doing so silently drops the base unit’s TMPDIR creation and its container-removal step for this instance, buying the gate by removing two commands the unit depends on. The gate is therefore ineffective where it can be placed safely, and unsafe where it would be effective. Both placements lose.

2. Duplication. #1787 already performs both checks at the only place they can run before anything is touched.

3. Risk. Building it means substituting checks inside a shipped outage fix, for a gate that would not fire in the placement that is safe.

The forfeited residual, named rather than dropped: the hook’s two prerequisite failure branches are not environment-overridable and have no safe isolated invocation, so they are left unexercised. That is an accepted residual against HEAL-03, not a silent omission. Reaching them for real would mean unmounting /extdata or stopping dockerd on the estate’s authoritative internal name service and Vault unseal-chain participant — a test that can break production to prove production is safe has inverted its own purpose.

Decision two — no mount requirement at instance level, in any of three forms

All three evaluated by name; none declared.

Form	Disposition	Reason
RequiresMountsFor=/extdata	REJECTED	/extdata has no fstab entry, so there is no mount unit systemd could start; a requirement on an unstartable unit fails the boot transaction. (#1787’s own recorded reasoning one layer down.)
ConditionPathIsMountPoint=/extdata	REJECTED	/extdata is never mounted at multi-user.target — the hook that mounts it has not run — so the condition would fail on every boot. A unit skipped by a failed condition is not started at all, so Restart= has nothing to restart. The unit would sit inactive until this role’s own hook enabled it, re-creating exactly the dependence HEAL-03 exists to discharge.
Requires=/After=extdata.mount	REJECTED	The named mount unit does not exist.

The condition form was measured on the box, not taken from the manual, using a throwaway probe unit with Restart=always / RestartSec=1:

Probe	ConditionResult	ActiveState	NRestarts after 8 s
Condition fails	no	inactive	0
Condition satisfied (control)	yes	failed	5

Identical unit, identical restart policy; the condition alone flips the restart count from 0 to 5. A skipped unit really is never restarted.

None is needed: the mount dimension is already gated one layer down at docker.service, as a condition rather than a requirement, re-evaluated on every start attempt.

Open question — the dependency form on docker.service is NOT settled

Plan 01-08 intended a third drop-in replacing the base unit’s Requires=docker.service with the optional Wants= form. That is not implementable as a drop-in, and no such file ships.

Measured on this box (systemd 249.11) with an isolated probe unit:

Drop-in	Effective Requires=	Effective Wants=
none (control)	docker.service …	chronyd.service
Requires= (empty)	docker.service still present	chronyd.service
Wants= (empty)	docker.service	chronyd.service still present
Requires= then Wants=docker.service	docker.service still present	gains docker.service

Confirmed against upstream source: config_parse_unit_deps in src/core/load-fragment.c — the parser hardwired to Requires=, Wants=, After= and every other dependency directive in load-fragment-gperf.gperf.in — has no isempty(rvalue) guard. On an empty value it returns without clearing anything. By contrast config_parse_documentation does carry that guard, so the omission is deliberate rather than an oversight. This is not a version limitation: a dependency inherited from a shared base unit cannot be removed by any drop-in on any systemd.

The three available routes, none of which was taken unilaterally:

1. Accept the mandatory form as a documented residual. Cheapest and, by the plan’s own reasoning, nearly free: at boot the two forms behave identically, because docker’s extdata-gate condition skips rather than fails and a skip does not propagate as a failure under either form. The difference is confined to the non-boot case — a dockerd that genuinely fails, or that an operator stops, drags the collector down with it.
2. A full per-instance unit file at /etc/systemd/system/docker-compose@otel-collector.service, overriding the template for this instance. Structural: it duplicates a unit shared with the estate’s name service and will drift from it.
3. Edit the shared base unit — rejected outright. It is shared with the other compose instances.

Decision three — route 1: the mandatory form is an ACCEPTED RESIDUAL

Settled by the operator 2026-08-04 at plan 01-08’s reboot-1 checkpoint. Route 2 was declined explicitly: duplicating a unit shared with the estate’s name service creates a permanent drift surface on the DNS path, in exchange for a difference that is unobservable at boot for the measured reason above.

The residual, named rather than implied: docker-compose@otel-collector carries Requires=docker.service, exactly as its two siblings do, and no drop-in can change that on any systemd version. The consequence is confined to the non-boot case: a dockerd that genuinely fails, or that an operator stops, takes the collector down with it rather than leaving it to retry. No boot-ordering claim in this phase depends on the dependency form.

Do not re-litigate this by adding a Wants= drop-in. It would satisfy a naive rg -q 'Wants=docker.service' check while Requires= remained in force — a shipped directive with no effect under a banner claiming a benefit it does not deliver. If the mandatory form ever becomes genuinely costly, route 2 is the only mechanism that works, and it must be taken with the DNS-drift cost accepted openly.