NAS (TrueNAS) Ansible Operations

Operational guide for the TrueNAS 26 nas host (192.168.20.200) under Ansible management.

Principle: manage the appliance through the middleware API only

TrueNAS is an appliance. Its configuration lives in a database that reverts hand edits to /etc. All host configuration is therefore done through the middleware API (midclt call <method> …) over SSH with become, never by writing config files. ZFS/ACL operations use zfs/setfacl/midclt filesystem.*. (ADR hl-akt3.)

Containers are libvirt-LXC via the container.* middleware (Incus is retired in TrueNAS 26; ADR hl-hghu). Custom Docker workloads use app.*.

Connection model

Phase	Connect as	Notes
One-time bootstrap (Task 1–2)	sean (-e ansible_ssh_user=sean -e ansible_become=true)	nas-automation does not exist yet
Everything after	nas-automation (inventory default)	key is in Vault, not on disk

The nas-automation private key lives in Vault (secret/fzymgc-house/infrastructure/nas/automation-ssh, field private_key), not on the control node. The SSH agent is deliberately not used: a typical agent holds many unrelated keys, and offering them all trips sshd MaxAuthTries → “Too many authentication failures” before the right key is reached. So the key is materialized to a temp file and IdentitiesOnly (+ IdentityAgent=none) forces ssh to offer only it.

Preferred: the wrapper script

scripts/nas-playbook.sh does all of the above in one command — it checks the Vault session, materializes the key to a 0600 temp file (scrubbed on exit), and runs ansible-playbook with the correct connection flags. Pass through any ansible-playbook args:

# dry-run
scripts/nas-playbook.sh --tags nas-sandbox-tools --check --diff
# apply
scripts/nas-playbook.sh --tags nas-sandbox-tools
# single host
scripts/nas-playbook.sh --tags nas-sandbox-tools --limit nas-support

Prereq: vault status succeeds (run vault login or scripts/vault-helper.sh login first).

Ad-hoc commands (one-off diagnostics)

scripts/nas-adhoc.sh is the ad-hoc sibling — same Vault-key handling, but it wraps ansible <pattern> ... instead of ansible-playbook, for one-off checks without writing a throwaway playbook. Everything after the host pattern passes straight through to ansible, so add -b when the target needs root (most NAS diagnostics do):

# Kopia B2 repo maintenance state on the sandbox (dedicated config path; root):
scripts/nas-adhoc.sh nas-support -b -m shell \
  -a 'KOPIA_CONFIG_PATH=/etc/nas-kopia-b2/repository.config /home/linuxbrew/.linuxbrew/bin/kopia maintenance info'
# Kopia timers armed + spaced:
scripts/nas-adhoc.sh nas-support -b -m shell -a "systemctl list-timers 'kopia-*' --all"

Gotchas: -m shell runs via the box’s dash (no bashisms/PIPESTATUS — prefer -m command when a shell isn’t needed), and ad-hoc module args are Jinja-templated by Ansible, so avoid literal {{ }} in the command.

Manual fallback (if you need to run a different playbook)

vault kv get -mount=secret -field=private_key \
  fzymgc-house/infrastructure/nas/automation-ssh > /tmp/nas.key && chmod 600 /tmp/nas.key

ansible-playbook -i inventory nas-playbook.yml --tags <tag> \
  -e ansible_ssh_private_key_file=/tmp/nas.key \
  -e "ansible_ssh_common_args='-o IdentitiesOnly=yes -o IdentityAgent=none'"

The inner single quotes around the -o args are required: they keep the args together as one value for Ansible’s k=v parser. Without them, ansible’s ssh TTY-parser mis-splits on the spaces and fails with argument -o: expected one argument. This is why the wrapper hard-codes the quoted form — don’t “simplify” the quoting.

nas-automation has NOPASSWD: ALL sudo. The scoped allowlist in the spec is incompatible with Ansible become (which wraps commands in sudo … /bin/sh -c '…', so NOPASSWD must match the shell); midclt is already the appliance root API, so a per-binary allowlist adds no real isolation — the credential boundary is the Vault-held key.

Bootstrap order (first run)

cd ansible
# 1. Create the automation user (as sean)
ansible-playbook -i inventory nas-playbook.yml --tags nas-automation-user \
  -e ansible_ssh_user=sean -e ansible_become=true
# 2+. Everything else runs as nas-automation (see Connection model)

Prerequisite (Task 0, operator): seed Vault — secret/fzymgc-house/infrastructure/nas/automation-ssh (ed25519 keypair) and …/kopia-r2 (R2 creds + repo password).

Tag map

Tag	What it does	Target
nas-automation-user	Reconcile the nas-automation middleware user	nas
nas-drift	Render docs/reference/nas-config-baseline.md from middleware queries	nas
nas-otel	Deploy the otelcol-contrib custom app (host metrics → ClickStack over mTLS)	nas
nas-mail	Point native alert email at Cloudflare Email Sending (retires Mailgun) + set the Email alert recipient	nas
nas-sandbox-instance	Create/start the nas-support LXC + NIC + mounts	nas
nas-sandbox-bootstrap	nsenter: in-container user + sshd + key	nas
nas-sandbox-acls	Grant container-root rw on curation targets	nas
nas-sandbox-tools	Interior curation tooling	nas_sandbox_hosts
nas-game-lxc-instance	Create/start the precipice LXC + NIC + mounts	nas
nas-game-lxc-bootstrap	nsenter: in-container user + sshd + key	nas
nas-aresmush	Interior: users, valkey, caddy, tailscale, engine unit	nas_game_hosts
nas-forgejo	Deploy the Forgejo custom app (private git hosting at git.nas.fzymgc.house)	nas

The tooling sandbox (nas-support)

A libvirt-LXC (container.*, DEFAULT idmap) on macvlan-over-bond0 with a fixed MAC (00:16:3e:20:02:02) reserved to 192.168.20.202 in Firewalla DHCP, DNS nas-support.fzymgc.house.

Mounts — leaf datasets only. A FILESYSTEM device can bind-mount only a leaf dataset. Pool roots (/mnt/main, /mnt/apps) and any dataset whose subtree contains other ZFS mountpoints fail at start (Failed to bind mount … Invalid argument). List specific leaf datasets in nas_sandbox_mounts; for a parent dataset, list its leaf children individually.

Write access — DEFAULT idmap + per-dataset ACLs (ADR hl-3ek7). The DEFAULT idmap maps container-root (uid 0 inside) to host UID 2147000001. The FILESYSTEM device mounts rw at the VFS layer, but ZFS ACLs gate actual writes. nas-sandbox-acls grants rwx to 2147000001 on each nas_sandbox_curation_targets entry, acltype-aware:

• POSIX1E datasets: setfacl -R -m u:2147000001:rwx (+ default ACL for inheritance).
• NFSv4 datasets (e.g. multimedia, software, time-machine): midclt filesystem.setacl appending a USER ACE (FULL_CONTROL + INHERIT, recursive). setfacl does not work here.

The grant is scoped to container-root, not the nas-automation login uid (which maps to a different host UID). Run curation tooling as root inside the container via sudo -i (a login shell, so Homebrew is on PATH — see Interior tooling): ssh nas-support 'sudo -i rmlint /mnt/main/multimedia'.

Add a curation target

1. Add the leaf dataset to both nas_sandbox_mounts and nas_sandbox_curation_targets in inventory/host_vars/nas.yml.
2. Re-run --tags nas-sandbox-instance (adds the mount; restart the container to mount it — newly attached FILESYSTEM devices mount only at start) then --tags nas-sandbox-acls.

AresMUSH game host (precipice)

The precipice AresMUSH game (telnet 4201, websocket 4202, web API 4203, web portal via caddy at precipice.hound-skate.ts.net, public route precipice.muxserver.net → edge-node.muxserver.net over the tailnet) ran as an Incus container until TrueNAS 26 removed Incus (last Valkey write 2026-05-11). It is restored as a libvirt-LXC (nas-game-lxc role, base ubuntu:resolute) with the retired container’s MAC (00:16:3e:ae:0c:f1), so the standing Kea reservation and PowerDNS records (tf/dns-records/hosts.yml → 192.168.22.109, precipice.fzymgc.house) apply unchanged.

Data is the single leaf dataset main/fzymgc-house/incus/storage/precipice (weekly snapshots + Kopia→B2 via nas_kopia_leaves); two directories inside it bind-mount into the container:

Source (on nas)	Target (in container)	Owner (container ids)
…/precipice/server	/home/ares	ares 2000:2000
…/precipice/database	/var/lib/valkey	valkey 102:103

The nas-aresmush role pins those uids/gids before installing valkey-server — the package must adopt the existing ids or the mounted DB is unreadable under the DEFAULT idmap.

Secrets: secret/fzymgc-house/infrastructure/nas/precipice holds valkey_password (the game config’s requirepass); …/nas/ares-tailscale-auth holds the tag-scoped Tailscale OAuth client (tailscale_oauth_client.ares_game in tf/tailscale, tag:ares-game-node). Two tailscale realities of the unprivileged LXC:

• Userspace networking is mandatory — the container.* middleware has no TUN passthrough (device dtypes are FILESYSTEM/GPU/NIC/USB only), so /etc/default/tailscaled runs --tun=userspace-networking. Inbound tailnet traffic (edge-node → caddy) arrives via netstack; nothing on this host dials out over the tailnet.
• The node registers TAGGED (tag:ares-game-node), never as a user device. The role probes tailscale status --json and (re)registers with the Vault OAuth client whenever the node is logged out or missing an expected tag — tagged nodes have node-key expiry disabled, so the May–Aug failure mode (a restored identity whose key expired while the game was down) cannot recur. The tag’s ACLs (edge-node → 80/443/4201/4202) already stand in tailscale/policy.hujson; caddy fetches the precipice.hound-skate.ts.net cert once tailscale is up. Two enrolment gotchas, both hit live: a re-register that changes ownership (user → tagged) creates a new device, so the old one must be deleted in the admin console and the bare MagicDNS name reclaimed with a hostname bounce (tailscale set --hostname=precipice-tmp then back); and none of this repo’s OAuth clients carry a devices scope, so that deletion is operator-only.

The public route lives on edge-node (operator-owned DigitalOcean box, 157.230.201.90 — NOT in this repo’s ansible/terraform, and it was rebuilt as a RustDesk host after the game died, so the game routing had to be re-authored there 2026-08-22): /etc/caddy/sites/precipice.muxserver.net.conf (docker caddy; reverse_proxy → https://precipice.hound-skate.ts.net over the tailnet, MagicDNS resolver 100.100.100.100 works from inside the container) and /etc/haproxy/config/20-precipice-telnet.cfg (TCP :4201 → game host). Reload with docker exec caddy caddy reload --config /etc/caddy/Caddyfile and systemctl reload haproxy.

Engine lifecycle is the recovered aresmush.service (oneshot, User=ares, bash --login so rvm’s ruby-3.1.2 resolves): startares runs bundle install + rake migrate on every start, then deploys the Ember portal from ~/ares-webportal.

ssh precipice 'sudo systemctl restart aresmush'    # engine restart
ssh precipice 'sudo systemctl status aresmush valkey-server caddy tailscaled'

Game interior facts (health-checked 2026-08-23, zero errors post-restore):

• Daily local backup at 11:25 UTC (game/config/backup.yml) zips the game dir into ~/aresmush/backups/ (5 kept). GOTCHA, fixed 2026-08-23: the backup plugin reads the nested database → path in game/config/database.yml, which still said /var/lib/redis/dump.rdb — a mis-indented valkey-migration fix had left a dead top-level path: key, so every backup zip from Mar 2025 to May 11 silently lacked the DB dump. Corrected to /var/lib/valkey/dump.rdb (original kept as database.yml.bak-2026-08-23); the running engine picks it up on config/reload or the next engine restart.
• Images live in ~/aresmush/game/uploads/ (theme images + character galleries) — inside the mounted dataset, so snapshots and Kopia cover them — and are served by the container Caddyfile’s handle /game/* file-server block. No external image host is configured.

Recovery: the game data has ZFS snapshots (zfs list -t snapshot -r main/fzymgc-house/incus/storage/precipice) — including @pre-libvirt-restore-2026-08-22 taken immediately before the first libvirt-LXC start. The retired Incus rootfs (apps/incus-pool/containers/precipice, final snapshot @auto-daily-2026-05-12_00-00) is retained for forensics until P3 cleanup; the interior configs recovered from it live in the nas-aresmush role.

Do not verify reachability from nas itself — macvlan-over-bond0 blocks host↔container traffic. Test nc 192.168.22.109 4201 from another LAN host.

Interior tooling

nas-sandbox-tools installs all curation tooling via a Homebrew bundle (brew bundle). The package set lives in nas_brew_packages (default ncdu, rmlint, rclone, jq, kopia), rendered to a Brewfile at /home/nas-automation/.Brewfile; only Homebrew’s own bootstrap prerequisites (build-essential, curl, git, …) come from apt. Idempotence is gated on brew bundle check.

Homebrew installs under /home/linuxbrew/.linuxbrew; /etc/profile.d/homebrew.sh (eval of brew shellenv) puts the tools on PATH for login shells — an interactive ssh nas-support session, and root via sudo -i <tool>. Note: ssh nas-support '<tool> …' (a one-shot, non-login command) will not find them — use ssh nas-support 'sudo -i <tool> …' or an interactive session. The Kopia repo env is rendered to /etc/nas-kopia.env (0600) from the Vault kopia-r2 secret (R2/S3 backend). Defining backup sets is a separate task.

Add a tool: append its Homebrew formula to nas_brew_packages in host_vars/nas.yml (or the role defaults) and re-run --tags nas-sandbox-tools.

Telemetry

nas-otel-collector (--tags nas-otel) deploys otelcol-contrib as a TrueNAS custom app (app.*) shipping host metrics + journald logs to otel-gateway.fzymgc.house over mTLS → ClickStack (ADR hl-wo1m). The Firewalla otel-collector role is left untouched.

• mTLS cert is issued from the control node via Vault PKI (otel-collector-nas-client, a 720h client cert) and written to {{ nas_otel_base_dir }}/certs; the role re-issues only when the cert is missing or within 7 days of expiry.
• Ingest token (otel_ingest_api_key from secret/…/cluster/clickstack) and the cert are rendered into the collector config; the compose is delivered as the custom-app definition. app.update fires only on cert/config/compose drift (idempotent; no needless container cycle).
• Host root is mounted /:/hostfs:ro with hostmetrics root_path: /hostfs (cpu, memory, load, disk, filesystem, network). Resource attributes set service.name=otelcol-nas, host.name=nas.
• Verify metrics in ClickStack: SELECT count() FROM otel_metrics_sum WHERE ServiceName='otelcol-nas'.
• Logs ship via the journald receiver, tailing a curated allowlist of systemd units (nas_otel_journal_units; tune in host_vars/nas.yml) at priority: info. The receiver reads the persistent journal at /var/log/journal (bind-mounted :ro) and resumes across restarts via a file_storage cursor at {{ nas_otel_base_dir }}/cursor (bind-mounted :rw). /etc/machine-id is mounted for journald host identification. A transform/journald processor lifts journal fields into attributes and sets the body to the MESSAGE text so ClickStack shows readable log lines. No filter processor yet — add filter/nas-noise once real volume data is available. Verify in ClickStack: SELECT count() FROM default.otel_logs WHERE ServiceName='otelcol-nas' AND TimestampTime > now() - INTERVAL 1 HOUR.

nas-support (.202) collector

The sandbox host (nas-support, 192.168.20.202) runs its own collector, separate from the TrueNAS nas-otel-collector above. nas-support-otel-collector (--tags nas-support-otel-collector) deploys otelcol-contrib as a native systemd unit (otelcol-nas-support.service) — no container, no TrueNAS app machinery — shipping journald logs + host metrics + a backup-freshness series to otel-gateway.fzymgc.house over mTLS → ClickStack. The collector binary is SHA256-pinned (arch-keyed against the official otelcol-contrib checksums) and installed via get_url with a fail-closed checksum:; the entire deploy is Ansible-only with no manual on-box edits.

• mTLS identity: the cert is issued control-node-side from the Vault PKI role otel-collector-nas-support-client (a 720h client cert, CN otelcol-nas-support.fzymgc.house) and written under the collector’s cert dir 0640 root; the role re-issues only when the cert is missing or within 7 days of expiry.
• Host metrics: hostmetrics scrapes the native LXC’s own lxcfs /proc directly (no root_path — that breaks the filesystem scraper inside the container). Resource attributes set host.name=nas-support.
• journald logs: the journald receiver live-tails a curated systemd-unit allowlist (no directory: — it reads the running journal via journalctl); a transform/journald processor lifts journal fields into attributes and sets the body to the MESSAGE text.
• Backup freshness (BKP-08): a file_stats receiver emits file.mtime for the last-success heartbeat /var/lib/nas-kopia-b2/last-success. This is a non-monotonic Sum (epoch-seconds), so it lands in otel_metrics_sum, not otel_metrics_gauge; freshness age is now() - Value at query time.
• Loopback OTLP ingest (BKP-09): a 127.0.0.1:4318 OTLP/HTTP receiver (loopback-only, off-box actors cannot reach it) accepts the end-of-run gauges that kopia-b2-backup.sh pushes (nas.kopia.backup.duration.seconds, nas.kopia.snapshot.failed{source}). The health_check extension binds loopback 127.0.0.1:13133.

Deploy:

# dry-run first
scripts/nas-playbook.sh --tags nas-support-otel-collector --check --diff

# apply
scripts/nas-playbook.sh --tags nas-support-otel-collector

Verify the series arrive in ClickStack with the runbook in docs/operations/clickstack.md (§ “nas-support (.202) collector — verification queries”) — the BKP-07 / BKP-08 / BKP-09 assertions keyed on ResourceAttributes['host.name']='nas-support', plus the on-box smoke checks (systemctl is-active otelcol-nas-support, curl -fsS http://127.0.0.1:13133/).

Pre-apply gates (deploy-time, operator): confirm the target arch resolves the expected amd64 asset (a wrong arch fails the get_url checksum loudly); note whether /var/log/journal is persistent vs volatile on .202 (either is fine — the receiver live-tails; persistent only adds cross-reboot history); and confirm the control-node deploy token holds create on the Vault issue path .../issue/otel-collector-nas-support-client before applying. The Plan 01 tf/vault PKI role must have landed (GitOps merge → main-cluster-vault run) or the cert issue returns 403.

Forgejo (private git hosting)

nas-forgejo (--tags nas-forgejo) deploys Forgejo as a TrueNAS custom app at https://git.nas.fzymgc.house. It hosts repositories that must not leave the house, such as the Obsidian vault. It is not a mirror of GitHub and it runs no Actions. The design is in docs/engineering/specs/2026-09-21-nas-forgejo-design.md.

The role owns four things:

• The leaf dataset main/forgejo, created as its own encryption root with a TrueNAS-managed key, so it unlocks at boot. The role refuses an existing unencrypted dataset at that name.
• The secret files secret_key and internal_token, read from secret/fzymgc-house/infrastructure/nas/forgejo and written under /mnt/main/forgejo/secrets as uid 568, mode 0600. The container reads them through FORGEJO__security__*__FILE. Generate both with docker run --rm codeberg.org/forgejo/forgejo:<tag> forgejo generate secret SECRET_KEY and ... INTERNAL_TOKEN on any machine with Docker.
• The data directory /mnt/main/forgejo/data, mounted at /var/lib/gitea. The rootless image writes app.ini to custom/conf/app.ini under it, so there is no separate config mount.
• The compose definition, pushed with app.create or app.update. An update fires only when the rendered compose or a secret file changed.
• The reach. HTTPS goes through the NAS Traefik proxy on the nas-internal network with the cloudflare certificate resolver. SSH is the built-in server, published on host port 2222. Both use the one name, because the Traefik VIP is an alias on the NAS host interface.

Registration is off and every page needs a sign-in. Two steps run once, after the first converge, because Forgejo has no API for either:

1. Create the accounts. Two exist: forgejo-admin is the break-glass administrator and never signs in through Keycloak, and sean is the everyday account that links to Keycloak. Generate each password on the control node and pass it as --password, so nothing prints. Store them in secret/fzymgc-house/infrastructure/nas/forgejo as admin_username, admin_password and user_sean_initial_password.

   scripts/nas-adhoc.sh nas -b -m command \
     -a "docker exec ix-forgejo-forgejo-1 forgejo admin user create --admin --username forgejo-admin --email <admin email> --password $PW --must-change-password=false"
   scripts/nas-adhoc.sh nas -b -m command \
     -a "docker exec ix-forgejo-forgejo-1 forgejo admin user create --username sean --email <email> --password $PW2 --must-change-password=true"

   TrueNAS names the container ix-<app>-<service>-1, so the container is ix-forgejo-forgejo-1, not forgejo.

2. Add the Keycloak sign-in at https://git.nas.fzymgc.house/admin/auths/new: type OAuth2, provider OpenID Connect, name keycloak, discovery URL https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration, client id and secret from secret/fzymgc-house/infrastructure/nas/forgejo/oidc. Then link the Keycloak identity to the admin account from the account settings page.

Back up the dataset key once. The appliance keystore holds it, and Vault holds the copy for a rebuild:

scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j pool.dataset.export_key main/forgejo'
vault kv put secret/fzymgc-house/infrastructure/nas/forgejo-zfs-key key=<hex>

Kopia backs up /mnt/main/forgejo with the other leaves. A new FILESYSTEM device mounts at container start, so restart nas-support after the first converge, then run scripts/nas-playbook.sh --tags nas-kopia-acls. Kopia reads live data, so one nightly copy of the SQLite file can be torn. The main pool’s recursive snapshot tasks hold the consistent local copies.

Verify:

dig +short git.nas.fzymgc.house                                                        # 192.168.20.201
curl -sS -o /dev/null -w '%{http_code}\n' https://git.nas.fzymgc.house/api/healthz     # 200
ssh -p 2222 git@git.nas.fzymgc.house                                                    # "Hi there, sean!"

Outbound email / alerts (Cloudflare Email Sending)

TrueNAS sends its native alerts (SMART, scrub, pool-health, update notices) over authenticated SMTP. The nas-mail role (--tags nas-mail) points that transport at Cloudflare Email Sending — the cluster standard (docs/operations/outbound-email.md) — replacing the deprecated shared Mailgun transport. Because the appliance is managed through the middleware API only, the config is pushed via midclt call mail.update, never the UI.

• Credential: a per-app Cloudflare API token (Email Sending: Edit), minted and written to Vault (secret/fzymgc-house/infrastructure/nas/cloudflare-email, fields cf_email_token + email_from) by tf/cloudflare/nas_email.tf. The role reads it control-node-side over the operator’s Vault session and sets user=api_token, pass=<token>, fromemail=nas@hl.fzymgc.house.
• Transport: smtp.mx.cloudflare.net:465, security: SSL (implicit TLS — Cloudflare does not support STARTTLS/587). TrueNAS was already on SSL/465, so only the host, credentials, and sender change. fromemail must be on the onboarded hl.fzymgc.house domain or Cloudflare rejects the send.
• Recipient: the role also sets the Email alert service’s recipient explicitly (nas_mail_alert_recipient, default sean@fzymgc.email) — an empty recipient silently falls back to admin-user email fields.
• Idempotent: a field-by-field compare of mail.config; mail.update fires only on drift, and the transport is re-applied cleanly on every run.

Prerequisites (operator, one-time): the tf/cloudflare apply that mints the token requires Email Sending: Edit on the Cloudflare bootstrap token, and the main-cluster-vault policy grant for infrastructure/nas/cloudflare-email must apply before main-cluster-cloudflare. See tf/cloudflare/CLAUDE.md.

Verify: trigger a test from the TrueNAS UI (System Settings → General → Email → Send Test Mail), then confirm delivery in the Cloudflare dashboard (Email Service → Email Sending → Activity Log — the per-app token makes the NAS send attributable). Once verified, revoke the old Mailgun SMTP key.

Drift baseline

--tags nas-drift re-renders docs/reference/nas-config-baseline.md — a compact, sorted projection of shares, datasets, snapshot tasks, and apps. Commit it to surface configuration drift in version control.

Brew autoupdate timer (nas-support sandbox)

The nas-sandbox-tools role installs a weekly systemd timer that refreshes the sandbox Homebrew bundle unattended. Managed by Ansible over the nas-automation SSH connection; gated behind nas_brew_autoupdate (default true; disable per-host via host vars).

• Schedule: OnCalendar=Sun 04:00 with RandomizedDelaySec=30m and Persistent=true (a missed run replays on next boot).
• Command: brew update && brew bundle upgrade --file ~/.Brewfile && brew autoremove && brew cleanup (runs as nas-automation via a login shell so /etc/profile.d/homebrew.sh loads brew onto PATH).
• Failure handling: OnFailure=brew-update-notify.service curls the Pushover API (cluster-infra app token) with the failing unit name + hostname. The token + user key live in a 0600 root-owned env file rendered from Vault at play time.

Verify

Run over SSH as nas-automation (or via ansible nas-support -m command -a ...):

1. systemctl list-timers brew-update* — timer listed, NEXT ≈ next Sunday 04:00 ± jitter.
2. systemctl start brew-update.service — trigger a one-off run.
3. systemctl status brew-update.service — Active: inactive (dead) with ExecStart= exit 0 (success path).
4. Failure-path check (optional): run systemctl start brew-update-notify.service directly and confirm a Pushover notification arrives (validates the notify unit’s curl + creds + network path end-to-end without breaking the real upgrade; the notify unit is stateless so no revert is needed).

Telemetry note: unit stdout/stderr lands in the systemd journal, but nas-otel-collector does not yet ship journald to ClickStack (the distroless otel-contrib image lacks journalctl; a custom image is a follow-up). Until then, inspect via journalctl -u brew-update.service. The Pushover alert is the active failure surface.

Kopia→B2 backup timers (nas-support sandbox)

The nas-kopia-backup role runs three systemd timers on nas-support (192.168.20.202), spaced ≥9h apart so they never contend for Kopia’s single exclusive maintenance lock:

• Snapshot @ 03:00 (kopia-b2-backup.timer) — the nightly snapshot create of the irreplaceable datasets to B2, plus a quick opportunistic maintenance pass. Writes the last-success heartbeat before maintenance so freshness is decoupled from maintenance duration.
• Full maintenance @ 12:00 (kopia-maint-full.timer) — the dedicated daily kopia maintenance run --full (GC + pack compaction). Kept off the 03:00 snapshot critical path so the expensive full pass never delays the snapshot loop or the heartbeat. Isolation from the snapshot unit relies on the 03:00↔12:00 timing separation plus Kopia’s exclusive maintenance lock and default safety — there is deliberately no systemd force-stop directive.
• Freshness dead-man’s-switch (hourly) (kopia-freshness.timer) — checks the last-success heartbeat age against nas_kopia_freshness_max_age_hours (36h, the daily-cadence + margin threshold) and fires a Pushover alert if the last successful snapshot is stale.

All three units source Pushover creds from a shared root-owned 0600 pushover.env (rendered from Vault) via EnvironmentFile=; the token/user_key never appear in the on-disk script bodies.

IMPORTANT — repo-state caveat (not in git): the role lengthens Kopia’s auto full-maintenance interval via kopia maintenance set --full-interval (so opportunistic full never fires inside a 03:00 snapshot). This value is stored inside the B2 repo, not in git — a role/PR revert does not reset it. Verify the live value with kopia maintenance info (shows the Full Cycle interval and the root@nas-support owner).

Stale-alert triage (one unreadable path fails the whole leaf)

Kopia treats a single permission denied directory as a fatal error for that entire source; a failed source suppresses the heartbeat, so the hourly dead-man’s-switch then fires every hour until a fully clean run. Triage order:

1. Which source, since when: ClickStack gauge nas.kopia.snapshot.failed (attribute source = the failing leaf); wrapper log() lines are in otel_logs under ServiceName = 'otelcol-nas-support'.
2. The actual Kopia error is only in the on-host log: scripts/nas-adhoc.sh nas-support -b -m shell -a 'tail -60 "$(ls -t /var/log/nas-kopia-b2/backup-*.log | head -1)"'

The usual cause is a path the sandbox uid (2147000001) cannot read. The nas-kopia-acls walk grants rX access ACLs plus inheritable default ACLs on every directory, so paths created after the walk inherit the grant — but mv-ed-in trees keep their source ACLs and never inherit (2026-08-05 incident: a new TrueNAS user home, 700, no ACL). Remedy either way:

scripts/nas-playbook.sh --tags nas-kopia-acls   # re-run the grant walk