NAS (TrueNAS) Ansible Operations

Operational guide for the TrueNAS 26 nas host (192.168.20.200) under Ansible management.

Principle: manage the appliance through the middleware API only

TrueNAS is an appliance. Its configuration lives in a database that reverts hand edits to /etc. All host configuration is therefore done through the middleware API (midclt call <method> …) over SSH with become, never by writing config files. ZFS/ACL operations use zfs/setfacl/midclt filesystem.*. (ADR hl-akt3.)

Containers are libvirt-LXC via the container.* middleware (Incus is retired in TrueNAS 26; ADR hl-hghu). Custom Docker workloads use app.*.

Connection model

Phase	Connect as	Notes
One-time bootstrap (Task 1–2)	sean (-e ansible_ssh_user=sean -e ansible_become=true)	nas-automation does not exist yet
Everything after	nas-automation (inventory default)	key is in Vault, not on disk

The nas-automation private key lives in Vault (secret/fzymgc-house/infrastructure/nas/automation-ssh, field private_key), not on the control node. The SSH agent is deliberately not used: a typical agent holds many unrelated keys, and offering them all trips sshd MaxAuthTries → “Too many authentication failures” before the right key is reached. So the key is materialized to a temp file and IdentitiesOnly (+ IdentityAgent=none) forces ssh to offer only it.

Preferred: the wrapper script

scripts/nas-playbook.sh does all of the above in one command — it checks the Vault session, materializes the key to a 0600 temp file (scrubbed on exit), and runs ansible-playbook with the correct connection flags. Pass through any ansible-playbook args:

# dry-run
scripts/nas-playbook.sh --tags nas-sandbox-tools --check --diff
# apply
scripts/nas-playbook.sh --tags nas-sandbox-tools
# single host
scripts/nas-playbook.sh --tags nas-sandbox-tools --limit nas-support

Prereq: vault status succeeds (run vault login or scripts/vault-helper.sh login first).

Ad-hoc commands (one-off diagnostics)

scripts/nas-adhoc.sh is the ad-hoc sibling — same Vault-key handling, but it wraps ansible <pattern> ... instead of ansible-playbook, for one-off checks without writing a throwaway playbook. Everything after the host pattern passes straight through to ansible, so add -b when the target needs root (most NAS diagnostics do):

# Kopia B2 repo maintenance state on the sandbox (dedicated config path; root):
scripts/nas-adhoc.sh nas-support -b -m shell \
  -a 'KOPIA_CONFIG_PATH=/etc/nas-kopia-b2/repository.config /home/linuxbrew/.linuxbrew/bin/kopia maintenance info'
# Kopia timers armed + spaced:
scripts/nas-adhoc.sh nas-support -b -m shell -a "systemctl list-timers 'kopia-*' --all"

Gotchas: -m shell runs via the box’s dash (no bashisms/PIPESTATUS — prefer -m command when a shell isn’t needed), and ad-hoc module args are Jinja-templated by Ansible, so avoid literal {{ }} in the command.

Manual fallback (if you need to run a different playbook)

vault kv get -mount=secret -field=private_key \
  fzymgc-house/infrastructure/nas/automation-ssh > /tmp/nas.key && chmod 600 /tmp/nas.key

ansible-playbook -i inventory nas-playbook.yml --tags <tag> \
  -e ansible_ssh_private_key_file=/tmp/nas.key \
  -e "ansible_ssh_common_args='-o IdentitiesOnly=yes -o IdentityAgent=none'"

The inner single quotes around the -o args are required: they keep the args together as one value for Ansible’s k=v parser. Without them, ansible’s ssh TTY-parser mis-splits on the spaces and fails with argument -o: expected one argument. This is why the wrapper hard-codes the quoted form — don’t “simplify” the quoting.

nas-automation has NOPASSWD: ALL sudo. The scoped allowlist in the spec is incompatible with Ansible become (which wraps commands in sudo … /bin/sh -c '…', so NOPASSWD must match the shell); midclt is already the appliance root API, so a per-binary allowlist adds no real isolation — the credential boundary is the Vault-held key.

Bootstrap order (first run)

cd ansible
# 1. Create the automation user (as sean)
ansible-playbook -i inventory nas-playbook.yml --tags nas-automation-user \
  -e ansible_ssh_user=sean -e ansible_become=true
# 2+. Everything else runs as nas-automation (see Connection model)

Prerequisite (Task 0, operator): seed Vault — secret/fzymgc-house/infrastructure/nas/automation-ssh (ed25519 keypair) and …/kopia-r2 (R2 creds + repo password).

Tag map

Tag	What it does	Target
nas-automation-user	Reconcile the nas-automation middleware user	nas
nas-drift	Render docs/reference/nas-config-baseline.md from middleware queries	nas
nas-otel	Deploy the otelcol-contrib custom app (host metrics → ClickStack over mTLS)	nas
nas-mail	Point native alert email at Cloudflare Email Sending (retires Mailgun) + set the Email alert recipient	nas
nas-sandbox-instance	Create/start the nas-support LXC + NIC + mounts	nas
nas-sandbox-bootstrap	nsenter: in-container user + sshd + key	nas
nas-sandbox-acls	Grant container-root rw on curation targets	nas
nas-sandbox-tools	Interior curation tooling	nas_sandbox_hosts

The tooling sandbox (nas-support)

A libvirt-LXC (container.*, DEFAULT idmap) on macvlan-over-bond0 with a fixed MAC (00:16:3e:20:02:02) reserved to 192.168.20.202 in Firewalla DHCP, DNS nas-support.fzymgc.house.

Mounts — leaf datasets only. A FILESYSTEM device can bind-mount only a leaf dataset. Pool roots (/mnt/main, /mnt/apps) and any dataset whose subtree contains other ZFS mountpoints fail at start (Failed to bind mount … Invalid argument). List specific leaf datasets in nas_sandbox_mounts; for a parent dataset, list its leaf children individually.

Write access — DEFAULT idmap + per-dataset ACLs (ADR hl-3ek7). The DEFAULT idmap maps container-root (uid 0 inside) to host UID 2147000001. The FILESYSTEM device mounts rw at the VFS layer, but ZFS ACLs gate actual writes. nas-sandbox-acls grants rwx to 2147000001 on each nas_sandbox_curation_targets entry, acltype-aware:

• POSIX1E datasets: setfacl -R -m u:2147000001:rwx (+ default ACL for inheritance).
• NFSv4 datasets (e.g. multimedia, software, time-machine): midclt filesystem.setacl appending a USER ACE (FULL_CONTROL + INHERIT, recursive). setfacl does not work here.

The grant is scoped to container-root, not the nas-automation login uid (which maps to a different host UID). Run curation tooling as root inside the container via sudo -i (a login shell, so Homebrew is on PATH — see Interior tooling): ssh nas-support 'sudo -i rmlint /mnt/main/multimedia'.

Add a curation target

1. Add the leaf dataset to both nas_sandbox_mounts and nas_sandbox_curation_targets in inventory/host_vars/nas.yml.
2. Re-run --tags nas-sandbox-instance (adds the mount; restart the container to mount it — newly attached FILESYSTEM devices mount only at start) then --tags nas-sandbox-acls.

Interior tooling

nas-sandbox-tools installs all curation tooling via a Homebrew bundle (brew bundle). The package set lives in nas_brew_packages (default ncdu, rmlint, rclone, jq, kopia), rendered to a Brewfile at /home/nas-automation/.Brewfile; only Homebrew’s own bootstrap prerequisites (build-essential, curl, git, …) come from apt. Idempotence is gated on brew bundle check.

Homebrew installs under /home/linuxbrew/.linuxbrew; /etc/profile.d/homebrew.sh (eval of brew shellenv) puts the tools on PATH for login shells — an interactive ssh nas-support session, and root via sudo -i <tool>. Note: ssh nas-support '<tool> …' (a one-shot, non-login command) will not find them — use ssh nas-support 'sudo -i <tool> …' or an interactive session. The Kopia repo env is rendered to /etc/nas-kopia.env (0600) from the Vault kopia-r2 secret (R2/S3 backend). Defining backup sets is a separate task.

Add a tool: append its Homebrew formula to nas_brew_packages in host_vars/nas.yml (or the role defaults) and re-run --tags nas-sandbox-tools.

Telemetry

nas-otel-collector (--tags nas-otel) deploys otelcol-contrib as a TrueNAS custom app (app.*) shipping host metrics + journald logs to otel-gateway.fzymgc.house over mTLS → ClickStack (ADR hl-wo1m). The Firewalla otel-collector role is left untouched.

• mTLS cert is issued from the control node via Vault PKI (otel-collector-nas-client, a 720h client cert) and written to {{ nas_otel_base_dir }}/certs; the role re-issues only when the cert is missing or within 7 days of expiry.
• Ingest token (otel_ingest_api_key from secret/…/cluster/clickstack) and the cert are rendered into the collector config; the compose is delivered as the custom-app definition. app.update fires only on cert/config/compose drift (idempotent; no needless container cycle).
• Host root is mounted /:/hostfs:ro with hostmetrics root_path: /hostfs (cpu, memory, load, disk, filesystem, network). Resource attributes set service.name=otelcol-nas, host.name=nas.
• Verify metrics in ClickStack: SELECT count() FROM otel_metrics_sum WHERE ServiceName='otelcol-nas'.
• Logs ship via the journald receiver, tailing a curated allowlist of systemd units (nas_otel_journal_units; tune in host_vars/nas.yml) at priority: info. The receiver reads the persistent journal at /var/log/journal (bind-mounted :ro) and resumes across restarts via a file_storage cursor at {{ nas_otel_base_dir }}/cursor (bind-mounted :rw). /etc/machine-id is mounted for journald host identification. A transform/journald processor lifts journal fields into attributes and sets the body to the MESSAGE text so ClickStack shows readable log lines. No filter processor yet — add filter/nas-noise once real volume data is available. Verify in ClickStack: SELECT count() FROM default.otel_logs WHERE ServiceName='otelcol-nas' AND TimestampTime > now() - INTERVAL 1 HOUR.

nas-support (.202) collector

The sandbox host (nas-support, 192.168.20.202) runs its own collector, separate from the TrueNAS nas-otel-collector above. nas-support-otel-collector (--tags nas-support-otel-collector) deploys otelcol-contrib as a native systemd unit (otelcol-nas-support.service) — no container, no TrueNAS app machinery — shipping journald logs + host metrics + a backup-freshness series to otel-gateway.fzymgc.house over mTLS → ClickStack. The collector binary is SHA256-pinned (arch-keyed against the official otelcol-contrib checksums) and installed via get_url with a fail-closed checksum:; the entire deploy is Ansible-only with no manual on-box edits.

• mTLS identity: the cert is issued control-node-side from the Vault PKI role otel-collector-nas-support-client (a 720h client cert, CN otelcol-nas-support.fzymgc.house) and written under the collector’s cert dir 0640 root; the role re-issues only when the cert is missing or within 7 days of expiry.
• Host metrics: hostmetrics scrapes the native LXC’s own lxcfs /proc directly (no root_path — that breaks the filesystem scraper inside the container). Resource attributes set host.name=nas-support.
• journald logs: the journald receiver live-tails a curated systemd-unit allowlist (no directory: — it reads the running journal via journalctl); a transform/journald processor lifts journal fields into attributes and sets the body to the MESSAGE text.
• Backup freshness (BKP-08): a file_stats receiver emits file.mtime for the last-success heartbeat /var/lib/nas-kopia-b2/last-success. This is a non-monotonic Sum (epoch-seconds), so it lands in otel_metrics_sum, not otel_metrics_gauge; freshness age is now() - Value at query time.
• Loopback OTLP ingest (BKP-09): a 127.0.0.1:4318 OTLP/HTTP receiver (loopback-only, off-box actors cannot reach it) accepts the end-of-run gauges that kopia-b2-backup.sh pushes (nas.kopia.backup.duration.seconds, nas.kopia.snapshot.failed{source}). The health_check extension binds loopback 127.0.0.1:13133.

Deploy:

# dry-run first
scripts/nas-playbook.sh --tags nas-support-otel-collector --check --diff

# apply
scripts/nas-playbook.sh --tags nas-support-otel-collector

Verify the series arrive in ClickStack with the runbook in docs/operations/clickstack.md (§ “nas-support (.202) collector — verification queries”) — the BKP-07 / BKP-08 / BKP-09 assertions keyed on ResourceAttributes['host.name']='nas-support', plus the on-box smoke checks (systemctl is-active otelcol-nas-support, curl -fsS http://127.0.0.1:13133/).

Pre-apply gates (deploy-time, operator): confirm the target arch resolves the expected amd64 asset (a wrong arch fails the get_url checksum loudly); note whether /var/log/journal is persistent vs volatile on .202 (either is fine — the receiver live-tails; persistent only adds cross-reboot history); and confirm the control-node deploy token holds create on the Vault issue path .../issue/otel-collector-nas-support-client before applying. The Plan 01 tf/vault PKI role must have landed (GitOps merge → main-cluster-vault run) or the cert issue returns 403.

Outbound email / alerts (Cloudflare Email Sending)

TrueNAS sends its native alerts (SMART, scrub, pool-health, update notices) over authenticated SMTP. The nas-mail role (--tags nas-mail) points that transport at Cloudflare Email Sending — the cluster standard (docs/operations/outbound-email.md) — replacing the deprecated shared Mailgun transport. Because the appliance is managed through the middleware API only, the config is pushed via midclt call mail.update, never the UI.

• Credential: a per-app Cloudflare API token (Email Sending: Edit), minted and written to Vault (secret/fzymgc-house/infrastructure/nas/cloudflare-email, fields cf_email_token + email_from) by tf/cloudflare/nas_email.tf. The role reads it control-node-side over the operator’s Vault session and sets user=api_token, pass=<token>, fromemail=nas@hl.fzymgc.house.
• Transport: smtp.mx.cloudflare.net:465, security: SSL (implicit TLS — Cloudflare does not support STARTTLS/587). TrueNAS was already on SSL/465, so only the host, credentials, and sender change. fromemail must be on the onboarded hl.fzymgc.house domain or Cloudflare rejects the send.
• Recipient: the role also sets the Email alert service’s recipient explicitly (nas_mail_alert_recipient, default sean@fzymgc.email) — an empty recipient silently falls back to admin-user email fields.
• Idempotent: a field-by-field compare of mail.config; mail.update fires only on drift, and the transport is re-applied cleanly on every run.

Prerequisites (operator, one-time): the tf/cloudflare apply that mints the token requires Email Sending: Edit on the Cloudflare bootstrap token, and the main-cluster-vault policy grant for infrastructure/nas/cloudflare-email must apply before main-cluster-cloudflare. See tf/cloudflare/CLAUDE.md.

Verify: trigger a test from the TrueNAS UI (System Settings → General → Email → Send Test Mail), then confirm delivery in the Cloudflare dashboard (Email Service → Email Sending → Activity Log — the per-app token makes the NAS send attributable). Once verified, revoke the old Mailgun SMTP key.

Drift baseline

--tags nas-drift re-renders docs/reference/nas-config-baseline.md — a compact, sorted projection of shares, datasets, snapshot tasks, and apps. Commit it to surface configuration drift in version control.

Brew autoupdate timer (nas-support sandbox)

The nas-sandbox-tools role installs a weekly systemd timer that refreshes the sandbox Homebrew bundle unattended. Managed by Ansible over the nas-automation SSH connection; gated behind nas_brew_autoupdate (default true; disable per-host via host vars).

• Schedule: OnCalendar=Sun 04:00 with RandomizedDelaySec=30m and Persistent=true (a missed run replays on next boot).
• Command: brew update && brew bundle upgrade --file ~/.Brewfile && brew autoremove && brew cleanup (runs as nas-automation via a login shell so /etc/profile.d/homebrew.sh loads brew onto PATH).
• Failure handling: OnFailure=brew-update-notify.service curls the Pushover API (cluster-infra app token) with the failing unit name + hostname. The token + user key live in a 0600 root-owned env file rendered from Vault at play time.

Verify

Run over SSH as nas-automation (or via ansible nas-support -m command -a ...):

1. systemctl list-timers brew-update* — timer listed, NEXT ≈ next Sunday 04:00 ± jitter.
2. systemctl start brew-update.service — trigger a one-off run.
3. systemctl status brew-update.service — Active: inactive (dead) with ExecStart= exit 0 (success path).
4. Failure-path check (optional): run systemctl start brew-update-notify.service directly and confirm a Pushover notification arrives (validates the notify unit’s curl + creds + network path end-to-end without breaking the real upgrade; the notify unit is stateless so no revert is needed).

Telemetry note: unit stdout/stderr lands in the systemd journal, but nas-otel-collector does not yet ship journald to ClickStack (the distroless otel-contrib image lacks journalctl; a custom image is a follow-up). Until then, inspect via journalctl -u brew-update.service. The Pushover alert is the active failure surface.

Kopia→B2 backup timers (nas-support sandbox)

The nas-kopia-backup role runs three systemd timers on nas-support (192.168.20.202), spaced ≥9h apart so they never contend for Kopia’s single exclusive maintenance lock:

• Snapshot @ 03:00 (kopia-b2-backup.timer) — the nightly snapshot create of the irreplaceable datasets to B2, plus a quick opportunistic maintenance pass. Writes the last-success heartbeat before maintenance so freshness is decoupled from maintenance duration.
• Full maintenance @ 12:00 (kopia-maint-full.timer) — the dedicated daily kopia maintenance run --full (GC + pack compaction). Kept off the 03:00 snapshot critical path so the expensive full pass never delays the snapshot loop or the heartbeat. Isolation from the snapshot unit relies on the 03:00↔12:00 timing separation plus Kopia’s exclusive maintenance lock and default safety — there is deliberately no systemd force-stop directive.
• Freshness dead-man’s-switch (hourly) (kopia-freshness.timer) — checks the last-success heartbeat age against nas_kopia_freshness_max_age_hours (36h, the daily-cadence + margin threshold) and fires a Pushover alert if the last successful snapshot is stale.

All three units source Pushover creds from a shared root-owned 0600 pushover.env (rendered from Vault) via EnvironmentFile=; the token/user_key never appear in the on-disk script bodies.

IMPORTANT — repo-state caveat (not in git): the role lengthens Kopia’s auto full-maintenance interval via kopia maintenance set --full-interval (so opportunistic full never fires inside a 03:00 snapshot). This value is stored inside the B2 repo, not in git — a role/PR revert does not reset it. Verify the live value with kopia maintenance info (shows the Full Cycle interval and the root@nas-support owner).