Skip to content

NAS (TrueNAS) Ansible Operations

Operational guide for the TrueNAS 26 nas host (192.168.20.200) under Ansible management. The roles live under ansible/roles/nas-*, the host values in ansible/inventory/host_vars/nas.yml, and the playbook is ansible/nas-playbook.yml.

Principle: manage the appliance through the middleware API only

Section titled “Principle: manage the appliance through the middleware API only”

TrueNAS is an appliance. Its configuration lives in a database that reverts hand edits to /etc. All host configuration goes through the middleware API (midclt call <method> ...) over SSH with become, never through config files. ZFS and ACL operations use zfs, setfacl and midclt filesystem.*. The decision is the ADR Manage TrueNAS exclusively via the middleware API.

Containers are libvirt-LXC through the container.* middleware, because TrueNAS 26 has no Incus (ADR Build NAS containers on libvirt-LXC). Custom Docker workloads use app.*.

Phase Connect as Notes
One-time bootstrap sean (-e ansible_ssh_user=sean -e ansible_become=true) nas-automation does not exist yet
Everything after nas-automation (inventory default) the key is in Vault, not on disk

The nas-automation private key lives in Vault at secret/fzymgc-house/infrastructure/nas/automation-ssh, field private_key, not on the control node. The SSH agent is not used. An agent holds many unrelated keys, and offering them all trips sshd MaxAuthTries (“Too many authentication failures”) before the right key is reached. The key is written to a temp file, and IdentitiesOnly with IdentityAgent=none makes ssh offer only that key.

scripts/nas-playbook.sh does all of this in one command. It checks the Vault session, writes the key to a 0600 temp file that it removes on exit, and runs ansible-playbook with the connection flags. Pass through any ansible-playbook arguments:

Terminal window
# dry-run
scripts/nas-playbook.sh --tags nas-sandbox-tools --check --diff
# apply
scripts/nas-playbook.sh --tags nas-sandbox-tools
# single host
scripts/nas-playbook.sh --tags nas-sandbox-tools --limit nas-support

Prerequisite: vault status succeeds. If it does not, run vault login or scripts/vault-helper.sh login first.

scripts/nas-adhoc.sh is the ad-hoc sibling. It handles the Vault key the same way and wraps ansible <pattern> ... instead of ansible-playbook. Everything after the host pattern passes through to ansible. Add -b when the target needs root, which most NAS diagnostics do:

Terminal window
# Kopia B2 repo maintenance state on the sandbox (dedicated config path; root):
scripts/nas-adhoc.sh nas-support -b -m shell \
-a 'KOPIA_CONFIG_PATH=/etc/nas-kopia-b2/repository.config /home/linuxbrew/.linuxbrew/bin/kopia maintenance info'
# Kopia timers armed + spaced:
scripts/nas-adhoc.sh nas-support -b -m shell -a "systemctl list-timers 'kopia-*' --all"

Two constraints: -m shell runs through the box’s dash, so use no bashisms and prefer -m command when no shell is needed. Ad-hoc module arguments are Jinja-templated by Ansible, so write no literal {{ }} in a command.

Manual fallback (if you need to run a different playbook)

Section titled “Manual fallback (if you need to run a different playbook)”
Terminal window
vault kv get -mount=secret -field=private_key \
fzymgc-house/infrastructure/nas/automation-ssh > /tmp/nas.key && chmod 600 /tmp/nas.key
ansible-playbook -i inventory nas-playbook.yml --tags <tag> \
-e ansible_ssh_private_key_file=/tmp/nas.key \
-e "ansible_ssh_common_args='-o IdentitiesOnly=yes -o IdentityAgent=none'"

The inner single quotes around the -o arguments are required. They keep the arguments together as one value for Ansible’s k=v parser. Without them, the parser splits on the spaces and fails with argument -o: expected one argument. The wrapper hard-codes the quoted form for this reason.

nas-automation has NOPASSWD: ALL sudo. Ansible become wraps commands in sudo ... /bin/sh -c '...', so a per-binary allowlist cannot match. midclt is already the appliance root API, so a narrower allowlist adds no isolation. The credential boundary is the Vault-held key.

Seed Vault first: secret/fzymgc-house/infrastructure/nas/automation-ssh (an ed25519 keypair).

Terminal window
cd ansible
# 1. Create the automation user (as sean)
ansible-playbook -i inventory nas-playbook.yml --tags nas-automation-user \
-e ansible_ssh_user=sean -e ansible_become=true
# 2+. Everything else runs as nas-automation (see Connection model)
Tag What it does Target
nas-automation-user Reconcile the nas-automation middleware user nas
nas-core-vlan Create the VLAN interface vlan1 on bond0 for Core, with no address, as the macvlan parent of the Core containers nas
nas-drift Render docs/reference/nas-config-baseline.md from middleware queries nas
nas-docker-networks Set the Docker address pools and the shared app networks nas-internal and nas-public nas
nas-otel Deploy the otelcol-contrib custom app (host metrics and journald logs to ClickStack over mTLS) and its Vault Agent sidecar nas
nas-tailscale Deploy the Tailscale custom app (tailnet node truenas-scale, subnet routes and exit node) nas
nas-rustfs Deploy the RustFS custom app, its Vault Agent sidecar and its encrypted data dataset. See RustFS nas
nas-forgejo Deploy the Forgejo custom app (private git hosting at git.nas.fzymgc.house) nas
nas-ha-backups The dataset main/home-assistant-backups and its NFS export for the Home Assistant VM. See Home Assistant nas
nas-mail Point native alert email at Cloudflare Email Sending and set the Email alert recipient nas
nas-sandbox-instance Create and start the nas-support LXC, its NIC and its mounts nas
nas-sandbox-bootstrap Through nsenter: in-container user, sshd and key nas
nas-sandbox-acls Grant container-root read-write on the curation targets nas
nas-kopia-acls Grant container-root read-only on every Kopia leaf nas
nas-sandbox-tools Interior curation tooling and the brew autoupdate timer nas_sandbox_hosts
nas-kopia-backup The Kopia to B2 backup, maintenance and freshness timers nas_sandbox_hosts
nas-support-otel-collector The native otelcol-contrib unit on the sandbox nas_sandbox_hosts
nas-game-lxc-instance Create and start the precipice LXC, its NIC and its mounts nas
nas-game-lxc-bootstrap Through nsenter: in-container user, sshd and key nas
nas-aresmush Interior: users, valkey, caddy, tailscale, the engine unit nas_game_hosts
nas-vault-unseal-lxc Create and start the vault-unseal LXC and its NIC, and bootstrap it through nsenter. nas-game-lxc does not select it nas
vault-unseal Interior: the vault-unseal binary, its configuration and its unit nas_vault_unseal_hosts

Docker address pools and shared app networks

Section titled “Docker address pools and shared app networks”

TrueNAS gives every app network a subnet from its Docker address pools. The nas-docker-networks tag sets the pools and creates the two shared networks. The values live in ansible/inventory/group_vars/nas_hosts.yml.

Item Value
IPv4 pool 172.16.0.0/12, one /24 per network
IPv6 pool fdd0::/48, one /64 per network
nas-internal 172.16.100.0/24 and fdd0:0:0:100::/64, internal, no route out
nas-public 172.16.101.0/24 and fdd0:0:0:101::/64

Two rules protect the pools:

  • Every base must be an aligned prefix. Docker skips an unaligned base such as 172.17.0.0/12 without a log line and takes the next pool.
  • No pool can sit inside 100.64.0.0/10. The Tailscale app drops any packet from that range that does not arrive on tailscale0. With app networks in that range, a published port such as 192.168.20.201:443 fails for tunnel clients and for the NAS itself, while wired LAN clients still work. Ping still answers, because the address is a host alias.

A pool change restarts Docker and every app, the Tailscale app included. When the NAS’s Tailscale goes down, the tailnet moves the subnet routes to another advertiser and does not move them back on its own. After the apps return, follow Fail the routes back to the NAS in Tailscale Subnet Routers. Existing networks keep their subnets. To move a shared network to a new subnet:

  1. Stop the apps on it: midclt call app.stop <name> for traefik, forgejo, rustfs, dozzle and otelcol-nas.

  2. Run the tag with force. Force disconnects the containers and recreates the network.

    Terminal window
    scripts/nas-playbook.sh --tags nas-docker-networks -e nas_docker_networks_force=true
  3. Start the apps again with midclt call app.start <name>.

  4. Run scripts/nas-playbook.sh --tags nas-rustfs, because RustFS trusts the proxies by subnet.

To make sure that no network is left in the old range, run this on the NAS:

Terminal window
docker network ls -q | xargs -n1 docker network inspect --format '{{.Name}} {{range .IPAM.Config}}{{.Subnet}} {{end}}'

A libvirt-LXC (container.*, DEFAULT idmap) on macvlan over vlan1, the Core VLAN interface of the NAS, with the fixed MAC 00:16:3e:20:02:02. The dns-control entry nas_support reserves 192.168.40.32 on Core for that MAC and publishes the name nas-support.fzymgc.house.

vlan1 holds no address. The NAS reaches Core through the gateway. A frame for the container arrives on vlan1 from the wire, and macvlan hands it to the container. A host address on vlan1 routes Core on-link instead. Macvlan never passes a frame from the host’s own stack to a child on the same parent. So that address cuts the container off from the host’s forwarding. Without it, the container is reachable from a tailnet client through truenas-scale. It stays unreachable from nas itself. The tag nas-core-vlan of nas-common creates the interface. Run it before nas-sandbox-instance, which fails when the parent is missing.

Mounts are leaf datasets only. A FILESYSTEM device can bind-mount only a leaf dataset. A pool root (/mnt/main, /mnt/apps) or a dataset whose subtree holds other ZFS mountpoints fails at start with Failed to bind mount ... Invalid argument. List the leaf datasets in nas_sandbox_mounts. For a parent dataset, list its leaf children one by one.

Write access is DEFAULT idmap plus per-dataset ACLs (ADR Grant sandbox dataset access via the DEFAULT idmap plus per-dataset ACLs). The DEFAULT idmap maps container-root (uid 0 inside) to host UID 2147000001 (nas_sandbox_mapped_uid). The FILESYSTEM device mounts read-write at the VFS layer, and ZFS ACLs gate the writes. nas-sandbox-acls grants rwx to 2147000001 on each entry of nas_sandbox_curation_targets, by acltype:

  • POSIX1E datasets: setfacl -R -m u:2147000001:rwx, plus a default ACL for inheritance.
  • NFSv4 datasets: midclt filesystem.setacl appends a USER ACE with FULL_CONTROL and INHERIT, recursive. setfacl does not work on them.

The grant covers container-root, not the nas-automation login uid, which maps to a different host UID. Run curation tooling as root inside the container through sudo -i, a login shell that puts Homebrew on PATH: ssh nas-support 'sudo -i rmlint /mnt/main/multimedia'.

  1. Add the leaf dataset to both nas_sandbox_mounts and nas_sandbox_curation_targets in inventory/host_vars/nas.yml.
  2. Run --tags nas-sandbox-instance, which adds the mount. Restart the container, because a newly attached FILESYSTEM device mounts only at start. Then run --tags nas-sandbox-acls.

The precipice AresMUSH game runs in a libvirt-LXC that the nas-game-lxc role creates from ubuntu:resolute, with the MAC 00:16:3e:ae:0c:f1. The Kea reservation and the PowerDNS records in tf/dns-records/hosts.yml give it 192.168.22.109 and precipice.fzymgc.house. The engine listens on 4201 (telnet), 4202 (websocket) and 4203 (web API). Caddy serves the web portal at precipice.hound-skate.ts.net. The public route precipice.muxserver.net terminates on edge-node, a host outside this repository, and reaches the game over the tailnet.

The data is the single leaf dataset main/fzymgc-house/incus/storage/precipice. It is in nas_kopia_leaves, so the weekly snapshots and the Kopia to B2 backup cover the game directory, the database and the uploaded images. Two directories inside it bind-mount into the container (nas_game_mounts):

Source (on nas) Target (in container) Owner (container ids)
.../precipice/server /home/ares ares 2000:2000
.../precipice/database /var/lib/valkey valkey 102:103

The nas-aresmush role pins those uids and gids before it installs valkey-server. The package must adopt the existing ids, or the mounted database is unreadable under the DEFAULT idmap.

Secrets: secret/fzymgc-house/infrastructure/nas/precipice holds valkey_password, the requirepass of the game configuration. secret/fzymgc-house/infrastructure/nas/ares-tailscale-auth holds the tag-scoped Tailscale OAuth client (tailscale_oauth_client.ares_game in tf/tailscale, tag tag:ares-game-node).

Two constraints of tailscale in this unprivileged LXC:

  • Userspace networking is mandatory. The container.* middleware has no TUN passthrough, so /etc/default/tailscaled runs --tun=userspace-networking. Inbound tailnet traffic reaches caddy through netstack, and nothing on this host dials out over the tailnet.
  • The node registers tagged (tag:ares-game-node), never as a user device. The role reads tailscale status --json and registers again with the OAuth client whenever the node is logged out or lacks an expected tag. A tagged node has no node-key expiry. The tag’s grants stand in tailscale/policy.hujson, and caddy fetches the precipice.hound-skate.ts.net certificate once tailscale is up.
  • A registration that changes ownership from user to tagged creates a new device. Delete the old device in the admin console and reclaim the bare MagicDNS name with a hostname bounce (tailscale set --hostname=precipice-tmp, then back). None of this repository’s OAuth clients carries a devices scope, so that deletion is an operator action.

The engine runs as aresmush.service (a file in the role): oneshot, User=ares, bash --login so rvm’s ruby resolves. startares runs bundle install and rake migrate on every start, then deploys the Ember portal from ~/ares-webportal.

Terminal window
ssh precipice 'sudo systemctl restart aresmush' # engine restart
ssh precipice 'sudo systemctl status aresmush valkey-server caddy tailscaled'

Recovery: the dataset has ZFS snapshots. List them with zfs list -t snapshot -r main/fzymgc-house/incus/storage/precipice.

Do not test reachability from nas itself. Macvlan over bond0 blocks traffic between the host and its containers. Test nc 192.168.22.109 4201 from another LAN host.

The container vault-unseal runs one of the Vault unsealers. Vault Operations lists the others. The play NAS Vault unsealer container (instance lifecycle) in ansible/nas-playbook.yml includes the task files instance.yml and bootstrap.yml of the nas-game-lxc role, with these task vars:

Parameter Value
nas_game_name vault-unseal
nas_game_nic_mac 00:16:3e:20:02:04
nas_game_mounts []

Task vars outrank the nas_game_* values in ansible/inventory/host_vars/nas.yml, so the play leaves precipice unchanged. The play does not use the main.yml of the role, because that file tags every task nas-game-lxc. Only nas-vault-unseal-lxc selects this play, and a run for precipice does not touch the container. The image and the pool come from host_vars, the same as precipice. The task vars also set nas_game_nic_parent to vlan1, the Core VLAN interface of the NAS. So the container is a macvlan child on Core, the same as nas-support. The dns-control entry vault_unseal_nas reserves 192.168.40.34 on Core for the container MAC and publishes vault-unseal-nas.fzymgc.house. The inventory host is vault-unseal-nas. The container name in the middleware stays vault-unseal.

The container has no bind mount. Its root filesystem is on the apps pool, and no nas_kopia_leaves entry covers that pool. So the key shares in /etc/vault-unseal/vault-unseal.yaml never reach the Kopia to B2 backup.

The second play applies ansible/roles/vault-unseal to the group nas_vault_unseal_hosts. That role is the same role that runs on heimdall.

Converge the container in this order after the reservation is live on both Kea nodes. The first command creates vlan1, the NIC parent, and the instance play fails when that parent is missing:

Terminal window
scripts/nas-playbook.sh --tags nas-core-vlan
scripts/nas-playbook.sh --tags nas-vault-unseal-lxc
scripts/nas-playbook.sh --tags vault-unseal --limit vault-unseal-nas

Read the state of the unsealer:

Terminal window
scripts/nas-adhoc.sh vault-unseal-nas -b -m ansible.builtin.shell \
-a 'systemctl is-active vault-unseal; journalctl -u vault-unseal --since -15min --no-pager | tail -20'

To roll back, disable the unit, then revert the pull request. The container stays and runs no unsealer:

Terminal window
scripts/nas-adhoc.sh vault-unseal-nas -b -m ansible.builtin.command -a 'systemctl disable --now vault-unseal'

Do not test reachability from nas itself. Macvlan over bond0 blocks traffic between the host and its containers.

nas-sandbox-tools installs every curation tool through a Homebrew bundle (brew bundle). The package set is nas_brew_packages (default ncdu, rmlint, rclone, jq, kopia), rendered to /home/nas-automation/.Brewfile. Only Homebrew’s own bootstrap prerequisites (build-essential, curl, git and the rest of nas_brew_apt_prereqs) come from apt. Idempotence is gated on brew bundle check.

Homebrew installs under /home/linuxbrew/.linuxbrew. /etc/profile.d/homebrew.sh puts the tools on PATH for login shells: an interactive ssh nas-support session, and root through sudo -i <tool>. A one-shot ssh nas-support '<tool> ...' is not a login shell and does not find them. Use ssh nas-support 'sudo -i <tool> ...' or an interactive session.

To add a tool, append its Homebrew formula to nas_brew_packages in host_vars/nas.yml or the role defaults, then run --tags nas-sandbox-tools.

nas-otel-collector (--tags nas-otel) deploys otelcol-contrib as a TrueNAS custom app (app.*). It ships host metrics and journald logs to otel-gateway.fzymgc.house over mTLS into ClickStack (ADR Ship NAS telemetry via a dedicated collector).

  • mTLS identity: the app has a second container, the sidecar vault-agent. It runs Vault Agent from the image hashicorp/vault at vault_agent_version (group_vars/all.yml) with the AppRole cert-agent-nas-otelcol. The agent renders the certificate of the Vault PKI role otel-collector-nas-client (720h, CN otelcol-nas.fzymgc.house) as one bundle, /mnt/apps/otel-collector/certs/tls.pem, and renews it at half of its life. The collector reads the bundle again every hour. It starts only after the healthcheck of the sidecar finds the bundle. Host certificates describes the agent.
  • The sidecar runs as uid 0 with no capability and a read-only root file system. The role vault-cert-agent (tasks/container.yml) writes its AppRole files, its CA and agent.hcl in /mnt/apps/otel-collector/vault-agent. The AppRole files are 0600 for root. The image entrypoint changes to the user vault with su-exec, which needs CAP_SETUID, so the compose runs /bin/vault agent directly.
  • Certificate time left: a tls_check receiver reports tlscheck.time_left for the bundle every 5 minutes. The HyperDX dashboard Certificates shows it, and the alert Certificate expires in under 7 days pages on it.
  • The ingest token (otel_ingest_api_key from secret/fzymgc-house/cluster/clickstack) is rendered into the collector configuration. The compose file is the custom-app definition. app.update pushes the whole compose and restarts the app. It runs when the collector configuration, the compose, an AppRole file, the CA or agent.hcl changed.
  • The host root is mounted /:/hostfs:ro, and hostmetrics reads root_path: /hostfs (cpu, memory, load, disk, filesystem, network). Resource attributes set service.name=otelcol-nas and host.name=nas.
  • OTLP from apps: the collector also joins the nas-internal Docker network and receives OTLP over HTTP on port nas_otel_otlp_http_port (4318). No host port is published. An app on nas-internal sends to http://otelcol:4318. The resource/otlp processor sets host.name and keeps the service name the sender set. RustFS sends its metrics and logs this way as ServiceName='rustfs'. The collector keeps its default network for egress, because nas-internal has no route out.
  • Logs come from the journald receiver, which tails the units in nas_otel_journal_units (host_vars/nas.yml) at priority: info. It reads the persistent journal at /var/log/journal (mounted :ro) and resumes across restarts through a file_storage cursor under {{ nas_otel_base_dir }}/cursor (mounted :rw). /etc/machine-id is mounted for host identification. The transform/journald processor lifts the journal fields into attributes and sets the body to the MESSAGE text. There is no filter processor.

The image ghcr.io/fzymgc-house/nas-otel:<nas_otel_version> comes from .github/workflows/build-nas-otel.yml. That workflow calls .github/workflows/build-image.yml, which signs the image keyless with cosign. The certificate names build-image.yml, so the workflow name binds the signature to this image. Before you raise nas_otel_version, make sure that the signature is valid:

Terminal window
cosign verify ghcr.io/fzymgc-house/nas-otel:<version> \
--certificate-identity 'https://github.com/fzymgc-house/selfhosted-cluster/.github/workflows/build-image.yml@refs/heads/main' \
--certificate-github-workflow-name 'Build NAS OTel Image' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com

Deploy:

Terminal window
# dry-run first
export VAULT_TOKEN="$(vault print token)"
scripts/nas-playbook.sh --tags nas-otel --check --diff
# apply
scripts/nas-playbook.sh --tags nas-otel

The control-node Vault token must read the role id of the AppRole cert-agent-nas-otelcol and create its secret id. The first run creates the secret id, and later runs reuse the one on the NAS.

Verify on the NAS:

Terminal window
# Both containers run, and the status of ix-otelcol-nas-vault-agent-1 is healthy.
scripts/nas-adhoc.sh nas -b -m command -a 'docker ps --filter name=ix-otelcol-nas'
# The two hashes are equal when the certificate and the key in the bundle match.
scripts/nas-adhoc.sh nas -b -m shell -a 'b=/mnt/apps/otel-collector/certs/tls.pem; openssl x509 -noout -pubkey -in $b | openssl sha256; openssl pkey -pubout -in $b | openssl sha256; openssl x509 -noout -enddate -in $b'
# The agent log shows each render of the bundle.
scripts/nas-adhoc.sh nas -b -m command -a 'docker logs --since 1h ix-otelcol-nas-vault-agent-1'

Verify in ClickStack:

SELECT count() FROM otel_metrics_sum WHERE ServiceName = 'otelcol-nas'
SELECT count() FROM default.otel_logs WHERE ServiceName = 'otelcol-nas' AND Timestamp > now() - INTERVAL 1 HOUR

The sandbox host nas-support (192.168.40.32) runs its own collector, separate from the TrueNAS one. nas-support-otel-collector (--tags nas-support-otel-collector) deploys otelcol-contrib as a native systemd unit, otelcol-nas-support.service, with no container and no TrueNAS app machinery. It ships journald logs, host metrics and a backup-freshness series to otel-gateway.fzymgc.house over mTLS. The binary is SHA256-pinned against the official otelcol-contrib checksums and installed with get_url and a fail-closed checksum:. The whole deploy is Ansible-only.

  • mTLS identity: the role vault-cert-agent (--tags vault-cert-agent) runs a Vault Agent with the AppRole cert-agent-nas-support. The agent renders the certificate of the Vault PKI role otel-collector-nas-support-client (720h, CN otelcol-nas-support.fzymgc.house) as one bundle, /etc/otelcol-nas-support/certs/tls.pem, and renews it at half of its life. The collector reads the bundle again every hour. Host certificates describes the agent.
  • Certificate time left: a tls_check receiver reports tlscheck.time_left for the bundle every 5 minutes. The HyperDX dashboard Certificates shows it, and the alert Certificate expires in under 7 days pages on it.
  • Host metrics: host_metrics scrapes the LXC’s own lxcfs /proc with no root_path, because root_path breaks the filesystem scraper inside the container. Resource attributes set host.name=nas-support.
  • journald logs: the receiver live-tails the units in nas_support_otel_journal_units through journalctl, with no directory:. The transform/journald processor lifts the fields into attributes and sets the body to the MESSAGE text.
  • Backup freshness: a file_stats receiver emits file.mtime for the last-success heartbeat /var/lib/nas-kopia-b2/last-success. It is a non-monotonic Sum in epoch seconds, so it lands in otel_metrics_sum, not otel_metrics_gauge. Freshness age is now() - Value at query time.
  • Loopback OTLP ingest: an OTLP/HTTP receiver on 127.0.0.1:4318 accepts the end-of-run gauges that kopia-b2-backup pushes (nas.kopia.backup.duration.seconds, nas.kopia.snapshot.failed{source}). Off-box actors cannot reach it. The health_check extension binds 127.0.0.1:13133.

Deploy:

Terminal window
# dry-run first
export VAULT_TOKEN="$(vault print token)"
scripts/nas-playbook.sh --tags nas-support-otel-collector,vault-cert-agent --limit nas-support --check --diff
# apply
scripts/nas-playbook.sh --tags nas-support-otel-collector,vault-cert-agent --limit nas-support

Two preconditions: the target architecture must resolve the amd64 asset, or the get_url checksum fails loudly, and the control-node Vault token must read the role id of the AppRole cert-agent-nas-support and create its secret id. The first run creates the secret id, and later runs reuse the one on the box.

Verify on the box, then in ClickStack with the queries in ClickStack keyed on ResourceAttributes['host.name'] = 'nas-support':

Terminal window
scripts/nas-adhoc.sh nas-support -b -m command -a 'systemctl is-active otelcol-nas-support vault-cert-agent' # active active
scripts/nas-adhoc.sh nas-support -b -m command -a 'openssl x509 -noout -serial -enddate -in /etc/otelcol-nas-support/certs/tls.pem'
scripts/nas-adhoc.sh nas-support -m command -a 'curl -fsS http://127.0.0.1:13133/' # {"status":"Server available",...}

nas-forgejo (--tags nas-forgejo) deploys Forgejo as a TrueNAS custom app at https://git.nas.fzymgc.house. It hosts repositories that must not leave the house, such as the Obsidian vault. It is not a mirror of GitHub and it runs no Actions. The design is in docs/engineering/specs/2026-09-21-nas-forgejo-design.md.

The role owns four things:

  • The leaf dataset main/forgejo, created as its own encryption root with a TrueNAS-managed key, so it unlocks at boot. The role refuses an existing unencrypted dataset at that name.
  • The secret files secret_key and internal_token, read from secret/fzymgc-house/infrastructure/nas/forgejo and written under /mnt/main/forgejo/secrets as uid 568, mode 0600. The container reads them through FORGEJO__security__*__FILE. Generate both with docker run --rm codeberg.org/forgejo/forgejo:<tag> forgejo generate secret SECRET_KEY and ... INTERNAL_TOKEN on any machine with Docker.
  • The data directory /mnt/main/forgejo/data, mounted at /var/lib/gitea. The rootless image writes app.ini to custom/conf/app.ini under it, so there is no separate config mount.
  • The compose definition, pushed with app.create or app.update. An update fires only when the rendered compose or a secret file changed.

HTTPS goes through the NAS Traefik proxy on the nas-internal network with the cloudflare certificate resolver. SSH is the built-in server, published on host port 2222. Both use the one name, because the Traefik VIP is an alias on the NAS host interface.

Registration is off and every page needs a sign-in. Two steps run once, after the first converge, because Forgejo has no API for either:

  1. Create the accounts. Two exist: forgejo-admin is the break-glass administrator and never signs in through Keycloak, and sean is the everyday account that links to Keycloak. Generate each password on the control node and pass it as --password, so nothing prints. Store them in secret/fzymgc-house/infrastructure/nas/forgejo as admin_username, admin_password and user_sean_initial_password.

    Terminal window
    scripts/nas-adhoc.sh nas -b -m command \
    -a "docker exec ix-forgejo-forgejo-1 forgejo admin user create --admin --username forgejo-admin --email <admin email> --password $PW --must-change-password=false"
    scripts/nas-adhoc.sh nas -b -m command \
    -a "docker exec ix-forgejo-forgejo-1 forgejo admin user create --username sean --email <email> --password $PW2 --must-change-password=true"

    TrueNAS names the container ix-<app>-<service>-1, so the container is ix-forgejo-forgejo-1, not forgejo.

  2. Add the Keycloak sign-in at https://git.nas.fzymgc.house/admin/auths/new: type OAuth2, provider OpenID Connect, name keycloak, discovery URL https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration, client id and secret from secret/fzymgc-house/infrastructure/nas/forgejo/oidc. Then link the Keycloak identity to the admin account from the account settings page.

Back up the dataset key once. The appliance keystore holds it, and Vault holds the copy for a rebuild:

Terminal window
scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j pool.dataset.export_key main/forgejo'
vault kv put secret/fzymgc-house/infrastructure/nas/forgejo-zfs-key key=<hex>

Kopia backs up /mnt/main/forgejo with the other leaves. A new FILESYSTEM device mounts at container start, so restart nas-support after the first converge, then run scripts/nas-playbook.sh --tags nas-kopia-acls. Then run scripts/nas-playbook.sh --limit nas-support --tags nas-kopia-backup, which writes the list of leaves into the backup script. Without that run, the nightly backup skips the new leaf and still reports success. Kopia reads live data, so one nightly copy of the SQLite file can be torn. The main pool’s recursive snapshot tasks hold the consistent local copies.

Verify:

Terminal window
dig +short git.nas.fzymgc.house # 192.168.20.201
curl -sS -o /dev/null -w '%{http_code}\n' https://git.nas.fzymgc.house/api/healthz # 200
ssh -p 2222 git@git.nas.fzymgc.house # "Hi there, sean!"

The role nas-ha-backups creates the dataset main/home-assistant-backups and an NFS export of it. The export allows 192.168.20.220, the Home Assistant VM, and no other client. Every client user maps to nobody. The NFS service was already enabled and running, and the role changes no service state. The dataset is a leaf in nas_kopia_leaves, so Kopia copies it to B2.

A new leaf needs four steps, in this order:

  1. Run scripts/nas-playbook.sh --tags nas-ha-backups,nas-sandbox-instance. It creates the dataset, the export and the mount device of nas-support.
  2. Restart nas-support, because a new FILESYSTEM device mounts only at container start. Do it outside the 03:00 snapshot and the 12:00 full maintenance.
  3. Run scripts/nas-playbook.sh --tags nas-kopia-acls.
  4. Run scripts/nas-playbook.sh --limit nas-support --tags nas-kopia-backup, which writes the list of leaves into the backup script. Without this step, the nightly backup skips the new leaf and still reports success.

The Home Assistant side of the share is in Home Assistant.

Outbound email / alerts (Cloudflare Email Sending)

Section titled “Outbound email / alerts (Cloudflare Email Sending)”

TrueNAS sends its native alerts (SMART, scrub, pool health, update notices) over authenticated SMTP. The nas-mail role (--tags nas-mail) points that transport at Cloudflare Email Sending, the cluster standard (see Outbound email). The configuration goes through midclt call mail.update, never the UI.

  • Credential: a per-app Cloudflare API token (Email Sending: Edit), minted and written to Vault at secret/fzymgc-house/infrastructure/nas/cloudflare-email (fields cf_email_token and email_from) by tf/cloudflare/nas_email.tf. The role reads it on the control node over the operator’s Vault session and sets user=api_token, pass=<token> and fromemail=nas@hl.fzymgc.house.
  • Transport: smtp.mx.cloudflare.net:465 with security: SSL (implicit TLS). Cloudflare does not support STARTTLS on 587. fromemail must be on the onboarded hl.fzymgc.house domain, or Cloudflare rejects the send.
  • Recipient: the role sets the Email alert service’s recipient explicitly (nas_mail_alert_recipient, default sean@fzymgc.email). An empty recipient falls back to the admin user’s email fields without a warning.
  • Idempotent: the role compares mail.config field by field, and mail.update fires only on drift.

Two prerequisites: the tf/cloudflare apply that mints the token needs Email Sending: Edit on the Cloudflare bootstrap token, and the main-cluster-vault policy grant for infrastructure/nas/cloudflare-email must apply before main-cluster-cloudflare. See tf/cloudflare/CLAUDE.md.

Verify: send a test from the TrueNAS UI (System Settings, General, Email, Send Test Mail), then read the delivery in the Cloudflare dashboard (Email Service, Email Sending, Activity Log). The per-app token makes the NAS send attributable.

--tags nas-drift renders docs/reference/nas-config-baseline.md, a compact sorted projection of shares, datasets, snapshot tasks and apps. Commit it, so that configuration drift shows in version control.

Brew autoupdate timer (nas-support sandbox)

Section titled “Brew autoupdate timer (nas-support sandbox)”

The nas-sandbox-tools role installs a weekly systemd timer that refreshes the sandbox Homebrew bundle unattended. It is gated behind nas_brew_autoupdate, default true. Set it to false in host vars to disable it on one host.

  • Schedule: OnCalendar=Sun 04:00 (nas_brew_autoupdate_schedule) with RandomizedDelaySec=30m and Persistent=true, so a missed run replays at the next boot.
  • Command: brew update && brew bundle upgrade --file ~/.Brewfile && brew autoremove && brew cleanup, run as nas-automation through a login shell so that /etc/profile.d/homebrew.sh loads brew onto PATH.
  • Failure handling: OnFailure=brew-update-notify.service posts to the Pushover API with the cluster-infra app token, the failing unit name and the hostname. The token and user key live in a 0600 root-owned env file rendered from Vault at play time (nas_pushover_token_vault_path, nas_pushover_user_vault_path).

Run over SSH as nas-automation, or through scripts/nas-adhoc.sh nas-support -b -m command -a ...:

  1. systemctl list-timers brew-update*: the timer is listed, and NEXT is the next Sunday 04:00 plus jitter.
  2. systemctl start brew-update.service: a one-off run.
  3. systemctl status brew-update.service: Active: inactive (dead) with ExecStart= exit 0.
  4. Optional failure-path check: systemctl start brew-update-notify.service and confirm that a Pushover notification arrives. The notify unit is stateless, so nothing needs a revert.

Unit output lands in the journal on nas-support. Read it with journalctl -u brew-update.service. The .202 collector ships only the units in nas_support_otel_journal_units, so add the unit there to see it in ClickStack. The Pushover alert is the failure surface.

Kopia to B2 backup timers (nas-support sandbox)

Section titled “Kopia to B2 backup timers (nas-support sandbox)”

The nas-kopia-backup role runs three systemd timers on nas-support (192.168.40.32), spaced so that they never contend for Kopia’s single exclusive maintenance lock. The sources are the leaves in nas_kopia_leaves (inventory/group_vars/all.yml), mapped to their /mnt paths.

  • Snapshot at 03:00 (kopia-b2-backup.timer, nas_kopia_backup_oncalendar): the nightly snapshot of the irreplaceable datasets to B2, plus a quick maintenance pass. It writes the last-success heartbeat before maintenance, so freshness does not depend on maintenance duration.
  • Full maintenance at 12:00 (kopia-maint-full.timer, nas_kopia_full_maint_oncalendar): the daily kopia maintenance run --full (GC and pack compaction), off the snapshot’s critical path. Isolation from the snapshot unit rests on the timing separation and Kopia’s exclusive maintenance lock. There is no systemd force-stop directive, on purpose.
  • Freshness dead-man’s-switch, hourly (kopia-freshness.timer): compares the heartbeat age with nas_kopia_freshness_max_age_hours (36) and sends a Pushover alert when the last successful snapshot is older.

All three units source Pushover credentials from one root-owned 0600 pushover.env (rendered from Vault) through EnvironmentFile=. The token and user key never appear in the script bodies.

One value lives outside git. The role sets Kopia’s automatic full-maintenance interval with kopia maintenance set --full-interval (nas_kopia_full_interval, 168h), so an opportunistic full pass never fires inside a 03:00 snapshot. Kopia stores that value inside the B2 repository, so a role revert does not reset it. Read the live value with kopia maintenance info, which shows the full cycle interval and the root@nas-support owner.

Stale-alert triage (one unreadable path fails the whole leaf)

Section titled “Stale-alert triage (one unreadable path fails the whole leaf)”

Kopia treats one permission denied directory as a fatal error for that whole source. A failed source withholds the heartbeat, so the hourly dead-man’s-switch fires every hour until a clean run. Triage in this order:

  1. Which source, and since when: the ClickStack gauge nas.kopia.snapshot.failed (attribute source is the failing leaf). The wrapper’s log() lines are in otel_logs under ServiceName = 'otelcol-nas-support'.
  2. The Kopia error itself is only in the on-host log: scripts/nas-adhoc.sh nas-support -b -m shell -a 'tail -60 "$(ls -t /var/log/nas-kopia-b2/backup-*.log | head -1)"'

The usual cause is a path that the sandbox uid 2147000001 cannot read. The nas-kopia-acls walk grants rX access ACLs plus inheritable default ACLs on every directory, so a path created after the walk inherits the grant. A tree moved in with mv keeps its source ACLs and never inherits, for example a new user home created 700 with no ACL. The remedy is the same either way:

Terminal window
scripts/nas-playbook.sh --tags nas-kopia-acls # re-run the grant walk

Subtrees that the walk must skip are listed in nas_kopia_acl_prune_subpaths.