NAS (TrueNAS) Ansible Operations
Operational guide for the TrueNAS 26 nas host (192.168.20.200) under Ansible management. The
roles live under ansible/roles/nas-*, the host values in ansible/inventory/host_vars/nas.yml,
and the playbook is ansible/nas-playbook.yml.
Principle: manage the appliance through the middleware API only
Section titled “Principle: manage the appliance through the middleware API only”TrueNAS is an appliance. Its configuration lives in a database that reverts hand edits to
/etc. All host configuration goes through the middleware API (midclt call <method> ...) over
SSH with become, never through config files. ZFS and ACL operations use zfs, setfacl and
midclt filesystem.*. The decision is the ADR
Manage TrueNAS exclusively via the middleware API.
Containers are libvirt-LXC through the container.* middleware, because TrueNAS 26 has no Incus
(ADR Build NAS containers on libvirt-LXC).
Custom Docker workloads use app.*.
Connection model
Section titled “Connection model”| Phase | Connect as | Notes |
|---|---|---|
| One-time bootstrap | sean (-e ansible_ssh_user=sean -e ansible_become=true) |
nas-automation does not exist yet |
| Everything after | nas-automation (inventory default) |
the key is in Vault, not on disk |
The nas-automation private key lives in Vault at
secret/fzymgc-house/infrastructure/nas/automation-ssh, field private_key, not on the control
node. The SSH agent is not used. An agent holds many unrelated keys, and offering them all trips
sshd MaxAuthTries (“Too many authentication failures”) before the right key is reached. The
key is written to a temp file, and IdentitiesOnly with IdentityAgent=none makes ssh offer
only that key.
Preferred: the wrapper script
Section titled “Preferred: the wrapper script”scripts/nas-playbook.sh does all of this in one command. It checks the Vault session, writes
the key to a 0600 temp file that it removes on exit, and runs ansible-playbook with the
connection flags. Pass through any ansible-playbook arguments:
# dry-runscripts/nas-playbook.sh --tags nas-sandbox-tools --check --diff# applyscripts/nas-playbook.sh --tags nas-sandbox-tools# single hostscripts/nas-playbook.sh --tags nas-sandbox-tools --limit nas-supportPrerequisite: vault status succeeds. If it does not, run vault login or
scripts/vault-helper.sh login first.
Ad-hoc commands (one-off diagnostics)
Section titled “Ad-hoc commands (one-off diagnostics)”scripts/nas-adhoc.sh is the ad-hoc sibling. It handles the Vault key the same way and wraps
ansible <pattern> ... instead of ansible-playbook. Everything after the host pattern passes
through to ansible. Add -b when the target needs root, which most NAS diagnostics do:
# Kopia B2 repo maintenance state on the sandbox (dedicated config path; root):scripts/nas-adhoc.sh nas-support -b -m shell \ -a 'KOPIA_CONFIG_PATH=/etc/nas-kopia-b2/repository.config /home/linuxbrew/.linuxbrew/bin/kopia maintenance info'# Kopia timers armed + spaced:scripts/nas-adhoc.sh nas-support -b -m shell -a "systemctl list-timers 'kopia-*' --all"Two constraints: -m shell runs through the box’s dash, so use no bashisms and prefer
-m command when no shell is needed. Ad-hoc module arguments are Jinja-templated by Ansible, so
write no literal {{ }} in a command.
Manual fallback (if you need to run a different playbook)
Section titled “Manual fallback (if you need to run a different playbook)”vault kv get -mount=secret -field=private_key \ fzymgc-house/infrastructure/nas/automation-ssh > /tmp/nas.key && chmod 600 /tmp/nas.key
ansible-playbook -i inventory nas-playbook.yml --tags <tag> \ -e ansible_ssh_private_key_file=/tmp/nas.key \ -e "ansible_ssh_common_args='-o IdentitiesOnly=yes -o IdentityAgent=none'"The inner single quotes around the -o arguments are required. They keep the arguments together
as one value for Ansible’s k=v parser. Without them, the parser splits on the spaces and fails
with argument -o: expected one argument. The wrapper hard-codes the quoted form for this reason.
nas-automation has NOPASSWD: ALL sudo. Ansible become wraps commands in
sudo ... /bin/sh -c '...', so a per-binary allowlist cannot match. midclt is already the
appliance root API, so a narrower allowlist adds no isolation. The credential boundary is the
Vault-held key.
Bootstrap order (first run)
Section titled “Bootstrap order (first run)”Seed Vault first: secret/fzymgc-house/infrastructure/nas/automation-ssh (an ed25519 keypair).
cd ansible# 1. Create the automation user (as sean)ansible-playbook -i inventory nas-playbook.yml --tags nas-automation-user \ -e ansible_ssh_user=sean -e ansible_become=true# 2+. Everything else runs as nas-automation (see Connection model)Tag map
Section titled “Tag map”| Tag | What it does | Target |
|---|---|---|
nas-automation-user |
Reconcile the nas-automation middleware user |
nas |
nas-core-vlan |
Create the VLAN interface vlan1 on bond0 for Core, with no address, as the macvlan parent of the Core containers |
nas |
nas-drift |
Render docs/reference/nas-config-baseline.md from middleware queries |
nas |
nas-docker-networks |
Set the Docker address pools and the shared app networks nas-internal and nas-public |
nas |
nas-otel |
Deploy the otelcol-contrib custom app (host metrics and journald logs to ClickStack over mTLS) and its Vault Agent sidecar | nas |
nas-tailscale |
Deploy the Tailscale custom app (tailnet node truenas-scale, subnet routes and exit node) |
nas |
nas-rustfs |
Deploy the RustFS custom app, its Vault Agent sidecar and its encrypted data dataset. See RustFS | nas |
nas-forgejo |
Deploy the Forgejo custom app (private git hosting at git.nas.fzymgc.house) |
nas |
nas-ha-backups |
The dataset main/home-assistant-backups and its NFS export for the Home Assistant VM. See Home Assistant |
nas |
nas-mail |
Point native alert email at Cloudflare Email Sending and set the Email alert recipient | nas |
nas-sandbox-instance |
Create and start the nas-support LXC, its NIC and its mounts |
nas |
nas-sandbox-bootstrap |
Through nsenter: in-container user, sshd and key |
nas |
nas-sandbox-acls |
Grant container-root read-write on the curation targets | nas |
nas-kopia-acls |
Grant container-root read-only on every Kopia leaf | nas |
nas-sandbox-tools |
Interior curation tooling and the brew autoupdate timer | nas_sandbox_hosts |
nas-kopia-backup |
The Kopia to B2 backup, maintenance and freshness timers | nas_sandbox_hosts |
nas-support-otel-collector |
The native otelcol-contrib unit on the sandbox | nas_sandbox_hosts |
nas-game-lxc-instance |
Create and start the precipice LXC, its NIC and its mounts |
nas |
nas-game-lxc-bootstrap |
Through nsenter: in-container user, sshd and key |
nas |
nas-aresmush |
Interior: users, valkey, caddy, tailscale, the engine unit | nas_game_hosts |
nas-vault-unseal-lxc |
Create and start the vault-unseal LXC and its NIC, and bootstrap it through nsenter. nas-game-lxc does not select it |
nas |
vault-unseal |
Interior: the vault-unseal binary, its configuration and its unit |
nas_vault_unseal_hosts |
Docker address pools and shared app networks
Section titled “Docker address pools and shared app networks”TrueNAS gives every app network a subnet from its Docker address pools. The nas-docker-networks
tag sets the pools and creates the two shared networks. The values live in
ansible/inventory/group_vars/nas_hosts.yml.
| Item | Value |
|---|---|
| IPv4 pool | 172.16.0.0/12, one /24 per network |
| IPv6 pool | fdd0::/48, one /64 per network |
nas-internal |
172.16.100.0/24 and fdd0:0:0:100::/64, internal, no route out |
nas-public |
172.16.101.0/24 and fdd0:0:0:101::/64 |
Two rules protect the pools:
- Every base must be an aligned prefix. Docker skips an unaligned base such as
172.17.0.0/12without a log line and takes the next pool. - No pool can sit inside
100.64.0.0/10. The Tailscale app drops any packet from that range that does not arrive ontailscale0. With app networks in that range, a published port such as192.168.20.201:443fails for tunnel clients and for the NAS itself, while wired LAN clients still work. Ping still answers, because the address is a host alias.
A pool change restarts Docker and every app, the Tailscale app included. When the NAS’s Tailscale goes down, the tailnet moves the subnet routes to another advertiser and does not move them back on its own. After the apps return, follow Fail the routes back to the NAS in Tailscale Subnet Routers. Existing networks keep their subnets. To move a shared network to a new subnet:
-
Stop the apps on it:
midclt call app.stop <name>fortraefik,forgejo,rustfs,dozzleandotelcol-nas. -
Run the tag with force. Force disconnects the containers and recreates the network.
Terminal window scripts/nas-playbook.sh --tags nas-docker-networks -e nas_docker_networks_force=true -
Start the apps again with
midclt call app.start <name>. -
Run
scripts/nas-playbook.sh --tags nas-rustfs, because RustFS trusts the proxies by subnet.
To make sure that no network is left in the old range, run this on the NAS:
docker network ls -q | xargs -n1 docker network inspect --format '{{.Name}} {{range .IPAM.Config}}{{.Subnet}} {{end}}'The tooling sandbox (nas-support)
Section titled “The tooling sandbox (nas-support)”A libvirt-LXC (container.*, DEFAULT idmap) on macvlan over vlan1, the Core VLAN interface of
the NAS, with the fixed MAC 00:16:3e:20:02:02. The dns-control entry nas_support reserves
192.168.40.32 on Core for that MAC and publishes the name nas-support.fzymgc.house.
vlan1 holds no address. The NAS reaches Core through the gateway. A frame for the container
arrives on vlan1 from the wire, and macvlan hands it to the container. A host address on vlan1
routes Core on-link instead. Macvlan never passes a frame from the host’s own stack to a child on
the same parent. So that address cuts the container off from the host’s forwarding. Without it,
the container is reachable from a tailnet client through truenas-scale. It stays unreachable
from nas itself. The tag nas-core-vlan of nas-common creates the interface. Run it before
nas-sandbox-instance, which fails when the parent is missing.
Mounts are leaf datasets only. A FILESYSTEM device can bind-mount only a leaf dataset. A pool
root (/mnt/main, /mnt/apps) or a dataset whose subtree holds other ZFS mountpoints fails at
start with Failed to bind mount ... Invalid argument. List the leaf datasets in
nas_sandbox_mounts. For a parent dataset, list its leaf children one by one.
Write access is DEFAULT idmap plus per-dataset ACLs (ADR
Grant sandbox dataset access via the DEFAULT idmap plus per-dataset ACLs).
The DEFAULT idmap maps container-root (uid 0 inside) to host UID 2147000001
(nas_sandbox_mapped_uid). The FILESYSTEM device mounts read-write at the VFS layer, and ZFS ACLs
gate the writes. nas-sandbox-acls grants rwx to 2147000001 on each entry of
nas_sandbox_curation_targets, by acltype:
- POSIX1E datasets:
setfacl -R -m u:2147000001:rwx, plus a default ACL for inheritance. - NFSv4 datasets:
midclt filesystem.setaclappends a USER ACE withFULL_CONTROLandINHERIT, recursive.setfacldoes not work on them.
The grant covers container-root, not the nas-automation login uid, which maps to a different
host UID. Run curation tooling as root inside the container through sudo -i, a login shell that
puts Homebrew on PATH: ssh nas-support 'sudo -i rmlint /mnt/main/multimedia'.
Add a curation target
Section titled “Add a curation target”- Add the leaf dataset to both
nas_sandbox_mountsandnas_sandbox_curation_targetsininventory/host_vars/nas.yml. - Run
--tags nas-sandbox-instance, which adds the mount. Restart the container, because a newly attached FILESYSTEM device mounts only at start. Then run--tags nas-sandbox-acls.
AresMUSH game host (precipice)
Section titled “AresMUSH game host (precipice)”The precipice AresMUSH game runs in a libvirt-LXC that the nas-game-lxc role creates from
ubuntu:resolute, with the MAC 00:16:3e:ae:0c:f1. The Kea reservation and the PowerDNS records
in tf/dns-records/hosts.yml give it 192.168.22.109 and precipice.fzymgc.house. The engine
listens on 4201 (telnet), 4202 (websocket) and 4203 (web API). Caddy serves the web portal
at precipice.hound-skate.ts.net. The public route precipice.muxserver.net terminates on
edge-node, a host outside this repository, and reaches the game over the tailnet.
The data is the single leaf dataset main/fzymgc-house/incus/storage/precipice. It is in
nas_kopia_leaves, so the weekly snapshots and the Kopia to B2 backup cover the game directory,
the database and the uploaded images. Two directories inside it bind-mount into the container
(nas_game_mounts):
| Source (on nas) | Target (in container) | Owner (container ids) |
|---|---|---|
.../precipice/server |
/home/ares |
ares 2000:2000 |
.../precipice/database |
/var/lib/valkey |
valkey 102:103 |
The nas-aresmush role pins those uids and gids before it installs valkey-server. The package
must adopt the existing ids, or the mounted database is unreadable under the DEFAULT idmap.
Secrets: secret/fzymgc-house/infrastructure/nas/precipice holds valkey_password, the
requirepass of the game configuration. secret/fzymgc-house/infrastructure/nas/ares-tailscale-auth
holds the tag-scoped Tailscale OAuth client (tailscale_oauth_client.ares_game in tf/tailscale,
tag tag:ares-game-node).
Two constraints of tailscale in this unprivileged LXC:
- Userspace networking is mandatory. The
container.*middleware has no TUN passthrough, so/etc/default/tailscaledruns--tun=userspace-networking. Inbound tailnet traffic reaches caddy through netstack, and nothing on this host dials out over the tailnet. - The node registers tagged (
tag:ares-game-node), never as a user device. The role readstailscale status --jsonand registers again with the OAuth client whenever the node is logged out or lacks an expected tag. A tagged node has no node-key expiry. The tag’s grants stand intailscale/policy.hujson, and caddy fetches theprecipice.hound-skate.ts.netcertificate once tailscale is up. - A registration that changes ownership from user to tagged creates a new device. Delete the old
device in the admin console and reclaim the bare MagicDNS name with a hostname bounce
(
tailscale set --hostname=precipice-tmp, then back). None of this repository’s OAuth clients carries adevicesscope, so that deletion is an operator action.
The engine runs as aresmush.service (a file in the role): oneshot, User=ares, bash --login
so rvm’s ruby resolves. startares runs bundle install and rake migrate on every start, then
deploys the Ember portal from ~/ares-webportal.
ssh precipice 'sudo systemctl restart aresmush' # engine restartssh precipice 'sudo systemctl status aresmush valkey-server caddy tailscaled'Recovery: the dataset has ZFS snapshots. List them with
zfs list -t snapshot -r main/fzymgc-house/incus/storage/precipice.
Do not test reachability from nas itself. Macvlan over bond0 blocks traffic between the host
and its containers. Test nc 192.168.22.109 4201 from another LAN host.
Vault unsealer container (vault-unseal)
Section titled “Vault unsealer container (vault-unseal)”The container vault-unseal runs one of the Vault unsealers. Vault Operations
lists the others. The play NAS Vault unsealer container (instance lifecycle) in
ansible/nas-playbook.yml includes the task files instance.yml and bootstrap.yml of the
nas-game-lxc role, with these task vars:
| Parameter | Value |
|---|---|
nas_game_name |
vault-unseal |
nas_game_nic_mac |
00:16:3e:20:02:04 |
nas_game_mounts |
[] |
Task vars outrank the nas_game_* values in ansible/inventory/host_vars/nas.yml, so the play
leaves precipice unchanged. The play does not use the main.yml of the role, because that file
tags every task nas-game-lxc. Only nas-vault-unseal-lxc selects this play, and a run for
precipice does not touch the container. The image and the pool come from host_vars, the same as
precipice. The task vars also set nas_game_nic_parent to vlan1, the Core VLAN interface of
the NAS. So the container is a macvlan child on Core, the same as nas-support. The dns-control
entry vault_unseal_nas reserves 192.168.40.34 on Core for the container MAC and publishes
vault-unseal-nas.fzymgc.house. The inventory host is vault-unseal-nas. The container name in
the middleware stays vault-unseal.
The container has no bind mount. Its root filesystem is on the apps pool, and no
nas_kopia_leaves entry covers that pool. So the key shares in
/etc/vault-unseal/vault-unseal.yaml never reach the Kopia to B2 backup.
The second play applies ansible/roles/vault-unseal to the group nas_vault_unseal_hosts. That
role is the same role that runs on heimdall.
Converge the container in this order after the reservation is live on both Kea nodes. The first
command creates vlan1, the NIC parent, and the instance play fails when that parent is missing:
scripts/nas-playbook.sh --tags nas-core-vlanscripts/nas-playbook.sh --tags nas-vault-unseal-lxcscripts/nas-playbook.sh --tags vault-unseal --limit vault-unseal-nasRead the state of the unsealer:
scripts/nas-adhoc.sh vault-unseal-nas -b -m ansible.builtin.shell \ -a 'systemctl is-active vault-unseal; journalctl -u vault-unseal --since -15min --no-pager | tail -20'To roll back, disable the unit, then revert the pull request. The container stays and runs no unsealer:
scripts/nas-adhoc.sh vault-unseal-nas -b -m ansible.builtin.command -a 'systemctl disable --now vault-unseal'Do not test reachability from nas itself. Macvlan over bond0 blocks traffic between the host
and its containers.
Interior tooling
Section titled “Interior tooling”nas-sandbox-tools installs every curation tool through a Homebrew bundle (brew bundle). The
package set is nas_brew_packages (default ncdu, rmlint, rclone, jq, kopia), rendered
to /home/nas-automation/.Brewfile. Only Homebrew’s own bootstrap prerequisites
(build-essential, curl, git and the rest of nas_brew_apt_prereqs) come from apt.
Idempotence is gated on brew bundle check.
Homebrew installs under /home/linuxbrew/.linuxbrew. /etc/profile.d/homebrew.sh puts the tools
on PATH for login shells: an interactive ssh nas-support session, and root through
sudo -i <tool>. A one-shot ssh nas-support '<tool> ...' is not a login shell and does not find
them. Use ssh nas-support 'sudo -i <tool> ...' or an interactive session.
To add a tool, append its Homebrew formula to nas_brew_packages in host_vars/nas.yml or the
role defaults, then run --tags nas-sandbox-tools.
Telemetry
Section titled “Telemetry”nas-otel-collector (--tags nas-otel) deploys otelcol-contrib as a TrueNAS custom app
(app.*). It ships host metrics and journald logs to otel-gateway.fzymgc.house over mTLS into
ClickStack (ADR
Ship NAS telemetry via a dedicated collector).
- mTLS identity: the app has a second container, the sidecar
vault-agent. It runs Vault Agent from the imagehashicorp/vaultatvault_agent_version(group_vars/all.yml) with the AppRolecert-agent-nas-otelcol. The agent renders the certificate of the Vault PKI roleotel-collector-nas-client(720h, CNotelcol-nas.fzymgc.house) as one bundle,/mnt/apps/otel-collector/certs/tls.pem, and renews it at half of its life. The collector reads the bundle again every hour. It starts only after the healthcheck of the sidecar finds the bundle. Host certificates describes the agent. - The sidecar runs as uid 0 with no capability and a read-only root file system. The role
vault-cert-agent(tasks/container.yml) writes its AppRole files, its CA andagent.hclin/mnt/apps/otel-collector/vault-agent. The AppRole files are 0600 for root. The image entrypoint changes to the uservaultwithsu-exec, which needsCAP_SETUID, so the compose runs/bin/vault agentdirectly. - Certificate time left: a
tls_checkreceiver reportstlscheck.time_leftfor the bundle every 5 minutes. The HyperDX dashboardCertificatesshows it, and the alertCertificate expires in under 7 dayspages on it. - The ingest token (
otel_ingest_api_keyfromsecret/fzymgc-house/cluster/clickstack) is rendered into the collector configuration. The compose file is the custom-app definition.app.updatepushes the whole compose and restarts the app. It runs when the collector configuration, the compose, an AppRole file, the CA oragent.hclchanged. - The host root is mounted
/:/hostfs:ro, and hostmetrics readsroot_path: /hostfs(cpu, memory, load, disk, filesystem, network). Resource attributes setservice.name=otelcol-nasandhost.name=nas. - OTLP from apps: the collector also joins the
nas-internalDocker network and receives OTLP over HTTP on portnas_otel_otlp_http_port(4318). No host port is published. An app onnas-internalsends tohttp://otelcol:4318. Theresource/otlpprocessor setshost.nameand keeps the service name the sender set. RustFS sends its metrics and logs this way asServiceName='rustfs'. The collector keeps itsdefaultnetwork for egress, becausenas-internalhas no route out. - Logs come from the journald receiver, which tails the units in
nas_otel_journal_units(host_vars/nas.yml) atpriority: info. It reads the persistent journal at/var/log/journal(mounted:ro) and resumes across restarts through afile_storagecursor under{{ nas_otel_base_dir }}/cursor(mounted:rw)./etc/machine-idis mounted for host identification. Thetransform/journaldprocessor lifts the journal fields into attributes and sets the body to theMESSAGEtext. There is no filter processor.
The image ghcr.io/fzymgc-house/nas-otel:<nas_otel_version> comes from
.github/workflows/build-nas-otel.yml. That workflow calls .github/workflows/build-image.yml,
which signs the image keyless with cosign. The certificate names build-image.yml, so the
workflow name binds the signature to this image. Before you raise nas_otel_version, make sure
that the signature is valid:
cosign verify ghcr.io/fzymgc-house/nas-otel:<version> \ --certificate-identity 'https://github.com/fzymgc-house/selfhosted-cluster/.github/workflows/build-image.yml@refs/heads/main' \ --certificate-github-workflow-name 'Build NAS OTel Image' \ --certificate-oidc-issuer https://token.actions.githubusercontent.comDeploy:
# dry-run firstexport VAULT_TOKEN="$(vault print token)"scripts/nas-playbook.sh --tags nas-otel --check --diff
# applyscripts/nas-playbook.sh --tags nas-otelThe control-node Vault token must read the role id of the AppRole cert-agent-nas-otelcol and
create its secret id. The first run creates the secret id, and later runs reuse the one on the NAS.
Verify on the NAS:
# Both containers run, and the status of ix-otelcol-nas-vault-agent-1 is healthy.scripts/nas-adhoc.sh nas -b -m command -a 'docker ps --filter name=ix-otelcol-nas'
# The two hashes are equal when the certificate and the key in the bundle match.scripts/nas-adhoc.sh nas -b -m shell -a 'b=/mnt/apps/otel-collector/certs/tls.pem; openssl x509 -noout -pubkey -in $b | openssl sha256; openssl pkey -pubout -in $b | openssl sha256; openssl x509 -noout -enddate -in $b'
# The agent log shows each render of the bundle.scripts/nas-adhoc.sh nas -b -m command -a 'docker logs --since 1h ix-otelcol-nas-vault-agent-1'Verify in ClickStack:
SELECT count() FROM otel_metrics_sum WHERE ServiceName = 'otelcol-nas'SELECT count() FROM default.otel_logs WHERE ServiceName = 'otelcol-nas' AND Timestamp > now() - INTERVAL 1 HOURnas-support collector
Section titled “nas-support collector”The sandbox host nas-support (192.168.40.32) runs its own collector, separate from the
TrueNAS one. nas-support-otel-collector (--tags nas-support-otel-collector) deploys
otelcol-contrib as a native systemd unit, otelcol-nas-support.service, with no container and no
TrueNAS app machinery. It ships journald logs, host metrics and a backup-freshness series to
otel-gateway.fzymgc.house over mTLS. The binary is SHA256-pinned against the official
otelcol-contrib checksums and installed with get_url and a fail-closed checksum:. The whole
deploy is Ansible-only.
- mTLS identity: the role
vault-cert-agent(--tags vault-cert-agent) runs a Vault Agent with the AppRolecert-agent-nas-support. The agent renders the certificate of the Vault PKI roleotel-collector-nas-support-client(720h, CNotelcol-nas-support.fzymgc.house) as one bundle,/etc/otelcol-nas-support/certs/tls.pem, and renews it at half of its life. The collector reads the bundle again every hour. Host certificates describes the agent. - Certificate time left: a
tls_checkreceiver reportstlscheck.time_leftfor the bundle every 5 minutes. The HyperDX dashboardCertificatesshows it, and the alertCertificate expires in under 7 dayspages on it. - Host metrics:
host_metricsscrapes the LXC’s own lxcfs/procwith noroot_path, becauseroot_pathbreaks the filesystem scraper inside the container. Resource attributes sethost.name=nas-support. - journald logs: the receiver live-tails the units in
nas_support_otel_journal_unitsthroughjournalctl, with nodirectory:. Thetransform/journaldprocessor lifts the fields into attributes and sets the body to theMESSAGEtext. - Backup freshness: a
file_statsreceiver emitsfile.mtimefor the last-success heartbeat/var/lib/nas-kopia-b2/last-success. It is a non-monotonic Sum in epoch seconds, so it lands inotel_metrics_sum, nototel_metrics_gauge. Freshness age isnow() - Valueat query time. - Loopback OTLP ingest: an OTLP/HTTP receiver on
127.0.0.1:4318accepts the end-of-run gauges thatkopia-b2-backuppushes (nas.kopia.backup.duration.seconds,nas.kopia.snapshot.failed{source}). Off-box actors cannot reach it. Thehealth_checkextension binds127.0.0.1:13133.
Deploy:
# dry-run firstexport VAULT_TOKEN="$(vault print token)"scripts/nas-playbook.sh --tags nas-support-otel-collector,vault-cert-agent --limit nas-support --check --diff
# applyscripts/nas-playbook.sh --tags nas-support-otel-collector,vault-cert-agent --limit nas-supportTwo preconditions: the target architecture must resolve the amd64 asset, or the get_url
checksum fails loudly, and the control-node Vault token must read the role id of the AppRole
cert-agent-nas-support and create its secret id. The first run creates the secret id, and later
runs reuse the one on the box.
Verify on the box, then in ClickStack with the queries in ClickStack
keyed on ResourceAttributes['host.name'] = 'nas-support':
scripts/nas-adhoc.sh nas-support -b -m command -a 'systemctl is-active otelcol-nas-support vault-cert-agent' # active activescripts/nas-adhoc.sh nas-support -b -m command -a 'openssl x509 -noout -serial -enddate -in /etc/otelcol-nas-support/certs/tls.pem'scripts/nas-adhoc.sh nas-support -m command -a 'curl -fsS http://127.0.0.1:13133/' # {"status":"Server available",...}Forgejo (private git hosting)
Section titled “Forgejo (private git hosting)”nas-forgejo (--tags nas-forgejo) deploys Forgejo as a TrueNAS custom app at
https://git.nas.fzymgc.house. It hosts repositories that must not leave the house, such as the
Obsidian vault. It is not a mirror of GitHub and it runs no Actions. The design is in
docs/engineering/specs/2026-09-21-nas-forgejo-design.md.
The role owns four things:
- The leaf dataset
main/forgejo, created as its own encryption root with a TrueNAS-managed key, so it unlocks at boot. The role refuses an existing unencrypted dataset at that name. - The secret files
secret_keyandinternal_token, read fromsecret/fzymgc-house/infrastructure/nas/forgejoand written under/mnt/main/forgejo/secretsas uid 568, mode 0600. The container reads them throughFORGEJO__security__*__FILE. Generate both withdocker run --rm codeberg.org/forgejo/forgejo:<tag> forgejo generate secret SECRET_KEYand... INTERNAL_TOKENon any machine with Docker. - The data directory
/mnt/main/forgejo/data, mounted at/var/lib/gitea. The rootless image writesapp.initocustom/conf/app.iniunder it, so there is no separate config mount. - The compose definition, pushed with
app.createorapp.update. An update fires only when the rendered compose or a secret file changed.
HTTPS goes through the NAS Traefik proxy on the nas-internal network with the cloudflare
certificate resolver. SSH is the built-in server, published on host port 2222. Both use the one
name, because the Traefik VIP is an alias on the NAS host interface.
Registration is off and every page needs a sign-in. Two steps run once, after the first converge, because Forgejo has no API for either:
-
Create the accounts. Two exist:
forgejo-adminis the break-glass administrator and never signs in through Keycloak, andseanis the everyday account that links to Keycloak. Generate each password on the control node and pass it as--password, so nothing prints. Store them insecret/fzymgc-house/infrastructure/nas/forgejoasadmin_username,admin_passwordanduser_sean_initial_password.Terminal window scripts/nas-adhoc.sh nas -b -m command \-a "docker exec ix-forgejo-forgejo-1 forgejo admin user create --admin --username forgejo-admin --email <admin email> --password $PW --must-change-password=false"scripts/nas-adhoc.sh nas -b -m command \-a "docker exec ix-forgejo-forgejo-1 forgejo admin user create --username sean --email <email> --password $PW2 --must-change-password=true"TrueNAS names the container
ix-<app>-<service>-1, so the container isix-forgejo-forgejo-1, notforgejo. -
Add the Keycloak sign-in at
https://git.nas.fzymgc.house/admin/auths/new: type OAuth2, provider OpenID Connect, namekeycloak, discovery URLhttps://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration, client id and secret fromsecret/fzymgc-house/infrastructure/nas/forgejo/oidc. Then link the Keycloak identity to the admin account from the account settings page.
Back up the dataset key once. The appliance keystore holds it, and Vault holds the copy for a rebuild:
scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j pool.dataset.export_key main/forgejo'vault kv put secret/fzymgc-house/infrastructure/nas/forgejo-zfs-key key=<hex>Kopia backs up /mnt/main/forgejo with the other leaves. A new FILESYSTEM device mounts at
container start, so restart nas-support after the first converge, then run
scripts/nas-playbook.sh --tags nas-kopia-acls. Then run
scripts/nas-playbook.sh --limit nas-support --tags nas-kopia-backup, which writes the list of
leaves into the backup script. Without that run, the nightly backup skips the new leaf and still
reports success. Kopia reads live data, so one nightly copy of the SQLite file can be torn. The
main pool’s recursive snapshot tasks hold the consistent local copies.
Verify:
dig +short git.nas.fzymgc.house # 192.168.20.201curl -sS -o /dev/null -w '%{http_code}\n' https://git.nas.fzymgc.house/api/healthz # 200ssh -p 2222 git@git.nas.fzymgc.house # "Hi there, sean!"Home Assistant backups (nas-ha-backups)
Section titled “Home Assistant backups (nas-ha-backups)”The role nas-ha-backups creates the dataset main/home-assistant-backups and an NFS export of it.
The export allows 192.168.20.220, the Home Assistant VM, and no other client. Every client user
maps to nobody. The NFS service was already enabled and running, and the role changes no service
state. The dataset is a leaf in nas_kopia_leaves, so Kopia copies it to B2.
A new leaf needs four steps, in this order:
- Run
scripts/nas-playbook.sh --tags nas-ha-backups,nas-sandbox-instance. It creates the dataset, the export and the mount device ofnas-support. - Restart
nas-support, because a new FILESYSTEM device mounts only at container start. Do it outside the 03:00 snapshot and the 12:00 full maintenance. - Run
scripts/nas-playbook.sh --tags nas-kopia-acls. - Run
scripts/nas-playbook.sh --limit nas-support --tags nas-kopia-backup, which writes the list of leaves into the backup script. Without this step, the nightly backup skips the new leaf and still reports success.
The Home Assistant side of the share is in Home Assistant.
Outbound email / alerts (Cloudflare Email Sending)
Section titled “Outbound email / alerts (Cloudflare Email Sending)”TrueNAS sends its native alerts (SMART, scrub, pool health, update notices) over authenticated
SMTP. The nas-mail role (--tags nas-mail) points that transport at Cloudflare Email Sending,
the cluster standard (see Outbound email). The configuration goes
through midclt call mail.update, never the UI.
- Credential: a per-app Cloudflare API token (
Email Sending: Edit), minted and written to Vault atsecret/fzymgc-house/infrastructure/nas/cloudflare-email(fieldscf_email_tokenandemail_from) bytf/cloudflare/nas_email.tf. The role reads it on the control node over the operator’s Vault session and setsuser=api_token,pass=<token>andfromemail=nas@hl.fzymgc.house. - Transport:
smtp.mx.cloudflare.net:465withsecurity: SSL(implicit TLS). Cloudflare does not support STARTTLS on 587.fromemailmust be on the onboardedhl.fzymgc.housedomain, or Cloudflare rejects the send. - Recipient: the role sets the
Emailalert service’s recipient explicitly (nas_mail_alert_recipient, defaultsean@fzymgc.email). An empty recipient falls back to the admin user’s email fields without a warning. - Idempotent: the role compares
mail.configfield by field, andmail.updatefires only on drift.
Two prerequisites: the tf/cloudflare apply that mints the token needs Email Sending: Edit on
the Cloudflare bootstrap token, and the main-cluster-vault policy grant for
infrastructure/nas/cloudflare-email must apply before main-cluster-cloudflare. See
tf/cloudflare/CLAUDE.md.
Verify: send a test from the TrueNAS UI (System Settings, General, Email, Send Test Mail), then read the delivery in the Cloudflare dashboard (Email Service, Email Sending, Activity Log). The per-app token makes the NAS send attributable.
Drift baseline
Section titled “Drift baseline”--tags nas-drift renders docs/reference/nas-config-baseline.md, a compact sorted projection of
shares, datasets, snapshot tasks and apps. Commit it, so that configuration drift shows in version
control.
Brew autoupdate timer (nas-support sandbox)
Section titled “Brew autoupdate timer (nas-support sandbox)”The nas-sandbox-tools role installs a weekly systemd timer that refreshes the sandbox Homebrew
bundle unattended. It is gated behind nas_brew_autoupdate, default true. Set it to false in
host vars to disable it on one host.
- Schedule:
OnCalendar=Sun 04:00(nas_brew_autoupdate_schedule) withRandomizedDelaySec=30mandPersistent=true, so a missed run replays at the next boot. - Command:
brew update && brew bundle upgrade --file ~/.Brewfile && brew autoremove && brew cleanup, run asnas-automationthrough a login shell so that/etc/profile.d/homebrew.shloads brew ontoPATH. - Failure handling:
OnFailure=brew-update-notify.serviceposts to the Pushover API with thecluster-infraapp token, the failing unit name and the hostname. The token and user key live in a0600root-owned env file rendered from Vault at play time (nas_pushover_token_vault_path,nas_pushover_user_vault_path).
Verify
Section titled “Verify”Run over SSH as nas-automation, or through scripts/nas-adhoc.sh nas-support -b -m command -a ...:
systemctl list-timers brew-update*: the timer is listed, andNEXTis the next Sunday 04:00 plus jitter.systemctl start brew-update.service: a one-off run.systemctl status brew-update.service:Active: inactive (dead)withExecStart=exit 0.- Optional failure-path check:
systemctl start brew-update-notify.serviceand confirm that a Pushover notification arrives. The notify unit is stateless, so nothing needs a revert.
Unit output lands in the journal on nas-support. Read it with journalctl -u brew-update.service.
The .202 collector ships only the units in nas_support_otel_journal_units, so add the unit
there to see it in ClickStack. The Pushover alert is the failure surface.
Kopia to B2 backup timers (nas-support sandbox)
Section titled “Kopia to B2 backup timers (nas-support sandbox)”The nas-kopia-backup role runs three systemd timers on nas-support (192.168.40.32), spaced
so that they never contend for Kopia’s single exclusive maintenance lock. The sources are the
leaves in nas_kopia_leaves (inventory/group_vars/all.yml), mapped to their /mnt paths.
- Snapshot at 03:00 (
kopia-b2-backup.timer,nas_kopia_backup_oncalendar): the nightly snapshot of the irreplaceable datasets to B2, plus a quick maintenance pass. It writes the last-success heartbeat before maintenance, so freshness does not depend on maintenance duration. - Full maintenance at 12:00 (
kopia-maint-full.timer,nas_kopia_full_maint_oncalendar): the dailykopia maintenance run --full(GC and pack compaction), off the snapshot’s critical path. Isolation from the snapshot unit rests on the timing separation and Kopia’s exclusive maintenance lock. There is no systemd force-stop directive, on purpose. - Freshness dead-man’s-switch, hourly (
kopia-freshness.timer): compares the heartbeat age withnas_kopia_freshness_max_age_hours(36) and sends a Pushover alert when the last successful snapshot is older.
All three units source Pushover credentials from one root-owned 0600 pushover.env (rendered
from Vault) through EnvironmentFile=. The token and user key never appear in the script bodies.
One value lives outside git. The role sets Kopia’s automatic full-maintenance interval with
kopia maintenance set --full-interval (nas_kopia_full_interval, 168h), so an opportunistic
full pass never fires inside a 03:00 snapshot. Kopia stores that value inside the B2 repository,
so a role revert does not reset it. Read the live value with kopia maintenance info, which
shows the full cycle interval and the root@nas-support owner.
Stale-alert triage (one unreadable path fails the whole leaf)
Section titled “Stale-alert triage (one unreadable path fails the whole leaf)”Kopia treats one permission denied directory as a fatal error for that whole source. A failed
source withholds the heartbeat, so the hourly dead-man’s-switch fires every hour until a clean run.
Triage in this order:
- Which source, and since when: the ClickStack gauge
nas.kopia.snapshot.failed(attributesourceis the failing leaf). The wrapper’slog()lines are inotel_logsunderServiceName = 'otelcol-nas-support'. - The Kopia error itself is only in the on-host log:
scripts/nas-adhoc.sh nas-support -b -m shell -a 'tail -60 "$(ls -t /var/log/nas-kopia-b2/backup-*.log | head -1)"'
The usual cause is a path that the sandbox uid 2147000001 cannot read. The nas-kopia-acls
walk grants rX access ACLs plus inheritable default ACLs on every directory, so a path created
after the walk inherits the grant. A tree moved in with mv keeps its source ACLs and never
inherits, for example a new user home created 700 with no ACL. The remedy is the same either
way:
scripts/nas-playbook.sh --tags nas-kopia-acls # re-run the grant walkSubtrees that the walk must skip are listed in nas_kopia_acl_prune_subpaths.