NAS Ansible Management — Foundation + Tooling Sandbox (P0+P1)

This design brings the nas host (TrueNAS 26) under Ansible management. The NAS is already a cluster citizen in Terraform (Authentik SSO), DNS (router-hosts), and monitoring (Uptime Kuma) — but nothing in ansible/ touches it. This spec closes the Ansible gap for the first two phases of a larger epic (tracked under bead hl-36m6): P0 the appliance-host foundation + observability parity, and P1 a managed libvirt-LXC tooling sandbox with access to the data pools.

Context & background

nas is TrueNAS 26.0.0-BETA.2 — a Debian-derived appliance, not a general-purpose Linux host. Two facts drive the entire design:

Config is middleware-database-driven, not file-driven. The TrueNAS middleware regenerates /etc (Samba, NFS, cron, users) from a SQLite config DB on boot/upgrade. Direct file edits — Ansible’s template/lineinfile reflex — are silently reverted. The supported path is the middleware API (midclt call … / JSON-RPC).
Incus is being removed; libvirt is the future. TrueNAS’s virtualization story zig-zagged (25.04 introduced Incus-backed Instances → community backlash → 25.04.2 reverted VMs to libvirt/KVM → 25.10 kept Incus only for experimental LXC). TrueNAS 26 makes LXC containers fully supported and switches the LXC backend from Incus to libvirt (“incus has been removed” per 26 nightlies and forums). The container.* middleware namespace is the supported, UI-visible, upgrade-safe path; on 26 it is libvirt-backed (container.create returns a libvirt UUID).

On the box today: the Incus daemon is gone (dead socket — bare incus commands hang), and the existing hand-rolled Incus setup under /mnt/main/fzymgc-house/incus/ plus the user storage pool apps/incus-pool are orphaned. See the migration inventory below; full disposition is in the epic decomposition.

Why not the alternatives

Raw incus CLI / community.general.incus connection plugin — dead end; the backend is being removed.
Raw docker-compose@ systemd units on the host (as the Firewalla otel-collector role does) — fights the appliance; the middleware owns Docker via app.*. A non-middleware compose stack risks being reverted on upgrade.
libvirt KVM VM for the sandbox — full-OS isolation is overkill for tooling.

Goals

G1 — Observability parity. NAS host metrics + logs flow into the cluster’s ClickStack pipeline (the same otel-gateway.fzymgc.house → ClickStack path the router uses), via an appliance-native collector.
G2 — Inventory + drift visibility. nas is a first-class inventory host; facts are gathered; current shares/datasets are snapshotted as a drift baseline.
G3 — Managed tooling sandbox. A libvirt-LXC container (nas-support reborn) with access to the data pools, fully Ansible-managed, hosting space-curation tooling (Kopia, ncdu, rmlint) and ad-hoc jobs.

Non-goals (deferred to later epic phases)

Declarative management of shares/datasets/snapshot tasks (future phase).
AresMUSH game-server migration — precipice + doorsportal1 (P2, high risk: idmap + MongoDB restore).
Full cleanup/reclaim of orphaned Incus storage (P3).
TrueNAS’s own internal rclone cloud-sync — left untouched; it is the appliance’s concern, not ours.

Epic decomposition (hl-36m6)

Phase	Sub-project	Status
P0	Host foundation + telemetry	this spec
P1	Tooling sandbox LXC (`nas-support`)	this spec
P2	AresMUSH migration (`precipice`, `doorsportal1`)	deferred (high risk)
P3	Cleanup/reclaim (orphaned Incus storage + dead containers)	deferred; partial reclaim safe now

This spec covers P0 + P1. P0 is reversible and unblocks everything; P1 proves the “Ansible drives midclt to lifecycle a libvirt-LXC, then configures the interior over SSH” toolchain on a zero-data-loss workload before P2 touches the irreplaceable game data.

Architecture

 Ansible control node (LAN) ── SSH (automation user) ──► nas (TrueNAS 26)
                                  │ become → midclt call …   (API access path)
                                  ▼
 ┌─ TrueNAS host — managed via middleware API only ────────────────────────┐
 │ P0                                                                       │
 │  • inventory: nas replaces stale nas-charlie-1                           │
 │  • user.create → automation user + SSH key (Vault)                       │
 │  • setup/facts; snapshot shares+datasets → drift baseline                │
 │  • otel-collector = custom TrueNAS Docker app                            │
 │       hostmetrics(/proc,/sys) + journald/syslog ──mTLS──► otel-gateway   │
 │                                                          → ClickStack    │
 │ P1                                                                       │
 │  • container.create (libvirt-LXC) "nas-support"                          │
 │    ┌───────────────────────────────────────────────────────────────┐   │
 │    │ Ubuntu :default · SSH key via mounted authorized_keys (no c-i)  │   │
 │    │ NIC: macvlan on bond0 · static 192.168.20.202 · router-hosts DNS│   │
 │    │ mounts: /mnt/main, /mnt/apps (recursive)                        │   │
 │    │ interior role: homebrew · kopia · ncdu · rmlint · curation      │   │
 │    └───────────────────────────────────────────────────────────────┘   │
 └──────────────────────────────────────────────────────────────────────────┘

P0 — Host foundation & telemetry

Inventory

Replace the stale nas_hosts entry (nas-charlie-1, user fzymgc) with the real host:

nas_hosts:
  hosts:
    nas:
      ansible_host: 192.168.20.200   # bond0 primary
      ansible_ssh_user: nas-automation

host_vars/nas.yml carries pool names, the static sandbox IP, and telemetry vars.

Automation user (middleware-managed)

A dedicated account, not the personal sean login, created via the middleware so it persists in the config DB:

Bootstrap (one-time): connect as sean (passwordless sudo confirmed), run midclt call user.create to create nas-automation with an injected SSH public key (private key stored in Vault) and scoped passwordless sudo for the commands Ansible needs:

{
  "username": "nas-automation",
  "full_name": "Ansible automation",
  "group_create": true,
  "sshpubkey": "<from Vault>",
  "password_disabled": true,
  "sudo_commands_nopasswd": ["/usr/bin/midclt", "/usr/sbin/zfs", "/usr/bin/docker"]
}

Steady-state: Ansible connects as nas-automation; an Ansible task asserts the user/key/sudo are present and correct (idempotent reconcile via user.query → user.update).

API access pattern

midclt-over-SSH with become (recommended): Ansible SSHes in and runs ansible.builtin.command: midclt call <method> <json> under become. midclt talks to the local middleware socket, so no API key or WebSocket client is needed. Wrap calls in a small nas_api helper (a task file or action that shells midclt and parses JSON via from_json).

Alternative considered: remote JSON-RPC WebSocket + API key (api_key.create, truenas/api_client). Rejected for P0 — it adds a non-Ansible-native client and a key to manage, for no benefit when SSH is already the transport.

Facts & drift baseline

Gather Ansible facts (interpreter_python: auto_silent is already set; Python 3.13 present).
Snapshot current config to the repo as a documented baseline (read-only): sharing.smb.query, sharing.nfs.query, pool.dataset.query, pool.snapshottask.query, app.query. Written under docs/reference/ (or an Ansible-rendered artifact) — establishes drift visibility without mutating anything.

Telemetry — custom TrueNAS Docker app

Run otelcol-contrib as a custom TrueNAS app (appliance-native, UI-visible, upgrade-safe), reusing the config + Vault-mTLS pattern from the Firewalla otel-collector role but not its boot-script/docker-compose@ delivery.

Delivery: Ansible defines the app via app.* (custom app / compose YAML) and reconciles idempotently: app.query → app.create (absent) or app.update (config drift). Verified: app.update itself runs a compose reconcile, so it is the steady-state path; app.redeploy (verified: stop → pull images → restart) is used only to force an image re-pull, not in the normal config reconcile.
Collection: hostmetrics receiver mounting host /proc,/sys (HOST_PROC/HOST_SYS env, standard in-container pattern) + the systemd journal for logs. Verified on the box: the journal is at the standard /var/log/journal (persistent) + /run/log/journal (volatile) — mount the journal dir into the collector (the Firewalla role’s /run/log/journal pattern applies), not a .system/syslog-* dataset.
Egress: mTLS to otel-gateway.fzymgc.house:443 → ClickStack, with the HyperDX ingest token from Vault (fzymgc-house/cluster/clickstack, property otel_ingest_api_key) — identical to the router path.
Identity: new client cert CN otelcol-nas.fzymgc.house; new Vault PKI role otel-collector-nas-client and AppRole otel-collector-nas (mirroring the Firewalla naming). Resource attribute service.name=otelcol-nas / host.name=nas.

Decision: Alloy is not used. The cluster is mid-sunset of Grafana/Alloy toward ClickStack (epic hl-nlxl); the NAS ships straight to otel→ClickStack.

P1 — Tooling sandbox LXC

Instance lifecycle

Create the container via the middleware so it is libvirt-backed and UI-visible. Schema verified against the live box (spike 2026-06-27 — container created + deleted):

container.create (an async job — input validation surfaces as a failed job with job.error, not synchronously) with:
- name;
- image — a nested object {"name":"ubuntu:noble:amd64:default", "version":"<dated, from container.image.query_registry>"} (there is no top-level version);
- idmap — an object {"type":"DEFAULT"} (see Mounts & idmap);
- pool — from container.pool_choices (verified: {"apps","main"});
- autostart: true.
container.device.create per device — the NIC and the filesystem mounts — against the new container id (devices are not accepted inline in container.create).
container.start.

Ansible reconciles idempotently (container.query → container.create / container.update / container.device.*; container.delete is synchronous). Lifecycle is the only thing driven through midclt; everything inside the container is normal Ansible over SSH.

Schema note: the published v26.04 API reference is wrong here — it shows a flat image+version with inline devices[]. The spiked live schema (nested image, object idmap, separate synchronous container.device.create) is authoritative. Verified by spike: a FILESYSTEM mount device takes {dtype:"FILESYSTEM", source:"/mnt/<pool>/<ds>", target:"/path"} — source must be an absolute /mnt path within a pool mountpoint (the rumored “not /mnt” restriction is false; a pool-relative path is rejected). The NIC device’s exact field set wasn’t fully spiked (only nic_attach_choices), so the plan confirms the NIC payload on a scratch container.

Image & bootstrap

Image: ubuntu:noble:amd64:default (verified available; note: ubuntu offers only :default, no :cloud variant — alpine has :tinycloud). The verified container.create schema exposes init/initdir/initenv/inituser/initgroup but no cloud-init user-data field, so the old Incus cloud-init.user-data path is unavailable.

Bootstrap (SSH key + baseline): create an in-container nas-automation user (matching the host automation identity; the container inventory entry uses ansible_ssh_user: nas-automation) and inject its SSH public key by mounting an authorized_keys file into that user’s ~/.ssh via a FILESYSTEM device — or a one-shot post-start container-exec to create the user + write the key, since the ubuntu:noble:amd64:default image defaults to root/ubuntu and our user must be created explicitly. Then hand off to SSH-based Ansible for the rest (chrony → 192.168.20.{1,151,152,153}, packages, etc.). The plan confirms the exact mechanism on the scratch container.

Networking — macvlan on bond0 + static IP

NIC device: macvlan parent bond0, address 192.168.20.202 on 192.168.20.0/22 — reclaimed from the retired container-apps container. Verified on the box: container.device.nic_attach_choices → {"BRIDGE":["truenasbr0"],"MACVLAN":["bond0"]}, so macvlan-on-bond0 is a supported attach mode. Address assignment: because there is no in-guest cloud-init network-config path (see Bootstrap), pin a fixed mac on the NIC device and reserve .202 via DHCP (router-hosts/Firewalla) rather than static-configuring inside the guest. IPv6 via SLAAC/accept-ra.
Exposure tier — interactive/outbound-only. The sandbox is a tool box you SSH into from your workstation; it pushes backups out to R2 and exposes no service. So it gets only a router-hosts LAN DNS record (nas-support.fzymgc.house) for SSH convenience — no Authentik SSO, no ingress, no Cloudflare tunnel/Access. Adding exposure later is additive and non-breaking, so this is the conservative default; revisit only if the sandbox later hosts a service others reach.
macvlan host-isolation caveat: the NAS host cannot talk to its own macvlan child. This is acceptable — the Ansible control node (your workstation) is a separate LAN host and reaches the container fine. We do not reintroduce the old /32 host route hack unless a host↔container path is later required.

Mounts & idmap

FILESYSTEM mounts of the datasets the sandbox needs — broad read of /mnt/main and /mnt/apps for analysis/backup, plus read-write on the specific datasets targeted for curation (prune, dedup, reorganize). Write is gated by ACLs, not by the mount alone (see idmap below).
idmap — verified: there is no privileged mode. The idmap object’s type accepts only DEFAULT or ISOLATED (spiked); omitting it defaults to DEFAULT. There is no None/disable option — a container.* LXC is always UID-shifted, so container-root maps to a high host UID (observed TrueNAS subuid ranges 568/3000/3500/4000) and cannot write host trees owned by 0:0 — verified: /mnt/main and /mnt/apps roots are 0:0.
Chosen: idmap: {type: DEFAULT} + POSIX1E ACLs on the specific curation datasets — proven end-to-end by spike. Under DEFAULT, container-root (uid 0) maps to host UID 2147000001 on this box (a fixed-namespace base; the plan re-confirms it after any recreate). The apps pool uses POSIX1E ACLs, so a grant is setfacl -R -m u:2147000001:rwx /mnt/<pool>/<ds> plus setfacl -d -m u:2147000001:rwx (default ACL so new files inherit). Verified: pre-ACL the container is denied on a 0700 root-owned dir; post-ACL it reads existing files and writes new ones (files land owned by 2147000001 on the host). This is exactly specific access to the data shares — everything else stays read-only or unmounted. ISOLATED (per-container unique slice) is the alternative; DEFAULT keeps the mapped UID predictable for ACLs.
Datasets are a mix of ACL types — the grant must branch (both proven). A survey of both pools found ~87 POSIX1E vs 15 NFSv4 datasets, and the highest-value curation targets are NFSv4 (multimedia, software/ISOs/Downloads, time-machine, family/shares). The grant for UID 2147000001 differs by acltype, both validated by spike:
- POSIX1E → setfacl -R -m u:2147000001:rwx <path> + setfacl -d -m u:2147000001:rwx <path> (default ACL for inheritance).
- NFSv4 → midclt call filesystem.setacl adding a USER ACE (id 2147000001, perms:{BASIC:FULL_CONTROL}, flags:{BASIC:INHERIT}, options.recursive). setfacl does not work on NFSv4 datasets.
The ACL-granting Ansible task therefore reads each target’s acltype (zfs get acltype) and applies the matching mechanism. (NFSv4 was validated via a host-UID-2147000001 proxy — equivalent to container-root for VFS ACL checks; the plan’s scratch-validation does one combined container+NFSv4 pass for completeness.)
Validation gate: the rw mechanism is proven; the plan still (a) re-confirms the DEFAULT-mapped root UID after creating the real container (read /proc/<pid>/uid_map) in case the namespace base differs, and (b) confirms read coverage on representative real paths — any 0700 root-owned subtree is invisible to ncdu/rmlint under the shifted UID until ACL’d. (The separate, harder idmap case — mapped UIDs 2147xxx on AresMUSH data — is a P2 concern.)
Scope note: write reach is bounded to the ACL’d targets, not blanket access — a smaller risk surface than the originally-imagined privileged container, and the secure-by-default posture.

Interior role — `nas-sandbox-tools`

A normal Ansible role applied over SSH:

Homebrew (Linuxbrew) for user tooling.
Kopia for personal backup/curation (the user’s chosen tool; the repo already standardizes on Kopia via router-kopia-backup). Wire a Kopia repo to Cloudflare R2 with creds from Vault (new path, new prefix — e.g. nas-backups/ on the existing fzymgc-cluster-storage bucket). The backup set is user-defined; this spec only stands up the tool + repo.
Space-curation utilities: ncdu, rmlint, rclone (for the user’s own use, distinct from TrueNAS’s internal rclone).
Optionally the generalized otel agent for in-container telemetry (host telemetry is already covered by the P0 collector).

Secrets / Vault paths (new)

Path / property	Contents
`fzymgc-house/infrastructure/nas/automation-ssh`	`nas-automation` private SSH key
Vault PKI role `otel-collector-nas-client` (path `fzymgc-house/v1/ica1/v1`)	mTLS client cert for `otelcol-nas.fzymgc.house`
AppRole `otel-collector-nas`	otel collector app Vault auth
`fzymgc-house/cluster/clickstack` · `otel_ingest_api_key`	reused (existing) ingest token
`fzymgc-house/infrastructure/nas/kopia-r2`	sandbox Kopia R2 creds + repo password

Vault policies must be updated for each new path/role (per tf/CLAUDE.md).

Repo structure

Mirror the router-* convention:

ansible/
  inventory/hosts.yml          # fix nas_hosts → real nas
  host_vars/nas.yml            # pools, sandbox IP, telemetry vars
  nas-playbook.yml             # plays: foundation, telemetry, sandbox
  roles/
    nas-common/                # automation user, facts, baseline, drift snapshot
    nas-otel-collector/        # otelcol-contrib as a TrueNAS custom app
    nas-lxc-sandbox/           # container.* lifecycle for nas-support
    nas-sandbox-tools/         # interior: homebrew, kopia, curation utils

The nas-lxc-sandbox role is named for the libvirt-LXC backend (not Incus).

Decision: dedicated nas-otel-collector role (the fallback anticipated below, now chosen). Generalizing the existing otel-collector role in place proved impractical: it is Firewalla-coupled — otel_collector_user defaults to pi (a getent precondition), it unconditionally creates /home/pi/.firewalla/... dirs, and configure.yml binds 19 copies to that user. Rather than thread delivery-mode conditionals through a working deployment, the NAS gets its own role that reuses the otel config + Vault-PKI-mTLS approach but owns TrueNAS-appropriate delivery (custom app via app.*). The Firewalla otel-collector role is left untouched.

All playbooks/roles carry the required headers (# SPDX-License-Identifier: MIT-0, # code: language=ansible) and FQCN modules, per ansible/CLAUDE.md.

Cleanup interactions

Confirmed-dead containers (halon 42G, gh-runner, nas-app-proxy, container-apps/nas-container-apps) are safe to reclaim now (~46G), but that work belongs to P3 and is out of scope here. The orphaned apps/incus-pool container rootfs and the fzymgc-house/incus tree for precipice/doorsportal1 must be retained until the P2 AresMUSH restore is verified.

Risks & mitigations

Risk	Mitigation
BETA `container.*` API churns under us	P0 doesn’t depend on it; P1 reconciles idempotently and is recreatable. Re-validate against `system.version` before runs.
Custom-app otel can’t see host metrics	Mount `/proc`,`/sys` into the collector (standard hostmetrics pattern); fall back to TrueNAS reporting API if gaps appear.
Bootstrap chicken-and-egg (automation user)	One-time `sean`-driven `user.create`; thereafter connect as `nas-automation`.
macvlan host-isolation surprises	Documented; control node is a separate LAN host. No `/32` hack unless needed.
Non-middleware changes reverted on upgrade	Everything host-side goes through `midclt`; the sandbox is a `container.*` instance.

Verification

ansible-playbook nas-playbook.yml --syntax-check and --check --diff first.
P0: confirm nas facts gather; confirm otel app RUNNING (app.query); confirm NAS telemetry visible in ClickStack (query default.otel_metrics_* / otel_logs filtered to host.name=nas).
P1: confirm container.query shows nas-support running; SSH in as nas-automation; confirm /mnt/main + /mnt/apps visible; confirm brew, kopia, curation tools installed; confirm nas-support.fzymgc.house resolves.

Decisions resolved during review

Sandbox static IP: reuse 192.168.20.202, reclaimed from the retired container-apps.
otel role: dedicated nas-otel-collector role — generalizing the Firewalla-coupled otel-collector in place proved impractical (see Repo structure).
Sandbox mounts: read-write via idmap: {type: DEFAULT} + ZFS ACLs on specific curation datasets — spike verified there is no privileged/None idmap; ACL-scoped access matches the “specific access to the data shares” intent. Validate the mapped UID on a scratch dataset before ACLing real targets (see Mounts & idmap).
Exposure: interactive/outbound-only — router-hosts LAN DNS for SSH; no SSO/ingress/tunnel (see Networking).

No blocking open questions remain; finer details are expected to surface during writing-plans.