NAS Ansible Foundation + Tooling Sandbox — Implementation Plan (P0+P1)

NAS Ansible Foundation + Tooling Sandbox Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Bring the TrueNAS 26 nas host under Ansible management — inventory, a dedicated automation user, observability parity, and an idmap-shifted LXC tooling sandbox with read-write access (POSIX1E and NFSv4) to targeted datasets.

Architecture: The TrueNAS appliance is managed only through the middleware API (midclt call … over SSH with become), never by editing /etc (the config DB reverts file edits). Telemetry runs as a custom TrueNAS Docker app (otelcol-contrib → otel-gateway.fzymgc.house → ClickStack). The sandbox is a libvirt-LXC created via container.*, with idmap:{type:DEFAULT} (container-root → host UID 2147000001) and per-dataset ACLs (acltype-aware) granting that UID read-write on curation targets.

Tech Stack: Ansible (FQCN modules, SOPS + community.hashi_vault), TrueNAS 26 middleware (midclt JSON-RPC), ZFS ACLs (setfacl / filesystem.setacl), otelcol-contrib, Kopia, Homebrew.

Conventions (from ansible/CLAUDE.md): every playbook/role file starts with # SPDX-License-Identifier: MIT-0 then # code: language=ansible; FQCN module names; no_log: true on secret-handling tasks; snake_case vars prefixed by role.

Testing model (Ansible, not unit tests): each task’s “test” cycle is (1) ansible-playbook --syntax-check, (2) --check --diff dry-run showing the intended change, (3) apply, (4) idempotence — re-run and assert changed=0, (5) post-conditions — midclt/SSH/ClickHouse assertions. “Write the failing test first” maps to asserting the pre-condition is absent (e.g. user/container/ACL not present) before the task creates it.

Model assignment (for plan-to-beads / dispatch): mechanical tasks → haiku (Task 0 vault seed, Task 1 inventory, Task 3 drift loop, Task 9 tooling install, Task 10 docs); middleware-schema-sensitive tasks → sonnet (Task 2 user reconcile, Task 4 vault TF, Task 5 telemetry app, Task 6 container lifecycle, Task 7 nsenter bootstrap, Task 8 ACL granting).

Bootstrap ordering note: the dedicated nas-automation user does not exist yet, so Task 1–2 connect as sean (passwordless sudo confirmed). After Task 2, all runs connect as nas-automation.

Spike-verified facts this plan depends on (bead hl-36m6):

container.create is an async job; image is nested {name,version}; idmap is {type} (DEFAULT|ISOLATED only — no privileged mode); pool from container.pool_choices. Devices are added via synchronous container.device.create (not inline).
FILESYSTEM device source MUST be an absolute /mnt/... path within a pool mountpoint.
DEFAULT idmap maps container-root → host UID 2147000001 on this box.
Datasets are mixed ACL type (~87 POSIX1E / 15 NFSv4); curation targets like multimedia, software, time-machine are NFSv4. Grant rw to 2147000001 via setfacl (POSIX1E) or midclt filesystem.setacl (NFSv4).
container.create exposes no cloud-init user-data field; ubuntu images are :default. Inject SSH key via a mounted authorized_keys file; assign the static IP via fixed NIC MAC + DHCP reservation.

Files created / modified

Path	Responsibility
`ansible/inventory/hosts.yml`	Replace stale `nas-charlie-1` with real `nas` host
`ansible/inventory/host_vars/nas.yml`	NAS vars: pools, sandbox IP/MAC, telemetry, curation targets
`ansible/nas-playbook.yml`	Orchestrates the `nas-*` roles in phases
`ansible/roles/nas-common/`	Automation user (via `midclt user.*`), facts, drift baseline
`ansible/roles/nas-otel-collector/`	New role — otelcol-contrib as a TrueNAS custom app (the Firewalla `otel-collector` role is left untouched)
`ansible/roles/nas-lxc-sandbox/`	`container.*` lifecycle + acltype-aware ACL granting
`ansible/roles/nas-sandbox-tools/`	Interior config over SSH: homebrew, kopia, ncdu/rmlint
`tf/vault/` (or existing vault policy module)	New PKI role `otel-collector-nas-client`, AppRole `otel-collector-nas`, policies for new KV paths

External prerequisites (not Ansible — note in PR description):

DHCP reservation for 192.168.20.202 keyed on the sandbox NIC MAC (Firewalla).
DNS nas-support.fzymgc.house via the router-hosts annotation/entry path.
Vault KV secrets seeded (Task 0).

Phase P0 — Host foundation + telemetry

Task 0: Seed Vault secrets (prerequisite)

Files: none (operator-run vault commands; documented for reproducibility).

Step 1: Generate the automation SSH keypair

Run:

ssh-keygen -t ed25519 -N '' -C nas-automation@fzymgc -f /tmp/nas-automation

Step 2: Store the keypair in Vault

Run:

vault kv put fzymgc-house/infrastructure/nas/automation-ssh \
  private_key=@/tmp/nas-automation \
  public_key="$(cat /tmp/nas-automation.pub)"
shred -u /tmp/nas-automation /tmp/nas-automation.pub

Step 3: Seed the sandbox Kopia R2 secret (reuses the R2 creds pattern from router-kopia-backup/README.md)

Run:

R2_USER=$(vault kv get -field=username fzymgc-house/cluster/longhorn/cloudflare-r2)
R2_PASS=$(vault kv get -field=password fzymgc-house/cluster/longhorn/cloudflare-r2)
vault kv put fzymgc-house/infrastructure/nas/kopia-r2 \
  username="$R2_USER" password="$R2_PASS" repo_password="$(openssl rand -base64 32)"

Step 4: Verify

Run: vault kv get -field=public_key fzymgc-house/infrastructure/nas/automation-ssh Expected: prints the ssh-ed25519 … public key.

Task 1: Inventory + host_vars

Files:

Modify: ansible/inventory/hosts.yml (the nas_hosts group)
Create: ansible/inventory/host_vars/nas.yml
Step 1: Assert the stale entry is what we expect (pre-condition)

Run: rg -n 'nas-charlie-1|nas_hosts' ansible/inventory/hosts.yml Expected: shows nas_hosts: → nas-charlie-1: ansible_ssh_user: fzymgc (the entry we will replace).

Step 2: Replace the nas_hosts group

In ansible/inventory/hosts.yml, replace the nas_hosts block with:

nas_hosts:
  hosts:
    nas:
      ansible_host: 192.168.20.200
      ansible_ssh_user: nas-automation

Step 3: Create ansible/inventory/host_vars/nas.yml

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
# TrueNAS 26 appliance — managed via middleware API only.
nas_pools:
  - apps
  - main

# Dedicated automation user (created via midclt in nas-common).
nas_automation_user: nas-automation

# --- Sandbox LXC (P1) ---
nas_sandbox_name: nas-support
nas_sandbox_image_name: "ubuntu:noble:amd64:default"
# Pin a recent image version; refresh from `midclt call container.image.query_registry`.
nas_sandbox_image_version: "20260625_07:42"
nas_sandbox_pool: apps
nas_sandbox_nic_parent: bond0
nas_sandbox_nic_mac: "00:16:3e:20:02:02"   # fixed → DHCP reservation for .202
nas_sandbox_ip: "192.168.20.202"
nas_sandbox_fqdn: nas-support.fzymgc.house
# DEFAULT idmap base on this host (re-confirmed at runtime from /proc/<pid>/uid_map).
nas_sandbox_mapped_uid: 2147000001

# Datasets mounted into the sandbox. NOTE: `mode` is documentation-only — the
# FILESYSTEM device mounts at the kernel default; actual write capability is gated
# by ACLs on nas_sandbox_curation_targets (Task 8), NOT by this field.
nas_sandbox_mounts:
  - { source: /mnt/main, target: /mnt/main, mode: ro }
  - { source: /mnt/apps, target: /mnt/apps, mode: ro }
nas_sandbox_curation_targets: []   # e.g. [/mnt/main/multimedia, /mnt/main/software]

# --- Telemetry (P0) ---
nas_otel_cert_cn: otelcol-nas.fzymgc.house
nas_otel_vault_pki_role: otel-collector-nas-client
nas_otel_vault_approle_name: otel-collector-nas
nas_otel_gateway_endpoint: "otel-gateway.fzymgc.house:443"

Step 4: Verify reachability (as sean for now — nas-automation doesn’t exist yet)

Run:

cd ansible
ansible -i inventory nas -m ansible.builtin.ping \
  -e ansible_ssh_user=sean -e ansible_become=true

Expected: nas | SUCCESS => {"ping": "pong"}.

Step 5: Commit

Commit using VCS-appropriate commands per references/vcs-preamble.md — message: feat(nas): add nas host to inventory + host_vars [hl-36m6].

Task 2: `nas-common` role — dedicated automation user

Files:

Create: ansible/roles/nas-common/tasks/main.yml
Create: ansible/roles/nas-common/tasks/automation-user.yml
Create: ansible/roles/nas-common/defaults/main.yml
Create: ansible/roles/nas-common/meta/main.yml
Create: ansible/nas-playbook.yml
Step 1: Assert the user is absent (pre-condition / “failing test”)

Run:

ssh nas "sudo -n midclt call user.query '[[\"username\",\"=\",\"nas-automation\"]]'"

Expected: [] (user does not exist yet).

Step 2: Create ansible/roles/nas-common/meta/main.yml

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
galaxy_info:
  role_name: nas_common
  author: fzymgc-house
  description: TrueNAS host baseline — automation user (via middleware), facts, drift baseline.
  license: MIT-0
  min_ansible_version: "2.14"
dependencies: []

Step 3: Create ansible/roles/nas-common/defaults/main.yml

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
nas_automation_user: nas-automation
# NOTE: spec listed /usr/bin/docker; replaced with /usr/bin/setfacl — the app.*
# middleware owns Docker (no direct docker access needed), and setfacl is required
# for POSIX1E ACL grants in Task 8.
nas_automation_sudo_nopasswd:
  - /usr/bin/midclt
  - /usr/sbin/zfs
  - /usr/bin/setfacl
nas_automation_ssh_vault_path: fzymgc-house/infrastructure/nas/automation-ssh

Step 4: Create ansible/roles/nas-common/tasks/automation-user.yml

This reconciles the user through the middleware. midclt is invoked via ansible.builtin.command under become; results parsed with from_json.

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Read automation SSH public key from Vault
  ansible.builtin.set_fact:
    nas_automation_pubkey: >-
      {{ lookup('community.hashi_vault.vault_kv2_get',
                nas_automation_ssh_vault_path).data.data.public_key }}
  no_log: true

- name: Query whether the automation user exists
  ansible.builtin.command:
    cmd: midclt call user.query [["username","=","{{ nas_automation_user }}"]]
  become: true
  changed_when: false
  register: nas_user_q

- name: Parse user query
  ansible.builtin.set_fact:
    nas_user_existing: "{{ (nas_user_q.stdout | from_json) }}"

- name: Create the automation user (middleware)
  ansible.builtin.command:
    cmd: >-
      midclt call user.create {{ {
        "username": nas_automation_user,
        "full_name": "Ansible automation",
        "group_create": true,
        "home": "/var/empty",
        "password_disabled": true,
        "ssh_password_enabled": false,
        "sshpubkey": nas_automation_pubkey,
        "sudo_commands_nopasswd": nas_automation_sudo_nopasswd
      } | to_json }}
  become: true
  when: nas_user_existing | length == 0
  no_log: true
  register: nas_user_created

- name: Reconcile sudo + key on existing user (idempotent)
  ansible.builtin.command:
    cmd: >-
      midclt call user.update {{ nas_user_existing[0].id }} {{ {
        "sshpubkey": nas_automation_pubkey,
        "sudo_commands_nopasswd": nas_automation_sudo_nopasswd,
        "ssh_password_enabled": false
      } | to_json }}
  become: true
  when: >-
    nas_user_existing | length > 0 and
    (nas_user_existing[0].sshpubkey != nas_automation_pubkey or
     nas_user_existing[0].sudo_commands_nopasswd | sort != nas_automation_sudo_nopasswd | sort)
  no_log: true

Step 5: Create ansible/roles/nas-common/tasks/main.yml

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Configure automation user
  ansible.builtin.include_tasks: automation-user.yml
  tags: [nas-common, nas-automation-user]

Step 6: Create ansible/nas-playbook.yml

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: NAS host foundation
  hosts: nas
  gather_facts: true
  roles:
    - role: nas-common
      tags: [nas-common]

Step 7: Syntax-check + dry-run the bootstrap (as sean)

Run:

cd ansible
ansible-playbook -i inventory nas-playbook.yml --syntax-check
ansible-playbook -i inventory nas-playbook.yml --tags nas-automation-user \
  -e ansible_ssh_user=sean -e ansible_become=true --check --diff

Expected: syntax OK; check-mode shows the user.create task would run.

Step 8: Apply the bootstrap (one-time, as sean)

Run:

ansible-playbook -i inventory nas-playbook.yml --tags nas-automation-user \
  -e ansible_ssh_user=sean -e ansible_become=true

Expected: changed on the create task.

Step 9: Verify the user + connect as nas-automation

Run:

ssh nas "sudo -n midclt call user.query '[[\"username\",\"=\",\"nas-automation\"]]'" | python3 -m json.tool | head
ansible -i inventory nas -m ansible.builtin.ping        # now uses nas-automation

Expected: user present with sudo_commands_nopasswd; ping pong as nas-automation.

Step 10: Idempotence

Run: ansible-playbook -i inventory nas-playbook.yml --tags nas-automation-user Expected: changed=0.

Step 11: Commit — feat(nas): nas-common role + automation user via middleware [hl-36m6].

Task 3: `nas-common` — drift baseline snapshot

Files:

Create: ansible/roles/nas-common/tasks/drift-baseline.yml
Create: ansible/roles/nas-common/templates/drift-baseline.md.j2
Modify: ansible/roles/nas-common/tasks/main.yml (add include)
Step 1: Create ansible/roles/nas-common/tasks/drift-baseline.yml

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Snapshot current middleware config for drift baseline
  ansible.builtin.command:
    cmd: "midclt call {{ item }}"
  become: true
  changed_when: false
  loop:
    - sharing.smb.query
    - sharing.nfs.query
    - pool.dataset.query
    - pool.snapshottask.query
    - app.query
  register: nas_drift_raw

- name: Render drift baseline doc
  ansible.builtin.template:
    src: drift-baseline.md.j2
    dest: "{{ playbook_dir }}/../docs/reference/nas-config-baseline.md"
    mode: "0644"
  delegate_to: localhost
  become: false

Step 2: Create ansible/roles/nas-common/templates/drift-baseline.md.j2

---
title: "NAS Config Baseline (drift reference)"
description: "Snapshot of TrueNAS shares, datasets, and apps captured by nas-common. Regenerate with: ansible-playbook nas-playbook.yml --tags nas-drift."
---

Generated by `nas-common`. Read-only snapshot — do not hand-edit.

{% for r in nas_drift_raw.results %}
## `{{ r.item }}`

```json
{{ r.stdout }}

{% endfor %}

- [ ] **Step 3: Add the include to `tasks/main.yml`**

Append:

```yaml
- name: Capture drift baseline
  ansible.builtin.include_tasks: drift-baseline.yml
  tags: [nas-common, nas-drift]

Step 4: Run + verify

Run:

cd ansible
ansible-playbook -i inventory nas-playbook.yml --tags nas-drift
rg -c '"id"' ../docs/reference/nas-config-baseline.md

Expected: baseline file rendered with the five query sections.

Step 5: Lint the rendered doc

Run: uvx rumdl@0.2.17 check docs/reference/nas-config-baseline.md Expected: no issues (fix the template fence/heading spacing if flagged).

Step 6: Commit — feat(nas): drift baseline snapshot via middleware queries [hl-36m6].

Task 4: Telemetry — Vault PKI/AppRole prerequisites

Files:

Modify: the Vault Terraform module under tf/ that defines PKI roles + AppRoles (follow the existing otel-collector-firewalla definitions — search them out first).
Step 1: Locate the existing Firewalla otel Vault definitions

Run:

rg -n 'otel-collector-firewalla|otel_collector' tf/ -l

Expected: the .tf file(s) defining the otel-collector-firewalla-client PKI role + otel-collector-firewalla AppRole + policy.

Step 2: Add NAS analogues

Duplicate those definitions for the NAS, changing identifiers only:

PKI role otel-collector-nas-client (path fzymgc-house/v1/ica1/v1), allowed CN otelcol-nas.fzymgc.house, TTL 720h.
AppRole otel-collector-nas bound to a policy granting: issue against the NAS PKI role, and read fzymgc-house/cluster/clickstack (property otel_ingest_api_key).
Step 3: Plan + apply (per fzymgc-house:terraform skill / HCP workflow)

Use the fzymgc-house:terraform skill. Expected: terraform plan shows only the new PKI role + AppRole + policy; apply succeeds.

Step 4: Verify

Run: vault read auth/approle/role/otel-collector-nas/role-id Expected: returns a role-id (AppRole exists).

Step 5: Commit — feat(nas): vault PKI role + approle for nas otel-collector [hl-36m6].

Task 5: Telemetry — dedicated `nas-otel-collector` role (TrueNAS custom app)

Model: sonnet (middleware app.* + Vault PKI issuance).

Why a new role, not generalize-in-place: the existing otel-collector role is Firewalla-coupled — otel_collector_user defaults to pi (a getent precondition in directories.yml), it unconditionally creates /home/pi/.firewalla/... compose + post-main dirs, and configure.yml binds 19 copy tasks to that user. Threading delivery-mode conditionals through it would risk the working Firewalla deployment. Per the spec’s fallback clause, P1 telemetry gets its own role that reuses the otel config + Vault-PKI-mTLS approach but owns TrueNAS-appropriate delivery. The Firewalla role is left untouched (so its deployment is unaffected by construction).

Files:

Create: ansible/roles/nas-otel-collector/meta/main.yml
Create: ansible/roles/nas-otel-collector/defaults/main.yml
Create: ansible/roles/nas-otel-collector/tasks/main.yml
Create: ansible/roles/nas-otel-collector/templates/otel-config.yaml.j2 (adapted from roles/otel-collector/templates/otel-config.yaml.j2)
Create: ansible/roles/nas-otel-collector/templates/compose.yaml.j2
Modify: ansible/nas-playbook.yml (add a telemetry play)
Step 1: meta + defaults

meta/main.yml mirrors nas-common/meta/main.yml (role_name: nas_otel_collector). defaults/main.yml:

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
nas_otel_app_name: otelcol-nas
nas_otel_base_dir: /mnt/apps/otel-collector
nas_otel_image: otel/opentelemetry-collector-contrib
nas_otel_version: "0.119.0"
nas_otel_cert_cn: otelcol-nas.fzymgc.house
nas_otel_vault_pki_path: fzymgc-house/v1/ica1/v1
nas_otel_vault_pki_role: otel-collector-nas-client
nas_otel_gateway_endpoint: "otel-gateway.fzymgc.house:443"
nas_otel_ingest_token_vault_path: fzymgc-house/cluster/clickstack
nas_otel_ingest_token_property: otel_ingest_api_key

Step 2: Adapt the collector config template

Copy roles/otel-collector/templates/otel-config.yaml.j2 → roles/nas-otel-collector/templates/otel-config.yaml.j2 and adapt:

Remove the otlp receiver and its pipeline reference (the NAS has no router-hosts push source).
Keep the hostmetrics receiver (reads /hostfs via HOST_PROC/HOST_SYS) and the journald receiver (/var/log/journal).
Keep the otlp exporter to {{ nas_otel_gateway_endpoint }} over mTLS (/etc/otelcol/certs/{tls.crt,tls.key,ca.crt}) with the authorization header = {{ nas_otel_ingest_token }}.
Set resource attributes service.name=otelcol-nas, host.name=nas.
Step 3: Compose template (templates/compose.yaml.j2)

services:
  otelcol:
    image: {{ nas_otel_image }}:{{ nas_otel_version }}
    restart: unless-stopped
    command: ["--config=/etc/otelcol/config.yaml"]
    environment:
      HOST_PROC: /hostfs/proc
      HOST_SYS: /hostfs/sys
    volumes:
      - {{ nas_otel_base_dir }}/config.yaml:/etc/otelcol/config.yaml:ro
      - {{ nas_otel_base_dir }}/certs:/etc/otelcol/certs:ro
      - /proc:/hostfs/proc:ro
      - /sys:/hostfs/sys:ro
      - /var/log/journal:/var/log/journal:ro

Step 4: tasks/main.yml — issue cert, render config + compose, deploy

The mTLS cert is issued from the control node via Vault PKI (the Firewalla role uses a vault-agent sidecar; a control-node issue is simpler for a 30-day TTL and re-issues on each run).

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Read HyperDX ingest token from Vault
  ansible.builtin.set_fact:
    nas_otel_ingest_token: >-
      {{ lookup('community.hashi_vault.vault_kv2_get',
                nas_otel_ingest_token_vault_path).data.data[nas_otel_ingest_token_property] }}
  no_log: true

- name: Ensure cert dir
  ansible.builtin.file:
    path: "{{ nas_otel_base_dir }}/certs"
    state: directory
    mode: "0750"
  become: true

- name: Check existing cert validity (rc!=0 if missing or expiring within 7 days)
  ansible.builtin.command:
    cmd: "openssl x509 -checkend 604800 -noout -in {{ nas_otel_base_dir }}/certs/tls.crt"
  become: true
  register: nas_otel_cert_ok
  changed_when: false
  failed_when: false

- name: Decide whether to (re)issue the cert
  ansible.builtin.set_fact:
    nas_otel_issue: "{{ (nas_otel_cert_ok.rc | default(1)) != 0 }}"

- name: Issue mTLS client cert from Vault PKI
  community.hashi_vault.vault_write:
    path: "{{ nas_otel_vault_pki_path }}/issue/{{ nas_otel_vault_pki_role }}"
    data: { common_name: "{{ nas_otel_cert_cn }}", ttl: "720h" }
  delegate_to: localhost
  become: false
  register: nas_otel_pki
  no_log: true
  when: nas_otel_issue

- name: Write cert material (only when (re)issued)
  ansible.builtin.copy:
    content: "{{ item.content }}"
    dest: "{{ nas_otel_base_dir }}/certs/{{ item.name }}"
    mode: "0640"
  become: true
  no_log: true
  when: nas_otel_issue
  loop:
    - { name: tls.crt, content: "{{ nas_otel_pki.data.certificate }}" }
    - { name: tls.key, content: "{{ nas_otel_pki.data.private_key }}" }
    - { name: ca.crt, content: "{{ nas_otel_pki.data.issuing_ca }}" }

- name: Render collector config
  ansible.builtin.template:
    src: otel-config.yaml.j2
    dest: "{{ nas_otel_base_dir }}/config.yaml"
    mode: "0640"
  become: true
  register: nas_otel_config_render

- name: Render compose
  ansible.builtin.template:
    src: compose.yaml.j2
    dest: "{{ nas_otel_base_dir }}/compose.yaml"
    mode: "0640"
  become: true
  register: nas_otel_compose_render

- name: Query existing app
  ansible.builtin.command:
    cmd: midclt call app.query [["name","=","{{ nas_otel_app_name }}"]]
  become: true
  changed_when: false
  register: nas_otel_app_q

- name: Create custom app (absent)
  ansible.builtin.command:
    cmd: >-
      midclt call -job app.create {{ {
        "app_name": nas_otel_app_name, "custom_app": true,
        "custom_compose_config_string": lookup('file', nas_otel_base_dir ~ '/compose.yaml')
      } | to_json }}
  become: true
  when: (nas_otel_app_q.stdout | from_json) | length == 0

- name: Update custom app — ONLY when the rendered config or compose changed
  ansible.builtin.command:
    cmd: >-
      midclt call -job app.update {{ nas_otel_app_name }} {{ {
        "custom_compose_config_string": lookup('file', nas_otel_base_dir ~ '/compose.yaml')
      } | to_json }}
  become: true
  when:
    - (nas_otel_app_q.stdout | from_json) | length > 0
    - (nas_otel_config_render is changed) or (nas_otel_compose_render is changed)
  changed_when: true

Idempotence: the two template renders carry their own change detection, so the app.update fires only when config.yaml or compose.yaml actually changed. A steady-state re-run renders identical files (not changed) → app.update skipped → changed=0, and the telemetry container is not cycled. (A config change does redeploy, which is correct — otelcol needs a restart to reload its mounted config.)

Step 5: Add the telemetry play to nas-playbook.yml

- name: NAS telemetry
  hosts: nas
  gather_facts: true
  roles:
    - role: nas-otel-collector
      tags: [nas-otel]

Step 6: Syntax-check; confirm Firewalla role untouched

Run:

cd ansible
ansible-playbook -i inventory nas-playbook.yml --tags nas-otel --syntax-check
git -C .. diff --stat -- ansible/roles/otel-collector   # must be EMPTY

Expected: syntax OK; zero changes to roles/otel-collector (the Firewalla deployment is untouched by construction).

Step 7: Apply to NAS + verify the app is running

Run:

ansible-playbook -i inventory nas-playbook.yml --tags nas-otel
ssh nas "sudo -n midclt call app.query '[[\"name\",\"=\",\"otelcol-nas\"]]'" \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)[0]["state"])'

Expected: RUNNING.

Step 8: Verify telemetry reaches ClickStack

Query ClickHouse (per docs/operations/clickstack.md) for recent rows with host.name = nas or service.name = otelcol-nas in otel_logs / otel_metrics_*. Expected: rows present within a few minutes.

Step 9: Idempotence

Run: ansible-playbook -i inventory nas-playbook.yml --tags nas-otel Expected: changed=0 (cert re-issue is gated on the 7-day expiry check, so it skips when the cert is fresh; app.update fires only on config/compose drift via the template register change-detection).

Step 10: Commit — feat(nas): nas-otel-collector custom app → ClickStack [hl-36m6].

Phase P1 — Tooling sandbox LXC

Task 6: `nas-lxc-sandbox` role — create container + devices + start

Files:

Create: ansible/roles/nas-lxc-sandbox/tasks/main.yml
Create: ansible/roles/nas-lxc-sandbox/tasks/instance.yml
Create: ansible/roles/nas-lxc-sandbox/defaults/main.yml
Create: ansible/roles/nas-lxc-sandbox/meta/main.yml
Modify: ansible/nas-playbook.yml (add sandbox play)
Step 1: Assert the container is absent (pre-condition)

Run: ssh nas "sudo -n midclt call container.query '[[\"name\",\"=\",\"nas-support\"]]'" Expected: [].

Step 2: Create meta/main.yml (same header/shape as nas-common/meta/main.yml, role_name: nas_lxc_sandbox, description “TrueNAS libvirt-LXC sandbox lifecycle + ACL grants”).
Step 3: Create defaults/main.yml

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
nas_sandbox_name: nas-support
nas_sandbox_pool: apps
nas_sandbox_mapped_uid: 2147000001
nas_sandbox_mounts: []
nas_sandbox_curation_targets: []

Step 4: Create tasks/instance.yml (uses the spike-verified schema; create is an async job, devices are separate sync calls)

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Query container
  ansible.builtin.command:
    cmd: midclt call container.query [["name","=","{{ nas_sandbox_name }}"]]
  become: true
  changed_when: false
  register: nas_ct_q

- name: Create container (async job)
  ansible.builtin.command:
    cmd: >-
      midclt call -job container.create {{ {
        "name": nas_sandbox_name,
        "image": {"name": nas_sandbox_image_name, "version": nas_sandbox_image_version},
        "idmap": {"type": "DEFAULT"},
        "pool": nas_sandbox_pool,
        "autostart": true
      } | to_json }}
  become: true
  when: (nas_ct_q.stdout | from_json) | length == 0
  register: nas_ct_create

- name: Re-query container id
  ansible.builtin.command:
    cmd: midclt call container.query [["name","=","{{ nas_sandbox_name }}"]]
  become: true
  changed_when: false
  register: nas_ct_q2

- name: Set container id fact
  ansible.builtin.set_fact:
    nas_ct_id: "{{ (nas_ct_q2.stdout | from_json)[0].id }}"
    nas_ct_devices: "{{ (nas_ct_q2.stdout | from_json)[0].devices | default([]) }}"

- name: Add NIC device (macvlan on bond0, fixed MAC)
  ansible.builtin.command:
    cmd: >-
      midclt call container.device.create {{ {
        "container": nas_ct_id | int,
        "attributes": {"dtype": "NIC", "nic_attach": nas_sandbox_nic_parent,
                       "mac": nas_sandbox_nic_mac}
      } | to_json }}
  become: true
  when: nas_ct_devices | selectattr('attributes.dtype','equalto','NIC') | list | length == 0

- name: Add filesystem mount devices
  ansible.builtin.command:
    cmd: >-
      midclt call container.device.create {{ {
        "container": nas_ct_id | int,
        "attributes": {"dtype": "FILESYSTEM", "source": item.source, "target": item.target}
      } | to_json }}
  become: true
  loop: "{{ nas_sandbox_mounts }}"
  when: >-
    nas_ct_devices
    | selectattr('attributes.dtype','equalto','FILESYSTEM')
    | selectattr('attributes.target','equalto', item.target)
    | list | length == 0

- name: Start container
  ansible.builtin.command:
    cmd: "midclt call container.start {{ nas_ct_id | int }}"
  become: true
  changed_when: true
  when: ((nas_ct_q2.stdout | from_json)[0].status.state | default('STOPPED')) != 'RUNNING'

NIC schema note: the NIC device’s exact field set (e.g. whether macvlan vs bridge is a separate selector) was not fully spiked. Before relying on this task, confirm the accepted NIC attributes by creating a scratch container and inspecting container.device.query; adjust the attributes map if the live schema differs. nic_attach_choices confirms MACVLAN: [bond0] is valid.

Step 5: Create tasks/main.yml

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Reconcile sandbox instance
  ansible.builtin.include_tasks: instance.yml
  tags: [nas-sandbox, nas-sandbox-instance]

Step 6: Add the sandbox play to nas-playbook.yml

- name: NAS sandbox (instance lifecycle)
  hosts: nas
  gather_facts: true
  roles:
    - role: nas-lxc-sandbox
      tags: [nas-sandbox]

Step 7: Syntax-check + apply + verify running

Run:

cd ansible
ansible-playbook -i inventory nas-playbook.yml --tags nas-sandbox-instance --syntax-check
ansible-playbook -i inventory nas-playbook.yml --tags nas-sandbox-instance
ssh nas "sudo -n midclt call container.query '[[\"name\",\"=\",\"nas-support\"]]'" \
  | python3 -c 'import sys,json;x=json.load(sys.stdin)[0];print(x["status"]["state"], x["id"])'

Expected: RUNNING <id>.

Step 8: Idempotence — re-run --tags nas-sandbox-instance; expected changed=0 (query/create/device guards all skip).
Step 9: Commit — feat(nas): nas-lxc-sandbox instance via container.* [hl-36m6].

Task 7: Sandbox bootstrap — SSH reachability as `nas-automation`

Files:

Create: ansible/roles/nas-lxc-sandbox/tasks/bootstrap.yml
Modify: ansible/roles/nas-lxc-sandbox/tasks/main.yml (add include)
Modify: ansible/inventory/hosts.yml (add nas-support host)

External prereq: the Firewalla DHCP reservation for 192.168.20.202 keyed on nas_sandbox_nic_mac must exist so the macvlan NIC gets .202.

Model: sonnet (namespace entry + first-boot bootstrap).

Step 1: Create the in-container user + SSH access via nsenter (proven path)

The ubuntu:noble:amd64:default image defaults to root/ubuntu and ships no sshd. There is no confirmed container.exec middleware method (the method survey did not surface one), so we enter the running container’s namespaces from the host with nsenter — the exact mechanism proven in the read/write spike (nsenter -t <init-pid> -a). bootstrap.yml is self-contained — it re-queries the container id/pid, so it works under --tags nas-sandbox-bootstrap in isolation (does not rely on nas_ct_id from instance.yml):

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Read automation SSH public key from Vault
  ansible.builtin.set_fact:
    nas_automation_pubkey: >-
      {{ lookup('community.hashi_vault.vault_kv2_get',
                'fzymgc-house/infrastructure/nas/automation-ssh').data.data.public_key }}
  no_log: true

- name: Query container (self-contained; do not assume instance.yml ran this play)
  ansible.builtin.command:
    cmd: midclt call container.query [["name","=","{{ nas_sandbox_name }}"]]
  become: true
  changed_when: false
  register: nas_bootstrap_q

- name: Capture init pid; require RUNNING
  ansible.builtin.set_fact:
    nas_ct_pid: "{{ (nas_bootstrap_q.stdout | from_json)[0].status.pid }}"
  failed_when: (nas_bootstrap_q.stdout | from_json)[0].status.state != 'RUNNING'

- name: Probe whether the sandbox is already bootstrapped (idempotence guard)
  ansible.builtin.command:
    cmd: "nsenter -t {{ nas_ct_pid }} -a -- id nas-automation"
  become: true
  changed_when: false
  failed_when: false
  register: nas_bootstrap_probe

- name: Bootstrap user + sshd + key inside the container (via nsenter)
  ansible.builtin.command:
    cmd: "nsenter -t {{ nas_ct_pid }} -a -- /bin/bash -s"
    stdin: |
      set -euo pipefail
      export DEBIAN_FRONTEND=noninteractive
      command -v sshd >/dev/null || { apt-get update && apt-get install -y openssh-server sudo; }
      id nas-automation >/dev/null 2>&1 || useradd -m -s /bin/bash nas-automation
      install -d -m700 -o nas-automation -g nas-automation /home/nas-automation/.ssh
      printf '%s\n' '{{ nas_automation_pubkey }}' > /home/nas-automation/.ssh/authorized_keys
      chmod 600 /home/nas-automation/.ssh/authorized_keys
      chown nas-automation:nas-automation /home/nas-automation/.ssh/authorized_keys
      printf 'nas-automation ALL=(ALL) NOPASSWD:ALL\n' > /etc/sudoers.d/nas-automation
      systemctl enable --now ssh
  become: true
  no_log: true
  when: nas_bootstrap_probe.rc != 0
  changed_when: true

Mechanism note: nsenter reaching the container namespaces is spike-proven (Spike 2 used nsenter -t <pid> -a to act as container-root). If a future TrueNAS release adds a supported container.exec/container.shell middleware method, prefer it over host nsenter.

Step 2: Add nas-support to inventory

In ansible/inventory/hosts.yml, add:

nas_sandbox_hosts:
  hosts:
    nas-support:
      ansible_host: 192.168.20.202
      ansible_ssh_user: nas-automation

Step 3: Add the include to nas-lxc-sandbox/tasks/main.yml

- name: Bootstrap sandbox SSH reachability
  ansible.builtin.include_tasks: bootstrap.yml
  tags: [nas-sandbox, nas-sandbox-bootstrap]

Step 4: Apply + verify SSH

Run:

cd ansible
ansible-playbook -i inventory nas-playbook.yml --tags nas-sandbox-bootstrap
ansible -i inventory nas-support -m ansible.builtin.ping

Expected: nas-support | SUCCESS => {"ping": "pong"}.

Step 5: Commit — feat(nas): bootstrap sandbox ssh as nas-automation [hl-36m6].

Task 8: Acltype-aware ACL granting for curation targets

Files:

Create: ansible/roles/nas-lxc-sandbox/tasks/acls.yml
Modify: ansible/roles/nas-lxc-sandbox/tasks/main.yml (add include)
Step 1: Re-confirm the DEFAULT-mapped UID (in case the namespace base differs)

Add to acls.yml:

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Get sandbox init pid
  ansible.builtin.command:
    cmd: midclt call container.query [["name","=","{{ nas_sandbox_name }}"]]
  become: true
  changed_when: false
  register: nas_ct_q3

- name: Read container-root host UID from uid_map
  ansible.builtin.shell:
    cmd: "awk 'NR==1{print $2}' /proc/{{ (nas_ct_q3.stdout | from_json)[0].status.pid }}/uid_map"
  become: true
  changed_when: false
  register: nas_mapped_uid_live

- name: Assert mapped UID matches expectation
  ansible.builtin.assert:
    that: nas_mapped_uid_live.stdout | int == nas_sandbox_mapped_uid | int
    fail_msg: >-
      DEFAULT idmap mapped container-root to {{ nas_mapped_uid_live.stdout }},
      expected {{ nas_sandbox_mapped_uid }}. Update nas_sandbox_mapped_uid and re-run.

Step 2: Detect acltype per target and grant rw

Append to acls.yml:

- name: Detect acltype per curation target
  ansible.builtin.command:
    cmd: "zfs get -H -o value acltype {{ item | regex_replace('^/mnt/', '') }}"
  become: true
  changed_when: false
  loop: "{{ nas_sandbox_curation_targets }}"
  register: nas_acltypes

- name: Grant POSIX1E ACL (setfacl)
  ansible.builtin.command:
    cmd: "setfacl -R -m u:{{ nas_sandbox_mapped_uid }}:rwx {{ item.item }}"
  become: true
  loop: "{{ nas_acltypes.results }}"
  loop_control: { label: "{{ item.item }}" }
  when: item.stdout == 'posix'
  register: nas_setfacl

- name: Grant POSIX1E default ACL (inheritance)
  ansible.builtin.command:
    cmd: "setfacl -d -m u:{{ nas_sandbox_mapped_uid }}:rwx {{ item.item }}"
  become: true
  loop: "{{ nas_acltypes.results }}"
  loop_control: { label: "{{ item.item }}" }
  when: item.stdout == 'posix'

- name: Build flat list of NFSv4 curation targets
  ansible.builtin.set_fact:
    nas_nfsv4_targets: >-
      {{ nas_acltypes.results | selectattr('stdout', 'equalto', 'nfsv4')
         | map(attribute='item') | list }}

- name: Fetch current ACL for each NFSv4 target (so the grant is additive)
  ansible.builtin.command:
    cmd: "midclt call filesystem.getacl {{ item }}"
  become: true
  changed_when: false
  loop: "{{ nas_nfsv4_targets }}"
  register: nas_nfsv4_getacl

- name: Grant NFSv4 ACL — append USER ACE only when absent (idempotent)
  ansible.builtin.command:
    cmd: >-
      midclt call -job filesystem.setacl {{ {
        "path": item.item,
        "dacl": (item.stdout | from_json).acl + [{
          "tag": "USER", "type": "ALLOW", "id": nas_sandbox_mapped_uid | int, "who": None,
          "perms": {"BASIC": "FULL_CONTROL"}, "flags": {"BASIC": "INHERIT"}}],
        "options": {"recursive": true, "traverse": true}
      } | to_json }}
  become: true
  loop: "{{ nas_nfsv4_getacl.results }}"
  loop_control: { label: "{{ item.item }}" }
  when:
    - (item.stdout | from_json).acl
      | selectattr('tag', 'equalto', 'USER')
      | selectattr('id', 'equalto', nas_sandbox_mapped_uid | int)
      | list | length == 0

The getacl fetch (item.item = path, item.stdout = current ACL JSON) makes the grant additive — it appends the USER ACE to the existing ACL rather than replacing it — and the when guard skips targets that already carry the ACE, so re-runs are changed=0.

Step 3: Make the POSIX1E grant idempotent

ansible.builtin.command: setfacl reports changed on every run. Replace the two POSIX setfacl tasks from Step 2 with this single self-checking task (NFSv4 already self-guards via its when in Step 2):

- name: Grant POSIX1E ACL (rwx + default) — idempotent
  ansible.builtin.shell:
    cmd: |
      set -euo pipefail
      before=$(getfacl -pcE {{ item.item }} 2>/dev/null || true)
      setfacl -R -m u:{{ nas_sandbox_mapped_uid }}:rwx {{ item.item }}
      setfacl -d -m u:{{ nas_sandbox_mapped_uid }}:rwx {{ item.item }}
      after=$(getfacl -pcE {{ item.item }} 2>/dev/null || true)
      [ "$before" = "$after" ] && echo UNCHANGED || echo CHANGED
  become: true
  loop: "{{ nas_acltypes.results }}"
  loop_control: { label: "{{ item.item }}" }
  when: item.stdout == 'posix'
  register: nas_setfacl
  changed_when: "'CHANGED' in (stdout | default(''))"

(In a loop, changed_when is evaluated per iteration; stdout resolves to the current item’s output.)

Step 4: Add the include to main.yml

- name: Grant acltype-aware ACLs on curation targets
  ansible.builtin.include_tasks: acls.yml
  when: nas_sandbox_curation_targets | length > 0
  tags: [nas-sandbox, nas-sandbox-acls]

Step 5: End-to-end verify (read + write from inside the sandbox)

Set a real target in host_vars/nas.yml (e.g. nas_sandbox_curation_targets: [/mnt/main/multimedia]), add it to nas_sandbox_mounts as rw, apply, then:

ssh nas-support "ls /mnt/main/multimedia | head; touch /mnt/main/multimedia/.nas-support-write-test && echo WRITE_OK && rm /mnt/main/multimedia/.nas-support-write-test"

Expected: directory lists; WRITE_OK. Repeat for one POSIX1E and one NFSv4 target.

Step 6: Commit — feat(nas): acltype-aware ACL grants for sandbox curation targets [hl-36m6].

Task 9: `nas-sandbox-tools` role — interior tooling

Files:

Create: ansible/roles/nas-sandbox-tools/tasks/main.yml
Create: ansible/roles/nas-sandbox-tools/defaults/main.yml
Create: ansible/roles/nas-sandbox-tools/templates/kopia-repo.env.j2
Modify: ansible/nas-playbook.yml (add interior play targeting nas-support)
Step 1: Create defaults/main.yml

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
nas_tools_apt:
  - ncdu
  - rmlint
  - rclone
  - jq
nas_kopia_version: "0.18.2"
nas_kopia_r2_vault_path: fzymgc-house/infrastructure/nas/kopia-r2
nas_kopia_s3_endpoint: "40753dbbbbd1540f02bd0707935ddb3f.r2.cloudflarestorage.com"
nas_kopia_s3_bucket: fzymgc-cluster-storage
nas_kopia_s3_prefix: nas-backups

Step 2: Create tasks/main.yml (apt tools + Homebrew + Kopia)

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Install curation/apt tooling
  ansible.builtin.apt:
    name: "{{ nas_tools_apt }}"
    state: present
    update_cache: true
  become: true

- name: Check Homebrew present
  ansible.builtin.stat:
    path: /home/linuxbrew/.linuxbrew/bin/brew
  register: nas_brew

- name: Install Homebrew (non-interactive)
  ansible.builtin.shell:
    cmd: >-
      NONINTERACTIVE=1 /bin/bash -c
      "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  become: true
  become_user: nas-automation
  when: not nas_brew.stat.exists

- name: Install Kopia binary
  ansible.builtin.get_url:
    url: "https://github.com/kopia/kopia/releases/download/v{{ nas_kopia_version }}/kopia-{{ nas_kopia_version }}-linux-x64.tar.gz"
    dest: /tmp/kopia.tar.gz
    mode: "0644"
  become: true

- name: Unpack Kopia to /usr/local/bin
  ansible.builtin.unarchive:
    src: /tmp/kopia.tar.gz
    dest: /usr/local/bin
    remote_src: true
    extra_opts: ["--strip-components=1", "--wildcards", "*/kopia"]
    creates: /usr/local/bin/kopia
  become: true

- name: Read Kopia R2 creds from Vault
  ansible.builtin.set_fact:
    nas_kopia_creds: "{{ lookup('community.hashi_vault.vault_kv2_get', nas_kopia_r2_vault_path).data.data }}"
  no_log: true

- name: Render Kopia repo env
  ansible.builtin.template:
    src: kopia-repo.env.j2
    dest: /etc/nas-kopia.env
    mode: "0600"
  become: true
  no_log: true

Step 3: Create templates/kopia-repo.env.j2

KOPIA_PASSWORD={{ nas_kopia_creds.repo_password }}
AWS_ACCESS_KEY_ID={{ nas_kopia_creds.username }}
AWS_SECRET_ACCESS_KEY={{ nas_kopia_creds.password }}
KOPIA_S3_ENDPOINT={{ nas_kopia_s3_endpoint }}
KOPIA_S3_BUCKET={{ nas_kopia_s3_bucket }}
KOPIA_S3_PREFIX={{ nas_kopia_s3_prefix }}/

Step 4: Add the interior play to nas-playbook.yml

- name: NAS sandbox interior tooling
  hosts: nas_sandbox_hosts
  gather_facts: true
  roles:
    - role: nas-sandbox-tools
      tags: [nas-sandbox-tools]

Step 5: Apply + verify

Run:

cd ansible
ansible-playbook -i inventory nas-playbook.yml --tags nas-sandbox-tools
ssh nas-support "/usr/local/bin/kopia --version; ncdu --version; /home/linuxbrew/.linuxbrew/bin/brew --version"

Expected: versions print for kopia, ncdu, brew.

Step 6: Idempotence — re-run; expected changed=0 (stat/creates guards skip installs).
Step 7: Commit — feat(nas): nas-sandbox-tools (homebrew, kopia, curation utils) [hl-36m6].

Task 10: Documentation + roles reference

Files:

Modify: ansible/CLAUDE.md (Roles Reference table — add the four nas-* roles + nas-playbook.yml)
Create: docs/operations/nas.md (operational runbook: bootstrap order, re-run tags, ACL-grant model, sandbox access)
Step 1: Add the roles to ansible/CLAUDE.md

Add rows for nas-common, nas-otel-collector (new role; the Firewalla otel-collector role is unchanged), nas-lxc-sandbox, nas-sandbox-tools, and a nas-playbook.yml phases section mirroring the existing k3s-playbook.yml table.

Step 2: Write docs/operations/nas.md with Starlight frontmatter (title, description), covering: appliance-via-API principle, one-time sean bootstrap → nas-automation, the --tags map, the DEFAULT-idmap + acltype-aware ACL model (UID 2147000001), and how to add a curation target.
Step 3: Lint

Run: uvx rumdl@0.2.17 fmt docs/operations/nas.md && uvx rumdl@0.2.17 check docs/operations/nas.md ansible/CLAUDE.md Expected: no issues.

Step 4: Commit — docs(nas): roles reference + operations runbook [hl-36m6].

Verification summary (whole-plan acceptance)

P0: nas reachable as nas-automation; drift baseline rendered; otelcol-nas app RUNNING; NAS telemetry visible in ClickStack (host.name=nas); Firewalla otel-collector unchanged (check-mode clean).
P1: nas-support LXC RUNNING; SSH in as nas-automation; /mnt/main + /mnt/apps visible; read+write proven on one POSIX1E and one NFSv4 curation target; brew/kopia/ncdu installed; nas-support.fzymgc.house resolves.
Idempotence: every ansible-playbook nas-playbook.yml tag re-run reports changed=0.

NAS Ansible Foundation + Tooling Sandbox — Implementation Plan (P0+P1)