NAS Kopia → Backblaze B2 backup — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Stand up a self-managed, client-encrypted, Object-Lock-immutable Kopia backup of the NAS’s ~2.3 TiB of irreplaceable data to Backblaze B2, with silent-failure alerting and a verified restore, then retire the failing rclone + Storj tasks.

Architecture: A host-autonomous TrueNAS periodic snapshot task (pool.snapshottask) snapshots each in-scope leaf dataset nightly; a scheduled Kopia CLI inside the merged hl-36m6 nas-support LXC reads the newest .zfs snapshot per leaf (selected by a --before-folder-action) and uploads to B2 via the S3-compatible endpoint. No second NIC, no host↔sandbox SSH, no sandbox→host privilege — the sandbox only reads its bind-mounted leaves.

Tech Stack: Ansible (FQCN, midclt-over-SSH), Kopia (Homebrew-installed in the sandbox), Backblaze B2 (S3-compat + Object Lock), HashiCorp Vault, TrueNAS 26 middleware (pool.snapshottask, filesystem.setacl), systemd timers, Pushover (alerting — ntfy decommissioned, hl-ney2.20).

Design of record: docs/engineering/specs/2026-06-27-nas-kopia-b2-backup-design.md (bead hl-ney2). This plan implements the post-merge Approach Z (§4): snapshot-task + per-leaf .zfs read. hl-36m6 merged (#1404); all nas-* roles below already exist on main.

Post-merge grounding (read before starting)

hl-36m6 (#1404) is on main. This plan reuses and populates its shipped mechanisms — it does not reinvent them:

hl-36m6 artifact (exists)	How this plan uses it
`ansible/roles/nas-sandbox-tools` — installs `kopia` via Homebrew at `/home/linuxbrew/.linuxbrew/bin/kopia`; renders an R2 stub env `/etc/nas-kopia.env` (no repo/backup)	Reuse the binary. The B2 backup is a distinct Kopia repo with its own `KOPIA_CONFIG_PATH`; the R2 stub is left untouched (coexist).
`ansible/roles/nas-lxc-sandbox/tasks/instance.yml` — container + NIC + `FILESYSTEM` mounts from `nas_sandbox_mounts`	Populate `nas_sandbox_mounts` with the in-scope leaves. CONSTRAINT: only LEAF datasets mount (pool root / parents-with-children fail at container start).
`ansible/roles/nas-lxc-sandbox/tasks/acls.yml` — grants UID `2147000001` rwx to `nas_sandbox_curation_targets` (posix `setfacl` / nfsv4 `filesystem.setacl`)	Clone its branch logic for a read-only (`rX`) grant over the backup leaves (a new task file in the new role).
`ansible/inventory/host_vars/nas.yml` — sandbox vars; `nas_sandbox_mounts: [multimedia]`, `nas_sandbox_curation_targets: [multimedia]`	Expand `nas_sandbox_mounts` to all in-scope leaves. `nas_kopia_leaves` goes in `group_vars/all.yml`; B2/policy vars in the new role’s `defaults/main.yml` (variable scoping — see Task 1).
`ansible/inventory/hosts.yml` — group `nas_sandbox_hosts` → `nas-support` @ `192.168.20.202`, user `nas-automation`	Interior backup play targets this group.
`ansible/nas-playbook.yml` — plays for `nas-common`, `nas-lxc-sandbox` (hosts: `nas`), `nas-sandbox-tools` (hosts: `nas_sandbox_hosts`)	Add a host-side snapshot/ACL role to the `nas` plays and an interior backup role to the `nas_sandbox_hosts` play.
`ansible/roles/router-kopia-backup/` — the proven Kopia-on-S3 role (templates, systemd units)	Clone-and-adapt the systemd service/timer + backup-script shape.

The sandbox runs curation/backup AS ROOT (sudo -i), because the ACL grant is to container-root (host UID 2147000001), not the nas-automation login uid. The Kopia backup systemd unit therefore runs as root inside the sandbox.

File structure

ansible/
  inventory/group_vars/all.yml                # MODIFY: add nas_kopia_leaves (shared by `nas` + `nas_sandbox_hosts`)
  inventory/host_vars/nas.yml                 # MODIFY: expand nas_sandbox_mounts (leaves); leave curation_targets
  nas-playbook.yml                            # MODIFY: add nas-kopia-snapshot (hosts: nas) + nas-kopia-backup (hosts: nas_sandbox_hosts)
  roles/
    nas-kopia-snapshot/                       # CREATE (host-side, runs on nas via midclt)
      defaults/main.yml                       #   snapshot-task schedule/lifetime; read-ACL list derivation
      meta/main.yml
      tasks/main.yml                          #   pool.snapshottask create/reconcile; snapdir=visible; read-only ACL grants
    nas-kopia-backup/                         # CREATE (interior, runs on nas_sandbox_hosts over SSH)
      defaults/main.yml                       #   B2 repo, retention, paths, sources, monitoring
      meta/main.yml
      tasks/main.yml                          #   repo connect/create + object-lock; policies; deploy units; monitoring
      handlers/main.yml                       #   "Reload systemd nas-kopia"
      templates/
        kopia-b2-backup.sh.j2                 #   per-leaf `snapshot create` + maintenance
        kopia-snapshot-pick.sh.j2             #   --before-folder-action: echo newest .zfs snapshot path
        kopia-b2-backup.service.j2            #   systemd oneshot (User=root)
        kopia-b2-backup.timer.j2              #   nightly 03:00
        kopia-freshness.sh.j2                 #   dead-man's-switch: alert Pushover if last-success stale
        kopia-freshness.service.j2 / .timer.j2
tf/
  vault/policy-nas-kopia.tf                   # CREATE: read policy for the kopia-b2 secret path
docs/
  reference/secrets.md                        # MODIFY: document fzymgc-house/infrastructure/nas/kopia-b2

Task 0: Spike — prove the load-bearing assumptions (BUILD GATE)

Nothing below Task 0 may begin until this passes. This is the spec’s Cutover step 0. It is a manual spike run from the workstation against the live NAS + sandbox; it writes no repo code.

Files: none (exploratory). Record findings in bd note hl-ney2.

Step 1: Reach the NAS and sandbox

Run (the 1Password agent auto-locks — offer only the correct key):

NAS="ssh -o IdentitiesOnly=yes -o IdentityFile=$HOME/.ssh/seanb4t_ed25519.pub -o ConnectTimeout=10 nas"
$NAS 'echo nas-ok; sudo zfs --version | head -1'
ssh -o ConnectTimeout=10 nas-automation@192.168.20.202 'echo sandbox-ok; id'

Expected: nas-ok + zfs version; sandbox-ok + the nas-automation id.

Step 2: Enumerate in-scope LEAF datasets authoritatively

Run on the NAS:

$NAS 'sudo zfs list -H -o name -t filesystem -r main/paperless-docs main/fzymgc-house \
  main/homes main/multimedia main/cloud-sync-backup main/bwcgroup-data main/immich \
  main/family_share main/shared'

Expected: the full dataset tree. A dataset is a leaf iff no other returned dataset has it as a prefix. Record the leaf list — it drives nas_kopia_leaves (Task 1). Confirm/replace the candidate list in Task 1.

Step 3: Confirm idmap base UID inside the running sandbox

$NAS 'pid=$(sudo midclt call container.query "[[\"name\",\"=\",\"nas-support\"]]" | python3 -c "import sys,json;print(json.load(sys.stdin)[0][\"status\"][\"pid\"])"); awk "NR==1{print \$2}" /proc/$pid/uid_map'

Expected: 2147000001. If different, update nas_sandbox_mapped_uid everywhere before proceeding.

Step 4: THE linchpin — prove a host snapshot is readable in the sandbox via .zfs

On the NAS, snapshot the one already-mounted leaf (multimedia) and set snapdir=visible:

$NAS 'sudo zfs set snapdir=visible main/multimedia && sudo zfs snapshot main/multimedia@kopia-spike'

In the sandbox, confirm the snapshot is listable and readable:

ssh nas-automation@192.168.20.202 \
  'sudo ls -d /mnt/main/multimedia/.zfs/snapshot/kopia-spike && sudo ls /mnt/main/multimedia/.zfs/snapshot/kopia-spike | head'

Expected: the snapshot directory lists and its contents are readable.

If it FAILS (auto-mount does not propagate through the bind-mount): switch to the spec’s §4 persistent-device fallback (mount one persistent snapshot per leaf as a dedicated FILESYSTEM device). Record the decision in bd note and adjust Tasks 6/11 accordingly before continuing.
Step 5: Clean up the spike + record result

$NAS 'sudo zfs destroy main/multimedia@kopia-spike'
cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster
bd note hl-ney2 "spike(step0) result: leaves=<list>; mapped_uid=<n>; .zfs-in-sandbox=PASS|FAIL(+fallback)"

Task 1: Inventory vars — leaves (group_vars/all) + sandbox mounts

Files:

Modify: ansible/inventory/group_vars/all.yml
Modify: ansible/inventory/host_vars/nas.yml

Variable scoping (critical). The two plays target different host groups — host-side roles run on nas, the interior backup role on nas_sandbox_hosts (nas-support). host_vars/nas.yml is not visible to nas-support. So: nas_kopia_leaves (needed by both) goes in group_vars/all.yml; B2/retention/ policy vars (interior-only) go in nas-kopia-backup/defaults/main.yml (Task 6); snapshot-task schedule vars (host-only) live in nas-kopia-snapshot/defaults (Task 5); only nas_sandbox_mounts / nas_sandbox_curation_targets stay in host_vars/nas.yml.

Step 1: Add the canonical leaf list to group_vars/all.yml

Append to ansible/inventory/group_vars/all.yml (replace the candidate leaves with Task 0 Step 2’s authoritative output — a dataset is a leaf iff no other returned dataset has it as a prefix):

# Canonical in-scope LEAF datasets for the NAS Kopia→B2 backup (hl-ney2).
# Shared by host-side roles (mounts, snapshot task, ACLs on `nas`) and the interior
# backup role (derives nas_kopia_sources, on `nas-support`) — hence group_vars/all.yml.
# Only leaf datasets bind-mount into the sandbox (libvirt-LXC + DEFAULT idmap constraint).
nas_kopia_leaves:
  - main/homes
  - main/multimedia
  - main/cloud-sync-backup
  - main/bwcgroup-data
  - main/immich
  - main/family_share
  - main/shared
  - main/paperless-docs/data
  - main/paperless-docs/export
  - main/paperless-docs/ingestion/sean
  - main/paperless-docs/storage
  - main/fzymgc-house/incus/storage/precipice
  - main/fzymgc-house/incus/storage/doorsportal1
  - main/fzymgc-house/incus/storage/container-apps

Step 2: Set the explicit sandbox mounts in host_vars/nas.yml

Set nas_sandbox_mounts (consumed by nas-lxc-sandbox on nas) to the explicit list below — one entry per leaf (keep this projection in sync with nas_kopia_leaves). Leave nas_sandbox_curation_targets as-is ([/mnt/main/multimedia] — hl-36m6’s rwx curation grant; the backup uses a separate read-only grant in Task 5):

nas_sandbox_mounts:
  - { source: /mnt/main/homes, target: /mnt/main/homes }
  - { source: /mnt/main/multimedia, target: /mnt/main/multimedia }
  - { source: /mnt/main/cloud-sync-backup, target: /mnt/main/cloud-sync-backup }
  - { source: /mnt/main/bwcgroup-data, target: /mnt/main/bwcgroup-data }
  - { source: /mnt/main/immich, target: /mnt/main/immich }
  - { source: /mnt/main/family_share, target: /mnt/main/family_share }
  - { source: /mnt/main/shared, target: /mnt/main/shared }
  - { source: /mnt/main/paperless-docs/data, target: /mnt/main/paperless-docs/data }
  - { source: /mnt/main/paperless-docs/export, target: /mnt/main/paperless-docs/export }
  - { source: /mnt/main/paperless-docs/ingestion/sean, target: /mnt/main/paperless-docs/ingestion/sean }
  - { source: /mnt/main/paperless-docs/storage, target: /mnt/main/paperless-docs/storage }
  - { source: /mnt/main/fzymgc-house/incus/storage/precipice, target: /mnt/main/fzymgc-house/incus/storage/precipice }
  - { source: /mnt/main/fzymgc-house/incus/storage/doorsportal1, target: /mnt/main/fzymgc-house/incus/storage/doorsportal1 }
  - { source: /mnt/main/fzymgc-house/incus/storage/container-apps, target: /mnt/main/fzymgc-house/incus/storage/container-apps }

Step 3: Lint + syntax-check

cd ansible && uvx yamllint inventory/group_vars/all.yml inventory/host_vars/nas.yml && \
  ansible-playbook -i inventory/hosts.yml nas-playbook.yml --syntax-check

Expected: clean yamllint; syntax-check OK.

Step 4: Commit

jj commit -m "feat(nas-kopia): inventory vars — leaves (group_vars/all) + sandbox mounts [hl-ney2]"

Task 2: Create the B2 bucket (Object Lock + versioning) + scoped key

Files: none (B2 CLI; record outputs into Vault in Task 3). Run from the workstation (b2 CLI v4.7.1 is authorized).

Step 1: Create the Object-Lock bucket

b2 bucket create --file-lock-enabled fzymgc-nas-kopia allPrivate
b2 bucket get fzymgc-nas-kopia   # confirm fileLockConfiguration.isFileLockEnabled == true
b2 account get | python3 -c 'import sys,json;print(json.load(sys.stdin)["s3ApiUrl"])'   # → region for nas_kopia_b2_region

Expected: bucket created; file lock enabled; note the S3 endpoint region.

Step 2: Create a bucket-scoped application key

b2 key create --bucket fzymgc-nas-kopia nas-kopia-b2 \
  listBuckets,listFiles,readFiles,writeFiles,deleteFiles,readBucketReplications,readFileRetentions,writeFileRetentions,readBucketEncryption

Expected: prints applicationKeyId and applicationKey. Copy both for Task 3 (shown once). The writeFileRetentions capability lets Kopia maintenance extend Object Lock.

Step 3: Record (no commit — secrets go to Vault next)

bd note hl-ney2 "B2 bucket fzymgc-nas-kopia created (file-lock on, region <r>); scoped key nas-kopia-b2 created."

Task 3: Vault secret + offline repo-password copy

Files: none (Vault). The repo password is the one irrecoverable secret.

Step 1: Generate the repo password and write the Vault secret

export VAULT_ADDR=https://vault.fzymgc.house
REPO_PASS=$(openssl rand -base64 32)
vault kv put secret/fzymgc-house/infrastructure/nas/kopia-b2 \
  b2_key_id="<applicationKeyId from Task 2>" \
  b2_application_key="<applicationKey from Task 2>" \
  repo_password="$REPO_PASS"
echo "OFFLINE-COPY THIS repo_password NOW (1Password): $REPO_PASS"

Step 2: Store the offline copy

Save repo_password to 1Password (item “NAS Kopia B2 repo password”) and confirm it is retrievable independently of Vault. Lose this → B2 is unreadable ciphertext.

Step 3: Verify readback

vault kv get -format=json secret/fzymgc-house/infrastructure/nas/kopia-b2 \
  | python3 -c 'import sys,json;d=json.load(sys.stdin)["data"]["data"];print(sorted(d.keys()))'

Expected: ['b2_application_key', 'b2_key_id', 'repo_password'].

Task 4: Vault read policy for the new secret path

Files:

Create: tf/vault/policy-nas-kopia.tf
Modify: docs/reference/secrets.md

The Vault read policy lets the backup role’s vault_kv2_get lookup read the B2 creds.

Step 1: Add the policy (model on tf/vault/policy-otel-collector-firewalla.tf)

# Read-only access to the NAS Kopia→B2 backup credentials.
resource "vault_policy" "nas_kopia" {
  name   = "nas-kopia"
  policy = <<-EOT
    path "secret/data/fzymgc-house/infrastructure/nas/kopia-b2" {
      capabilities = ["read"]
    }
  EOT
}

Step 2: Document the path

Add a row to docs/reference/secrets.md under the NAS section: fzymgc-house/infrastructure/nas/kopia-b2 — b2_key_id, b2_application_key, repo_password (Kopia→B2 backup; repo_password also held offline).

Step 3: Plan + apply (per docs/operations/hcp-terraform.md)

cd tf/vault && terraform plan   # via HCP workflow; confirm only the new policy added

Expected: plan shows vault_policy.nas_kopia to add, nothing destroyed.

Step 4: Commit

jj commit -m "feat(vault): read policy for nas kopia-b2 secret [hl-ney2]"

Task 5: `nas-kopia-snapshot` role — host-side snapshot task

v1 RE-SCOPE (2026-06-28, ADR hl-zrbh): snapshots are dropped for v1 (live backup). This task is implemented as the nas-kopia-acls role — host-side read-only ACL grants only (no pool.snapshottask, no snapdir=visible). The snapshot-task steps below are retained for the deferred consistency fast-follow.

Files:

Create: ansible/roles/nas-kopia-snapshot/{meta,defaults,tasks}/main.yml

The host play creates a TrueNAS periodic snapshot task per in-scope leaf (autonomous; no sandbox involvement).

Step 1: meta + defaults

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
galaxy_info:
  author: fzymgc-house
  description: Host-side TrueNAS snapshot task + read-only ACLs for the NAS Kopia→B2 backup.
  license: MIT-0
  min_ansible_version: "2.14"
dependencies: []

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
nas_kopia_snapshot_naming: "kopia-%Y%m%d-%H%M"
nas_kopia_snapshot_schedule_hour: 2
nas_kopia_snapshot_schedule_minute: 0
nas_kopia_snapshot_lifetime_days: 3
nas_sandbox_mapped_uid: 2147000001

Step 2: tasks/main.yml — snapdir=visible, snapshot task, read-only ACLs

midclt calls use the argv form (string form corrupts JSON — see hl-36m6’s automation-user.yml).

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Make snapshot dirs explicit on in-scope leaves
  ansible.builtin.command:
    argv: [zfs, set, snapdir=visible, "{{ item }}"]
  become: true
  loop: "{{ nas_kopia_leaves }}"
  changed_when: false   # idempotent set; value rarely changes

- name: Query existing periodic snapshot tasks
  ansible.builtin.command:
    argv: [midclt, call, pool.snapshottask.query]
  become: true
  changed_when: false
  check_mode: false
  register: nas_snaptask_q

- name: Create snapshot task per in-scope leaf (when absent)
  ansible.builtin.command:
    argv:
      - midclt
      - call
      - pool.snapshottask.create
      - >-
        {{ {
          "dataset": item,
          "recursive": false,
          "lifetime_value": nas_kopia_snapshot_lifetime_days,
          "lifetime_unit": "DAY",
          "naming_schema": nas_kopia_snapshot_naming,
          "schedule": {"hour": nas_kopia_snapshot_schedule_hour | string,
                       "minute": nas_kopia_snapshot_schedule_minute | string,
                       "dom": "*", "month": "*", "dow": "*"},
          "enabled": true
        } | to_json }}
  become: true
  loop: "{{ nas_kopia_leaves }}"
  loop_control: { label: "{{ item }}" }
  when: >-
    (nas_snaptask_q.stdout | from_json)
    | selectattr('dataset', 'equalto', item)
    | selectattr('naming_schema', 'equalto', nas_kopia_snapshot_naming)
    | list | length == 0
  register: nas_snaptask_create

- name: Read-only ACL grant for container-root (posix/nfsv4 branch)
  ansible.builtin.include_tasks: acls-readonly.yml

Step 3: tasks/acls-readonly.yml — clone hl-36m6 acls.yml with rX (read) perms

Clone the structure of ansible/roles/nas-lxc-sandbox/tasks/acls.yml, changing rwx→rX and FULL_CONTROL→read, and iterating over nas_kopia_leaves mapped to /mnt/<leaf> minus those already in nas_sandbox_curation_targets (which already have rwx):

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Compute read-only ACL targets (in-scope leaves not already rwx-granted)
  ansible.builtin.set_fact:
    nas_kopia_ro_targets: >-
      {{ nas_kopia_leaves | map('regex_replace', '^(.+)$', '/mnt/\1') | list
         | difference(nas_sandbox_curation_targets) }}

- name: Detect acltype per read-only target
  ansible.builtin.command:
    argv: [zfs, get, -H, -o, value, acltype, "{{ item | regex_replace('^/mnt/', '') }}"]
  become: true
  changed_when: false
  check_mode: false
  loop: "{{ nas_kopia_ro_targets }}"
  register: nas_ro_acltypes

- name: Grant POSIX1E read ACL (rX + default) — idempotent
  ansible.builtin.shell:
    cmd: |
      set -euo pipefail
      before=$(getfacl -pcE {{ item.item }} 2>/dev/null || true)
      setfacl -R -m u:{{ nas_sandbox_mapped_uid }}:rX {{ item.item }}
      setfacl -d -m u:{{ nas_sandbox_mapped_uid }}:rX {{ item.item }}
      after=$(getfacl -pcE {{ item.item }} 2>/dev/null || true)
      [ "$before" = "$after" ] && echo UNCHANGED || echo CHANGED
  become: true
  loop: "{{ nas_ro_acltypes.results }}"
  loop_control: { label: "{{ item.item }}" }
  when: item.stdout == 'posix'
  changed_when: "'CHANGED' in (stdout | default(''))"

- name: Build NFSv4 read-only target list
  ansible.builtin.set_fact:
    nas_ro_nfsv4: >-
      {{ nas_ro_acltypes.results | selectattr('stdout', 'equalto', 'nfsv4')
         | map(attribute='item') | list }}

- name: Fetch current ACL for each NFSv4 target (additive grant)
  ansible.builtin.command:
    argv: [midclt, call, filesystem.getacl, "{{ item }}"]
  become: true
  changed_when: false
  check_mode: false
  loop: "{{ nas_ro_nfsv4 }}"
  register: nas_ro_getacl

- name: Append NFSv4 READ ACE for container-root when absent (idempotent)
  ansible.builtin.command:
    argv:
      - midclt
      - call
      - --job
      - filesystem.setacl
      - >-
        {{ {
          "path": item.item,
          "dacl": (item.stdout | from_json).acl + [{
            "tag": "USER", "type": "ALLOW", "id": nas_sandbox_mapped_uid | int, "who": None,
            "perms": {"BASIC": "READ"}, "flags": {"BASIC": "INHERIT"}}],
          "options": {"recursive": true, "traverse": true}
        } | to_json }}
  become: true
  loop: "{{ nas_ro_getacl.results }}"
  loop_control: { label: "{{ item.item }}" }
  when:
    - (item.stdout | from_json).acl
      | selectattr('tag', 'equalto', 'USER')
      | selectattr('id', 'equalto', nas_sandbox_mapped_uid | int)
      | list | length == 0

Schema confirmation needed at build (Task 0 follow-up): verify pool.snapshottask.create field names (lifetime_value/lifetime_unit, naming_schema, schedule) against the live box: midclt call pool.snapshottask.create accepts a single dict — dump an existing task with midclt call pool.snapshottask.query first to confirm the exact shape, exactly as hl-36m6 spiked container.create. Also confirm the NFSv4 perms.BASIC: READ token name via filesystem.getacl on a sample dataset.

Step 4: Syntax-check

cd ansible && ansible-playbook -i inventory/hosts.yml nas-playbook.yml --syntax-check

Expected: OK (after Task 8 wires the play, re-run).

Step 5: Commit

jj commit -m "feat(nas-kopia): host-side snapshot task + read-only ACL role [hl-ney2]"

Task 6: `nas-kopia-backup` role skeleton + defaults

Files:

Create: ansible/roles/nas-kopia-backup/{meta,defaults}/main.yml
Step 1: meta

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
galaxy_info:
  author: fzymgc-house
  description: Interior Kopia→B2 backup (Object Lock) for the nas-support sandbox.
  license: MIT-0
  min_ansible_version: "2.14"
dependencies: []
collections:
  - community.hashi_vault

Step 2: defaults

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
nas_kopia_bin: /home/linuxbrew/.linuxbrew/bin/kopia
nas_kopia_config_dir: /etc/nas-kopia-b2          # dedicated KOPIA_CONFIG_PATH (distinct from /etc/nas-kopia.env R2 stub)
nas_kopia_config_path: "{{ nas_kopia_config_dir }}/repository.config"
nas_kopia_cache_dir: /var/cache/nas-kopia-b2
nas_kopia_log_dir: /var/log/nas-kopia-b2
nas_kopia_bin_dir: /usr/local/bin
nas_kopia_state_dir: /var/lib/nas-kopia-b2       # last-success heartbeat
# Sources = each in-scope leaf at its real path.
nas_kopia_sources: "{{ nas_kopia_leaves | map('regex_replace', '^(.+)$', '/mnt/\\1') | list }}"
nas_kopia_snapshot_prefix: "kopia-"              # literal prefix of host snapshot-task names (grep '^...')

# B2 repository (S3-compatible endpoint + Object Lock). Interior-only — lives here,
# NOT in host_vars/nas.yml, because this role runs on nas_sandbox_hosts (see Task 1 scoping note).
nas_kopia_b2_bucket: fzymgc-nas-kopia
nas_kopia_b2_region: us-west-004                 # confirm via `b2 account get` (s3ApiUrl)
nas_kopia_b2_endpoint: "s3.{{ nas_kopia_b2_region }}.backblazeb2.com"
nas_kopia_b2_vault_path: fzymgc-house/infrastructure/nas/kopia-b2

# Object Lock / retention (spec §3).
nas_kopia_retention_mode: GOVERNANCE
nas_kopia_retention_period: 30d
nas_kopia_keep_daily: 7
nas_kopia_keep_weekly: 4
nas_kopia_keep_monthly: 12
nas_kopia_keep_annual: 3
nas_kopia_compression: zstd
nas_kopia_backup_oncalendar: "*-*-* 03:00:00"

# homes sub-path excludes (redundant backup systems + caches).
nas_kopia_homes_ignores:
  - /AcronisBackups/
  - /restic-backups/
  - /kopia-backups/
  - /Disk Images/
  - /.cache/
  - /.vscode-server/

# Pushover alerting (cluster standard; ntfy decommissioned — hl-ney2.20) + freshness
# dead-man's-switch. Creds read from Vault + deployed to a root-owned 0600 env file
# the systemd units source via EnvironmentFile= (see hl-ney2.19 / PR #1448).
nas_kopia_pushover_api_url: https://api.pushover.net/1/messages.json
nas_kopia_pushover_token_vault_path: fzymgc-house/cluster/pushover/app/cluster-infra
nas_kopia_pushover_user_key_vault_path: fzymgc-house/cluster/pushover
nas_kopia_freshness_max_age_hours: 26

Step 3: Commit

jj commit -m "feat(nas-kopia): nas-kopia-backup role skeleton + defaults [hl-ney2]"

Task 7: Connect/create the B2 repo (S3-compat + Object Lock)

Files:

Create: ansible/roles/nas-kopia-backup/tasks/main.yml (repo section)

Creds read from Vault on the control node, injected over SSH (no_log), exactly as router-kopia-backup does.

Step 1: tasks/main.yml — Vault read + dirs + repo connect/create

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Read B2 creds from Vault (control node)
  ansible.builtin.set_fact:
    nas_kopia_b2: >-
      {{ lookup('community.hashi_vault.vault_kv2_get',
                nas_kopia_b2_vault_path, ca_cert=vault_ca_cert_bundle).data.data }}
  delegate_to: localhost
  become: false
  no_log: true

- name: Create kopia dirs (root-owned, 0700 for config/cache)
  ansible.builtin.file:
    path: "{{ item.path }}"
    state: directory
    owner: root
    group: root
    mode: "{{ item.mode }}"
  become: true
  loop:
    - { path: "{{ nas_kopia_config_dir }}", mode: "0700" }
    - { path: "{{ nas_kopia_cache_dir }}", mode: "0700" }
    - { path: "{{ nas_kopia_log_dir }}", mode: "0750" }
    - { path: "{{ nas_kopia_state_dir }}", mode: "0750" }

- name: Check repository connection
  ansible.builtin.command:
    cmd: "{{ nas_kopia_bin }} repository status"
  environment:
    KOPIA_CONFIG_PATH: "{{ nas_kopia_config_path }}"
  become: true
  register: nas_kopia_status
  changed_when: false
  failed_when: false

- name: Connect to existing B2 repo (S3-compatible endpoint)
  ansible.builtin.command:
    cmd: >
      {{ nas_kopia_bin }} repository connect s3
      --bucket={{ nas_kopia_b2_bucket }}
      --endpoint={{ nas_kopia_b2_endpoint }}
      --access-key={{ nas_kopia_b2.b2_key_id }}
      --secret-access-key={{ nas_kopia_b2.b2_application_key }}
      --password={{ nas_kopia_b2.repo_password }}
      --cache-directory={{ nas_kopia_cache_dir }}
      --enable-actions
  environment: { KOPIA_CONFIG_PATH: "{{ nas_kopia_config_path }}" }
  become: true
  when: nas_kopia_status.rc != 0
  register: nas_kopia_connect
  changed_when: false
  failed_when: false
  no_log: true

- name: Create B2 repo with Object Lock (first run only)
  ansible.builtin.command:
    cmd: >
      {{ nas_kopia_bin }} repository create s3
      --bucket={{ nas_kopia_b2_bucket }}
      --endpoint={{ nas_kopia_b2_endpoint }}
      --access-key={{ nas_kopia_b2.b2_key_id }}
      --secret-access-key={{ nas_kopia_b2.b2_application_key }}
      --password={{ nas_kopia_b2.repo_password }}
      --cache-directory={{ nas_kopia_cache_dir }}
      --retention-mode={{ nas_kopia_retention_mode }}
      --retention-period={{ nas_kopia_retention_period }}
      --enable-actions
  environment: { KOPIA_CONFIG_PATH: "{{ nas_kopia_config_path }}" }
  become: true
  when:
    - nas_kopia_status.rc != 0
    - nas_kopia_connect.rc != 0
  changed_when: true
  no_log: true

--enable-actions is mandatory (grounded, plan-review r1): Kopia disables actions by default — without --enable-actions at repository connect/create, the --before-folder-action (Task 8) is silently skipped and Kopia backs up the live tree instead of the .zfs snapshot, defeating §4 consistency with no error. Both commands above set it.

Build-time confirmation: verify the brew Kopia supports repository create s3 --retention-mode/--retention-period and --enable-actions ({{ nas_kopia_bin }} repository create s3 --help | rg 'retention|enable-actions'). If --retention-mode is absent, pin a newer Kopia in the nas-sandbox-tools Brewfile (spec open-decision 2) before this task.

Step 2: Syntax-check + commit

cd ansible && ansible-playbook -i inventory/hosts.yml nas-playbook.yml --syntax-check
jj commit -m "feat(nas-kopia): B2 repo connect/create with Object Lock [hl-ney2]"

Task 8: `--before-folder-action` newest-snapshot selector

Files:

Create: ansible/roles/nas-kopia-backup/templates/kopia-snapshot-pick.sh.j2
Modify: ansible/roles/nas-kopia-backup/tasks/main.yml (deploy + policy)

The action runs inside the sandbox for each source, picks the newest host-made .zfs snapshot, and redirects Kopia’s read there. Stable source identity = the invoked /mnt/main/<leaf> path (no --override-source).

Step 1: The action script

#!/usr/bin/env bash
# {{ ansible_managed }}
# Kopia --before-folder-action. Kopia sets KOPIA_SOURCE_PATH to the source dir.
# Echo KOPIA_SNAPSHOT_PATH pointing at the newest host-made ZFS snapshot for that leaf.
set -euo pipefail
src="${KOPIA_SOURCE_PATH:?}"
snapdir="${src%/}/.zfs/snapshot"
# Literal prefix match (grep -F-style anchor); names sort lexically by the %Y%m%d-%H%M schema.
newest="$(ls -1 "$snapdir" 2>/dev/null | grep "^{{ nas_kopia_snapshot_prefix }}" | sort | tail -1 || true)"
if [ -z "$newest" ]; then
  echo "no kopia snapshot found under $snapdir" >&2
  exit 1
fi
echo "KOPIA_SNAPSHOT_PATH=$snapdir/$newest"

Step 2: Deploy it + register the folder policy per source

Append to tasks/main.yml:

- name: Deploy snapshot-pick action
  ansible.builtin.template:
    src: kopia-snapshot-pick.sh.j2
    dest: "{{ nas_kopia_bin_dir }}/kopia-snapshot-pick"
    owner: root
    group: root
    mode: "0755"
  become: true

- name: Set per-source before-folder-action
  ansible.builtin.command:
    cmd: >
      {{ nas_kopia_bin }} policy set {{ item }}
      --before-folder-action {{ nas_kopia_bin_dir }}/kopia-snapshot-pick
  environment: { KOPIA_CONFIG_PATH: "{{ nas_kopia_config_path }}" }
  become: true
  loop: "{{ nas_kopia_sources }}"
  loop_control: { label: "{{ item }}" }
  changed_when: true

Failure behavior (grounded, plan-review r1): the action defaults to --action-command-mode=essential, so if the script exits non-zero (no .zfs snapshot found) Kopia aborts that source’s snapshot rather than silently backing up live data — exactly what we want, so no extra flag is needed. There is no --persist-action-script-on-failure flag in Kopia (it does not exist). Actions are enabled on the repo at connect/create (Task 7, --enable-actions).

Step 3: Commit

jj commit -m "feat(nas-kopia): before-folder-action newest-.zfs-snapshot selector [hl-ney2]"

Task 9: Backup script + systemd units

Files:

Create: templates/kopia-b2-backup.sh.j2, kopia-b2-backup.service.j2, kopia-b2-backup.timer.j2
Modify: tasks/main.yml

Clone the shape of ansible/roles/router-kopia-backup/templates/*. Differences: per-source loop over nas_kopia_sources; writes a heartbeat on success; runs as root.

Step 1: Backup script

#!/usr/bin/env bash
# {{ ansible_managed }}
set -euo pipefail
export KOPIA_CONFIG_PATH="{{ nas_kopia_config_path }}"
export KOPIA_CACHE_DIRECTORY="{{ nas_kopia_cache_dir }}"
export KOPIA_LOG_DIR="{{ nas_kopia_log_dir }}"
LOG="{{ nas_kopia_log_dir }}/backup-$(date +%Y%m%d-%H%M%S).log"
log() { echo "[$(date '+%F %T')] $*" | tee -a "$LOG"; }

log "Starting Kopia→B2 backup"
if ! {{ nas_kopia_bin }} repository status >>"$LOG" 2>&1; then
  log "ERROR: repository not connected"; exit 1
fi
rc=0
{% for src in nas_kopia_sources %}
log "Backing up {{ src }}"
if ! {{ nas_kopia_bin }} snapshot create "{{ src }}" >>"$LOG" 2>&1; then
  log "ERROR: snapshot failed for {{ src }}"; rc=1
fi
{% endfor %}
log "Running quick maintenance"
{{ nas_kopia_bin }} maintenance run >>"$LOG" 2>&1 || log "WARNING: maintenance failed"
if [ "$rc" -eq 0 ]; then
  date +%s > "{{ nas_kopia_state_dir }}/last-success"
  log "Backup completed successfully"
fi
find "{{ nas_kopia_log_dir }}" -name 'backup-*.log' -mtime +30 -delete 2>/dev/null || true
exit "$rc"

Step 2: systemd service + timer (clone router-kopia-backup units, adjust paths)

# {{ ansible_managed }}
[Unit]
Description=Kopia backup of NAS irreplaceable data to B2
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart={{ nas_kopia_bin_dir }}/kopia-b2-backup
User=root
Nice=19
IOSchedulingClass=idle
[Install]
WantedBy=multi-user.target

# {{ ansible_managed }}
[Unit]
Description=Nightly Kopia→B2 backup timer
[Timer]
OnCalendar={{ nas_kopia_backup_oncalendar }}
RandomizedDelaySec=900
Persistent=true
[Install]
WantedBy=timers.target

Step 3: Deploy units + script (append to tasks/main.yml)

- name: Deploy backup script
  ansible.builtin.template:
    src: kopia-b2-backup.sh.j2
    dest: "{{ nas_kopia_bin_dir }}/kopia-b2-backup"
    owner: root
    group: root
    mode: "0755"
  become: true

- name: Deploy systemd service + timer
  ansible.builtin.template:
    src: "{{ item }}.j2"
    dest: "/etc/systemd/system/{{ item }}"
    owner: root
    group: root
    mode: "0644"
  become: true
  loop:
    - kopia-b2-backup.service
    - kopia-b2-backup.timer
  notify: Reload systemd nas-kopia

- name: Enable + start backup timer
  ansible.builtin.systemd:
    name: kopia-b2-backup.timer
    enabled: true
    state: started
    daemon_reload: true
  become: true
  when: not ansible_check_mode

Create ansible/roles/nas-kopia-backup/handlers/main.yml with the exact handler name the notify: above references (a literal clone of router-kopia-backup/handlers/main.yml names it Reload systemd, which would NOT match — define it explicitly):

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
- name: Reload systemd nas-kopia
  ansible.builtin.systemd:
    daemon_reload: true
  become: true
  when: not ansible_check_mode

Step 4: Commit

jj commit -m "feat(nas-kopia): backup script + systemd timer (per-leaf, heartbeat) [hl-ney2]"

Task 10: Kopia policies — retention, compression, ignores

Files: Modify tasks/main.yml

Step 1: Global retention + compression + maintenance retention-mode

- name: Set global retention + compression
  ansible.builtin.command:
    cmd: >
      {{ nas_kopia_bin }} policy set --global
      --keep-daily {{ nas_kopia_keep_daily }}
      --keep-weekly {{ nas_kopia_keep_weekly }}
      --keep-monthly {{ nas_kopia_keep_monthly }}
      --keep-annual {{ nas_kopia_keep_annual }}
      --compression {{ nas_kopia_compression }}
  environment: { KOPIA_CONFIG_PATH: "{{ nas_kopia_config_path }}" }
  become: true
  register: nas_kopia_policy
  changed_when: "'setting' in nas_kopia_policy.stdout"

- name: Ensure Object Lock retention on maintenance (idempotent set)
  ansible.builtin.command:
    cmd: >
      {{ nas_kopia_bin }} maintenance set
      --retention-mode {{ nas_kopia_retention_mode }}
      --retention-period {{ nas_kopia_retention_period }}
  environment: { KOPIA_CONFIG_PATH: "{{ nas_kopia_config_path }}" }
  become: true
  changed_when: false

Step 2: homes sub-path ignores (only on the homes source)

- name: Set homes sub-path ignores
  ansible.builtin.command:
    cmd: >
      {{ nas_kopia_bin }} policy set /mnt/main/homes
      {% for ig in nas_kopia_homes_ignores %}--add-ignore "{{ ig }}" {% endfor %}
  environment: { KOPIA_CONFIG_PATH: "{{ nas_kopia_config_path }}" }
  become: true
  changed_when: true

Step 3: Commit

jj commit -m "feat(nas-kopia): retention, compression, Object-Lock + homes ignores [hl-ney2]"

Task 11: Monitoring — Pushover failure alerting + freshness dead-man’s-switch

Files:

Create: templates/kopia-freshness.sh.j2, kopia-freshness.service.j2, kopia-freshness.timer.j2, templates/pushover.env.j2
Modify: tasks/main.yml, templates/kopia-b2-backup.sh.j2, templates/kopia-b2-backup.service.j2, templates/kopia-freshness.service.j2

Two independent signals (spec §7). The freshness switch is self-contained (no dependency on hl-36m6’s optional in-container otel); a follow-up bead can add a ClickStack gauge once the sandbox’s otel egress is confirmed. Alerting is Pushover (the cluster standard — ntfy decommissioned, hl-ney2.20); creds read from Vault + deployed to a root-owned 0600 pushover.env the units source via EnvironmentFile=.

Step 1: Pushover creds + env file

- name: Read Pushover creds from Vault (control node)
  ansible.builtin.set_fact:
    nas_kopia_pushover_token: "{{ (lookup('community.hashi_vault.vault_kv2_get',
                nas_kopia_pushover_token_vault_path, ca_cert=vault_ca_cert_bundle).data.data)[nas_kopia_pushover_token_field] }}"
    nas_kopia_pushover_user_key: "{{ (lookup('community.hashi_vault.vault_kv2_get',
                nas_kopia_pushover_user_key_vault_path, ca_cert=vault_ca_cert_bundle).data.data)[nas_kopia_pushover_user_key_field] }}"
  delegate_to: localhost
  become: false
  no_log: true

- name: Deploy Pushover env file (root-owned 0600; sourced via EnvironmentFile=)
  ansible.builtin.template:
    src: pushover.env.j2
    dest: "{{ nas_kopia_config_dir }}/pushover.env"
    owner: root
    group: root
    mode: "0600"
  become: true
  no_log: true

Kopia’s native webhook notification profile was dropped — it can’t shape a Pushover-compatible payload (token+user in the body) cleanly, and would have baked a secret into kopia config. The backup-script pushover_alert() wrapper (Task 10’s kopia-b2-backup.sh.j2) covers per-source + maintenance failure immediately; the freshness switch covers “never ran.” Both units add EnvironmentFile={{ nas_kopia_config_dir }}/pushover.env.

Step 2: Freshness dead-man’s-switch script (catches “never ran”)

#!/usr/bin/env bash
# {{ ansible_managed }}
set -euo pipefail
hb="{{ nas_kopia_state_dir }}/last-success"
max=$(( {{ nas_kopia_freshness_max_age_hours }} * 3600 ))
now=$(date +%s)
last=$(cat "$hb" 2>/dev/null || echo 0)
age=$(( now - last ))
if [ "$age" -gt "$max" ]; then
  curl -fsS --form-string token="$PUSHOVER_TOKEN" \
    --form-string user="$PUSHOVER_USER_KEY" \
    --form-string title="NAS Kopia backup STALE" \
    --form-string priority=1 \
    --form-string message="No successful NAS→B2 backup in $(( age / 3600 ))h (threshold {{ nas_kopia_freshness_max_age_hours }}h)." \
    "$PUSHOVER_API_URL" || true
fi

Step 3: Freshness service/timer (runs hourly) + deploy all

# {{ ansible_managed }}
[Unit]
Description=Hourly NAS Kopia backup freshness check
[Timer]
OnCalendar=hourly
Persistent=true
[Install]
WantedBy=timers.target

Deploy kopia-freshness.sh + a Type=oneshot kopia-freshness.service (clone the backup service, ExecStart=…/kopia-freshness) + the timer, and enable the timer (mirror Task 9 Step 3). Also add OnFailure= is optional; the freshness switch is the primary “never ran” alarm.

Step 4: Commit

jj commit -m "feat(nas-kopia): Pushover alerting + freshness dead-man's-switch [hl-ney2]"

Task 12: Wire the plays into `nas-playbook.yml`

Files: Modify ansible/nas-playbook.yml

Step 1: Add the host-side snapshot/ACL role to the nas host play, and the interior backup role to the sandbox play

# append to the "NAS sandbox (instance lifecycle)" play (hosts: nas), after nas-lxc-sandbox:
    - role: nas-kopia-snapshot
      tags: [nas-kopia-snapshot]

# append to the "NAS sandbox interior tooling" play (hosts: nas_sandbox_hosts):
    - role: nas-kopia-backup
      tags: [nas-kopia-backup]

Step 2: Syntax-check + check-mode dry run

cd ansible
ansible-playbook -i inventory/hosts.yml nas-playbook.yml --syntax-check
ansible-playbook -i inventory/hosts.yml nas-playbook.yml --tags nas-kopia-snapshot,nas-kopia-backup --check --diff

Expected: syntax OK; check-diff shows the intended changes, no errors. (Vault + live midclt calls may limit --check; review diff for the file/template tasks.)

Step 3: Commit

jj commit -m "feat(nas-kopia): wire snapshot + backup roles into nas-playbook [hl-ney2]"

Task 13: Deploy + add the sandbox mounts (container restart)

Files: none (apply).

Adding FILESYSTEM mounts to an already-RUNNING sandbox does not take effect until a restart (hl-36m6 instance.yml note). The expanded nas_sandbox_mounts requires a one-time sandbox restart.

Step 1: Apply the sandbox lifecycle (adds the new leaf mounts) + restart

cd ansible
ansible-playbook -i inventory/hosts.yml nas-playbook.yml --tags nas-sandbox
# Restart the container so the new FILESYSTEM devices mount:
ssh -o IdentitiesOnly=yes -o IdentityFile=$HOME/.ssh/seanb4t_ed25519.pub nas \
  'id=$(sudo midclt call container.query "[[\"name\",\"=\",\"nas-support\"]]" | python3 -c "import sys,json;print(json.load(sys.stdin)[0][\"id\"])"); sudo midclt call container.stop "$id"; sudo midclt call container.start "$id"'

Step 2: Verify all leaves are mounted + readable in the sandbox

ssh nas-automation@192.168.20.202 \
  'sudo ls -d /mnt/main/{homes,multimedia,cloud-sync-backup,bwcgroup-data,immich,family_share,shared} \
     /mnt/main/paperless-docs/storage /mnt/main/fzymgc-house/incus/storage/precipice'

Expected: each in-scope leaf path lists.

Step 3: Apply snapshot task + ACLs + backup role

ansible-playbook -i inventory/hosts.yml nas-playbook.yml --tags nas-kopia-snapshot,nas-kopia-backup

Expected: snapshot tasks created (verify in TrueNAS UI / pool.snapshottask.query); repo connected; units enabled.

Step 4: Commit any var fixups discovered during apply

jj commit -m "chore(nas-kopia): apply fixups from first deploy [hl-ney2]" || true

Task 14: Seed the repository (initial backup)

Files: none.

Step 1: Trigger the snapshot task once + run the backup manually

# Force a snapshot now (or wait for 02:00), then run the backup:
ssh nas-automation@192.168.20.202 'sudo systemctl start kopia-b2-backup.service; \
  journalctl -u kopia-b2-backup.service -f'

Expected: per-leaf snapshot create runs; first run uploads ~unique content (~days). Resumable — re-running continues.

Step 2: Confirm snapshots landed

ssh nas-automation@192.168.20.202 \
  'sudo env KOPIA_CONFIG_PATH={{ nas_kopia_config_path }} {{ nas_kopia_bin }} snapshot list'

Expected: one source per in-scope leaf, recent snapshot times.

Step 3: Record seed completion

bd note hl-ney2 "Seed complete: <N> sources, <size> uploaded, repo <stats>."

Task 15: VERIFY RESTORE (the gate — retire nothing before this passes)

Files: none.

Step 1: Restore a sample to scratch (include a child-dataset path)

Export KOPIA_CONFIG_PATH for the whole remote shell so the inner snapshot list inherits it (jq is on PATH via hl-36m6’s Brewfile + bash -lc):

ssh nas-automation@192.168.20.202 'sudo env KOPIA_CONFIG_PATH={{ nas_kopia_config_path }} bash -lc "
  sid=\$({{ nas_kopia_bin }} snapshot list /mnt/main/paperless-docs/storage --json | jq -r .[-1].id)
  {{ nas_kopia_bin }} restore \"\$sid\" /tmp/restore-paperless"'

Step 2: Checksum-compare restore vs source

ssh nas-automation@192.168.20.202 \
  'sudo diff -r /mnt/main/paperless-docs/storage/.zfs/snapshot/$(ls /mnt/main/paperless-docs/storage/.zfs/snapshot | tail -1) /tmp/restore-paperless && echo RESTORE-OK'

Expected: RESTORE-OK (no diffs). Also spot-check a multimedia file restores byte-identical.

Step 3: Verify Object Lock is active

b2 file-info b2://fzymgc-nas-kopia/<a kopia blob path>   # confirm fileRetention.mode == governance, retainUntil in ~30d

Expected: blobs carry retention. Record: bd note hl-ney2 "RESTORE VERIFIED + Object Lock active. Cutover authorized."

Task 16: Cutover — retire rclone + Storj (only after Task 15 passes)

Files: none (midclt).

Step 1: Disable + delete the rclone cloud-sync task id=2

ssh -o IdentitiesOnly=yes -o IdentityFile=$HOME/.ssh/seanb4t_ed25519.pub nas \
  'sudo midclt call cloudsync.query "[[\"description\",\"=\",\"b2-fzymgc-nas\"]]"'   # confirm id
# then: sudo midclt call cloudsync.delete <id>

Step 2: Delete the four disabled Storj cloud_backup tasks

ssh ... nas 'sudo midclt call cloud_backup.query' # list the 4 disabled Storj tasks; delete each by id

Step 3: Record + file fast-follow beads

bd note hl-ney2 "Cutover done: rclone task + 4 Storj tasks deleted."
# Fast-follow beads (separate): prune old b2://fzymgc-nas/backups tree; 2nd cold copy (Glacier/R2 replication);
# rmlint media consolidation; graduate Object Lock to COMPLIANCE; ClickStack freshness gauge.
bd close hl-ney2 --reason="Kopia→B2 backup deployed, restore-verified, old paths retired"

Notes for the implementer

midclt schema is BETA / spike-confirmed. For pool.snapshottask.create, filesystem.setacl, and cloudsync/cloud_backup, dump an existing record via the matching .query first and mirror its exact field shape (hl-36m6’s discipline). Use the argv command form for every midclt call.
Everything in the sandbox runs as root (sudo/User=root) because the ACL grant is to container-root (2147000001).
The seed is the long pole (~3 days). Tasks 14–16 are operational, not code; pace them around the upload.
Do not point a B2 lifecycle rule at fzymgc-nas-kopia — Kopia maintenance owns deletion (spec §2).

NAS Kopia → Backblaze B2 backup — Implementation Plan

Post-merge grounding (read before starting)

File structure

Task 0: Spike — prove the load-bearing assumptions (BUILD GATE)

Task 1: Inventory vars — leaves (group_vars/all) + sandbox mounts

Task 2: Create the B2 bucket (Object Lock + versioning) + scoped key

Task 3: Vault secret + offline repo-password copy

Task 4: Vault read policy for the new secret path

Task 5: nas-kopia-snapshot role — host-side snapshot task

Task 6: nas-kopia-backup role skeleton + defaults

Task 7: Connect/create the B2 repo (S3-compat + Object Lock)

Task 8: --before-folder-action newest-snapshot selector

Task 9: Backup script + systemd units

Task 10: Kopia policies — retention, compression, ignores

Task 11: Monitoring — Pushover failure alerting + freshness dead-man’s-switch

Task 12: Wire the plays into nas-playbook.yml

Task 13: Deploy + add the sandbox mounts (container restart)

Task 14: Seed the repository (initial backup)

Task 15: VERIFY RESTORE (the gate — retire nothing before this passes)

Task 16: Cutover — retire rclone + Storj (only after Task 15 passes)

Notes for the implementer

Task 5: `nas-kopia-snapshot` role — host-side snapshot task

Task 6: `nas-kopia-backup` role skeleton + defaults

Task 8: `--before-folder-action` newest-snapshot selector

Task 12: Wire the plays into `nas-playbook.yml`