Self-managed Kopia → Backblaze B2 backup for NAS irreplaceable data

This design replaces the chronically-failing TrueNAS-native rclone cloud-sync (b2-fzymgc-nas, last_run=null, stored unencrypted) with a self-managed Kopia backup of the NAS’s irreplaceable personal data to Backblaze B2 — client-side encrypted, deduplicated, compressed, and protected against ransomware by Object Lock. It retires the four disabled TrueCloud → Storj tasks (Storj exit) and, after a verified restore, the old rclone task and its unwanted B2 file-tree.

Tracked under bead hl-ney2. Builds additively on the hl-36m6 NAS-Ansible foundation (P0 host + P1 nas-support libvirt-LXC sandbox; merged in PR #1404).

Context & motivation

The current offsite story is broken and over-scoped:

• TrueNAS native rclone cloud-sync task id=2 pushed 7.5 TiB of /mnt/main (including 5.5 TiB of Acronis machine images) to b2://fzymgc-nas/backups, unencrypted, with chronic FAILED runs (30–43 transient errors each) and last_run=null. The task scope was tightened in the UI, but the architecture is wrong: a file-tree push with no encryption, no dedup, and no verified restore.
• The four TrueCloud → Storj cloud_backup tasks are already disabled (Storj exit).
• A workstation survey (2026-06-27) reclaimed ~8.7 TiB; the pool is now ~57% used. This backup work protects the irreplaceable subset and defers a capacity upgrade.

We pivot to a self-managed Kopia instance — not the TrueNAS TrueCloud wrapper (which errored adding B2 creds and hides control) — giving client-side encryption, content-addressable dedup, compression, verified restore, and immutability.

Relationship to hl-36m6 (merged — additive)

hl-36m6 merged in #1404. This spec treats its nas-support LXC sandbox as the substrate hl-ney2 extends additively (verified against the shipped roles):

• hl-36m6 merged (#1404) during this design; the spec was rebased onto it and re-grounded against the shipped roles. hl-ney2 adds a dedicated nas-kopia-backup role plus configuration that populates hl-36m6’s existing sandbox mechanisms (nas_sandbox_mounts, nas_sandbox_curation_targets / acls.yml) — additive, not edits to hl-36m6’s role logic.
• The plan re-verifies the live sandbox at build time — idmap base UID, the leaf-only mount constraint, and the brew-installed kopia — rather than trusting spec text.
• hl-36m6’s nas-sandbox-tools may wire a Kopia→R2 repo (fzymgc-house/infrastructure/nas/kopia-r2) for personal/curation use. hl-ney2’s backup is a distinct Kopia repository (B2) and coexists with it via a dedicated KOPIA_CONFIG_PATH (and ideally a dedicated backup identity), exactly as Kopia supports multiple repos per host. hl-ney2 does not retire the R2 repo — that is hl-36m6’s decision. If hl-36m6 instead landed only a binary install or an empty stub, hl-ney2’s role owns the full B2 backup config at implementation time. The choice is resolved by plan-time re-verification of the live sandbox.

hl-ney2 sequences after the base sandbox (P1) exists.

Goals

• G1 — Protect irreplaceable data now. Client-encrypted, deduplicated Kopia snapshots of the irreplaceable NAS datasets land in B2, on a nightly schedule, with a proven restore before anything is retired.
• G2 — Immutable / ransomware-resistant. The B2 repository is Object-Lock protected so a compromised NAS cannot destroy recent backup history.
• G3 — Never fail silently. A freshness signal + failure notification make a dead or stale backup loud — the failure mode the old rclone task hid.
• G4 — Retire the old paths. After verification, retire the rclone task and the disabled Storj tasks, and prune the unwanted B2 file-tree.

Non-goals (deferred to fast-follow beads)

• Second cold copy (3-2-1 third leg): replicate the B2 repo bucket to AWS Glacier Deep Archive (cheapest, ~$1/TiB-mo) or Cloudflare R2 (zero egress). Repo-bucket replication — not kopia-direct-on-Glacier (Glacier objects are not instantly readable).
• Media consolidation (rmlint dedup of near-duplicate photos/video). Runs after the first backup, using it as an undo safety net.
• immich application database backup (albums/faces/metadata). The NAS Kopia covers immich’s photo files; the immich Postgres DB on the cluster is a separate backup track.

Scope — what gets backed up (measured 2026-06-27, zfs … used)

The whole point is to back up the irreplaceable subset, not the 7.5 TiB the rclone task was pushing. Measured in-scope datasets:

Dataset	Size	acltype	Notes
main/homes (in-scope subset)	~0.8 T	posix	personal files (~166 G) + machine-archives (~633 G, in-flight Mac migration). Sub-path excludes below
main/multimedia	763 G	nfsv4	family photos/video — 99.99% irreplaceable, heavily duplicated
main/cloud-sync-backup	537 G	posix	local pull of remote cloud (Google etc.)
main/bwcgroup-data	126 G	nfsv4	business data
main/fzymgc-house	71.5 G	posix	incl. 66.7 G incus/AresMUSH game data (precipice, doorsportal1)
main/paperless-docs (+children)	11.6 G	posix/nfsv4	documents (has child datasets)
main/immich	~0 (grows)	posix	ultimate photo destination; empty today, in-scope so it is covered as it fills
main/family_share	3.6 G	nfsv4
main/shared	3.4 G	nfsv4
Total	≈ 2.3 TiB raw		→ ~1.9–2.1 TiB after dedup

Excluded (redundant or re-creatable, not irreplaceable): inside homes — AcronisBackups (~5.5 T), restic-backups (~303 G), kopia-backups (~70 G), Disk Images (~238 G), caches (.cache, .vscode-server); and pool-level time-machine (7.97 T, a Mac TM backup), storj-node (945 G, retiring), software/shares/software (~122 G ISOs), virtual-machines (145 G), garage-s3, rustfs, .system.

The homes sub-path excludes are enforced with Kopia ignore rules (policy set --add-ignore and/or a committed .kopiaignore).

Child datasets — back up leaves, not parents. ZFS snapshots are per-dataset: a snapshot of a parent does not include its child datasets (their mount-over points appear as empty directories inside the parent’s snapshot). In-scope datasets with children — notably main/paperless-docs (children data, export, ingestion[/sean], storage) and main/fzymgc-house (incus subtree) — are therefore expanded to their leaf datasets: the plan enumerates each leaf dataset, snapshots it, and backs it up as its own Kopia source. The leaf enumeration is resolved at plan time against pool.dataset.query (acltype branches per §5). Restore verification (§Cutover) must include a child-dataset path to prove coverage.

Cost & seed estimate

~1.9–2.1 TiB stored on B2 at $6/TiB-mo ≈ **$12–13/month**. Initial seed of unique content at ~10 MiB/s ≈ ~3 days (duplicates are hashed locally but never uploaded). Incrementals are near-free — the data is ~99% cold.

Why dedup makes the duplication a non-problem: Kopia is content-addressable — identical chunks are stored once regardless of how many times they appear in the tree. The survey’s “bulk copies + numbered duplicate sets” of photos cost storage once. Near-duplicates (re-encoded video, resized photos) won’t fold — that is what the later rmlint consolidation pass cleans up.

Architecture

TrueNAS host (appliance; middleware-managed only)
  pool.snapshottask (midclt, host-autonomous — no sandbox involvement):
    nightly  main/<leaf>@kopia-<ts>   · ~3-day lifetime · UI-visible
  │
  └─ nas-support LXC  (hl-36m6 P1; idmap DEFAULT → host UID 2147000001)
       NIC: macvlan bond0 .202   → workstation SSH + B2 egress (hl-36m6; unchanged)
       mounts: each in-scope LEAF dataset at /mnt/main/<leaf>   (read-only via ACLs)

       systemd timer (nightly):  kopia snapshot create /mnt/main/<leaf>   (stable source identity)
         --before-folder-action (runs IN sandbox, no host access):
             pick newest /mnt/main/<leaf>/.zfs/snapshot/kopia-*  → echo KOPIA_SNAPSHOT_PATH
         Kopia reads from:  /mnt/main/<leaf>/.zfs/snapshot/<newest>/  ──────► B2 (client-encrypted)
         kopia maintenance (quick nightly / full weekly) — extends Object Lock

   ▼  S3-compatible endpoint
Backblaze B2 · bucket fzymgc-nas-kopia  (Object Lock + versioning)
  NO object-deleting lifecycle — Kopia maintenance owns retention

Component design

1. Run model & host

Scheduled Kopia CLI inside the nas-support LXC, cloning the proven ansible/roles/router-kopia-backup shape but reusing the Kopia binary hl-36m6 already installs via Homebrew in nas-sandbox-tools (/home/linuxbrew/.linuxbrew/bin/kopia): idempotent repository connect/create, a backup script looping snapshot create per source + maintenance, driven by a systemd timer. No long-running daemon, no web UI, no ingress — outbound-only, matching the bead’s posture. The same sandbox later hosts rmlint/ncdu consolidation.

Why not a Kopia server or a dedicated Docker app: a server adds a long-running TLS/auth surface for visibility we get more cheaply from telemetry; a Docker app sidesteps the idmap ACLs but splits the NAS tooling across two runtimes. The LXC + scheduled-CLI keeps one host for backup + curation and reuses a battle-tested role. (Decision locked during brainstorming.)

2. Repository & backend (Backblaze B2 via S3-compatible endpoint)

• Endpoint: Kopia’s S3-compatible repository against the B2 S3 endpoint (s3.<region>.backblazeb2.com), not the native b2 provider. Object Lock on B2 is only supported through Kopia’s S3 option (the native b2 create command is deprecated). This also matches the router-kopia-backup role’s repository create s3 shape.
• Bucket: dedicated fzymgc-nas-kopia (separate from the old rclone file-tree), created with Object Lock enabled + versioning. Object Lock is simplest to establish at bucket creation; enabling it up front avoids any window of unprotected history — hence v1 rather than a fast-follow.
• Encryption: Kopia default client-side encryption — B2 only ever holds ciphertext.
• Lifecycle: no object-deleting lifecycle. Kopia’s maintenance GC owns deletion; a B2 lifecycle that hides/deletes objects would corrupt the repo.
• Compression: zstd global policy (wins on docs/cloud-sync-backup; harmless on already-compressed media).

3. Immutability / ransomware protection (Object Lock)

• Repository created with object-lock support; kopia maintenance set --retention-mode <MODE> --retention-period <period>. Each maintenance run re-extends the lock on referenced blobs, so they stay immutable on a rolling horizon.
• Mode (v1 default — GOVERNANCE + offline bypass key): governance mode is bypassable only with a special permission, which is held in an offline B2 key never placed on the host. This defeats a compromised-host smash-and-grab while leaving an offline escape hatch — safer for a first deployment than compliance (which nobody can undo before expiry). Graduate to COMPLIANCE once confident.
• Retention period (v1 default — 30 days): must exceed the full-maintenance interval so locks never lapse between extensions.
• Threat model: repo_password protects reading; Object Lock protects deletion. A compromised NAS can encrypt/delete the source and even issue delete calls to B2, but cannot remove blobs locked within the window — restore to a pre-attack snapshot is always possible inside it.
• Cost note: locked blobs cannot be GC’d until their lock expires, so the repo carries up to ~one retention-window of otherwise-deletable data. Negligible for cold data; a bounded, temporary bump during the later consolidation (heavy-delete) phase.

4. Consistency — TrueNAS snapshot task + per-leaf .zfs read

v1 UPDATE (2026-06-28) — this section is superseded by ADR hl-zrbh. The build spike (hl-ney2.1) and salvage spike (hl-ney2.6) proved this section’s load-bearing assumption false and unrecoverable: the unprivileged libvirt/LXC sandbox bind-mounts are private and kernel-locked (MNT_LOCKED), so a host ZFS snapshot can never become visible in the sandbox without a container restart — not via .zfs automount, not via host-submount propagation, not via runtime make-rslave (fails, err 5005), and the TrueNAS FILESYSTEM device schema exposes no mount-propagation option. v1 therefore backs up LIVE datasets — no ZFS snapshots. The host-side role nas-kopia-acls grants read-only ACLs (POSIX1E rX / NFSv4 READ) for uid 2147000001, and Kopia reads /mnt/main/<leaf> directly. Point-in-time / application consistency for the actively changing datasets (fzymgc-house/incus AresMUSH, paperless, future immich) is a deferred fast-follow; the snapshot design below is retained for that fast-follow’s reference. See ADR hl-zrbh (supersedes hl-9uzu).

Post-merge grounding update (2026-06-27). hl-36m6 merged (#1404) during this design. Re-grounding against its shipped nas-lxc-sandbox role revealed a hard constraint that reshapes this section: only LEAF datasets bind-mount into the sandbox — the pool root /mnt/main and any parent-with-children fail to bind-mount (“Failed to bind mount … Invalid argument”). The earlier mechanism (a host-mounted snapshot under a /mnt/main/.kopia-snap-mounts/ shared dir, reached over a second truenasbr0 NIC + ssh) is therefore infeasible — there is no pool-root mount to host the shared dir. The simpler, now-primary mechanism below removes the second NIC, the host↔sandbox SSH channel, and any sandbox→host privilege entirely.

Each in-scope leaf dataset is individually bind-mounted into the sandbox at its real path (source=/mnt/main/<leaf>, target=/mnt/main/<leaf> — hl-36m6’s nas_sandbox_mounts mechanism, populated by this design). Consistency is achieved with two decoupled pieces that never communicate directly:

• Host (autonomous): a TrueNAS periodic snapshot task (pool.snapshottask, created via midclt by this design’s host play) snapshots the in-scope leaves nightly into main/<leaf>@kopia-<timestamp>, with a short lifetime (e.g. 3 days) and middleware-managed cleanup. UI-visible, upgrade-safe, and needs no sandbox involvement or privilege.
• Sandbox (read-only): Kopia is invoked with the stable source path /mnt/main/<leaf>, and a --before-folder-action script (running inside the sandbox, no host access) lists /mnt/main/<leaf>/.zfs/snapshot/, selects the newest kopia-* snapshot, and echoes KOPIA_SNAPSHOT_PATH=/mnt/main/<leaf>/.zfs/snapshot/<newest>. Kopia reads the backup from the snapshot while recording the source identity as the stable /mnt/main/<leaf> — so incrementals stay clean without --override-source.

This is strictly simpler and safer than the previous host↔sandbox bridge: no second NIC, no SSH channel, no zfs/mount privilege in the sandbox. The sandbox reads its mounted data and nothing else; the host snapshots autonomously via its own blessed mechanism. The macvlan .202 NIC (workstation SSH + B2 egress) is hl-36m6’s and is unchanged.

Load-bearing prerequisite — MUST be spiked before the role is built

The one unproven assumption: that <leaf>/.zfs/snapshot/<snap>/ is visible and readable inside the sandbox for a bind-mounted leaf. ZFS auto-mounts a snapshot on access; whether that auto-mount propagates through the libvirt-LXC bind-mount was not spiked by hl-36m6. The plan MUST gate role-build on a spike that proves it (see Cutover step 0). Setting snapdir=visible on in-scope leaves makes the snapshot dir listable (it is reachable by explicit path even when hidden, but visible removes doubt). Fallback if .zfs does not propagate: mount one persistent per-leaf snapshot as a dedicated FILESYSTEM device at container-(re)configure time (present from container start, no post-start mount), refreshed on a cadence by the host — trading per-run freshness for zero propagation dependency.

No sandbox→host privilege (a security improvement)

Because the host snapshots autonomously via pool.snapshottask and the sandbox only reads .zfs, the sandbox holds no credential that can act on the host — a strict improvement over any ssh-to-host-zfs scheme. The snapshot task is created and reconciled by this design’s host-side Ansible play (midclt), exactly as hl-36m6 manages the container itself. The ransomware backstop remains the offsite, Object-Lock-protected B2 repo, not local ZFS snapshots.

5. Permissions (idmap DEFAULT + read-only ACLs)

Under idmap: {type: DEFAULT} the sandbox reads as host UID 2147000001 (proven in hl-36m6; the plan re-confirms the base after sandbox (re)creation). An Ansible task grants read-only ACLs for that UID on each in-scope dataset, branching by acltype (both proven in hl-36m6):

• POSIX1E → setfacl -R -m u:2147000001:rX <path> + setfacl -d -m u:2147000001:rX <path> (default ACL so new files inherit).
• NFSv4 → midclt call filesystem.setacl adding a USER ACE (id 2147000001, read perms, inherit, recursive). setfacl does not work on NFSv4.

Applied once, idempotent. The grant propagates to the dataset’s snapshots (snapshot ACLs mirror the live dataset at snap time). Read is broader than hl-36m6’s curation grant (backup reads the whole in-scope set), but it is read-only and bounded to in-scope datasets.

6. Secrets / Vault

Path / property	Contents
fzymgc-house/infrastructure/nas/kopia-b2	b2_key_id, b2_application_key (S3 access/secret), repo_password
(offline, not in Vault)	Object Lock governance bypass key — disaster/escape only

• The B2 application key is scoped to the fzymgc-nas-kopia bucket only and carries the object-lock retention permission (so maintenance can extend locks).
• repo_password = openssl rand -base64 32, stored in Vault and kept as an offline copy — it is the one irrecoverable secret (lose it and B2 is unreadable ciphertext; Vault being gone must not mean data loss).
• Creds are read from Vault on the control node and injected over SSH (no_log), exactly as the router role does. Kopia’s repository.config in the sandbox holds the connection; the config dir is 0700, owned by the backup user.
• Vault policy updated for the new path (per tf/CLAUDE.md).

7. Monitoring — never fail silently (G3)

Two independent signals, because the old rclone failure (last_run=null) was the job never running — a failure notification cannot fire for a job that never executes:

• Freshness dead-man’s-switch → Pushover (v1): a timer compares a last-success heartbeat against a max-age threshold (e.g. > 26 h) and alerts Pushover if stale. Self-contained — no dependency on the sandbox’s (optional) otel egress. This catches “didn’t run at all” — the exact gap rclone fell into. Fast-follow: ship the same snapshot-age as a ClickStack gauge once the sandbox otel path is confirmed.
• Failure notification → Pushover: the backup script’s pushover_alert() wrapper posts to the Pushover API on repository-not-connected, per-source snapshot failure, and maintenance failure — immediate per-failure notices for the “ran but failed” case. (Kopia’s native webhook notification profile was dropped: it can’t shape a Pushover-compatible payload cleanly, and the cluster’s ntfy is decommissioned — hl-ney2.20.) Pushover creds come from Vault (secret/fzymgc-house/cluster/pushover, infra-domain app token) via a root-owned 0600 env file the systemd units source.

8. Role structure

A dedicated nas-kopia-backup role owns the backup system; it is additive to hl-36m6 and reconciled against the live sandbox at build time (see “Relationship to hl-36m6”). Split by where the work runs:

• Host-side (via midclt on the TrueNAS host): create the TrueNAS periodic snapshot task for the in-scope leaves; ensure the read-only ACL grants for UID 2147000001 (extending hl-36m6’s acls.yml mechanism); populate the in-scope leaf bind-mounts (nas_sandbox_mounts). No NIC change; no host privilege handed to the sandbox.
• Sandbox interior (over SSH, play targets nas_sandbox_hosts): reuse the brew-installed Kopia; connect/create the S3-compat B2 repo with object lock under a dedicated KOPIA_CONFIG_PATH (distinct from hl-36m6’s R2 stub); deploy the backup script + the --before-folder-action newest-snapshot selector + systemd service/timer; set retention, compression, and ignore policies; configure the notification profile + freshness metric.

ansible/
  roles/
    nas-kopia-backup/        # host-side: snapshot task, ACL read grants, leaf mounts (midclt)
                             # interior: repo connect, backup script + .zfs action, timer, policies, monitoring

9. Retention & maintenance

• Snapshot retention (Kopia policy): --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-annual 3 (cold data + dedup make deep history cheap; tunable).
• Maintenance: quick nightly, full weekly (a full GC every night is heavy on a TB-class repo). Full maintenance is what extends Object Lock retention — its interval must stay under the retention period (§3).

Cutover & verification (no retirement until restore is proven)

0. Spike (build gate) — on the live sandbox, prove the load-bearing assumptions before writing the role: (a) the newest main/<leaf>@kopia-* snapshot is readable inside the sandbox via /mnt/main/<leaf>/.zfs/snapshot/ (auto-mount through the bind-mount — §4); (b) the idmap base UID is 2147000001 (/proc/<pid>/uid_map); (c) the read-ACL grant works on a representative POSIX1E and NFSv4 dataset (§5); (d) enumerate in-scope leaf datasets (pool.dataset.query). If (a) fails, switch to the §4 persistent-device fallback before proceeding. No role-build until this passes.
1. Build — create the B2 bucket (Object Lock + versioning), the scoped B2 app key, the Vault secret; add NIC2 + scoped snapshot identity + ACL grants; create the Kopia repo.
2. Seed — initial backup of all in-scope sources (~3 days). Resumable.
3. VERIFY RESTORE (the gate) — restore a sample (a known photo set + paperless docs) to a scratch location and checksum-compare against source. Retire nothing until this passes.
4. Retire — disable + delete the rclone cloud-sync task id=2; delete the four disabled Storj cloud_backup tasks; later prune the old b2://fzymgc-nas/backups file-tree (~3.24 TiB unwanted + remainder) once Kopia is the system of record.
5. Fast-follow beads — (a) graduate Object Lock to COMPLIANCE if desired; (b) second cold copy via repo-bucket replication (Glacier Deep Archive / R2); (c) rmlint media consolidation with Kopia as the undo button.

Risks & mitigations

Risk	Mitigation
hl-36m6 substrate — sandbox built by a separate (merged) role	hl-ney2 is additive (populates mounts/ACLs, adds a role); plan re-verifies the live sandbox (idmap base UID, leaf mounts, brew kopia) at build time
.zfs snapshot not readable inside sandbox (auto-mount-through-bind-mount unspiked — the §4 linchpin)	Mandatory build-gate spike (Cutover step 0); snapdir=visible; fallback = persistent per-leaf snapshot mounted as a FILESYSTEM device
Child datasets silently omitted (per-dataset snapshots)	In-scope set expanded to leaf datasets; restore verification includes a child-dataset path
repo_password loss = total loss	Vault + offline copy
Object Lock cost / irreversibility	Governance + offline bypass + 30 d for v1 (not compliance); bounded extra storage during heavy-delete consolidation
B2 lifecycle vs Kopia GC	Bucket has no object-deleting lifecycle; Kopia maintenance owns retention
Silent backup death (the rclone failure)	Snapshot-age freshness alert and kopia→Pushover failure notification
Broad read-ACL on live pool	Read-only, idempotent, branch posix/nfsv4, bounded to in-scope datasets
BETA container.* API churn	Backup correctness does not depend on it once the sandbox exists; re-validate against system.version

Open decisions (defaults stated; reviewer/user may override)

1. Object Lock mode — default GOVERNANCE + offline bypass key, 30-day retention. Alternative: COMPLIANCE (stricter, irreversible).
2. Kopia version — reuse the Homebrew-installed Kopia from hl-36m6’s nas-sandbox-tools (currently unpinned/latest). Confirm at build time that it supports S3 Object Lock (maintenance set --retention-mode); pin the Brewfile version if reproducibility is wanted.

Verification checklist

• Spike passed (Cutover step 0): .zfs snapshot readable in sandbox; idmap UID confirmed; ACL grant works on POSIX1E + NFSv4; leaf datasets enumerated.
• ansible-playbook … --syntax-check and --check --diff first.
• Repo reachable: kopia repository status from the sandbox.
• A nightly run produces a new snapshot; kopia snapshot list shows expected sources (one per in-scope leaf dataset).
• Object Lock active: blobs carry retention; a delete attempt within the window is refused.
• Freshness dead-man’s-switch fires on a stale heartbeat; failure path notifies Pushover (ClickStack gauge is a fast-follow).
• Restore verified (step 3) — including a child-dataset path (e.g. under paperless-docs) — before any retirement.