NAS Sandbox — Scheduled Homebrew Update/Upgrade

This design adds a scheduled, unattended refresh of the Homebrew bundle inside the nas-support libvirt-LXC sandbox. The sandbox’s curation tooling is installed through a Brewfile (ansible/roles/nas-sandbox-tools) but nothing keeps it current once bootstrapped — formulae drift until an operator runs a manual upgrade. This spec closes that gap with a systemd timer + oneshot service, managed by Ansible over the existing nas-automation SSH connection, and a direct Pushover notification on failure.

Context & background

The sandbox is a ubuntu:noble:amd64:default LXC container provisioned by ansible/roles/nas-lxc-sandbox (created via TrueNAS midclt call container.create, libvirt-backed on TrueNAS 26). Two facts drive this design:

1. The container runs systemd as PID 1. The sandbox bootstrap role enters the running container with nsenter -t <init-pid> -a and successfully runs systemctl enable --now ssh in-container (nas-lxc-sandbox/tasks/bootstrap.yml). systemctl operating against the container’s init PID is decisive evidence that systemd is the init — so a timer-in-container is legitimate here (it would not be for a distroless or init-less image). The image is ubuntu:noble:amd64:default (pinned in ansible/inventory/host_vars/nas.yml; :default shorthand is used in prose below for brevity).
2. nas-sandbox-tools connects over SSH as nas-automation (not via nsenter), so Ansible’s ansible.builtin.systemd module manages in-container units natively. Homebrew is owned by nas-automation (the installer self-elevates via sudo but the prefix lives under that user’s home) and refuses to run as root — so the upgrade service must run as nas-automation.

Brew is on PATH only in login shells: nas-sandbox-tools installs /etc/profile.d/homebrew.sh which evals brew shellenv. The upgrade service’s ExecStart therefore uses bash -lc so the login profile loads brew onto PATH. The brew binary lives at /home/linuxbrew/.linuxbrew/bin/brew; the Brewfile at /home/nas-automation/.Brewfile.

Why not the alternatives

• Host-side cron on the TrueNAS appliance — TrueNAS’s /etc is middleware-database-driven and reverts file edits; cron is not a supported surface. The sandbox container is the correct, supported place to schedule.
• A metric-based Alertmanager alert for failures — nas-otel-collector’s hostmetrics receiver ships only cpu/load/memory/disk/filesystem/network scrapers (no systemd scraper), and journald log shipping is deferred (the upstream otel-contrib image is distroless and lacks journalctl). A metric/log-based alert is not viable today and would couple this work to an unbuilt telemetry follow-up. Direct Pushover notification via a systemd OnFailure= unit is decoupled and grounded in an existing in-repo pattern.

Goals & non-goals

Goals

• Unattended weekly refresh of all Brewfile formulae inside the sandbox.
• Install newly-added Brewfile formulae on the same schedule (so a Brewfile edit lands without a manual play).
• Surface a failed refresh to the operator immediately via Pushover.
• Idempotent enablement via the existing nas-sandbox-tools SSH play, gated behind a flag so it can be disabled per-host.

Non-goals

• The journald-to-ClickStack telemetry follow-up (separate bead; the nas-otel-collector image lacks journalctl).
• Pinning specific formula versions (Homebrew keeps latest by design).
• Upgrading formulae installed outside the Brewfile (none exist in this sandbox — all tools come from the Brewfile).
• Cross-host scheduling coordination (single sandbox host only).

Design

Components

All artifacts are added to ansible/roles/nas-sandbox-tools/:

Artifact	Purpose
templates/brew-update.service.j2	oneshot unit running the upgrade as nas-automation
templates/brew-update.timer.j2	weekly calendar trigger
templates/brew-update-notify.service.j2	oneshot unit curling Pushover on failure
defaults/main.yml (additions)	flag, schedule, jitter, vault paths
tasks/main.yml (additions)	render units + env file, enable the timer

The upgrade command

brew update && brew bundle upgrade --file ~/.Brewfile && brew autoremove && brew cleanup

Grounded in current Homebrew docs (Context7 /homebrew/brew): brew bundle attempts to upgrade all formulae by default, and brew bundle upgrade is shorthand for brew bundle install --upgrade — it installs newly-listed formulae and upgrades existing ones in one idempotent pass. This subsumes the bead’s original brew upgrade && brew bundle install pair (which overlap, since bundle upgrades by default). brew update refreshes formula metadata first; brew autoremove removes no-longer-needed dependencies; brew cleanup clears old downloads. Every sandbox tool is Brewfile-managed, so this covers the full upgrade scope.

brew-update.service.j2

# {{ ansible_managed }}
[Unit]
Description=nas-support Homebrew bundle refresh
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=nas-automation
ExecStart=/bin/bash -lc 'set -euo pipefail; brew update && brew bundle upgrade --file {{ nas_brewfile_path }} && brew autoremove && brew cleanup'
OnFailure=brew-update-notify.service

[Install]
WantedBy=multi-user.target

Notes:

• User=nas-automation — Homebrew refuses root; the brew prefix is owned by this user. No ProtectSystem=strict (the router-kopia-backup service uses it, but brew writes across /home/linuxbrew/.linuxbrew and ~/.cache/Homebrew; a strict read-only system filter would break it). Hardening is intentionally minimal to match brew’s filesystem model.
• bash -lc — the -l login shell sources /etc/profile.d/homebrew.sh, putting brew on PATH. set -euo pipefail makes any sub-step failure exit nonzero, which triggers OnFailure=. The Brewfile path uses the existing nas_brewfile_path var (/home/nas-automation/.Brewfile) rather than shell ~ expansion, matching how the rest of the role references the file.
• The [Install] section on this oneshot is intentionally inert: the play enables only the timer (below), never the service. It is kept for parity with the router-kopia-backup.service.j2 convention (which carries the same unused [Install]). A plan author should not wire the service to be enabled — that would run the upgrade on every boot.
• OnFailure=brew-update-notify.service — fires the Pushover notify unit only on failure, not on success.

brew-update.timer.j2

Mirrors the router-kopia-backup timer convention:

# {{ ansible_managed }}
[Unit]
Description=Weekly Homebrew bundle refresh timer

[Timer]
OnCalendar={{ nas_brew_autoupdate_schedule }}
RandomizedDelaySec={{ nas_brew_autoupdate_jitter }}
Persistent=true

[Install]
WantedBy=timers.target

Persistent=true ensures a missed run (sandbox was down at 04:00 Sunday) is replayed on next boot rather than skipped.

brew-update-notify.service.j2

# {{ ansible_managed }}
[Unit]
Description=Pushover notification for failed brew-update

[Service]
Type=oneshot
User=root
EnvironmentFile={{ nas_brew_pushover_env_path }}
ExecStart=/usr/bin/curl -fsS -X POST https://api.pushover.net/1/messages.json \
  --data-urlencode "token=${PUSHOVER_TOKEN}" \
  --data-urlencode "user=${PUSHOVER_USER_KEY}" \
  --data-urlencode "message=nas-support: brew-update.service failed on %H — see: journalctl -u brew-update.service"

• User=root only because EnvironmentFile is 0600 root-owned (holds the Pushover token + user key). The curl itself is a stateless HTTPS POST.
• %H is systemd’s hostname specifier, expanded by the manager before exec — not a shell substitution. systemd does not run ExecStart through a shell unless the line starts with a shell binary, so a literal $(hostname) would be passed unexpanded into the curl body. %H avoids spawning a shell.
• curl is already present in the container — it is in nas_brew_apt_prereqs (build-essential procps curl file git). No new dependency.
• Pushover is the cluster’s active notifier (ntfy was decommissioned). Creds live in Vault at secret/fzymgc-house/cluster/pushover (user_key) and secret/fzymgc-house/cluster/pushover/app/cluster-infra (token). The cluster-infra app token is used because the sandbox is infrastructure (curation/backup tooling); per-app tokens let the sandbox’s credential be revoked in isolation.

/etc/nas-pushover.env (rendered at play time)

Reuses the exact pattern already proven by /etc/nas-kopia.env in this role: play-time Vault lookup via community.hashi_vault.vault_kv2_get, render to a root-owned 0600 file, become: true, no_log: true:

PUSHOVER_USER_KEY=<from vault>
PUSHOVER_TOKEN=<from vault>

The Vault paths are already documented in docs/reference/secrets.md, so no secrets reference update is required.

defaults/main.yml additions

# Scheduled Homebrew bundle refresh (brew-update.timer). Gated on brew being
# installed. Disable per-host via host vars to opt out of auto-upgrade.
nas_brew_autoupdate: true
nas_brew_autoupdate_schedule: "Sun 04:00"
nas_brew_autoupdate_jitter: "30m"

# Pushover failure notification. Creds rendered from Vault to a 0600 env file.
nas_brew_pushover_env_path: /etc/nas-pushover.env
nas_pushover_user_vault_path: fzymgc-house/cluster/pushover
nas_pushover_token_vault_path: fzymgc-house/cluster/pushover/app/cluster-infra

tasks/main.yml additions

Appended as a single block: gated on both brew install and the autoupdate flag (matching the per-task when: nas_brew_install | bool pattern already used throughout tasks/main.yml — here one guard wraps the whole addition):

- name: Scheduled Homebrew bundle refresh
  when: nas_brew_install | bool and nas_brew_autoupdate | bool
  block:
    - name: Read Pushover user key from Vault
      ansible.builtin.set_fact:
        nas_pushover_user_key: >-
          {{ lookup('community.hashi_vault.vault_kv2_get',
                    nas_pushover_user_vault_path,
                    ca_cert=vault_ca_cert_bundle).data.data.user_key }}
      no_log: true

    - name: Render Pushover env file
      ansible.builtin.copy:
        dest: "{{ nas_brew_pushover_env_path }}"
        content: |
          PUSHOVER_USER_KEY={{ nas_pushover_user_key }}
          PUSHOVER_TOKEN={{ lookup('community.hashi_vault.vault_kv2_get',
                    nas_pushover_token_vault_path,
                    ca_cert=vault_ca_cert_bundle).data.data.token }}
        mode: "0600"
      become: true
      no_log: true

    - name: Render brew-update units
      ansible.builtin.template:
        src: "{{ item }}.j2"
        dest: "/etc/systemd/system/{{ item }}"
        mode: "0644"
      loop:
        - brew-update.service
        - brew-update.timer
        - brew-update-notify.service
      become: true
      notify: Reload systemd and enable brew-update timer

The Pushover token lookup is split into its own vault_kv2_get call because it lives at a different Vault path (the app sub-path) than the user key. Both use the existing vault_ca_cert_bundle trust anchor already in use by the Kopia creds lookup.

The handler goes in the role’s first handlers/main.yml (the role has no handlers/ directory today — meta/main.yml lists dependencies: []). It does a daemon_reload + enabled: true + state: started on the timer only:

- name: Reload systemd and enable brew-update timer
  ansible.builtin.systemd:
    name: brew-update.timer
    daemon_reload: true
    enabled: true
    state: started
  become: true

The service is never state: started by the play — starting the oneshot would trigger an immediate upgrade on every apply (a side effect). The service runs only via the timer (weekly) or a manual systemctl start for verification.

Idempotence

• template/copy are idempotent on content.
• ansible.builtin.systemd is idempotent on enabled/state.
• The notify handler fires only when a template changes.
• daemon_reload: true absorbs unit-file changes safely on re-apply.

Telemetry (stated honestly)

Unit stdout/stderr lands in the systemd journal by default. The sandbox’s nas-otel-collector ships host metrics today but not journald logs (the distroless otel-contrib image lacks journalctl; a custom image is a follow-up). Until that follow-up lands, the journal is inspectable only via journalctl -u brew-update.service over SSH — it will not appear in ClickStack. The Pushover failure notification is the active observability surface; the journal is the forensic record. This design does not block on the journald follow-up.

Verification

Documented in the NAS operations runbook (bead hl-36m6.11, closed):

1. systemctl list-timers brew-update* — timer listed, NEXT ≈ next Sunday 04:00 ± jitter.
2. systemctl start brew-update.service — trigger a one-off run.
3. systemctl status brew-update.service — Active: inactive (dead) with ExecStart= exit 0 (success path).
4. Failure-path check (optional): temporarily set ExecStart to a command that exits nonzero, systemctl start brew-update.service, confirm brew-update-notify.service runs and a Pushover notification arrives, then revert.

Steps 1–3 are run over the nas-automation SSH connection. They are not wired as play-time tasks (a play-time systemctl start is a side effect that would run an upgrade on every apply).

Risks & mitigations

Risk	Mitigation
A formula upgrade breaks a curation tool mid-week	Failure → Pushover alert immediately; Persistent=true retries next Sunday. No auto-rollback (Homebrew has none); operator rolls back manually via brew pin/reinstall if needed.
brew bundle upgrade upgrades an unpinned dependency the operator relies on	Acceptable: the sandbox is ephemeral curation tooling, not a workload runtime. brew pin is the escape hatch if a specific version matters.
Pushover token leak via the env file	0600 root-owned, no_log: true on the lookup/render tasks, per-app (cluster-infra) token revocable in isolation.
Sandbox down at scheduled time	Persistent=true replays on next boot.
brew update fails on network blip	set -euo pipefail → nonzero → Pushover; transient, self-heals next week.

Open questions

None — all decisions settled during brainstorming. The brew command (brew bundle upgrade), the direct-Pushover failure path (not Alertmanager), and the schedule (weekly Sun 04:00, cluster-infra token) are confirmed.

Follow-ups / out of scope

• journald log shipping to ClickStack via a custom otel-contrib image (separate bead; would make the journal visible in telemetry).
• Extending the same timer pattern to other in-container scheduled work if more emerges.