NAS Sandbox Homebrew Autoupdate — Implementation Plan

NAS Sandbox Homebrew Autoupdate Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add a scheduled, unattended weekly Homebrew bundle refresh to the nas-support LXC sandbox, with a direct Pushover notification on failure, all managed by the existing nas-sandbox-tools Ansible role over SSH.

Architecture: A systemd timer + oneshot service inside the ubuntu:noble:amd64:default LXC container (systemd is PID 1, confirmed in nas-lxc-sandbox/tasks/bootstrap.yml). The service runs brew bundle upgrade as the nas-automation user via a login shell (so /etc/profile.d/homebrew.sh loads brew onto PATH). An OnFailure= notify unit curls the Pushover API directly, with creds rendered from Vault to a 0600 env file at play time (reusing the existing /etc/nas-kopia.env pattern). All artifacts are added to ansible/roles/nas-sandbox-tools/, gated behind nas_brew_autoupdate (default true).

Tech Stack: Ansible (FQCN, production lint profile), systemd unit files (Jinja2 templates), Homebrew (brew bundle upgrade), Vault (community.hashi_vault.vault_kv2_get), Pushover HTTP API.

Spec: docs/engineering/specs/2026-06-27-nas-brew-autoupdate-design.md Bead: hl-36m6.12 (phase:design, design-review READY)

TDD note (documented exception): This repo’s Ansible roles have no unit-test framework (no molecule). Per the writing-plans skill’s TDD exception for config/infra, each task uses a four-rung test ladder instead of a unit test: --syntax-check (structural) → ansible-lint + yamllint (style/safety) → --check --diff (dry-run idempotence) → live systemctl verification (behavioral acceptance, Task 7). The behavioral rung is the real “test passes” gate.

Working directory: All commands run from the worktree root /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster/.worktrees/nas-brew-autoupdate unless noted. The role path below is given relative to that root.

Commit cadence: One commit per task, conventional-commit format (feat(nas): … [hl-36m6.12]), with the Co-Authored-By: Claude <noreply@anthropic.com> byline. VCS is jj (jj-first policy); use jj describe -m then jj commit -m per references/vcs-preamble.md. Do not push — push/PR happens at landing time.

File Structure

File	Action	Responsibility
`ansible/roles/nas-sandbox-tools/defaults/main.yml`	Modify (append)	`nas_brew_autoupdate` flag, schedule, jitter, Pushover vault paths
`ansible/roles/nas-sandbox-tools/templates/brew-update.service.j2`	Create	oneshot unit running `brew bundle upgrade` as `nas-automation`
`ansible/roles/nas-sandbox-tools/templates/brew-update.timer.j2`	Create	weekly calendar trigger (mirrors `router-kopia-backup` timer)
`ansible/roles/nas-sandbox-tools/templates/brew-update-notify.service.j2`	Create	oneshot unit curling Pushover on failure
`ansible/roles/nas-sandbox-tools/handlers/main.yml`	Create (role’s first handler file)	`daemon_reload` + enable the timer
`ansible/roles/nas-sandbox-tools/tasks/main.yml`	Modify (append block)	Vault lookup → render env file → render units → notify handler
`docs/operations/nas.md`	Modify (append section)	Verification runbook for the brew-update timer

The role today has defaults/, meta/, tasks/, templates/ — no handlers/. This plan creates the role’s first handlers/main.yml.

Task 1: Add autoupdate defaults

Files:

Modify: ansible/roles/nas-sandbox-tools/defaults/main.yml (append after the existing Kopia block, line ~28)
Step 1: Append the defaults block

Open ansible/roles/nas-sandbox-tools/defaults/main.yml and append after the final Kopia line (nas_kopia_s3_prefix: nas-backups):

# Scheduled Homebrew bundle refresh (brew-update.timer). Gated on brew being
# installed. Disable per-host via host vars to opt out of auto-upgrade.
nas_brew_autoupdate: true
nas_brew_autoupdate_schedule: "Sun 04:00"
nas_brew_autoupdate_jitter: "30m"

# Pushover failure notification. Creds rendered from Vault to a 0600 env file
# (same pattern as /etc/nas-kopia.env above). cluster-infra app token — sandbox
# is infrastructure (curation/backup tooling); per-app token revocable in
# isolation.
nas_brew_pushover_env_path: /etc/nas-pushover.env
nas_pushover_user_vault_path: fzymgc-house/cluster/pushover
nas_pushover_token_vault_path: fzymgc-house/cluster/pushover/app/cluster-infra

Step 2: Verify YAML is well-formed and lints

Run:

cd ansible && ansible-lint --config-file=.ansible-lint roles/nas-sandbox-tools/defaults/main.yml
yamllint -c ../.yamllint.yaml roles/nas-sandbox-tools/defaults/main.yml

Expected: ansible-lint reports no new issues; yamllint exits 0. (The file already has the SPDX/code header; do not remove it.)

Step 3: Commit

jj describe -m "feat(nas): brew autoupdate defaults [hl-36m6.12]

Add nas_brew_autoupdate flag (default true), weekly Sun 04:00 schedule,
30m jitter, and Pushover vault paths (cluster-infra app token) to
nas-sandbox-tools defaults.

Co-Authored-By: Claude <noreply@anthropic.com>"
jj commit -m "feat(nas): brew autoupdate defaults [hl-36m6.12]"

Note: jj describe sets the message on the working-copy commit; jj commit then commits it and opens a fresh working copy. If the working copy already has a description from a prior task, jj describe overwrites it — verify with jj log -r '@ | @-' before committing.

Task 2: Create the brew-update service + timer templates

Files:

Create: ansible/roles/nas-sandbox-tools/templates/brew-update.service.j2
Create: ansible/roles/nas-sandbox-tools/templates/brew-update.timer.j2

Step 1: Create brew-update.service.j2

# {{ ansible_managed }}
[Unit]
Description=nas-support Homebrew bundle refresh
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=nas-automation
ExecStart=/bin/bash -lc 'set -euo pipefail; brew update && brew bundle upgrade --file {{ nas_brewfile_path }} && brew autoremove && brew cleanup'
OnFailure=brew-update-notify.service

[Install]
WantedBy=multi-user.target

Step 2: Create brew-update.timer.j2

# {{ ansible_managed }}
[Unit]
Description=Weekly Homebrew bundle refresh timer

[Timer]
OnCalendar={{ nas_brew_autoupdate_schedule }}
RandomizedDelaySec={{ nas_brew_autoupdate_jitter }}
Persistent=true

[Install]
WantedBy=timers.target

Step 3: Verify the role still syntax-checks (structural gate for templates)

ansible-lint and yamllint do not lint .j2/.service templates — ansible-lint only parses playbook/role YAML, and yamllint only handles .yml/.yaml extensions, so running either on a .service.j2 exits 0 regardless of content (a no-op that cannot fail). The real structural gate for template files is ansible-playbook --syntax-check, which parses the whole role including its templates directory:

cd ansible && ansible-playbook -i inventory/hosts.yml ../nas-playbook.yml --tags nas-sandbox-tools --syntax-check

Expected: playbook: nas-playbook.yml with no errors. (Templates are not yet referenced by tasks, so they’re inert — the role remains valid. The template content is exercised later by the --check --diff dry run in Task 5 and the live apply in Task 7.)

Step 4: Commit

jj describe -m "feat(nas): brew-update service+timer templates [hl-36m6.12]

oneshot service runs 'brew bundle upgrade' as nas-automation via login
shell (brew shellenv on PATH); OnFailure -> notify unit. Weekly timer
mirrors router-kopia-backup (OnCalendar, RandomizedDelaySec, Persistent).

Co-Authored-By: Claude <noreply@anthropic.com>"
jj commit -m "feat(nas): brew-update service+timer templates [hl-36m6.12]"

Task 3: Create the brew-update-notify template (Pushover on failure)

Files:

Create: ansible/roles/nas-sandbox-tools/templates/brew-update-notify.service.j2
Step 1: Create brew-update-notify.service.j2

# {{ ansible_managed }}
[Unit]
Description=Pushover notification for failed brew-update

[Service]
Type=oneshot
User=root
EnvironmentFile={{ nas_brew_pushover_env_path }}
ExecStart=/usr/bin/curl -fsS -X POST https://api.pushover.net/1/messages.json \
  --data-urlencode "token=${PUSHOVER_TOKEN}" \
  --data-urlencode "user=${PUSHOVER_USER_KEY}" \
  --data-urlencode "message=nas-support: brew-update.service failed on %H — see: journalctl -u brew-update.service"

Step 2: Verify the role still syntax-checks (structural gate)

ansible-lint does not lint .j2/.service templates (no-op — see Task 2 Step 3 note). Use --syntax-check, which parses the whole role:

cd ansible && ansible-playbook -i inventory/hosts.yml ../nas-playbook.yml --tags nas-sandbox-tools --syntax-check

Expected: playbook: nas-playbook.yml, no errors. (%H is systemd’s hostname specifier, expanded by the manager before exec — not a shell substitution. ${PUSHOVER_TOKEN} / ${PUSHOVER_USER_KEY} are expanded by systemd from EnvironmentFile. Template content is exercised by the Task 5 dry run and Task 7 live apply.)

Step 3: Commit

jj describe -m "feat(nas): brew-update-notify Pushover template [hl-36m6.12]

OnFailure unit curls Pushover API directly (cluster-infra token from
EnvironmentFile). %H specifier for hostname (no shell). User=root only
to read the 0600 env file.

Co-Authored-By: Claude <noreply@anthropic.com>"
jj commit -m "feat(nas): brew-update-notify Pushover template [hl-36m6.12]"

Task 4: Create the role’s first handlers/main.yml

Files:

Create: ansible/roles/nas-sandbox-tools/handlers/main.yml
Step 1: Create handlers/main.yml (this is the role’s first handler file — the role has no handlers/ directory today)

# SPDX-License-Identifier: MIT-0
# code: language=ansible
---
# Handler for the brew-update timer. Fired by the 'Render brew-update units'
# task in tasks/main.yml only when a unit template changed. Enables the timer
# (NOT the oneshot service — enabling the service would run the upgrade on
# every boot).
- name: Reload systemd and enable brew-update timer
  ansible.builtin.systemd:
    name: brew-update.timer
    daemon_reload: true
    enabled: true
    state: started
  become: true

Step 2: Verify lint + syntax-check

cd ansible && ansible-lint --config-file=.ansible-lint roles/nas-sandbox-tools/handlers/main.yml
yamllint -c ../.yamllint.yaml roles/nas-sandbox-tools/handlers/main.yml
ansible-playbook -i inventory/hosts.yml ../nas-playbook.yml --tags nas-sandbox-tools --syntax-check

Expected: all clean. (The handler is not yet notified by any task, so it’s inert — but Ansible must parse it. ansible-lint production profile accepts handler files.)

Step 3: Commit

jj describe -m "feat(nas): nas-sandbox-tools handler for brew-update timer [hl-36m6.12]

Role's first handlers/main.yml. daemon_reload + enable/start the timer
only (never the oneshot service — would run upgrade on every boot).

Co-Authored-By: Claude <noreply@anthropic.com>"
jj commit -m "feat(nas): nas-sandbox-tools handler for brew-update timer [hl-36m6.12]"

Task 5: Wire the tasks block (Vault lookup → render → notify)

Files:

Modify: ansible/roles/nas-sandbox-tools/tasks/main.yml (append after the final Kopia env task, line ~94)
Step 1: Append the gated block

Open ansible/roles/nas-sandbox-tools/tasks/main.yml and append after the final task (Render Kopia repo env):

# --- Scheduled Homebrew bundle refresh (brew-update.timer) ---------------
# Gated on brew being installed AND the autoupdate flag. Renders the Pushover
# env file (0600, root) + the three systemd units, then notifies the handler
# to reload systemd and enable the timer. The oneshot service is never started
# by the play — it runs via the timer (weekly) or manual 'systemctl start'.
- name: Scheduled Homebrew bundle refresh
  when: nas_brew_install | bool and nas_brew_autoupdate | bool
  block:
    - name: Read Pushover user key from Vault
      ansible.builtin.set_fact:
        nas_pushover_user_key: >-
          {{ lookup('community.hashi_vault.vault_kv2_get',
                    nas_pushover_user_vault_path,
                    ca_cert=vault_ca_cert_bundle).data.data.user_key }}
      no_log: true

    - name: Render Pushover env file
      ansible.builtin.copy:
        dest: "{{ nas_brew_pushover_env_path }}"
        content: |
          PUSHOVER_USER_KEY={{ nas_pushover_user_key }}
          PUSHOVER_TOKEN={{ lookup('community.hashi_vault.vault_kv2_get',
                    nas_pushover_token_vault_path,
                    ca_cert=vault_ca_cert_bundle).data.data.token }}
        mode: "0600"
      become: true
      no_log: true

    - name: Render brew-update units
      ansible.builtin.template:
        src: "{{ item }}.j2"
        dest: "/etc/systemd/system/{{ item }}"
        mode: "0644"
      loop:
        - brew-update.service
        - brew-update.timer
        - brew-update-notify.service
      become: true
      notify: Reload systemd and enable brew-update timer

Step 2: Verify lint + syntax-check (structural test)

cd ansible && ansible-lint --config-file=.ansible-lint roles/nas-sandbox-tools/tasks/main.yml
yamllint -c ../.yamllint.yaml roles/nas-sandbox-tools/tasks/main.yml
ansible-playbook -i inventory/hosts.yml ../nas-playbook.yml --tags nas-sandbox-tools --syntax-check

Expected: all clean. ansible-lint production will flag any FQCN violation, command-vs-shell misuse, or missing no_log on the Vault lookups — the block above satisfies all three.

Step 3: Dry-run (idempotence test, no live Vault/SSH required to fail-closed)

cd ansible && ansible-playbook -i inventory/hosts.yml ../nas-playbook.yml --tags nas-sandbox-tools --check --diff --limit nas-support

Expected: the play connects to nas-support over SSH as nas-automation, reads Vault (the controller’s Vault auth is required), and shows a --diff of the three units + the env file being created (changed). The handler fires in --check as a notification (Ansible reports “1 handler would have run”). If Vault auth fails, stop and resolve the controller’s Vault session before proceeding — do not bypass.

Step 4: Commit

jj describe -m "feat(nas): wire brew-update tasks block [hl-36m6.12]

Gated block: Vault lookup (user_key + cluster-infra token) -> render
0600 Pushover env file -> render 3 systemd units -> notify handler.
Reuses vault_ca_cert_bundle trust anchor from the Kopia lookup.

Co-Authored-By: Claude <noreply@anthropic.com>"
jj commit -m "feat(nas): wire brew-update tasks block [hl-36m6.12]"

Task 6: Document verification in the NAS runbook

Files:

Modify: docs/operations/nas.md (append a new section)
Step 1: Append the verification section

Open docs/operations/nas.md and append (matching the file’s existing heading style):

## Brew autoupdate timer (nas-support sandbox)

The `nas-sandbox-tools` role installs a weekly systemd timer that refreshes
the sandbox Homebrew bundle unattended. Managed by Ansible over the
`nas-automation` SSH connection; gated behind `nas_brew_autoupdate` (default
`true`; disable per-host via host vars).

- **Schedule:** `OnCalendar=Sun 04:00` with `RandomizedDelaySec=30m` and
  `Persistent=true` (a missed run replays on next boot).
- **Command:** `brew update && brew bundle upgrade --file ~/.Brewfile &&
  brew autoremove && brew cleanup` (runs as `nas-automation` via a login
  shell so `/etc/profile.d/homebrew.sh` loads brew onto PATH).
- **Failure handling:** `OnFailure=brew-update-notify.service` curls the
  Pushover API (`cluster-infra` app token) with the failing unit name +
  hostname. The token + user key live in a `0600` root-owned env file
  rendered from Vault at play time.

### Verify

Run over SSH as `nas-automation` (or via `ansible nas-support -m command -a ...`):

1. `systemctl list-timers brew-update*` — timer listed, `NEXT` ≈ next Sunday
   04:00 ± jitter.
2. `systemctl start brew-update.service` — trigger a one-off run.
3. `systemctl status brew-update.service` — `Active: inactive (dead)` with
   `ExecStart=` exit 0 (success path).
4. **Failure-path check (optional):** temporarily set the service
   `ExecStart` to a command that exits nonzero, `systemctl start
   brew-update.service`, confirm `brew-update-notify.service` runs and a
   Pushover notification arrives, then revert with the next play.

> **Telemetry note:** unit stdout/stderr lands in the systemd journal, but
> `nas-otel-collector` does not yet ship journald to ClickStack (the distroless
> otel-contrib image lacks `journalctl`; a custom image is a follow-up). Until
> then, inspect via `journalctl -u brew-update.service`. The Pushover alert is
> the active failure surface.

Step 2: Lint the markdown

rumdl check --config docs/.rumdl.toml docs/operations/nas.md

Expected: no new issues. (If rumdl flags MD040/MD032-style issues in pre-existing content, only fix the lines you added.)

Step 3: Commit

jj describe -m "docs(nas): brew autoupdate verification runbook [hl-36m6.12]

Document the brew-update timer schedule, command, Pushover failure path,
and verification steps in docs/operations/nas.md. Note journald telemetry
gap honestly.

Co-Authored-By: Claude <noreply@anthropic.com>"
jj commit -m "docs(nas): brew autoupdate verification runbook [hl-36m6.12]"

Task 7: Full quality gates + live behavioral verification

Files: (none — verification only)

This task is the behavioral acceptance test (the TDD-exception “test passes” rung). All prior tasks’ structural/lint/dry-run gates must already be green.

Step 1: Run the lint suite (YAML role files + markdown docs)

ansible-lint lints the role holistically (playbook/role YAML); yamllint covers the .yml role files (defaults/, tasks/, handlers/). Neither lints .j2/.service templates — those are covered by the --syntax-check in Step 2 and the live apply in Step 3. rumdl covers the markdown docs.

cd ansible && ansible-lint --config-file=.ansible-lint roles/nas-sandbox-tools/
yamllint -c ../.yamllint.yaml roles/nas-sandbox-tools/
cd .. && rumdl check --config docs/.rumdl.toml docs/engineering/specs/2026-06-27-nas-brew-autoupdate-design.md docs/engineering/plans/2026-06-27-nas-brew-autoupdate.md docs/operations/nas.md

Expected: all clean.

Step 2: Full playbook syntax-check

cd ansible && ansible-playbook -i inventory/hosts.yml ../nas-playbook.yml --syntax-check

Expected: playbook: nas-playbook.yml, no errors.

Step 3: Live apply to nas-support

cd ansible && ansible-playbook -i inventory/hosts.yml ../nas-playbook.yml --tags nas-sandbox-tools --diff

Expected: connects as nas-automation, renders /etc/nas-pushover.env (0600) + the three units under /etc/systemd/system/, handler reloads systemd and enables+starts brew-update.timer. --diff shows the new files; the handler reports changed.

Step 4: Verify the timer is registered and enabled

ansible nas-support -i ansible/inventory/hosts.yml -m command -a 'systemctl list-timers brew-update*' -u nas-automation --become

Expected: brew-update.timer listed with NEXT ≈ next Sunday 04:00 ± jitter, ACTIVATES=brew-update.service.

Step 5: Trigger a one-off run and confirm exit 0 (success path)

ansible nas-support -i ansible/inventory/hosts.yml -m command -a 'systemctl start brew-update.service && systemctl show brew-update.service -p ExecMainStatus -p ActiveState --value' -u nas-automation --become

Expected: ActiveState=inactive (oneshot finished) and ExecMainStatus=0 (success). The first run may download/upgrade formulae — allow a few minutes. If ExecMainStatus is nonzero, run ansible nas-support ... -m command -a 'journalctl -u brew-update.service -n 100 --no-pager' -u nas-automation --become and triage via ClickStack/logs before declaring failure.

Step 6: (Optional) Failure-path check

Confirm the Pushover notify path end-to-end by triggering the notify unit directly (it is stateless, so this needs no revert). This validates the notify unit’s curl + creds + network path without breaking the real upgrade:

ansible nas-support -i ansible/inventory/hosts.yml -m command -a 'systemctl start brew-update-notify.service && systemctl show brew-update-notify.service -p ExecMainStatus --value' -u nas-automation --become

Expected: ExecMainStatus=0 (the curl succeeded — Pushover returned 200) and a Pushover notification arrives on the operator’s device with the nas-support: brew-update.service failed on … message body. A nonzero ExecMainStatus means the curl failed (check journalctl -u brew-update-notify.service for the response code — likely an empty/missing Vault secret at the cluster-infra path or a network block).

Step 7: Confirm idempotence — re-apply shows no changes

cd ansible && ansible-playbook -i inventory/hosts.yml ../nas-playbook.yml --tags nas-sandbox-tools --diff

Expected: changed=0 for the render tasks (content unchanged), handler does not fire. This confirms idempotence.

Step 8: Final state check

jj --no-pager log -r 'trunk()..@' --no-graph -T 'change_id.short(6) ++ " " ++ description.first_line()'

Expected: 6 commits on top of main (Tasks 1–6; Task 7 is verification-only, no commit). All gate outputs above are green and recorded.

Spec coverage map

Spec section	Implementing task
Components table (defaults, 3 templates, tasks, handler)	Tasks 1–5
The upgrade command (`brew bundle upgrade`)	Task 2 (service template)
`brew-update.service.j2`	Task 2
`brew-update.timer.j2`	Task 2
`brew-update-notify.service.j2`	Task 3
`/etc/nas-pushover.env` (Vault render)	Task 5
`defaults/main.yml` additions	Task 1
`tasks/main.yml` block + handler	Tasks 4, 5
Idempotence	Task 7 Step 7
Telemetry (stated honestly)	Task 6 (runbook note)
Verification procedure	Task 6 (documented) + Task 7 (executed)
Risks & mitigations	Reflected in Task 6 runbook + Task 7 failure-path check

No spec section is unaccounted for.

Risks during implementation

Risk	Mitigation in plan
Controller Vault session not authenticated → Step 3 of Task 5 fails	Step 3 explicitly says stop and resolve Vault auth; do not bypass.
`ansible-lint` `production` flags a new rule (e.g. `command-instead-of-shell`)	All tasks use `ansible.builtin.template`/`copy`/`set_fact`/`systemd` — no raw `command`/`shell` introduced.
Live `brew bundle upgrade` is slow / network-flaky on first run	Task 7 Step 5 notes “allow a few minutes” + gives the `journalctl` triage command.
Pushover token not yet in Vault at `cluster-infra` path	`docs/reference/secrets.md` already lists it (verified in spec grounding); if the secret is empty, the notify unit returns non-200 — caught in Task 7 Step 6.
Forgetting the `nas-automation` brew-PATH login-shell context	Enforced by `bash -lc` in the service template (Task 2); documented in runbook (Task 6).