NAS Journald Logs → ClickStack Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Ship the TrueNAS nas host’s journald logs into ClickStack (default.otel_logs, service.name=otelcol-nas) alongside the existing host metrics, by switching the role to a CI-built, cosign-signed custom otelcol-contrib image that includes journalctl.

Architecture: Two PRs. PR 1 adds images/nas-otel/Dockerfile (debian-slim + systemd + otelcol-contrib tarball — the validated Firewalla recipe) and .github/workflows/build-nas-otel.yml (modeled on build-headroom-otel.yml, amd64-only) that builds, pushes to ghcr.io/fzymgc-house/nas-otel:0.119.0, and keyless-signs with cosign. PR 2 (gated on the signed image existing) switches nas_otel_image to the GHCR image and adds a journald receiver + transform/journald body-shaping processor + file_storage cursor extension + a logs pipeline to the role config, plus journal/machine-id/cursor mounts to the compose, plus docs.

Tech Stack: OpenTelemetry Collector Contrib 0.119.0 (journaldreceiver, file_storage extension, transform processor), Docker (debian:13-slim base), GitHub Actions (docker/build-push-action, sigstore/cosign-installer), Ansible (TrueNAS app.* middleware via midclt).

Spec: docs/engineering/specs/2026-06-27-nas-journald-logs-design.md (bead hl-36m6.13).

Conventions: Conventional commits (type(scope): description [hl-36m6.13]). This is a jj repo — commit via jj describe -m "..." / jj new -m "..." (never mutating git; the guard hook blocks it). Every commit message MUST end with the AI-authorship byline Co-Authored-By: Claude <noreply@anthropic.com> per global CLAUDE.md — append it as the final line of each commit-message block below (use jj describe -m "<subject>" -m "Co-Authored-By: Claude <noreply@anthropic.com>" or the commit-commands:commit skill, which adds it automatically). All YAML files pass yamllint -c .yamllint.yaml; Ansible files pass cd ansible && ansible-lint --config-file=.ansible-lint; workflows pass actionlint; docs pass rumdl. images/ Dockerfiles use # SPDX-License-Identifier: MIT header.

File Structure

File (PR 1)	Responsibility
`images/nas-otel/Dockerfile`	Custom otelcol-contrib image with `journalctl` (debian-slim + systemd + release tarball)

File (PR 1)	Responsibility
`.github/workflows/build-nas-otel.yml`	Build + push + cosign-sign the image to GHCR on `main`

File (PR 2)	Responsibility
`ansible/roles/nas-otel-collector/defaults/main.yml`	Switch `nas_otel_image`; add journal-dir/cursor-dir/units vars
`ansible/roles/nas-otel-collector/templates/otel-config.yaml.j2`	Add journald receiver, transform/journald, file_storage ext, logs pipeline
`ansible/roles/nas-otel-collector/templates/compose.yaml.j2`	Add journal/machine-id/cursor mounts
`ansible/roles/nas-otel-collector/tasks/main.yml`	Add cursor-dir creation task
`docs/operations/nas.md`	Telemetry section: logs now ship; add logs verify query
`ansible/CLAUDE.md`	Role row + follow-up sentence updates

PR 1 — Custom image + CI workflow

Task 1: Create the custom image Dockerfile

Files:

Create: images/nas-otel/Dockerfile
Step 1: Create the Dockerfile

Create images/nas-otel/Dockerfile with exactly this content:

# SPDX-License-Identifier: MIT
# Custom otel-collector-contrib image with journalctl available for the
# journaldreceiver. Mirrors the official upstream example at:
#   https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/journaldreceiver/examples/container
#
# Why a custom image: otel/opentelemetry-collector-contrib is FROM scratch
# (distroless) and ships only the binary + ca-certs + config.
# journaldreceiver shells out to `journalctl`, which doesn't exist there.
# Bind-mounting the host's journalctl can't work — journalctl is dynamically
# linked and resolves its dynamic linker from the CONTAINER's filesystem, not
# the host's, so it can't find its shared libraries. (Lesson learned on the
# Firewalla role, ansible/roles/otel-collector/templates/Dockerfile.j2, 2026-05.)
# Base is debian-slim with systemd installed (provides journalctl + shared libs);
# the otelcol-contrib release tarball is pulled from the project's GitHub
# releases, pinned to the role's nas_otel_version.

FROM debian:13-slim

ARG OTEL_COL_VERSION=0.119.0

WORKDIR /opt

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        systemd \
        wget \
        ca-certificates \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /usr/share/doc/* /usr/share/man/*

RUN wget -qO- "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTEL_COL_VERSION}/otelcol-contrib_${OTEL_COL_VERSION}_linux_amd64.tar.gz" \
    | tar -xz otelcol-contrib

ENTRYPOINT ["/opt/otelcol-contrib"]

Step 2: Build the image locally to verify it builds and journalctl exists

Run: docker build -t nas-otel:0.119.0-local images/nas-otel Expected: Build succeeds (downloads the otelcol-contrib tarball, installs systemd).

Step 3: Verify journalctl is present in the built image

Run: docker run --rm --entrypoint /bin/sh nas-otel:0.119.0-local -c 'command -v journalctl && journalctl --version' Expected: prints a path like /usr/bin/journalctl and a systemd 257 (or similar) version line. This is the core requirement the distroless upstream image fails.

Step 4: Verify the collector binary runs

Run: docker run --rm nas-otel:0.119.0-local --version Expected: prints otelcol-contrib version 0.119.0 (confirming the tarball extraction + ENTRYPOINT are correct).

Step 5: Commit

jj describe -m "feat(nas-otel): custom journalctl image Dockerfile [hl-36m6.13]

debian:13-slim + systemd (provides journalctl) + otelcol-contrib 0.119.0
release tarball. Mirrors the Firewalla otel-collector Dockerfile.j2 recipe
(validated in prod since 2026-05). Replaces the distroless upstream image
that lacks journalctl, which the journald receiver shells out to."

Task 2: Create the CI build + sign workflow

Files:

Create: .github/workflows/build-nas-otel.yml
Step 1: Determine the current SHA-pinned action versions

The plan pins GitHub Actions to SHAs (repo convention — supply-chain hardening). The versions below match .github/workflows/build-headroom-otel.yml (verified at plan-writing time). Re-confirm each is current before merging by checking the marketplace / build-headroom-otel.yml:

actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
sigstore/cosign-installer — pick the latest SHA-pinned release tag at implementation time (e.g. sigstore/cosign-installer@<sha> # cosign-installer/<release>). Record the chosen SHA in the commit message.
Step 2: Create the workflow file

Create .github/workflows/build-nas-otel.yml with exactly this content (substitute the sigstore/cosign-installer SHA chosen in Step 1 for <COSIGN_INSTALLER_SHA> and its release tag for <COSIGN_RELEASE_TAG>):

# SPDX-License-Identifier: MIT
# yaml-language-server: $schema=https://json.schemastore.org/github-workflow
---
name: Build NAS OTel Image

on:
  push:
    branches:
      - main
    paths:
      - 'images/nas-otel/**'
      - '.github/workflows/build-nas-otel.yml'
    tags:
      - 'nas-otel-*'
  workflow_dispatch:

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository_owner }}/nas-otel

permissions:
  contents: read
  packages: write
  id-token: write  # required for keyless OIDC cosign signing

jobs:
  build-and-sign-amd64:
    runs-on: namespace-profile-linux-amd64-2x4
    outputs:
      image_tag: ${{ steps.vars.outputs.image_tag }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3

      - name: Set image tag from Dockerfile ARG
        id: vars
        run: |
          # The version lives in the ARG OTEL_COL_VERSION line (NOT the FROM
          # line, unlike build-headroom-otel which parses FROM). A copy-paste
          # of headroom's regex would yield an empty tag.
          IMAGE_TAG=$(grep '^ARG OTEL_COL_VERSION=' images/nas-otel/Dockerfile \
            | sed -E 's/^ARG OTEL_COL_VERSION=([0-9]+\.[0-9]+\.[0-9]+).*/\1/')
          echo "image_tag=${IMAGE_TAG}" >> "$GITHUB_OUTPUT"
          echo "Image tag: ${IMAGE_TAG}"

      - name: Log in to GHCR
        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push amd64 image
        id: build
        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
        with:
          context: images/nas-otel
          push: true
          provenance: true
          sbom: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.vars.outputs.image_tag }}
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest

      - name: Install cosign
        uses: sigstore/cosign-installer@<COSIGN_INSTALLER_SHA> # <COSIGN_RELEASE_TAG>

      - name: Sign image (keyless, OIDC)
        env:
          IMAGE_TAG: ${{ steps.vars.outputs.image_tag }}
        run: |
          set -euxo pipefail
          cosign sign --yes "${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}"
          cosign sign --yes "${REGISTRY}/${IMAGE_NAME}:latest"

      - name: Print verification command
        run: |
          echo "Verify with:"
          echo "  cosign verify ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} \\"
          echo "    --certificate-identity-regexp 'https://github.com/${{ github.repository }}/.github/workflows/build-nas-otel.yml@refs/heads/main' \\"
          echo "    --certificate-oidc-issuer https://token.actions.githubusercontent.com"

Step 3: Lint the workflow

Run: actionlint .github/workflows/build-nas-otel.yml Expected: no output, exit 0. (If actionlint flags the placeholder <COSIGN_INSTALLER_SHA> — that’s expected pre-merge; the real SHA from Step 1 must be substituted first. Re-run after substitution.)

Step 4: Lint the workflow YAML

Run: yamllint -c .yamllint.yaml .github/workflows/build-nas-otel.yml Expected: no output, exit 0.

Step 5: Commit

jj new -m "feat(nas-otel): GHCR build + cosign-sign workflow [hl-36m6.13]

Modeled on build-headroom-otel.yml (amd64-only — NAS is x86-64). Builds
images/nas-otel/Dockerfile to ghcr.io/fzymgc-house/nas-otel:0.119.0 + :latest
with provenance + SBOM, then keyless-signs both tags with cosign (id-token:write).
First cosign-signed image in the repo (precedent; repo-wide signing is a
flagged follow-up). Verify binds to the workflow path+ref (fail-closed)."

Note on commits: Step 5 uses jj new -m to start a new change on top of Task 1’s commit, keeping each task as its own commit. If you prefer a single squashed commit for PR 1, use jj describe -m instead and squash before push.

Task 3: Open PR 1, merge, and gate on the signed image

Files: none (VCS + CI operations)

Step 1: Push PR 1’s branch and open the PR

jj bookmark set feat/nas-otel-image -r @
jj git push -b feat/nas-otel-image
gh pr create --head feat/nas-otel-image \
  --title "feat(nas-otel): custom journalctl image + cosign-signed GHCR build [hl-36m6.13]" \
  --body "PR 1 of 2 for hl-36m6.13. Adds images/nas-otel/Dockerfile (debian-slim + systemd + otelcol-contrib 0.119.0) and .github/workflows/build-nas-otel.yml (builds + cosign-signs to GHCR). PR 2 (role changes) is gated on this image existing and being signed. Spec: docs/engineering/specs/2026-06-27-nas-journald-logs-design.md"

Step 2: Merge PR 1

After CI (pr-review, docs-audit, the new build workflow’s dry-validate) passes, merge via gh pr merge <num> --squash. The merge push to main triggers build-nas-otel.yml.

Step 3: Confirm the workflow run completed and the image is published

Run: gh run list --workflow=build-nas-otel.yml --limit 1 Expected: a completed (success) run after merge. Then check the package: Run: gh api /orgs/fzymgc-house/packages/container/nas-otel/versions --jq '.[0].metadata.container.tags' Expected: includes 0.119.0 and latest.

Step 4: Gate — verify the image is signed (criterion 1)

cosign is not installed on the control node by default. Install it once (Homebrew): brew install cosign, then:

Run:

cosign verify ghcr.io/fzymgc-house/nas-otel:0.119.0 \
  --certificate-identity-regexp 'https://github.com/fzymgc-house/selfhosted-cluster/.github/workflows/build-nas-otel.yml@refs/heads/main' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Expected: Verification for ghcr.io/fzymgc-house/nas-otel:0.119.0 ... followed by The signatures were verified under the provided constraints (exit 0).

This is the merge gate for PR 2. If this fails (image missing or unsigned), stop — do not start PR 2. File an issue on hl-36m6.13 and diagnose the workflow.

Step 5: Record the gate result on the bead

bd note hl-36m6.13 "PR 1 merged. Image ghcr.io/fzymgc-house/nas-otel:0.119.0 published + cosign-verified (criterion 1 PASS). PR 2 unblocked."

PR 2 — Role changes + docs

Start a fresh worktree off latest main (now containing PR 1) before Task 4, so the role’s nas_otel_image points at an image that actually exists in GHCR. Run jj git fetch then create the PR 2 branch off main@origin.

Task 4: Add the unit discovery note to the role defaults

Files:

Modify: ansible/roles/nas-otel-collector/defaults/main.yml

This task does the defaults changes that don’t require box access. The unit-list tuning (Task 9) requires SSH access to the nas and is therefore the operator-gated discovery step called out in the spec.

Step 1: Edit defaults/main.yml

In ansible/roles/nas-otel-collector/defaults/main.yml, make three changes:

Change the image line (line 6):

# before:
nas_otel_image: otel/opentelemetry-collector-contrib
# after:
nas_otel_image: ghcr.io/fzymgc-house/nas-otel

After the existing nas_otel_deployment_environment: production line (line 22), append the new journald vars:

# --- journald logs ---
# Persistent journal (survives reboot) — receiver reads via journalctl.
nas_otel_journal_dir: /var/log/journal
# file_storage cursor dir (receiver resumes mid-stream across restarts).
nas_otel_cursor_dir: "{{ nas_otel_base_dir }}/cursor"
# Tailed systemd units. Best-effort default; TUNE in host_vars/nas.yml after
# running `systemctl list-units` / `journalctl --list-boots` on the nas (Task 9).
# otelcol-nas itself runs as a custom_app, not a systemd unit — its container
# logs are not in journald (tailing it is a no-op).
nas_otel_journal_units:
  - sshd.service
  - midcltd.service        # TrueNAS middleware (the appliance's core API)
  - middleware.service
  - containerd.service

Step 2: Lint

Run: cd ansible && ansible-lint --config-file=.ansible-lint roles/nas-otel-collector/defaults/main.yml Expected: no findings.

Run: yamllint -c .yamllint.yaml ansible/roles/nas-otel-collector/defaults/main.yml Expected: no output, exit 0.

Step 3: Commit

jj new -m "feat(nas-otel): switch nas_otel_image to GHCR + journald defaults [hl-36m6.13]"

Task 5: Add journald receiver + transform + file_storage + logs pipeline to the collector config

Files:

Modify: ansible/roles/nas-otel-collector/templates/otel-config.yaml.j2
Step 1: Replace the header comment

Replace lines 1-5 of templates/otel-config.yaml.j2 (the current comment that says “Journald LOGS are deferred”) with:

# {{ ansible_managed }}
# otelcol-contrib (custom journalctl image) on the TrueNAS `nas` host: host metrics
# AND journald logs → otel-gateway (mTLS) → ClickStack. No OTLP receiver (nothing
# pushes to the nas). The image is ghcr.io/fzymgc-house/nas-otel (CI-built,
# cosign-signed) because the upstream otel-contrib image is distroless and lacks
# journalctl, which the journald receiver shells out to.

Step 2: Add the journald receiver

Under receivers:, after the hostmetrics: block (after the network: scraper’s closing lines), append:

  journald:
    directory: {{ nas_otel_journal_dir }}
    units:
{% for unit in nas_otel_journal_units %}
      - {{ unit }}
{% endfor %}
    priority: info
    start_at: end
    storage: file_storage

Step 3: Add the transform/journald processor

Under processors:, after the resource: block (after the deployment.environment attribute line) and before batch:, insert:

  transform/journald:
    log_statements:
      # journald native — body is a Map. Lift fields into attributes, set body
      # to MESSAGE so ClickStack shows readable log lines. (Firewalla role,
      # ansible/roles/otel-collector/templates/otel-config.yaml.j2, empirically
      # debugged 2026-05-14: IsString(Map) is false, so an IsString(body) gate
      # never matched.) No Case 2 (JSON-string body) — the nas has no OTLP
      # receiver, so no log record can arrive as a JSON-string body.
      - context: log
        conditions:
          - IsMap(body)
        statements:
          - merge_maps(attributes, body, "upsert")
          - set(body, attributes["MESSAGE"]) where attributes["MESSAGE"] != nil

Step 4: Add the file_storage extension

Replace the existing extensions: block (currently just health_check):

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"
  file_storage:
    directory: {{ nas_otel_cursor_dir }}

Step 5: Add the logs pipeline + register file_storage

Replace the service: block’s extensions: line and add a logs pipeline. The final service: block should read:

service:
  extensions: [health_check, file_storage]
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection/system, resource, batch]
      exporters: [otlp/gateway]
    logs:
      receivers: [journald]
      processors: [resourcedetection/system, resource, transform/journald, batch]
      exporters: [otlp/gateway]

(The metrics pipeline is unchanged — keep its existing content verbatim; only add the logs: block beneath it and file_storage to the extensions: list.)

Step 6: Verify the rendered config is valid YAML + valid collector config

Render and validate locally. The role renders the config on the nas, but you can render-check on the control node:

Run:

ansible-playbook -i ansible/inventory/hosts.yml ansible/nas-playbook.yml --tags nas-otel --check --diff --limit nas

Expected: dry-run renders the templates without applying; no template/Jinja errors. (This requires Vault + nas SSH access; if unavailable, fall back to a syntax-only check: yamllint -c .yamllint.yaml ansible/roles/nas-otel-collector/templates/otel-config.yaml.j2 — note Jinja {{ }} makes the raw file non-strict-YAML, so yamllint may flag those lines; focus only on structural errors, not the Jinja delimiters.)

Step 7: Lint

Run: cd ansible && ansible-lint --config-file=.ansible-lint roles/nas-otel-collector/templates/otel-config.yaml.j2 Expected: no findings (or only Jinja-delimiter notices, which are acceptable for a .j2 template).

Step 8: Commit

jj new -m "feat(nas-otel): journald receiver + transform + file_storage + logs pipeline [hl-36m6.13]"

Task 6: Add journal/machine-id/cursor mounts to the compose

Files:

Modify: ansible/roles/nas-otel-collector/templates/compose.yaml.j2
Step 1: Add the three new volume mounts

In templates/compose.yaml.j2, append to the volumes: list (after the existing - /:/hostfs:ro line):

      # --- journald logs (read via journalctl in the custom image) ---
      - {{ nas_otel_journal_dir }}:{{ nas_otel_journal_dir }}:ro
      - /etc/machine-id:/etc/machine-id:ro
      # --- file_storage cursor persistence (rw — receiver writes cursors here) ---
      - {{ nas_otel_cursor_dir }}:{{ nas_otel_cursor_dir }}:rw

Leave the existing image:, user:, command:, restart:, and the first three volume lines unchanged. The image line already renders ghcr.io/fzymgc-house/nas-otel:0.119.0 via the nas_otel_image/nas_otel_version vars changed in Task 4.

Step 2: Lint

Run: yamllint -c .yamllint.yaml ansible/roles/nas-otel-collector/templates/compose.yaml.j2 Expected: no structural errors (Jinja delimiters may flag — acceptable for .j2).

Step 3: Commit

jj new -m "feat(nas-otel): journal/machine-id/cursor mounts in compose [hl-36m6.13]"

Task 7: Add the cursor-dir creation task

Files:

Modify: ansible/roles/nas-otel-collector/tasks/main.yml:16-21 (the existing single-path “Ensure base + cert directories” task)
Step 1: Add a second file task for the cursor dir

After the existing “Ensure base + cert directories” task (lines 16-21, which create {{ nas_otel_base_dir }}/certs), insert a new task:

- name: Ensure cursor directory (file_storage)
  ansible.builtin.file:
    path: "{{ nas_otel_cursor_dir }}"
    state: directory
    mode: "0750"
  become: true

(The existing task is single-path, not a loop — a second task is the minimal change. Do not convert the existing task to a loop; it stays as-is.)

Step 2: Lint

Run: cd ansible && ansible-lint --config-file=.ansible-lint roles/nas-otel-collector/tasks/main.yml Expected: no findings.

Step 3: Syntax-check the role

Run: ansible-playbook -i ansible/inventory/hosts.yml ansible/nas-playbook.yml --tags nas-otel --syntax-check Expected: playbook: nas-playbook.yml with no syntax errors.

Step 4: Commit

jj new -m "feat(nas-otel): create file_storage cursor directory [hl-36m6.13]"

Task 8: Update docs (`nas.md` + `ansible/CLAUDE.md`)

Files:

Modify: docs/operations/nas.md (Telemetry section, lines ~118-135)
Modify: ansible/CLAUDE.md (nas-otel-collector role row + the “left untouched” sentence)
Step 1: Update the nas.md telemetry section

In docs/operations/nas.md, replace the “Logs are a follow-up” blockquote (lines ~134-135):

> **Logs are a follow-up:** the journald receiver needs `journalctl`, absent from the distroless
> upstream image — a custom image is tracked separately. Metrics ship today.

with:

- **Logs** ship via the journald receiver, tailing a curated allowlist of systemd units
  (`nas_otel_journal_units`; tune in `host_vars/nas.yml`) at `priority: info`. The receiver
  reads the persistent journal at `/var/log/journal` (bind-mounted `:ro`) and resumes across
  restarts via a `file_storage` cursor at `{{ nas_otel_base_dir }}/cursor` (bind-mounted `:rw`).
  `/etc/machine-id` is mounted for journald host identification. A `transform/journald`
  processor lifts journal fields into attributes and sets the body to the `MESSAGE` text so
  ClickStack shows readable log lines. No filter processor yet — add `filter/nas-noise` once
  real volume data is available. Verify in ClickStack:
  `SELECT count() FROM default.otel_logs WHERE ServiceName='otelcol-nas' AND TimestampTime > now() - INTERVAL 1 HOUR`.

Also update the first telemetry bullet (line ~119) from “shipping host metrics” to “shipping host metrics + journald logs”, and the verify line (line ~132) to mention both queries.

Step 2: Update ansible/CLAUDE.md role row + sentence

In ansible/CLAUDE.md, change the nas-otel-collector role row (line ~38) from:

| `nas-otel-collector` | otelcol-contrib custom app → host metrics → ClickStack (mTLS) | `nas` |

to:

| `nas-otel-collector` | otelcol-contrib custom app → host metrics + journald logs → ClickStack (mTLS, signed custom image) | `nas` |

And update the sentence at the end of the Roles Reference block (line ~41) from “…journald logs are a follow-up — distroless image lacks journalctl).” to “…journald logs ship via a CI-built, cosign-signed custom image (ghcr.io/fzymgc-house/nas-otel) that adds journalctl.”

Step 3: Lint the docs

Run: rumdl check --config docs/.rumdl.toml docs/operations/nas.md Run: rumdl check ansible/CLAUDE.md Expected: no findings (or fix any flagged).

Step 4: Commit

jj new -m "docs(nas): journald logs ship to ClickStack via signed custom image [hl-36m6.13]"

Task 9: Discover real unit names on the nas (operator-gated)

Files:

Modify (conditionally): ansible/inventory/host_vars/nas.yml (add nas_otel_journal_units override)

This is the box-verification step the spec flagged. It requires SSH to the nas as nas-automation.

Step 1: List the nas systemd units that actually log

Run (per docs/operations/nas.md connection model, materialize the Vault key first):

ssh nas-automation@nas 'systemctl list-units --type=service --state=running --no-pager'

Expected: a list of running services. Identify the high-signal units (e.g. sshd.service, TrueNAS middleware units, containerd.service). Confirm midcltd.service/middleware.service exist as named (these are inferred defaults — correct if wrong).

Step 2: Confirm the persistent journal exists at /var/log/journal

Run: ssh nas-automation@nas 'ls -la /var/log/journal /run/log/journal 2>&1; journalctl --list-boots --no-pager | head' Expected: /var/log/journal exists and journalctl --list-boots shows boots (confirms persistent journal + the directory choice).

Step 3: Override the unit list if the defaults are wrong

If Step 1 shows different unit names than the defaults, add to ansible/inventory/host_vars/nas.yml:

nas_otel_journal_units:
  - sshd.service
  - <real-middleware-unit>.service
  - <other-real-unit>.service
  - containerd.service

If the defaults are correct, skip this step (the role defaults apply).

Step 4: Commit (only if host_vars changed)

jj new -m "chore(nas): tune nas_otel_journal_units from box discovery [hl-36m6.13]"

Task 10: Open PR 2, deploy, and verify all acceptance criteria

Files: none (VCS + deploy + verify)

Step 1: Push PR 2’s branch and open the PR

jj bookmark set feat/nas-otel-logs -r @
jj git push -b feat/nas-otel-logs
gh pr create --head feat/nas-otel-logs \
  --title "feat(nas): ship journald logs to ClickStack via signed custom image [hl-36m6.13]" \
  --body "PR 2 of 2 for hl-36m6.13. Switches nas-otel-collector to ghcr.io/fzymgc-house/nas-otel (PR 1, merged + cosign-verified), adds journald receiver + transform/journald + file_storage + logs pipeline, journal/machine-id/cursor mounts, cursor-dir task, and docs. Spec: docs/engineering/specs/2026-06-27-nas-journald-logs-design.md. Acceptance: metrics non-regression + logs in otel_logs + no journalctl crash."

Step 2: Merge PR 2 after CI passes

gh pr merge <num> --squash

Step 3: Pull main and deploy the role

jj git fetch
jj new main
ansible-playbook -i ansible/inventory/hosts.yml ansible/nas-playbook.yml --tags nas-otel \
  -e ansible_ssh_private_key_file=/tmp/nas.key \
  -e "ansible_ssh_common_args='-o IdentitiesOnly=yes -o IdentityAgent=none'"

Expected: the role renders the new config/compose, app.update fires once (compose drift detected), the TrueNAS app pulls ghcr.io/fzymgc-house/nas-otel:0.119.0 and restarts. No task failure.

Step 4: Verify criterion 2 — app running the GHCR image, healthy

ssh nas-automation@nas 'midclt call app.query [["name","=","otelcol-nas"]]'

Expected: one app, image ghcr.io/fzymgc-house/nas-otel:0.119.0, state RUNNING (or equivalent healthy state).

Step 5: Verify criterion 5 — no journalctl-missing crash

ssh nas-automation@nas 'midclt call app.logs otelcol-nas 2>&1 | tail -50'

Expected: collector logs show normal startup (Everything is ready. Begin running and processing data. or similar), no journalctl: not found / executable not found errors.

Step 6: Verify criterion 3 — metrics regression guard (the canary)

Via ClickStack (ClickHouse MCP or your usual query path):

SELECT count() FROM otel_metrics_sum WHERE ServiceName='otelcol-nas' AND TimeUnix > now() - INTERVAL 10 MINUTE

Expected: > 0. If 0: the collector failed to start (the original failure mode) — see Rollback below.

Step 7: Trigger a journald event and verify criterion 4 — logs land

Generate an SSH login to the nas (a real log event for sshd.service), wait ~60s for the 10s batch cycle, then:

SELECT count() FROM default.otel_logs
WHERE ServiceName='otelcol-nas' AND TimestampTime > now() - INTERVAL 5 MINUTE

Expected: > 0. Optionally inspect one record to confirm the body is the readable MESSAGE text (transform/journald working):

SELECT TimestampTime, Body, attributes['MESSAGE'] FROM default.otel_logs
WHERE ServiceName='otelcol-nas' ORDER BY TimestampTime DESC LIMIT 3

Step 8: Record outcome and close the bead at merge

bd note hl-36m6.13 "PR 2 merged + deployed. Criteria 1-5 PASS: image cosign-verified; app RUNNING on GHCR image; metrics non-regression (otel_metrics_sum > 0); logs landing in otel_logs (service.name=otelcol-nas); no journalctl crash. Logs design complete."
bd close hl-36m6.13

Rollback (if criterion 3 fails): revert PR 2 (gh pr revert <num>) and re-run the playbook — the app reverts to the upstream distroless image (otel/opentelemetry-collector-contrib) on the next app.update, metrics resume, logs go away (back to pre-change state). PR 1 can remain merged (inert if nothing references the image). Diagnose the startup failure before re-attempting.

Follow-ups (out of scope, file as separate beads)

Retrofit cosign signing onto the other 6 existing build-* workflows (repo-wide hardening).
Add a filter/nas-noise processor once real ClickStack log-volume data is available post-deploy.

NAS Journald Logs → ClickStack Implementation Plan

File Structure

PR 1 — Custom image + CI workflow

Task 1: Create the custom image Dockerfile

Task 2: Create the CI build + sign workflow

Task 3: Open PR 1, merge, and gate on the signed image

PR 2 — Role changes + docs

Task 4: Add the unit discovery note to the role defaults

Task 5: Add journald receiver + transform + file_storage + logs pipeline to the collector config

Task 6: Add journal/machine-id/cursor mounts to the compose

Task 7: Add the cursor-dir creation task

Task 8: Update docs (nas.md + ansible/CLAUDE.md)

Task 9: Discover real unit names on the nas (operator-gated)

Task 10: Open PR 2, deploy, and verify all acceptance criteria

Follow-ups (out of scope, file as separate beads)

Task 8: Update docs (`nas.md` + `ansible/CLAUDE.md`)