NAS Journald Logs → ClickStack Design

For Claude: This is a design spec. Implement via superpowers:writing-plans → executing-plans.

Goal: Extend nas-otel-collector to ship the TrueNAS nas host’s journald logs into ClickStack (default.otel_logs, service.name=otelcol-nas), alongside the existing host metrics (unchanged). Today the role ships metrics only — the upstream otel/opentelemetry-collector-contrib image is distroless and lacks journalctl, which the journald receiver shells out to (the receiver has no internal journal reader; confirmed via OTel contrib docs). Adding the receiver to the distroless image crashes collector startup.

Architecture: A custom otelcol-contrib image (debian-slim + systemd for journalctl, + the otelcol-contrib release binary) is built by GitHub Actions to GHCR and keyless-signed with cosign. The Ansible role switches nas_otel_image to that GHCR image and adds a journald receiver + transform/journald body-shaping processor + file_storage cursor extension + a logs pipeline to otel-config.yaml.j2, plus journal/machine-id/cursor mounts to compose.yaml.j2. The TrueNAS custom-app runtime pulls the signed image from the registry as it does for any image — no on-box build step.

Bead: hl-36m6.13 (parent epic hl-36m6).

Problem (authoritative)

The bead states the problem: ship nas journald LOGS to ClickStack. The bead’s “mirror the Firewalla otel-collector role’s Dockerfile.j2 + custom-image build” text is a candidate solution, treated as a hypothesis and validated against root-cause analysis during this design. Findings:

• The Dockerfile recipe (debian-slim + systemd + tarball) is reusable and correct — validated in production by the Firewalla role since 2026-05.
• The build/delivery mechanism (“custom-image build” on-box) does not transfer: the Firewalla role builds via docker build on a Linux host and runs under systemd docker-compose@; the nas role deploys via the TrueNAS app.* custom-app model, whose runtime pulls images from a registry and exposes no docker build step. The on-box build approach is incompatible with the nas delivery path.
• The repo already has the canonical registry-delivery pattern: .github/workflows/build-headroom-otel.yml builds a derived otel image to GHCR (per-arch jobs + buildx imagetools multi-arch manifest). That mechanism — not the Firewalla build mechanism — is the right fit for the nas.

Design

Component 1 — Custom image (images/nas-otel/Dockerfile)

Adapts the Firewalla ansible/roles/otel-collector/templates/Dockerfile.j2 content to a non-templated CI-build form:

# Custom otel-collector-contrib image with journalctl available for the
# journaldreceiver. Mirrors the official upstream example at:
#   https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/journaldreceiver/examples/container
#
# Why a custom image: otel/opentelemetry-collector-contrib is FROM scratch
# (distroless) and ships only the binary + ca-certs + config.
# journaldreceiver shells out to `journalctl`, which doesn't exist there.
# Bind-mounting the host's journalctl can't work — journalctl is dynamically
# linked and resolves its dynamic linker from the CONTAINER's filesystem, not
# the host's, so it can't find its shared libraries. (Lesson learned on the
# Firewalla role, 2026-05.) Base is debian-slim with systemd installed
# (provides journalctl + shared libs); the otelcol-contrib release tarball is
# pulled from the project's GitHub releases, pinned to the role's version.

FROM debian:13-slim

ARG OTEL_COL_VERSION=0.119.0

WORKDIR /opt

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        systemd \
        wget \
        ca-certificates \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /usr/share/doc/* /usr/share/man/*

RUN wget -qO- "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTEL_COL_VERSION}/otelcol-contrib_${OTEL_COL_VERSION}_linux_amd64.tar.gz" \
    | tar -xz otelcol-contrib

ENTRYPOINT ["/opt/otelcol-contrib"]

• Pinned to 0.119.0 (the role’s nas_otel_version); the version is the image tag.
• debian:13-slim + systemd — identical to the validated Firewalla recipe.
• amd64 tarball only — NAS is x86-64 (confirmed: nas_sandbox_image_name: ubuntu:noble:amd64). Single-arch build avoids the multi-arch manifest complexity build-headroom-otel.yml carries (which exists only because headroom serves ≥2 arches). A one-line note in the workflow records that headroom’s per-arch split is the template if arm64 is ever needed.

Component 2 — CI workflow (.github/workflows/build-nas-otel.yml)

Modeled on build-headroom-otel.yml, amd64-only, with image signing & attestation (this is the first signed image in the repo — precedent-setting):

• Triggers: push to main on images/nas-otel/** or the workflow file; tags nas-otel-*; workflow_dispatch.
• Env: REGISTRY: ghcr.io, IMAGE_NAME: ${{ github.repository_owner }}/nas-otel.
• Permissions: contents: read, packages: write, id-token: write (required for keyless OIDC cosign signing).
• Runner: namespace-profile-linux-amd64-2x4.
• Steps:
  1. actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 (SHA-pinned, repo convention).
  2. Set image tag from the Dockerfile OTEL_COL_VERSION ARG → 0.119.0. (Same grep+sed approach as headroom’s version extraction, but the regex targets the ARG OTEL_COL_VERSION= line, not the ^FROM line headroom parses — the nas version lives in the ARG, not the base image. A copy-paste of headroom’s regex would yield an empty tag.)
  3. docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0 to GHCR via ${{ secrets.GITHUB_TOKEN }}.
  4. docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0 with context: images/nas-otel, push: true, provenance: true, sbom: true, tags ghcr.io/fzymgc-house/nas-otel:0.119.0 and :latest.
  5. sigstore/cosign-installer (SHA-pinned to latest at implementation time).
  6. cosign sign --yes on both :0.119.0 and :latest (keyless — uses the GitHub OIDC token; no key material to store in Vault).

The signature, SLSA provenance attestation, and SBOM together form the three-layer OCI attestation stack (who built it / how it was built / what’s in it). All are free under buildkit + keyless cosign.

Verification identity (trust anchor): the signature is bound to the exact workflow path and ref:

--certificate-identity-regexp 'https://github.com/fzymgc-house/selfhosted-cluster/.github/workflows/build-nas-otel.yml@refs/heads/main'
--certificate-oidc-issuer https://token.actions.githubusercontent.com

Fail-closed: an image built by any other workflow (e.g. a future staging workflow publishing to the same package) fails this verify step.

Scope guardrail: signing is specified for this image only. Retrofitting signing onto the other 6 existing build-* workflows is a separate, repo-wide hardening task — out of scope for this bead, flagged as a follow-up so the precedent is intentional.

Component 3 — Role defaults (ansible/roles/nas-otel-collector/defaults/main.yml)

nas_otel_image: ghcr.io/fzymgc-house/nas-otel   # was: otel/opentelemetry-collector-contrib
nas_otel_version: "0.119.0"                      # unchanged — matches the GHCR tag

# --- new: journald logs ---
nas_otel_journal_dir: /var/log/journal           # persistent journal (survives reboot)
nas_otel_cursor_dir: "{{ nas_otel_base_dir }}/cursor"
nas_otel_journal_units:
  - sshd.service
  - midcltd.service          # TrueNAS middleware (the appliance's core API)
  - middleware.service
  - containerd.service
  # otelcol-nas itself runs as a custom_app, not a systemd unit; its container
  # logs are not in journald. Tailing it is a no-op. Tune in host_vars/nas.yml.

The unit list is a best-effort inference, not box-verified — the implementation plan includes a discovery step (journalctl --list-boots, systemctl list-units) on the nas before first deploy, with nas_otel_journal_units tuned in host_vars/nas.yml. It is a variable so the operator can tune it without touching the role (same extensibility as the Firewalla role).

Component 4 — Collector config (templates/otel-config.yaml.j2)

Four additions to the existing metrics-only config; the metrics pipeline is untouched.

file_storage extension (new, alongside health_check):

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"
  file_storage:
    directory: {{ nas_otel_cursor_dir }}

journald receiver (under receivers:):

  journald:
    directory: {{ nas_otel_journal_dir }}
    units:
{% for unit in nas_otel_journal_units %}
      - {{ unit }}
{% endfor %}
    priority: info
    start_at: end
    storage: file_storage

• start_at: end — first-ever run starts at “now” (no back-fill of the historical journal into ClickStack on first deploy).
• storage: file_storage — subsequent restarts resume from the persisted cursor.
• directory is /var/log/journal (persistent journal).

transform/journald processor (body-shaping only — no firewalla Case 3 service.name setter; the nas resource processor already upserts service.name=otelcol-nas on all telemetry):

  transform/journald:
    log_statements:
      # journald native — body is a Map. Lift fields into attributes, set body
      # to MESSAGE so ClickStack shows readable log lines. (Firewalla role,
      # empirically debugged 2026-05-14: IsString(Map) is false, so the prior
      # IsString(body) gate never matched.)
      - context: log
        conditions:
          - IsMap(body)
        statements:
          - merge_maps(attributes, body, "upsert")
          - set(body, attributes["MESSAGE"]) where attributes["MESSAGE"] != nil

Case 2 (defensive JSON-string body parse) is omitted — the nas collector has no OTLP receiver (nothing pushes to it; confirmed in the existing config comment), so no log record can arrive as a JSON-string body. Carrying that branch would be unreachable code.

No filter processor by design. The Firewalla filter/firewalla-noise is Firewalla-specific (dnsmasq audit chatter). TrueNAS chattiness is different and unknown. A wrong filter drops signal (silent data loss — the worst failure mode for a telemetry pipeline); no filter only costs ClickStack storage (recoverable, visible). Ship without a filter; add filter/nas-noise in a follow-up bead once real volume data exists. The unit allowlist is the first volume-control knob.

logs pipeline (under service.pipelines:):

    logs:
      receivers: [journald]
      processors: [resourcedetection/system, resource, transform/journald, batch]
      exporters: [otlp/gateway]

Pipeline order: resourcedetection/system + resource (populate host.name, service.name=otelcol-nas, deployment.environment) → transform/journald (shape body) → batch. Shares the existing otlp/gateway exporter (mTLS, already configured) — logs and metrics share the gateway + ClickStack path. service.extensions becomes [health_check, file_storage].

Component 5 — Compose (templates/compose.yaml.j2)

Three new mounts; image already switched via the variable; no other changes:

services:
  otelcol:
    image: {{ nas_otel_image }}:{{ nas_otel_version }}
    # ...existing user: "0:0", command, restart, /hostfs mount...
    volumes:
      - {{ nas_otel_base_dir }}/config.yaml:/etc/otelcol/config.yaml:ro
      - {{ nas_otel_base_dir }}/certs:/etc/otelcol/certs:ro
      - /:/hostfs:ro
      # --- new: journald logs ---
      - {{ nas_otel_journal_dir }}:{{ nas_otel_journal_dir }}:ro
      - /etc/machine-id:/etc/machine-id:ro
      # --- new: cursor persistence (rw — file_storage writes cursors here) ---
      - {{ nas_otel_cursor_dir }}:{{ nas_otel_cursor_dir }}:rw

• Journal :ro — receiver reads via journalctl, never writes.
• /etc/machine-id :ro — journald uses it for host identification; a mismatch makes journalctl refuse to read the files.
• Cursor :rw — the only new writable mount, under nas_otel_base_dir on the apps pool.

The role continues to run as user: "0:0" (root) — already required for /hostfs access; journalctl reading + cursor writing both work under root. No group_add needed (the Firewalla needed systemd-journal GID only because it ran as a non-root pi user).

Component 6 — Role task flow (tasks/main.yml)

Minimal deltas; the existing metrics-working flow is unchanged:

1. The “Ensure base + cert directories” task → add a second ansible.builtin.file task (the existing one is single-path, not a loop) creating {{ nas_otel_cursor_dir }} (mode 0750, become: true).
2. The Dockerfile.j2 template is not added to this role — the image is built by CI, not on-box. (Explicit departure from the Firewalla role: Dockerfile content borrowed, build mechanism not.)
3. Everything downstream (render config → render compose drift marker → build compose string → app.query → app.create/app.update) is unchanged. The existing app.update drift gate fires on config/compose/cert change, so the new mounts + image are picked up as a compose change → one app.update cycle on first deploy.

Documentation changes

File	Change
docs/operations/nas.md	Telemetry section: flip “Logs are a follow-up” block → describe the custom GHCR image + journald receiver + units allowlist + cursor persistence. Add the logs verify query.
ansible/CLAUDE.md	nas-otel-collector role row: host metrics → ClickStack → host metrics + journald logs → ClickStack (mTLS, signed custom image). Update the “journald logs are a follow-up” sentence.

Logs verify query (parallel to the existing metrics one):

SELECT count() FROM default.otel_logs
WHERE ServiceName='otelcol-nas' AND TimestampTime > now() - INTERVAL 1 HOUR;

Acceptance criteria

1. Image signed: cosign verify ghcr.io/fzymgc-house/nas-otel:0.119.0 --certificate-identity-regexp 'https://github.com/fzymgc-house/selfhosted-cluster/.github/workflows/build-nas-otel.yml@refs/heads/main' --certificate-oidc-issuer https://token.actions.githubusercontent.com succeeds.
2. App deploy: midclt call app.query shows otelcol-nas running the GHCR image; app status healthy.
3. Metrics regression guard: SELECT count() FROM otel_metrics_sum WHERE ServiceName='otelcol-nas' still returns > 0 (the metrics pipeline must not regress — this is the canary for the original “collector crashes on startup” failure mode).
4. Logs: SELECT count() FROM default.otel_logs WHERE ServiceName='otelcol-nas' AND TimestampTime > now() - INTERVAL 1 HOUR returns > 0 after a journald-producing event (e.g. an SSH login to the nas).
5. No startup crash: container logs show no journalctl-missing error.

Sequencing — two PRs with a merge gate

The image must exist in GHCR (and be signed) before the Ansible role pulls it.

• PR 1: images/nas-otel/Dockerfile + .github/workflows/build-nas-otel.yml. Merge → workflow fires on push to main touching images/nas-otel/** → publishes + signs the image.
• Gate: confirm the image exists in GHCR and cosign verify (criterion 1) passes before merging PR 2.
• PR 2: role defaults/config/compose changes + docs.

If PR 1’s workflow run fails or the image is unsigned, PR 2 must not merge — the role would point at a non-existent/unsigned image and the TrueNAS app would fail to start (the metrics regression would be the visible symptom).

Rollback: if the deploy regresses metrics (criterion 3 fails), revert PR 2; the TrueNAS app reverts to the upstream distroless image on the next app.update (metrics resume, logs go away — back to the pre-change state). PR 1 (image + workflow) can remain merged; it’s inert if nothing references the image.

Risks & mitigations

Risk	Mitigation
Unit allowlist names wrong (inferred, not box-verified)	Plan step: journalctl --list-boots / systemctl list-units on nas before first deploy; tune nas_otel_journal_units in host_vars/nas.yml.
First deploy dumps historical journal	start_at: end + storage: file_storage — first run starts at “now”, no back-fill.
journalctl crashes despite custom image (host journal / in-image systemd version skew)	Recipe validated in prod on the Firewalla since 2026-05. Lighter fallback first: the OTel contrib journaldreceiver README documents a chroot option (root_path + journalctl_path to use the host’s journalctl via the custom image’s own linker) to dodge version skew. Heavier fallback: bump debian:13-slim base or pin systemd to match the TrueNAS journal version — a follow-up, not a design change.
ClickStack log volume excessive	No filter shipped by design; follow-up bead adds filter/nas-noise once real volume data exists.
Signing precedent drift (only this image signed)	Follow-up note flags repo-wide signing as a separate task; precedent is intentional.

Follow-ups (out of scope for this bead)

• Retrofit cosign signing onto the other 6 existing build-* workflows (repo-wide hardening).
• Add a filter/nas-noise processor once real ClickStack volume data is available.