Alloy Observability Design

Revision (2026-01-04): Network/security logs (Zeek, FW_ADT firewall audit, ACL DNS logs) removed from Loki pipeline per PR #592. These high-cardinality security event logs are better suited for specialized SIEM tools. The architecture below reflects the original design; current implementation collects only systemd journal and Docker container logs from Firewalla.

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Unified observability for fzymgc-house infrastructure, collecting logs and metrics from both the Kubernetes cluster and the Firewalla router.

Architecture: Two-tier Alloy deployment with K8s Alloy acting as a relay gateway for external collectors.

Tech Stack: Grafana Alloy, OTLP/HTTP, Vault AppRole, Ansible, Docker Compose

Issues: Closes #335 (disable Grafana stats reporting), #494 (install Alloy on Firewalla)

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                         Firewalla Router                            │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ Alloy Container                                              │   │
│  │  ├─ loki.source.journal (systemd + kernel logs)             │   │
│  │  ├─ loki.source.docker (container logs)                     │   │
│  │  ├─ loki.source.file (Zeek network logs)                    │   │
│  │  ├─ prometheus.exporter.unix (system metrics)               │   │
│  │  └─ prometheus.exporter.cadvisor (container metrics)        │   │
│  └──────────────────────────┬──────────────────────────────────┘   │
└─────────────────────────────┼───────────────────────────────────────┘
                              │ OTLP/HTTP + TLS + Bearer Token
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         K8s Cluster                                 │
│                                                                     │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ Alloy-Ingest (Deployment, 2 replicas)                        │   │
│  │  ├─ otelcol.receiver.otlp (external ingestion)              │   │
│  │  ├─ otelcol.auth.bearer (token validation)                  │   │
│  │  ├─ otelcol.exporter.prometheus → prometheus.remote_write   │   │
│  │  └─ otelcol.exporter.loki → loki.write                      │   │
│  └──────────────────────────┬──────────────────────────────────┘   │
│                              │                                      │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ Alloy (DaemonSet, all nodes)                                 │   │
│  │  ├─ loki.source.kubernetes (pod logs)                       │   │
│  │  ├─ loki.source.kubernetes_events                           │   │
│  │  └─ loki.write                                              │   │
│  └──────────────────────────┬──────────────────────────────────┘   │
│                              ▼                                      │
│              ┌──────────────────────────────┐                      │
│              │   Loki + Prometheus          │                      │
│              └──────────────────────────────┘                      │
└─────────────────────────────────────────────────────────────────────┘

Key Decisions:

Separated K8s deployments for failure isolation:
- monitoring-alloy (DaemonSet) - Cluster monitoring only
- monitoring-alloy-ingest (Deployment) - OTLP ingestion only
Bearer token authentication via Vault
OTLP/HTTP via Traefik IngressRoute to alloy-ingest Service
Static Alloy config on Firewalla via Ansible
Dedicated alloy-agent AppRole for Vault Agent on Firewalla

Why Separated:

Failure in OTLP auth doesn’t affect cluster monitoring
Independent scaling (2 replicas vs every node)
Cleaner configuration per use case
Easier debugging and troubleshooting

K8s Alloy Changes

Chart Version

Chart: alloy from https://grafana.github.io/helm-charts
Version: 1.5.1 (appVersion v1.12.1)

Deployment Architecture

Two separate ArgoCD Applications using the same Helm chart with different configurations:

1. Cluster Monitoring (`monitoring-alloy`)

Controller: DaemonSet (runs on all nodes)

// Pod logs via Kubernetes API
loki.source.kubernetes "pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [loki.process.pods.receiver]
}

// Kubernetes events
loki.source.kubernetes_events "events" {
  job_name   = "kubernetes-events"
  log_format = "json"
  forward_to = [loki.process.events.receiver]
}

// Ship to Loki
loki.write "default" {
  endpoint {
    url = "http://loki-gateway.loki.svc.cluster.local/loki/api/v1/push"
  }
}

2. OTLP Ingestion (`monitoring-alloy-ingest`)

Controller: Deployment (2 replicas)

// Bearer token from Vault secret
local.file "bearer_token" {
  filename  = "/var/secrets/bearer/bearer-token"
  is_secret = true
}

otelcol.auth.bearer "default" {
  token = local.file.bearer_token.content
}

// OTLP receiver with auth
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
    auth     = otelcol.auth.bearer.default.handler
  }
  http {
    endpoint = "0.0.0.0:4318"
    auth     = otelcol.auth.bearer.default.handler
  }
  output {
    logs    = [otelcol.processor.batch.default.input]
    metrics = [otelcol.processor.batch.default.input]
  }
}

// Batch and export
otelcol.processor.batch "default" {
  output {
    logs    = [otelcol.exporter.loki.default.input]
    metrics = [otelcol.exporter.prometheus.default.input]
  }
}

otelcol.exporter.loki "default" {
  forward_to = [loki.write.default.receiver]
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://prometheus-kube-prometheus-prometheus.prometheus.svc.cluster.local:9090/api/v1/write"
  }
}

New Resources

Resource	Purpose
ArgoCD Application	`monitoring-alloy` (DaemonSet for cluster monitoring)
ArgoCD Application	`monitoring-alloy-ingest` (Deployment for OTLP ingestion)
IngressRoute	`alloy-ingest.fzymgc.house` → alloy-ingest Service (port 4318)
ExternalSecret	Bearer token from `secret/fzymgc-house/cluster/alloy`
Certificate	TLS via cert-manager (`cloudflare-acme-issuer`)

Files

File	Purpose
`argocd/cluster-app/templates/monitoring-alloy.yaml`	Cluster monitoring DaemonSet
`argocd/cluster-app/templates/monitoring-alloy-ingest.yaml`	OTLP ingestion Deployment
`argocd/app-configs/monitoring-alloy/ingress.yaml`	IngressRoute + Certificate
`argocd/app-configs/monitoring-alloy/external-secret.yaml`	Bearer token ExternalSecret
`argocd/app-configs/monitoring-alloy/kustomization.yaml`	Kustomize configuration

Firewalla Alloy (Ansible Role)

Role Structure

ansible/roles/alloy/
├── defaults/main.yml
├── tasks/main.yml
├── templates/
│   ├── docker-compose.yml.j2
│   ├── config.alloy.j2
│   └── vault-agent.hcl.j2
└── files/
    └── fzymgc-ca-bundle.pem

Docker Compose (with Vault Agent sidecar)

services:
  vault-agent:
    image: "{{ alloy_vault_image }}"
    restart: unless-stopped
    volumes:
      - "{{ alloy_base_dir }}/vault-agent.hcl:/etc/vault/agent.hcl:ro"
      - "{{ alloy_base_dir }}/credentials:/credentials:rw"
      - "{{ alloy_base_dir }}/certs:/certs:ro"
    command: ["agent", "-config=/etc/vault/agent.hcl"]

  alloy:
    image: "{{ alloy_image }}:{{ alloy_version }}"
    restart: unless-stopped
    depends_on:
      - vault-agent
    volumes:
      - "{{ alloy_base_dir }}/config.alloy:/etc/alloy/config.alloy:ro"
      - "{{ alloy_base_dir }}/credentials:/credentials:ro"
      - "/run/log/journal:/run/log/journal:ro"
      - "/var/log:/var/log:ro"
      - "/log/blog:/log/blog:ro"  # Zeek logs
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    command: ["run", "/etc/alloy/config.alloy"]
    network_mode: host

Firewalla-Specific Paths

Source	Path	Notes
Systemd journal	`/run/log/journal`	Volatile (RAM), not `/var/log/journal`
Zeek network logs	`/log/blog/`	Compressed JSON logs
Docker socket	`/var/run/docker.sock`	Container discovery

Log Collection Configuration

Sources

Source	Method	Notes
Systemd journal	`loki.source.journal`	Includes kernel via `_TRANSPORT=kernel`
Docker containers	`loki.source.docker`	router-hosts, vault-agent
Zeek logs	`loki.source.file`	Compressed JSON, needs decompression

Filtering (Drop High-Volume Noise)

loki.process "journal_filter" {
  forward_to = [otelcol.receiver.loki.default.input]

  // Drop dnsmasq [ACL][Allowed] - too verbose (every DNS query)
  stage.match {
    selector = "{job=\"firewalla-journal\"}"
    stage.drop {
      expression = "\\[ACL\\]\\[Allowed\\]"
      drop_counter_reason = "dnsmasq_acl_allowed"
    }
  }

  // Drop miniupnpd SSDP spam
  stage.match {
    selector = "{job=\"firewalla-journal\"}"
    stage.drop {
      expression = "miniupnpd.*SSDP packet sender.*not from a LAN"
      drop_counter_reason = "miniupnpd_ssdp_noise"
    }
  }

  // Drop FW_ADT continues (routine packet flow)
  stage.match {
    selector = "{job=\"firewalla-journal\"}"
    stage.drop {
      expression = "\\[FW_ADT\\]A=C"
      drop_counter_reason = "fw_adt_continue"
    }
  }
}

Keeps:

[ACL][Blocked] - Security events (DNS blocks)
[FW_ADT]A=A - Accepts
[FW_ADT]A=RD - Route denied
[FW_ADT]D=W - WAN blocks
All kernel messages
Docker container logs
Zeek network logs

Metrics Collection Configuration

// System metrics (CPU, memory, disk, network)
prometheus.exporter.unix "system" {
  include_exporter_metrics = true
  enable_collectors = [
    "cpu", "diskstats", "filesystem", "loadavg",
    "meminfo", "netdev", "stat", "time", "uname",
  ]
}

// Docker container metrics
prometheus.exporter.cadvisor "docker" {
  docker_host = "unix:///var/run/docker.sock"
  storage_duration = "5m"
}

prometheus.scrape "local" {
  targets = concat(
    prometheus.exporter.unix.system.targets,
    prometheus.exporter.cadvisor.docker.targets,
  )
  forward_to = [otelcol.receiver.prometheus.default.input]
  scrape_interval = "30s"
}

Terraform (Vault)

New Resources

resource "random_password" "alloy_external_token" {
  length  = 32
  special = false
}

resource "vault_kv_secret_v2" "alloy" {
  mount = vault_mount.secret.path
  name  = "fzymgc-house/cluster/alloy"
  data_json = jsonencode({
    external_token = random_password.alloy_external_token.result
  })
}

resource "vault_policy" "alloy_agent" {
  name   = "alloy-agent"
  policy = <<EOT
path "secret/data/fzymgc-house/cluster/alloy" {
  capabilities = ["read"]
}
path "secret/metadata/fzymgc-house/cluster/alloy" {
  capabilities = ["read", "list"]
}
EOT
}

resource "vault_approle_auth_backend_role" "alloy_agent" {
  backend        = vault_auth_backend.approle.path
  role_name      = "alloy-agent"
  token_policies = [vault_policy.alloy_agent.name]
  token_ttl      = 3600
  token_max_ttl  = 86400
}

Implementation Tasks

Phase 1: K8s Alloy (fixes #335)

Upgrade Alloy Helm chart to 1.5.1
Add enableReporting: false
Verify existing log collection still works

Phase 2: K8s Relay Infrastructure

Create Vault secret for bearer token
Add OTLP receiver to Alloy config
Create IngressRoute + ExternalSecret
Test relay endpoint responds

Phase 3: Terraform (Vault)

Create alloy-agent AppRole
Create policy with secret/data/fzymgc-house/cluster/alloy read access
Deploy via HCP Terraform

Phase 4: Ansible Role

Create alloy role structure
Add Vault Agent sidecar config
Add Alloy River config with filters
Add docker-compose template
Add boot script for persistence
Test on Firewalla

Phase 5: Validation

Verify logs appear in Loki with host=Firewalla
Verify metrics appear in Prometheus
Verify Zeek logs are searchable
Close issues #335 and #494

Files Summary

Create

File	Purpose
`argocd/app-configs/alloy/ingressroute.yaml`	Traefik IngressRoute for OTLP ingestion
`argocd/app-configs/alloy/externalsecret.yaml`	Bearer token from Vault
`tf/vault/alloy.tf`	AppRole, policy, secret
`ansible/roles/alloy/`	Complete Ansible role

Modify

File	Changes
`argocd/app-configs/alloy/values.yaml`	Chart upgrade, OTLP receiver, enableReporting
`argocd/app-configs/alloy/kustomization.yaml`	Add new resources
`ansible/router-playbook.yml`	Add alloy role

Alloy Observability Design

Architecture Overview

K8s Alloy Changes

Chart Version

Deployment Architecture

1. Cluster Monitoring (monitoring-alloy)

2. OTLP Ingestion (monitoring-alloy-ingest)

New Resources

Files

Firewalla Alloy (Ansible Role)

Role Structure

Docker Compose (with Vault Agent sidecar)

Firewalla-Specific Paths

Log Collection Configuration

Sources

Filtering (Drop High-Volume Noise)

Metrics Collection Configuration

Terraform (Vault)

New Resources

Implementation Tasks

Phase 1: K8s Alloy (fixes #335)

Phase 2: K8s Relay Infrastructure

Phase 3: Terraform (Vault)

Phase 4: Ansible Role

Phase 5: Validation

Files Summary

Create

Modify

1. Cluster Monitoring (`monitoring-alloy`)

2. OTLP Ingestion (`monitoring-alloy-ingest`)