Alloy Ingest Log Parsing Design

Date: 2026-01-03 Status: Approved

Revision (2026-01-04): Zeek log parsing removed per PR #592. Network/security logs (Zeek, FW_ADT, ACL) are better suited for SIEM tools and are no longer collected into Loki. The detect_source and parse_logs stages below show the original design; current implementation handles only journal and docker sources.

Problem

Logs from Firewalla arrive at the K8s alloy-ingest via OTLP, but otelcol.exporter.loki wraps them in {"body": "..."} JSON format. This makes logs hard to read in Grafana and loses structured information.

Solution

Add a multi-stage loki.process pipeline on the K8s alloy-ingest to:

1. Unwrap the OTLP JSON body
2. Detect log source from labels
3. Apply source-specific parsing
4. Normalize labels and write to Loki

Architecture

OTLP Receiver
     ↓
otelcol.exporter.loki → loki.process "unwrap_otlp"
                              ↓
                        loki.process "detect_source"
                              ↓
                        loki.process "parse_logs"
                              ↓
                        loki.process "finalize"
                              ↓
                        loki.write

Processing Stages

Stage 1: OTLP Unwrap

Extracts the body field from the JSON wrapper created by otelcol.exporter.loki.

• Input: {"body": "2026-01-03 13:28:46 magicsock: derp-27..."}
• Output: 2026-01-03 13:28:46 magicsock: derp-27...

Stage 2: Source Detection

Detects log source from the job label set by router Alloy:

Job Label	Source Label
firewalla-journal	journal
docker	docker
zeek	zeek
(other)	unknown

Stage 3: Source-Specific Parsing

Journal logs:

• Extract log level from message text using regex
• Normalize level variants (warning→warn, critical→crit)

Docker logs:

• Parse JSON if present
• Extract message from common field names (msg, message)
• Extract level from various field names (level, log_level, severity)

Zeek logs:

• Parse JSON network connection data
• Extract protocol, service as labels
• Store IPs and ports as structured metadata (high cardinality)

Stage 4: Finalization

• Set default level = "info" for logs without extracted level
• Drop high-cardinality labels (filename, uid)
• Move Zeek connection details to structured metadata
• Add cluster = "fzymgc-house" external label

Label Schema

Label	Source	Example Values
host	All	firewalla
job	All	firewalla-journal, docker, zeek
source	All	journal, docker, zeek, unknown
level	All	debug, info, warn, error, crit
unit	Journal	docker.service, firewalla.service
container	Docker	container name
proto	Zeek	tcp, udp, icmp
service	Zeek	http, dns, ssh

Implementation

Modify argocd/cluster-app/templates/monitoring-alloy-ingest.yaml to add the processing stages between otelcol.exporter.loki and loki.write.

Testing

1. Deploy updated config
2. Query Loki for {host="firewalla"}
3. Verify logs show plain text (not JSON wrapped)
4. Verify level label is populated
5. Verify source-specific labels appear (proto, service for Zeek)