Use the native experimental OTLP access-log exporter on cluster ingress
Date: 2026-06-13 Status: Accepted Decision: hl-5z4x Deciders: sean@fzymgc.email
Context
Section titled “Context”Traefik v3 can deliver access logs to ClickHouse two ways: (a) JSON/CLF on stdout scraped by the existing Vector DaemonSet into otel_logs, or (b) the experimental.otlpLogs flag pushing access logs via OTLP directly to cs-otel-collector. OTel-first prefers (b). The exporter is upstream-flagged experimental and Traefik is the cluster’s single ingress proxy.
Decision
Section titled “Decision”Enable experimental.otlpLogs: true and configure logs.general.otlp + logs.access.otlp to push via gRPC to cs-otel-collector. Keep logs.*.format: json on stdout as a kubectl-logs fallback. If the exporter proves unstable, disable the flag and route access logs via the Vector stdout pipeline — no app-side rework, same otel_logs destination.
Rationale
Section titled “Rationale”Putting access logs on the same OTLP transport as traces and metrics removes a separate Vector pipeline and keeps the telemetry architecture uniform (the explicit OTel-first / no-new-Vector directive). The experimental flag affects log DELIVERY only — a fault in the OTLP log exporter does not affect request routing or the traces/metrics exporters. JSON stdout is always present as a fallback, so visibility is never fully lost. Homelab risk tolerance for experimental features on a non-routing path is explicitly accepted.
Alternatives Considered
Section titled “Alternatives Considered”(1) Vector stdout scrape (CLF or JSON) — rejected as the primary path: adds Vector to Traefik’s telemetry path against the directive, needs a new Vector source/transform, CLF is hard to parse — but retained as the documented fallback.
Consequences
Section titled “Consequences”POSITIVE: single transport for all three signals, no Vector dependency for Traefik logs; structured access logs in otel_logs enable HyperDX 5xx alerting with no CLF parse. NEGATIVE: experimental upstream flag on cluster ingress, may need rollback if a Traefik release breaks the exporter; the exact otel_logs attribute carrying HTTP status is unknown until Phase 1 deploys (gates the Phase 3 5xx alert). NEUTRAL: fallback is a Helm values change reusing the same destination.