Traefik metrics: pure-OTLP push; disable the Prometheus endpoint
Date: 2026-06-13 Status: Accepted Decision: hl-fvje Deciders: sean@fzymgc.email
Context
Section titled “Context”Traefik’s Helm chart enables a Prometheus scrape endpoint by default. The cluster observability backend is ClickStack (HyperDX + ClickHouse) fed by cs-otel-collector, under an OTel-first directive. No ServiceMonitor scrapes Traefik today, so the chart’s Prometheus endpoint (and the Grafana dashboard #17346 that depends on it) have no live data source.
Decision
Section titled “Decision”Disable the Traefik Prometheus endpoint (metrics.prometheus: null — the chart has no prometheus.enabled field) and enable metrics.otlp.grpc push to cs-otel-collector. Decommission the now-empty Prometheus-based Grafana dashboard as a direct consequence.
Rationale
Section titled “Rationale”Cluster observability target is ClickStack, not Prometheus/Grafana; a scrape endpoint with no scraper is dead weight. OTLP push collapses traces, metrics, and logs onto one transport, one auth secret, and one destination. The Grafana dashboard was already empty (no ServiceMonitor), so removing it is no functional regression. OTel-first is an explicit operator directive.
Alternatives Considered
Section titled “Alternatives Considered”(1) Keep Prometheus + add an otel-collector prometheus receiver scrape — rejected: two pipelines for the same data, contradicts OTel-first, marginal value. (2) Keep Prometheus only, no OTLP — rejected: metrics stay siloed from traces/logs, still needs a scrape target that does not exist.
Consequences
Section titled “Consequences”POSITIVE: all three Traefik signals share one OTLP/gRPC transport and one ingest token; HyperDX becomes the single pane. NEGATIVE: re-enabling Prometheus needs both a config change and a new ServiceMonitor (not a one-liner); addRoutersLabels:true raises cardinality with route count. NEUTRAL: trace sampling defaults to 1.0, tuning deferred until otel_traces volume is observed.