Inject Traefik OTLP auth headers via TRAEFIK_* env vars, not Helm values
Date: 2026-06-13 Status: Accepted Decision: hl-aek4 Deciders: sean@fzymgc.email
Context
Section titled “Context”cs-otel-collector requires the raw ingest token in an authorization header on every OTLP push. In Traefik Helm chart v40 the otlp.grpc config block has NO headers field (only otlp.http does). The token must not be committed to Git. Traefik static config supports env-var overrides for any key via the TRAEFIK_* convention.
Decision
Section titled “Decision”Inject the ingest token via four env vars — TRAEFIK_{TRACING,METRICS,LOG,ACCESSLOG}_OTLP_GRPC_HEADERS_AUTHORIZATION — each sourced from a Kubernetes Secret provisioned by an ExternalSecret (ESO -> Vault fzymgc-house/cluster/clickstack#otel_ingest_api_key).
Rationale
Section titled “Rationale”The gRPC block has no headers field, so env-var injection is the only path that keeps gRPC (the cluster’s OTLP convention) without diverging to OTLP/HTTP. ESO/Vault is the established secret-injection pattern (reused from monitoring-otel-scraper). gRPC lowercases metadata keys, so HEADERS_AUTHORIZATION lands as authorization:
Alternatives Considered
Section titled “Alternatives Considered”(1) Switch to OTLP/HTTP to use the Helm headers map — rejected: diverges from the cluster gRPC convention for all three signals. (2) Unauthenticated OTLP relying on network policy — rejected: collector rejects unauthenticated pushes; token-exposure risk.
Consequences
Section titled “Consequences”POSITIVE: token Vault-sourced, rotates via ESO without a deploy, never in Git; consistent with otel-ingest-token in monitoring-otel-scraper. NEGATIVE: four env vars must each be correct — a typo silently drops that one signal; the compound TRAEFIK_ACCESSLOG_* casing is the highest-risk unknown and the primary post-deploy verification gate; an auth mismatch echoes the token into Traefik’s own (OTLP-exported) logs -> treat as exposure and rotate.