Traefik ↔ OpenTelemetry Integration

Design: Maximal Traefik ↔ OpenTelemetry Integration

Bead: hl-pwwf Date: 2026-06-13 Status: Design (pre-implementation)

Problem

Traefik is the cluster’s single ingress proxy — every external and internal HTTP request flows through it — yet it is effectively dark in our observability story:

• No tracing. Traefik emits no spans, so it never appears in default.otel_traces. The request path through ingress is invisible, and backend spans (e.g. engram) have no upstream root to attach to.
• Metrics are unconsumed. The Helm chart’s default Prometheus endpoint is not scraped (no ServiceMonitor exists), so Traefik’s RED metrics (rate/errors/duration), connection counts, and TLS-cert expiry go nowhere.
• Access logs are unstructured. logs.access.enabled: true is set, but the default Common Log Format (CLF) text does not parse cleanly into ClickHouse, so access-log analysis in HyperDX is impractical.

The cluster’s observability backend is ClickStack (HyperDX + ClickHouse), fed by the cs-otel-collector OTLP receiver. The operator’s directive is OTel-first: prefer OTLP push to the collector over Prometheus scrape or new Vector configuration wherever Traefik supports it.

Goals

1. Traces — Traefik pushes a span for every request via OTLP/gRPC to cs-otel-collector, becoming the root span of cluster request flows.
2. Metrics — Traefik pushes RED + connection + TLS metrics via OTLP/gRPC; the default Prometheus endpoint is disabled (pure OTel).
3. Structured access logs — access logs pushed natively via OTLP to cs-otel-collector → otel_logs (Traefik v3’s experimental.otlpLogs), keeping all three signals on one transport. Vector is not in Traefik’s path.
4. Dashboards — a HyperDX “Traefik — ingress health” dashboard.
5. Alerts — HyperDX alert(s) → the existing Pushover webhook channel.
6. Trace propagation — Traefik propagates W3C traceparent so OTel-instrumented backends continue the trace.

Non-Goals (YAGNI)

• No Prometheus ServiceMonitor or PrometheusRule for Traefik — metrics and alerting are pure OTel/ClickStack.
• No forwarding/sidecar OTel collector — Traefik pushes directly to cs-otel-collector (the established sender convention).
• No changes to backend applications’ instrumentation.
• No new Grafana Traefik dashboard — HyperDX is the home for Traefik telemetry. The existing Prometheus-based Grafana dashboard is decommissioned in Phase 1 (it goes permanently empty once the Prometheus endpoint is disabled) — see “Decommission the Prometheus-based Grafana dashboard”.

Background & Grounding

Fact	Source
Traefik deployed via Helm chart traefik v40.0.0 (Traefik v3), ns traefik, 2 replicas + PDB	argocd/cluster-app/templates/traefik.yaml
OTLP collector: cs-otel-collector.clickstack.svc.cluster.local:4317 (gRPC); requires the raw ingest token (no Bearer) in the authorization header	memory 6f616bef; monitoring-otel-scraper/scrape-collector-values.yaml
Token at Vault fzymgc-house/cluster/clickstack#otel_ingest_api_key, projected by ESO	monitoring-otel-scraper/otel-ingest-token-externalsecret.yaml
clickstack-default CiliumNetworkPolicy already whitelists the traefik namespace with no port restriction; the traefik ns has no egress CNP	argocd/app-configs/clickstack/networkpolicy.yaml
Traefik v3 supports OTLP tracing (tracing.otlp.grpc), OTLP metrics (metrics.otlp.grpc), and OTLP access logs (logs.access.otlp.grpc, gated by experimental.otlpLogs: true); static config merges file ← env ← CLI, and arbitrary config-map keys are settable via TRAEFIK_* env vars	helm show values traefik/traefik --version 40.0.0
In chart v40, each signal needs both otlp.enabled: true and otlp.grpc.enabled: true; the otlp.grpc block has no headers field (only otlp.http does), so auth headers MUST be injected via env vars. The Prometheus endpoint has no enabled field — disable it with metrics.prometheus: null	helm show values traefik/traefik --version 40.0.0
HyperDX alerting is GitOps-as-code via a mongosh bootstrap Job (db.webhooks/db.savedsearches/db.alerts); a Pushover (cluster-infra) webhook → api.pushover.net/1/messages.json already exists	argocd/app-configs/clickstack-alerts/bootstrap-script.yaml
HyperDX dashboards are MongoDB-stored, driven via the tools/hyperdx agent-browser dashboards-as-code flow	memory c6c4c846; tools/hyperdx/
ntfy is decommissioned; notifications go to Pushover (the li alertmanager migration)	docs/reference/secrets.md; repo plan archive
The live wildcard-fzymgc-house-tls Certificate in the traefik ns is a GitOps-unowned orphan: it carries stale kustomize.toolkit.fluxcd.io/name=infra-controllers / ns flux-system labels from the cluster’s original (since-removed) Flux bootstrap, but flux-system no longer exists and the Flux GitOps-Toolkit CRDs are not installed — cert-manager simply keeps renewing it (cloudflare-acme-issuer, 373d old). argocd/app-configs/traefik/wildcard-cert.yaml is a dead ArgoCD-side duplicate whose spec matches the live cert exactly. Flux is vestigial, not running.	k8s MCP (live Certificate; flux-system ns absent; no kustomize.toolkit.fluxcd.io CRD)
A live Prometheus-based Grafana Traefik dashboard exists (Grafana.com #17346)	argocd/app-configs/grafana/dashboards/infrastructure/traefik.yaml

Architecture

                          ┌─ traces      (OTLP/gRPC) ─┐
 every request ─► Traefik ─┼─ metrics     (OTLP/gRPC) ─┼─► cs-otel-collector ─► ClickHouse
                          └─ access logs (OTLP/gRPC) ─┘     :4317              (otel_traces,
                                                       authorization: <token>  otel_metrics_*,
                                                                                otel_logs)
                                                                                  │
                          HyperDX ◄── dashboard tiles ◄── ClickHouse ◄────────────┘
                             │
                             └─ alert (threshold) ─► webhook ─► Pushover (cluster-infra)

All three signals share one transport (OTLP/gRPC → :4317) and one auth header. Vector is no longer in Traefik’s telemetry path.

Traefik already propagates W3C traceparent to backends when tracing is enabled, so instrumented services (engram, etc.) continue the trace; uninstrumented backends still receive a Traefik root span.

Emit Side

All emit-side configuration is added to the Traefik Application’s Helm valuesObject in argocd/cluster-app/templates/traefik.yaml, plus one new ExternalSecret and a new git source on the Application.

Traces

tracing:
  otlp:
    enabled: true
    grpc:
      enabled: true           # both otlp.enabled AND otlp.grpc.enabled required
      endpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
      insecure: true          # in-cluster ClusterIP; no transport security

The top-level tracing.sampleRate key (sibling of tracing.otlp, not nested under it) defaults to 1.0 (100%) and is left unset. It is the documented tuning knob — homelab traffic is low, so full fidelity is the starting point; dial it down if otel_traces volume grows (Open Item #2).

Metrics (pure OTel — Prometheus endpoint OFF)

metrics:
  prometheus: null            # OTel-first: disable the default endpoint
                              # (chart has no prometheus.enabled; null disables it)
  otlp:
    enabled: true
    addEntryPointsLabels: true
    addRoutersLabels: true    # richer per-route RED metrics (chart default: false)
    addServicesLabels: true
    grpc:
      enabled: true
      endpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
      insecure: true

addRoutersLabels: true is non-default and raises metric cardinality (one label set per IngressRoute). This is acceptable here given the bounded number of routes in a homelab; revisit if route count grows substantially.

Access logs (native OTLP — experimental.otlpLogs)

experimental:
  otlpLogs: true              # required to enable the OTLP log/access-log exporters

logs:
  general:
    format: json
    otlp:
      enabled: true
      grpc:
        enabled: true
        endpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
        insecure: true
  access:
    enabled: true
    format: json
    fields:
      headers:
        names:
          Authorization: drop   # never log auth headers
    otlp:
      enabled: true
      grpc:
        enabled: true
        endpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
        insecure: true

Access logs (and Traefik’s own general logs) push as OTLP straight to cs-otel-collector → otel_logs — the same backend a Vector scrape would reach, but on the shared OTLP transport and auth. The exporter is flagged experimental upstream; this is an accepted risk for the homelab in exchange for a single, consistent telemetry path. format: json keeps stdout readable as a kubectl logs fallback.

OTLP authentication (token stays out of Git)

cs-otel-collector requires the raw ingest token in the authorization header on every OTLP push. Traefik cannot interpolate ${env:...} inside its config headers map (unlike the OTel collector), so the header is supplied via environment variables that Traefik merges into its static configuration. The Helm values set the endpoint; pod env supplies the header:

All four OTLP exporters (one per static-config root: tracing, metrics, log, accessLog) need the header, each sourced from the same Secret key:

env:
  - name: TRAEFIK_TRACING_OTLP_GRPC_HEADERS_AUTHORIZATION
    valueFrom:
      secretKeyRef: { name: otel-ingest-token, key: INGEST_TOKEN }
  - name: TRAEFIK_METRICS_OTLP_GRPC_HEADERS_AUTHORIZATION
    valueFrom:
      secretKeyRef: { name: otel-ingest-token, key: INGEST_TOKEN }
  - name: TRAEFIK_LOG_OTLP_GRPC_HEADERS_AUTHORIZATION
    valueFrom:
      secretKeyRef: { name: otel-ingest-token, key: INGEST_TOKEN }
  - name: TRAEFIK_ACCESSLOG_OTLP_GRPC_HEADERS_AUTHORIZATION
    valueFrom:
      secretKeyRef: { name: otel-ingest-token, key: INGEST_TOKEN }

gRPC lowercases metadata keys, so ...HEADERS_AUTHORIZATION lands as authorization: <token> — exactly what the collector’s bearertokenauth/hyperdx extension expects. The token never enters Git. The exact env-var casing for the compound accessLog root (TRAEFIK_ACCESSLOG_...) is the primary deploy-time verification gate (see Risks).

ExternalSecret + Application source

argocd/app-configs/traefik/ is currently orphaned (no Application or ApplicationSet references it). Following the agent-memory multi-source pattern, a git source is added to the Traefik Application so local manifests are synced alongside the Helm chart:

# new third source on the traefik Application
- repoURL: https://github.com/fzymgc-house/selfhosted-cluster
  targetRevision: HEAD
  path: argocd/app-configs/traefik

A new ExternalSecret (clone of the otel-scraper pattern) is added to that path:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: otel-ingest-token
  namespace: traefik
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: otel-ingest-token
    creationPolicy: Owner
    deletionPolicy: Delete
  data:
    - secretKey: INGEST_TOKEN
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: otel_ingest_api_key

Critical — the kustomization must NOT sync the orphaned cert. The live wildcard-fzymgc-house-tls Certificate in the traefik namespace is a GitOps-unowned orphan from the cluster’s original (since-removed) Flux bootstrap — it carries stale kustomize.toolkit.fluxcd.io labels, but Flux is no longer running (flux-system absent, Toolkit CRDs uninstalled); cert-manager just keeps renewing it. The argocd/app-configs/traefik/wildcard-cert.yaml file is a dead ArgoCD-side duplicate (spec matches the live cert). Pulling it into the new ArgoCD source would make ArgoCD adopt/reconcile the cluster’s default TLS cert — the highest-blast-radius object in the cluster — inside an unrelated ingress PR.

Therefore app-configs/traefik/kustomization.yaml is rewritten to list only otel-ingest-token.yaml (dropping the wildcard-cert.yaml resource entry), and the dead wildcard-cert.yaml file is deleted from the repo as cleanup. ArgoCD then manages only the ExternalSecret in the traefik namespace; the orphaned cert is left untouched (cert-manager keeps renewing it). Properly adopting the orphan into ArgoCD and removing the remaining vestigial-Flux remnants is deliberately deferred to follow-up bead hl-e4ny, to keep this ingress PR focused.

Decommission the Prometheus-based Grafana dashboard

Disabling metrics.prometheus (above) makes the existing Grafana Traefik dashboard (argocd/app-configs/grafana/dashboards/infrastructure/traefik.yaml, Grafana.com

17346, a Prometheus-datasource dashboard) permanently empty. As part of Phase 1

the GrafanaDashboard resource is removed and its entry dropped from argocd/app-configs/grafana/dashboards/infrastructure/kustomization.yaml. HyperDX becomes the sole home for Traefik telemetry (the dashboard has no live scrape target today, so nothing functional is lost).

Consume Side

Dashboard

A “Traefik — ingress health” HyperDX dashboard, built via the existing tools/hyperdx dashboards-as-code (agent-browser) flow and committed as a JSON artifact under tools/hyperdx/dashboards/. Tiles drawn from the OTLP metrics/traces:

• Request rate by entrypoint / router / service
• 5xx (and 4xx) error rate
• Request-duration percentiles (p50 / p90 / p99) — histogram quantile aggFn per the HyperDX metrics-tile conventions
• Open connections
• TLS certificate expiry (cert-not-after gauge)

Alerts (GitOps via clickstack-alerts/bootstrap-script.yaml)

The clickstack-alerts mongosh bootstrap Job is the source of truth for HyperDX alerts. It is extended with Traefik saved-searches + alerts, reusing the already-provisioned Pushover (cluster-infra) webhook channel. Initial set:

• Traefik 5xx error rate — a saved_search over otel_logs (where ServiceName = 'traefik' AND <status field> >= 500), thresholdType: "above", interval: 5m. Identical mechanism to the existing firewalla / router-hosts-operator log-flow alerts.

• (Phase 3b, optional) TLS certificate expiry — a metric-tile-based alert on the cert-not-after gauge. Deferred/optional because tile-based alerts add complexity beyond the log-saved-search pattern.

Trace propagation

No application change is required for Traefik’s own spans. Correlation with backends is automatic for OTel-instrumented services via W3C traceparent propagation.

Phasing

Phase	Deliverable	Mechanism
1 — Emit	Traces + metrics + native OTLP access logs (experimental.otlpLogs) + 4 env-auth headers; metrics.prometheus: null; ExternalSecret; git source; remove the Prometheus Grafana dashboard	edit traefik.yaml; new app-configs/traefik/otel-ingest-token.yaml + kustomization rewrite (cert excluded); delete grafana/.../infrastructure/traefik.yaml
2 — Dashboard	HyperDX “Traefik — ingress health” dashboard	tools/hyperdx agent-browser flow; commit JSON artifact
3 — Alerts	Traefik 5xx alert → Pushover (3b: cert-expiry tile alert)	extend clickstack-alerts/bootstrap-script.yaml

Phase 1 is the gate; phases 2–3 are read-only consumers and follow once emission is verified. Phase 3 is additionally gated on Open Item #1 (the exact otel_logs field carrying the Traefik response status) — the 5xx saved-search where clause cannot be finalized until that field is confirmed against live Traefik logs in ClickHouse, which only exist after Phase 1 deploys.

Risks & Verification

• Blast radius (Phase 1). Phase 1 touches cluster ingress; a malformed Traefik static configuration breaks all routing. Mitigations: render-check (helm template of the cluster-app), watch the rollout (2 replicas + PDB allow safe rolling), and verify ingress still serves before/after.

• Env-injected header merge (approach A) — primary gate. Four env vars inject the authorization header (one per OTLP root: TRAEFIK_TRACING_…, TRAEFIK_METRICS_…, TRAEFIK_LOG_…, TRAEFIK_ACCESSLOG_…). The compound ACCESSLOG root casing is the riskiest of the four. Confirm post-deploy that cs-otel-collector shows no Unauthenticated drops and that rows with ServiceName = 'traefik' appear in all three ClickHouse tables (default.otel_traces, otel_metrics_*, otel_logs) — a signal missing from exactly one table fingerprints a wrong env-var name for that root. Token-leak note: a scheme/auth mismatch causes the collector to echo the presented token into the sender’s logs (which now flow via OTLP) — treat any such episode as token exposure and rotate.

• Experimental OTLP-logs exporter on ingress. experimental.otlpLogs is an upstream-flagged experimental feature; we run it on cluster ingress. Mitigation: logs.*.format: json keeps stdout readable (kubectl logs) as a fallback if the exporter misbehaves, and the rollout is watched. If the exporter proves unstable, the fallback path is to disable experimental.otlpLogs and route access logs via the Vector stdout pipeline (no app-side rework — same otel_logs destination).

• Access-log status field. Verify which otel_logs attribute carries the Traefik response status in OTLP-exported access logs (e.g. LogAttributes['DownstreamStatus']) so the Phase 3 5xx saved-search where clause is correct.

• Orphaned cert adoption (resolved in design). The traefik-ns wildcard-fzymgc-house-tls Certificate is a vestigial-Flux orphan (no live GitOps owner). The new ArgoCD source must exclude wildcard-cert.yaml (kustomization lists only otel-ingest-token.yaml; the dead file is deleted). Verify post-merge that ArgoCD reports the Traefik Application healthy/synced and does not show the Certificate as a managed or OutOfSync resource (i.e. ArgoCD did not inadvertently adopt the orphan).

• /api/v2/alerts vs Mongo bootstrap. Alerts are provisioned via the Mongo bootstrap Job (not the REST API), so REST auth is not on the critical path; the dashboard flow uses session-cookie auth via agent-browser as today.

Open Items

1. Exact otel_logs field for the Traefik response status (gates the Phase 3 5xx where clause). Resolve against live logs after Phase 1.
2. Trace-sampling rate review once real otel_traces volume is observed.
3. Follow-up bead hl-e4ny (out of scope): Flux is vestigial in this cluster — it was the original bootstrap tool, since replaced by ArgoCD (flux-system ns gone, GitOps-Toolkit CRDs uninstalled). Remnants: the orphaned wildcard cert with stale kustomize.toolkit.fluxcd.io labels, and leftover fluxcd.controlplane.io / tf-controller CRDs. hl-e4ny tracks adopting the orphaned cert into ArgoCD and removing the vestigial Flux remnants.