Traefik ↔ OpenTelemetry Integration Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make Traefik emit traces, metrics, and access logs to the ClickStack OTel collector over OTLP/gRPC, then consume them in a HyperDX dashboard and a Pushover-routed alert — fully OTel-first, no Prometheus, no Vector in Traefik’s path.

Architecture: Traefik v3 (Helm chart traefik v40.0.0, ns traefik) gets tracing.otlp, metrics.otlp, and experimental.otlpLogs + logs.{general,access}.otlp pointed at cs-otel-collector.clickstack.svc.cluster.local:4317. The collector’s bearertokenauth requires the raw ingest token in an authorization header; since the chart’s otlp.grpc blocks have no headers field, the header is injected via four TRAEFIK_*_OTLP_GRPC_HEADERS_AUTHORIZATION env vars sourced from an ExternalSecret. The default Prometheus endpoint is disabled (metrics.prometheus: null) and the now-dead Prometheus Grafana dashboard is removed. Consumption is a HyperDX dashboard (MongoDB, via the tools/hyperdx flow) and a 5xx saved-search alert appended to the clickstack-alerts bootstrap Job (reusing the existing Pushover webhook).

Tech Stack: ArgoCD (multi-source Application + Helm valuesObject), Kustomize, External Secrets Operator, Traefik v3 / chart v40, ClickStack (HyperDX + ClickHouse + OTel collector), MongoDB bootstrap Job, Pushover.

Reference spec: docs/engineering/specs/2026-06-13-traefik-otel-integration-design.md

Design bead: hl-pwwf · Follow-up: hl-e4ny (vestigial-Flux cleanup: adopt orphaned cert, remove leftover Flux CRDs)

File Structure

File	Responsibility	Action
`argocd/app-configs/traefik/otel-ingest-token.yaml`	ExternalSecret projecting the OTLP ingest token into ns `traefik`	Create
`argocd/app-configs/traefik/kustomization.yaml`	Sync only the ExternalSecret (NOT the orphaned cert)	Modify
`argocd/app-configs/traefik/wildcard-cert.yaml`	Dead ArgoCD-side duplicate of an orphaned (vestigial-Flux) cert	Delete
`argocd/cluster-app/templates/traefik.yaml`	Traefik Application: add git source + OTLP `valuesObject` + env auth	Modify
`argocd/app-configs/grafana/dashboards/infrastructure/traefik.yaml`	Prometheus-based Grafana dashboard (goes empty)	Delete
`argocd/app-configs/grafana/dashboards/infrastructure/kustomization.yaml`	Drop the `traefik.yaml` resource entry	Modify
`tools/hyperdx/dashboards/traefik-ingress-health.json`	HyperDX dashboard artifact	Create (Phase 2)
`argocd/app-configs/clickstack-alerts/bootstrap-script.yaml`	Append Traefik 5xx saved-search + alert	Modify (Phase 3)

Phase boundaries: Tasks 1–6 = Phase 1 (emit), one PR, the gate. Task 7 = Phase 2 (dashboard). Tasks 8–10 = Phase 3 (alerts), gated on Task 8.

Model hints (for bead materialization, Rule 5): Tasks 1, 2, 4 = model:haiku (mechanical YAML edits); Tasks 3, 6, 8 = model:sonnet (ingress-critical config + ClickHouse verification reasoning); Tasks 5, 9 = model:sonnet; Task 7, 10 = model:sonnet (agent-browser HyperDX flow).

Phase 1 — Emit (one PR)

Task 1: ExternalSecret for the OTLP ingest token

Files:

Create: argocd/app-configs/traefik/otel-ingest-token.yaml
Step 1: Create the ExternalSecret manifest

Create argocd/app-configs/traefik/otel-ingest-token.yaml with exactly:

---
# OTLP ingest token for Traefik's OTLP exporters (traces, metrics, logs).
# cs-otel-collector's bearertokenauth/hyperdx extension requires this token on
# every OTLP push. Same Vault key as the otel-scraper + Firewalla collectors —
# rotation is a single Vault KV update + pod restart.
#
# Vault path: secret/fzymgc-house/cluster/clickstack
# Property:   otel_ingest_api_key
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: otel-ingest-token
  namespace: traefik
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: otel-ingest-token
    creationPolicy: Owner
    deletionPolicy: Delete
  data:
    - secretKey: INGEST_TOKEN
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: otel_ingest_api_key

Step 2: Lint the manifest

Run: yamllint argocd/app-configs/traefik/otel-ingest-token.yaml Expected: no errors (clean exit 0).

Step 3: Validate schema client-side

Run: kubectl --context fzymgc-house apply --dry-run=client -f argocd/app-configs/traefik/otel-ingest-token.yaml Expected: externalsecret.external-secrets.io/otel-ingest-token created (dry run).

Step 4: Commit

Commit using VCS-appropriate commands per references/vcs-preamble.md. Message: feat(traefik): add otel-ingest-token ExternalSecret for OTLP auth [hl-pwwf]

Task 2: Rewrite the traefik kustomization (exclude the orphaned cert)

Files:

Modify: argocd/app-configs/traefik/kustomization.yaml
Delete: argocd/app-configs/traefik/wildcard-cert.yaml

Context: The live wildcard-fzymgc-house-tls Certificate in ns traefik is a vestigial-Flux orphan — it carries stale kustomize.toolkit.fluxcd.io/name=infra-controllers labels from the cluster’s original (since-removed) Flux bootstrap, but Flux is no longer running (flux-system ns absent, Toolkit CRDs uninstalled); cert-manager keeps renewing it. wildcard-cert.yaml here is a dead ArgoCD-side duplicate (spec matches the live cert). When Task 3 wires this directory as an ArgoCD source, syncing the cert would make ArgoCD adopt/reconcile the cluster’s default TLS cert inside an unrelated ingress PR. So the kustomization must list only the ExternalSecret. (Adopting the orphan + removing vestigial Flux is deferred to bead hl-e4ny.)

Step 1: Replace the kustomization contents

Overwrite argocd/app-configs/traefik/kustomization.yaml with exactly:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik
resources:
  - ./otel-ingest-token.yaml

Step 2: Delete the dead cert file

Run: jj file untrack argocd/app-configs/traefik/wildcard-cert.yaml 2>/dev/null; rm argocd/app-configs/traefik/wildcard-cert.yaml (In jj, rm is sufficient — the deletion is auto-tracked into the working-copy commit.)

Step 3: Verify the kustomization builds and references no deleted file

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/traefik Expected: renders ONLY the ExternalSecret/otel-ingest-token (no Certificate); no “file not found” error for wildcard-cert.yaml.

Step 4: Commit

Message: refactor(traefik): kustomization syncs only the ExternalSecret; drop dead duplicate wildcard-cert [hl-pwwf]

Task 3: Wire the git source + OTLP valuesObject on the Traefik Application

Files:

Modify: argocd/cluster-app/templates/traefik.yaml

Context: The Application currently has two chart sources (traefik-crds, traefik) and no git source. Add a third (git) source for the kustomize path, and extend the traefik chart’s valuesObject. The current valuesObject ends with a logs: block and an additionalArguments: block — those are the anchors for the edits below.

Step 1: Add the git source

In argocd/cluster-app/templates/traefik.yaml, the spec.sources list currently ends after the traefik chart source. Append a third source as the last list item under sources: (sibling of the two - chart: entries):

    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/traefik

Step 2: Replace the logs: block with the OTLP-enabled version

Find this block in the valuesObject:

          logs:
            access:
              enabled: true

Replace it with:

          logs:
            general:
              format: json
              otlp:
                enabled: true
                grpc:
                  enabled: true
                  endpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
                  insecure: true
            access:
              enabled: true
              format: json
              fields:
                headers:
                  names:
                    Authorization: drop
              otlp:
                enabled: true
                grpc:
                  enabled: true
                  endpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
                  insecure: true

Step 3: Add tracing, metrics, experimental, and env blocks

Immediately after the additionalArguments: block (which is the last key in the current valuesObject):

          additionalArguments:
            - --api.insecure=true

append these four sibling keys (same indentation as logs: / additionalArguments:):

          experimental:
            otlpLogs: true
          tracing:
            otlp:
              enabled: true
              grpc:
                enabled: true
                endpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
                insecure: true
          metrics:
            prometheus: null
            otlp:
              enabled: true
              addEntryPointsLabels: true
              addRoutersLabels: true
              addServicesLabels: true
              grpc:
                enabled: true
                endpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
                insecure: true
          env:
            - name: TRAEFIK_TRACING_OTLP_GRPC_HEADERS_AUTHORIZATION
              valueFrom:
                secretKeyRef:
                  name: otel-ingest-token
                  key: INGEST_TOKEN
            - name: TRAEFIK_METRICS_OTLP_GRPC_HEADERS_AUTHORIZATION
              valueFrom:
                secretKeyRef:
                  name: otel-ingest-token
                  key: INGEST_TOKEN
            - name: TRAEFIK_LOG_OTLP_GRPC_HEADERS_AUTHORIZATION
              valueFrom:
                secretKeyRef:
                  name: otel-ingest-token
                  key: INGEST_TOKEN
            - name: TRAEFIK_ACCESSLOG_OTLP_GRPC_HEADERS_AUTHORIZATION
              valueFrom:
                secretKeyRef:
                  name: otel-ingest-token
                  key: INGEST_TOKEN

Step 4: Lint the file

Run: yamllint argocd/cluster-app/templates/traefik.yaml Expected: clean exit 0.

Step 5: Verify the cluster-app Helm template still renders

Run: helm template argocd/cluster-app | rg 'app-configs/traefik|otlpLogs|prometheus: null|HEADERS_AUTHORIZATION' | head Expected: matches for the git source path, otlpLogs, prometheus: null, and the auth env vars — confirming the new valuesObject and third source rendered without template errors.

Step 6: Commit

Message: feat(traefik): emit OTLP traces+metrics+logs to ClickStack; disable Prometheus [hl-pwwf]

Task 4: Remove the dead Prometheus Grafana dashboard

Files:

Delete: argocd/app-configs/grafana/dashboards/infrastructure/traefik.yaml
Modify: argocd/app-configs/grafana/dashboards/infrastructure/kustomization.yaml

Context: metrics.prometheus: null (Task 3) removes Traefik’s Prometheus endpoint, so this GrafanaDashboard (Grafana.com #17346, Prometheus datasource) goes permanently empty. HyperDX (Phase 2) becomes its replacement.

Step 1: Delete the dashboard manifest

Run: rm argocd/app-configs/grafana/dashboards/infrastructure/traefik.yaml

Step 2: Drop the resource entry from the kustomization

In argocd/app-configs/grafana/dashboards/infrastructure/kustomization.yaml, remove the line - traefik.yaml so resources: becomes:

resources:
  - alloy.yaml
  - longhorn.yaml
  - metallb.yaml
  - node-exporter.yaml

Step 3: Verify the kustomization builds without the removed file

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/grafana/dashboards/infrastructure | rg -c 'kind: GrafanaDashboard' Expected: 4 (was 5; traefik removed), and no “file not found” error.

Step 4: Commit

Message: chore(grafana): remove Prometheus-based Traefik dashboard (superseded by HyperDX) [hl-pwwf]

Task 5: Open the Phase 1 PR

Files: none (VCS only)

Step 1: Push the branch and open the PR

Push the bookmark and open a PR per references/vcs-preamble.md. PR title: feat(traefik): OTel observability integration — emit traces/metrics/logs [hl-pwwf]

PR body must include: the link to the spec, the four-env-var auth note, the metrics.prometheus: null + Grafana-dashboard-removal rationale, and the post-merge verification checklist from Task 6.

Step 2: Confirm CI / cluster-app render check passes

Run: gh pr checks <pr-number> Expected: checks pass (or no blocking checks configured).

Task 6: Post-merge deploy verification (the primary gate)

Files: none (verification only). Run AFTER the PR merges and ArgoCD syncs.

Context: This is where the env-injected-header approach is proven. A signal missing from exactly one ClickHouse table fingerprints a wrong env-var name for that OTLP root. ClickHouse access: HTTPS https://ch.fzymgc.house, user mcp_readonly (password Vault secret/fzymgc-house/cluster/clickstack#mcp_readonly_password).

Step 1: Confirm the ExternalSecret materialized

Run: kubectl --context fzymgc-house -n traefik get secret otel-ingest-token Expected: secret exists with key INGEST_TOKEN.

Step 2: Confirm ArgoCD did NOT adopt the orphaned cert

Run: kubectl --context fzymgc-house get application traefik -n argocd -o json | jq '.status.resources[] | select(.kind=="Certificate")' Expected: empty (no Certificate under ArgoCD management); the Traefik Application is Synced/Healthy.

Step 3: Confirm no auth failures in the collector

Run: kubectl --context fzymgc-house -n clickstack logs deploy/cs-otel-collector --since=10m | rg -i 'unauthenticated|missing or empty authorization' Expected: NO matches. (If matches appear, an env-var name is wrong — see Step 5.)

Step 4: Confirm all three signals land in ClickHouse

Run each query against https://ch.fzymgc.house as mcp_readonly (use the ClickHouse MCP run_query or curl):

SELECT count() FROM default.otel_traces  WHERE ServiceName = 'traefik' AND Timestamp > now() - INTERVAL 10 MINUTE;
SELECT count() FROM default.otel_logs    WHERE ServiceName = 'traefik' AND Timestamp > now() - INTERVAL 10 MINUTE;
SELECT count() FROM default.otel_metrics_sum WHERE ServiceName = 'traefik' AND TimeUnix > now() - INTERVAL 10 MINUTE LIMIT 1;

Expected: all three return > 0. A 0 in exactly one table means that root’s TRAEFIK_*_OTLP_GRPC_HEADERS_AUTHORIZATION env var name is wrong (most likely TRAEFIK_ACCESSLOG_... casing).

Step 5: If a signal is missing — diagnose the env-var name

Run: kubectl --context fzymgc-house -n traefik exec deploy/traefik -- printenv | rg TRAEFIK_.*OTLP Confirm all four vars are present and non-empty. If the access-log/log signal is the missing one, the static-config root casing may differ — check Traefik startup logs (kubectl -n traefik logs deploy/traefik | rg -i 'otlp\|accesslog\|error') for the parsed config, adjust the env-var name in traefik.yaml, and re-deploy. Record the confirmed working names in a bd note hl-pwwf.

Step 6: Confirm ingress still serves

Run: curl -sS -o /dev/null -w '%{http_code}\n' https://grafana.fzymgc.house (or any known IngressRoute host) Expected: 200/302 (routing intact — the static-config change did not break ingress).

Step 7: Resolve Open Item #1 (status field) for Phase 3

Run against ClickHouse:

SELECT LogAttributes FROM default.otel_logs WHERE ServiceName = 'traefik' LIMIT 1 FORMAT Vertical;

Record which attribute key carries the HTTP response status (candidates: DownstreamStatus, OriginStatus, status). Note it on bd note hl-pwwf — Task 9 needs it.

Phase 2 — Dashboard

Task 7: Build the HyperDX “Traefik — ingress health” dashboard

Files:

Create: tools/hyperdx/dashboards/traefik-ingress-health.json

Context: Per tools/hyperdx/README.md and .claude/skills/hyperdx-dashboards.md, dashboards are MongoDB-stored and driven via agent-browser on a logged-in HyperDX session (the dashboard API is session-cookie auth, NOT the team apiKey). Sources: Traces 6a0381ba6d27c492599151a0 (otel_traces, ServiceName), Metrics 6a0381ba6d27c492599151a2. Metric histograms support quantile (NOT literal p95); counters (sum) use aggFn: 'sum'. Build each tile in the UI and click Run to render before Save, then GET /api/dashboards and commit the resulting tile JSON.

Step 1: Create the dashboard via the agent-browser flow

Follow .claude/skills/hyperdx-dashboards.md to create a dashboard named Traefik — ingress health with these tiles (all filtered where ServiceName = 'traefik', whereLanguage: 'sql'):

Request rate by service — metric tile, source Metrics, metricName: traefik_service_requests_total, metricType: sum, aggFn: 'sum', valueExpression: 'Value', groupBy: "Attributes['service']", displayType line.
5xx rate by service — same as #1 with where "ServiceName='traefik' AND Attributes['code'] >= '500'".
Request duration p50/p90/p99 — metric tile, metricName: traefik_service_request_duration_seconds, metricType: histogram, multi-series select with three quantile entries (level 0.5/0.9/0.99, aliases p50/p90/p99).
Open connections — metric tile, metricName: traefik_open_connections, metricType: sum (gauge-style), aggFn: 'sum', groupBy: "Attributes['entrypoint']".
TLS cert expiry — metric tile, metricName: traefik_tls_certs_not_after, aggFn: 'max', valueExpression: 'Value'.

Metric names follow Traefik v3 OTLP naming; confirm exact names in HyperDX’s metric picker (Task 6 must have landed metrics first). If a name differs, use the picker’s value and note it.

Step 2: Export and commit the dashboard JSON

Run the GET /api/dashboards export per the skill, save the Traefik dashboard object to tools/hyperdx/dashboards/traefik-ingress-health.json.

Step 3: Verify tiles render data

In the HyperDX UI, confirm each tile shows non-empty data over a 1h window. Expected: request-rate and duration tiles populated (proves metrics are flowing and the queries are correct).

Step 4: Commit

Message: feat(hyperdx): add Traefik ingress-health dashboard [hl-pwwf]

Phase 3 — Alerts (gated on Task 6 Step 7)

Task 8: Confirm the status-field attribute

Files: none (verification only).

Step 1: Read the resolved status attribute

From Task 6 Step 7’s bd note, take the confirmed attribute key (e.g. LogAttributes['DownstreamStatus']). If not yet recorded, re-run the SELECT LogAttributes ... FORMAT Vertical query from Task 6 Step 7. Expected: a concrete attribute name to use in Task 9’s where clause. Do not proceed to Task 9 without it.

Task 9: Add the Traefik 5xx alert to the clickstack-alerts bootstrap

Files:

Modify: argocd/app-configs/clickstack-alerts/bootstrap-script.yaml

Context: This ConfigMap holds bootstrap.js, run by the clickstack-alerts-bootstrap PostSync Job (mongo:8.0). It upserts-by-name into db.savedsearches and db.alerts, reusing the Pushover (cluster-infra) webhook (channel variable already defined in the script). Append the new blocks just before the final print("bootstrap complete"); line.

Step 1: Append the Traefik saved-search + alert to bootstrap.js

In the data: bootstrap.js: | script, immediately before print("bootstrap complete");, insert (replace DownstreamStatus with the attribute confirmed in Task 8 if different):

    // ---- Cluster ingress alert: Traefik 5xx rate ----
    const traefik5xxSearchId = upsert(
      db.savedsearches,
      { team: teamId, name: "Traefik 5xx responses" },
      {
        team: teamId,
        name: "Traefik 5xx responses",
        select: "",
        where: "ServiceName = 'traefik' AND toInt32OrZero(LogAttributes['DownstreamStatus']) >= 500",
        whereLanguage: "sql",
        orderBy: "TimestampTime DESC",
        source: logsSource._id,
        tags: ["cluster","traefik","ingress"],
        filters: []
      }
    );
    print("savedSearch (traefik-5xx): " + traefik5xxSearchId);

    upsert(
      db.alerts,
      { team: teamId, name: "Traefik 5xx spike" },
      {
        team: teamId,
        source: "saved_search",
        savedSearch: traefik5xxSearchId,
        name: "Traefik 5xx spike",
        message: "More than {{threshold}} Traefik 5xx responses in {{interval}}. An upstream service or ingress route may be failing.",
        threshold: 25,
        thresholdType: "above",
        interval: "5m",
        channel: channel,
        state: "OK"
      }
    );

Step 2: Lint the manifest

Run: yamllint argocd/app-configs/clickstack-alerts/bootstrap-script.yaml Expected: clean exit 0.

Step 3: Sanity-check the embedded JS parses

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/clickstack-alerts | rg -c 'Traefik 5xx' Expected: >= 2 (saved-search name + alert name present in the rendered ConfigMap).

Step 4: Commit and open PR

Message: feat(clickstack-alerts): add Traefik 5xx → Pushover alert [hl-pwwf] Open a PR per references/vcs-preamble.md.

Step 5: Post-merge — confirm the bootstrap Job re-ran and created the alert

After merge + ArgoCD sync, run: kubectl --context fzymgc-house -n clickstack logs job/clickstack-alerts-bootstrap | rg 'traefik-5xx\|Traefik 5xx' Expected: savedSearch (traefik-5xx): <id> printed; Job completed successfully.

Step 6: Verify the alert exists in HyperDX

In the HyperDX UI (Alerts), confirm “Traefik 5xx spike” is listed with the Pushover channel. Expected: alert present, channel = Pushover (cluster-infra).

Task 10 (optional, Phase 3b): TLS cert-expiry alert

Files:

Modify: tools/hyperdx/dashboards/traefik-ingress-health.json (alert bound to the cert-expiry tile)

Context: Cert-expiry is a metric, so its alert must bind to a dashboard tile (source: "tile"), not a log saved-search. This is deferred/optional — implement only if metric-tile alerting is wanted.

Step 1: Add a tile-source alert on the cert-expiry tile

In the HyperDX UI, on the “TLS cert expiry” tile (Task 7), create an alert: threshold = expiry within 14 days ((traefik_tls_certs_not_after - now) < 14d), channel = Pushover (cluster-infra). Export and update tools/hyperdx/dashboards/traefik-ingress-health.json.

Step 2: Verify and commit

Confirm the alert renders in HyperDX. Commit: feat(hyperdx): add Traefik TLS cert-expiry alert [hl-pwwf]

Notes for the implementer

Phase 1 is ingress-critical. A malformed Traefik static config breaks ALL cluster routing. Always run the Task 5 render check and the Task 6 verification; the 2-replica + PDB setup makes the rollout safe to watch.
The riskiest single line is the TRAEFIK_ACCESSLOG_OTLP_GRPC_HEADERS_AUTHORIZATION env-var name (compound static-config root casing). Task 6 Step 4–5 exists specifically to catch and fix it.
Do not sync wildcard-cert.yaml — it duplicates an orphaned (vestigial-Flux) cert that has no live GitOps owner (see Task 2 context).
metrics.prometheus: null is the chart-documented way to disable the endpoint (there is no prometheus.enabled field). Confirm via the Task 3 Step 5 render that no metrics entrypoint is configured.