Scrape agentgateway metrics via otel-scraper; push traces directly
Date: 2026-06-22 Status: Accepted Decision: hl-1pwl Deciders: Sean Brandt
Context
Section titled “Context”agentgateway is the cluster’s unified AI gateway for LLM and MCP traffic. Its Rust data plane emits metrics only via a Prometheus pull endpoint on the admin listener (:15000) — it has no OTLP metrics exporter, no OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, and no metrics rawConfig block. Traces, by contrast, are pushed natively via OTLP. This asymmetric telemetry surface must be reconciled so all signals land in ClickStack (ClickHouse/HyperDX), the cluster’s observability target.
Decision
Section titled “Decision”Ingest agentgateway metrics via a static scrape job in monitoring-otel-scraper (Prometheus pull → OTLP push to cs-otel-collector → ClickHouse). Push traces directly from the data plane to cs-otel-collector via OTLP/gRPC, authenticated with a raw ingest token injected through the OTLP_HEADERS env var.
Rationale
Section titled “Rationale”- agentgateway’s Rust data plane is pull-only for metrics; OTLP metrics push is architecturally unavailable upstream — a forced constraint, not a preference.
- The chart’s monitoring.enabled (ServiceMonitor) path terminates at kube-prometheus-stack/Grafana and never forwards app metrics to ClickHouse; it is incompatible with the OTel-first observability target.
- monitoring-otel-scraper is the established Prometheus→OTLP→ClickStack bridge (20 existing static jobs); adding a 21st is consistent and avoids standing up a new pipeline.
- Trace auth must ride OTLP_HEADERS as a raw token (no Bearer prefix): rawConfig header values do not perform env-var substitution, and cs-otel-collector rejects the Bearer scheme.
- The asymmetry is explicitly tracked: if agentgateway gains an OTLP metrics exporter upstream, collapse both lanes into the rawConfig tracing block and retire the scrape job.
Alternatives Considered
Section titled “Alternatives Considered”- Scrape via monitoring-otel-scraper + push traces (chosen): the only path that lands both signals in ClickHouse; reuses the established bridge under ServiceName=‘agentgateway’.
- Enable chart ServiceMonitor / monitoring.enabled (rejected): routes metrics only to kps/Grafana; never reaches ClickStack.
- OTLP push for both metrics and traces (rejected — impossible): the Rust data plane has no OTLP metrics exporter; the OTel Go metric modules in go.sum belong to the Go controller, not the data plane.
Consequences
Section titled “Consequences”- Positive: gen_ai_* and mcp_requests metrics reach ClickHouse (LLM and MCP usage become queryable and dashboardable); traces land under ServiceName=‘agentgateway’, validating ADR hl-l875; no new Vault paths or ClusterSecretStore policy changes (the ExternalSecret mirrors an existing key).
- Negative: two intake paths (scrape + push) for a single service — contributors must understand the asymmetry and its upstream cause; the scrape job becomes dead weight if/when upstream adds OTLP metrics, until explicitly retired.
- Neutral: 30s static scrape target, identical operational posture to the 20 existing monitoring-otel-scraper jobs; clickstack-default CNP already opens :4317 to all senders, so no new egress rules.