agentgateway full telemetry → ClickStack (distinct MCP/LLM usage)

agentgateway full telemetry → ClickStack

Bead: hl-rpm9.11 (design) — child of epic hl-rpm9 (openrouter-gw lane). Closes acceptance for: hl-rpm9.10 (confirm Detect-lane LLM telemetry lands in observability).

Problem

agentgateway is the cluster’s dual-purpose AI gateway — llm-gw.fzymgc.house (LLM proxy for engram, octopus, karakeep, mealie, claude-code), mcp-gw.fzymgc.house (MCP proxy for firewalla, clickhouse, engram), and the new openrouter-gw.fzymgc.house Detect lane. Despite carrying all in-cluster AI traffic, it emits zero telemetry to ClickStack.

Verified 2026-06-20 against live ClickHouse:

otel_traces (last 7d) ServiceNames: traefik, hdx-oss-api, engram, fovea — no agentgateway.
otel_metrics_sum (last 1d) ServiceNames: 34 services incl. apiserver, coredns, vault, dolt — no agentgateway.

This is not a broken transport (the classic “pull-metrics work, push-signals dark” fingerprint). Telemetry was simply never enabled:

argocd/app-configs/agentgateway/parameters.yaml (AgentgatewayParameters) configures only logging, TZ, and the :15000 admin listener — no tracing/OTLP block.
argocd/app-configs/agentgateway-controller/values-controller.yaml sets monitoring.enabled: false — no ServiceMonitor/PodMonitor.

Without telemetry, the cluster cannot answer “how much is each client spending on which model” (LLM) or “which MCP servers/tools are being called” (MCP) — and the Detect-over-Passthrough choice (ADR hl-l875), whose entire justification is preserved OTel telemetry, is unvalidated.

Goals

Land agentgateway metrics in ClickStack, separating LLM usage (gen_ai_*) from MCP usage (mcp_requests).
Land agentgateway traces in ClickStack (per-request spans: model, latency, status).
Provide a HyperDX dashboard reporting LLM and MCP usage distinctly.
Satisfy hl-rpm9.10’s acceptance: provider/model/token telemetry from the Detect lane is observable.

Non-goals (YAGNI)

No alerts in this iteration (dashboard-only; alerting is a follow-up using the clickstack-alerts bootstrap pattern if desired).
No trace-sampling cap — run full fidelity (homelab volume is low).
No kube-prometheus-stack ServiceMonitor for agentgateway — see Decision 1.
No NetworkPolicy changes — see Decision 3.

Grounding (Rule 7)

deepwiki agentgateway/agentgateway — telemetry model: traces via OTLP push (rawConfig.config.tracing.otlpEndpoint/otlpProtocol, or OTEL_EXPORTER_OTLP_* env); metrics via Prometheus /metrics (pull) only. Metric families: LLM gen_ai_token_usage /gen_ai_request_duration /gen_ai_time_to_first_token; MCP mcp_requests.
deepwiki — agentgateway’s Rust data plane has no OTLP metrics exporter, no OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, no metrics rawConfig. Metrics are prometheus_client-crate, pull-only. (The OTel Go metric modules in go.sum belong to the Go controller, not the Rust data plane.) This forces the scrape lane for metrics.
deepwiki — AgentgatewayParameters.spec.env is corev1.EnvVar, so it supports valueFrom.secretKeyRef. The data plane reads OTLP_HEADERS (comma-separated key=value or JSON) via parse_otlp_headers. rawConfig header values do not do env-substitution — auth must ride OTLP_HEADERS.
Live ClickHouse — confirmed agentgateway absent from traces + metrics.
Live k8s — agentgateway-admin ClusterIP :15000 exists (metrics target).
Repo — monitoring-otel-scraper/scrape-collector-values.yaml uses static scrape targets (20 jobs) → OTLP push to cs-otel-collector:4317. The otel-ingest-token ExternalSecret reads Vault fzymgc-house/cluster/clickstack#otel_ingest_api_key.
Repo — clickstack CiliumNetworkPolicy now opens :4317/:4318 to fromEntities: [all] (hl-pwwf); auth is the ingest token (raw, no Bearer). Per-namespace whitelists are documentation-only.
Repo — HyperDX dashboards are MongoDB-stored, session-cookie API only; no ArgoCD/Vault headless path. Source of truth = tools/hyperdx/dashboards/*.json applied via the agent-browser/devtools session workflow. Metrics source id 6a0381ba6d27c492599151a2; histogram tiles use aggFn: quantile + level + alias.

Architecture

                  ┌──────────── agentgateway pod (ns: agentgateway) ────────────┐
                  │  data plane (Rust)                                           │
  LLM + MCP   ───▶│   • Prometheus /metrics on admin :15000 (gen_ai_*, mcp_*)   │
  clients         │   • OTLP traces (push) ───────────────────────┐             │
                  └────────────────────────────────────────────────┼────────────┘
                                                                    │
   metrics: PULL                                      traces: PUSH  │ raw-token auth
        │                                                           ▼
        ▼                                            cs-otel-collector.clickstack:4317
  monitoring-otel-scraper (otelcol-k8s)                             │
   • static job "agentgateway" → :15000/metrics                    ▼
   • PUSH OTLP ─────────────────────────────────────▶  ClickHouse otel_metrics_* / otel_traces
                                                                    │
                                          HyperDX dashboard ◀───────┘  (LLM vs MCP tiles)

Two intake paths because agentgateway offers two surfaces and only one (traces) supports OTLP push. Both converge on ClickStack as OTLP — the Prometheus hop is the only metrics intake agentgateway exposes, transcoded to OTLP by the scrape collector (identical to how vault/coredns/apiserver reach ClickHouse).

Decisions

Metrics via the static-scrape collector, not the chart’s monitoring.enabled. Enabling the chart ServiceMonitor would surface metrics only in kube-prometheus-stack/Grafana; kps does not forward app metrics to ClickStack, so the data would never reach ClickHouse. The monitoring-otel-scraper is the cluster’s only Prometheus→OTLP→ClickStack bridge and already lands 20 targets this way.
Metrics are scraped, traces are pushed — asymmetry forced by upstream. agentgateway has no OTLP metrics exporter (grounded). If/when it gains one, collapse both lanes into the single telemetry config block and retire the scrape job (tracked as a follow-up bead).
No NetworkPolicy change for traces. clickstack-default already opens :4317 to all senders; the ingest token gates access. (Metrics: verify the agentgateway namespace has no ingress CNP blocking otel-scraper → :15000; if one exists, add an allow rule — expected to be a no-op.)
Auth header via OTLP_HEADERS env, raw token (no Bearer). rawConfig headers can’t reference env vars, and cs-otel-collector rejects the Bearer scheme. The token is injected from a new ExternalSecret.

Components

A. Metrics lane — `monitoring-otel-scraper/scrape-collector-values.yaml`

Append one scrape job under receivers.prometheus/scrape.config.scrape_configs:

# agentgateway — LLM (gen_ai_*) + MCP (mcp_requests) usage metrics.
# Pull-only emitter (no OTLP metrics exporter upstream); the collector
# transcodes to OTLP → ClickStack. Lands as ServiceName='agentgateway'.
- job_name: agentgateway
  scrape_interval: 30s
  static_configs:
    - targets: ["agentgateway-admin.agentgateway.svc.cluster.local:15000"]

Lands under ServiceName='agentgateway':

LLM: gen_ai_token_usage{gen_ai_system, gen_ai_request_model, gen_ai_response_model, gen_ai_token_type} (sum → otel_metrics_sum), gen_ai_request_duration (histogram → otel_metrics_histogram), gen_ai_time_to_first_token.
MCP: mcp_requests{method, server, resource, route} (sum).

B. Traces lane — `agentgateway/parameters.yaml`

Add OTLP tracing config + token env to the existing AgentgatewayParameters:

spec:
  env:
    - { name: TZ, value: "America/Los_Angeles" }
    - { name: ADMIN_ADDR, value: "0.0.0.0:15000" }
    # OTLP ingest token for cs-otel-collector (raw token, NOT "Bearer").
    - name: OTLP_AUTH_TOKEN
      valueFrom:
        secretKeyRef:
          name: agentgateway-otel-ingest
          key: INGEST_TOKEN
    - { name: OTLP_HEADERS, value: "authorization=$(OTLP_AUTH_TOKEN)" }
  rawConfig:
    # NOTE: `$(OTLP_AUTH_TOKEN)` is Kubernetes downward env-var interpolation
    # (the referenced var must be declared earlier in the same list — it is).
    # This works but is mildly discouraged upstream. Alternative if preferred:
    # have the ExternalSecret template the full header string
    # (`authorization=<token>`) into one key and point `OTLP_HEADERS` at it via
    # `valueFrom.secretKeyRef` directly, dropping the two-step interpolation.
    config:
      tracing:
        otlpEndpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
        otlpProtocol: grpc
        randomSampling: true   # full fidelity; low homelab volume

C. ExternalSecret — `agentgateway/otel-ingest-token-externalsecret.yaml` (new)

Mirror the otel-scraper’s secret into the agentgateway namespace:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: agentgateway-otel-ingest
  namespace: agentgateway
spec:
  refreshInterval: 1h
  secretStoreRef: { name: vault, kind: ClusterSecretStore }
  target: { name: agentgateway-otel-ingest, creationPolicy: Owner }
  data:
    - secretKey: INGEST_TOKEN
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: otel_ingest_api_key

Add to agentgateway/kustomization.yaml. No new Vault path or policy: the cluster’s ClusterSecretStore (vault) uses a single ESO Vault role with a broad secret/data/* policy that already covers all fzymgc-house/cluster/* paths — there is no per-namespace role to extend. Verification reduces to confirming the ExternalSecret reaches SecretSynced=True.

D. Dashboard — `tools/hyperdx/dashboards/agentgateway-llm-mcp-usage.json` (new)

Version-controlled definition {name, tiles}, applied via the documented agent-browser/devtools import workflow against the logged-in hyperdx.fzymgc.house session (Metrics source 6a0381ba6d27c492599151a2).

Tiles:

LLM — tokens by provider+model (stacked sum of gen_ai_token_usage grouped by gen_ai_system/gen_ai_request_model/gen_ai_token_type); request rate by model; gen_ai_request_duration p95 (quantile, level: 0.95); TTFT p95.
MCP — calls by server+method (sum of mcp_requests grouped by server/method); tool-call rate by server.

Data flow

A client calls llm-gw/mcp-gw/openrouter-gw. The data plane records Prometheus metrics (gen_ai_* or mcp_requests) and, if tracing is enabled, emits an OTLP span.
Metrics: monitoring-otel-scraper scrapes :15000/metrics every 30s, enriches with k8s attributes, batches, and pushes OTLP/gRPC to cs-otel-collector:4317 → otel_metrics_{sum,histogram}.
Traces: the data plane pushes spans directly to cs-otel-collector:4317 with the raw ingest token → otel_traces.
HyperDX reads both from ClickHouse; the dashboard renders LLM/MCP tiles.

Error handling & risks

Wrong /metrics path or port. Mitigation: verify :15000/metrics returns Prometheus text at implementation (live curl/port-forward) before merge.
rawConfig.tracing clobbers generated config. Mitigation: agentgateway server dry-run + confirm the merged config still renders all listeners/routes.
Bearer vs raw token. Encoded as raw in OTLP_HEADERS; a Bearer prefix would 401 silently and drop all spans.
ExternalSecret fails to sync. Low risk — the shared ClusterSecretStore role already reads all fzymgc-house/cluster/* paths. Mitigation: confirm the ExternalSecret reaches SecretSynced=True after sync.
Admin-port exposure. :15000 is ClusterIP-only and already used for the UI; scraping it adds no external surface.
Dashboard drift. HyperDX dashboards aren’t GitOps; the JSON file is the restore source of truth and must be re-exported after UI edits.

Verification (acceptance)

After ArgoCD sync:

-- metrics: both families present under agentgateway
SELECT MetricName, count() FROM default.otel_metrics_sum
WHERE ServiceName = 'agentgateway' AND TimeUnix > now() - INTERVAL 1 HOUR
  AND (MetricName LIKE 'gen_ai%' OR MetricName LIKE 'mcp%')
GROUP BY MetricName;

-- metrics: histogram families (gen_ai_request_duration, _time_to_first_token)
SELECT MetricName, count() FROM default.otel_metrics_histogram
WHERE ServiceName = 'agentgateway' AND TimeUnix > now() - INTERVAL 1 HOUR
  AND MetricName LIKE 'gen_ai%'
GROUP BY MetricName;

-- traces: spans landing
SELECT count(), max(Timestamp) FROM default.otel_traces
WHERE ServiceName = 'agentgateway' AND Timestamp > now() - INTERVAL 1 HOUR;

Acceptance = non-zero gen_ai_token_usage and mcp_requests rows + spans in otel_traces. This is exactly the confirmation hl-rpm9.10 requires; closing this validates ADR hl-l875.

Follow-ups

Collapse metrics+traces into one telemetry block + retire the scrape job if agentgateway adds an OTLP metrics exporter upstream.
Optional Pushover alerts (error-rate spike, token-spend threshold) via clickstack-alerts.
Document the agentgateway telemetry surfaces in docs/operations/clickstack.md and docs/reference/services.md.

agentgateway full telemetry → ClickStack (distinct MCP/LLM usage)