agentgateway Telemetry Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Wire agentgateway’s metrics (Prometheus scrape) and traces (OTLP push) into ClickStack and add a HyperDX dashboard that reports distinct LLM (gen_ai_*) vs MCP (mcp_requests) usage.

Architecture: Two intake paths converging on ClickStack. Metrics are scraped from the admin :15000/metrics endpoint by the existing monitoring-otel-scraper (one new static job) and transcoded to OTLP. Traces are pushed natively by the agentgateway data plane to cs-otel-collector:4317 (config + token env on the AgentgatewayParameters). No NetworkPolicy change — :4317 is already open to all senders, token-gated.

Tech Stack: Kubernetes (GitOps via ArgoCD), agentgateway v1.3.0-alpha.1 CRDs (agentgateway.dev/v1alpha1), OpenTelemetry Collector (otelcol-k8s), External Secrets Operator, ClickHouse/HyperDX (ClickStack), jj VCS.

Spec: docs/engineering/specs/2026-06-20-agentgateway-telemetry-design.md Design bead: hl-rpm9.11 · Acceptance target: hl-rpm9.10

File Structure

File	Action	Responsibility
`argocd/app-configs/agentgateway/otel-ingest-token-externalsecret.yaml`	Create	ESO secret holding the OTLP ingest token for the traces lane
`argocd/app-configs/agentgateway/kustomization.yaml`	Modify	Register the new ExternalSecret
`argocd/app-configs/agentgateway/parameters.yaml`	Modify	Enable OTLP tracing + inject the ingest token via env
`argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml`	Modify	Add the `agentgateway` static scrape job (metrics lane)
`docs/reference/services.md`	Modify	Document the telemetry surfaces in the agentgateway section
`docs/operations/clickstack.md`	Modify	Add agentgateway as an OTLP/scrape sender
`tools/hyperdx/dashboards/agentgateway-llm-mcp-usage.json`	Create	Version-controlled dashboard definition (LLM + MCP tiles)

Sequencing: Tasks 1–4 are the PR content (manifests + docs). Tasks 5–6 are post-merge, gated on ArgoCD sync — Task 5 verifies telemetry flows (closes hl-rpm9.10), Task 6 authors the dashboard against the now-live metrics.

Task 1: ExternalSecret for the OTLP ingest token (traces auth)

Files:

Create: argocd/app-configs/agentgateway/otel-ingest-token-externalsecret.yaml
Modify: argocd/app-configs/agentgateway/kustomization.yaml:5-23
Step 1: Create the ExternalSecret

Create argocd/app-configs/agentgateway/otel-ingest-token-externalsecret.yaml:

---
# OTLP ingest token (HyperDX team API key) for agentgateway's OTLP trace
# exporter. Reuses the same Vault key the otel-scraper collectors use
# (fzymgc-house/cluster/clickstack#otel_ingest_api_key). cs-otel-collector
# validates this as the `authorization` header (RAW token, no Bearer scheme).
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: agentgateway-otel-ingest
  namespace: agentgateway
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: agentgateway-otel-ingest
    creationPolicy: Owner
  data:
    - secretKey: INGEST_TOKEN
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: otel_ingest_api_key

Step 2: Register it in the kustomization

In argocd/app-configs/agentgateway/kustomization.yaml, add the new file to the resources list (place it after secrets.yaml so secret-type resources group):

resources:
  - namespace.yaml
  - certificate.yaml
  - secrets.yaml
  - otel-ingest-token-externalsecret.yaml
  - parameters.yaml
  - gateway.yaml
  - readiness-service.yaml
  - llm-backends.yaml
  - llm-routes.yaml
  - llm-policies.yaml
  - openrouter-generic.yaml
  - mcp-engram.yaml
  - mcp-saas.yaml
  - mcp-clickhouse.yaml
  - mcp-firewalla.yaml
  - modern-auth.yaml
  - ui-ingress.yaml
  - hostmapping.yaml

Step 3: Validate lint + kustomize build

Run: yamllint -c .yamllint.yaml argocd/app-configs/agentgateway/otel-ingest-token-externalsecret.yaml argocd/app-configs/agentgateway/kustomization.yaml Expected: no output (exit 0).

Run: kubectl kustomize argocd/app-configs/agentgateway | grep -A3 'name: agentgateway-otel-ingest' Expected: the ExternalSecret/agentgateway-otel-ingest renders in the built output.

Step 4: Commit

Run: jj commit -m "feat(agentgateway): ExternalSecret for OTLP ingest token [hl-rpm9.11]" (Colocated jj repo — jj commit describes @ and opens the next empty change. The guard hook blocks mutating git.)

Task 2: Enable OTLP traces in AgentgatewayParameters

Files:

Modify: argocd/app-configs/agentgateway/parameters.yaml:19-23
Step 1: Add the token env vars and the tracing rawConfig

Replace the env: block (and append a rawConfig: block) in argocd/app-configs/agentgateway/parameters.yaml. The full spec becomes:

spec:
  logging:
    level: info
    format: json
  image:
    registry: cr.agentgateway.dev
    repository: agentgateway
    tag: "v1.3.0-alpha.1"
    pullPolicy: IfNotPresent
  env:
    - name: TZ
      value: "America/Los_Angeles"
    - name: ADMIN_ADDR          # expose admin/UI listener (default loopback-only)
      value: "0.0.0.0:15000"
    # OTLP ingest token for cs-otel-collector. RAW token (no "Bearer"). The data
    # plane reads OTLP_HEADERS (parse_otlp_headers); $(OTLP_AUTH_TOKEN) is k8s
    # downward env interpolation of the sibling var declared just above.
    - name: OTLP_AUTH_TOKEN
      valueFrom:
        secretKeyRef:
          name: agentgateway-otel-ingest
          key: INGEST_TOKEN
    - name: OTLP_HEADERS
      value: "authorization=$(OTLP_AUTH_TOKEN)"
  rawConfig:
    config:
      tracing:
        otlpEndpoint: cs-otel-collector.clickstack.svc.cluster.local:4317
        otlpProtocol: grpc
        randomSampling: true    # full fidelity; low homelab volume
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: "1"
      memory: 512Mi

Step 2: Validate lint + kustomize build

Run: yamllint -c .yamllint.yaml argocd/app-configs/agentgateway/parameters.yaml Expected: no output (exit 0).

Run: kubectl kustomize argocd/app-configs/agentgateway | grep -A2 otlpEndpoint Expected: the tracing.otlpEndpoint: cs-otel-collector...:4317 renders in the AgentgatewayParameters output.

Note: the agentgateway controller validates rawConfig at reconcile time (it merges into the generated data-plane config). A schema error would surface as a controller-rejected AgentgatewayParameters and a non-progressing Deployment — checked live in Task 5, not catchable by kubectl kustomize.

Step 3: Commit

Run: jj commit -m "feat(agentgateway): enable OTLP trace export to ClickStack [hl-rpm9.11]"

Task 3: Add the agentgateway metrics scrape job

Files:

Modify: argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml:262-266
Step 1: Append the scrape job

In argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml, under receivers.prometheus/scrape.config.scrape_configs, add a new job. Insert it immediately after the node-exporter job (around line 266), before the nats/nats job:

          # agentgateway — LLM (gen_ai_*) + MCP (mcp_requests) usage metrics.
          # Pull-only emitter: the Rust data plane has no OTLP metrics exporter,
          # so we scrape /metrics on the admin endpoint and the collector
          # transcodes to OTLP → ClickStack. Lands as ServiceName='agentgateway'.
          - job_name: agentgateway
            scrape_interval: 30s
            static_configs:
              - targets: ["agentgateway-admin.agentgateway.svc.cluster.local:15000"]

Step 2: Validate lint

Run: yamllint -c .yamllint.yaml argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml Expected: no output (exit 0).

Note: this is a Helm values file consumed by the OTel collector chart at deploy time. A malformed scrape config surfaces as a collector pod CrashLoopBackOff (config parse error) after ArgoCD sync — verified live in Task 5. There is no pre-merge helm template gate in this repo for it.

Step 3: Commit

Run: jj commit -m "feat(monitoring-otel-scraper): scrape agentgateway metrics [hl-rpm9.11]"

Task 4: Document the telemetry surfaces

Files:

Modify: docs/reference/services.md (agentgateway section, ~line 279)
Modify: docs/operations/clickstack.md (senders, ~line 30/124)
Step 1: Add an Observability note to services.md

In docs/reference/services.md, in the agentgateway section, add a paragraph after the “Deployment model:” paragraph (~line 281):

**Observability:** agentgateway exports telemetry to ClickStack via two paths
(its Rust data plane has no OTLP metrics exporter, so the surfaces are
asymmetric). **Metrics** (`gen_ai_token_usage`, `gen_ai_request_duration`,
`gen_ai_time_to_first_token` for LLM; `mcp_requests` for MCP) are exposed at the
admin `:15000/metrics` endpoint and scraped by `monitoring-otel-scraper` (static
job `agentgateway` → `cs-otel-collector` → `otel_metrics_{sum,histogram}`,
`ServiceName='agentgateway'`). **Traces** are pushed natively (OTLP/gRPC to
`cs-otel-collector:4317`, configured via `AgentgatewayParameters.rawConfig...tracing`,
authed with the `otel_ingest_api_key` raw token). The HyperDX dashboard
`agentgateway-llm-mcp-usage` reports LLM-vs-MCP usage distinctly.

Step 2: Add agentgateway as a sender in clickstack.md

In docs/operations/clickstack.md, in the senders/Traces-Ingestion area (the section that lists which apps push OTLP, ~line 124), add agentgateway to the sender list. Add this bullet under the trace-producers enumeration:

- **agentgateway** — traces via native OTLP/gRPC push (`AgentgatewayParameters`
  `rawConfig.config.tracing`); metrics via `monitoring-otel-scraper` scrape of
  `:15000/metrics` (no OTLP metrics exporter upstream). `ServiceName='agentgateway'`.

Step 3: Validate docs lint

Run: rumdl check docs/reference/services.md docs/operations/clickstack.md Expected: no errors (exit 0). Fix any MD rule violations rumdl reports.

Step 4: Commit

Run: jj commit -m "docs(agentgateway): document telemetry surfaces [hl-rpm9.11]"

Task 5: Post-merge live verification (closes hl-rpm9.10)

Gated on: PR merged → ArgoCD synced agentgateway + monitoring-otel-scraper → Reloader/controller restarted the agentgateway pod (picks up new env + rawConfig).

Files: none (verification only).

Step 1: Confirm the agentgateway pod restarted with tracing config

Run: kubectl -n agentgateway get pods -l app.kubernetes.io/name=agentgateway Expected: a single Running pod with a recent age (post-sync restart).

Run: kubectl -n agentgateway logs -l app.kubernetes.io/name=agentgateway --tail=200 | grep -iE 'otlp|tracing|export' | head (label selector, not deploy/... — the data-plane Deployment is controller-named and may carry a release prefix) Expected: no repeating dial tcp ...:4317 / export-error lines. (An auth/scheme error — e.g. a stray Bearer — would show as silent span drops; if so, recheck OTLP_HEADERS is a raw token.)

Step 2: Confirm the scrape collector is healthy (metrics lane)

Run: kubectl -n otel-scraper get pods Expected: otel-scrape-... Deployment pod Running (a bad scrape config would CrashLoopBackOff).

Step 3: Verify metrics landed (the hl-rpm9.10 acceptance)

Run the ClickHouse query (via mcp__clickhouse-ro__run_query or clickhouse-client at ch.fzymgc.house, user mcp_readonly):

SELECT MetricName, count() FROM default.otel_metrics_sum
WHERE ServiceName = 'agentgateway' AND TimeUnix > now() - INTERVAL 1 HOUR
  AND (MetricName LIKE 'gen_ai%' OR MetricName LIKE 'mcp%')
GROUP BY MetricName;

Expected: non-zero rows for at least gen_ai_token_usage and mcp_requests (generate traffic first if idle — e.g. a curl LLM chat completion through llm-gw and an MCP tools/list through mcp-gw).

SELECT MetricName, count() FROM default.otel_metrics_histogram
WHERE ServiceName = 'agentgateway' AND TimeUnix > now() - INTERVAL 1 HOUR
  AND MetricName LIKE 'gen_ai%'
GROUP BY MetricName;

Expected: non-zero rows for gen_ai_request_duration (and/or gen_ai_time_to_first_token).

Step 4: Verify traces landed

SELECT count(), max(Timestamp) FROM default.otel_traces
WHERE ServiceName = 'agentgateway' AND Timestamp > now() - INTERVAL 1 HOUR;

Expected: non-zero span count with a recent max(Timestamp).

Step 5: Record the exact metric names + label keys for Task 6

From the Step 3/4 output, note the exact MetricName strings (the Prometheus→OTLP transcoding may append _total or alter separators) and run:

SELECT DISTINCT arrayJoin(mapKeys(Attributes)) AS label
FROM default.otel_metrics_sum
WHERE ServiceName = 'agentgateway' AND MetricName LIKE 'gen_ai%'
  AND TimeUnix > now() - INTERVAL 1 HOUR;

Expected: the LLM label keys (gen_ai_system, gen_ai_request_model, gen_ai_token_type, …). Repeat with MetricName LIKE 'mcp%' for MCP labels (server, method, …). Save this output — Task 6 tiles reference these exact names.

Step 6: Close the acceptance bead

Run: bd close hl-rpm9.10 --reason="agentgateway gen_ai_* + mcp_requests metrics and traces confirmed in ClickStack; validates ADR hl-l875"

Task 6: Author and commit the HyperDX dashboard

Gated on: Task 5 (metrics must be flowing so tiles are authored against live data, and the exact MetricName/label strings from Task 5 Step 5 are known).

Files:

Create: tools/hyperdx/dashboards/agentgateway-llm-mcp-usage.json
Step 1: Author the dashboard JSON

Create tools/hyperdx/dashboards/agentgateway-llm-mcp-usage.json. Use the Metrics source id 6a0381ba6d27c492599151a2. Counters use aggFn: "sum"; histograms use aggFn: "quantile" + level + alias (per tools/hyperdx/README.md). Substitute the exact MetricName / label strings recorded in Task 5 Step 5 (shown here with the expected names):

{
  "name": "agentgateway — LLM + MCP usage",
  "tiles": [
    {
      "name": "LLM tokens by provider + model",
      "config": {
        "displayType": "stacked_bar",
        "source": "6a0381ba6d27c492599151a2",
        "select": [{ "aggFn": "sum", "valueExpression": "Value",
                     "metricType": "sum", "metricName": "gen_ai_token_usage" }],
        "where": "ServiceName = 'agentgateway'", "whereLanguage": "sql",
        "groupBy": "Attributes['gen_ai_request_model']"
      }
    },
    {
      "name": "LLM tokens by type (input/output)",
      "config": {
        "displayType": "stacked_bar",
        "source": "6a0381ba6d27c492599151a2",
        "select": [{ "aggFn": "sum", "valueExpression": "Value",
                     "metricType": "sum", "metricName": "gen_ai_token_usage" }],
        "where": "ServiceName = 'agentgateway'", "whereLanguage": "sql",
        "groupBy": "Attributes['gen_ai_token_type']"
      }
    },
    {
      "name": "LLM request latency p95",
      "config": {
        "displayType": "line",
        "source": "6a0381ba6d27c492599151a2",
        "select": [{ "aggFn": "quantile", "level": 0.95, "valueExpression": "Value",
                     "metricType": "histogram", "metricName": "gen_ai_request_duration",
                     "alias": "p95" }],
        "where": "ServiceName = 'agentgateway'", "whereLanguage": "sql"
      }
    },
    {
      "name": "MCP calls by server + method",
      "config": {
        "displayType": "stacked_bar",
        "source": "6a0381ba6d27c492599151a2",
        "select": [{ "aggFn": "sum", "valueExpression": "Value",
                     "metricType": "sum", "metricName": "mcp_requests" }],
        "where": "ServiceName = 'agentgateway'", "whereLanguage": "sql",
        "groupBy": "Attributes['server']"
      }
    },
    {
      "name": "MCP calls by method",
      "config": {
        "displayType": "stacked_bar",
        "source": "6a0381ba6d27c492599151a2",
        "select": [{ "aggFn": "sum", "valueExpression": "Value",
                     "metricType": "sum", "metricName": "mcp_requests" }],
        "where": "ServiceName = 'agentgateway'", "whereLanguage": "sql",
        "groupBy": "Attributes['method']"
      }
    }
  ]
}

If Task 5 Step 5 showed a different MetricName (e.g. a _total suffix on the counters) or different label keys, update metricName / groupBy accordingly before saving — the live names are authoritative.

Step 2: Validate JSON

Run: jq empty tools/hyperdx/dashboards/agentgateway-llm-mcp-usage.json Expected: no output (exit 0 = valid JSON).

Step 3: Apply to HyperDX via the logged-in session

Per tools/hyperdx/README.md import workflow, against a logged-in hyperdx.fzymgc.house session (devtools console or agent-browser eval):

const DEF = /* contents of agentgateway-llm-mcp-usage.json */;
const created = await (await fetch('/api/dashboards', {
  method:'POST', headers:{'Content-Type':'application/json'},
  body: JSON.stringify({name: DEF.name, tiles: []})
})).json();
await fetch('/api/dashboards/' + (created._id||created.id), {
  method:'PATCH', headers:{'Content-Type':'application/json'},
  body: JSON.stringify({tiles: DEF.tiles})
});

Step 4: Verify tiles render, then re-export as source of truth

In the HyperDX UI, open the dashboard and confirm each tile renders non-empty (click Run on each). If a tile errors or is blank, fix the metricName/groupBy against the live names, re-apply, then re-export {name, tiles} over the JSON file (the rendered config is authoritative).

Step 5: Commit

Run: jj commit -m "feat(hyperdx): agentgateway LLM+MCP usage dashboard [hl-rpm9.11]"

Notes for the implementer

VCS: colocated jj repo — use jj commands (the guard hook blocks mutating git). Work on bookmark feat/agentgateway-telemetry. jj commit -m per task; jj git push -b feat/agentgateway-telemetry when ready for the PR.
Do not kubectl apply — ArgoCD owns deployment. Tasks 1–4 take effect only after merge + sync; Tasks 5–6 run against the live cluster post-sync.
Raw token, not Bearer: the single most likely failure is an authorization: Bearer <token> header — cs-otel-collector wants the raw token. Keep OTLP_HEADERS as authorization=<token>.
Idempotency: the metrics/traces config is declarative; re-sync is safe.