Collect Keycloak metrics via OTel-native OTLP push, not Prometheus scrape
Date: 2026-06-30 Status: Accepted Decision: hl-hpbv Deciders: Sean Brandt, Claude
Context
Section titled “Context”Keycloak 26.6.4 exposes Prometheus-format metrics at the management :9000/metrics endpoint; after kps removal a static OTel scrape job could replace the old ServiceMonitor. Keycloak 26+ also provides an experimental opentelemetry-metrics feature (Micrometer-to-OTel bridge) that pushes metrics natively via OTLP gRPC, eliminating the scrape path entirely.
Decision
Section titled “Decision”Enable the Keycloak opentelemetry-metrics experimental feature with OTLP gRPC push (telemetry-metrics-enabled=true, telemetry-metrics-endpoint -> cs-otel-collector.clickstack:4317) in the Keycloak CR. No static scrape job. W0 verifies which series land; fall back to a :9000/metrics scrape job if the feature proves unreliable.
Rationale
Section titled “Rationale”- OTLP push is the preferred collection model per existing ADRs (hl-fvje pure-OTLP, agentgateway/traefik OTLP-native precedent); scrape is the fallback, not the default.
- Eliminates a scrape job and ServiceMonitor for Keycloak, consistent with the OTel-first direction.
- The scrape fallback is explicitly preserved at the W0 verification gate, bounding the experimental-feature risk.
Alternatives Considered
Section titled “Alternatives Considered”- OTLP push via opentelemetry-metrics experimental feature (chosen): push-based, no scrape job, OTel-semconv native, consistent with OTel-first ADRs. Cost: upstream-labeled experimental (“not recommended for production”); Micrometer-to-OTel name differences need W0 verification.
- Static OTel scrape job targeting :9000/metrics (rejected as default, kept as fallback): proven pattern, identical Prometheus-name metrics, but retains a scrape dependency and Prometheus-name (non-semconv) metrics.
Consequences
Section titled “Consequences”- Positive: Keycloak metrics OTel-native; no scrape job or ServiceMonitor to maintain.
- Negative: experimental feature; metric names/families may differ from the Prometheus scrape output and must be confirmed before kps teardown.
- Neutral: W0 verification confirms the series before kps teardown; operator accepts the experimental risk with a documented scrape fallback.