Skip to content
fzymgc-house Cluster Docs

Adopt HyperDX/ClickStack as cluster observability platform; retire Grafana-Labs

Date: 2026-06-25 Status: Accepted Decision: hl-5xgq Deciders: sean

Context

Section titled “Context”

The cluster ran the Grafana-Labs observability stack — Grafana UI, Loki (log backend), grafana-operator, and grafana-mcp — alongside Prometheus. HyperDX/ClickStack was adopted as the OTLP-native replacement: dashboards live in MongoDB-backed HyperDX, and logs/metrics/traces land in ClickHouse (default.otel_{logs,metrics_*,traces}). The 2026-06-15 operator decision (engram memory 67a2c263) directed full retirement of all Grafana-Labs components. Loki was additionally pinned to chart 7.0.0 — a GEL-only dead release with no OSS upgrade path.

Decision

Section titled “Decision”

Replace Grafana UI, Loki, grafana-operator, and grafana-mcp with HyperDX/ClickStack as the cluster’s unified observability UI and log/trace/metric backend. Prometheus (kube-prometheus-stack) is retained separately as the scrape-based metrics/alerting backbone (its Grafana subchart is already disabled).

Rationale

Section titled “Rationale”
  • ClickStack already receives all cluster OTLP telemetry; Grafana/Loki adds no unique data coverage.
  • Loki 7.0.0 is pinned to a GEL-only dead release with no viable upgrade path.
  • Dashboards and alerts were already fully migrated to HyperDX prior to this teardown.
  • OTel-first is an explicit operator directive, corroborated by the narrower OTLP-routing ADRs hl-fvje, hl-5z4x, and hl-1pwl.

Alternatives Considered

Section titled “Alternatives Considered”
  • Adopt HyperDX/ClickStack; retire all Grafana-Labs components (chosen): single OTLP-native platform for logs/metrics/traces/UI; dashboards and alerts already migrated; eliminates the dead Loki chart; reduces operational surface (no grafana-operator CRDs).
  • Continue the Grafana-Labs stack (rejected): mature ecosystem and existing plugins, but redundant with ClickStack which already receives all OTLP telemetry; Loki 7.0.0 is a dead chart; keeps a log pipeline separate from traces/metrics; ongoing grafana-operator / CRD / finalizer overhead.

Consequences

Section titled “Consequences”
  • Positive: single observability platform for logs, metrics, traces, and UI; eliminates the grafana-operator CRD surface and its finalizer complexity; removes the unmaintainable Loki chart dependency.
  • Negative: the Grafana plugin ecosystem is unavailable — all visualization, dashboards, and alerts must be built in HyperDX.
  • Neutral: Prometheus is retained as a separate metrics/alerting backbone; Vector’s clickhouse sink is unaffected (only the loki sink is removed).