ClickStack vs Loki+Prom Comparison
Evaluation runbook for the vector + ClickStack migration.
Evaluation window: 1 month minimum, 3 months maximum from PR1 merge.
Decision date: TBD (set when PR1 merges)
Known Divergence Classes (Not Data Loss)
Section titled “Known Divergence Classes (Not Data Loss)”These differences are expected and do not count toward dimension 3’s “zero data-loss incidents”:
- Log serialization: Loki LogQL line vs OTel LogRecord body/attributes
- Log transport: Loki HTTP push vs OTLP gRPC
- Metric semantics: Prom remote_write preserves data model exactly; OTLP path converts to OTel model (histogram re-bucketing, summary lossy mapping, staleness markers absent)
- Timestamp precision: Prom ms vs OTel ns
- Counter resets: Prom transmits resets; OTel cumulative conversion can mask or amplify reset artifacts
Dimension Scorecard (Live During Eval)
Section titled “Dimension Scorecard (Live During Eval)”Dimension 1: Query Parity (target: all 10 queries expressible)
Section titled “Dimension 1: Query Parity (target: all 10 queries expressible)”| # | Query | Loki/Prom expression | HyperDX expression | Equivalence |
|---|---|---|---|---|
| 1 | Last 1h pod crash count | count_over_time({namespace="X"} |= "CrashLoopBackOff"[1h]) |
kubernetes.namespace_name:X AND message:CrashLoopBackOff |
TBD |
| 2 | Cert expiring in 7d | (probe_ssl_earliest_cert_expiry - time()) / 86400 < 7 |
SQL on otel_metrics_gauge | TBD |
| 3 | (to be filled pre-evaluation) | TBD | ||
| 4 | (to be filled pre-evaluation) | TBD | ||
| 5 | (to be filled pre-evaluation) | TBD | ||
| 6 | (to be filled pre-evaluation) | TBD | ||
| 7 | (to be filled pre-evaluation) | TBD | ||
| 8 | (to be filled pre-evaluation) | TBD | ||
| 9 | (to be filled pre-evaluation) | TBD | ||
| 10 | (to be filled pre-evaluation) | TBD |
Dimension 2: Investigation Speed (target: HyperDX first-reach >50% weeks 3+, n>=10)
Section titled “Dimension 2: Investigation Speed (target: HyperDX first-reach >50% weeks 3+, n>=10)”| Date | Question | Stack opened first | Stack that resolved | Notes |
|---|---|---|---|---|
| (filled in during eval) |
Dimension 3: Operational Cost
Section titled “Dimension 3: Operational Cost”Resource consumption snapshots (weekly):
| Week | ClickStack CPU/mem/disk | Loki+Prom CPU/mem/disk | Ratio | Notes |
|---|---|---|---|---|
| 1 | baseline |
Mongo-down incidents: count and timestamps (per “Mongo-down incident” definition in spec).
Disruption baseline (Loki LogQL):
count_over_time({job="kubernetes-events"} |~ "(?i)node draining|Cordoned|Drained" [<BASELINE_WINDOW>])Result captured in audit doc; eval-window incidents rate-normalized against this baseline.
Dimension 4: IaC Story
Section titled “Dimension 4: IaC Story”Attempted: (describe HyperDX dashboard-as-code attempt here)
Outcome: (pass/fail with rationale)
Dimension i (Informational): AI Features
Section titled “Dimension i (Informational): AI Features”Auto-clustering value:
- (date, observation, did it surface something Grafana would not?)
NL-to-query value:
- (date, observation)
Anomaly detection:
- (date, observation)
Decision (Filled In at End of Window)
Section titled “Decision (Filled In at End of Window)”Hard dimensions won (out of 4 or 3 depending on dim-2 sample size):
- Dim 1: WIN / LOSS / UNCLEAR
- Dim 2 (hard if n>=10): WIN / LOSS / INFORMATIONAL
- Dim 3: WIN / LOSS / UNCLEAR
- Dim 4: WIN / LOSS / UNCLEAR
Path selected: A / B / C
Rationale: …
Next-step PR: …