Skip to content
fzymgc-house Cluster Docs

ClickStack scalable rebuild: replicated ClickHouse + telemetry-pipeline hardening

  • Date: 2026-07-05
  • Bead: hl-uj69 (design); supersedes the upstream-PR framing of hl-p7z
  • Status: Approved

Problem

Section titled “Problem”

The 2026-07-05 disk-full incident (bd hl-z93i, PR #1604) exposed four systemic weaknesses in the observability stack, beyond the immediate 100Gi PVC:

  1. Single-node ClickHouse is the availability and scale ceiling. The 2026-05 shard experiment failed because HyperDX’s schema migrator creates plain MergeTree tables only — sharded queries went shard-local, extra replicas received zero writes (bd hl-p7z). That constraint is unchanged upstream through HyperDX 2.29.0 (ClickStack-helm-charts#92, dormant since 2025-08).
  2. The collector’s OOM protection cannot work as deployed. The ClickStack image ships memory_limiter: limit_mib: 1500 — sized for a 2Gi container — while ours runs a 1Gi limit, so the limiter sits above the cgroup OOM threshold and never engages. A ClickHouse write outage therefore OOM-crashloops the collector (188 restarts during the incident) instead of buffering or shedding load.
  3. The stack amplifies its own failures. ClickHouse logs a ~20-frame stack trace per failed insert; Vector ships clickhouse-server logs back into ClickHouse unfiltered and un-throttled; system.text_log (25.9 GiB, no TTL) additionally re-stores every log line in-table.
  4. Backups are the wrong tool for this data. Velero captures the 200Gi CH volume via Longhorn CSI snapshots + kopia data-mover (snapshotMoveData: true in argocd/app-configs/velero/backup-schedule.yaml) — crash-consistent at best, never restore-tested, requiring manual Keeper reconciliation (SYSTEM RESTORE REPLICA) to recover Replicated tables, and churning 200Gi of disposable telemetry to R2 daily. The state that actually matters (HyperDX dashboards/alerts/sources) lives in MongoDB.

A dashboard OOM class rides along: 7-day histogram-rate charts peak ~1.7 GiB against a 3.6 GiB server-wide memory cap (0.9 x 4Gi pod limit) shared with ingest — the per-query spill valve never triggers because the server total exhausts first.

Telemetry data is declared disposable for this program: the design optimizes for the end state, not for preserving history.

Grounding

Section titled “Grounding”

Four-agent research sweep 2026-07-05 (traces on bd hl-uj69; durable records in engram, tags research-2026-07-05):

  • Auto-conversion disproven at source level: ClickHouse OSS Replicated database engine replicates metadata only; it never rewrites MergeTree -> ReplicatedMergeTree. database_replicated_allow_only_replicated_engine throws (guard, not converter); default_table_engine applies only when the ENGINE clause is absent (HyperDX seeds declare it); convert_to_replicated is Atomic-database-only.
  • Split-brain hazard: replicas > 1 with the stock migrator creates tables everywhere (metadata DDL propagation) but replicates no data — round-robin Services split rows into disjoint per-replica subsets (Langfuse#12589 is the same failure class).
  • 1 shard x N replicas needs no Distributed table and no HyperDX changes. HyperDX 2.28/2.29’s Distributed-table read support targets sharded setups only. Community precedent: ClickStack-helm-charts#92 user and a published Altinity-operator walkthrough, both with pre-created replicated schemas.
  • Seed schema churn ~2 commits/month, new tables ~2/year; upstream cannot retrofit existing tables either (IF NOT EXISTS), so all long-lived deployments drift. Disposable data permits drop-and-rebootstrap adoption.
  • No OSS shared storage: zero-copy replication is officially “not ready for production”; s3_plain_rewritable cannot replicate; SharedMergeTree is Cloud-only. Hot path = replicated local volumes.
  • RustFS caution: zero ClickHouse compatibility reports; known S3-semantics gaps (rustfs#2540, #2079, #2607). Suitable as a bounded backup target only, never as a live S3 data disk.
  • Operator verified live: official ClickHouse operator (clickhouse.com/v1alpha1, ClickHouseCluster/KeeperCluster) propagates dataVolumeClaimSpec growth to existing PVCs; Longhorn expands online (proven during incident recovery, PR #1604).
  • Per-replica StatefulSets with preserved volumeClaimTemplates (deepwiki vs ClickHouse/clickhouse-operator, confirmed independently in design review round 3): the operator creates one single-pod StatefulSet per replica and preserves each replica’s PVC spec on reconcile — which is why changing dataVolumeClaimSpec.storageClassName affects only new replicas’ PVCs (Workstream 2 item 4) while replica-0’s existing PVC keeps its class until recreated.
  • Replicated database engine verified live (2026-07-05, system.databases on the running cluster): default is already Replicated('/clickhouse/databases/default', '{shard}', '{replica}') — the operator’s enableDatabaseSync (set in our CR) did this at deploy time. Workstream 2’s “new replica receives schema via metadata sync” therefore rests on observed state, not assumed operator behavior. (Tables inside it remain plain MergeTree; that is exactly the gap the bootstrap Job closes.)
  • Citations verified against GitHub (2026-07-05, gh api): langfuse#12589 “ClickHouse schema migrations are flaky with Replicated database”; ClickStack-helm-charts#92 “Improvements needed for external MongoDB, OpenTelemetry metrics, and ClickHouse schema support”; hyperdx PR #2351 “Allow CUSTOM_OTELCOL_CONFIG_FILE to override default processors”; hyperdx PR #2545 “fix: change metrics PK for better memory usage”.

Decisions

Section titled “Decisions”
DecisionChoice
ScopeOne rebuild-centered program absorbing interim fixes and hygiene
Cutover modelTables-only greenfield: keep namespace, MongoDB, URLs; drop/recreate CH tables
Topology1 shard x 2 replicas, KeeperCluster 1 -> 3
CH data storageNew longhorn-single StorageClass (numberOfReplicas: 1); Keeper stays on standard longhorn
Retentionlogs 14d / metrics 30d / traces 14d / sessions 14d; system.* logs ~7d
BackupsVelero excludes CH data PVCs; MongoDB stays Velero-covered; weekly native BACKUP DATABASE ... TO S3(RustFS) + daily incrementals, restore rehearsal as acceptance gate
Merge sequencingThree PRs: PR1 values-only hardening, PR2 chart+topology — each its own ArgoCD sync with validation between — then PR3 (bootstrap Job + collector-user guard + Velero exclusion) merged at the start of the cutover window

Non-goals

Section titled “Non-goals”
  • Sharding (two orders of magnitude below the workload threshold; wrong for arm64 boards).
  • S3-tiered hot storage (fights replication 2x; RustFS unqualified). Revisit post-qualification if long retention becomes a goal.
  • Swapping to the Altinity operator (official operator just proved its key day-2 op; alpha API is a watch-item, not a blocker).
  • Preserving telemetry history across the cutover.

Architecture

Section titled “Architecture”
                     otel-node DaemonSet (metrics)      Vector DaemonSet (logs)
                              \                          | throttle: clickstack ns
                               v                         v
   OTLP clients ---> cs-otel-collector (2Gi, memory_limiter effective,
                     disk-backed bounded export queue)   |
                              \__________________________|
                                          v
                              ClusterIP Service (round-robin)
                               /                        \
                    CH replica 0-0-0             CH replica 0-1-0
                    (200Gi longhorn-single)      (200Gi longhorn-single)
                          \   ReplicatedMergeTree sync   /
                           \____ KeeperCluster x3 _____/
                                          ^
                     bootstrap Job (PreSync hook): Replicated schema,
                     per-signal TTLs; migrator seeds no-op via IF NOT EXISTS
  • Writes: any replica accepts inserts (async replication, insert_quorum=0 — correct trade for telemetry). Exporter async_insert disabled (Workstream 1 item 2) to restore idempotent block dedup on retries. Note the two writers use different Service objects: the collector targets the round-robin ClusterIP Service; Vector targets the headless Service (cs-clickstack-clickhouse-clickhouse-headless...) and resolves pod IPs directly. Correctness is unaffected — ReplicatedMergeTree replicates regardless of which pod receives an insert — but validation steps must not assume a single shared write path.
  • Reads: HyperDX -> same Service. Contingency for live-tail flicker from replica lag: sessionAffinity: ClientIP on a HyperDX-facing Service.
  • Guard: database_replicated_allow_only_replicated_engine=1 on the otelcollector user profile. A future upstream seed adding a new table fails loudly (collector migration crashloop) instead of silently creating a plain table with round-robin-split data. Drift alert on system.tables WHERE engine = 'MergeTree' AND database = 'default' as belt-and-suspenders.
  • Documented exception: metrics_ts (TimeSeries engine, experimental PromQL path). Its internal DATA/TAGS/METRICS target tables can use Replicated-family engines, but TimeSeries DDL does not round-trip cleanly through the Replicated database engine’s DDL mechanism — so the bootstrap pre-creates it as-is under an admin user and the guard never evaluates it. Its data splits across replicas; acceptable while the PromQL feature is unused.

Workstream 1 — PR1: values-only hardening (one sync, no interaction risk)

Section titled “Workstream 1 — PR1: values-only hardening (one sync, no interaction risk)”

All in argocd/app-configs/clickstack-chart/values.yaml + argocd/app-configs/clickstack/ + argocd/app-configs/monitoring-vector/:

  1. Collector memory 1Gi -> 2Gi (request 1Gi / limit 2Gi). This makes the shipped memory_limiter (1500 MiB, designed for 2Gi) correct as-is — less config than re-tuning limits.

  2. CUSTOM_OTELCOL_CONFIG_FILE plumbing: ConfigMap mounted via otel-collector.extraVolumes/extraVolumeMounts, env via extraEnvs. Contents (all leaf-level additions that survive the koanf key merge under the 2.27 OpAMP remote config):

    • extensions.file_storage/queue (emptyDir at /var/lib/otelcol/queue);
    • exporters.clickhouse.sending_queue and exporters.clickhouse/rrweb.sending_queue: {sizer: items, queue_size: 50000, block_on_overflow: true, storage: file_storage/queue};
    • exporters.clickhouse.async_insert: false (likewise for clickhouse/rrweb) — the contrib exporter defaults this on, and async inserts bypass ReplicatedMergeTree’s idempotent block dedup; disabling it is what makes retry-after-timeout non-duplicating (the Architecture section depends on this key);
    • processors.batch.send_batch_max_size: 10000 — caps single-request memory. No HYPERDX_OTEL_BATCH_* env vars exist upstream (the collector consumes exactly three HYPERDX_OTEL_* vars, all schema-related), so this cap MUST live in the config file, not in env.

    Re-verify the effective config after the PR2 image bump (2.28 moved pipeline processor lists local — these leaf additions remain valid).

  3. ClickHouse containerTemplate.resources: 2/4Gi -> 4Gi request / 8Gi limit (fits 16GB beta workers; fixes the dashboard-OOM class together with item 4).

  4. clickhouse.cluster.spec.settings.extraUsersConfig (full path — this chart silently drops values under wrong paths, the exact failure class documented at values.yaml lines 65-72): app user profile max_memory_usage: 2147483648 (2 GiB) so dashboard queries spill or fail alone instead of exhausting the server-wide cap.

  5. clickhouse.cluster.spec.settings.extraConfig (same full-path caution): TTLs on system log tables (text_log, query_log, part_log, trace_log, metric_log) at ~7 days.

  6. Disable the unused, unauthenticated fluentd receiver port (24225) — no consumer exists in this cluster.

  7. Vector: throttle transform on kubernetes_logs, keyed per-container, ~100 events/s for the clickstack namespace (breaks the self-amplification loop independently of the collector).

Workstream 2 — PR2: chart bump + topology (one sync)

Section titled “Workstream 2 — PR2: chart bump + topology (one sync)”
  1. Prerequisite — delete the four orphaned 100Gi PVCs (...clickhouse-0-1-0/0-2-0/1-0-0/2-0-0, relics of the 2026-05 shard experiment). Critical, not hygiene: the operator derives PVC names from shard/replica indices, so the new replica in step 4 would silently adopt the stale 0-1-0 volume — wrong StorageClass, old experiment data.
  2. Chart 3.0.0 -> 3.0.1 (app + collector 2.29.0): follow the render-diff protocol in argocd/CLAUDE.md; drop the 2.27.0 collector image-tag override (upstream #219 fixed); diff docker/otel-collector/schema/seed/ between 2.27.0 and 2.29.0 tags to refresh the bootstrap DDL source (new metrics PK #2545, rollup + timeseries tables).
  3. longhorn-single StorageClass (numberOfReplicas: "1", allowVolumeExpansion: true) — added to argocd/app-configs/shared-resources/longhorn-storage-classes.yaml, alongside the existing longhorn-retain class.
  4. KeeperCluster 1 -> 3 replicas (raft membership change; the riskiest single op in this PR — validate quorum health post-sync before proceeding).
  5. ClickHouseCluster replicas: 1 -> 2; dataVolumeClaimSpec.storageClassName: longhorn-single (affects the new replica’s PVC only; replica-0’s class is immutable and is handled in the cutover window).

Post-sync state: replica 0-1-0 runs with schema but no data — a bounded, known split-data window; hold the cutover to the same day. The schema arrives via Replicated-database metadata sync, which is grounded in observed state (see Grounding: default is already Replicated(...) engine on the live cluster), not assumed operator behavior.

Workstream 3 — PR3 + cutover window (minutes, telemetry-history reset)

Section titled “Workstream 3 — PR3 + cutover window (minutes, telemetry-history reset)”

The cutover changes are their own merge, PR3, landed at the start of the maintenance window: the bootstrap Job manifest, the otelcollector-user guard, and the Velero exclusion from Workstream 4 item 2 (it must be active before the new empty tables start filling).

Bootstrap Job (new, in argocd/app-configs/clickstack/): ArgoCD PreSync hook with argocd.argoproj.io/hook-delete-policy: BeforeHookCreation. The delete-policy has repo precedent (argocd/app-configs/clickstack-alerts/bootstrap-job.yaml and the keycloak dcr-trusted-hosts job use BeforeHookCreation; nats uses HookSucceeded) and is what lets the hook re-fire on every sync — but the PreSync phase is a novel pattern in this repo: all three existing hooks are PostSync. PreSync is nonetheless required, not preference: the collector’s migrator runs at pod start, so on any future sync that rolls the collector (e.g. a chart bump shipping a new seed table) the Job must complete before the new pod starts. A PostSync hook would deadlock — the guard makes the new pod crashloop on the missing table, the app never reports healthy, and the PostSync hook that would create the table never fires. Because the clickstack Application is multi-source (chart-inflated + plain kustomize) and hook ordering across multi-source apps is unexercised in this repo, PreSync ordering gets its own pre-cutover validation (see Validation). clickhouse/clickhouse-server image running clickhouse-client against one replica with admin credentials from clickstack-secret. DDL = the 3.0.1 image’s seed SQL verbatim with:

  • ENGINE = MergeTree -> ENGINE = ReplicatedMergeTree (and SummingMergeTree -> ReplicatedSummingMergeTree for rollups) — no engine arguments (Replicated DB {uuid}-based defaults; explicit paths are rejected by default anyway).
  • Per-signal TTLs baked in: logs 14d, metrics tables 30d, traces 14d, sessions 14d.
  • Materialized views unchanged (TO-table MVs store no data).
  • metrics_ts created as-is (see exception above).
  • Everything IF NOT EXISTS — the Job is idempotent and safe on every sync; the collector’s own migrator then no-ops forever.

Runbook:

  1. Scale cs-otel-collector deployment to 0 (Vector keeps buffering on disk; OTLP clients retry).
  2. DROP TABLE ... SYNC all otel_*, rollup, MV, and hyperdx_sessions tables on the Replicated database (propagates to both replicas).
  3. Trigger the bootstrap Job; verify system.replicas lists all tables on both replicas.
  4. Delete replica-0’s PVC + pod: recreated on longhorn-single; empty tables sync trivially from replica 1.
  5. Confirm the otelcollector-user guard (database_replicated_allow_only_replicated_engine=1, set via clickhouse.cluster.spec.settings.extraUsersConfig in PR3) is live before the collector returns.
  6. Scale the collector back up; watch ingest resume on both replicas.

Vector needs no change (same Service, same user, INSERT-only). HyperDX sources reference unchanged table names; dashboards and alerts continue working against empty-then-refilling tables.

Workstream 4 — Post-cutover: backups, safety nets, hygiene

Section titled “Workstream 4 — Post-cutover: backups, safety nets, hygiene”
  1. CH backups: weekly CronJob BACKUP DATABASE default TO S3('https://s3.nas.fzymgc.house/<bucket>/clickstack-backups', ...) + daily incrementals via base_backup + use_same_s3_credentials_for_base_backup. RustFS qualification settings: path-style endpoint, support_batch_delete: false initially, s3_check_objects_after_upload: 1. Acceptance gate: one rehearsed RESTORE into a scratch database. Documented fallback if RustFS fails qualification: drop CH backups entirely (replication is the availability story) and re-evaluate with MinIO/Garage.
  2. Velero (manifest change rides in PR3 so it is active before the new tables fill; listed here as it is operationally a backup concern): exclude CH data volumes via a Velero resource-policies ConfigMap with a volume policy storageClasses: [longhorn-single] -> action: skip, referenced by both Schedules (resourcePolicy). Keying on the CH-only StorageClass is self-maintaining — it survives PVC recreation, unlike per-PVC labels. MongoDB volumes stay covered; optional mongodump CronJob for a logical dashboard backup.
  3. Watchdog (drift detector + dead-man’s switch, one mechanism): a CronJob (every 5 min) checks (a) ingest freshness (max(TimeUnix) staleness) and (b) engine drift (system.tables WHERE engine IN ('MergeTree','SummingMergeTree') AND database='default'), and pushes an uptime-kuma heartbeat only when both pass; uptime-kuma (already pushover-wired) alerts on missed heartbeats. Rationale for implementing drift detection here rather than as a ClickStack tile alert (amended during plan-writing): an alert inside the stack cannot fire when the stack is dark — the external watchdog covers both failure modes with one moving part.
  4. Optional: collector replicaCount: 2 + podDisruptionBudget (CPU-only HPA if ever; never memory-based for queue-driven collectors).

Risks and rollback

Section titled “Risks and rollback”
RiskMitigation
Seed churn on future chart bumps desyncs bootstrap DDLDiff schema/seed/ per upgrade (release checklist); the guard turns a missed new table into a visible collector crashloop, never silent split data; disposable data permits drop-and-rebootstrap
Keeper 1->3 membership change wedges DDLOwn sync (PR2), validated before cutover; rollback = revert to 1 replica
Live-tail flicker from replica lagsessionAffinity: ClientIP contingency on HyperDX’s read path
RustFS backup qualification failsDocumented fallback: no CH backups; MinIO/Garage re-evaluation
Chart 3.0.1 remote-config changes interact with custom collector configPR1 validated under 2.27 first; effective-config re-verified after PR2 (HyperDX collector page)
metrics_ts guard exception confuses future readersDocumented here and inline in the bootstrap DDL
PreSync hook is novel in this repo + multi-source Application hook ordering unexercisedValidated on a benign sync pre-cutover (Validation section); fallback: run the DDL manually via clickhouse-client in the window and demote the Job to PostSync-with-documented-caveat
Community-thin precedent for this exact HyperDX topologyValidation phase exercises every read/write path before the old PVC is reclaimed; rollback below

Rollback: PR1/PR2 are git reverts (ArgoCD auto-syncs back). Cutover worst case: re-drop tables, re-run bootstrap with plain engines at replicas: 1 — cost is telemetry history, already written off.

Validation

Section titled “Validation”
  • PreSync hook ordering in the multi-source Application verified on a benign sync before the cutover relies on it: the Job runs to completion before Sync-phase resources (collector Deployment) apply. First PreSync hook in this repo — do not assume.
  • Collector effective config (HyperDX collector page / OpAMP view) shows the custom-file keys applied: sending_queue (with storage), async_insert: false, send_batch_max_size — checked after PR1 and re-checked after the PR2 image bump (2.27 -> 2.29 merge-semantics change).
  • system.replicas: all otel tables present on both replicas, queue_size ~0, no is_readonly.
  • Row-count convergence checked per-pod (not via the Service) after 15 minutes of ingest.
  • Pod-kill test: delete one CH pod; ingest and dashboards continue; replica rejoins and catches up.
  • Dashboard smoke: the 7d histogram chart that OOM’d renders under the 2 GiB app-user cap.
  • Alert smoke: one tile alert fires end-to-end (pushover).
  • Backup rehearsal: RESTORE into scratch DB from RustFS succeeds.
  • Collector resilience re-test: brief CH write outage (readonly toggle) shows queue-to-disk buffering, no OOM, no restart.

References

Section titled “References”
  • Beads: hl-uj69 (this design), hl-z93i (incident, closed), hl-p7z (updated with research), ClickStack-helm-charts#92, hyperdx PR #2351/#2545.
  • Engram: memories tagged research-2026-07-05, incident-2026-07-05 (mechanism details, collector config merge semantics, operator PVC-resize proof, RustFS evidence).
  • PRs: #1604 (PVC expansion, merged).