Replicate ClickHouse without sharding via bootstrap Job
Date: 2026-07-05 Status: Accepted Decision: hl-4pmu Deciders: Sean Brandt
Context
Section titled “Context”HyperDX’s schema migrator only creates plain MergeTree tables (no CLICKHOUSE_CLUSTER / ON CLUSTER support through 2.29.0) and ClickHouse OSS never auto-converts MergeTree to ReplicatedMergeTree (the Replicated database engine replicates metadata only; the guard setting throws rather than converts). The 2026-05 shard experiment split round-robin writes across un-replicated shards (bd hl-p7z), and the 2026-07-05 disk-full incident (bd hl-z93i) exposed single-node ClickHouse as the availability ceiling.
Decision
Section titled “Decision”Adopt 1 shard x 2 ReplicatedMergeTree replicas (KeeperCluster x3), with the schema pre-created by an idempotent PreSync bootstrap Job (generated from the pinned upstream seeds); enforce database_replicated_allow_only_replicated_engine=1 on the otelcollector writer profile; drop the ClickHouse data volumes to a numberOfReplicas:1 Longhorn class (longhorn-single) since durability now lives at the ClickHouse layer.
Rationale
Section titled “Rationale”- The Replicated database engine replicates metadata only, never rewrites MergeTree to ReplicatedMergeTree — something other than the migrator must own schema creation (auto-conversion disproven at source level, engram research-2026-07-05).
- 1 shard x N replicas needs no Distributed table and no HyperDX changes; every replica holds all data and answers any query.
- The engine guard converts a future silent split-brain (a new upstream seed table created plain, with round-robin writes) into a visible collector crashloop.
- Longhorn 3x under app-replicated tables adds write amplification, not durability (6 copies of every byte on Turing Pi NVMe).
Alternatives Considered
Section titled “Alternatives Considered”- 1x2 ReplicatedMergeTree + bootstrap Job (chosen): no upstream changes needed; operator’s Replicated-database metadata sync verified live; community precedent in ClickStack-helm-charts#92.
- Sharded topology with Distributed tables (rejected): two orders of magnitude below the workload threshold; the migrator still seeds plain per-shard tables — the same split-brain class that failed in 2026-05.
- Single-node status quo (rejected): hard availability ceiling, exposed by the 2026-07-05 incident.
Consequences
Section titled “Consequences”- Positive: no Distributed/HyperDX changes; schema drift fails loudly; storage write amplification halved.
- Negative: the bootstrap Job must be regenerated on every chart bump (seed churn ~2/month, new tables ~2/year); metrics_ts (TimeSeries engine) is a permanent documented guard exception.
- Neutral: sharding remains a non-goal until sustained >20-30k events/s.