Skip to content
fzymgc-house Cluster Docs

ClickStack 2.37.0 schema rehearsal — the bootstrap DDL generator fix and the three unexecuted ClickHouse mechanics, 2026-08-30

Captured against context fzymgc-house with clickhouse-client inside the ClickHouse pods cs-clickstack-clickhouse-clickhouse-0-0-0 and cs-clickstack-clickhouse-clickhouse-0-1-0 in namespace clickstack (server 26.7.1.1315), against the two scratch Replicated databases bootstrap_rehearsal and bootstrap_rehearsal_src, plus local runs of scripts/clickstack/generate-bootstrap-ddl.sh against the upstream seed set at hyperdxio/hyperdx tag @hyperdx/otel-collector@2.37.0.

Why this file exists: the ClickStack 2.37.0 change window rests on three ClickHouse mechanics that had been read in the server source but never executed against this cluster — that EXCHANGE TABLES is atomic on the Replicated database engine and must not carry a cluster clause, that INSERT … SELECT coerces Array(DateTime64(9)) to Array(DateTime) without error, and that a full-text text(tokenizer = 'array') index is accepted on an ALIAS column at 26.7. Reasoning about a database engine is not the same as running against it, and the migration this evidence supports is not reversible in the cheap sense. This is a dated record of work performed on one day rather than standing operational prose, so it lives in evidence/ rather than in docs/operations/clickstack.md. The runbook says what to do; this file says what actually happened when it was tried.

Where the planning artifacts cited below live. Several passages cite a phase planning artifact by bare filename (03-RESEARCH.md, 03-01-PLAN.md, 03-CONTEXT.md). Every one of them lives in .planning/workstreams/cluster-health/phases/03-clickstack-currency/. That directory is a planning record kept in the repository and is deliberately not published to the documentation site, so a reader on the site should expect to open those artifacts from a checkout rather than follow a link from here.

Status of the three assumptions before this capture. 03-RESEARCH.md records A1, A2 and A3 as source-verified but unexecuted — §A.4 states plainly that the EXCHANGE path “was NOT executed this session — I read the implementation rather than mutating a production cluster”. This file is that execution. Nothing in default was touched; every statement below names bootstrap_rehearsal or bootstrap_rehearsal_src.

What was proven

Section titled “What was proven”
ID Assumption Verdict
— The bootstrap DDL generator emits substitution-complete SQL at the 2.37.0 seed tag, and fails loudly when it does not Proven, with a RED positive control
A1 EXCHANGE TABLES is atomic on a Replicated database at 26.7, without ON CLUSTER Proven, plus a negative control showing ON CLUSTER is rejected
A2 INSERT … SELECT with an explicit column list coerces Array(DateTime64(9)) to Array(DateTime) and preserves row count Proven on otel_metrics_gauge (1,200 rows) and otel_metrics_summary (1,100 rows)
A3 ADD INDEX … TYPE text(tokenizer = 'array') is accepted on an ALIAS column of a ReplicatedMergeTree table at 26.7 Proven by DROP INDEX then ADD INDEX on bootstrap_rehearsal.otel_logs

Generator run and RED positive control, verbatim

Section titled “Generator run and RED positive control, verbatim”

The generator’s macro guard was a bare ! rg -q '$\{' … under set -e. Bash explicitly does not exit on a command whose status is inverted with !, so the guard detected incomplete substitution and then discarded the detection — the exact defect 03-RESEARCH.md §B.4a reproduced. The fix rewrites it into the file’s own <test> || { echo "ERROR: …" >&2; exit 1; } idiom and extends both sed invocations to substitute all five TTL macro names upstream now uses.

The TTL clauses below are the byte-for-byte reproduction of today’s live intervals: logs, traces and sessions at 14 days, metrics at 30 days.

=== generator capture: 2026-08-30T20:39:51Z ===
$ rg -n "^TAG=" scripts/clickstack/generate-bootstrap-ddl.sh
36:TAG="@hyperdx/otel-collector@2.37.0"


$ bash scripts/clickstack/generate-bootstrap-ddl.sh bootstrap_rehearsal
CH 26.7: skipping seed files matching: _compat
skip: 00002_otel_logs_compat.sql (version-gated alternate)
skip: 00005_otel_traces_compat.sql (version-gated alternate)
Wrote /tmp/bootstrap-bootstrap_rehearsal.sql (rehearsal; not a ConfigMap)
  [exit 0]


$ rg -o "\$\{[A-Z_]+\}" /tmp/bootstrap-bootstrap_rehearsal.sql | wc -l      # GREEN: unsubstituted macros
0
$ rg -o "CREATE (TABLE|MATERIALIZED VIEW) IF NOT EXISTS" /tmp/bootstrap-bootstrap_rehearsal.sql | wc -l
13
$ rg -o "ON CLUSTER" /tmp/bootstrap-bootstrap_rehearsal.sql | wc -l
0
$ printf 'x ON CLUSTER y\n' | rg -o 'ON CLUSTER' | wc -l                  # positive control for the pattern above
1
$ rg -o "TYPE text\(tokenizer" /tmp/bootstrap-bootstrap_rehearsal.sql | wc -l
10
$ rg -n "TTL .*toIntervalDay" /tmp/bootstrap-bootstrap_rehearsal.sql
47:TTL toDateTime(Timestamp) + toIntervalDay(14)
85:TTL toDateTime(TimeUnix) + toIntervalDay(30)
125:TTL toDateTime(TimeUnix) + toIntervalDay(30)
170:TTL toDateTime(TimeUnix) + toIntervalDay(30)
219:TTL toDateTime(TimeUnix) + toIntervalDay(30)
256:TTL toDateTime(TimeUnix) + toIntervalDay(30)
293:TTL TimestampTime + toIntervalDay(14)
338:TTL toDate(Timestamp) + toIntervalDay(14)
358:TTL Timestamp + toIntervalDay(14)
481:TTL Timestamp + toIntervalDay(14)


=== RED positive control: 2026-08-30T20:39:53Z ===
$ bash /tmp/red-control.sh scripts/clickstack/generate-bootstrap-ddl.sh
non-comment LOGS_TTL substitution arms at lines: 85 92
  line 85: MUTATION INERT - deletion landed but changed no output (this arm feeds the 00008/TimeSeries branch, whose seed carries no TTL macro); says NOTHING about the guard
  line 92: RED CONTROL FIRED (exit 1) - ERROR: unsubstituted macro(s) remain: ${LOGS_TTL}
summary: fired=1 vacuous=0 inert=1
VERDICT: the macro guard is proven RED - it fires on every load-bearing deletion
  [exit 0]

The control’s own control. 03-01-PLAN.md specified the RED control as “delete the first non-comment ${LOGS_TTL} substitution arm”. Run verbatim, that control reports GUARD IS VACUOUS against a guard that demonstrably works. The first arm lives in the 00008/TimeSeries branch, and 00008_otel_metrics_timeseries.sql carries no TTL macro at all — so deleting it changes no output and exercises nothing. A control that judges by line position cannot tell “the guard did not fire” from “the mutation did nothing”, which is precisely the failure mode this whole task exists to close. The control was therefore rewritten to judge by the artifact the mutant emits: it walks every non-comment arm and classifies each as fired, vacuous or inert, with distinct exit codes. Both readings are recorded here.

=== the plan's control, run verbatim, misdiagnosing itself: 2026-08-30T20:40:12Z ===
$ n=$(rg -nF "LOGS_TTL}/" "$src" | rg -v "^[0-9]+:[[:space:]]*#" | head -1 | cut -d: -f1)   # ...then sed "${n}d"
mutation landed: removed line 85 of scripts/clickstack/generate-bootstrap-ddl.sh; non-comment LOGS_TTL arms 2 -> 1
GUARD IS VACUOUS: line 85 was provably removed and the generator still did not fail with ERROR: unsubstituted macro
  [exit 1]


$ rg -o "\$\{[A-Z_]+\}" /tmp/seed237/00008_otel_metrics_timeseries.sql | sort -u   # why line 85 is inert
${DATABASE}

The rewritten control, in full, so the result above is reproducible from a checkout:

#!/usr/bin/env bash
# RED positive control for the macro guard in generate-bootstrap-ddl.sh.
#
# Contract: deleting a load-bearing ${LOGS_TTL} substitution arm must make the
# generator exit non-zero with "ERROR: unsubstituted macro".
#
# Three outcomes, three exit codes, never reported as each other:
#   0 = guard fired on every load-bearing deletion
#   1 = GUARD IS VACUOUS  (mutant emitted an unsubstituted macro and the
#                          generator still exited without the ERROR line)
#   2 = MUTATION FAILED   (the deletion did not land, or no arm was
#                          load-bearing, so the guard was never exercised)
#
# The arbiter is the artifact the mutant emits, not the line number deleted:
# one of the two arms lives in the 00008/TimeSeries branch, whose only seed
# file carries no TTL macro at all, so deleting it changes no output. A
# control that judged by position would call that "GUARD IS VACUOUS".
set -u
src=${1:-scripts/clickstack/generate-bootstrap-ddl.sh}
arms=$(rg -nF 'LOGS_TTL}/' "$src" | rg -v '^[0-9]+:[[:space:]]*#' | cut -d: -f1)
[ -n "$arms" ] || { echo "MUTATION FAILED: no non-comment LOGS_TTL arm in $src"; exit 2; }
echo "non-comment LOGS_TTL substitution arms at lines: $(echo $arms | tr '\n' ' ')"
fired=0; vacuous=0; inert=0
for n in $arms; do
  rm -f /tmp/bootstrap-probe_red.sql
  sed "${n}d" "$src" > /tmp/gen-broken.sh
  if diff -q /tmp/gen-broken.sh "$src" >/dev/null 2>&1; then
    echo "  line $n: MUTATION FAILED - mutant is byte-identical to source"; exit 2; fi
  ls=$(wc -l < "$src"); lb=$(wc -l < /tmp/gen-broken.sh)
  [ "$((ls - lb))" -eq 1 ] || { echo "  line $n: MUTATION FAILED - mutant is $((ls-lb)) lines shorter, want exactly 1"; exit 2; }
  out=$(bash /tmp/gen-broken.sh probe_red 2>&1); rc=$?
  if printf '%s' "$out" | rg -q 'ERROR: unsubstituted macro'; then
    [ "$rc" -ne 0 ] || { echo "  line $n: guard printed ERROR but exited 0"; exit 1; }
    echo "  line $n: RED CONTROL FIRED (exit $rc) - $(printf '%s' "$out" | rg -o 'ERROR: unsubstituted macro.*')"
    fired=$((fired+1))
  elif [ -f /tmp/bootstrap-probe_red.sql ] && [ "$(rg -o '\$\{[A-Z_]+\}' /tmp/bootstrap-probe_red.sql | wc -l | tr -d ' ')" -gt 0 ]; then
    echo "  line $n: GUARD IS VACUOUS - mutant emitted $(rg -o '\$\{[A-Z_]+\}' /tmp/bootstrap-probe_red.sql | sort -u | tr '\n' ' ')and the generator exited $rc without the ERROR line"
    vacuous=$((vacuous+1))
  else
    echo "  line $n: MUTATION INERT - deletion landed but changed no output (this arm feeds the 00008/TimeSeries branch, whose seed carries no TTL macro); says NOTHING about the guard"
    inert=$((inert+1))
  fi
done
echo "summary: fired=$fired vacuous=$vacuous inert=$inert"
[ "$vacuous" -eq 0 ] || { echo "VERDICT: GUARD IS VACUOUS"; exit 1; }
[ "$fired" -ge 1 ] || { echo "VERDICT: MUTATION FAILED - no arm was load-bearing, the guard was never exercised"; exit 2; }
echo "VERDICT: the macro guard is proven RED - it fires on every load-bearing deletion"

Provenance and server identity, verbatim

Section titled “Provenance and server identity, verbatim”
=== capture start: 2026-08-30T20:38:35Z ===
$ clickhouse-client --query "SELECT version()"
26.7.1.1315
  [exit 0]
$ clickhouse-client --query "SELECT name, engine, engine_full FROM system.databases WHERE name LIKE 'bootstrap_rehearsal%' ORDER BY name FORMAT TSV"
bootstrap_rehearsal  Replicated  Replicated(\'/clickhouse/databases/bootstrap_rehearsal\', \'{shard}\', \'{replica}\')
bootstrap_rehearsal_src  Replicated  Replicated(\'/clickhouse/databases/bootstrap_rehearsal_src\', \'{shard}\', \'{replica}\')
  [exit 0]

A2 — INSERT … SELECT nested-array coercion, verbatim

Section titled “A2 — INSERT … SELECT nested-array coercion, verbatim”

otel_metrics_summary is the table 03-01-PLAN.md named for this proof, but it carries no Exemplars.* columns — the OpenTelemetry Summary point type has no exemplars, and neither the 2.29.0 nor the 2.37.0 seed defines them there. The nested-array half of A2 is therefore proven on otel_metrics_gauge, which does carry Exemplars.TimeUnix and which 03-RESEARCH.md §A.3 names as the reference case for the identical delta. otel_metrics_summary is exercised too, for the String → LowCardinality(String) and DateTime64(9) → DateTime halves.

The source rows were written with a deliberate sub-second component (.987654321) so the truncation is visible rather than inferred.

########## A2 - nested array coercion, Array(DateTime64(9)) -> Array(DateTime) ##########
$ clickhouse-client --query "SELECT 'src', name, type FROM system.columns WHERE database='bootstrap_rehearsal_src' AND table='otel_metrics_gauge' AND name IN ('MetricName','TimeUnix','Exemplars.TimeUnix') UNION ALL SELECT 'tgt', name, type FROM system.columns WHERE database='bootstrap_rehearsal' AND table='otel_metrics_gauge' AND name IN ('MetricName','TimeUnix','Exemplars.TimeUnix') ORDER BY 1, 2 FORMAT TSV"
src  MetricName  String
src  TimeUnix  DateTime64(9)
src  Exemplars.TimeUnix  Array(DateTime64(9))
tgt  Exemplars.TimeUnix  Array(DateTime)
tgt  MetricName  LowCardinality(String)
tgt  TimeUnix  DateTime
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM bootstrap_rehearsal_src.otel_metrics_gauge"
1200
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge"
0
  [exit 0]
$ clickhouse-client --query "INSERT INTO bootstrap_rehearsal.otel_metrics_gauge (ResourceAttributes, ResourceSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, ScopeDroppedAttrCount, ScopeSchemaUrl, ServiceName, MetricName, MetricDescription, MetricUnit, Attributes, StartTimeUnix, TimeUnix, Value, Flags, `Exemplars.FilteredAttributes`, `Exemplars.TimeUnix`, `Exemplars.Value`, `Exemplars.SpanId`, `Exemplars.TraceId`) SELECT ResourceAttributes, ResourceSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, ScopeDroppedAttrCount, ScopeSchemaUrl, ServiceName, MetricName, MetricDescription, MetricUnit, Attributes, StartTimeUnix, TimeUnix, Value, Flags, `Exemplars.FilteredAttributes`, `Exemplars.TimeUnix`, `Exemplars.Value`, `Exemplars.SpanId`, `Exemplars.TraceId` FROM bootstrap_rehearsal_src.otel_metrics_gauge"
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge"
1200
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM bootstrap_rehearsal_src.otel_metrics_gauge"
1200
  [exit 0]
--- round-tripped exemplar timestamp, source (DateTime64(9)) vs target (DateTime) ---
$ clickhouse-client --query "SELECT toString(`Exemplars.TimeUnix`) FROM bootstrap_rehearsal_src.otel_metrics_gauge WHERE notEmpty(`Exemplars.TimeUnix`) ORDER BY TimeUnix LIMIT 3 FORMAT TSV"
[\'2026-08-01 12:34:56.987654321\']
[\'2026-08-01 12:36:36.987654321\']
[\'2026-08-01 12:38:16.987654321\']
  [exit 0]
$ clickhouse-client --query "SELECT toString(`Exemplars.TimeUnix`) FROM bootstrap_rehearsal.otel_metrics_gauge WHERE notEmpty(`Exemplars.TimeUnix`) ORDER BY TimeUnix LIMIT 3 FORMAT TSV"
[\'2026-08-01 12:34:56\']
[\'2026-08-01 12:36:36\']
[\'2026-08-01 12:38:16\']
  [exit 0]
--- and the same coercion on the plan's named table, otel_metrics_summary (no Exemplars: OTel Summary has none) ---
$ clickhouse-client --query "INSERT INTO bootstrap_rehearsal.otel_metrics_summary (ResourceAttributes, ResourceSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, ScopeDroppedAttrCount, ScopeSchemaUrl, ServiceName, MetricName, MetricDescription, MetricUnit, Attributes, StartTimeUnix, TimeUnix, Count, Sum, `ValueAtQuantiles.Quantile`, `ValueAtQuantiles.Value`, Flags) SELECT ResourceAttributes, ResourceSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, ScopeDroppedAttrCount, ScopeSchemaUrl, ServiceName, MetricName, MetricDescription, MetricUnit, Attributes, StartTimeUnix, TimeUnix, Count, Sum, `ValueAtQuantiles.Quantile`, `ValueAtQuantiles.Value`, Flags FROM bootstrap_rehearsal_src.otel_metrics_summary"
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM bootstrap_rehearsal.otel_metrics_summary"
1100
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM bootstrap_rehearsal_src.otel_metrics_summary"
1100
  [exit 0]
$ clickhouse-client --query "SELECT database, type FROM system.columns WHERE table='otel_metrics_gauge' AND name='Exemplars.TimeUnix' AND database IN ('bootstrap_rehearsal','bootstrap_rehearsal_src') ORDER BY database FORMAT TSV"
bootstrap_rehearsal  Array(DateTime)
bootstrap_rehearsal_src  Array(DateTime64(9))
  [exit 0]

A1 — EXCHANGE TABLES on a Replicated database, verbatim

Section titled “A1 — EXCHANGE TABLES on a Replicated database, verbatim”

Two readings are recorded. In the first, the EXCHANGE itself succeeded but both row-count queries failed with UNKNOWN_IDENTIFIER — a UNION ALL … ORDER BY scoping error in the query, not in the statement under test — so the transposition went unrecorded. Rather than reconstruct the counts from memory, the proof was re-run: a second EXCHANGE on the same pair, with the counts wrapped in a subquery, swaps the rows back and records all four values. The failed reading is left in place because a transcript that quietly drops its own failures is not evidence.

The negative control is the important half: the same statement with ON CLUSTER default is rejected with Code: 80 … ON CLUSTER is not allowed for Replicated database, matching DatabaseReplicated.cpp:1469 exactly. A cluster named default does exist in system.clusters, so the wrong form is syntactically available and would double-execute.

########## A1 - EXCHANGE TABLES on a Replicated database, no cluster clause ##########
$ clickhouse-client --multiquery < /tmp/create-new.sql   # ENGINE = ReplicatedMergeTree, no engine arguments
0  0  OK  1  0
0  1  OK  0  0
  [exit 0]
$ clickhouse-client --query "SELECT engine_full FROM system.tables WHERE database='bootstrap_rehearsal' AND name='otel_metrics_summary__new' FORMAT TSV"
ReplicatedMergeTree(\'/clickhouse/tables/{uuid}/{shard}\', \'{replica}\') PARTITION BY toDate(TimeUnix) ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix) TTL toDateTime(TimeUnix) + toIntervalDay(30) SETTINGS ttl_only_drop_parts = 1, index_granularity = 8192
  [exit 0]
--- load a deliberately different row count into __new so the swap is visible ---
$ clickhouse-client --query "INSERT INTO bootstrap_rehearsal.otel_metrics_summary__new (ResourceAttributes, ResourceSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, ScopeDroppedAttrCount, ScopeSchemaUrl, ServiceName, MetricName, MetricDescription, MetricUnit, Attributes, StartTimeUnix, TimeUnix, Count, Sum, `ValueAtQuantiles.Quantile`, `ValueAtQuantiles.Value`, Flags) SELECT ResourceAttributes, ResourceSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, ScopeDroppedAttrCount, ScopeSchemaUrl, ServiceName, MetricName, MetricDescription, MetricUnit, Attributes, StartTimeUnix, TimeUnix, Count, Sum, `ValueAtQuantiles.Quantile`, `ValueAtQuantiles.Value`, Flags FROM bootstrap_rehearsal_src.otel_metrics_summary LIMIT 300"
  [exit 0]
--- BEFORE ---
$ clickhouse-client --query "SELECT 'otel_metrics_summary' AS name, count() FROM bootstrap_rehearsal.otel_metrics_summary UNION ALL SELECT 'otel_metrics_summary__new', count() FROM bootstrap_rehearsal.otel_metrics_summary__new ORDER BY name FORMAT TSV"
Received exception from server (version 26.7.1):
Code: 47. DB::Exception: Received from localhost:9001. DB::Exception: Unknown expression identifier `name` in scope SELECT 'otel_metrics_summary__new', count() FROM bootstrap_rehearsal.otel_metrics_summary__new ORDER BY name ASC. (UNKNOWN_IDENTIFIER)
(query: SELECT 'otel_metrics_summary' AS name, count() FROM bootstrap_rehearsal.otel_metrics_summary UNION ALL SELECT 'otel_metrics_summary__new', count() FROM bootstrap_rehearsal.otel_metrics_summary__new ORDER BY name FORMAT TSV)
command terminated with exit code 47
  [exit 47]
$ clickhouse-client --query "EXCHANGE TABLES bootstrap_rehearsal.otel_metrics_summary AND bootstrap_rehearsal.otel_metrics_summary__new"
0  0  OK  1  0
0  1  OK  0  0
  [exit 0]
--- AFTER ---
$ clickhouse-client --query "SELECT 'otel_metrics_summary' AS name, count() FROM bootstrap_rehearsal.otel_metrics_summary UNION ALL SELECT 'otel_metrics_summary__new', count() FROM bootstrap_rehearsal.otel_metrics_summary__new ORDER BY name FORMAT TSV"
Received exception from server (version 26.7.1):
Code: 47. DB::Exception: Received from localhost:9001. DB::Exception: Unknown expression identifier `name` in scope SELECT 'otel_metrics_summary__new', count() FROM bootstrap_rehearsal.otel_metrics_summary__new ORDER BY name ASC. (UNKNOWN_IDENTIFIER)
(query: SELECT 'otel_metrics_summary' AS name, count() FROM bootstrap_rehearsal.otel_metrics_summary UNION ALL SELECT 'otel_metrics_summary__new', count() FROM bootstrap_rehearsal.otel_metrics_summary__new ORDER BY name FORMAT TSV)
command terminated with exit code 47
  [exit 47]
$ clickhouse-client --query "SELECT count() FROM system.tables WHERE database='bootstrap_rehearsal' AND name IN ('otel_metrics_summary','otel_metrics_summary__new')"
2
  [exit 0]
$ clickhouse-client --query "SELECT name, is_detached_permanently FROM system.tables WHERE database='bootstrap_rehearsal' AND name IN ('otel_metrics_summary','otel_metrics_summary__new') ORDER BY name FORMAT TSV"
Received exception from server (version 26.7.1):
Code: 47. DB::Exception: Received from localhost:9001. DB::Exception: Unknown expression identifier `is_detached_permanently` in scope SELECT name, is_detached_permanently FROM system.tables WHERE (database = 'bootstrap_rehearsal') AND (name IN ('otel_metrics_summary', 'otel_metrics_summary__new')) ORDER BY name ASC. (UNKNOWN_IDENTIFIER)
(query: SELECT name, is_detached_permanently FROM system.tables WHERE database='bootstrap_rehearsal' AND name IN ('otel_metrics_summary','otel_metrics_summary__new') ORDER BY name FORMAT TSV)
command terminated with exit code 47
  [exit 47]
--- negative control: the same statement WITH a cluster clause must be rejected ---
$ clickhouse-client --query "EXCHANGE TABLES bootstrap_rehearsal.otel_metrics_summary AND bootstrap_rehearsal.otel_metrics_summary__new ON CLUSTER default"
cs-clickstack-clickhouse-clickhouse-0-0-0.cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc.cluster.local  9001  80  Code: 80. DB::Exception: It\'s not initial query. ON CLUSTER is not allowed for Replicated database. (INCORRECT_QUERY) (version 26.7.1.1315 (official build))  1  0
cs-clickstack-clickhouse-clickhouse-0-1-0.cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc.cluster.local  9001  80  Code: 80. DB::Exception: It\'s not initial query. ON CLUSTER is not allowed for Replicated database. (INCORRECT_QUERY) (version 26.7.1.1315 (official build))  0  0
Received exception from server (version 26.7.1):
  [rejected as expected]
=== A1 re-capture start: 2026-08-30T20:39:09Z ===
--- state left by the first EXCHANGE (its two count queries above errored on ORDER BY scoping, so this is the first recorded reading) ---
$ clickhouse-client --query "SELECT * FROM (SELECT 'otel_metrics_summary' AS t, count() AS rows FROM bootstrap_rehearsal.otel_metrics_summary UNION ALL SELECT 'otel_metrics_summary__new' AS t, count() AS rows FROM bootstrap_rehearsal.otel_metrics_summary__new) ORDER BY t FORMAT TSV"
otel_metrics_summary  300
otel_metrics_summary__new  1100
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.detached_tables WHERE database='bootstrap_rehearsal'"
0
  [exit 0]
--- BEFORE (second EXCHANGE) ---
$ clickhouse-client --query "SELECT * FROM (SELECT 'otel_metrics_summary' AS t, count() AS rows FROM bootstrap_rehearsal.otel_metrics_summary UNION ALL SELECT 'otel_metrics_summary__new' AS t, count() AS rows FROM bootstrap_rehearsal.otel_metrics_summary__new) ORDER BY t FORMAT TSV"
otel_metrics_summary  300
otel_metrics_summary__new  1100
  [exit 0]
$ clickhouse-client --query "EXCHANGE TABLES bootstrap_rehearsal.otel_metrics_summary AND bootstrap_rehearsal.otel_metrics_summary__new"
0  0  OK  1  0
0  1  OK  0  0
  [exit 0]
--- AFTER (second EXCHANGE) ---
$ clickhouse-client --query "SELECT * FROM (SELECT 'otel_metrics_summary' AS t, count() AS rows FROM bootstrap_rehearsal.otel_metrics_summary UNION ALL SELECT 'otel_metrics_summary__new' AS t, count() AS rows FROM bootstrap_rehearsal.otel_metrics_summary__new) ORDER BY t FORMAT TSV"
otel_metrics_summary  1100
otel_metrics_summary__new  300
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.tables WHERE database='bootstrap_rehearsal' AND name IN ('otel_metrics_summary','otel_metrics_summary__new')"
2
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.detached_tables WHERE database='bootstrap_rehearsal'"
0
  [exit 0]
$ clickhouse-client --query "SELECT name, engine FROM system.tables WHERE database='bootstrap_rehearsal' AND name IN ('otel_metrics_summary','otel_metrics_summary__new') ORDER BY name FORMAT TSV"
otel_metrics_summary  ReplicatedMergeTree
otel_metrics_summary__new  ReplicatedMergeTree
  [exit 0]
=== A1 re-capture end: 2026-08-30T20:39:12Z ===


=== replica-1 read-back: 2026-08-30T20:39:23Z ===
$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-1-0 -- clickhouse-client --query "..."
otel_metrics_summary  1100
otel_metrics_summary__new  300
  [exit 0]
idx_log_attr_items  text(tokenizer = \'array\')
  [exit 0]
0
  [exit 0]

A3 — ADD INDEX … TYPE text(tokenizer = 'array') on an ALIAS column, verbatim

Section titled “A3 — ADD INDEX … TYPE text(tokenizer = 'array') on an ALIAS column, verbatim”

The 2.37.0 seed already defines idx_log_attr_items inside CREATE TABLE, so a bare ADD INDEX would have collided rather than proven anything. The index is dropped first, its absence asserted, and then added through the ALTER path — which is the path the live migration will take against default.otel_logs, where the table is altered rather than recreated.

########## A3 - ADD INDEX ... TYPE text(tokenizer = 'array') on an ALIAS column ##########
$ clickhouse-client --query "SELECT name, type, default_kind, default_expression FROM system.columns WHERE database='bootstrap_rehearsal' AND table='otel_logs' AND name='LogAttributeItems' FORMAT TSV"
LogAttributeItems  Array(String)  ALIAS  arrayMap(arr -> concat(arr.1, \'=\', arr.2), CAST(LogAttributes, \'Array(Tuple(String, String))\'))
  [exit 0]
$ clickhouse-client --query "SELECT name, type_full FROM system.data_skipping_indices WHERE database='bootstrap_rehearsal' AND table='otel_logs' AND name='idx_log_attr_items' FORMAT TSV"
idx_log_attr_items  text(tokenizer = \'array\')
  [exit 0]
$ clickhouse-client --query "ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX idx_log_attr_items"
0  0  OK  1  0
0  1  OK  0  0
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.data_skipping_indices WHERE database='bootstrap_rehearsal' AND table='otel_logs' AND name='idx_log_attr_items'"
0
  [exit 0]
$ clickhouse-client --query "ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_log_attr_items LogAttributeItems TYPE text(tokenizer = 'array')"
0  0  OK  1  0
0  1  OK  0  0
  [exit 0]
$ clickhouse-client --query "SELECT name, type_full, expr FROM system.data_skipping_indices WHERE database='bootstrap_rehearsal' AND table='otel_logs' AND name='idx_log_attr_items' FORMAT TSV"
idx_log_attr_items  text(tokenizer = \'array\')  arrayMap(arr -> concat(arr.1, \'=\', arr.2), CAST(LogAttributes, \'Array(Tuple(String, String))\'))
  [exit 0]

Cleanup and post-state, verbatim

Section titled “Cleanup and post-state, verbatim”

bootstrap_rehearsal_src is dropped. bootstrap_rehearsal is left holding the thirteen freshly created 2.37.0-shaped objects, zero plain-MergeTree tables, and the synthetic row set, for the next plan to reuse. It also retains otel_metrics_summary__new — the EXCHANGE partner, which the proof requires to still exist afterwards.

########## cleanup ##########
$ clickhouse-client --query "DROP DATABASE bootstrap_rehearsal_src SYNC"
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.databases WHERE name='bootstrap_rehearsal_src'"
0
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.tables WHERE database='bootstrap_rehearsal' AND engine IN ('MergeTree','SummingMergeTree') AND name NOT LIKE '.inner%'"
0
  [exit 0]
=== capture end: 2026-08-30T20:38:47Z ===

default was not touched, verbatim

Section titled “default was not touched, verbatim”

The live production database still carries the 2.29.0 shape: MetricName String, TimeUnix DateTime64(9), no idx_time_minmax, and no ALIAS columns on otel_logs. The committed bootstrap ConfigMap was not regenerated by this work either — the generator was run only in scratch mode, which writes plain SQL to /tmp and no ConfigMap.

=== default is untouched: 2026-08-30T20:40:13Z ===
$ clickhouse-client --query "SELECT name, type FROM system.columns WHERE database='default' AND table='otel_metrics_gauge' AND name IN (...)"
Exemplars.TimeUnix  Array(DateTime64(9))
MetricName  String
TimeUnix  DateTime64(9)
$ clickhouse-client --query "SELECT count() FROM system.data_skipping_indices WHERE database='default' AND table='otel_metrics_gauge' AND name='idx_time_minmax'"
0
$ clickhouse-client --query "SELECT count() FROM system.columns WHERE database='default' AND table='otel_logs' AND default_kind='ALIAS'"
0
$ git status --short argocd/app-configs/clickstack/bootstrap-schema-configmap.yaml
  [empty = the committed ConfigMap was not regenerated by this plan]

Interpretation

Section titled “Interpretation”
  • The generator is no longer capable of reporting success while emitting invalid SQL. The guard fires on every load-bearing deletion of a substitution arm, and names the surviving macros.
  • EXCHANGE TABLES is safe to use as the single-statement swap primitive in the metrics recreate, on the strict condition that no cluster clause is attached and that one statement covers exactly one pair.
  • The DateTime64(9) → DateTime coercion is implicit, lossy in the intended direction only (sub-second truncation), and preserves row count. No explicit CAST is needed in the migration’s INSERT … SELECT, though the explicit column list is what makes the conversion legible.
  • A full-text text index on an ALIAS column is accepted through the ALTER path at 26.7, so the otel_logs and otel_traces index migration does not require a table recreate.

Open question

Section titled “Open question”

The rehearsal ran as the in-pod default user, whose profile matches schema_admin’s (profile: default in argocd/app-configs/clickstack-chart/values.yaml), so the privilege posture is reproduced — but it is reproduced, not identical. The live migration should still assert ENGINE = ReplicatedMergeTree explicitly on every CREATE, because database_replicated_allow_only_replicated_engine is 0 for that profile and a plain MergeTree would succeed silently.

The 2.37.0 migration script rehearsal — a second capture on the same day

Section titled “The 2.37.0 migration script rehearsal — a second capture on the same day”

Everything above records the generator fix and the three ClickHouse mechanics the change window rests on. What follows records the artifact those mechanics exist to support: scripts/clickstack/migrate-schema-2.37.0.sh, run end to end against a genuinely pre-migration bootstrap_rehearsal, twice.

Why the script exists at all. 03-RESEARCH.md §B.5 establishes that the regenerated bootstrap ConfigMap is 100 % CREATE ... IF NOT EXISTS, so the PreSync Job can never recreate an existing table, drop an object upstream removed, replace a materialized-view body, or add an index. It reports success against an already-existing, differently-shaped table. That is the precise false positive this script is the answer to: 100 % of the live schema change comes from here, and nothing in the sync path can substitute for it.

What --dry-run means here. It is not a print-only mode. It is the same code path --apply runs, pointed at the scratch Replicated database bootstrap_rehearsal instead of default. The only difference between the run captured below and the change window is which database name the statements carry.

How the pre-migration state was re-established, and a deviation worth stating. 03-02-PLAN.md asked for the saved pre-edit 2.29.0 generator to be re-run against bootstrap_rehearsal. It was not used, and could not be: the generator picks the compat versus full-text seed variant from the ClickHouse version pinned in values.yaml, which today reads 26.7. Re-running the 2.29.0 generator now therefore selects the non-compat 00002/00005 seeds and produces otel_logs/otel_traces that already carry the full-text indexes — which would have made the index migration a no-op and the whole rehearsal vacuous. The committed ConfigMap header says Compat selection: CH 25.7, so the committed artifact — the one that actually created today’s live default — is the faithful pre-migration shape. The reset applies that committed SQL with default. rewritten to bootstrap_rehearsal.. The starting state captured below confirms it: 2.29.0 ORDER BY, MetricName String, TimeUnix DateTime64(9), eight bloom-filter/tokenbf_v1 indexes on otel_logs, zero ALIAS columns, and all four upstream-removed objects present.

The collector was not paused, and that is the point. 03-RESEARCH.md §A.2 recommends against pausing, because every clean way to do it costs extra ArgoCD syncs or means pausing the app-of-apps parent. To exercise that decision rather than assume it, a writer inserted rows into the live bootstrap_rehearsal.otel_metrics_gauge and …otel_metrics_summary names every three seconds for the whole run, standing in for the collector. The catch-up insert and its parity assertion are what make that safe, and both are visible in the transcript with non-zero counts (live=10 old=10 for gauge, live=15 old=15 for summary).

One correction to the plan’s parity window. 03-02-PLAN.md specifies the parity window as [t_cut, t_exch) where t_exch is SELECT now() taken after the EXCHANGE. That formulation has a race: the now() read is a separate round trip, and any row the collector writes into the new table inside that gap carries a timestamp below t_exch, inflating the live side and failing parity on an otherwise correct migration — after the exchange, which 03-RESEARCH.md §E.14 names as the point of no return. The script instead derives the window’s upper bound from the data: the newest row the old table ever received at or after t_cut. The old table is frozen from the instant of the exchange, so that value is exactly the last write before the swap, and it cannot race a clock. The catch-up INSERT itself is deliberately left unbounded above, so that no row the old table holds can be missed. Both sides are compared at second granularity, which is the precision the recreate imposes anyway.

Migration rehearsal — pre-migration reset and starting state, verbatim

Section titled “Migration rehearsal — pre-migration reset and starting state, verbatim”

Every object dropped first, then the committed 2.29.0 DDL applied, then synthetic rows spanning wider than the three-day buffer window so the deliberate tail discard is observable. otel_metrics_exponential_histogram is left empty, matching live.

=== pre-migration state: 2026-08-30T21:10:30Z ===
# bootstrap_rehearsal has been reset to the shape the COMMITTED 2.29.0 ConfigMap
# creates - the artifact that actually created today's live default database -
# re-pointed at the scratch database. See the note in the prose above on why the
# pre-edit generator could not be used for this.
$ clickhouse-client --query "SELECT name, engine FROM system.tables WHERE database='bootstrap_rehearsal' AND name NOT LIKE '.inner%' ORDER BY name FORMAT TSV"
hyperdx_sessions  ReplicatedMergeTree
metrics_ts  TimeSeries
otel_logs  ReplicatedMergeTree
otel_logs_attr_kv_rollup_15m_mv  MaterializedView
otel_logs_key_rollup_15m  ReplicatedSummingMergeTree
otel_logs_key_rollup_15m_mv  MaterializedView
otel_logs_kv_rollup_15m  ReplicatedSummingMergeTree
otel_metrics_exponential_histogram  ReplicatedMergeTree
otel_metrics_gauge  ReplicatedMergeTree
otel_metrics_histogram  ReplicatedMergeTree
otel_metrics_sum  ReplicatedMergeTree
otel_metrics_summary  ReplicatedMergeTree
otel_traces  ReplicatedMergeTree
otel_traces_key_rollup_15m  ReplicatedSummingMergeTree
otel_traces_key_rollup_15m_mv  MaterializedView
otel_traces_kv_rollup_15m  ReplicatedSummingMergeTree
otel_traces_kv_rollup_15m_mv  MaterializedView
  [exit 0]
$ clickhouse-client --query "SELECT name, type, default_kind FROM system.columns WHERE database='bootstrap_rehearsal' AND table='otel_metrics_gauge' AND name IN ('MetricName','TimeUnix','Exemplars.TimeUnix') ORDER BY name FORMAT TSV"
Exemplars.TimeUnix  Array(DateTime64(9))
MetricName  String
TimeUnix  DateTime64(9)
  [exit 0]
$ clickhouse-client --query "SELECT sorting_key FROM system.tables WHERE database='bootstrap_rehearsal' AND name='otel_metrics_gauge'"
ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
  [exit 0]
$ clickhouse-client --query "SELECT name, type_full FROM system.data_skipping_indices WHERE database='bootstrap_rehearsal' AND table='otel_logs' ORDER BY name FORMAT TSV"
idx_log_attr_key  bloom_filter(0.01)
idx_log_attr_value  bloom_filter(0.01)
idx_lower_body  tokenbf_v1(32768, 3, 0)
idx_res_attr_key  bloom_filter(0.01)
idx_res_attr_value  bloom_filter(0.01)
idx_scope_attr_key  bloom_filter(0.01)
idx_scope_attr_value  bloom_filter(0.01)
idx_trace_id  bloom_filter(0.001)
  [exit 0]
$ clickhouse-client --query "SELECT name, type_full FROM system.data_skipping_indices WHERE database='bootstrap_rehearsal' AND table='otel_traces' ORDER BY name FORMAT TSV"
idx_duration  minmax
idx_lower_span_name  tokenbf_v1(32768, 3, 0)
idx_res_attr_key  bloom_filter(0.01)
idx_res_attr_value  bloom_filter(0.01)
idx_rum_session_id  bloom_filter(0.001)
idx_span_attr_key  bloom_filter(0.01)
idx_span_attr_value  bloom_filter(0.01)
idx_trace_id  bloom_filter(0.001)
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.columns WHERE database='bootstrap_rehearsal' AND table='otel_logs' AND default_kind='ALIAS'"
0
  [exit 0]
$ clickhouse-client --query "SELECT countIf(position(create_table_query,'LogAttributes')>0) FROM system.tables WHERE database='bootstrap_rehearsal' AND name='otel_logs_attr_kv_rollup_15m_mv'"
1
  [exit 0]
# the buffer-bound control value: rows the 3-day copy will deliberately discard
$ clickhouse-client --query "SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge WHERE TimeUnix < now() - INTERVAL 4 DAY"
604
  [exit 0]
$ clickhouse-client --query "SELECT name, total_rows FROM system.tables WHERE database='bootstrap_rehearsal' AND name IN ('otel_metrics_gauge','otel_metrics_sum','otel_metrics_histogram','otel_metrics_summary','otel_metrics_exponential_histogram','otel_logs','otel_traces') ORDER BY name FORMAT TSV"
otel_logs  5000
otel_metrics_exponential_histogram  0
otel_metrics_gauge  1012
otel_metrics_histogram  1000
otel_metrics_sum  1000
otel_metrics_summary  1000
otel_traces  5000
  [exit 0]

Migration rehearsal — first --dry-run pass, verbatim

Section titled “Migration rehearsal — first --dry-run pass, verbatim”

Entire stdout, unedited. The simulated collector was writing to the live table names throughout.

CH 26.7: skipping seed files matching: _compat
skip: 00002_otel_logs_compat.sql (version-gated alternate)
skip: 00005_otel_traces_compat.sql (version-gated alternate)
Wrote /tmp/bootstrap-bootstrap_rehearsal.sql (rehearsal; not a ConfigMap)
[INFO] target database: bootstrap_rehearsal   mode: dry-run   transport: pod
[INFO] buffer window: 3 day(s)   skip-drop: false


=== Phase 0: preflight (read-only, all fatal) ===
[INFO] server 26.7.1.1315 (>= 26.2)
[INFO] unfinished mutations: 0
[INFO] replication queue depth: 0
[INFO] free space: 69.86 GiB
[INFO] engine drift (watchdog Q_DRIFT): 0
[INFO] t_cut  = 2026-08-30 21:10:48   (fixed once; never recomputed)
[INFO] t_floor= 2026-08-27 21:10:48   (t_cut minus 3 day(s))


=== Phase 1: otel_logs - 3 ALIAS columns and 8 index definitions ===
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD COLUMN IF NOT EXISTS `ResourceAttributeItems` Array(String) ALIAS arrayMap((arr) -> concat(arr
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_log_attr_key
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_log_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_lower_body
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_res_attr_key
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_res_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_scope_attr_key
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_scope_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_trace_id
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_trace_id TraceId TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_res_attr_items ResourceAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_scope_attr_items ScopeAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_log_attr_items LogAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_lower_body lower(Body) TYPE text(tokenizer = 'splitByNonAlpha')
[WARN] otel_logs text indexes are added without materializing them - deliberate (OP-02).
[WARN]   Accepted cost: full-text search over history written before this change is
[WARN]   incomplete for up to 14 days, after which the existing TTL has aged the
[WARN]   backlog out. Materializing 8 text indexes over 551.6 M rows under an 8 GiB
[WARN]   pod cap is the largest unbounded risk in this migration, so it is not done.


=== Phase 2: otel_traces - 2 ALIAS columns and exactly 2 index swaps ===
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces ADD COLUMN IF NOT EXISTS `ResourceAttributeItems` Array(String) ALIAS arrayMap((arr) -> concat(a
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces DROP INDEX IF EXISTS idx_res_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces DROP INDEX IF EXISTS idx_span_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces ADD INDEX idx_res_attr_items ResourceAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces MATERIALIZE INDEX idx_res_attr_items
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces ADD INDEX idx_span_attr_items SpanAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces MATERIALIZE INDEX idx_span_attr_items
[INFO] otel_traces is 672 MiB, so these two ARE materialized; waiting for the mutations
[INFO] otel_traces index materialization complete
[INFO]   six bloom-filter indexes were NOT touched: re-adding them without
[INFO]   materializing would strip the skip index from every existing part


=== Phase 3: replace the two rewritten materialized-view bodies ===
[DDL]  DROP VIEW IF EXISTS bootstrap_rehearsal.otel_logs_attr_kv_rollup_15m_mv
[DDL]  CREATE MATERIALIZED VIEW IF NOT EXISTS bootstrap_rehearsal.otel_logs_attr_kv_rollup_15m_mv TO bootstrap_rehearsal.otel_logs_kv_rollup_15m
[WARN] otel_logs_attr_kv_rollup_15m_mv: the 2.37.0 body is native-column-only, so the kv rollup STOPS
[WARN]   receiving map attribute keys and values. HyperDX attribute autocomplete
[WARN]   is served by the new text indexes instead. Existing rows age out on TTL.
[DDL]  DROP VIEW IF EXISTS bootstrap_rehearsal.otel_traces_kv_rollup_15m_mv
[DDL]  CREATE MATERIALIZED VIEW IF NOT EXISTS bootstrap_rehearsal.otel_traces_kv_rollup_15m_mv TO bootstrap_rehearsal.otel_traces_kv_rollup_15m
[WARN] otel_traces_kv_rollup_15m_mv: the 2.37.0 body is native-column-only, so the kv rollup STOPS
[WARN]   receiving map attribute keys and values. HyperDX attribute autocomplete
[WARN]   is served by the new text indexes instead. Existing rows age out on TTL.


=== Phase 4: recreate the five otel_metrics_* tables (ORDER BY is not ALTER-able) ===
[INFO] otel_metrics_gauge: 1023 rows before
[DDL]  CREATE TABLE IF NOT EXISTS bootstrap_rehearsal.otel_metrics_gauge__new
[INFO]   explicit column list (bootstrap_rehearsal.otel_metrics_gauge__new intersect bootstrap_rehearsal.otel_metrics_gauge): `ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes`, `ScopeDroppedAttrCount`, `ScopeSchemaUrl`, `ServiceName`, `MetricName`, `MetricDescription`, `MetricUnit`, `Attributes`, `StartTimeUnix`, `TimeUnix`, `Value`, `Flags`, `Exemplars.FilteredAttributes`, `Exemplars.TimeUnix`, `Exemplars.Value`, `Exemplars.SpanId`, `Exemplars.TraceId`
[INFO]   MetricName String -> LowCardinality(String) and Exemplars.TimeUnix
[INFO]   Array(DateTime64(9)) -> Array(DateTime) are stated by that list, not inferred
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_gauge__new (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttri
[DDL]  EXCHANGE TABLES bootstrap_rehearsal.otel_metrics_gauge AND bootstrap_rehearsal.otel_metrics_gauge__new
[INFO]   exchanged at 2026-08-30 21:11:24; otel_metrics_gauge__new now holds the OLD table and can no longer be written to
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_gauge (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes
[INFO]   catch-up parity [2026-08-30 21:10:48 .. 2026-08-30 21:11:22] (last write before the swap): live=10 old=10
[INFO]   catch-up parity OK - only now is the staging table eligible for dropping
[DDL]  DROP TABLE IF EXISTS bootstrap_rehearsal.otel_metrics_gauge__new SYNC
[INFO] otel_metrics_gauge: 1023 rows before, 318 rows after
[INFO] otel_metrics_sum: 1000 rows before
[DDL]  CREATE TABLE IF NOT EXISTS bootstrap_rehearsal.otel_metrics_sum__new
[INFO]   explicit column list (bootstrap_rehearsal.otel_metrics_sum__new intersect bootstrap_rehearsal.otel_metrics_sum): `ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes`, `ScopeDroppedAttrCount`, `ScopeSchemaUrl`, `ServiceName`, `MetricName`, `MetricDescription`, `MetricUnit`, `Attributes`, `StartTimeUnix`, `TimeUnix`, `Value`, `Flags`, `Exemplars.FilteredAttributes`, `Exemplars.TimeUnix`, `Exemplars.Value`, `Exemplars.SpanId`, `Exemplars.TraceId`, `AggregationTemporality`, `IsMonotonic`
[INFO]   MetricName String -> LowCardinality(String) and Exemplars.TimeUnix
[INFO]   Array(DateTime64(9)) -> Array(DateTime) are stated by that list, not inferred
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_sum__new (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttribu
[DDL]  EXCHANGE TABLES bootstrap_rehearsal.otel_metrics_sum AND bootstrap_rehearsal.otel_metrics_sum__new
[INFO]   exchanged at 2026-08-30 21:11:30; otel_metrics_sum__new now holds the OLD table and can no longer be written to
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_sum (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes`,
[INFO]   catch-up parity [2026-08-30 21:10:48 .. 2026-08-30 21:10:48] (last write before the swap): live=0 old=0
[INFO]   catch-up parity OK - only now is the staging table eligible for dropping
[DDL]  DROP TABLE IF EXISTS bootstrap_rehearsal.otel_metrics_sum__new SYNC
[INFO] otel_metrics_sum: 1000 rows before, 300 rows after
[INFO] otel_metrics_histogram: 1000 rows before
[DDL]  CREATE TABLE IF NOT EXISTS bootstrap_rehearsal.otel_metrics_histogram__new
[INFO]   explicit column list (bootstrap_rehearsal.otel_metrics_histogram__new intersect bootstrap_rehearsal.otel_metrics_histogram): `ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes`, `ScopeDroppedAttrCount`, `ScopeSchemaUrl`, `ServiceName`, `MetricName`, `MetricDescription`, `MetricUnit`, `Attributes`, `StartTimeUnix`, `TimeUnix`, `Count`, `Sum`, `BucketCounts`, `ExplicitBounds`, `Exemplars.FilteredAttributes`, `Exemplars.TimeUnix`, `Exemplars.Value`, `Exemplars.SpanId`, `Exemplars.TraceId`, `Flags`, `Min`, `Max`, `AggregationTemporality`
[INFO]   MetricName String -> LowCardinality(String) and Exemplars.TimeUnix
[INFO]   Array(DateTime64(9)) -> Array(DateTime) are stated by that list, not inferred
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_histogram__new (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeA
[DDL]  EXCHANGE TABLES bootstrap_rehearsal.otel_metrics_histogram AND bootstrap_rehearsal.otel_metrics_histogram__new
[INFO]   exchanged at 2026-08-30 21:11:36; otel_metrics_histogram__new now holds the OLD table and can no longer be written to
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_histogram (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttrib
[INFO]   catch-up parity [2026-08-30 21:10:48 .. 2026-08-30 21:10:48] (last write before the swap): live=0 old=0
[INFO]   catch-up parity OK - only now is the staging table eligible for dropping
[DDL]  DROP TABLE IF EXISTS bootstrap_rehearsal.otel_metrics_histogram__new SYNC
[INFO] otel_metrics_histogram: 1000 rows before, 300 rows after
[INFO] otel_metrics_summary: 1016 rows before
[DDL]  CREATE TABLE IF NOT EXISTS bootstrap_rehearsal.otel_metrics_summary__new
[INFO]   explicit column list (bootstrap_rehearsal.otel_metrics_summary__new intersect bootstrap_rehearsal.otel_metrics_summary): `ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes`, `ScopeDroppedAttrCount`, `ScopeSchemaUrl`, `ServiceName`, `MetricName`, `MetricDescription`, `MetricUnit`, `Attributes`, `StartTimeUnix`, `TimeUnix`, `Count`, `Sum`, `ValueAtQuantiles.Quantile`, `ValueAtQuantiles.Value`, `Flags`
[INFO]   MetricName String -> LowCardinality(String) and Exemplars.TimeUnix
[INFO]   Array(DateTime64(9)) -> Array(DateTime) are stated by that list, not inferred
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_summary__new (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAtt
[DDL]  EXCHANGE TABLES bootstrap_rehearsal.otel_metrics_summary AND bootstrap_rehearsal.otel_metrics_summary__new
[INFO]   exchanged at 2026-08-30 21:11:41; otel_metrics_summary__new now holds the OLD table and can no longer be written to
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_summary (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttribut
[INFO]   catch-up parity [2026-08-30 21:10:48 .. 2026-08-30 21:11:40] (last write before the swap): live=15 old=15
[INFO]   catch-up parity OK - only now is the staging table eligible for dropping
[DDL]  DROP TABLE IF EXISTS bootstrap_rehearsal.otel_metrics_summary__new SYNC
[INFO] otel_metrics_summary: 1016 rows before, 317 rows after
[INFO] otel_metrics_exponential_histogram: 0 rows before
[DDL]  CREATE TABLE IF NOT EXISTS bootstrap_rehearsal.otel_metrics_exponential_histogram__new
[INFO]   explicit column list (bootstrap_rehearsal.otel_metrics_exponential_histogram__new intersect bootstrap_rehearsal.otel_metrics_exponential_histogram): `ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes`, `ScopeDroppedAttrCount`, `ScopeSchemaUrl`, `ServiceName`, `MetricName`, `MetricDescription`, `MetricUnit`, `Attributes`, `StartTimeUnix`, `TimeUnix`, `Count`, `Sum`, `Scale`, `ZeroCount`, `PositiveOffset`, `PositiveBucketCounts`, `NegativeOffset`, `NegativeBucketCounts`, `Exemplars.FilteredAttributes`, `Exemplars.TimeUnix`, `Exemplars.Value`, `Exemplars.SpanId`, `Exemplars.TraceId`, `Flags`, `Min`, `Max`, `AggregationTemporality`
[INFO]   MetricName String -> LowCardinality(String) and Exemplars.TimeUnix
[INFO]   Array(DateTime64(9)) -> Array(DateTime) are stated by that list, not inferred
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_exponential_histogram__new (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersi
[DDL]  EXCHANGE TABLES bootstrap_rehearsal.otel_metrics_exponential_histogram AND bootstrap_rehearsal.otel_metrics_exponential_histogram__new
[INFO]   exchanged at 2026-08-30 21:11:46; otel_metrics_exponential_histogram__new now holds the OLD table and can no longer be written to
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_exponential_histogram (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`,
[INFO]   catch-up parity [2026-08-30 21:10:48 .. 2026-08-30 21:10:48] (last write before the swap): live=0 old=0
[INFO]   catch-up parity OK - only now is the staging table eligible for dropping
[DDL]  DROP TABLE IF EXISTS bootstrap_rehearsal.otel_metrics_exponential_histogram__new SYNC
[INFO] otel_metrics_exponential_histogram: 0 rows before, 0 rows after


=== Phase 5: drop the four objects removed upstream at 2.32.0 ===
[DDL]  DROP VIEW IF EXISTS bootstrap_rehearsal.otel_logs_key_rollup_15m_mv
[DDL]  DROP VIEW IF EXISTS bootstrap_rehearsal.otel_traces_key_rollup_15m_mv
[DDL]  DROP TABLE IF EXISTS bootstrap_rehearsal.otel_logs_key_rollup_15m SYNC
[DDL]  DROP TABLE IF EXISTS bootstrap_rehearsal.otel_traces_key_rollup_15m SYNC


=== Phase 6: postflight (read from system tables, never from this summary) ===
[INFO] engine drift (watchdog Q_DRIFT): 0
[INFO] the four upstream-removed objects are absent
[INFO] all five otel_metrics_* sorting keys match the 2.37.0 seed
[INFO] otel_logs and otel_traces each carry 8 indexes
[INFO] otel_logs index name set matches the 2.37.0 seed exactly
[INFO] otel_traces carries both *_items indexes and neither *_value index


=== Summary ===
database:            bootstrap_rehearsal
mode:                dry-run
t_cut:               2026-08-30 21:10:48
buffer floor:        2026-08-27 21:10:48 (3 day window)
DDL statements issued: 57
steps already applied: 0


metrics table                           rows before   rows after  disposition
otel_metrics_gauge                             1023          318  recreated
otel_metrics_sum                               1000          300  recreated
otel_metrics_histogram                         1000          300  recreated
otel_metrics_summary                           1016          317  recreated
otel_metrics_exponential_histogram                0            0  recreated


[INFO] migration complete: 57 DDL statement(s) issued against bootstrap_rehearsal

Migration rehearsal — post-migration state read from system tables, verbatim

Section titled “Migration rehearsal — post-migration state read from system tables, verbatim”

Read from system.tables, system.columns and system.data_skipping_indices, never from the script’s own summary. The sorting key and the eight index names are compared as strings against the 2.37.0 seed the generator emitted, not eyeballed.

=== post-migration state, first run: 2026-08-30T21:12:15Z ===
$ awk '...otel_metrics_gauge...' /tmp/bootstrap-bootstrap_rehearsal.sql | sed -n 's/^ORDER BY (\(.*\))$/\1/p'
ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
  [the 2.37.0 seed's ORDER BY, extracted from the generator's own output]
$ clickhouse-client --query "SELECT sorting_key = 'ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix' AS matches_seed, sorting_key FROM system.tables WHERE database='bootstrap_rehearsal' AND name='otel_metrics_gauge' FORMAT TSV"
1  ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
  [exit 0]


$ awk '...otel_logs...' /tmp/bootstrap-bootstrap_rehearsal.sql | rg '^\s*INDEX ' | awk '{print $2}' | sort | tr '\n' ','
idx_log_attr_items,idx_log_attr_key,idx_lower_body,idx_res_attr_items,idx_res_attr_key,idx_scope_attr_items,idx_scope_attr_key,idx_trace_id
  [the 2.37.0 seed's eight otel_logs index names]
$ clickhouse-client --query "SELECT count() AS n, arrayStringConcat(arraySort(groupArray(name)),',') = 'idx_log_attr_items,idx_log_attr_key,idx_lower_body,idx_res_attr_items,idx_res_attr_key,idx_scope_attr_items,idx_scope_attr_key,idx_trace_id' AS set_equals_seed, arrayStringConcat(arraySort(groupArray(name)),',') AS names FROM system.data_skipping_indices WHERE database='bootstrap_rehearsal' AND table='otel_logs' FORMAT TSV"
8  1  idx_log_attr_items,idx_log_attr_key,idx_lower_body,idx_res_attr_items,idx_res_attr_key,idx_scope_attr_items,idx_scope_attr_key,idx_trace_id
  [exit 0]
$ clickhouse-client --query "SELECT name, type_full FROM system.data_skipping_indices WHERE database='bootstrap_rehearsal' AND table='otel_logs' ORDER BY name FORMAT TSV"
idx_log_attr_items  text(tokenizer = \'array\')
idx_log_attr_key  text(tokenizer = \'array\')
idx_lower_body  text(tokenizer = \'splitByNonAlpha\')
idx_res_attr_items  text(tokenizer = \'array\')
idx_res_attr_key  text(tokenizer = \'array\')
idx_scope_attr_items  text(tokenizer = \'array\')
idx_scope_attr_key  text(tokenizer = \'array\')
idx_trace_id  text(tokenizer = \'array\')
  [exit 0]


# otel_traces: exactly two swapped, the six unchanged bloom filters left alone
$ clickhouse-client --query "SELECT count() FROM system.data_skipping_indices WHERE database='bootstrap_rehearsal' AND table='otel_traces'"
8
  [exit 0]
$ clickhouse-client --query "SELECT name, type_full FROM system.data_skipping_indices WHERE database='bootstrap_rehearsal' AND table='otel_traces' ORDER BY name FORMAT TSV"
idx_duration  minmax
idx_lower_span_name  tokenbf_v1(32768, 3, 0)
idx_res_attr_items  text(tokenizer = \'array\')
idx_res_attr_key  bloom_filter(0.01)
idx_rum_session_id  bloom_filter(0.001)
idx_span_attr_items  text(tokenizer = \'array\')
idx_span_attr_key  bloom_filter(0.01)
idx_trace_id  bloom_filter(0.001)
  [exit 0]


# the four objects removed upstream at 2.32.0
$ clickhouse-client --query "SELECT count() FROM system.tables WHERE database='bootstrap_rehearsal' AND name IN ('otel_logs_key_rollup_15m','otel_logs_key_rollup_15m_mv','otel_traces_key_rollup_15m','otel_traces_key_rollup_15m_mv')"
0
  [exit 0]


# the type conversions the explicit column list carried across
$ clickhouse-client --query "SELECT name, type FROM system.columns WHERE database='bootstrap_rehearsal' AND table='otel_metrics_gauge' AND name IN ('MetricName','TimeUnix','Exemplars.TimeUnix') ORDER BY name FORMAT TSV"
Exemplars.TimeUnix  Array(DateTime)
MetricName  LowCardinality(String)
TimeUnix  DateTime
  [exit 0]


# buffer bound: 604 rows sat past the 4-day mark before the run
$ clickhouse-client --query "SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge WHERE TimeUnix < now() - INTERVAL 4 DAY"
0
  [exit 0]


# engine drift, the watchdog's own query
$ clickhouse-client --query "SELECT count() FROM system.tables WHERE database='bootstrap_rehearsal' AND engine IN ('MergeTree','SummingMergeTree') AND name NOT LIKE '.inner%'"
0
  [exit 0]


# the MV bodies are native-column-only now
$ clickhouse-client --query "SELECT name, position(create_table_query,'LogAttributes')>0 OR position(create_table_query,'SpanAttributes')>0 AS still_maps FROM system.tables WHERE database='bootstrap_rehearsal' AND name IN ('otel_logs_attr_kv_rollup_15m_mv','otel_traces_kv_rollup_15m_mv') ORDER BY name FORMAT TSV"
otel_logs_attr_kv_rollup_15m_mv  0
otel_traces_kv_rollup_15m_mv  0
  [exit 0]


# no staging table survived
$ clickhouse-client --query "SELECT count() FROM system.tables WHERE database='bootstrap_rehearsal' AND name LIKE '%__new'"
0
  [exit 0]

Migration rehearsal — the identical second run, verbatim

Section titled “Migration rehearsal — the identical second run, verbatim”

Byte-identical command, immediately after the first.

CH 26.7: skipping seed files matching: _compat
skip: 00002_otel_logs_compat.sql (version-gated alternate)
skip: 00005_otel_traces_compat.sql (version-gated alternate)
Wrote /tmp/bootstrap-bootstrap_rehearsal.sql (rehearsal; not a ConfigMap)
[INFO] target database: bootstrap_rehearsal   mode: dry-run   transport: pod
[INFO] buffer window: 3 day(s)   skip-drop: false


=== Phase 0: preflight (read-only, all fatal) ===
[INFO] server 26.7.1.1315 (>= 26.2)
[INFO] unfinished mutations: 0
[INFO] replication queue depth: 0
[INFO] free space: 69.97 GiB
[INFO] engine drift (watchdog Q_DRIFT): 0
[INFO] t_cut  = 2026-08-30 21:12:36   (fixed once; never recomputed)
[INFO] t_floor= 2026-08-27 21:12:36   (t_cut minus 3 day(s))


=== Phase 1: otel_logs - 3 ALIAS columns and 8 index definitions ===
[STEP] SKIP: already applied - otel_logs already carries the 2.37.0 ALIAS columns and all 8 text indexes


=== Phase 2: otel_traces - 2 ALIAS columns and exactly 2 index swaps ===
[STEP] SKIP: already applied - otel_traces already carries idx_res_attr_items and idx_span_attr_items
[INFO]   the six unchanged bloom-filter indexes were left alone, as intended


=== Phase 3: replace the two rewritten materialized-view bodies ===
[STEP] SKIP: already applied - otel_logs_attr_kv_rollup_15m_mv already carries the native-column-only 2.37.0 body
[STEP] SKIP: already applied - otel_traces_kv_rollup_15m_mv already carries the native-column-only 2.37.0 body


=== Phase 4: recreate the five otel_metrics_* tables (ORDER BY is not ALTER-able) ===
[STEP] SKIP: already applied - otel_metrics_gauge already carries the 2.37.0 sorting key (318 rows)
[STEP] SKIP: already applied - otel_metrics_sum already carries the 2.37.0 sorting key (300 rows)
[STEP] SKIP: already applied - otel_metrics_histogram already carries the 2.37.0 sorting key (300 rows)
[STEP] SKIP: already applied - otel_metrics_summary already carries the 2.37.0 sorting key (317 rows)
[STEP] SKIP: already applied - otel_metrics_exponential_histogram already carries the 2.37.0 sorting key (0 rows)


=== Phase 5: drop the four objects removed upstream at 2.32.0 ===
[STEP] SKIP: already applied - all four upstream-removed objects are already absent


=== Phase 6: postflight (read from system tables, never from this summary) ===
[INFO] engine drift (watchdog Q_DRIFT): 0
[INFO] the four upstream-removed objects are absent
[INFO] all five otel_metrics_* sorting keys match the 2.37.0 seed
[INFO] otel_logs and otel_traces each carry 8 indexes
[INFO] otel_logs index name set matches the 2.37.0 seed exactly
[INFO] otel_traces carries both *_items indexes and neither *_value index


=== Summary ===
database:            bootstrap_rehearsal
mode:                dry-run
t_cut:               2026-08-30 21:12:36
buffer floor:        2026-08-27 21:12:36 (3 day window)
DDL statements issued: 0
steps already applied: 10


metrics table                           rows before   rows after  disposition
otel_metrics_gauge                              318          318  already applied
otel_metrics_sum                                300          300  already applied
otel_metrics_histogram                          300          300  already applied
otel_metrics_summary                            317          317  already applied
otel_metrics_exponential_histogram                0            0  already applied


[INFO] no DDL was issued - the schema was already at 2.37.0

Migration rehearsal — catch-up parity negative control, verbatim

Section titled “Migration rehearsal — catch-up parity negative control, verbatim”

Two readings are kept, the vacuous one first. A control that cannot fail proves nothing, and the first window chosen could not.

=== parity negative control, query level: 2026-08-30T21:07:14Z ===
# t_cut of the --skip-drop run above: 2026-08-30 21:05:46
# parity bound the script derived for otel_metrics_gauge: 2026-08-30 21:06:21
# t_floor (3-day buffer floor): 2026-08-27 21:05:46


########## the window the script actually used - the pair is EQUAL ##########
$ clickhouse-client --query "SELECT (SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge WHERE TimeUnix >= toDateTime('2026-08-30 21:05:46') AND toDateTime(TimeUnix) <= toDateTime('2026-08-30 21:06:21')) AS live, (SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge__new WHERE TimeUnix >= toDateTime('2026-08-30 21:05:46') AND toDateTime(TimeUnix) <= toDateTime('2026-08-30 21:06:21')) AS old, live = old AS parity_ok FORMAT TSV"
10  10  1
  [exit 0]


########## the same comparison with a deliberately wrong lower bound ##########
# t_floor instead of t_cut: the live table holds only the 3-day buffer copy,
# the old table holds every pre-migration row, so the pair MUST be unequal.
$ clickhouse-client --query "SELECT (SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge WHERE TimeUnix >= toDateTime('2026-08-27 21:05:46') AND toDateTime(TimeUnix) <= toDateTime('2026-08-30 21:06:21')) AS live, (SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge__new WHERE TimeUnix >= toDateTime('2026-08-27 21:05:46') AND toDateTime(TimeUnix) <= toDateTime('2026-08-30 21:06:21')) AS old, live = old AS parity_ok FORMAT TSV"
318  318  1
  [exit 0]


########## why that reading is vacuous, and the corrected control ##########
# The t_floor lower bound is NOT a discriminating window: the buffer copy put
# exactly the [t_floor, t_cut) rows into the live table, so both sides hold the
# same 318 rows and the pair is equal for a reason that has nothing to do with
# the parity check working. A control that cannot fail proves nothing. The
# discriminating window reaches BELOW t_floor, where the live table deliberately
# holds nothing and the old table still holds every pre-migration row.
$ clickhouse-client --query "SELECT (SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge WHERE TimeUnix >= now() - INTERVAL 15 DAY AND toDateTime(TimeUnix) <= toDateTime('2026-08-30 21:06:21')) AS live, (SELECT count() FROM bootstrap_rehearsal.otel_metrics_gauge__new WHERE TimeUnix >= now() - INTERVAL 15 DAY AND toDateTime(TimeUnix) <= toDateTime('2026-08-30 21:06:21')) AS old, live = old AS parity_ok FORMAT TSV"
318  1024  0
  [exit 0]

Migration rehearsal — parity guard mutation control, verbatim

Section titled “Migration rehearsal — parity guard mutation control, verbatim”

Stronger than the query-level control: the guard’s consequence is what matters, so the script’s parity window is mutated and the script itself is run. The mutant is asserted to differ from the source, and to parse and be non-empty, before any verdict is drawn — a mutation that failed to land would otherwise be indistinguishable from a guard that did not fire. Tail of the mutant run, then the state it left behind.

[INFO] target database: bootstrap_rehearsal   mode: dry-run   transport: pod
[INFO] buffer window: 3 day(s)   skip-drop: false


=== Phase 0: preflight (read-only, all fatal) ===
[INFO] server 26.7.1.1315 (>= 26.2)
[INFO] unfinished mutations: 0
[INFO] replication queue depth: 0
[INFO] free space: 69.81 GiB
[INFO] engine drift (watchdog Q_DRIFT): 0
[INFO] t_cut  = 2026-08-30 21:08:49   (fixed once; never recomputed)
[INFO] t_floor= 2026-08-27 21:08:49   (t_cut minus 3 day(s))


=== Phase 1: otel_logs - 3 ALIAS columns and 8 index definitions ===
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD COLUMN IF NOT EXISTS `ResourceAttributeItems` Array(String) ALIAS arrayMap((arr) -> concat(arr
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_log_attr_key
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_log_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_lower_body
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_res_attr_key
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_res_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_scope_attr_key
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_scope_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs DROP INDEX IF EXISTS idx_trace_id
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_trace_id TraceId TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_res_attr_items ResourceAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_scope_attr_items ScopeAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_log_attr_items LogAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_logs ADD INDEX idx_lower_body lower(Body) TYPE text(tokenizer = 'splitByNonAlpha')
[WARN] otel_logs text indexes are added without materializing them - deliberate (OP-02).
[WARN]   Accepted cost: full-text search over history written before this change is
[WARN]   incomplete for up to 14 days, after which the existing TTL has aged the
[WARN]   backlog out. Materializing 8 text indexes over 551.6 M rows under an 8 GiB
[WARN]   pod cap is the largest unbounded risk in this migration, so it is not done.


=== Phase 2: otel_traces - 2 ALIAS columns and exactly 2 index swaps ===
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces ADD COLUMN IF NOT EXISTS `ResourceAttributeItems` Array(String) ALIAS arrayMap((arr) -> concat(a
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces DROP INDEX IF EXISTS idx_res_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces DROP INDEX IF EXISTS idx_span_attr_value
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces ADD INDEX idx_res_attr_items ResourceAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces MATERIALIZE INDEX idx_res_attr_items
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces ADD INDEX idx_span_attr_items SpanAttributeItems TYPE text(tokenizer = 'array')
[DDL]  ALTER TABLE bootstrap_rehearsal.otel_traces MATERIALIZE INDEX idx_span_attr_items
[INFO] otel_traces is 672 MiB, so these two ARE materialized; waiting for the mutations
[INFO] otel_traces index materialization complete
[INFO]   six bloom-filter indexes were NOT touched: re-adding them without
[INFO]   materializing would strip the skip index from every existing part


=== Phase 3: replace the two rewritten materialized-view bodies ===
[DDL]  DROP VIEW IF EXISTS bootstrap_rehearsal.otel_logs_attr_kv_rollup_15m_mv
[DDL]  CREATE MATERIALIZED VIEW IF NOT EXISTS bootstrap_rehearsal.otel_logs_attr_kv_rollup_15m_mv TO bootstrap_rehearsal.otel_logs_kv_rollup_15m
[WARN] otel_logs_attr_kv_rollup_15m_mv: the 2.37.0 body is native-column-only, so the kv rollup STOPS
[WARN]   receiving map attribute keys and values. HyperDX attribute autocomplete
[WARN]   is served by the new text indexes instead. Existing rows age out on TTL.
[DDL]  DROP VIEW IF EXISTS bootstrap_rehearsal.otel_traces_kv_rollup_15m_mv
[DDL]  CREATE MATERIALIZED VIEW IF NOT EXISTS bootstrap_rehearsal.otel_traces_kv_rollup_15m_mv TO bootstrap_rehearsal.otel_traces_kv_rollup_15m
[WARN] otel_traces_kv_rollup_15m_mv: the 2.37.0 body is native-column-only, so the kv rollup STOPS
[WARN]   receiving map attribute keys and values. HyperDX attribute autocomplete
[WARN]   is served by the new text indexes instead. Existing rows age out on TTL.


=== Phase 4: recreate the five otel_metrics_* tables (ORDER BY is not ALTER-able) ===
[INFO] otel_metrics_gauge: 1012 rows before
[DDL]  CREATE TABLE IF NOT EXISTS bootstrap_rehearsal.otel_metrics_gauge__new
[INFO]   explicit column list (bootstrap_rehearsal.otel_metrics_gauge__new intersect bootstrap_rehearsal.otel_metrics_gauge): `ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes`, `ScopeDroppedAttrCount`, `ScopeSchemaUrl`, `ServiceName`, `MetricName`, `MetricDescription`, `MetricUnit`, `Attributes`, `StartTimeUnix`, `TimeUnix`, `Value`, `Flags`, `Exemplars.FilteredAttributes`, `Exemplars.TimeUnix`, `Exemplars.Value`, `Exemplars.SpanId`, `Exemplars.TraceId`
[INFO]   MetricName String -> LowCardinality(String) and Exemplars.TimeUnix
[INFO]   Array(DateTime64(9)) -> Array(DateTime) are stated by that list, not inferred
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_gauge__new (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttri
[DDL]  EXCHANGE TABLES bootstrap_rehearsal.otel_metrics_gauge AND bootstrap_rehearsal.otel_metrics_gauge__new
[INFO]   exchanged at 2026-08-30 21:09:29; otel_metrics_gauge__new now holds the OLD table and can no longer be written to
[DDL]  INSERT INTO bootstrap_rehearsal.otel_metrics_gauge (`ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes
[INFO]   catch-up parity [2026-08-30 21:08:49 .. 2026-08-30 21:08:49] (last write before the swap): live=306 old=1012
[ERROR] catch-up parity FAILED for otel_metrics_gauge: live=306 old=1012. Nothing dropped; otel_metrics_gauge__new still holds every old row.

Migration rehearsal — default was not touched, verbatim

Section titled “Migration rehearsal — default was not touched, verbatim”
=== default is untouched by this plan: 2026-08-30T21:14:07Z ===
$ clickhouse-client --query "SELECT name, type FROM system.columns WHERE database='default' AND table='otel_metrics_gauge' AND name IN ('MetricName','TimeUnix','Exemplars.TimeUnix') ORDER BY name FORMAT TSV"
Exemplars.TimeUnix  Array(DateTime64(9))
MetricName  String
TimeUnix  DateTime64(9)
  [exit 0]
$ clickhouse-client --query "SELECT sorting_key FROM system.tables WHERE database='default' AND name='otel_metrics_gauge'"
ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
  [exit 0]
$ clickhouse-client --query "SELECT arrayStringConcat(arraySort(groupArray(name)),',') FROM system.data_skipping_indices WHERE database='default' AND table='otel_logs'"
idx_log_attr_key,idx_log_attr_value,idx_lower_body,idx_res_attr_key,idx_res_attr_value,idx_scope_attr_key,idx_scope_attr_value,idx_trace_id
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.columns WHERE database='default' AND table='otel_logs' AND default_kind='ALIAS'"
0
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.tables WHERE database='default' AND name IN ('otel_logs_key_rollup_15m','otel_logs_key_rollup_15m_mv','otel_traces_key_rollup_15m','otel_traces_key_rollup_15m_mv')"
4
  [exit 0]
$ clickhouse-client --query "SELECT count() FROM system.tables WHERE database='default' AND name LIKE '%__new'"
0
  [exit 0]
$ git status --short argocd/app-configs/clickstack/bootstrap-schema-configmap.yaml
  [exit 0]
  [empty = the committed ConfigMap was not regenerated by this plan either]

Interpretation of the migration rehearsal

Section titled “Interpretation of the migration rehearsal”
  • The script performs a real forward migration, not a print. Starting from the committed 2.29.0 shape it issued 57 DDL statements and left bootstrap_rehearsal at the 2.37.0 shape object for object: four upstream-removed objects gone, five metrics tables on the new sorting key with MetricName LowCardinality(String), TimeUnix DateTime and Exemplars.TimeUnix Array(DateTime), otel_logs on the eight-name text-index set taken from the generator’s own output, otel_traces carrying both *_items indexes and neither *_value index with its six unchanged bloom filters untouched, both materialized-view bodies native-column-only, and zero engine drift.
  • The buffer bound discarded the tail deliberately, and provably. 604 otel_metrics_gauge rows sat past the four-day mark before the run and 0 after it, while the tables that received no live writes went from 1,000 rows to exactly the 300 inside the three-day window. A silently failed copy would have shown 0 rows after, not 300.
  • The second run is a provable no-op. It exited 0, reported all ten steps already applied, and printed DDL statements issued: 0 — and, independently of anything the script says about itself, the (name, metadata_modification_time) multiset for every object in the database was byte-identical before and after (diff exit 0, cmp byte-identical). The exchange step is never reached when the staging table is absent, because the sorting-key probe short-circuits the whole table first.
  • The parity assertion is not a vacuous green. Its first negative control was vacuous and that reading is kept above: bounding below at t_floor compares two sets that the buffer copy made identical by construction, so it reports equal for a reason unrelated to the check working. The discriminating control reaches below t_floor and reports 318 / 1024 / 0. The mutation control goes further and proves the guard’s consequence: with the window mutated, the script aborts with catch-up parity FAILED, exits 1, and drops nothing — the old table still holds all 1,012 pre-migration rows. That is the property that matters, because the drop is where the tail becomes unrecoverable.
  • default was not touched. Every statement in this plan named bootstrap_rehearsal. The live database still carries MetricName String, TimeUnix DateTime64(9), the 2.29.0 ORDER BY, the eight compat indexes, zero ALIAS columns, all four upstream-removed objects, and no staging table. The committed ConfigMap is unmodified — regenerating it is 03-03’s job.

What the change window still has to find out

Section titled “What the change window still has to find out”

The rehearsal database holds thousands of rows; default.otel_logs holds 551.6 M in 17.42 GiB and default.otel_metrics_gauge three days of real metrics at 11.69 GiB. Nothing here measures duration at that scale, and the three-day copy is the step whose runtime sets how long the catch-up window has to be. The rehearsal proves the mechanics and the guards; it does not predict the clock.