ClickStack 2.37.0 live schema migration — the one-way gate, the measured discard, and the operator's ruling, 2026-08-30

Captured against context fzymgc-house with kubectl against namespaces clickstack, longhorn-system and argocd, and with clickhouse-client inside the ClickHouse pod cs-clickstack-clickhouse-clickhouse-0-0-0 in namespace clickstack (server 26.7.1.1315), reading system.parts, system.tables, system.mutations, system.replication_queue, system.disks and default.otel_metrics_* in the live default database, plus the backups.longhorn.io custom resources in longhorn-system and the clickstack ArgoCD Application. Every reading in this file was taken read-only.

Why this file exists: D-12 makes the ClickStack metrics recreate an operator-run imperative migration, because the clickstack-schema-bootstrap PreSync Job is 100 % CREATE … IF NOT EXISTS and can never recreate an existing table, drop a removed object, replace a materialized-view body or add an index. The Job reports SUCCESS against an already-existing, differently-shaped table. This file is the record of that operator-run window: what the cluster looked like before it opened, what was irreversibly given up, who decided that and on what measured numbers, and what the running database looked like afterwards. It is a dated record of work performed on one day rather than standing operational prose, so it lives in evidence/ rather than in docs/operations/clickstack.md. The runbook says what to do; this file says what happened when it was done. It pairs with docs/operations/evidence/2026-08-30-clickstack-schema-rehearsal-2.37.0.md, which proved the three ClickHouse mechanics this migration depends on against a scratch database.

Where the planning artifacts cited below live. Passages here cite phase planning artifacts by bare filename (03-RESEARCH.md, 03-CONTEXT.md, 03-06-PLAN.md). Every one of them lives in .planning/workstreams/cluster-health/phases/03-clickstack-currency/. That directory is a planning record kept in the repository and deliberately not published to the documentation site, so a reader on the site should expect to open those from a checkout rather than follow a link from here.

Status of this record. The one-way gate is closed and recorded below. The migration run itself, the merge, and the post-sync verification are appended as they happen.

Stage	State
One-way gate — readings, decision, disposition	Closed — recorded below
migrate-schema-2.37.0.sh --apply --skip-drop against default, attempt 1	Failed closed 2026-08-30T23:21:52Z — schema_admin lacks ALTER; nothing applied, state verified clean
Privilege remediation	Resolved — declarative grants landed on main and synced; system.grants now carries the four ALTER grants for schema_admin on default
migrate-schema-2.37.0.sh --apply --skip-drop against default, attempt 2	Partial 2026-08-31T00:13:59Z — phases 0–3 applied; otel_metrics_gauge migrated correctly; aborted on a false catch-up parity failure before the four remaining metrics tables
Catch-up parity gate	Fixed — f3383d8de; equality replaced by containment, proven two-sided against the real gauge pair
migrate-schema-2.37.0.sh --apply --skip-drop, resume for the four remaining metrics tables	Pending — operator-run, otel_metrics_gauge is skipped by sorting key
One merge, one ArgoCD sync	Pending
Post-sync verification against live state	Pending

The decision, verbatim

The gate was presented with the discard measured in rows and gigabytes, with the fact that the Longhorn restore path has never been exercised in this estate stated plainly, and with the --skip-drop alternative offered. The operator’s answer:

DECISION: proceed-skip-drop
Migrate the live `default` schema to the 2.37.0 shape, and RETAIN the staging tables
(`otel_metrics_*__new`). Do NOT drop them in this run.

STAGING-TABLE DISPOSITION:
Owner: Sean.
When: after plan 03-11 completes — i.e. at phase end, once the MongoDB 5.0 -> 8.0 ladder
is done and the whole ClickStack plane is verified on current versions.
Rationale as given: keeps the fallback available through every remaining live change in
this phase, and lands well inside the ~27-day horizon after which the unchanged 30-day
TTL would remove that data anyway.

Option chosen: proceed-skip-drop — the second of the three offered (proceed, proceed-skip-drop, stop).

Mechanically this means the migration is run as scripts/clickstack/migrate-schema-2.37.0.sh --apply --skip-drop. --skip-drop is the flag the committed script actually defines (migrate-schema-2.37.0.sh:171); under it Phase 4 performs the staging create, the bounded copy, the EXCHANGE, the catch-up insert and the parity assertion exactly as it otherwise would, and then declines the final DROP TABLE default.otel_metrics_*__new SYNC, reporting the disposition recreated (staging kept) per table. Every other phase is unaffected: the four upstream-removed objects in Phase 5 are still dropped, because those are not the tail this decision is protecting.

Consequence to hold onto. After the exchange, default.otel_metrics_<x>__new holds the pre-migration table — all 30 days of it — and default.otel_metrics_<x> holds the new 2.37.0-shaped table containing only the 3-day continuity buffer plus everything written since the cut. The staging name is therefore the fallback, and nothing writes to it after the exchange.

Staging-table disposition

Field	Value
Tables	default.otel_metrics_gauge__new, otel_metrics_sum__new, otel_metrics_histogram__new, otel_metrics_summary__new, otel_metrics_exponential_histogram__new
Owner	Sean
Due	After plan 03-11 completes — phase end, once the MongoDB 5.0 → 8.0 ladder is done and the ClickStack plane is verified on current versions
Action	DROP TABLE default.otel_metrics_*__new SYNC
Backstop	The unchanged 30-day metrics TTL ages the retained tail out by approximately 2026-09-26 regardless

This is recorded because a forgotten staging table is a slow leak, and D-04’s discard was a deliberate decision rather than an accident. It is written here rather than left in a chat transcript so the obligation outlives the session that created it.

Executed 2026-09-19, on Sean’s instruction. The due condition was met: Phase 03 closed on 2026-09-02 with the MongoDB ladder at 8.0.29 / FCV 8.0. Pre-flight checks:

• default is a Replicated database, so one DROP reaches both replicas.
• No table or view depends on the five staging tables.
• No new rows had arrived since the exchange: the newest TimeUnix in the gauge staging table was 2026-08-31 01:25:57. The September part timestamps come from TTL merges, not inserts.

Each table was dropped with DROP TABLE default.<t> SYNC as the in-pod default user on replica 0-0. Every statement returned OK from both replicas. Afterwards both replicas list zero __new tables. Free disk rose from 56.8 GiB to 90.3 GiB per replica (71% to 54% used). Gauge ingest lag read 3–4 s immediately after the drop. The pre-migration rows from 2026-08-20 to 2026-08-28 are discarded, as D-04 intended.

The correction to the plan’s footprint claim, by measurement

03-06-PLAN.md states, as a cons line against the proceed-skip-drop option, that retaining the staging tables “doubles the metrics footprint on a disk already around 65 per cent used.” That claim is wrong, and it is wrong by a factor of about seven. It is corrected here rather than repeated, because the operator’s decision was taken against the measured number and the plan’s number would have argued against the choice that was actually made.

The claim would be true if the new table were a full copy of the old one. It is not. D-04 fixes the continuity buffer at 3 days, so the INSERT … SELECT copies only the rows in [t_cut − 3 days, t_cut). The retained staging table is the original table — it is not an extra copy of anything — and the only new bytes on the disk are the 3-day buffer.

Measured from system.parts on 2026-08-30, at partition granularity (PARTITION BY toDate(TimeUnix)):

Quantity	Measured
All five otel_metrics_* tables, total	7.89 B rows / 88.15 GiB
Rows inside the 3-day buffer (the copy)	1.08 B rows / 12.17 GiB
Footprint before the migration	~88.15 GiB
Footprint after, with --skip-drop	~100.4 GiB (88.15 retained under the staging name + 12.17 new)
Delta	+12.17 GiB, about +14 %
Free space before	69.31 GiB of 195.80 GiB (64.6 % used)
Free space after, projected	~57.1 GiB

A doubling would have been +88 GiB against 69 GiB free — it would not have fit, and the script’s own 20 GiB free-space preflight would have been the wrong guard for it. The actual cost is one 3-day buffer, held for the remainder of the phase, against a disk that ends the window with roughly 57 GiB free. The plan’s cons line described a 30-day copy that this migration never makes.

The measured discard, verbatim

The tail outside the continuity buffer is what proceed would have destroyed. Under proceed-skip-drop it is retained, not discarded — it survives under the staging name until the disposition above is executed, or until the 30-day TTL ages it out, whichever comes first. It is recorded here because it is the number the operator was shown, and because the decision only means something against it.

Buffer floor t_floor = t_cut − 3 days = 2026-08-27 22:09:37. Split computed from system.parts by partition.

| Table                              | Rows total    | Bytes total | Rows kept (3d) | Bytes kept | Rows in tail  | Bytes in tail |
|------------------------------------|---------------|-------------|----------------|------------|---------------|---------------|
| otel_metrics_gauge                 | 4.62 B        | 52.38 GiB   |    615.95 M    |  7.11 GiB  | 4.00 B        | 45.27 GiB     |
| otel_metrics_sum                   | 2.31 B        | 22.94 GiB   |    312.12 M    |  3.10 GiB  | 2.00 B        | 19.84 GiB     |
| otel_metrics_histogram             | 930.99 M      | 12.30 GiB   |    147.85 M    |  1.90 GiB  | 783.13 M      | 10.41 GiB     |
| otel_metrics_summary               |  29.31 M      | 535.01 MiB  |      3.81 M    | 70.23 MiB  |  25.51 M      | 464.78 MiB    |
| otel_metrics_exponential_histogram |         0     |          0  |          0     |         0  |         0     |            0  |
| TOTAL                              | 7,892,751,382 | 88.15 GiB   | 1,079,778,822  | 12.17 GiB  | 6,812,972,560 | 75.97 GiB     |

Oldest partition: 2026-07-31.  Newest: 2026-08-30.
TTL unchanged at 30 days on both sides, so the tail expires by ~2026-09-26 regardless.

Re-measured independently at 2026-08-30 22:43 before this record was written, as a check that the figures had not drifted materially under continuing ingest:

   ┌─table──────────────────┬─rows_total─┬─bytes_total─┬─rows_kept_3d─┬─bytes_kept_3d─┐
1. │ otel_metrics_gauge     │ 4623137276 │ 52.43 GiB   │    619603100 │ 7.16 GiB      │
2. │ otel_metrics_histogram │  932313056 │ 12.32 GiB   │    149178086 │ 1.91 GiB      │
3. │ otel_metrics_sum       │ 2314975594 │ 22.96 GiB   │    314179167 │ 3.12 GiB      │
4. │ otel_metrics_summary   │   29335751 │ 535.47 MiB  │      3828764 │ 70.69 MiB     │
   └────────────────────────┴────────────┴─────────────┴──────────────┴───────────────┘

The differences are the half hour of ingest between the two readings. otel_metrics_exponential_histogram holds no parts at all and so does not appear in the second reading; it exists, is empty, and is recreated along with the other four because its sorting key changes too.

The D-07 backup gate, verbatim

D-07’s falsifying condition is that if a fresh Longhorn backup does not exist for the ClickHouse volumes, the gate is unmet and the window does not open. It was met.

The enumeration is sorted by status.snapshotCreatedAt, never by name. That is not a stylistic preference: system-backup-* sorts above backup-*, so a name-descending sort silently returns a weeks-old system-backup as though it were the newest. That mistake was made and caught during the discussion that produced this phase, and it is recorded in 03-CONTEXT.md D-07 for that reason. The sort field is visible in the command below.

kubectl -n longhorn-system get backups.longhorn.io -o json \
| jq -r '.items[] | select(.spec.snapshotName|test("^daily-ba-"))
         | [.status.volumeName, .status.snapshotCreatedAt, .status.state, .status.size,
            .spec.snapshotName] | @tsv' \
| sort -t$'\t' -k1,1 -k2,2r \
| awk -F'\t' '!seen[$1]++'

The PVC-to-volume map came from kubectl -n clickstack get pvc -o json (.spec.volumeName).

| Volume (PVC)                                  | Newest daily-backup  | State     | Size       |
|-----------------------------------------------|---------------------|-----------|------------|
| clickhouse-storage … clickhouse-0-0-0          | 2026-08-30T07:03:27Z | Completed | 170.99 GiB |
| clickhouse-storage … clickhouse-0-1-0          | 2026-08-30T07:09:57Z | Completed | 168.48 GiB |
| clickhouse-storage … keeper-0-0                | 2026-08-30T07:03:58Z | Completed |   0.64 GiB |
| clickhouse-storage … keeper-1-0                | 2026-08-30T07:02:46Z | Completed |   0.62 GiB |
| clickhouse-storage … keeper-2-0                | 2026-08-30T07:01:03Z | Completed |   0.62 GiB |
| data-volume … mongodb-0                        | 2026-08-30T07:05:10Z | Completed |   0.92 GiB |
| data-volume … mongodb-1                        | 2026-08-30T07:00:33Z | Completed |   0.88 GiB |
| data-volume … mongodb-2                        | 2026-08-30T07:02:50Z | Completed |   0.91 GiB |
| logs-volume … mongodb-0                        | 2026-08-30T07:03:22Z | Completed |   0.22 GiB |
| logs-volume … mongodb-1                        | 2026-08-30T07:04:34Z | Completed |   0.20 GiB |
| logs-volume … mongodb-2                        | 2026-08-30T07:09:06Z | Completed |   0.21 GiB |

Gate MET: both ClickHouse volumes’ newest recurring-job backup is Completed and dated within the current day, as are all six MongoDB volumes. The enumeration was run twice — once when the gate was first assessed and once independently before this record was written — and returned identical timestamps and sizes both times.

Boundary, stated rather than glossed. Engram vzkfcz6tg8 records restore-not-yet-exercised for this estate. These backups complete; a restore from them has never been performed here. D-07 rests on a well-evidenced expectation, not a demonstrated capability, and this record does not claim otherwise. The operator was told this before answering.

The pre-window ClickHouse and workload readings, verbatim

Preflight, read from the live default database. Every one of these is also asserted by the migration script’s own Phase 0, which is fatal on any of them, so these are the operator’s independent look at the same facts rather than a substitute for the script’s check.

reading_taken_at:        2026-08-30 22:08:58        re-read 2026-08-30 22:43:25
server_version:          26.7.1.1315                26.7.1.1315     (>= 26.2 required)
unfinished_mutations:    0                          0               (must be 0)
replication_queue_depth: 0                          1               (must be < 10)
engine_drift:            0                          0               (must be 0)
free_space:              69.31 GiB of 195.80        69.12 GiB       (must be >= 20 GiB)

The replication queue moving from 0 to 1 between the two reads is ordinary background replication on a live cluster; the script’s threshold is 10.

Workload baseline — the values the post-migration verification is compared against:

otel-collector pod cs-otel-collector-7864f5c4fc-9mgvc, container otel-collector:
  restartCount = 0                     <-- the A4 baseline; a restart here falsifies A4
  image docker.clickhouse.com/clickhouse/clickstack-otel-collector:2.32.0
cs-clickstack-app-6f74fc98c7-4nqq2 / app:
  restartCount = 2, image hyperdx/hyperdx:2.32.0
Ingest freshness (the watchdog's own Q_STALE, threshold 300): 5 s   re-read: 3 s
Watchdog CronJob clickstack-watchdog: spec.suspend = false, schedule */5 * * * *
Last watchdog Job clickstack-watchdog-29802125: succeeded=1 at 2026-08-30T22:05:04Z
ArgoCD Application clickstack: Synced Healthy

Context tables, not recreated by this migration and recorded so a later reader can tell what was and was not in scope: otel_logs 550.63 M rows / 17.42 GiB; otel_traces 15.17 M rows / 678.11 MiB.

Pre-migration schema state, confirming the migration had not yet run when this record was opened — all five metrics tables still carry the 2.29.0 sorting key and no __new staging table exists:

   ┌─name───────────────────────────────┬─engine──────────────┬─sorting_key──────────────────────────────────────────────────────────┐
1. │ otel_metrics_exponential_histogram │ ReplicatedMergeTree │ ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix) │
2. │ otel_metrics_gauge                 │ ReplicatedMergeTree │ ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix) │
3. │ otel_metrics_histogram             │ ReplicatedMergeTree │ ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix) │
4. │ otel_metrics_sum                   │ ReplicatedMergeTree │ ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix) │
5. │ otel_metrics_summary               │ ReplicatedMergeTree │ ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix) │
   └────────────────────────────────────┴─────────────────────┴──────────────────────────────────────────────────────────────────────┘

The 2.37.0 target sorting key is ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix. That change is why ORDER BY cannot be ALTERed and why the tables are recreated rather than altered.

The copy clock, measured

Nothing had measured how long the copy would take. It is measured here, because “how long is the ingest gap” determines how the watchdog behaves during the window, and an unmeasured clock is how an operator mistakes correct behaviour for a fault.

Read-only full-column scans of the 3-day window, timed with clickhouse-client --time:

otel_metrics_summary    ( 70.23 MiB /   3.81 M rows) ->   1.265 s
otel_metrics_histogram  (  1.90 GiB / 147.85 M rows) ->  62.089 s   (~31 MiB/s, ~2.38 M rows/s)

Extrapolated read-only floor for the whole 12.17 GiB copy: ~400 s (~6.6 min). That is a floor, not an estimate. The real INSERT … SELECT additionally compresses, writes, replicates and re-sorts every block, because the sorting key genuinely changes. Realistic whole-run expectation: 15–40 minutes, including the otel_traces index materialization in Phase 2, which the script waits on for up to 10 minutes.

Expectation correction: expect several missed heartbeats, not one

03-06-PLAN.md and 03-RESEARCH.md §E.13 both say to “expect the watchdog to fire once.” Against the measured clock that is optimistic, and an operator holding it would read correct behaviour as a fault. Expect several missed heartbeats and several uptime-kuma alerts.

The mechanism, read from argocd/app-configs/clickstack/watchdog-cronjob.yaml:

Q_STALE = SELECT dateDiff('second', max(TimeUnix), now())
          FROM default.otel_metrics_gauge WHERE TimeUnix > now() - INTERVAL 1 HOUR
threshold: [ "$STALE" -lt 300 ]        schedule: */5 * * * *

t_cut is fixed once, in the script’s Phase 0, before the otel_logs index work (Phase 1), the otel_traces index materialization (Phase 2, up to 10 minutes) and the materialized-view swaps (Phase 3). otel_metrics_gauge — the one table the watchdog reads — is not exchanged until Phase 4. So at the instant of its exchange, the newest row in the live table is t_cut, and STALE reads now() − t_cut, which by then includes every preceding phase. That is plausibly 20–40 minutes, which is four to eight consecutive five-minute ticks over the 300 s threshold, until the catch-up INSERT lands and STALE drops back to single-digit seconds.

This is correct behaviour. Do not treat the repeated alerts as a fault, and do not silence the watchdog. The same CronJob runs Q_DRIFT — the live detector for the plain-MergeTree staging table mistake, which is the single most likely way this window goes wrong and is a mistake schema_admin is unguarded against. Suspending it would remove the only instrument watching the thing most likely to break, at exactly the moment it could break.

What was rejected and why, in one line each

• proceed — one window, no residue, disk returns to a single copy immediately. Rejected because the 6.81 B-row / 75.97 GiB tail would then be recoverable only by a whole-volume Longhorn restore that has never been exercised here, and plans 03-07 through 03-11 still have live changes to make against this same plane.
• stop — nothing applied, every wave-2 commit reverts for free. Rejected because CS-01 does not land and the schema drift the phase exists to repair persists.

Attempt 1 failed closed on a privilege gap, 2026-08-30T23:21:52Z

The operator ran the approved command. The script’s own Phase 0 preflight passed in full and is preserved in the capture at /tmp/clickstack-migrate-2.37.0-20260830T232148Z.log:

[INFO] target database: default   mode: apply   transport: pod
[INFO] buffer window: 3 day(s)   skip-drop: true
[INFO] server 26.7.1.1315 (>= 26.2)
[INFO] unfinished mutations: 0
[INFO] replication queue depth: 0
[INFO] free space: 68.90 GiB
[INFO] engine drift (watchdog Q_DRIFT): 0
[INFO] t_cut  = 2026-08-30 23:21:52   (fixed once; never recomputed)
[INFO] t_floor= 2026-08-27 23:21:52   (t_cut minus 3 day(s))

Phase 1 then died on its first DDL, before any statement had modified anything:

=== Phase 1: otel_logs - 3 ALIAS columns and 8 index definitions ===
[DDL]  ALTER TABLE default.otel_logs ADD COLUMN IF NOT EXISTS `ResourceAttributeItems` ...
Received exception from server (version 26.7.1):
Code: 497. DB::Exception: Received from localhost:9001. DB::Exception: schema_admin: Not enough
privileges. To execute this query, it's necessary to have the grant
ALTER ADD COLUMN(ResourceAttributeItems) ON default.otel_logs. (ACCESS_DENIED)
command terminated with exit code 241

The one-way door was never approached. The failure is entirely pre-EXCHANGE: Phase 1 is the first phase that writes, ALTER is its first statement, and it was refused. t_cut was derived but never used.

The state is clean, verified rather than inferred

Read from the running database, not from the script’s exit code:

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT count() FROM system.columns WHERE database='default' AND table='otel_logs' AND default_kind='ALIAS'"
0

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT name FROM system.tables WHERE database='default' AND name LIKE '%__new%'"
(no rows)

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT count() FROM system.data_skipping_indices WHERE database='default' AND table='otel_traces' AND name IN ('idx_res_attr_items','idx_span_attr_items')"
0

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT name, engine, sorting_key FROM system.tables WHERE database='default' AND name LIKE 'otel_metrics%' ORDER BY name"
otel_metrics_exponential_histogram  ReplicatedMergeTree  ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
otel_metrics_gauge                  ReplicatedMergeTree  ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
otel_metrics_histogram              ReplicatedMergeTree  ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
otel_metrics_sum                    ReplicatedMergeTree  ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
otel_metrics_summary                ReplicatedMergeTree  ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)

Zero ALIAS columns on otel_logs, no staging table under any __new name, neither replacement index on otel_traces, and all five metrics tables still carrying the 2.29.0 sorting key (toUnixTimestamp64Nano(TimeUnix)) on ReplicatedMergeTree. This is a failed attempt, not a partial migration, and there is nothing to unwind. The re-run is a clean first run.

Root cause, measured

schema_admin’s live grants, read from system.grants rather than from the chart values:

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT user_name, access_type, database, grant_option FROM system.grants WHERE user_name='schema_admin'"
schema_admin  SHOW    \N  0
schema_admin  SELECT  \N  0
schema_admin  INSERT  \N  0
schema_admin  CREATE  \N  0
schema_admin  DROP    \N  0
schema_admin  BACKUP  \N  0

Six privileges, all ON *.*, no ALTER of any kind. They correspond exactly, verb for verb, to the single codified line at argocd/app-configs/clickstack-chart/values.yaml:

- query: "GRANT CREATE, DROP, INSERT, SELECT, SHOW, BACKUP ON *.*"

The live set matching the codified set exactly is itself a useful finding: the chart values are the sole source of schema_admin’s privileges, so the gap is a gap in the codified line and nowhere else.

The privileges the script actually needs, enumerated against the script

Every ALTER in scripts/clickstack/migrate-schema-2.37.0.sh, by line, with no others present in the file:

Line	Statement	Privilege	Phase
429	ALTER TABLE default.otel_logs ADD COLUMN IF NOT EXISTS …	ALTER ADD COLUMN	1
434	ALTER TABLE default.otel_logs DROP INDEX IF EXISTS …	ALTER DROP INDEX	1
438	ALTER TABLE default.otel_logs ADD INDEX …	ALTER ADD INDEX	1
470	ALTER TABLE default.otel_traces ADD COLUMN IF NOT EXISTS …	ALTER ADD COLUMN	2
473	ALTER TABLE default.otel_traces DROP INDEX IF EXISTS …	ALTER DROP INDEX	2
478	ALTER TABLE default.otel_traces ADD INDEX …	ALTER ADD INDEX	2
479	ALTER TABLE default.otel_traces MATERIALIZE INDEX …	ALTER MATERIALIZE INDEX	2

Four distinct subtypes. All four exist in this server’s privilege vocabulary at COLUMN or TABLE level, so ON default.* is a legal and sufficient scope — a blanket ALTER ON *.* is not required:

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT privilege, level FROM system.privileges WHERE privilege IN ('ALTER ADD COLUMN','ALTER ADD INDEX','ALTER DROP INDEX','ALTER MATERIALIZE INDEX') ORDER BY privilege"
ALTER ADD COLUMN         COLUMN
ALTER ADD INDEX          TABLE
ALTER DROP INDEX         TABLE
ALTER MATERIALIZE INDEX  TABLE

Phases 3 through 6 need nothing beyond what schema_admin already holds. They issue CREATE TABLE / CREATE MATERIALIZED VIEW, DROP TABLE / DROP VIEW, INSERT … SELECT, and one EXCHANGE TABLES per metrics table. EXCHANGE deserves the explicit check because it looks like it might need its own verb: it does not. This server’s system.privileges enumeration contains no RENAME or EXCHANGE privilege at all, so EXCHANGE TABLES is authorised by CREATE TABLE plus DROP TABLE, both already held ON *.*. The script issues no OPTIMIZE, no TRUNCATE, no SYSTEM …, no DETACH/ATTACH and no ALTER UPDATE/ALTER DELETE.

Why the rehearsal did not catch this — profile is not grants

docs/operations/evidence/2026-08-30-clickstack-schema-rehearsal-2.37.0.md, under Open question, says:

The rehearsal ran as the in-pod default user, whose profile matches schema_admin’s (profile: default in argocd/app-configs/clickstack-chart/values.yaml), so the privilege posture is reproduced — but it is reproduced, not identical.

That sentence reads as a statement of coverage and is in fact a statement of the gap. A ClickHouse profile and a ClickHouse grant are unrelated subsystems. A profile carries resource settings — memory ceilings, timeouts, readonly, database_replicated_allow_only_replicated_engine. Grants carry privileges. Two users sharing profile: default tells you their query-resource behaviour matches and tells you nothing whatsoever about what either is permitted to execute. Here the two users’ grants are as far apart as they can be: default holds ALL, schema_admin holds six verbs and no ALTER.

So assumption A3 was genuinely proven as a mechanic — the DDL shapes are accepted by ClickHouse 26.7, twice, including a provable no-op second run — and never proven as the user who would run it. The rehearsal’s own hedge (“reproduced, not identical”) was correct and was the right instinct attached to the wrong axis.

This is the same shape as the standing rule that an offline render is evidence about production only when its environment matches the consumer’s, and that the equivalence must be asserted explicitly rather than assumed. The equivalence asserted here was real — the profiles do match — but it was asserted on an axis that does not carry the property under test.

The durable lesson, for any future rehearsal of a privileged change in this estate: rehearse as the principal that will run it, or state in writing which privileges were not exercised and why. A rehearsal that runs as a more-privileged user proves the mechanics and, by construction, cannot prove the authorisation. Both halves need evidence; only one of them was gathered.

The operator’s ruling on the remediation, verbatim

DECISION: Apply the four ALTER privileges to schema_admin imperatively NOW, scoped to default.*, AND codify the same change in argocd/app-configs/clickstack-chart/values.yaml in this same pull request.

RATIONALE AS GIVEN: this mirrors D-12’s own logic — imperative for the live change, declarative for the desired state. It preserves the plan’s migration-before-sync ordering (Truth 2), which a pure-GitOps path would invert, because ArgoCD serves main and every wave-2 commit is still unmerged on the milestone branch: making the grant live via sync would also carry the chart 3.2.0 bump and the 2.37.0 images. It keeps least privilege — scoped to default.*, only the four verbs the script actually uses, rather than a blanket ALTER ON *.*.

The declarative half is applied

argocd/app-configs/clickstack-chart/values.yaml now carries a second, additive - query: entry under schema_admin.grants, deliberately not folded into the existing ON *.* line so that the narrower scope stays visible in the diff and in the rendered config:

- query: "GRANT CREATE, DROP, INSERT, SELECT, SHOW, BACKUP ON *.*"
- query: "GRANT ALTER ADD COLUMN, ALTER ADD INDEX, ALTER DROP INDEX, ALTER MATERIALIZE INDEX ON default.*"

The imperative half cannot be performed on this cluster

This is the finding that blocks the re-run, and it was established by attempting the grant, not by reasoning about it. Issued as the in-pod default user, which holds ALL:

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "GRANT ALTER ADD COLUMN, ALTER ADD INDEX, ALTER DROP INDEX, ALTER MATERIALIZE INDEX ON default.* TO schema_admin"
Received exception from server (version 26.7.1):
Code: 497. DB::Exception: default: Not enough privileges. The required privileges have been granted,
but without grant option. To execute this query, it's necessary to have the grant
ALTER ADD COLUMN ON default.* WITH GRANT OPTION. (ACCESS_DENIED)
command terminated with exit code 241

default holds the privilege and still cannot pass it on, because ClickHouse requires WITH GRANT OPTION to grant. No principal on this cluster has it:

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT user_name, role_name, access_type, database, grant_option FROM system.grants WHERE grant_option=1"
(no rows)

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT user_name, access_type, database, grant_option FROM system.grants WHERE user_name='default'"
default  ALL  \N  0

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT name, storage FROM system.users ORDER BY name"
app             users_xml
default         users_xml
mcp_readonly    users_xml
mcp_readwrite   users_xml
operator        users_xml
otelcollector   users_xml
schema_admin    users_xml

Every user is users_xml-stored. Their grants are rendered by the Altinity operator from the chart values into /etc/clickhouse-server/users.d/99-extra-users-config.yaml inside the pod, whose schema_admin entry is byte-for-byte the codified line. A replicated access directory exists at precedence 2 — that is where the orphaned SQL-created MCP twins lived — but reaching it still requires a grantor holding GRANT OPTION, and none exists.

Therefore the ruling’s first half is not merely inadvisable here; it has no mechanism. The declarative path is not the preferred way to change schema_admin’s privileges on this cluster. It is the only way. The ruling’s second half stands unchanged and is committed.

What this costs, and the choice it forces

The rationale for going imperative was to preserve Truth 2 — migration strictly before sync — so that the only mismatched window is a 2.32.0 collector writing into 2.37.0-shaped columns and never the reverse. With no imperative path, granting the privilege necessarily involves an ArgoCD sync, and the CS-01 branch’s values.yaml also carries the two 2.37.0 image tags. Merging the milestone branch to make the grant live would therefore roll the collector to 2.37.0 before the schema migration runs — precisely the inversion D-11 and Truth 2 exist to prevent.

The open options, none of which an agent may choose:

• A privilege-only pull request straight to main. Carries the values.yaml grants line and nothing else, syncs, makes the grant live, and leaves the image and chart pins on the milestone branch untouched so Truth 2 holds. Costs a second merge; whether that offends D-11 is a judgement about whether a privilege prerequisite is part of CS-01’s content or a precondition of it.
• Run the migration as the in-pod default user. Requires no privilege change and no sync at all, and is exactly the principal the rehearsal proved end to end. It is broader than least privilege, and it needs a change to the script: line 237 hard-requires a non-empty SCHEMA_ADMIN_PASSWORD for --apply, and in-pod default authenticates without one.
• Merge CS-01 first, then migrate. Cheapest in mechanism, and it inverts Truth 2. Recorded for completeness, not recommended.

Nothing further has been applied to the live database. t_cut from attempt 1 is discarded; the script fixes it once per run in Phase 0, so the re-run derives its own and the three-day continuity buffer moves with it. The measured discard recorded earlier in this file was taken against a floor computed from the attempt-1 t_cut and will shift by the elapsed interval — the shape of the decision does not change, but the exact row and byte counts at re-run time will differ slightly.

The privilege remediation landed, verbatim

The declarative fix was merged straight to main as a privilege-only change and synced, leaving the image and chart pins on the milestone branch untouched so Truth 2 — migration strictly before sync — still holds. The grant is asserted at the interpreter, not from the ConfigMap and not from a sync status, because that distinction is the whole reason attempt 1 got as far as it did:

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT user_name, access_type, database FROM system.grants WHERE user_name='schema_admin' ORDER BY access_type"
schema_admin  SHOW                     \N
schema_admin  SELECT                   \N
schema_admin  INSERT                   \N
schema_admin  ALTER ADD COLUMN         default
schema_admin  ALTER ADD INDEX          default
schema_admin  ALTER DROP INDEX         default
schema_admin  ALTER MATERIALIZE INDEX  default
schema_admin  CREATE                   \N
schema_admin  DROP                     \N
schema_admin  BACKUP                   \N

Attempt 2 — phases 0–3 applied, otel_metrics_gauge migrated, then a halt, verbatim

Capture: /tmp/clickstack-migrate-2.37.0-20260831T001359Z.log, 78 lines. Phases 0, 1, 2 and 3 completed. Phase 4 migrated otel_metrics_gauge and stopped on the catch-up parity assertion.

[INFO] target database: default   mode: apply   transport: pod
[INFO] buffer window: 3 day(s)   skip-drop: true
[WARN] this run targets the LIVE database - the EXCHANGE is the point of no return

=== Phase 0: preflight (read-only, all fatal) ===
[INFO] server 26.7.1.1315 (>= 26.2)
[INFO] unfinished mutations: 0
[INFO] replication queue depth: 0
[INFO] free space: 71.75 GiB
[INFO] engine drift (watchdog Q_DRIFT): 0
[INFO] t_cut  = 2026-08-31 00:14:02   (fixed once; never recomputed)
[INFO] t_floor= 2026-08-28 00:14:02   (t_cut minus 3 day(s))
...
=== Phase 4: recreate the five otel_metrics_* tables (ORDER BY is not ALTER-able) ===
[INFO] otel_metrics_gauge: 4606240875 rows before
[DDL]  CREATE TABLE IF NOT EXISTS default.otel_metrics_gauge__new
[INFO]   explicit column list (default.otel_metrics_gauge__new intersect default.otel_metrics_gauge): `ResourceAttributes`, `ResourceSchemaUrl`, `ScopeName`, `ScopeVersion`, `ScopeAttributes`, `ScopeDroppedAttrCount`, `ScopeSchemaUrl`, `ServiceName`, `MetricName`, `MetricDescription`, `MetricUnit`, `Attributes`, `StartTimeUnix`, `TimeUnix`, `Value`, `Flags`, `Exemplars.FilteredAttributes`, `Exemplars.TimeUnix`, `Exemplars.Value`, `Exemplars.SpanId`, `Exemplars.TraceId`
[INFO]   MetricName String -> LowCardinality(String) and Exemplars.TimeUnix
[INFO]   Array(DateTime64(9)) -> Array(DateTime) are stated by that list, not inferred
[DDL]  INSERT INTO default.otel_metrics_gauge__new (`ResourceAttributes`, ...
[DDL]  EXCHANGE TABLES default.otel_metrics_gauge AND default.otel_metrics_gauge__new
[INFO]   exchanged at 2026-08-31 01:26:03; otel_metrics_gauge__new now holds the OLD table and can no longer be written to
[DDL]  INSERT INTO default.otel_metrics_gauge (`ResourceAttributes`, ...
[INFO]   catch-up parity [2026-08-31 00:14:02 .. 2026-08-31 01:25:57] (last write before the swap): live=7721841 old=7719386
[ERROR] catch-up parity FAILED for otel_metrics_gauge: live=7721841 old=7719386. Nothing dropped; otel_metrics_gauge__new still holds every old row.

The names inverted at the exchange, and that inversion is the single most important fact in this section. EXCHANGE TABLES swaps the two names, not the two tables. After 01:26:03:

• default.otel_metrics_gauge is the new 2.37.0-shaped table. It is live and taking writes.
• default.otel_metrics_gauge__new is the old 2.29.0 table. It is frozen; nothing writes to it.

Read from system.tables rather than inferred from the transcript:

$ kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-0-0-0 -- clickhouse-client \
    --query "SELECT name, sorting_key FROM system.tables WHERE database='default' AND name LIKE 'otel_metrics%' ORDER BY name"
otel_metrics_exponential_histogram  ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
otel_metrics_gauge                  ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
otel_metrics_gauge__new             ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
otel_metrics_histogram              ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
otel_metrics_sum                    ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)
otel_metrics_summary                ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)

otel_metrics_gauge alone carries the 2.37.0 sorting key. The four remaining tables are untouched. The column types confirm the same story from the other direction:

$ ... --query "SELECT table, name, type FROM system.columns WHERE database='default' AND table IN ('otel_metrics_gauge','otel_metrics_gauge__new') AND name IN ('TimeUnix','StartTimeUnix','MetricName','Exemplars.TimeUnix') ORDER BY table, name"
otel_metrics_gauge       Exemplars.TimeUnix  Array(DateTime)
otel_metrics_gauge       MetricName          LowCardinality(String)
otel_metrics_gauge       StartTimeUnix       DateTime
otel_metrics_gauge       TimeUnix            DateTime
otel_metrics_gauge__new  Exemplars.TimeUnix  Array(DateTime64(9))
otel_metrics_gauge__new  MetricName          String
otel_metrics_gauge__new  StartTimeUnix       DateTime64(9)
otel_metrics_gauge__new  TimeUnix            DateTime64(9)

Two behaviour changes that are already live, stated plainly

Phases 1 and 3 of attempt 2 applied and are in force on the running database now, ahead of the merge. Both are deliberate and both cost something. Neither is a defect; both are recorded here rather than left to be discovered.

OP-02 — the otel_logs text indexes were added without materializing them. Full-text search over history written before this window is incomplete for up to fourteen days, after which the existing TTL has aged the backlog out. Materializing eight text indexes over 551.6 M rows under an 8 GiB pod cap is the largest unbounded risk in this migration, so it is deliberately not done. The eight indexes apply to everything written from this point on. otel_traces is 672 MiB, so its two new indexes were materialized and the script waited for the mutations to drain.

RESEARCH §A.3.4 — the rewritten rollup views are native-column-only. Both otel_logs_attr_kv_rollup_15m_mv and otel_traces_kv_rollup_15m_mv now carry the 2.37.0 body, which no longer feeds map attribute keys and values into the kv rollup. HyperDX attribute autocomplete is served by the new text indexes instead. Existing rollup rows age out on their TTL. The spot check that distinguishes “the indexes landed” from “the rollup stopped feeding and nothing replaced it” is opening the log-search attribute autocomplete in HyperDX and confirming it still offers map attribute keys — that check belongs to the post-sync verification and has not been performed yet.

The copy is complete — four measurements, verbatim

The parity failure is false. Four independent readings, all taken read-only against the live pair over the exact window the script asserted, [2026-08-31 00:14:02 .. 2026-08-31 01:25:57].

1. The counts reproduce exactly, and the entire excess is in one minute.

$ ... "SELECT (SELECT count() FROM default.otel_metrics_gauge WHERE <window>) AS live_tail,
         (SELECT count() FROM default.otel_metrics_gauge__new WHERE <window>) AS old_tail"
live_tail  old_tail
7721841    7719386

A per-minute diff across the whole window, printing only minutes where the two sides disagree:

$ ... "SELECT m, sum(l) AS live, sum(o) AS old, sum(l)-sum(o) AS delta FROM (
         SELECT toStartOfMinute(toDateTime(TimeUnix)) AS m, 1 AS l, 0 AS o FROM default.otel_metrics_gauge      WHERE <window>
         UNION ALL
         SELECT toStartOfMinute(toDateTime(TimeUnix)) AS m, 0 AS l, 1 AS o FROM default.otel_metrics_gauge__new WHERE <window>
       ) GROUP BY m HAVING delta != 0 ORDER BY m"
m                    live    old    delta
2026-08-31 01:25:00  102009  99554  2455

One row. Every other minute in the 72-minute window matches exactly. The divergence is confined to the minute immediately before the exchange at 01:26:03 — which is precisely where a write-time leak would put it, and precisely where a broken copy would not.

2 and 3. Excluding that minute, the two sides are identical in count and in composition.

$ ... "SELECT 'live', sum(Value), countIf(<not the 01:25 minute>), uniqExactIf(toString(MetricName), <not the 01:25 minute>),
              groupBitXorIf(cityHash64(toString(ServiceName), toString(MetricName), toString(Value),
                                       toString(toDateTime(TimeUnix)), toString(Attributes)), <not the 01:25 minute>)
       FROM default.otel_metrics_gauge WHERE <window>
       UNION ALL SELECT 'old', ... FROM default.otel_metrics_gauge__new WHERE <window>"

side  sum_value  rows_ex_1125  metricnames_ex_1125  xor_ex_1125
live  inf        7619832       1890                 2372688998560378365
old   inf        7619832       1890                 2372688998560378365

Identical row count, identical distinct-MetricName count, and an identical order-independent XOR checksum. The checksum is a genuine instrument, not a tautology: a negative control removing a single live row changes it.

4. Multiset containment over the whole window, including the disputed minute, holds. This is the strongest of the four and it needs no exclusion at all — see the fix below, which reports 0.

Two dead instruments, and why each is dead, verbatim

Recorded because both look like obvious checks and both are worthless here.

sum(Value) is dead: it returns inf on both tables. Visible in the table above. A gauge series in this estate holds an infinite value, so the sum saturates and carries no information at all — including no information about a table that lost half its rows.

A hash over raw TimeUnix is dead: it differs across the two tables for an entirely benign reason. The 2.37.0 seed defines TimeUnix DateTime and StartTimeUnix DateTime, down from DateTime64(9); MetricName moves from String to LowCardinality(String) and Exemplars.TimeUnix from Array(DateTime64(9)) to Array(DateTime). Sub-second truncation is the intended schema change, proved as assumption A2 of the 03-01 rehearsal. Any cross-table comparison must normalise. Measured, on data proved intact by the three readings above:

un-normalised fingerprint  ->  2152687 phantom "missing" rows
normalised fingerprint     ->        0

That is not a rounding artefact; it is two million false positives from omitting one cast.

Root cause: the bound is on event time, the leak is in write time

03-02 identified correctly that taking the parity window’s upper bound from SELECT now() after the EXCHANGE races the collector, and moved the bound onto max(TimeUnix) over the frozen old table. That was a real improvement and it is recorded in 03-02-SUMMARY.md as deviation 1. It is not enough, and the reason is a category difference rather than a matter of degree:

• The bound is on event time — the timestamp a data point carries.
• The leak is in write time — when a batch arrives.

An OTel collector batches and flushes on an interval. A batch flushed after the exchange can carry data points stamped before any bound the script is able to compute, because those points were observed before the swap and merely delivered after it. They land in the new table, inside the parity window, and the frozen old table never saw them. No event-time bound can exclude them. The one minute of divergence measured above is exactly this population.

The plan prohibits the only mechanism that would remove it: “never scale the collector to zero or edit its replica count to create a quiet window.” So under this plan’s own constraints strict equality is unsatisfiable by construction, not merely flaky. It would have failed on otel_metrics_sum, otel_metrics_histogram, otel_metrics_summary and otel_metrics_exponential_histogram in turn, each time past the point of no return.

The operator’s ruling on the parity gate, verbatim

DECISION: Change the catch-up gate from `live == old` to the invariant that actually proves no
data loss and is satisfiable while ingest continues:
    live_count >= old_count  AND  no old row is missing from live
Commit the fix to `scripts/clickstack/migrate-schema-2.37.0.sh`, then resume the migration for the
four remaining tables.
RATIONALE AS GIVEN: fixes the defect once rather than overriding it by hand four times, and leaves
the script correct for the next migration.

The fix, and its two-sided proof, verbatim

Commit f3383d8de. The window derivation from 03-02 is unchanged — it is correct as far as it goes. What changed is the assertion made over that window:

[[ "$live_tail" -ge "$old_tail" ]] || die \
    "catch-up parity FAILED for ${t}: live=${live_tail} is BELOW old=${old_tail}. Nothing dropped; ${staging} still holds every old row."
missing=$(ch_int "SELECT count() FROM (SELECT fp, sum(is_old) AS n_old, sum(1 - is_old) AS n_live FROM (SELECT cityHash64(${fp_expr}) AS fp, toUInt8(0) AS is_old FROM ${DB}.${t} WHERE ${window} UNION ALL SELECT cityHash64(${fp_expr}) AS fp, toUInt8(1) AS is_old FROM ${DB}.${staging} WHERE ${window}) GROUP BY fp HAVING n_old > n_live)")
log_info "  catch-up no-loss: ${missing} old row fingerprint(s) not covered by the live table (must be 0)"
[[ "$missing" -eq 0 ]] || die \
    "catch-up parity FAILED for ${t}: ${missing} old row fingerprint(s) are missing from the live table. Nothing dropped; ${staging} still holds every old row."

The check is multiset containment: for every normalised row fingerprint, the live table must hold at least as many rows as the frozen old table. Extra live rows — the late-flushed batches that caused the false failure — can only ever increase a live count, so continuing ingest cannot make this fail. A dropped row always can. live >= old is subsumed by it but is kept and reported first, because it is one cheap count that names the failure in the operator’s own terms before the twenty-second containment scan runs.

The fingerprint casts every shared column to the migrated table’s own type before hashing:

toString(CAST(`ResourceAttributes` AS Map(LowCardinality(String), String))), ...,
toString(CAST(`MetricName` AS LowCardinality(String))), ...,
toString(CAST(`StartTimeUnix` AS DateTime)), toString(CAST(`TimeUnix` AS DateTime)), ...,
toString(CAST(`Exemplars.TimeUnix` AS Array(DateTime)))

That cast reproduces exactly the conversion the catch-up INSERT performs, so the gate asks the only question that matters: is every old row present in the live table as the copy would have written it. On the live side the cast is an identity, which is why one expression serves both sides. The CAST(x AS Type) form is used rather than CAST(x, 'Type') because the client’s default TabSeparated output escapes a quote as \', and that escape travels back into the next query as a syntax error — measured during this work, not theorised, and guarded by a check that refuses any column type containing a quote.

Two-sided proof against the real migrated gauge pair. Every executable line of the harness is extracted verbatim from the committed script with sed, so the proof cannot pass against a hand-tuned variant of the gate; a sed that matches nothing is a fatal error rather than an empty eval, and each mutant is asserted to differ from its source before it is run.

$ bash gate-harness.sh green
extracted: fp_arity=21 col_arity=21 quoted_types=0
extracted: t_bound=2026-08-31 01:25:57
[INFO]   catch-up parity [2026-08-31 00:14:02 .. 2026-08-31 01:25:57]: live=7721841 old=7719386
[INFO]   catch-up no-loss: 0 old row fingerprint(s) not covered by the live table (must be 0)
[INFO]   catch-up parity OK (live >= old, and every old row is present)
GATE VERDICT: PASS (mode=green)
EXIT=0

$ bash gate-harness.sh red-lossy          # ~1 live row in 997 removed
[INFO]   catch-up no-loss: 10295 old row fingerprint(s) not covered by the live table (must be 0)
[GATE-DIE] catch-up parity FAILED for otel_metrics_gauge: 10295 old row fingerprint(s) are missing from the live table.
EXIT=9

$ bash gate-harness.sh red-unnormalised   # normalisation removed, data provably intact
[INFO]   catch-up no-loss: 2152687 old row fingerprint(s) not covered by the live table (must be 0)
[GATE-DIE] catch-up parity FAILED for otel_metrics_gauge: 2152687 old row fingerprint(s) are missing from the live table.
EXIT=9

GREEN on the real data; RED on a lossy control that was actually executed; RED again with the normalisation removed, which is what makes the casts demonstrably load-bearing rather than decorative. For completeness, the old gate on this same data reads live=7721841 old=7719386 and fails — so the fix changes the outcome rather than merely restating it.

The drop still gates on the assertion. Both die calls precede the DROP TABLE, and the script runs under set -euo pipefail, so a failed gate leaves the staging table intact. That ordering is unchanged; this run is --skip-drop in any case, so no drop occurs either way.

One further change came out of the proof. The fingerprint is now derived after the EXCHANGE rather than before it. Both placements are correct — the same column set, the same target types — but the pre-exchange form reads those types from the other table name, which makes it impossible to replay against the finished tables. That is exactly how the first draft of this check was caught silently reading the old types.

The resume — what it touches and what it does not, verbatim

A naive re-run would be catastrophic if Phase 4 recreated and exchanged unconditionally: otel_metrics_gauge__new already exists and holds the old table, so a second EXCHANGE would put the 2.29.0 table back on the live name and strand every post-exchange write. It does not do that. The loop probes the sorting key first and continues before the staging table is even looked at:

have_sort=$(ch "SELECT sorting_key FROM system.tables WHERE database = '${DB}' AND name = '${t}'")
...
if [[ "$have_sort" == "$want_sort" ]]; then
    log_skip "${t} already carries the 2.37.0 sorting key (${rows_before} rows)"
    METRICS_REPORT+=("${t}|${rows_before}|${rows_before}|already applied")
    continue
fi

staging_present=$(ch_int "SELECT count() FROM system.tables WHERE database = '${DB}' AND name = '${staging}'")
[[ "$staging_present" -eq 0 ]] || die \
    "${DB}.${staging} already exists but ${t} is not migrated; a previous run left it behind - inspect and drop it before retrying"

otel_metrics_gauge reaches continue before the staging_present guard, so the retained otel_metrics_gauge__new neither triggers the guard nor is re-exchanged. Verified by comparing the 2.37.0 seed’s ORDER BY against the live sorting key for all five tables, using the script’s own extraction (seed_stmt + seed_sorting_key) against a scratch-token seed:

otel_metrics_gauge                     SKIP (already migrated)
   want=ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
   have=ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
otel_metrics_sum                       MIGRATE
otel_metrics_histogram                 MIGRATE
otel_metrics_summary                   MIGRATE
otel_metrics_exponential_histogram     MIGRATE
   want=ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
   have=ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)

The strings are compared, not eyeballed: have_sort for gauge is byte-identical to want_sort. Phases 1, 2 and 3 are likewise idempotent by probe — index-name sets, ALIAS-column counts and the materialized-view bodies are each read from system tables and reported as SKIP: already applied.

Resume blast radius.

Object	What the resume does
otel_logs, otel_traces	Nothing — Phase 1 and 2 report SKIP: already applied
The two rollup materialized views	Nothing — Phase 3 reports SKIP: already applied
otel_metrics_gauge	Nothing — skipped on the sorting key
otel_metrics_gauge__new (52.31 GiB)	Not touched, not exchanged, not dropped
otel_metrics_sum, _histogram, _summary, _exponential_histogram	Recreated: staging, bounded 3-day copy, one EXCHANGE each, catch-up, the new gate
otel_logs_key_rollup_15m{,_mv}, otel_traces_key_rollup_15m{,_mv}	Dropped by Phase 5 — the four upstream removals at 2.32.0. --skip-drop does not cover these; it governs the metrics staging tables only

Additional disk for the four copies, measured rather than guessed — the EXCHANGE is a rename and costs nothing, so the only new space is the bounded copy:

t                                   total_rows   rows_3d     total_size   est_copy_size
otel_metrics_exponential_histogram           0           0   0.00 B       0.00 B
otel_metrics_histogram               901372093   114503336   11.89 GiB    1.51 GiB
otel_metrics_sum                    2337036069   242629912   23.18 GiB    2.41 GiB
otel_metrics_summary                  29580679     2857177   540.13 MiB   52.17 MiB

Roughly 4 GiB against 63.65 GiB free. otel_metrics_exponential_histogram holds no rows at all, so its recreate is a schema change over an empty table.

Pre-resume cluster state, read from the cluster.

server_version           26.7.1.1315
replication_queue        2            (threshold 10)
unfinished_mutations     0
free_space               63.65 GiB    (threshold 20 GiB)
engine_drift             0
staging_tables_present   ['otel_metrics_gauge__new']
removed_objects_present  ['otel_logs_key_rollup_15m','otel_logs_key_rollup_15m_mv','otel_traces_key_rollup_15m','otel_traces_key_rollup_15m_mv']

$ kubectl -n clickstack get cronjob clickstack-watchdog -o jsonpath='{.spec.suspend}'
false

$ kubectl -n clickstack get pods -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[*].restartCount,IMAGE:.spec.containers[*].image'
cs-otel-collector-7864f5c4fc-9mgvc  0  docker.clickhouse.com/clickhouse/clickstack-otel-collector:2.32.0

The collector’s restart count is still 0, so research assumption A4 — that the swap requires no collector restart — survives the gauge migration intact. The watchdog was never suspended.

Staging-table retention under --skip-drop

default.otel_metrics_gauge__new is retained, holding 4,613,742,871 rows / 52.31 GiB of pre-migration gauge data. It is frozen: nothing writes to that name after the exchange.

Disposition	Retained under --skip-drop; drop it deliberately, do not let it be forgotten
Owner	Sean
When	After plan 03-11 — at phase end, once the MongoDB 5.0 → 8.0 ladder is done and the whole ClickStack plane is verified on current versions
Why then	Keeps the fallback available through every remaining live change in this phase, and lands well inside the ~27-day horizon after which the unchanged 30-day TTL would remove that data anyway

The same disposition applies to the three further staging tables the resume creates (otel_metrics_sum__new, otel_metrics_histogram__new, otel_metrics_summary__new) and to otel_metrics_exponential_histogram__new, which will be empty. A forgotten staging table is a doubled footprint on a disk already around two-thirds used, which is why this row exists rather than a sentence.

Attempt 3 — the resume completed, 2026-08-31T13:55:25Z, verbatim

Capture: /tmp/clickstack-migrate-2.37.0-20260831T135525Z.log (117 lines). The operator re-ran scripts/clickstack/migrate-schema-2.37.0.sh --apply --skip-drop after the containment fix landed in f3383d8de. t_cut is fixed once per run and this run took a fresh one, so it is later than attempt 2’s; that is the design, not drift.

[INFO] target database: default   mode: apply   transport: pod
[INFO] buffer window: 3 day(s)   skip-drop: true

=== Phase 0: preflight (read-only, all fatal) ===
[INFO] server 26.7.1.1315 (>= 26.2)
[INFO] unfinished mutations: 0
[INFO] replication queue depth: 0
[INFO] free space: 63.37 GiB
[INFO] engine drift (watchdog Q_DRIFT): 0
[INFO] t_cut  = 2026-08-31 13:55:24   (fixed once; never recomputed)
[INFO] t_floor= 2026-08-28 13:55:24   (t_cut minus 3 day(s))

=== Phase 1: otel_logs - 3 ALIAS columns and 8 index definitions ===
[STEP] SKIP: already applied - otel_logs already carries the 2.37.0 ALIAS columns and all 8 text indexes

=== Phase 2: otel_traces - 2 ALIAS columns and exactly 2 index swaps ===
[STEP] SKIP: already applied - otel_traces already carries idx_res_attr_items and idx_span_attr_items
[INFO]   the six unchanged bloom-filter indexes were left alone, as intended

=== Phase 3: replace the two rewritten materialized-view bodies ===
[STEP] SKIP: already applied - otel_logs_attr_kv_rollup_15m_mv already carries the native-column-only 2.37.0 body
[STEP] SKIP: already applied - otel_traces_kv_rollup_15m_mv already carries the native-column-only 2.37.0 body

=== Phase 4: recreate the five otel_metrics_* tables (ORDER BY is not ALTER-able) ===
[STEP] SKIP: already applied - otel_metrics_gauge already carries the 2.37.0 sorting key (558401913 rows)
[INFO] otel_metrics_sum: 2338611975 rows before
[DDL]  CREATE TABLE IF NOT EXISTS default.otel_metrics_sum__new
[DDL]  INSERT INTO default.otel_metrics_sum__new (...)
[DDL]  EXCHANGE TABLES default.otel_metrics_sum AND default.otel_metrics_sum__new
[INFO]   exchanged at 2026-08-31 14:34:33; otel_metrics_sum__new now holds the OLD table and can no longer be written to
[DDL]  INSERT INTO default.otel_metrics_sum (...)
[INFO]   catch-up parity [2026-08-31 13:55:24 .. 2026-08-31 14:34:33] (last write before the swap): live=2214212 old=2200693
[INFO]   catch-up no-loss: 0 old row fingerprint(s) not covered by the live table (must be 0)
[INFO]   catch-up parity OK (live >= old, and every old row is present) - only now is the staging table eligible for dropping
[WARN]   --skip-drop: default.otel_metrics_sum__new kept, holding the pre-migration rows
[INFO] otel_metrics_sum: 2338611975 rows before, 244946645 rows after
    ... otel_metrics_histogram, otel_metrics_summary and
    ... otel_metrics_exponential_histogram follow the identical shape; their
    ... parity readings are tabulated below

=== Phase 5: drop the four objects removed upstream at 2.32.0 ===
[DDL]  DROP VIEW IF EXISTS default.otel_logs_key_rollup_15m_mv
[DDL]  DROP VIEW IF EXISTS default.otel_traces_key_rollup_15m_mv
[DDL]  DROP TABLE IF EXISTS default.otel_logs_key_rollup_15m SYNC
[DDL]  DROP TABLE IF EXISTS default.otel_traces_key_rollup_15m SYNC

=== Phase 6: postflight (read from system tables, never from this summary) ===
[INFO] engine drift (watchdog Q_DRIFT): 0
[INFO] the four upstream-removed objects are absent
[INFO] all five otel_metrics_* sorting keys match the 2.37.0 seed
[INFO] otel_logs and otel_traces each carry 8 indexes
[INFO] otel_logs index name set matches the 2.37.0 seed exactly
[INFO] otel_traces carries both *_items indexes and neither *_value index

=== Summary ===
database:            default
mode:                apply
t_cut:               2026-08-31 13:55:24
buffer floor:        2026-08-28 13:55:24 (3 day window)
DDL statements issued: 20
steps already applied: 5

metrics table                           rows before   rows after  disposition
otel_metrics_gauge                        558401913    558401913  already applied
otel_metrics_sum                         2338611975    244946645  recreated (staging kept)
otel_metrics_histogram                    902818618    115826723  recreated (staging kept)
otel_metrics_summary                       29640089      2897725  recreated (staging kept)
otel_metrics_exponential_histogram                0            0  recreated (staging kept)

[INFO] migration complete: 20 DDL statement(s) issued against default

The five SKIP: already applied steps are the resume mechanic that the 03-06 segment summary predicted and verified in advance: phase 4 probes each table’s live sorting key against the seed’s ORDER BY and continues before the staging-table guard, so the retained otel_metrics_gauge__new was neither tripped over nor re-exchanged. Gauge’s rows before and rows after are identical (558401913) because nothing was done to it.

The four parity readings — the evidence the gate fix was correct

Every one of the four recreated tables reported live > old on the catch-up window. Under the old strict-equality assertion this run would have halted four times, each time past the point of no return, on data that was in every case provably complete.

Table	Catch-up window	live	old	Excess	Old rows not covered by live
otel_metrics_sum	13:55:24 .. 14:34:33	2,214,212	2,200,693	+13,519	0
otel_metrics_histogram	13:55:24 .. 14:59:35	1,385,128	1,381,541	+3,587	0
otel_metrics_summary	13:55:24 .. 15:00:31	41,691	41,644	+47	0
otel_metrics_exponential_histogram	13:55:24 .. 13:55:24	0	0	+0	0

The excesses are exactly the write-time leak the fix was built for: a collector batch flushed after the EXCHANGE carrying data points stamped before any event-time bound the script can compute. The containment half of the gate — no old-row fingerprint is missing from the live table — returned 0 on all four, which is the half that would actually catch a lost row.

otel_metrics_exponential_histogram is the zero-row control and it is worth naming as one: it is the only case in which the old and the new assertions agree, so it shows the fix relaxed the gate rather than disabling it. The containment clause is independent of the counts, so it does the same work at 0 == 0 as at live > old.

Each table’s [WARN] --skip-drop: … kept, holding the pre-migration rows line follows its parity line, which is the ordering the plan requires: the drop is gated on parity, and here the drop did not happen at all.

End-state verification against the running database, verbatim

Read from system.tables, system.columns, system.data_skipping_indices, system.parts, system.replicas, system.mutations and system.disks in the live default database, and from the running pods — never from the ConfigMap, never from the script’s own summary table, and never from a Job’s exit code. That prohibition is the whole point of this section: attempt 1 proved that six layers of this stack will report green while the fact underneath is false.

Taken at approximately 2026-08-31T15:25Z, some twenty minutes after migration complete.

All five metrics tables carry the 2.37.0 sorting key

$ SELECT name, sorting_key FROM system.tables
    WHERE database='default' AND name LIKE 'otel_metrics_%' AND name NOT LIKE '%__new' ORDER BY name

otel_metrics_exponential_histogram  ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
otel_metrics_gauge                  ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
otel_metrics_histogram              ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
otel_metrics_sum                    ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix
otel_metrics_summary                ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix

Compared programmatically against the ORDER BY clause extracted from the committed argocd/app-configs/clickstack/bootstrap-schema-configmap.yaml for the same table name — five string equalities, not a spot check. The toStartOfHour(TimeUnix) term is the 2.37.0 marker; the 2.29.0 key did not carry it.

The four objects upstream removed at 2.32.0 are gone

$ SELECT count() FROM system.tables WHERE database='default' AND name IN
    ('otel_logs_key_rollup_15m','otel_logs_key_rollup_15m_mv',
     'otel_traces_key_rollup_15m','otel_traces_key_rollup_15m_mv')
0

RED control - the identical query over four names that DO exist:
$ SELECT count() FROM system.tables WHERE database='default' AND name IN
    ('otel_logs','otel_traces','otel_metrics_gauge','otel_metrics_sum')
4

The control was executed, not described. A 0 from a query that returns 0 for every input is not evidence of absence, and this phase has already been bitten three times by a count-only gate that was vacuous. The 4 proves the predicate can see tables in this database at all.

Indexes, ALIAS columns and engine drift

$ SELECT count() FROM system.data_skipping_indices WHERE database='default' AND table='otel_logs'
8
$ index names, sorted
idx_log_attr_items, idx_log_attr_key, idx_lower_body, idx_res_attr_items,
idx_res_attr_key, idx_scope_attr_items, idx_scope_attr_key, idx_trace_id

$ SELECT count() FROM system.data_skipping_indices WHERE database='default' AND table='otel_traces'
8
$ index names, sorted
idx_duration, idx_lower_span_name, idx_res_attr_items, idx_res_attr_key,
idx_rum_session_id, idx_span_attr_items, idx_span_attr_key, idx_trace_id

$ superseded names still present on otel_traces (idx_res_attr_value, idx_span_attr_value)
0

$ SELECT count() FROM system.columns WHERE database='default' AND table='otel_logs'
    AND name IN ('ResourceAttributeItems','ScopeAttributeItems','LogAttributeItems')
3

$ idx_time_minmax present across the five live metrics tables
5

$ engine drift - plain MergeTree/SummingMergeTree in default, excluding .inner
0

The five retained staging tables are ReplicatedMergeTree, not plain MergeTree, so the data-splitting mistake this window was most likely to make did not happen. Asserted on the staging tables specifically, because they are the objects the script created:

otel_metrics_exponential_histogram__new  ReplicatedMergeTree
otel_metrics_gauge__new                  ReplicatedMergeTree
otel_metrics_histogram__new              ReplicatedMergeTree
otel_metrics_sum__new                    ReplicatedMergeTree
otel_metrics_summary__new                ReplicatedMergeTree

The two surviving rollup views, and their targets

$ SELECT name, engine, metadata_modification_time FROM system.tables
    WHERE database='default' AND name LIKE '%rollup%' ORDER BY name

otel_logs_attr_kv_rollup_15m_mv    MaterializedView            2026-08-31 00:16:12
otel_logs_kv_rollup_15m            ReplicatedSummingMergeTree  2026-07-05 21:28:55
otel_traces_kv_rollup_15m          ReplicatedSummingMergeTree  2026-07-05 21:28:56
otel_traces_kv_rollup_15m_mv       MaterializedView            2026-08-31 00:16:13

The two target tables still carry their July metadata timestamps, so they were not recreated — only the view bodies were replaced, which is what phase 3 is supposed to do. Both view bodies are native-column-only, measured rather than read:

View	NativeColumn occurrences	arrayJoin/mapKeys/mapValues	ResourceAttributes
otel_logs_attr_kv_rollup_15m_mv	14	0	0
otel_traces_kv_rollup_15m_mv	6	0	0

RED control for that counter: piping the literal text x arrayJoin y ResourceAttributes z through the same rg -o 'arrayJoin|mapKeys|mapValues' | wc -l returns 1.

Replication, disk and freshness

$ SELECT countIf(is_readonly), countIf(absolute_delay>60), sum(queue_size) FROM system.replicas
0   0   0
$ SELECT count() FROM system.mutations WHERE NOT is_done
0
$ SELECT name, free_space, total_space, pct_used FROM system.disks
default   59.95 GiB   195.80 GiB   69.38

ingest freshness, seconds behind now()
logs        4
gauge      10
sum         2
histogram   3
summary    14

The collector never restarted, and the watchdog was never silenced

cs-otel-collector-7864f5c4fc-9mgvc   restarts=0   startTime=2026-08-03T04:49:22Z

The pod has not restarted since 3 August — twenty-eight days, spanning every attempt in this record. Research assumption A4 (the exchange requires no collector restart) survives the whole window intact, not merely the gauge segment.

$ kubectl -n clickstack get cronjob clickstack-watchdog -o jsonpath='{.spec.suspend}'
false
$ lastSchedule=2026-08-31T15:20:00Z   lastSuccessful=2026-08-31T15:20:04Z
$ kubectl -n clickstack logs clickstack-watchdog-29803160-lq9mm
stale=0s drift=0
{"ok":true}

The watchdog was never suspended at any point in this window, and its heartbeat has resumed: stale=0s, drift=0, {"ok":true}.

Rule w7esnv785p re-verified after the schema regeneration

Read from MongoDB inside cs-clickstack-mongodb-0, with the connection string passed through the pod’s environment and never on a command line:

name      kind     from.tableName      timestampValueExpression
Logs      log      otel_logs           Timestamp
Traces    trace    otel_traces         Timestamp
Metrics   metric   (empty)             TimeUnix
Sessions  session  hyperdx_sessions    TimestampTime

--- connections ---
Local ClickHouse   http://cs-clickstack-clickhouse-clickhouse-headless:8123   app

The Logs source names Timestamp, not TimestampTime. Exactly one source — Sessions — names TimestampTime, and for hyperdx_sessions that is the correct column. The connections document still points at the ClickHouse headless service as user app. The rule holds, and this reading satisfies the re-verification w7esnv785p requires after a schema regeneration.

The Metrics source’s TimeUnix expression now resolves against a DateTime column rather than DateTime64(9). The expression is unchanged and still valid; the type change beneath it is assumption A2, the intended schema change, not drift.

The retained-staging figure, corrected by measurement

The ## Staging-table retention under --skip-drop section above was written when only otel_metrics_gauge__new existed, and it states the retention as 4,613,742,871 rows / 52.31 GiB — one table. --skip-drop on the resume kept all four further staging tables. The disposition is unchanged; the size it applies to is not. Restated against system.parts, measured now rather than projected:

$ SELECT table, sum(rows), formatReadableSize(sum(bytes_on_disk))
    FROM system.parts WHERE database='default' AND active AND table LIKE 'otel_metrics_%__new'
    GROUP BY table ORDER BY table

otel_metrics_gauge__new       4,613,742,871   52.31 GiB
otel_metrics_sum__new         2,340,814,622   23.22 GiB
otel_metrics_histogram__new     903,346,728   11.92 GiB
otel_metrics_summary__new        29,640,394  541.23 MiB
                              -------------  ----------
TOTAL                         7,887,544,615   87.97 GiB   (94,461,044,680 bytes)

otel_metrics_exponential_histogram__new exists and holds no parts, so it does not appear in a system.parts aggregate.

~88 GiB retained, not ~52 GiB. The 52.31 GiB figure was never wrong for what it described; it described one table because at the time only one existed. It is left in place above rather than edited in situ, because a number that silently changes teaches nobody why it moved.

What did not move: the +12.17 GiB footprint projection held

The ## The correction to the plan's footprint claim section projected the --skip-drop cost as +12.17 GiB, about +14 % — the 3-day buffer copy, against an 88.15 GiB pre-migration metrics footprint. That projection is now checkable rather than modelled, and it was close. It is recorded as confirmed rather than quietly restated, because a projection that survives contact with measurement is worth as much as one that does not:

Quantity	Projected 2026-08-30	Measured 2026-08-31
Pre-migration metrics footprint	88.15 GiB	87.97 GiB, now held under the staging names
The new 3-day copy	12.17 GiB	10.45 GiB (935,270,408 rows)
Combined metrics footprint	~100.4 GiB	98.42 GiB
Delta	+12.17 GiB / +14 %	+10.45 GiB / +11.9 %
Free space after	~57.1 GiB	59.95 GiB of 195.80 GiB (69.38 % used)

The copy came in 1.7 GiB under its projection. That is not measurement error: the recreate changes MetricName from String to LowCardinality(String) and reorders the primary key to lead with toStartOfHour(TimeUnix), and both improve compression over the same rows. The disk is better off than the projection said, not worse, and the plan’s original cons line — that retention would “double the metrics footprint” — remains wrong by roughly a factor of eight.

The live tables after the migration:

otel_metrics_gauge                  568,157,349    6.59 GiB
otel_metrics_sum                    247,810,689    2.37 GiB
otel_metrics_histogram              116,388,585    1.43 GiB
otel_metrics_summary                  2,913,785   52.69 MiB
otel_metrics_exponential_histogram            0           0

Retention disposition, restated

Field	Value
Tables	default.otel_metrics_gauge__new, otel_metrics_sum__new, otel_metrics_histogram__new, otel_metrics_summary__new, otel_metrics_exponential_histogram__new
Size	~88 GiB / 7.89 B rows, measured 2026-08-31 — not ~52 GiB, which covered gauge alone
Owner	Sean
Due	After plan 03-11 completes
Action	DROP TABLE default.otel_metrics_*__new SYNC
Backstop	The unchanged 30-day TTL ages the retained tail out by approximately 2026-09-26 regardless

Two live behaviour changes, stated plainly

Both are consequences of the 2.37.0 schema that are already in effect on the live database, and both are costs rather than defects. They are recorded here because a cost that lives only in a planning artifact is a cost nobody will find when it bites.

OP-02 — the otel_logs text indexes were added without materializing them

Phase 1 adds eight text index definitions to otel_logs and deliberately does not issue ALTER TABLE … MATERIALIZE INDEX. The script says so at the point where it does it:

[WARN] otel_logs text indexes are added without materializing them - deliberate (OP-02).

A skip index only covers the parts written after it was defined. The consequence, stated without softening: full-text search over otel_logs history written before this window is incomplete. A query relying on one of these indexes will not find matches in older parts, and it returns fewer rows rather than erroring — the failure mode is a quiet under-count, not a visible break. The condition clears itself as the 30-day retention window turns over, so the whole table is covered by roughly 2026-09-30.

Not materializing was chosen over materializing because otel_logs is the largest table in the estate and a MATERIALIZE INDEX mutation across it would run for hours inside a live window, competing with ingest for the same disk. otel_traces, at 672 MiB, was materialized — phase 2 issues the mutation and waits for it — which is why the two tables are treated differently, and the difference is a size judgement rather than an inconsistency.

Current index sizes on otel_logs, which reflect only post-definition parts:

idx_log_attr_items     text(tokenizer = 'array')             136.96 MiB
idx_log_attr_key       text(tokenizer = 'array')               1.42 MiB
idx_lower_body         text(tokenizer = 'splitByNonAlpha')   702.73 MiB
idx_res_attr_items     text(tokenizer = 'array')              14.93 MiB
idx_res_attr_key       text(tokenizer = 'array')             393.19 KiB
idx_scope_attr_items   text(tokenizer = 'array')             319.00 B
idx_scope_attr_key     text(tokenizer = 'array')             319.00 B
idx_trace_id           text(tokenizer = 'array')               1.47 MiB

The rollup views are native-column-only, and attribute autocomplete moved with them

The 2.37.0 bodies of otel_logs_attr_kv_rollup_15m_mv and otel_traces_kv_rollup_15m_mv project only native columns — SeverityText, ServiceName, ScopeName, SpanName, SpanKind, StatusCode and their siblings — each tagged ColumnIdentifier = 'NativeColumn'. They no longer iterate the ResourceAttributes / LogAttributes / SpanAttributes maps, which the 2.29.0 bodies did. Verified above: zero arrayJoin / mapKeys / mapValues and zero ResourceAttributes occurrences in either body.

The consequence: the kv rollup no longer receives map attribute keys and values, and HyperDX’s attribute autocomplete is now served by the new text indexes instead. That is upstream’s deliberate trade — the *_items text indexes are a better instrument for the same job — but it couples the two changes. If the text indexes were absent or unhelpful, autocomplete would degrade with nothing feeding it, because the old source has stopped.

This is why the one human spot check on this window is not “does a dashboard render” but “does log-search attribute autocomplete still offer map attribute keys”. A rendering dashboard distinguishes nothing; autocomplete is the single observation that separates the indexes landed from the rollup stopped feeding and nothing replaced it. It carries the same pre-window-history caveat as OP-02: autocomplete over older data fills in as retention turns over.

What this record does not yet cover, and why

The declarative half of CS-01 is open as pull request #2005 and is not merged. Agents do not merge pull requests in this repository. Everything above is therefore the migration and its end-state verification against the running database; the post-sync half is genuinely outstanding, and it is stated here rather than implied by omission.

Read from the live pods at the time of writing:

$ kubectl -n clickstack get pods -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}'
hyperdx/hyperdx:2.32.0
docker.clickhouse.com/clickhouse/clickstack-otel-collector:2.32.0

$ kubectl -n argocd get application clickstack -o jsonpath='{.status.sync.status} {.status.health.status}'
Synced Healthy

Synced Healthy here means the Application matches the unmerged manifest set, not the 2.37.0 one. The cluster is currently a 2.32.0 app and a 2.32.0 collector writing into a 2.37.0-shaped database — the benign direction of the mismatch, and the whole reason the migration was ordered before the merge.

Outstanding until #2005 merges and its sync completes:

Item	Why it cannot be closed now
The clickstack-schema-bootstrap PreSync Job succeeded for this change	No sync has been triggered for it
Both Deployments running the pinned 2.37.0 images	They run 2.32.0, read from the live pods above
Exactly one merged pull request contains all five plans’ commits	#2005 is open
clickstack-operators absent from the merged range	There is no merged range yet
HyperDX attribute autocomplete still offers map attribute keys	The spot check belongs after the app rolls to 2.37.0

Everything else this window was supposed to establish — the schema shape, the index sets, the ALIAS columns, the dropped objects, engine drift, replication health, the collector’s restart count, the watchdog’s heartbeat and rule w7esnv785p — is verified above against live state and does not depend on the merge.