Longhorn instance-manager drain — per-volume verdicts, 2026-08-29

Captured against context fzymgc-house with kubectl against instancemanagers.longhorn.io, engines.longhorn.io and volumes.longhorn.io in longhorn-system, the kubelet /api/v1/nodes/<node>/proxy/stats/summary endpoint, and default.otel_logs via the ClickHouse pod cs-clickstack-clickhouse-clickhouse-0-0-0.

Why this file exists: the drain IS the audit. Every Longhorn volume in this estate has to be detached and reattached to move its engine process off a pre-v1.12.1 instance-manager, and that restart is the only occasion on which each volume is exercised end-to-end. Rather than build a separate integrity investigation around third-party storage behaviour, the post-restart observation for each volume is recorded here as it is taken. This is a dated record of work performed on a specific day, not standing operational prose, so it lives in evidence/ rather than in docs/operations/longhorn.md.

Where the planning artifacts cited below live. Several passages cite a phase planning artifact by bare filename. Every one of them lives in .planning/workstreams/cluster-health/phases/02-longhorn-residue-the-beta-4-tail-and-the-v1-12-1-leftovers/. That directory is a planning record kept in the repository and is deliberately not published to the documentation site, so a reader on the site should expect to open those artifacts from a checkout rather than follow a link from here.

Instance-manager inventory at drain start, verbatim

The LH-03 gate is read two-sided. Query (a) counts engine processes still hosted outside a v1.12.1 instance-manager and its target is 0 — but an empty result from a broken query reads identically to a genuine zero. The inverted-predicate control immediately below it counts engine processes inside v1.12.1 instance-managers using the same traversal; a non-zero control proves the query can see engines at all.

=== capture start: 2026-08-29T20:41:31Z ===
--- (a) engine processes still outside v1.12.1 ---
39
--- (a-inverted) CONTROL: engine processes INSIDE v1.12.1 IMs ---
1
--- (b) stale instance-managers ---
instance-manager-1b8c1c4d55fc05f3f1a9af947ff100a4 v1.10.1 tpi-beta-3 engines=1 replicas=0
instance-manager-27af93a4524a66b6e9b1bc3a6e0a6920 v1.11.2 tpi-alpha-2 engines=5 replicas=0
instance-manager-58b7a48389b11c2c7750c5940cb6802b v1.10.1 tpi-alpha-4 engines=2 replicas=0
instance-manager-5c94660e40a52187b9a427c2cb679808 v1.11.2 tpi-alpha-1 engines=4 replicas=0
instance-manager-68fca8eecce9b1d85ba3f5986b1faae4 v1.11.2 tpi-beta-1 engines=3 replicas=0
instance-manager-996e594d45afc211165dc4b7bfca856b v1.11.2 tpi-beta-4 engines=4 replicas=0
instance-manager-9c18146a00a741ccff3e5d9983c2f788 v1.11.2 tpi-alpha-3 engines=6 replicas=0
instance-manager-9e303193f6af67a4962641022b0ed2ac v1.10.1 tpi-beta-2 engines=3 replicas=0
instance-manager-b48dac91b45fa0a6f4869005f61020b4 v1.11.2 tpi-alpha-4 engines=7 replicas=0
instance-manager-ddbf6cfd1f950d8cb36be16153997003 v1.10.1 tpi-alpha-2 engines=3 replicas=0
instance-manager-fc5f937c05ebc017992613b0f02d73ee v1.11.2 tpi-beta-3 engines=1 replicas=0
--- (c) volumes not attached+healthy on v1.12.1 image ---
--- (c) row count ---
0
--- total volumes ---
40

[VERIFIED: kubectl, 2026-08-29T20:41:31Z] — (a) returned 39, its control returned 1, (b) returned 11 rows, (c) returned 0 rows against 40 volumes. The estate reading is unchanged from 02-RESEARCH.md §1.2 at 2026-08-29T19:08Z: no drift in the intervening 93 minutes. All 86 replicas were already migrated (replicas=0 on every stale instance-manager), so what remains stale is engine processes only.

Tracer volume — mealie/mealie-data, end to end

The tracer was selected by property, not by name: the first tier-1 leaf still hosted in a pre-v1.12.1 instance-manager, owned by a single-replica Deployment with strategy: Recreate, holding exactly one PVC on its pod. mealie/mealie-data satisfied all three at capture time and sat on a v1.10.1 host, which exercises the deepest version hop available in the estate.

=== PRE-RESTART volume state 2026-08-29T20:42:05Z ===
{
  "state": "attached",
  "robustness": "healthy",
  "image": "docker.io/longhornio/longhorn-engine:v1.12.1",
  "node": "tpi-alpha-2",
  "actualSize": 409690112,
  "lastDegradedAt": "",
  "lastAutoSalvagedAt": "",
  "remountRequestedAt": "2026-05-12T13:05:07Z",
  "lastBackupAt": "2026-08-29T07:09:40Z"
}
=== PRE-RESTART engine ===
pvc-5472d87c-382c-4360-aee5-900715c5818e-e-0  currentState=running  IM=instance-manager-ddbf6cfd1f950d8cb36be16153997003  snapshotsError=""  specIM=""
=== PRE-RESTART IM image ===
longhornio/longhorn-instance-manager:v1.10.1

=== DELETE at 2026-08-29T20:42:49Z ===
pod "mealie-75f897d776-msqnp" deleted from mealie namespace

=== POST volume state 2026-08-29T20:44:27Z ===
attached/healthy/docker.io/longhornio/longhorn-engine:v1.12.1
{
  "state": "attached",
  "robustness": "healthy",
  "image": "docker.io/longhornio/longhorn-engine:v1.12.1",
  "node": "tpi-alpha-3",
  "actualSize": 352591872,
  "lastDegradedAt": "",
  "lastAutoSalvagedAt": "",
  "remountRequestedAt": "2026-05-12T13:05:07Z",
  "lastBackupAt": "2026-08-29T07:09:40Z"
}
=== POST engine (status.instanceManagerName, NOT spec) ===
pvc-5472d87c-382c-4360-aee5-900715c5818e-e-0  currentState=running  IM=instance-manager-88a77e8ce104cc546dcc35359557c2c5  snapshotsError=""
=== resolve that IM back to spec.image ===
docker.io/longhornio/longhorn-instance-manager:v1.12.1  tpi-alpha-3  2026-08-24T14:58:12Z

[VERIFIED: kubectl, 2026-08-29T20:42:05Z–20:44:27Z]

The engine moved from instance-manager-ddbf6cfd… (v1.10.1, tpi-alpha-2) to instance-manager-88a77e8ce… (v1.12.1, tpi-alpha-3). The verb was kubectl delete pod and nothing else: no rollout restart, no annotation patch, no kubectl apply, no instance-manager mutation. The replacement pod reached Ready 87 seconds after the delete, and the replacement landed on a different node — confirming that engine placement follows the attach rather than the node, so the move does not depend on rescheduling.

.status.instanceManagerName is the populated field. .spec.instanceManagerName is empty on every engine in this estate (shown above as specIM="") and reading it returns blanks that look like a clean result.

Instance-manager mutation control

--- diff of IM inventory before vs after (empty = none created/deleted) ---
NO DIFFERENCE — no instance-manager created or deleted during the task window
--- any IM created after 20:41Z today? ---
0

[VERIFIED: kubectl get instancemanagers.longhorn.io --sort-by=.metadata.creationTimestamp, diffed across the task window, 2026-08-29T20:44:41Z] — all 19 instance-managers present before the restart are present after it, unchanged. Deleting or patching an instance-manager is prohibited: one instance-manager hosts up to seven engines here, and instance-manager-996e594d… alone holds all three ClickHouse Keeper engines plus postgres/main-20, so a single delete would cost Keeper quorum and a Postgres instance.

Log delta across the restart

The verdict input is a per-pod delta against a recorded baseline, never an absolute zero. Ten namespaces in this cluster carry a nonzero steady-state error-line rate, and matching is done on Body because SeverityText is not populated for most workloads here.

=== error-class lines, mealie namespace, anchored at the delete (20:42:49Z) ===
   ┌─pod─────────────────────┬─baseline_30m_pre─┬─after_since_delete─┬─window_total─┐
1. │ mealie-75f897d776-cv7nv │                0 │                  2 │            2 │
   └─────────────────────────┴──────────────────┴────────────────────┴──────────────┘

=== CONTROL: all mealie lines, same buckets (proves the query reaches both pods) ===
   ┌─pod─────────────────────┬─baseline_30m_pre─┬─after_since_delete─┐
1. │ mealie-75f897d776-cv7nv │                0 │                134 │
2. │ mealie-75f897d776-msqnp │              571 │                  6 │
   └─────────────────────────┴──────────────────┴────────────────────┘

=== the two matched bodies, read rather than counted ===
Timestamp: 2026-08-29 20:44:14.609408156   Body:     "light_error": "#EF5350",
Timestamp: 2026-08-29 20:44:14.609431783   Body:     "dark_error": "#EF5350"

=== CONTROL: did previous pod incarnations emit the same two lines at THEIR startup? ===
   ┌─pod─────────────────────┬─n─┬─────────────────────────first─┐
1. │ mealie-6b6b8f874f-xqcpr │ 2 │ 2026-08-16 20:35:38.433252369 │
2. │ mealie-75f897d776-cv7nv │ 2 │ 2026-08-29 20:44:14.609408156 │
3. │ mealie-75f897d776-msqnp │ 4 │ 2026-08-16 22:48:52.181330352 │
   └─────────────────────────┴───┴───────────────────────────────┘

[VERIFIED: default.otel_logs via cs-clickstack-clickhouse-clickhouse-0-0-0, 2026-08-29T20:42:34Z (baseline) and 20:45:23Z (after)]

Baseline 0 → after 2, and both after-lines are Mealie’s startup theme-colour dump matched only because the substring error appears inside the CSS token names light_error / dark_error. They are not error records. The third control settles it: every prior pod incarnation emitted the same two lines at its own startup, so this body class was already present in the workload’s normal behaviour and is not a class introduced by the restart. The old pod carried 269 error-class lines over the retained window against the new pod’s 2, so the restart did not raise the rate.

Recording the raw counts and then reading the actual bodies is the point. A criterion of the form “zero errors after the restart” would have failed here on a colour swatch.

Gate movement attributable to the tracer

=== post-tracer LH-03 gate re-read, 2026-08-29T20:48:39Z ===
(a) engine processes outside v1.12.1 : 38   (was 39)
(a-inverted) control, inside v1.12.1 :  2   (was 1)

[VERIFIED: kubectl, 2026-08-29T20:48:39Z] — one engine moved, and it moved in the direction the drain intends. Both halves of the gate moved by exactly one, which is what a single successful detach/reattach should produce and is itself a check that the two queries partition the same set.

Per-volume verdicts

One row per Longhorn volume present in the live estate at capture time. The row set is derived from a live re-read of volumes.longhorn.io joined on status.kubernetesStatus.namespace + pvcName, not transcribed from research.

[VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io, 2026-08-29T20:48:39Z — 40 volumes, 40 rows]

ns/PVC	Tier / wave	state/robustness/image	FS mounted rw	Workload Ready	Log delta vs baseline	Verdict
mealie/mealie-data	1 / 02-01 (tracer)	attached/healthy/v1.12.1	100 MiB used of 9.75 GiB, usedBytes at 20:43:29Z on the new node	True, 87 s after delete	0 → 2, both startup theme tokens; no new body class	pass
agent-memory/qdrant-data	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-2 → tpi-alpha-4	144.53 MiB of 4.84 GiB, usedBytes on the new node	True, 40 s after delete	1 → 1; the one after-line is qdrant’s telemetry-egress ERROR, third control shows 197 + 151 identical lines on prior incarnations since 2026-08-15	pass
clickstack/clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-0-0	2 / 02-05	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, 68fca8ee… → 7e6aabe3…; ends on tpi-beta-1, its home node, via the two-step move-away-then-home verb this wave had to derive for single-replica volumes	123.5 GiB used of 195.8 GiB, df on the re-mounted /dev/longhorn/pvc-d1086c25…, ext4 rw,relatime in /proc/mounts at 00:50:22Z	True at 00:50:22Z, 116 s after the home-bound delete; the container’s own Application: Ready for connections at 00:50:21Z, 5 s after Starting ClickHouse 26.7.1.1315	anchored <Error>/<Fatal> 0 → 1, the one line an in-flight QUERY_WAS_CANCELLED caused by our own shutdown; prescribed_raw 31 → 9 and prescribed-minus-query-echo 0 → 7 (startup banner + peer-noticing on the survivor); corruption control 4 raw → 0 tightened, all four the system.metric_log DDL echo, read verbatim; all-lines control 179,944 → 23,208 proves the query reaches the replacement. Read through the surviving replica …0-1-0, so this cell is a real reading and not an n/a	pass — restarted 1st. Single-replica 142 GiB volume; the best-effort rebuild the cordon started on tpi-alpha-4 was discarded unfinished and the original replica kept its healthyAt=2026-07-05T21:14:35Z
clickstack/clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-1-0	2 / 02-05	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, fc5f937c… → 4e199918…; ends on tpi-beta-3, its home node. fc5f937c… held only this engine and was reaped by Longhorn once emptied	122.6 GiB used of 195.8 GiB, df on the re-mounted /dev/longhorn/pvc-5fa64d0c…, ext4 rw,relatime in /proc/mounts at 01:07:18Z	True at 01:07:14Z, 36 s after the home-bound delete (88 s on the away-bound delete before it)	anchored <Error>/<Fatal> 0 → 1, again a single QUERY_WAS_CANCELLED from our own shutdown; prescribed_raw 2 → 13 and prescribed-minus-query-echo 0 → 9; corruption control 4 raw → 0 tightened, the same system.metric_log DDL echo; all-lines control 4,421 → 39,132 proves the query reaches the replacement. Read through the surviving replica …0-0-0, so this cell is a real reading and not an n/a	pass — restarted 2nd, after a READY-True All shards are ready 2/2 reading at 00:53:24Z. Three deletes: one no-op that falsified the detach-in-place hypothesis, then the working two-step
clickstack/clickhouse-storage-volume-cs-clickstack-keeper-keeper-0-0	2 / 02-05	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, 996e594d… → 6abfc3c9… v1.12.1, tpi-beta-4 → tpi-alpha-1	175.5 MiB used of 9.8 GiB, df on the re-mounted /dev/longhorn/pvc-df53c8dd…, ext4 rw,relatime at 01:22:36Z	True at 01:15:15Z, 36 s after delete; KeeperCluster Cluster is ready 3/3 only at 01:16:59Z, 102 s later — the gate the wave was actually held on	anchored <Error>/<Fatal> 0 → 1, the single line a RaftInstance socket teardown at 01:19:52.576Z naming the peer being deleted; prescribed 0 → 13; corruption control 0 → 0; <Warning> 0 → 59, the chronic RaftInstance … took long time class already running at 5/day before this wave; all-lines control 10 → 279 proves the query reaches the replacement	pass — restarted 1st, role follower re-read at 01:14:40Z from keeper-1 and keeper-2. Recovered from its own on-disk log, receiving snapshot FALSE, Ready for connections 5 s after process start. remountRequestedAt byte-identical at its pre-phase 2026-07-03T18:05:37Z
clickstack/clickhouse-storage-volume-cs-clickstack-keeper-keeper-1-0	2 / 02-05	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, 996e594d… → 3536df4b… v1.12.1, tpi-beta-4 → tpi-alpha-4	156.1 MiB used of 9.7 GiB, df on the re-mounted /dev/longhorn/pvc-9b57e4e5…, ext4 rw,relatime at 01:22:36Z	True at 01:17:53Z, 32 s after delete; KeeperCluster ready 3/3 at 01:19:42Z, 107 s later	anchored 0 → 3, all three RaftInstance socket teardowns inside 1.1 s of a delete instant; prescribed 0 → 6; corruption control 0 → 0; <Warning> 0 → 21; all-lines control 9 → 246	pass — restarted 2nd, role follower re-read at 01:17:23Z from keeper-0 and keeper-2. No config in log store, will use config from snapshot, caught up 2,336 entries with receiving snapshot FALSE
clickstack/clickhouse-storage-volume-cs-clickstack-keeper-keeper-2-0	2 / 02-05	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, 996e594d… → 3536df4b… v1.12.1, tpi-beta-4 → tpi-alpha-4	144.4 MiB used of 9.7 GiB, df on the re-mounted /dev/longhorn/pvc-316983f6…, ext4 rw,relatime at 01:22:36Z	True at 01:20:21Z, 30 s after delete; KeeperCluster ready 3/3 at 01:22:09Z, 107 s later, with keeper-0 reading leader and zk_synced_followers 2	anchored 0 → 7, all seven RaftInstance teardowns in the three sub-second bursts around the deletes; prescribed 0 → 44; corruption control 0 → 0; <Warning> 0 → 87; all-lines control 18 → 287	pass — restarted LAST, role LEADER re-read at 01:19:51Z from keeper-0 and keeper-1. Stepped down cleanly; KeeperStateMachine: Preprocessing 2 uncommitted log entries … Preprocessing done then Ready for connections 4 s after process start
clickstack/data-volume-cs-clickstack-mongodb-0	2 / 02-04	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-3 → tpi-alpha-1	584.47 MiB of 4.84 GiB, usedBytes on the new node	True at 00:16:18Z, 76 s after delete; back in SECONDARY per a fresh rs.status() at 00:16:33Z	prescribed 8 → 13, mongo-aware "s":"E"/"s":"F" 0 → 0; all-lines control 2810 → 566 proves the query reaches the replacement	pass — restarted 1st, stateStr SECONDARY re-read at 00:15:00Z. Phase 3 CS-02 re-detaches this volume ≥3 more times (D-04, accepted overlap)
clickstack/data-volume-cs-clickstack-mongodb-1	2 / 02-04	attached/healthy/v1.12.1 — IM v1.10.1 → v1.12.1, tpi-alpha-2 → tpi-alpha-3	594.16 MiB of 4.84 GiB, usedBytes on the new node	True at 00:18:48Z, 75 s after delete; back in SECONDARY per a fresh rs.status() at 00:19:07Z	prescribed 216 → 16, mongo-aware 0 → 0; all-lines control 2836 → 533 proves the query reaches the replacement	pass — restarted 2nd, stateStr SECONDARY re-read at 00:17:31Z. Phase 3 CS-02 re-detaches this volume ≥3 more times (D-04, accepted overlap)
clickstack/data-volume-cs-clickstack-mongodb-2	2 / 02-04	attached/healthy/v1.12.1 — IM v1.10.1 → v1.12.1, tpi-alpha-4 → tpi-alpha-2	583.55 MiB of 4.84 GiB, usedBytes on the new node	True at 00:20:41Z, 75 s after delete; rejoined as SECONDARY at 00:21:06Z, cs-clickstack-mongodb-0 elected PRIMARY	prescribed 352 → 14, mongo-aware 0 → 0; all-lines control 4604 → 585 proves the query reaches the replacement	pass — restarted LAST, stateStr PRIMARY re-read at 00:19:24Z. Phase 3 CS-02 re-detaches this volume ≥3 more times (D-04, accepted overlap)
clickstack/logs-volume-cs-clickstack-mongodb-0	2 / 02-04	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-3 → tpi-alpha-1	67.69 MiB of 1.9 GiB, usedBytes on the new node	True at 00:16:18Z, 76 s after delete; back in SECONDARY at 00:16:33Z	prescribed 8 → 13, mongo-aware "s":"E"/"s":"F" 0 → 0; all-lines control 2810 → 566 proves the query reaches the replacement (same pod, one log stream)	pass — same single restart as data-volume-…-mongodb-0. Phase 3 CS-02 re-detaches this volume ≥3 more times (D-04, accepted overlap)
clickstack/logs-volume-cs-clickstack-mongodb-1	2 / 02-04	attached/healthy/v1.12.1 — IM v1.10.1 → v1.12.1, tpi-alpha-2 → tpi-alpha-3	67.46 MiB of 1.9 GiB, usedBytes on the new node	True at 00:18:48Z, 75 s after delete; back in SECONDARY at 00:19:07Z	prescribed 216 → 16, mongo-aware 0 → 0; all-lines control 2836 → 533 proves the query reaches the replacement (same pod, one log stream)	pass — same single restart as data-volume-…-mongodb-1. Phase 3 CS-02 re-detaches this volume ≥3 more times (D-04, accepted overlap)
clickstack/logs-volume-cs-clickstack-mongodb-2	2 / 02-04	attached/healthy/v1.12.1 — IM v1.10.1 → v1.12.1, tpi-alpha-4 → tpi-alpha-2	67.43 MiB of 1.9 GiB, usedBytes on the new node	True at 00:20:41Z, 75 s after delete; rejoined as SECONDARY at 00:21:06Z	prescribed 352 → 14, mongo-aware 0 → 0; all-lines control 4604 → 585 proves the query reaches the replacement (same pod, one log stream)	pass — same single restart as data-volume-…-mongodb-2. Phase 3 CS-02 re-detaches this volume ≥3 more times (D-04, accepted overlap)
dolt/data-dolt-0	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-2 → tpi-beta-2	1.99 MiB of 19.52 GiB, usedBytes on the new node	True, 61 s after delete (container startedAt 21:03:57Z)	0 → 0; all-lines control 0 → 31 proves the query reaches the pod	pass
fovea/fovea-audit-js	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-3 → tpi-alpha-1	65.44 MiB of 4.84 GiB, usedBytes on the new node	True, 35 s after delete	0 → 0; all-lines control 0 → 2 (old pod) and 0 → 30 (new)	pass
fovea/qdrant-fovea-data	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-4 → tpi-alpha-2	146.75 MiB of 19.52 GiB, usedBytes on the new node	True, 75 s after delete	0 → 1; the one after-line is the same qdrant telemetry-egress ERROR, third control shows 335 identical lines on the prior incarnation since 2026-08-15	pass
headroom-agents/headroom-agents-workspace	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-4 → tpi-beta-2	58.17 MiB of 0.95 GiB, usedBytes on the new node	True, 85 s after delete	0 → 0; all-lines control 0 → 47	pass
headroom-apps/headroom-apps-workspace	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-4 → tpi-alpha-1; needed the cordon fallback, see the same-node correction below	89.75 MiB of 0.95 GiB, usedBytes on the new node	True, 30 s after the third delete	0 → 0 across all three pod incarnations; all-lines control 47 on each	pass
karakeep/karakeep-data	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-3 → tpi-alpha-4	4.78 GiB of 48.91 GiB, usedBytes on the new node	True, 45 s after delete	0 → 0 on the karakeep pod itself; all-lines control 1094 → 17 (old) and 0 → 107 (new). The namespace’s browserless pod carries its own steady rate (12 → 2) and was not restarted	pass
karakeep/meilisearch-data	1 / 02-02	attached/healthy/v1.12.1	n/a — no restart performed	n/a — no restart performed	n/a — no restart performed	already-migrated, read 2026-08-29
mosquitto/mosquitto-data	1 / 02-02	attached/healthy/v1.12.1 — IM v1.10.1 → v1.12.1, tpi-beta-2 → tpi-beta-4	32 KiB of 0.97 GiB, usedBytes on the new node	True, 65 s after delete	0 → 0; all-lines control 309 → 2 (old) and 0 → 50 (new)	pass
nats/nats-js-nats-0	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-2 → tpi-alpha-3; needed the cordon fallback	452 KiB of 9.75 GiB, usedBytes on the new node	True, 60 s after the second delete (startedAt 22:17:03Z)	prescribed matcher 0 → 0; NATS-aware matcher 0 → 0 once the route-retry class and the chronic bridge class are named and excluded	pass — JetStream restored all 5 streams, Server is ready, re-read
nats/nats-js-nats-1	1 / 02-02	attached/healthy/v1.12.1 — IM v1.10.1 → v1.12.1, tpi-beta-2 → tpi-alpha-2; needed the cordon fallback and the vector-release remedy below	460 KiB of 9.75 GiB, usedBytes on the new node	True at 22:32:44Z, ~15 min after delete — blocked by the stale LUKS device, not by storage	prescribed 82 → 1100 and NATS-aware 82 → 1100, every line the peer-DNS route-retry class; 0 [FTL], 0 corruption strings	pass — first post-incident read; all 5 streams restored, Server is ready
nats/nats-js-nats-2	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-4 → tpi-beta-2; needed the cordon fallback and the vector-release remedy	456 KiB of 9.75 GiB, usedBytes on the new node	True at 22:39:14Z, ~5 min after delete (200 s of it the stale-LUKS stall)	prescribed 159 → 0 and NATS-aware 159 → 0 on its own anchor; the 159 baseline is the route-retry class from the nats-0/nats-1 restarts	pass — first post-incident read; all 5 streams restored, Server is ready
octopus/qdrant-octopus-data	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-4 → tpi-alpha-2; needed the cordon fallback	665.82 MiB of 19.52 GiB, usedBytes on the new node	True, 108 s after the second delete	1 → 1; the same qdrant telemetry-egress ERROR, third control shows 336 identical lines on the prior incarnation since 2026-08-15	pass
postgres/main-16	3 / 02-06	attached/healthy/v1.12.1 — IM v1.10.1 → v1.12.1, 1b8c1c4d… → 24264c66…, tpi-beta-3 → tpi-beta-4; 1b8c1c4d… held only this engine, was reaped by Longhorn once emptied, and was the last v1.10.1 instance-manager in the estate	2.23 GiB used of 48.91 GiB, kubelet /stats/summary on the new node at 01:53:55Z; /dev/longhorn/pvc-780c5703… ext4 rw,relatime in /proc/mounts at 01:52:50Z	True at 01:52:40Z, 54 s after the gated delete (uid 7da421db… → fce31f44…); Cluster CR main-20/main-20/3/Cluster in healthy state at 01:52:43Z; streaming, async, 17.4 ms replay lag read from the NEW PRIMARY main-20 — its own log shows restored log file "00000012.history", consistent recovery state reached at 1CD/1C0D87B0, then started streaming WAL from primary … on **timeline 18**	cnpg-level 0 → 1, the one line While terminating server 1.0 s after our own delete; pg-severity 0 → 94 across the switchover+delete window (9 in the delete-only window), six classes and every one of them present in the 01:05Z unaided-switchover control (terminating connection … administrator command 61, starting up 24, shutting down 8, streaming replication receiver 2, could not send end-of-streaming message 2, terminating walreceiver … 1) — zero new body classes; prescribed_raw 0 → 142 and discarded; corruption control 4 raw → 0 tightened; all-lines control 0 → 382 proves the query reaches the replacement	pass — restarted LAST, and the only volume in the phase whose detach needed two verbs. kubectl cnpg promote main main-20 confirmed on the live Cluster CR at 01:51:18Z (currentPrimary==targetPrimary==main-20, healthy, tl 17→18); the engine was then re-read and found UNMOVED — pod uid and creationTimestamp unchanged, restartCount 2→3 — so the gated pod delete was taken, not skipped. The pod was never deleted while it was the primary. Its remountRequestedAt stayed byte-identical at its pre-phase 2026-05-12T01:25:34Z
postgres/main-17	3 / 02-06	attached/healthy/v1.12.1 — IM v1.10.1 → v1.12.1, 9e303193… → 3536df4b…, tpi-beta-2 → tpi-alpha-4; 9e303193… held only this engine and was reaped by Longhorn once emptied	2.20 GiB used of 48.91 GiB, kubelet /stats/summary on the new node at 01:43:40Z; /dev/longhorn/pvc-b8815886… ext4 rw,relatime in /proc/mounts at 01:41:57Z	True at 01:42:07Z, 47 s after the cordon-gated delete (uid 9da43de8… → 757e57e6…); Cluster CR readyInstances 3 / Cluster in healthy state at 01:42:09Z; streaming, async, 4.3 ms replay lag read from the PRIMARY main-16 — its own log shows consistent recovery state reached at 1CD/190B1CC0 then started streaming WAL from primary … on timeline 17	cnpg-level "level":"error"/"fatal" 0 → 1, the one line While terminating server 1.4 s after our own delete; pg-severity error_severity ERROR/FATAL/PANIC 0 → 18, all three classes present in the 01:05Z unaided-switchover control (starting up 16, terminating walreceiver … administrator command 1, not yet accepting connections 1) — zero new body classes; prescribed_raw 0 → 38 and discarded, it matches the error_severity field NAME on 98.2% of hits; corruption control 8 raw → 0 tightened, every raw hit the pg_controldata Data page checksum version label at offset 2322; all-lines control 0 → 157 proves the query reaches the replacement	pass — restarted 1st. Two deletes: the first returned it to tpi-beta-2 and moved nothing (02-02 same-node non-migration, reproduced on CNPG), the second used the cordon verb. The best-effort rebuild on tpi-alpha-4 completed in ~2 min (healthyAt 2026-08-30T01:43:52Z) and the volume never read degraded
postgres/main-20	3 / 02-06	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, 996e594d… → 5f4ef201…, tpi-beta-4 → tpi-beta-2; 996e594d… was 02-05’s lone survivor holding only this engine and was reaped by Longhorn once emptied — the LH-03 reaping signal, observed not forced	2.22 GiB used of 48.91 GiB, kubelet /stats/summary on the new node at 01:45:45Z; /dev/longhorn/pvc-bc28579b… ext4 rw,relatime in /proc/mounts at 01:45:00Z	True at 01:44:45Z, 48 s after the cordon-first delete (uid bee9a8ba… → 671be9cb…); Cluster CR readyInstances 3 / Cluster in healthy state at 01:44:45Z; streaming, async, 4.6 ms replay lag read from the PRIMARY main-16 — its own log shows consistent recovery state reached at 1CD/1A0A2408 then started streaming WAL from primary … on timeline 17	cnpg-level 0 → 3, all three at 01:43:58Z ~1 s after our own delete (Error while getting cluster CA Client secret, Reconciler error, While terminating server — the in-pod instance-manager controller losing its API client during teardown); pg-severity 0 → 9, two classes both in the 01:05Z control (starting up 8, terminating walreceiver … administrator command 1) — zero new body classes; prescribed_raw 0 → 34 and discarded; corruption control 0 raw → 0; all-lines control 0 → 146 proves the query reaches the replacement	pass — restarted 2nd, driven cordon-first so it needed one delete. Landed on tpi-beta-2 where a replica from 2026-08-23T17:28:24Z already lived, so zero rebuild; three replicas throughout, every one with a non-empty healthyAt
renovate/renovate-ce-cache-hi	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-3 → tpi-alpha-4; needed the cordon fallback; both renovate engines moved together into instance-manager-3536df4b…	18.95 GiB of 29.70 GiB, usedBytes on the new node	True at 22:09:52Z; the delay is a kubelet fsGroup chown over 1.57 M files, not a storage fault — see the finding below	4 → 0; the 4 baseline lines were on the pre-delete pod, the replacement emitted none	pass
renovate/renovate-ce-db	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-3 → tpi-alpha-4; same single restart as renovate-ce-cache-hi	17.36 MiB of 0.95 GiB, usedBytes on the new node	True at 22:09:52Z, same pod	4 → 0, same namespace query as the row above (one pod, one log stream)	pass
uptime-kuma/uptime-kuma-data	1 / 02-02	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-1 → tpi-beta-4	15.54 MiB of 0.95 GiB, usedBytes on the new node	True, 105 s after delete	0 → 0; all-lines control 121 → 8 (old) and 0 → 33 (new)	pass
valkey/valkey-valkey-0-0-data	2 / 02-04	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-1 → tpi-alpha-4	0.05 MiB of 1.9 GiB, usedBytes on the new node	True at 00:08:15Z, 49 s after delete (startedAt 00:07:58Z)	11 → 2; both after-lines are the chronic # Error accepting a client connection class, the same class as all 11 baseline lines; all-lines control 11 → 65 proves the query reaches the replacement	pass — restarted 1st, role replica re-read at 00:07:13Z
valkey/valkey-valkey-0-1-data	2 / 02-04	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-1 → tpi-alpha-4	0.07 MiB of 1.9 GiB, usedBytes on the new node	True at 00:11:23Z, 37 s after delete (startedAt 00:11:16Z)	5 → 0; all-lines control 5 → 11 proves the query reaches the replacement	pass — restarted LAST, role primary re-read at 00:10:44Z
valkey/valkey-valkey-0-2-data	2 / 02-04	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, tpi-alpha-1 → tpi-alpha-4	0.05 MiB of 1.9 GiB, usedBytes on the new node	True at 00:10:02Z, 35 s after delete (startedAt 00:09:54Z)	13 → 0; all-lines control 13 → 63 proves the query reaches the replacement	pass — restarted 2nd, role replica re-read at 00:09:26Z
vault/audit-vault-0	3 / 02-07	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, b48dac91… → 24264c66…, tpi-alpha-4 → tpi-beta-4; b48dac91… held only vault-0’s two engines and was reaped by Longhorn once emptied	24.0 KiB used of 9.7 GiB, df on the re-mounted /dev/longhorn/pvc-7ec86f2c…, ext4 rw,relatime in /proc/mounts at 03:44:22Z. Plain Longhorn device, not dm-crypt — it detached and reattached cleanly at 03:27:57Z, 15 s, inside the same delete that left the data- volume stuck for 12.6 minutes	sealed: false, standby: true at 03:43:34Z, read from /v1/sys/health?standbyok=true — NOT the pod Ready condition, which went True at 03:42:50Z while the API still read sealed: true at 03:43:24Z (≥34 s Ready-while-sealed). Raft voter=true re-read from a witness at 03:44:22Z	anchored [ERROR] 2 → 0; prescribed 2 → 0; [WARN] 1 → 2; corruption control 0 → 0, and 0 across the whole 24-hour pre-wave window on every vault pod — the only wave in the phase whose negative control needed no tightening; all-lines control 10 → 56 proves the query reaches the replacement	pass — restarted 1st, role standby re-read at 03:27:33Z. Same single vault-0 delete as data-vault-0; both rows filled from that one restart
vault/audit-vault-1	3 / 02-07	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, 68fca8ee… → 5f4ef201…, tpi-beta-1 → tpi-beta-2; 68fca8ee… held only vault-1’s two engines and was reaped by Longhorn once emptied	24.0 KiB used of 9.7 GiB, df on the re-mounted /dev/longhorn/pvc-c60c89ef…, ext4 rw,relatime in /proc/mounts at 03:47:45Z	sealed: false, standby: true at 03:47:29Z from /v1/sys/health, NOT pod Ready — Ready=True at 03:46:36Z while the API still read sealed: true at 03:47:18Z (≥42 s Ready-while-sealed). Raft voter=true re-read at 03:47:45Z; restartCount stayed 0, the sealed window being shorter than the liveness cycle	anchored [ERROR] 0 → 3, all three at 03:45:10Z — 5 s after our own delete — and all three in the 24-hour control (Raft RPC layer closed ×1, transport shutdown ×2); prescribed 0 → 3; corruption control 0 → 0, and 0 across the whole 24-hour pre-wave window on every vault pod — the only wave in the phase whose negative control needed no tightening; all-lines control 3 → 42	pass — restarted 2nd, role standby re-read at 03:45:04Z. numberOfReplicas: 2. No cordon: its source node tpi-beta-1 holds the sole replica of the 142 GiB single-replica ClickHouse volume, which was re-read attached/healthy and untouched afterwards
vault/audit-vault-2	3 / 02-07	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, 27af93a4… → 6abfc3c9…, tpi-alpha-2 → tpi-alpha-1; 27af93a4… held only vault-2’s two engines and was reaped by Longhorn once emptied — the last pre-v1.12.1 instance-manager in the estate	24.0 KiB used of 9.7 GiB, df on the re-mounted /dev/longhorn/pvc-3e524a6f…, ext4 rw,relatime in /proc/mounts at 03:52:31Z	sealed: false, standby: true at 03:50:51Z from /v1/sys/health, NOT pod Ready — Ready=True at 03:50:13Z while the API still read sealed: true at 03:50:41Z (≥28 s Ready-while-sealed). Raft voter=true, leader=false re-read at 03:51:10Z after the election; restartCount 0	anchored [ERROR] 320 → 4; prescribed 320 → 4. The 320 is not a baseline — vault-2 was leader while this wave had vault-0 down, and 320 of those lines are its own heartbeats to the peer we had deleted. Of the four after-lines, three are in the 24-hour control and one is new: peer has newer term, stopping replication at 03:49:05Z, 6.2 s after the leader delete, naming vault-0 — the leadership handover itself. corruption control 0 → 0, and 0 across the whole 24-hour pre-wave window on every vault pod — the only wave in the phase whose negative control needed no tightening; all-lines control 749 → 53	pass — restarted LAST, role ACTIVE LEADER (standby: false) re-read at 03:48:59Z immediately before the delete, with both other members already confirmed sealed: false and Raft voters. vault operator step-down returned 403 permission denied (the admin policy lacks sudo on sys/step-down), so leadership was lost deliberately rather than moved; vault-0 took it in ≤19 s and quorum never dropped below 2 of 3
vault/data-vault-0	3 / 02-07	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, b48dac91… → 24264c66…, tpi-alpha-4 → tpi-beta-4	53.4 MiB used of 9.7 GiB, df on the re-mounted /dev/mapper/pvc-fdf2a3c3… — a live dm-crypt device — ext4 rw,relatime in /proc/mounts at 03:44:22Z	sealed: false, standby: true at 03:43:34Z from /v1/sys/health, NOT pod Ready (Ready=True 03:42:50Z, API still sealed: true 03:43:24Z, ≥34 s overlap). The unsealer submitted 3 of 3 shares at 03:43:31Z and logged (was sealed) … now unsealed with tokens. restartCount 0 → 1 at 03:42:44Z is the liveness probe killing the still-sealed pod at 120 s + 2×5 s, not a storage fault	anchored [ERROR] 2 → 0; prescribed 2 → 0; corruption control 0 → 0, and 0 across the whole 24-hour pre-wave window on every vault pod — the only wave in the phase whose negative control needed no tightening; all-lines control 10 → 56 (same pod, one log stream)	pass — restarted 1st. The only delete in the wave that stalled: luksClose returned Device … is still in use for 12.6 minutes until vector was released on the source node at 03:38:24Z, after which the detach completed in 114 s. Storage was never at fault
vault/data-vault-1	3 / 02-07	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, 68fca8ee… → 5f4ef201…, tpi-beta-1 → tpi-beta-2	41.5 MiB used of 9.7 GiB, df on the re-mounted /dev/mapper/pvc-d0b57c21… (dm-crypt), ext4 rw,relatime in /proc/mounts at 03:47:45Z	sealed: false, standby: true at 03:47:29Z from /v1/sys/health, NOT pod Ready (Ready=True 03:46:36Z, API sealed: true 03:47:18Z, ≥42 s overlap). Raft voter=true from a witness; restartCount 0	anchored [ERROR] 0 → 3, the same three teardown lines 5 s after our delete, all in the 24-hour control; prescribed 0 → 3; corruption control 0 → 0, and 0 across the whole 24-hour pre-wave window on every vault pod — the only wave in the phase whose negative control needed no tightening; all-lines control 3 → 42 (same pod, one log stream)	pass — restarted 2nd. numberOfReplicas: 2. Detached on the first delete with no LUKS stall, because vector was released on the source node pre-emptively 20 s after the pod delete rather than after a twelve-minute diagnosis
vault/data-vault-2	3 / 02-07	attached/healthy/v1.12.1 — IM v1.11.2 → v1.12.1, 27af93a4… → 6abfc3c9…, tpi-alpha-2 → tpi-alpha-1	46.4 MiB used of 9.7 GiB, df on the re-mounted /dev/mapper/pvc-3614a6c6… (dm-crypt), ext4 rw,relatime in /proc/mounts at 03:52:31Z	sealed: false, standby: true at 03:50:51Z from /v1/sys/health, NOT pod Ready (Ready=True 03:50:13Z, API sealed: true 03:50:41Z, ≥28 s overlap). Raft voter=true from a witness at 03:51:10Z; restartCount 0	anchored [ERROR] 320 → 4, same single log stream as audit-vault-2 and the same one new handover class; prescribed 320 → 4; corruption control 0 → 0, and 0 across the whole 24-hour pre-wave window on every vault pod — the only wave in the phase whose negative control needed no tightening; all-lines control 749 → 53	pass — restarted LAST, same single vault-2 delete as audit-vault-2. Its remountRequestedAt stayed byte-identical at its pre-phase 2026-05-12T12:02:01Z across the leader delete and the election

The degenerate cases, checked rather than assumed. No volume was detached at capture time and none lacked an owning workload: query (c) returned 0 rows against 40 volumes, and the kubelet /stats/summary sweep returned exactly 40 PVC rows across the eight nodes — one per volume, each naming a live owning pod. No row therefore needed an n/a cell for an absent input. The only n/a cells in the table belong to karakeep/meilisearch-data, whose inputs are unavailable for the stated reason that no restart was performed on it. Should a later wave meet a detached or unowned volume, its row keeps its place with the unavailable inputs marked n/a and a reason, and its verdict derived from the inputs that are available — never omitted, never defaulted to pass. [VERIFIED: kubectl + kubelet /stats/summary across all 8 nodes, 2026-08-29T20:44Z]

Restart work list, keyed by owning pod

The verdict record above is keyed by volume. The restart work is keyed by pod, because seven pods each hold two volumes and a naive per-volume loop would delete the same pod twice. The pairing below is re-derived live from the kubelet stats sweep (a pod appearing against more than one PVC), not carried over from research.

Owning pod	Node	Volumes on pod	IM version at drain start	Volumes	FS used at capture
mealie-75f897d776-cv7nv	tpi-alpha-3	1	v1.10.1 → v1.12.1, done	mealie/mealie-data	100 MiB
qdrant-5f5b979585-n42sh	tpi-alpha-2	1	v1.11.2	agent-memory/qdrant-data	143 MiB
dolt-0	tpi-alpha-2	1	v1.11.2	dolt/data-dolt-0	2 MiB
fovea-6bb56946b7-gxswk	tpi-alpha-3	1	v1.11.2	fovea/fovea-audit-js	65 MiB
qdrant-fovea-7dc87dff79-zglmz	tpi-alpha-4	1	v1.11.2	fovea/qdrant-fovea-data	147 MiB
headroom-agents-6fc5966848-sjf56	tpi-alpha-4	1	v1.11.2	headroom-agents/headroom-agents-workspace	58 MiB
headroom-apps-65d9cb6c-m4xlr	tpi-alpha-4	1	v1.11.2	headroom-apps/headroom-apps-workspace	89 MiB
karakeep-6c69c8cd44-vpkjf	tpi-alpha-3	1	v1.11.2	karakeep/karakeep-data	4.78 GiB
meilisearch-6ddff585c5-2qwd6	tpi-beta-4	1	v1.12.1 — already migrated	karakeep/meilisearch-data	83 MiB
mosquitto-64879cdd96-7lwhm	tpi-beta-2	1	v1.10.1	mosquitto/mosquitto-data	28 KiB
nats-0	tpi-alpha-2	1	v1.11.2	nats/nats-js-nats-0	452 KiB
nats-1	tpi-beta-2	1	v1.10.1	nats/nats-js-nats-1	460 KiB
nats-2	tpi-alpha-4	1	v1.11.2	nats/nats-js-nats-2	456 KiB
qdrant-octopus-6499d95748-6fsfx	tpi-alpha-4	1	v1.11.2	octopus/qdrant-octopus-data	666 MiB
renovate-ce-649d779bf7-s657h	tpi-alpha-3	2	v1.11.2, v1.11.2	renovate/renovate-ce-cache-hi, renovate/renovate-ce-db	18.92 GiB, 17 MiB
uptime-kuma-594cdb584-82qcz	tpi-alpha-1	1	v1.11.2	uptime-kuma/uptime-kuma-data	19 MiB
cs-clickstack-mongodb-0	tpi-alpha-3	2	v1.11.2, v1.11.2	clickstack/data-volume-…-mongodb-0, clickstack/logs-volume-…-mongodb-0	592 MiB, 65 MiB
cs-clickstack-mongodb-1	tpi-alpha-2	2	v1.10.1, v1.10.1	clickstack/data-volume-…-mongodb-1, clickstack/logs-volume-…-mongodb-1	592 MiB, 65 MiB
cs-clickstack-mongodb-2	tpi-alpha-4	2	v1.10.1, v1.10.1	clickstack/data-volume-…-mongodb-2, clickstack/logs-volume-…-mongodb-2	590 MiB, 65 MiB
valkey-valkey-0-0-0	tpi-alpha-1	1	v1.11.2	valkey/valkey-valkey-0-0-data	68 KiB
valkey-valkey-0-1-0	tpi-alpha-1	1	v1.11.2	valkey/valkey-valkey-0-1-data	64 KiB
valkey-valkey-0-2-0	tpi-alpha-1	1	v1.11.2	valkey/valkey-valkey-0-2-data	64 KiB
cs-clickstack-clickhouse-clickhouse-0-0-0	tpi-beta-1	1	v1.11.2	clickstack/clickhouse-storage-volume-…-clickhouse-0-0-0	126.85 GiB
cs-clickstack-clickhouse-clickhouse-0-1-0	tpi-beta-3	1	v1.11.2	clickstack/clickhouse-storage-volume-…-clickhouse-0-1-0	126.09 GiB
cs-clickstack-keeper-keeper-0-0	tpi-beta-4	1	v1.11.2	clickstack/clickhouse-storage-volume-…-keeper-0-0	214 MiB
cs-clickstack-keeper-keeper-1-0	tpi-beta-4	1	v1.11.2	clickstack/clickhouse-storage-volume-…-keeper-1-0	195 MiB
cs-clickstack-keeper-keeper-2-0	tpi-beta-4	1	v1.11.2	clickstack/clickhouse-storage-volume-…-keeper-2-0	183 MiB
main-16	tpi-beta-3	1	v1.10.1	postgres/main-16	2.19 GiB
main-17	tpi-beta-2	1	v1.10.1	postgres/main-17	2.17 GiB
main-20	tpi-beta-4	1	v1.11.2	postgres/main-20	2.17 GiB
vault-0	tpi-alpha-4	2	v1.11.2, v1.11.2	vault/audit-vault-0, vault/data-vault-0	24 KiB, 53 MiB
vault-1	tpi-beta-1	2	v1.11.2, v1.11.2	vault/audit-vault-1, vault/data-vault-1	24 KiB, 41 MiB
vault-2	tpi-alpha-2	2	v1.11.2, v1.11.2	vault/audit-vault-2, vault/data-vault-2	24 KiB, 46 MiB

[VERIFIED: derived live from kubelet /stats/summary + the instance-manager engine map, 2026-08-29T20:41Z–20:44Z] — 33 pod restarts clear 40 volumes, of which one (meilisearch) needs none and one (mealie) is done. Seven pods carry two volumes each.

Wave 02-02 — start-of-wave re-read and tier-1 work list

The estate self-drains. The reading in the drain-start inventory above is a baseline, not a target, so this wave opens by re-running the same two-sided gate and reconciling the result against 02-01’s post-tracer reading rather than assuming it still holds.

=== start-of-wave gate re-read: 2026-08-29T20:57:39Z ===
--- (a) engine processes still outside v1.12.1 ---
38
--- (a-inverted) CONTROL: engine processes INSIDE v1.12.1 IMs ---
2
--- (b) stale instance-managers ---
instance-manager-1b8c1c4d55fc05f3f1a9af947ff100a4 v1.10.1 tpi-beta-3 engines=1 replicas=0
instance-manager-27af93a4524a66b6e9b1bc3a6e0a6920 v1.11.2 tpi-alpha-2 engines=5 replicas=0
instance-manager-58b7a48389b11c2c7750c5940cb6802b v1.10.1 tpi-alpha-4 engines=2 replicas=0
instance-manager-5c94660e40a52187b9a427c2cb679808 v1.11.2 tpi-alpha-1 engines=4 replicas=0
instance-manager-68fca8eecce9b1d85ba3f5986b1faae4 v1.11.2 tpi-beta-1 engines=3 replicas=0
instance-manager-996e594d45afc211165dc4b7bfca856b v1.11.2 tpi-beta-4 engines=4 replicas=0
instance-manager-9c18146a00a741ccff3e5d9983c2f788 v1.11.2 tpi-alpha-3 engines=6 replicas=0
instance-manager-9e303193f6af67a4962641022b0ed2ac v1.10.1 tpi-beta-2 engines=3 replicas=0
instance-manager-b48dac91b45fa0a6f4869005f61020b4 v1.11.2 tpi-alpha-4 engines=7 replicas=0
instance-manager-ddbf6cfd1f950d8cb36be16153997003 v1.10.1 tpi-alpha-2 engines=2 replicas=0
instance-manager-fc5f937c05ebc017992613b0f02d73ee v1.11.2 tpi-beta-3 engines=1 replicas=0
--- (c) volumes not attached+healthy on v1.12.1 image ---
--- (c) row count ---
0
--- total volumes ---
40

[VERIFIED: kubectl, 2026-08-29T20:57:39Z] — the control returned 2, strictly greater than zero, so (a) is answered by a query that demonstrably reaches engine processes.

Reconciliation against 02-01

Reading	UTC	(a) outside v1.12.1	control, inside v1.12.1
02-01 post-tracer	2026-08-29T20:48:39Z	38	2
02-02 start-of-wave	2026-08-29T20:57:39Z	38	2

Drift since 02-01: zero. No engine migrated without our intervention in the intervening nine minutes, and no volume that 02-01 left pending has since moved on its own. The two engines already inside v1.12.1 instance-managers are karakeep/meilisearch-data (migrated unaided before this phase, already recorded already-migrated) and mealie/mealie-data (the 02-01 tracer). No row changes to already-migrated on this reading.

This is a measurement, not a guarantee: the set moved 31 → 30 unaided between 2026-08-24 and 2026-08-29, so a later wave may well find a nonzero drift. The reconciliation is re-run at the top of each wave for exactly that reason, and a nonzero result is recorded as drift this phase did not cause — never as our own progress, and never as a failure.

Volume → instance-manager map, re-derived live

The map is rebuilt for all 40 volumes by iterating instancemanagers.longhorn.io, joining .status.instanceEngines keys through each engine’s .spec.volumeName to .status.kubernetesStatus.namespace + pvcName. .status.instanceManagerName is the populated field; .spec.instanceManagerName is empty on every engine here and reading it returns blanks that look like a clean estate.

[VERIFIED: kubectl, 2026-08-29T20:57:39Z — 40 volumes resolved, 38 to a pre-v1.12.1 instance-manager, 2 to a v1.12.1 one]

Tier-1 work list, keyed by pod

Owning pods are resolved from volumes.longhorn.io .status.kubernetesStatus.workloadsStatus and independently corroborated by the kubelet /stats/summary sweep (a pod appearing against more than one PVC). Ordered ascending by filesystem size, so the cheapest restart runs first and the largest runs against a pipeline that has already worked.

Task 2 — Deployments and dolt (11 pod deletes, 12 volumes):

#	Pod delete	Namespace	Node	IM version	Volumes moved	FS used
1	mosquitto-64879cdd96-7lwhm	mosquitto	tpi-beta-2	v1.10.1	mosquitto/mosquitto-data	28 KiB
2	dolt-0	dolt	tpi-alpha-2	v1.11.2	dolt/data-dolt-0	1.99 MiB
3	uptime-kuma-594cdb584-82qcz	uptime-kuma	tpi-alpha-1	v1.11.2	uptime-kuma/uptime-kuma-data	19.02 MiB
4	headroom-agents-6fc5966848-sjf56	headroom-agents	tpi-alpha-4	v1.11.2	headroom-agents/headroom-agents-workspace	57.97 MiB
5	fovea-6bb56946b7-gxswk	fovea	tpi-alpha-3	v1.11.2	fovea/fovea-audit-js	65.44 MiB
6	headroom-apps-65d9cb6c-m4xlr	headroom-apps	tpi-alpha-4	v1.11.2	headroom-apps/headroom-apps-workspace	89.53 MiB
7	qdrant-5f5b979585-n42sh	agent-memory	tpi-alpha-2	v1.11.2	agent-memory/qdrant-data	143.42 MiB
8	qdrant-fovea-7dc87dff79-zglmz	fovea	tpi-alpha-4	v1.11.2	fovea/qdrant-fovea-data	146.75 MiB
9	qdrant-octopus-6499d95748-6fsfx	octopus	tpi-alpha-4	v1.11.2	octopus/qdrant-octopus-data	665.82 MiB
10	karakeep-6c69c8cd44-vpkjf	karakeep	tpi-alpha-3	v1.11.2	karakeep/karakeep-data	4.78 GiB
11	renovate-ce-649d779bf7-s657h	renovate	tpi-alpha-3	v1.11.2 ×2	renovate/renovate-ce-cache-hi + renovate/renovate-ce-db	18.83 GiB + 17.35 MiB

Task 3 — nats, one at a time (3 pod deletes, 3 volumes):

#	Pod delete	Node	IM version	Volume moved	FS used	Post-incident status
12	nats-0	tpi-alpha-2	v1.11.2	nats/nats-js-nats-0	452 KiB	re-read — started 2026-08-07T13:54:57Z, already clean once since the event
13	nats-1	tpi-beta-2	v1.10.1	nats/nats-js-nats-1	460 KiB	first post-incident read — started 2026-05-12T01:08:52Z
14	nats-2	tpi-alpha-4	v1.11.2	nats/nats-js-nats-2	456 KiB	first post-incident read — started 2026-07-03T18:09:03Z

[VERIFIED: kubectl volumes/engines/instancemanagers + kubelet /stats/summary + container startedAt, 2026-08-29T20:57:39Z–20:58:40Z] — 14 pod deletes clear 15 tier-1 volumes. Every listed volume resolved through .status.instanceManagerName to an instance-manager whose spec.image is not v1.12.1 at the moment the list was built. No pod identifier appears twice: renovate-ce is entered once and fills two verdict rows from one restart.

Workload shapes were re-derived live rather than carried from research, because a workload could have changed shape between planning and execution. All ten owning Deployments read replicas: 1, strategy: Recreate; dolt is a 1-replica OrderedReady StatefulSet; nats is a 3-replica Parallel StatefulSet with a PDB of maxUnavailable: 1 (disruptionsAllowed: 1). A Recreate single-replica Deployment on an RWO volume has no attach-conflict risk — the old pod is fully gone before the new one is created.

Correction: an engine does NOT migrate when the replacement pod lands on the same node

02-RESEARCH.md §4.0 states, as part of “Why a pod delete moves the engine”:

It moves even if the pod lands on the same node — placement follows the attach, not the node.

That is false, and this wave falsified it by measurement. headroom-apps was deleted, its replacement was scheduled back onto tpi-alpha-4 three seconds later, and the engine stayed in the v1.11.2 instance-manager. A second plain delete reproduced the same result.

=== headroom-apps after two plain pod deletes, 2026-08-29T21:25:45Z ===
volume : attached / healthy / v1.12.1 / node=tpi-alpha-4
engine : running  instance-manager-b48dac91b45fa0a6f4869005f61020b4  snapshotsError=""
IM     : docker.io/longhornio/longhorn-instance-manager:v1.11.2  tpi-alpha-4

=== the k8s VolumeAttachment that explains it ===
csi-2243e693db07fcd21c417e9a6865ff5b0512fe9dc8fe1c6eb1db1bb4f31206c3
  node=tpi-alpha-4  attached=true  created=2026-07-03T18:05:51Z

=== the Longhorn attachment ticket it holds ===
ticket csi-2243e693…  nodeID=tpi-alpha-4  type=csi-attacher  Satisfied=True

[VERIFIED: kubectl, 2026-08-29T21:24:15Z–21:25:45Z]

The mechanism. CSI ControllerPublish is scoped to a node, not to a pod. The VolumeAttachment above was created on 2026-07-03 and survived both deletes: while any pod for that PVC remains scheduled on the same node, kube-controller-manager never issues ControllerUnpublish, so Longhorn’s csi-attacher ticket stays satisfied, the volume never reaches detached, the engine process is never destroyed, and no migration into the node’s current-version instance-manager can occur. A pod delete is therefore a sufficient trigger only when the replacement lands elsewhere.

Research reached the opposite conclusion because its evidence was karakeep/meilisearch-data — an observation of a volume that had already migrated, not a controlled same-node test. The claim was an inference from one uncontrolled case.

The fallback, and why this verb. Force the replacement off the node by cordoning it for the duration of one delete, then uncordon:

kubectl cordon <node>
kubectl -n <ns> delete pod <pod>        # replacement must schedule elsewhere
# wait for Ready
kubectl uncordon <node>

kubectl cordon sets spec.unschedulable on the Node object. Nodes carry no ArgoCD labels or annotations and belong to no Application, so selfHeal is not in play — the objection that rules out rollout restart, an annotation patch and kubectl apply does not apply here. Cordon evicts nothing; it only stops new scheduling for the seconds the delete takes, and it is reversed immediately. No instance-manager is touched, so D-05 holds.

Applied to headroom-apps it worked first time:

=== cordon → delete → uncordon, 2026-08-29T21:26:29Z–21:27:00Z (31 s cordon window) ===
node/tpi-alpha-4 cordoned
pod "headroom-apps-65d9cb6c-ltkqb" deleted
READY after 30s: headroom-apps-65d9cb6c-k6lsk  tpi-alpha-1  True
node/tpi-alpha-4 uncordoned
engine : running  instance-manager-6abfc3c987e931e5e659a3ddbe768002
IM     : docker.io/longhornio/longhorn-instance-manager:v1.12.1  tpi-alpha-1
VolumeAttachment: csi-fe92497c…  node=tpi-alpha-1  attached=true  created=2026-08-29T21:26:34Z

[VERIFIED: kubectl, 2026-08-29T21:26:29Z–21:27:15Z] — a new VolumeAttachment dated today replaced the July one, which is the direct evidence that the volume genuinely detached this time.

Consequence for later waves. Migration must be asserted by re-reading .status.instanceManagerName and resolving it to spec.image after every restart — a Ready pod proves nothing about engine placement. Of the eleven Task 2 restarts, three (headroom-apps, qdrant-octopus, renovate-ce) needed the cordon fallback. Waves 02-04 through 02-07 work StatefulSets, whose pods have a much stronger tendency to return to the same node, so the fallback should be expected there rather than treated as exceptional.

Rescue fields across each restart — read as a delta, never as an emptiness test

Per the correction 02-01 recorded, the criterion is that the field must not advance into the task window. Identical before and after is the passing reading.

ns/PVC	lastAutoSalvagedAt before → after	remountRequestedAt before → after	Advanced?
mosquitto/mosquitto-data	"" → ""	2026-05-12T00:53:01Z → 2026-05-12T00:53:01Z	no
dolt/data-dolt-0	"" → ""	2026-05-12T12:02:01Z → 2026-05-12T12:02:01Z	no
uptime-kuma/uptime-kuma-data	"" → ""	2026-08-07T13:53:30Z → 2026-08-07T13:53:30Z	no
headroom-agents/headroom-agents-workspace	"" → ""	"" → ""	no
fovea/fovea-audit-js	"" → ""	"" → ""	no
headroom-apps/headroom-apps-workspace	"" → ""	"" → ""	no
agent-memory/qdrant-data	"" → ""	"" → ""	no
fovea/qdrant-fovea-data	"" → ""	"" → ""	no
octopus/qdrant-octopus-data	"" → ""	"" → ""	no
karakeep/karakeep-data	"" → ""	2026-08-07T13:53:30Z → 2026-08-07T13:53:30Z	no
renovate/renovate-ce-cache-hi	"" → ""	2026-08-07T13:53:30Z → 2026-08-07T13:53:30Z	no
renovate/renovate-ce-db	"" → ""	2026-08-07T13:53:30Z → 2026-08-07T13:53:30Z	no

[VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io, read immediately before each delete and again after each reattach, 2026-08-29T21:01Z–22:11Z] — no field advanced on any of the twelve. lastAutoSalvagedAt remains empty estate-wide; every non-empty remountRequestedAt is a historical node-restart date and is byte-identical across the restart. Longhorn did not have to rescue any of these volumes.

snapshotsError was empty on all twelve both before and after.

Gate movement attributable to Task 2

=== post-Task-2 LH-03 gate re-read, 2026-08-29T22:00:52Z ===
(a) engine processes outside v1.12.1 : 26   (was 38)
(a-inverted) control, inside v1.12.1 : 14   (was 2)

[VERIFIED: kubectl, 2026-08-29T22:00:52Z] — twelve engines moved, and both halves of the gate moved by exactly twelve. That the two queries change by the same magnitude in opposite directions is itself a check that they partition the same set.

Finding: renovate-ce restarts are slow for a reason that is not Longhorn

renovate-ce took from 21:40:02Z to 22:09:52Z to reach Ready. Both volumes attached promptly — the VolumeAttachment objects are stamped 21:50:29Z and 21:50:40Z and both engines were in v1.12.1 instance-managers well before the container started. The delay is entirely kubelet’s recursive fsGroup ownership change over the cache volume:

Warning  VolumePermissionChangeInProgress  kubelet
  Setting volume ownership for …/pvc-6991ee39-…/mount is taking longer than expected,
  consider using OnRootMismatch
  … processed 160684 files.
  … processed 563707 files.
  … processed 1570506 files.

[VERIFIED: kubectl -n renovate describe pod renovate-ce-649d779bf7-n45s4, 2026-08-29T22:06Z]

1.57 million files on an 18.95 GiB cache, chowned one at a time on every pod start. This is a pre-existing property of the workload’s pod spec — an fsGroup without fsGroupChangePolicy: OnRootMismatch — and it happens on every renovate-ce restart, not only this one. It is not incident residue and not a drain failure, and no verdict input is affected: the volume came back attached/healthy, the filesystem reported 18.95 GiB used, and the workload did reach Ready. Carried to ## Open question as a candidate todo.

Findings

nats/nats-js-nats-{0,1,2} — the only explicit kernel data-loss flag

These three volumes carry the sole EXT4-fs … potential data loss line from the 2026-07-27 event (inode 524325, under comm nats-server, docs/operations/evidence/2026-08-23-tpi-beta-4-console-ring.md). They get no special apparatus. Their restart in 02-02 is where NATS’ own JetStream startup consistency output runs, and that output is the evidence: a Restored N messages for stream … in Xms line per stream followed by Server is ready. Reading an instrument the workload already publishes is the whole method here; building a checksum harness around third-party storage is out of scope.

Two constraints on how that reading must be taken, both measured. First, default.otel_logs retains roughly 14 days, so the 2026-07-27 event is outside the window — there is no retrospective log audit available and none should be assumed. Second, NATS carries a chronic [ERR] … invalid character in client ID "mosquitto-…nats-bridge" from the mosquitto bridge that repeats every few seconds; it is a real pre-existing defect and not a drain failure, so the NATS verdict must be a delta against that standing rate.

postgres/main-15 — already resolved, confirmed by live re-read

main-15 volumes found: 0
main-16   Bound   pvc-780c5703-…   50Gi   RWO   postgres-storage   137d
main-17   Bound   pvc-b8815886-…   50Gi   RWO   postgres-storage   137d
main-20   Bound   pvc-bc28579b-…   50Gi   RWO   postgres-storage   6d3h

[VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io; kubectl -n postgres get pvc, 2026-08-29T20:45Z] — no Longhorn volume and no PVC named main-15 exists. It was destroyed and rebuilt as main-20 on 2026-08-23, whose 6-day PVC age corroborates the rebuild date. Closed.

No Longhorn volume matches vector — with a mechanism, not a bare negative

vector-matching Longhorn volumes: 0
PVCs matching vector: 0
--- vector DaemonSet volume kinds ---
vector  config  projected
vector  data  hostPath
vector  var-log  hostPath
vector  var-lib  hostPath
vector  procfs  hostPath
vector  sysfs  hostPath
vector  buffer  emptyDir

[VERIFIED: kubectl get ds -n vector -o json; kubectl get pvc -A, 2026-08-29T20:45Z]

A vector DaemonSet does exist, but it has no persistentVolumeClaim of any kind — only projected, hostPath and emptyDir volumes — and no PVC anywhere in the cluster matches vector. So the console ring’s dm-1 … comm vector line at −26.8 d cannot name a Longhorn volume owned by vector, and there is nothing to audit under that name.

[ASSUMED: the most likely explanation is that vector's var-libhostPath covers/var/lib, which is where the kubelet mounts Longhorn CSI volumes at /var/lib/kubelet/pods//volumes/kubernetes.io~csi/pvc-/mount, so the kernel named vectoras the writingcomm on a device belonging to some other workload. This is a plausible reading, not a verified one, and it does not change the disposition.]

Correction: remountRequestedAt is not empty estate-wide

Research recorded remountRequestedAt as set on 0 of 40 volumes. A live re-read finds it set on 19 of 40, mealie/mealie-data among them at 2026-05-12T13:05:07Z. lastAutoSalvagedAt is genuinely empty on 40 of 40.

Field	Research §1.2	Live re-read	Reading
lastAutoSalvagedAt set	0 / 40	0 / 40	confirmed
remountRequestedAt set	0 / 40	19 / 40	research premise falsified

[VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io -o json, 2026-08-29T20:42Z]

Every one of the 19 timestamps is historical — clustered on 2026-05-12, 2026-07-03 and 2026-08-07, all node-restart dates — and none is recent. The consequence for this phase is a change of criterion shape, not of verdict: the rescue-field input must be read as a delta across the restart (the field must not advance into the task window), never as an absolute emptiness test. An emptiness test would have failed the tracer on a value written three and a half months before this phase began. mealie/mealie-data read 2026-05-12T13:05:07Z both before and after its restart — unchanged, which is the passing reading.

This is the same correction the log input already required, arriving from a second direction: an absolute-zero criterion against a live estate is wrong by construction.

NATS JetStream restore output, verbatim

D-03 gives the nats-js-nats-{0,1,2} volumes no special apparatus. Their restart is where NATS’ own JetStream startup recovery runs, and that output is the evidence. Each block below is the untouched container log from kubectl -n nats logs <pod> -c nats, from Starting nats-server through Server is ready. No consistency check, checksum or restore-and-compare of our own was added — Longhorn’s snapshot-data-integrity cronjob already runs a checksum verification every 7 days cluster-wide, and building a second one beside it is the apparatus D-01 and D-09 forbid.

nats-0 is a re-read; nats-1 and nats-2 are the first post-incident reads. Container startedAt before this wave: nats-0 2026-08-07T13:54:57Z (eleven days after the 2026-07-27 event, already clean once), nats-1 2026-05-12T01:08:52Z and nats-2 2026-07-03T18:09:03Z — both predating the event, so their restarts here are the first time those two volumes have been read since it. nats-1 is also the only one of the three whose engine was still on a v1.10.1 instance-manager.

[VERIFIED: kubectl -n nats get pods -o json, container startedAt, 2026-08-29T22:15Z]

nats-0 — re-read of an already-clean post-incident start; engine v1.11.2 → v1.12.1

[7] 2026/08/29 22:17:03.129652 [INF] Starting nats-server
[7] 2026/08/29 22:17:03.129740 [INF]   Version:  2.12.6
[7] 2026/08/29 22:17:03.129743 [INF]   Git:      [0e06390]
[7] 2026/08/29 22:17:03.129745 [INF]   Cluster:  nats
[7] 2026/08/29 22:17:03.129747 [INF]   Name:     nats-0
[7] 2026/08/29 22:17:03.129750 [INF]   Node:     S1Nunr6R
[7] 2026/08/29 22:17:03.129752 [INF]   ID:       NBCXCRSP5TGORETIRV5MDITZK6DB6QQQ37AHUSKLWSKIQQGXOSDJK2KM
[7] 2026/08/29 22:17:03.129767 [INF] Using configuration file: /etc/nats-config/nats.conf (sha256:bf7d5d5eb0ffe36733e6afbdb4bf5fa045deeb3b2599e8a30f39a5cd57f46eb3)
[7] 2026/08/29 22:17:03.129774 [INF] Trusted Operators
[7] 2026/08/29 22:17:03.129775 [INF]   System  : ""
[7] 2026/08/29 22:17:03.129777 [INF]   Operator: "fzymgc-house"
[7] 2026/08/29 22:17:03.129779 [INF]   Issued  : 2026-01-05 01:06:46 +0000 UTC
[7] 2026/08/29 22:17:03.129795 [INF]   Expires : Never
[7] 2026/08/29 22:17:03.140020 [INF] Starting http monitor on 0.0.0.0:8222
[7] 2026/08/29 22:17:03.140335 [INF] Managing all jwt in exclusive directory /data/jwt
[7] 2026/08/29 22:17:03.140361 [INF] Starting JetStream
[7] 2026/08/29 22:17:03.148316 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[7] 2026/08/29 22:17:03.148349 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[7] 2026/08/29 22:17:03.148354 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[7] 2026/08/29 22:17:03.148358 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[7] 2026/08/29 22:17:03.148362 [INF]
[7] 2026/08/29 22:17:03.148366 [INF]          https://docs.nats.io/jetstream
[7] 2026/08/29 22:17:03.148371 [INF]
[7] 2026/08/29 22:17:03.148375 [INF] ---------------- JETSTREAM ----------------
[7] 2026/08/29 22:17:03.148382 [INF]   Strict:          true
[7] 2026/08/29 22:17:03.148407 [INF]   Max Memory:      256.00 MB
[7] 2026/08/29 22:17:03.148421 [INF]   Max Storage:     10.00 GB
[7] 2026/08/29 22:17:03.148426 [INF]   Store Directory: "/data/jetstream"
[7] 2026/08/29 22:17:03.148431 [INF]   Domain:          fzymgc
[7] 2026/08/29 22:17:03.148436 [INF]   API Level:       3
[7] 2026/08/29 22:17:03.148442 [INF] -------------------------------------------
[7] 2026/08/29 22:17:03.165342 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_msgs'
[7] 2026/08/29 22:17:03.172338 [INF]   Restored 0 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_msgs' in 7ms
[7] 2026/08/29 22:17:03.180943 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_out'
[7] 2026/08/29 22:17:03.186513 [INF]   Restored 0 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_out' in 6ms
[7] 2026/08/29 22:17:03.194782 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_qos2in'
[7] 2026/08/29 22:17:03.201157 [INF]   Restored 0 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_qos2in' in 6ms
[7] 2026/08/29 22:17:03.211667 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_rmsgs'
[7] 2026/08/29 22:17:03.220742 [INF]   Restored 3 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_rmsgs' in 9ms
[7] 2026/08/29 22:17:03.227497 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_sess'
[7] 2026/08/29 22:17:03.234151 [INF]   Restored 3 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_sess' in 7ms
[7] 2026/08/29 22:17:03.236155 [INF] Starting JetStream cluster
[7] 2026/08/29 22:17:03.236165 [INF] Creating JetStream metadata controller
[7] 2026/08/29 22:17:03.249182 [INF] JetStream cluster recovering state
[7] 2026/08/29 22:17:03.303814 [INF] Took 163.448904ms to start JetStream
[7] 2026/08/29 22:17:03.303881 [INF] Listening for MQTT clients on tls://0.0.0.0:1883
[7] 2026/08/29 22:17:03.303905 [INF] Listening for client connections on 0.0.0.0:4222
[7] 2026/08/29 22:17:03.303908 [INF] TLS required for client connections
[7] 2026/08/29 22:17:03.304095 [INF] Server is ready

nats-1 — FIRST post-incident read; engine v1.10.1 → v1.12.1; this volume carries the 2026-07-27 EXT4-fs … potential data loss flag on inode 524325 under comm nats-server

[7] 2026/08/29 22:32:44.709036 [INF] Starting nats-server
[7] 2026/08/29 22:32:44.709095 [INF]   Version:  2.12.6
[7] 2026/08/29 22:32:44.709098 [INF]   Git:      [0e06390]
[7] 2026/08/29 22:32:44.709100 [INF]   Cluster:  nats
[7] 2026/08/29 22:32:44.709101 [INF]   Name:     nats-1
[7] 2026/08/29 22:32:44.709105 [INF]   Node:     yrzKKRBu
[7] 2026/08/29 22:32:44.709106 [INF]   ID:       NDD5VCX5RVC7WTTM4JASGHRFZWT275QVKTUM4EJGDJ5ZTJSTT33FKHU6
[7] 2026/08/29 22:32:44.709117 [INF] Using configuration file: /etc/nats-config/nats.conf (sha256:0f5b66ce9cf9935ff4ece5003bfcdeb5a512fbd5790ae7a194f2d62436b1b7a5)
[7] 2026/08/29 22:32:44.709126 [INF] Trusted Operators
[7] 2026/08/29 22:32:44.709128 [INF]   System  : ""
[7] 2026/08/29 22:32:44.709130 [INF]   Operator: "fzymgc-house"
[7] 2026/08/29 22:32:44.709133 [INF]   Issued  : 2026-01-05 01:06:46 +0000 UTC
[7] 2026/08/29 22:32:44.709152 [INF]   Expires : Never
[7] 2026/08/29 22:32:44.718257 [INF] Starting http monitor on 0.0.0.0:8222
[7] 2026/08/29 22:32:44.718776 [INF] Managing all jwt in exclusive directory /data/jwt
[7] 2026/08/29 22:32:44.718815 [INF] Starting JetStream
[7] 2026/08/29 22:32:44.726405 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[7] 2026/08/29 22:32:44.726432 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[7] 2026/08/29 22:32:44.726437 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[7] 2026/08/29 22:32:44.726442 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[7] 2026/08/29 22:32:44.726445 [INF]
[7] 2026/08/29 22:32:44.726449 [INF]          https://docs.nats.io/jetstream
[7] 2026/08/29 22:32:44.726452 [INF]
[7] 2026/08/29 22:32:44.726456 [INF] ---------------- JETSTREAM ----------------
[7] 2026/08/29 22:32:44.726461 [INF]   Strict:          true
[7] 2026/08/29 22:32:44.726481 [INF]   Max Memory:      256.00 MB
[7] 2026/08/29 22:32:44.726490 [INF]   Max Storage:     10.00 GB
[7] 2026/08/29 22:32:44.726495 [INF]   Store Directory: "/data/jetstream"
[7] 2026/08/29 22:32:44.726500 [INF]   Domain:          fzymgc
[7] 2026/08/29 22:32:44.726505 [INF]   API Level:       3
[7] 2026/08/29 22:32:44.726511 [INF] -------------------------------------------
[7] 2026/08/29 22:32:44.744097 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_msgs'
[7] 2026/08/29 22:32:44.752273 [INF]   Restored 0 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_msgs' in 8ms
[7] 2026/08/29 22:32:44.767320 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_out'
[7] 2026/08/29 22:32:44.771934 [INF]   Restored 0 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_out' in 5ms
[7] 2026/08/29 22:32:44.780204 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_qos2in'
[7] 2026/08/29 22:32:44.785026 [INF]   Restored 0 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_qos2in' in 5ms
[7] 2026/08/29 22:32:44.794757 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_rmsgs'
[7] 2026/08/29 22:32:44.802360 [INF]   Restored 3 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_rmsgs' in 8ms
[7] 2026/08/29 22:32:44.810437 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_sess'
[7] 2026/08/29 22:32:44.816626 [INF]   Restored 3 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_sess' in 6ms
[7] 2026/08/29 22:32:44.818665 [INF] Starting JetStream cluster
[7] 2026/08/29 22:32:44.818674 [INF] Creating JetStream metadata controller
[7] 2026/08/29 22:32:44.833507 [INF] JetStream cluster recovering state
[7] 2026/08/29 22:32:44.891436 [INF] Took 172.612667ms to start JetStream
[7] 2026/08/29 22:32:44.891502 [INF] Listening for MQTT clients on tls://0.0.0.0:1883
[7] 2026/08/29 22:32:44.891542 [INF] Listening for client connections on 0.0.0.0:4222
[7] 2026/08/29 22:32:44.891545 [INF] TLS required for client connections
[7] 2026/08/29 22:32:44.891702 [INF] Server is ready

nats-2 — FIRST post-incident read; engine v1.11.2 → v1.12.1

[7] 2026/08/29 22:39:14.745685 [INF] Starting nats-server
[7] 2026/08/29 22:39:14.745843 [INF]   Version:  2.12.6
[7] 2026/08/29 22:39:14.745851 [INF]   Git:      [0e06390]
[7] 2026/08/29 22:39:14.745857 [INF]   Cluster:  nats
[7] 2026/08/29 22:39:14.745862 [INF]   Name:     nats-2
[7] 2026/08/29 22:39:14.745869 [INF]   Node:     cnrtt3eg
[7] 2026/08/29 22:39:14.745874 [INF]   ID:       NDO3ARPV2JHVGSPRNOIQDSQMDKDE3COQ7GLAQ6FOWD6LHQUZW2XBWVR2
[7] 2026/08/29 22:39:14.745913 [INF] Using configuration file: /etc/nats-config/nats.conf (sha256:4cbb7f66a00616a52ae25be61ed32e538dc752f5a704e5af37c759d958ad2189)
[7] 2026/08/29 22:39:14.745933 [INF] Trusted Operators
[7] 2026/08/29 22:39:14.745942 [INF]   System  : ""
[7] 2026/08/29 22:39:14.745947 [INF]   Operator: "fzymgc-house"
[7] 2026/08/29 22:39:14.745954 [INF]   Issued  : 2026-01-05 01:06:46 +0000 UTC
[7] 2026/08/29 22:39:14.745986 [INF]   Expires : Never
[7] 2026/08/29 22:39:14.757202 [INF] Starting http monitor on 0.0.0.0:8222
[7] 2026/08/29 22:39:14.757808 [INF] Managing all jwt in exclusive directory /data/jwt
[7] 2026/08/29 22:39:14.757856 [INF] Starting JetStream
[7] 2026/08/29 22:39:14.763458 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[7] 2026/08/29 22:39:14.763473 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[7] 2026/08/29 22:39:14.763475 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[7] 2026/08/29 22:39:14.763476 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[7] 2026/08/29 22:39:14.763478 [INF]
[7] 2026/08/29 22:39:14.763480 [INF]          https://docs.nats.io/jetstream
[7] 2026/08/29 22:39:14.763481 [INF]
[7] 2026/08/29 22:39:14.763482 [INF] ---------------- JETSTREAM ----------------
[7] 2026/08/29 22:39:14.763484 [INF]   Strict:          true
[7] 2026/08/29 22:39:14.763494 [INF]   Max Memory:      256.00 MB
[7] 2026/08/29 22:39:14.763497 [INF]   Max Storage:     10.00 GB
[7] 2026/08/29 22:39:14.763499 [INF]   Store Directory: "/data/jetstream"
[7] 2026/08/29 22:39:14.763501 [INF]   Domain:          fzymgc
[7] 2026/08/29 22:39:14.763502 [INF]   API Level:       3
[7] 2026/08/29 22:39:14.763505 [INF] -------------------------------------------
[7] 2026/08/29 22:39:14.772912 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_msgs'
[7] 2026/08/29 22:39:14.780475 [INF]   Restored 0 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_msgs' in 8ms
[7] 2026/08/29 22:39:14.787202 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_out'
[7] 2026/08/29 22:39:14.792303 [INF]   Restored 0 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_out' in 5ms
[7] 2026/08/29 22:39:14.799763 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_qos2in'
[7] 2026/08/29 22:39:14.803596 [INF]   Restored 0 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_qos2in' in 4ms
[7] 2026/08/29 22:39:14.811614 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_rmsgs'
[7] 2026/08/29 22:39:14.816893 [INF]   Restored 3 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_rmsgs' in 5ms
[7] 2026/08/29 22:39:14.824754 [INF]   Starting restore for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_sess'
[7] 2026/08/29 22:39:14.830041 [INF]   Restored 3 messages for stream 'ABQESRHVJWOVTMGUWYIPVZBXXVDBZMSD7EIW27PDUR243XNQUSHEUTJP > $MQTT_sess' in 5ms
[7] 2026/08/29 22:39:14.831634 [INF] Starting JetStream cluster
[7] 2026/08/29 22:39:14.831645 [INF] Creating JetStream metadata controller
[7] 2026/08/29 22:39:14.845902 [INF] JetStream cluster recovering state
[7] 2026/08/29 22:39:14.908051 [INF] Took 150.182451ms to start JetStream
[7] 2026/08/29 22:39:14.908114 [INF] Listening for MQTT clients on tls://0.0.0.0:1883
[7] 2026/08/29 22:39:14.908148 [INF] Listening for client connections on 0.0.0.0:4222
[7] 2026/08/29 22:39:14.908152 [INF] TLS required for client connections
[7] 2026/08/29 22:39:14.908313 [INF] Server is ready

[VERIFIED: kubectl -n nats logs <pod> -c nats, 2026-08-29T22:40Z]

All three are the clean sequence. Each restored the same five MQTT streams with the same message counts — $MQTT_msgs 0, $MQTT_out 0, $MQTT_qos2in 0, $MQTT_rmsgs 3, $MQTT_sess 3, six messages in total — then Starting JetStream cluster, JetStream cluster recovering state, a Took …ms to start JetStream line and Server is ready. Not one line between the first Starting restore and Server is ready is at a level other than [INF], on any of the three. No Filestore … encountered internal inconsistencies, no Rebuilding index, no Corrupt state file — and those strings were never used as a matcher, because RESEARCH marks them [ASSUMED]. The actual text was read; the criterion is that the sequence is the clean one, and it is.

Two facts that bound what this proves, recorded so a later reader does not over-read it. The JetStream store holds five MQTT streams totalling six messages, and each nats-js-* filesystem reports roughly 460 KiB used of a 10 GiB volume. These volumes carry MQTT session state, not a dataset. A clean restore of six messages is real evidence that the volumes read correctly, and it is not evidence about large-volume integrity.

Cluster state after all three restarts, from NATS’ own monitoring endpoint

=== /routez on nats-0, 2026-08-29T22:42:22Z ===
{"num_routes":8,"routes":[
  {"remote_name":"nats-1","ip":"10.245.2.241"},{"remote_name":"nats-1","ip":"10.245.2.241"},
  {"remote_name":"nats-1","ip":"10.245.2.241"},{"remote_name":"nats-1","ip":"10.245.2.241"},
  {"remote_name":"nats-2","ip":"10.245.6.118"},{"remote_name":"nats-2","ip":"10.245.6.118"},
  {"remote_name":"nats-2","ip":"10.245.6.118"},{"remote_name":"nats-2","ip":"10.245.6.118"}]}

=== /jsz on nats-0 ===
{"cluster":"nats","leader":"nats-1","streams":5,"messages":6}

[VERIFIED: kubectl -n nats exec nats-0 -c nats -- wget -qO- http://localhost:8222/{routez,jsz}, 2026-08-29T22:42:22Z] — full route mesh, a JetStream metadata leader elected, and 5 streams / 6 messages, identical to the pre-restart content. This is the strongest single reading in the nats set: it is JetStream’s own accounting of what survived, taken after all three volumes moved.

The nats restarts were strictly one at a time

Pod	Delete	Ready	Siblings Ready throughout
nats-0	22:15:30Z, and 22:16:4xZ for the cordon retry	22:17:03Z	nats-1, nats-2 both True
nats-1	22:17:59Z	22:32:44Z	nats-0, nats-2 both True
nats-2	22:34:38Z	22:39:14Z	nats-0, nats-1 both True

[VERIFIED: readiness polled between every delete; the next delete was issued only after the previous member reported Ready=True] — at no point were two nats pods simultaneously not Ready. The StatefulSet runs podManagementPolicy: Parallel with a PDB of maxUnavailable: 1 (disruptionsAllowed: 1), and a PDB constrains eviction rather than a direct pod delete, so the one-at-a-time rule here was operator discipline and not an enforced constraint. Quorum held at 2 of 3 for the whole wave, including the fifteen-minute nats-1 stall.

Correction: the prescribed error-class matcher is blind to NATS’ error format

The matcher RESEARCH §5.4 prescribes tests Body for error, fatal, panic and corrupt. NATS does not write any of those words. It writes [ERR], [WRN] and [FTL].

=== nats namespace, 2 hours to 2026-08-29T22:14Z ===
bridge_lines                            : 284
bridge_lines_ALSO_matching_error_class  :   0
error_class_total                       :   0

[VERIFIED: default.otel_logs, 2026-08-29T22:14Z] — the prescribed matcher scores zero against 284 real [ERR] lines. Run alone on this namespace it returns a flattering 0 → 0 that means nothing at all, on precisely the three volumes D-03 singled out.

Every nats row above therefore carries two readings: the prescribed matcher, for comparability with the other rows, and a NATS-aware matcher on [ERR]/[FTL] with the chronic bridge class excluded, which is the reading the verdict actually rests on.

The elevated nats log counts, read rather than counted

The NATS-aware matcher returns large after-counts — 4,620 on nats-0, 1,100 on nats-1. Reading them settles what they are:

=== distinct [ERR] body classes in nats since 22:15:30, normalised, bridge class excluded ===
[ERR] Error trying to connect to route (attempt N): lookup for host
  "nats-0.nats-headless.nats.svc.cluster.local": ... no such host
[ERR] Error trying to connect to route (attempt N): lookup for host
  "nats-2.nats-headless.nats.svc.cluster.local": ... no such host

=== the two negative controls ===
[FTL] lines since the drain began                                        : 0
lines matching corrupt|inconsistenc|rebuilding index|corrupt state file  : 0

=== when they stopped ===
last route error : 2026-08-29 22:39:15  (nats-2 rejoining)

[VERIFIED: default.otel_logs, 2026-08-29T22:42Z]

There is exactly one class, and it is the surviving members’ route-reconnect loop failing to resolve a peer’s headless DNS record while that peer was down. The count is large because NATS retries once a second and logs each attempt — the (attempt N) counter runs to 261 — not because there are many distinct faults. It is self-inflicted by our own restarts, self-resolving, and it stopped at 22:39:15Z the moment nats-2 rejoined; /routez then showed the full mesh. The two negative controls are the load-bearing part: zero [FTL] and zero lines matching any corruption vocabulary, across the whole drain window.

The vector hypothesis, upgraded from [ASSUMED] to verified

02-01 recorded, as [ASSUMED], that vector’s var-lib hostPath covers /var/lib — where the kubelet mounts Longhorn CSI volumes — and that this is why the 2026-07-27 console ring named comm vector on dm-1, a device belonging to some other workload. This wave verified that mechanism directly, because it stopped the drain twice.

nats-1 sat in ContainerCreating for fifteen minutes with Multi-Attach error. The volume was attached/healthy the whole time — on the wrong node. The chain:

=== NodeUnstageVolume on tpi-beta-2, repeating every ~2 min ===
level=error msg="NodeUnstageVolume: err: rpc error: code = Internal desc =
  failed after 10 attempts to refresh stale namespace directory: failed to execute:
  /usr/bin/nsenter [... cryptsetup luksClose pvc-8a4953ea-af0b-4a8c-9fee-f84a963bd42b],
  stderr Device pvc-8a4953ea-af0b-4a8c-9fee-f84a963bd42b is still in use.: exit status 5"

=== the dm devices it could not close ===
pvc-245682b6-23af-4fbc-a66e-6d5083b898aa (253:0)   <- mosquitto/mosquitto-data, ALREADY migrated
pvc-8a4953ea-af0b-4a8c-9fee-f84a963bd42b (253:1)   <- nats/nats-js-nats-1
  State: ACTIVE   Open count: 1   (both)

=== mounted anywhere in the host namespace? ===
NONE MOUNTED IN HOST NS

=== scan of /proc/*/mounts across every namespace on the node ===
pid 3387677: /usr/bin/vector --config-dir /etc/vector/

[VERIFIED: kubectl -n longhorn-system logs longhorn-csi-plugin-q2f2n; nsenter into the host mount namespace for dmsetup infoand the/proc/*/mounts scan, 2026-08-29T22:29Z–22:31Z]

vector was the sole holder. Its mount namespace, captured at its own start, still contained the kubelet’s CSI mount entries for both volumes. That pinned each LUKS device at Open count: 1, so cryptsetup luksClose failed, so NodeUnstageVolume failed, so kubelet never cleared node.status.volumesInUse, so the attach/detach controller never issued ControllerUnpublish, so the volume stayed attached to tpi-beta-2 and nats-1 could not start anywhere else.

The remedy is one pod delete of the vector DaemonSet pod on the source node:

=== after deleting vector-xgdmm on tpi-beta-2, 22:31Z ===
Open count: 0   (both devices)     holders: (none)

=== the very next NodeUnstage retry ===
22:32:23 "Closing LUKS device pvc-8a4953ea-af0b-4a8c-9fee-f84a963bd42b"
22:32:24 "Volume pvc-8a4953ea-af0b-4a8c-9fee-f84a963bd42b unmounted from node path ..."   <- no error
=> volume detached from tpi-beta-2, re-attached on tpi-alpha-2, nats-1 Ready 22:32:44Z

[VERIFIED: kubectl, 2026-08-29T22:31Z–22:32:44Z]

It then reproduced on a second node. nats-2 stalled identically for 200 s on tpi-alpha-4; deleting vector-m289j there released it and the pod was Ready 80 s later. Two independent occurrences, same mechanism, same remedy.

This is the mechanism the console ring’s dm-1 … comm vector line records, demonstrated rather than inferred: vector genuinely holds references to Longhorn CSI volume mounts belonging to other workloads. The [ASSUMED] marker on that attribution in 02-01 can be discharged. It does not change that finding’s disposition — there is still no Longhorn volume owned by vector, and nothing to audit under that name.

Consequence for later waves. Every volume in this estate is LUKS-encrypted (encrypted: "true" on the StorageClass), so every remaining detach can hit this. When a volume stays attached to its old node after the consuming pod is gone, and the pod on the new node reports Multi-Attach error, the sequence is: confirm Open count on the dm device, scan /proc/*/mounts for the holder, delete the vector pod on the source node, and wait one NodeUnstage retry cycle. It is not a Longhorn fault and no instance-manager is involved.

Rescue fields across the nats restarts

ns/PVC	lastAutoSalvagedAt before → after	remountRequestedAt before → after	Advanced?
nats/nats-js-nats-0	"" → ""	2026-08-07T13:53:30Z → 2026-08-07T13:53:30Z	no
nats/nats-js-nats-1	"" → ""	2026-05-12T00:53:00Z → 2026-05-12T00:53:00Z	no
nats/nats-js-nats-2	"" → ""	2026-07-03T18:05:37Z → 2026-07-03T18:05:37Z	no

[VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io, before each delete and after each reattach, 2026-08-29T22:15Z–22:40Z] — no field advanced, snapshotsError empty on all three both before and after. Longhorn did not have to rescue any nats volume across the drain, including the one carrying the kernel data-loss flag.

Gate movement attributable to Task 3

=== post-Task-3 LH-03 gate re-read, 2026-08-29T22:43:10Z ===
(a) engine processes outside v1.12.1 : 23   (was 26)
(a-inverted) control, inside v1.12.1 : 17   (was 14)

[VERIFIED: kubectl, 2026-08-29T22:43:10Z] — three engines moved, both halves by exactly three.

Wave 02-04 — start-of-wave re-read and the tier-2a work list

Tier 2a is the half of D-06 tier 2 that is not the audit’s own instrument: the three-node valkey cluster and the three-member ClickStack MongoDB replica set. ClickHouse and Keeper — the query path this record’s log verdicts are read through — are deliberately left to 02-05, so that every verdict in this wave is still read through a healthy telemetry stack.

=== start-of-wave gate re-read: 2026-08-29T23:54:07Z ===
--- (a) engine processes still outside v1.12.1 ---
23
--- (a-inverted) CONTROL: engine processes INSIDE v1.12.1 IMs ---
17
--- (b) stale instance-managers ---
instance-manager-1b8c1c4d55fc05f3f1a9af947ff100a4 v1.10.1 tpi-beta-3 engines=1 replicas=0
instance-manager-27af93a4524a66b6e9b1bc3a6e0a6920 v1.11.2 tpi-alpha-2 engines=2 replicas=0
instance-manager-58b7a48389b11c2c7750c5940cb6802b v1.10.1 tpi-alpha-4 engines=2 replicas=0
instance-manager-5c94660e40a52187b9a427c2cb679808 v1.11.2 tpi-alpha-1 engines=3 replicas=0
instance-manager-68fca8eecce9b1d85ba3f5986b1faae4 v1.11.2 tpi-beta-1 engines=3 replicas=0
instance-manager-996e594d45afc211165dc4b7bfca856b v1.11.2 tpi-beta-4 engines=4 replicas=0
instance-manager-9c18146a00a741ccff3e5d9983c2f788 v1.11.2 tpi-alpha-3 engines=2 replicas=0
instance-manager-9e303193f6af67a4962641022b0ed2ac v1.10.1 tpi-beta-2 engines=1 replicas=0
instance-manager-b48dac91b45fa0a6f4869005f61020b4 v1.11.2 tpi-alpha-4 engines=2 replicas=0
instance-manager-ddbf6cfd1f950d8cb36be16153997003 v1.10.1 tpi-alpha-2 engines=2 replicas=0
instance-manager-fc5f937c05ebc017992613b0f02d73ee v1.11.2 tpi-beta-3 engines=1 replicas=0
--- (c) volumes not attached+healthy on v1.12.1 image ---
--- (c) row count ---
0
--- total volumes ---
40

[VERIFIED: kubectl, 2026-08-29T23:54:07Z] — the inverted-predicate control returned 17, strictly greater than zero, so (a)’s reading of 23 is answered by a query that demonstrably reaches engine processes rather than by a query that has stopped working.

Reconciliation against 02-02

Reading	UTC	(a) outside v1.12.1	control, inside v1.12.1
02-02 close of wave	2026-08-29T22:43:10Z	23	17
02-04 start of wave	2026-08-29T23:54:07Z	23	17

Drift since 02-02: zero, across 71 minutes. No engine migrated without our intervention, and no volume this phase left pending has moved on its own. No row changes to already-migrated on this reading. As in 02-02 this is a measurement rather than a guarantee — the estate moved 31 → 30 unaided between 2026-08-24 and 2026-08-29 — so the reconciliation is re-run at the top of every wave and a nonzero result is recorded as drift this phase did not cause, never as our own progress and never as a failure.

Volume → instance-manager map for tier 2a, re-derived live

Rebuilt for all 40 volumes by joining each engine’s .status.instanceManagerName (never .spec.instanceManagerName, which is empty on every engine here and reads like a clean estate) to the instance-manager’s spec.image, then filtered to the nine tier-2a volumes.

=== 2026-08-29T23:54:25Z — 40/40 volumes resolved, 23 to a pre-v1.12.1 IM ===
clickstack/data-volume-cs-clickstack-mongodb-0  node=tpi-alpha-3  attached/healthy/v1.12.1  IM=instance-manager-9c18146a00a741ccff3e5d9983c2f788 [v1.11.2]
clickstack/data-volume-cs-clickstack-mongodb-1  node=tpi-alpha-2  attached/healthy/v1.12.1  IM=instance-manager-ddbf6cfd1f950d8cb36be16153997003 [v1.10.1]
clickstack/data-volume-cs-clickstack-mongodb-2  node=tpi-alpha-4  attached/healthy/v1.12.1  IM=instance-manager-58b7a48389b11c2c7750c5940cb6802b [v1.10.1]
clickstack/logs-volume-cs-clickstack-mongodb-0  node=tpi-alpha-3  attached/healthy/v1.12.1  IM=instance-manager-9c18146a00a741ccff3e5d9983c2f788 [v1.11.2]
clickstack/logs-volume-cs-clickstack-mongodb-1  node=tpi-alpha-2  attached/healthy/v1.12.1  IM=instance-manager-ddbf6cfd1f950d8cb36be16153997003 [v1.10.1]
clickstack/logs-volume-cs-clickstack-mongodb-2  node=tpi-alpha-4  attached/healthy/v1.12.1  IM=instance-manager-58b7a48389b11c2c7750c5940cb6802b [v1.10.1]
valkey/valkey-valkey-0-0-data                   node=tpi-alpha-1  attached/healthy/v1.12.1  IM=instance-manager-5c94660e40a52187b9a427c2cb679808 [v1.11.2]
valkey/valkey-valkey-0-1-data                   node=tpi-alpha-1  attached/healthy/v1.12.1  IM=instance-manager-5c94660e40a52187b9a427c2cb679808 [v1.11.2]
valkey/valkey-valkey-0-2-data                   node=tpi-alpha-1  attached/healthy/v1.12.1  IM=instance-manager-5c94660e40a52187b9a427c2cb679808 [v1.11.2]

[VERIFIED: kubectl volumes/engines/instancemanagers, 2026-08-29T23:54:25Z] — all nine are still outside v1.12.1 at the moment the work list is built, so none of them is an already-migrated row. The currentImage column reading v1.12.1 on every one of them is the engine binary, not the engine process placement — that distinction is the whole of LH-03 and the reason the gate is read through the instance-manager rather than through the volume.

D-05 is at its sharpest here. All three valkey engines share instance-manager-5c94660e40a52187b9a427c2cb679808 on tpi-alpha-1 (now 3 engines, down from 4 — uptime-kuma left it in 02-02). One delete of that pod would take the entire valkey cluster in a single action. The only sanctioned move remains a workload pod delete.

valkey live member roles

=== 2026-08-29T23:54:37Z ===
NAME         READY   ROLE      POD
valkey-0-0   true    replica   valkey-valkey-0-0-0
valkey-0-1   true    primary   valkey-valkey-0-1-0
valkey-0-2   true    replica   valkey-valkey-0-2-0

NAME     STATE   REASON           AGE
valkey   Ready   ClusterHealthy   54d

valkey-valkey-0-0-0   2/2   Running   10.245.1.216   tpi-alpha-1
valkey-valkey-0-1-0   2/2   Running   10.245.1.58    tpi-alpha-1
valkey-valkey-0-2-0   2/2   Running   10.245.1.44    tpi-alpha-1

[VERIFIED: kubectl -n valkey get valkeynodes.valkey.io + valkeyclusters.valkey.io, 2026-08-29T23:54:37Z] — exactly one primary (valkey-0-1, unchanged from the research reading) and two replicas. All three pods are on tpi-alpha-1, which is why all three engines share one instance-manager and why the cordon fallback is expected here rather than exceptional.

Correction: the valkey namespace DOES have a PodDisruptionBudget

02-RESEARCH.md §3.2 states:

There is a PDB for litellm’s valkey (valkey-litellm-valkey, maxUnavailable: 1) but none for the valkey namespace’s cluster — nothing enforces one-at-a-time there.

That is false. The valkey namespace carries pdb/valkey-valkey, created 2026-07-06 — seven weeks before this phase — owned by the ValkeyCluster/valkey CR itself:

=== kubectl -n valkey get pdb valkey-valkey, 2026-08-29T23:54:48Z ===
selector        : valkey.io/cluster=valkey
maxUnavailable  : 1
status          : currentHealthy=3 desiredHealthy=2 disruptionsAllowed=1 expectedPods=3
ownerReferences : ValkeyCluster/valkey
pods matched    : valkey-valkey-0-0-0, valkey-valkey-0-1-0, valkey-valkey-0-2-0

[VERIFIED: kubectl, 2026-08-29T23:54:48Z]

The operational conclusion is unchanged, and it is worth being precise about why. A PodDisruptionBudget constrains the Eviction API — kubectl drain, the node-pressure and descheduler paths. It does not constrain kubectl delete pod, which is the verb this wave uses. So one-at-a-time here is still the operator’s discipline, exactly as the plan assumes; it is simply not true that nothing would enforce it under a different verb. Recorded so that a later reader who runs kubectl get pdb -n valkey and finds one does not conclude the drain was gated by it, and so that the same reader does not conclude from the PDB’s presence that a careless second delete would have been refused. It would not have been.

MongoDB live member states

rs.status() is the only source: the mongodbcommunity CR exposes phase: Running and nothing about member roles. The credential is read from the clickstack namespace Secret and handed to the shell through the process environment only — never as a command-line argument, never echoed — because pod stdout lands in default.otel_logs, which is queryable with roughly 14 days of retention.

=== rs.status(), 2026-08-30T00:01:36Z, credential via _getEnv from the pod environment ===
SET=cs-clickstack-mongodb date=2026-08-30T00:01:36.926Z
cs-clickstack-mongodb-0.cs-clickstack-mongodb-svc.clickstack.svc.cluster.local:27017 SECONDARY health=1 self=true
cs-clickstack-mongodb-1.cs-clickstack-mongodb-svc.clickstack.svc.cluster.local:27017 SECONDARY health=1 self=false
cs-clickstack-mongodb-2.cs-clickstack-mongodb-svc.clickstack.svc.cluster.local:27017 PRIMARY   health=1 self=false

[VERIFIED: kubectl -n clickstack exec cs-clickstack-mongodb-0 -c mongod, 2026-08-30T00:01:36Z] — exactly one PRIMARY (cs-clickstack-mongodb-2) and two SECONDARY, all three health=1. Note this is not where research left it by implication: the primary is the last ordinal, so the OrderedReady StatefulSet ordering and the replica-set ordering happen to agree here, which they need not on the next reading.

Correction: mongosh hangs in this deployment even WITH credentials — use the legacy mongo shell

02-RESEARCH.md §4.2 and §13 attribute the observed two-minute mongosh hang to the absence of credentials, and prescribe “supply credentials and budget a timeout”. Supplying credentials does not fix it. Measured here, every one of these hangs until the timeout kills it:

mongosh --version                                              -> 2.6.0, exit=0
mongosh --nodb --quiet --eval 'print(1+1)'                     -> 2,     exit=0
mongosh 'mongodb://127.0.0.1:27017/?directConnection=true' ... -> no output at all, exit=124
mongosh --nodb --eval 'new Mongo("mongodb://127.0.0.1:27017")' -> no output at all, exit=124
mongosh --nodb --eval 'connect(<authenticated URI>)'           -> no output at all, exit=124

[VERIFIED: kubectl -n clickstack exec cs-clickstack-mongodb-0 -c mongod, 2026-08-29T23:56Z–23:58Z]

mongosh 2.6.0 hangs before it prints even its own startup banner the moment a connection string is involved, with --nodb evaluation working fine — so the failure is in its connection path, not in authentication and not in the server. The server is healthy and listening: /proc/net/tcp shows 0.0.0.0:27017 in state 0A, and the co-installed legacy mongo 5.0 shell answers immediately on the same socket.

The working verb, and why it is also the safer one:

# credential -> opaque base64 on stdin -> decoded inside the pod -> pod environment.
# Never argv, never echoed, and never decoded on the client at all.
PW_B64=$(kubectl -n clickstack get secret cs-clickstack-mongodb-password -o jsonpath='{.data.password}' | tr -d '\n')
kubectl -n clickstack exec -i cs-clickstack-mongodb-0 -c mongod -- sh -s <<EOF
HOME=/tmp
MONGO_USER=hyperdx
MONGO_AUTHDB=hyperdx
MONGO_PW=\$(printf %s '$PW_B64' | base64 -d)
export HOME MONGO_PW MONGO_USER MONGO_AUTHDB
timeout 30 mongo --quiet --host 127.0.0.1 --port 27017 --eval '
  var ok = db.getSiblingDB(_getEnv("MONGO_AUTHDB")).auth(_getEnv("MONGO_USER"), _getEnv("MONGO_PW"));
  if (!ok) { print("AUTH_FAILED"); quit(1); }
  var s = db.getSiblingDB("admin").runCommand({replSetGetStatus: 1});
  s.members.forEach(function(m) { print(m.name + " " + m.stateStr + " health=" + m.health); });
'
EOF

The legacy shell’s _getEnv() is what makes the credential-handling requirement satisfiable at all: mongosh’s process.env route was the only other way to keep the password out of argv, and mongosh does not work here. timeout 30 is present in the container (/usr/bin/timeout) and bounds every call. HOME=/tmp is required — the default HOME=/data/db is not writable by the shell’s uid and the client warns on every invocation.

Correction: the block’s original quoting did not generalise — three forms, measured

As first recorded, the block decoded the password on the client and interpolated it into a single-quoted assignment inside an unquoted heredoc. Against the chart-default literal actually in use that worked, and the argv and stdout guarantees held. It does not generalise. A password containing a single quote closes that quote, and because the heredoc body is the script sh -s reads on stdin, the remainder is swallowed as string rather than merely erroring.

A review of this record proposed emitting the assignment with printf ahead of a quoted heredoc body. That does not fix it, and recording why is the point of this correction rather than quietly substituting something else: the value still reaches the remote shell as a bare assignment right-hand side, so an unbalanced quote there fails in the same way and for the same reason.

Three candidate forms were measured against a synthetic value containing a single quote, a dollar sign and a space, with a local sh standing in for the pod and the delivered value compared by checksum rather than printed:

Form	Intact?	Result
1. as first recorded — client-decoded value interpolated into a single-quoted assignment	no	sh reports an unterminated string and consumes the rest of the heredoc
2. the reviewed proposal — emit the assignment with printf ahead of a quoted heredoc body	no	identical failure; still a bare assignment right-hand side in the remote shell’s source
3. carry .data.password as opaque base64, decode inside the pod	yes	value reproduced byte for byte

Form 3 is what is written above. Base64 output is alphanumerics plus +, / and =, so the string embedded in the remote shell’s source cannot contain a quote, a dollar or a space for any password. It also removes the client-side decode, so the plaintext never exists on the client at all. Two controls: form 3 delivers an ordinary alphanumeric value intact, and delivers a value containing a newline intact — base64 encodes a newline as ordinary alphanumerics and base64 -d restores it.

The residual boundary is on the client side, not on the password. What has to hold is that the base64 interpolated into the heredoc arrives as a single unwrapped line, which is what the tr -d '\n' on the kubectl get secret pipeline is for — GNU base64 wraps at 76 columns, so a locally re-encoded value would otherwise be folded. Quoting is not a constraint on the value: quotes, backslashes, $, backticks and interior newlines are all carried intact, so an operator has no reason to hand-edit a password to get it through. The one measured exception is a trailing newline — MONGO_PW=$(… | base64 -d) is a command substitution, and command substitution strips trailing newlines, so secret\n (7 bytes) arrives as 6.

Measured rather than assumed, the tr -d '\n' is a guard on the shape of the interpolation rather than the last thing standing between this form and a break: a deliberately folded encoding was fed through the same single-quoted printf %s and still decoded correctly, because the fold lands inside the quotes and base64 -d ignores newlines in its input. Keep the tr regardless — it is what stops the block silently depending on both of those tolerances at once.

Two smaller corrections in the same area.

1. The Secret the replica set actually authenticates against is not ExternalSecret-backed. cs-clickstack-mongodb-password — the passwordSecretRef on the mongodbcommunity CR, and the one read above — carries app.kubernetes.io/managed-by: Helm and argocd.argoproj.io/tracking-id: clickstack:/Secret:clickstack/cs-clickstack-mongodb-password. It is rendered by the clickstack-3.1.1 chart and reconciled by ArgoCD. None of the four ExternalSecrets in the namespace (clickstack-secret, clickstack-watchdog, pushover-hyperdx-tokens, vault-pki-ca-bundle) targets it. The clickstack-secret ExternalSecret does carry a MONGODB_PASSWORD key, but as a chart-default literal rather than a Vault property — argocd/app-configs/clickstack/external-secret.yaml:52-56 says so in a comment, and has since 2026-05-12. So both routes hold the same chart default and the plan’s “ExternalSecret-backed” sourcing sentence names a mechanism that is not the one in use. The trust boundary the prohibition cares about — in-cluster Secret to process environment, never to argv and never to stdout — is identical either way, and it is honoured. [VERIFIED: kubectl + repo at HEAD, 2026-08-29T23:55Z]

   Consequence for the credential-leak control. Because this password is a chart-default literal that has been committed in the repository since 2026-05-12 and is annotated there as deliberate, a literal-substring sweep of this record for its value is not a meaningful control: it neither detects a disclosure this wave caused nor would a clean result prove one did not happen. The load-bearing control is therefore the command shape, reproduced verbatim above: the value travels Secret → still-encoded base64 on the client → kubectl exec -i stdin → decoded inside the pod → pod environment → _getEnv(). It never appears in any argv, is never echoed, and never reaches pod stdout, so it cannot land in default.otel_logs. Under the corrected form above the plaintext additionally never exists on the client at all.

2. hyperdx@hyperdx is sufficient for replSetGetStatus. No fallback to the operator’s mms-automation agent credential was needed, so the agent password was never read.

A cheap second opinion exists and agrees. The mongodb-agent sidecar writes /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json, whose ReplicationStatus field is the same member-state enum (1 = PRIMARY, 2 = SECONDARY). It read 2 on cs-clickstack-mongodb-0 at the same moment rs.status() did. It is a useful fallback if the shell route ever fails, but it is per-pod and reports no view of the other members, so it does not replace the rs.status() read.

Tier-2a work list, keyed by owning pod

Pod-to-volume pairing is derived live from volumes.longhorn.io .status.kubernetesStatus.workloadsStatus and independently corroborated by the kubelet /stats/summary sweep across the four nodes involved. Each cs-clickstack-mongodb-N pod holds two volumes — one data-volume- and one logs-volume- — so one delete moves two engines and fills two verdict rows. A naive per-volume loop would delete each mongodb pod twice.

Task 2 — valkey, replicas first, primary last (3 pod deletes, 3 volumes):

#	Pod delete	Node	Role at 23:54:37Z	IM version	Volume moved	FS used
1	valkey-valkey-0-0-0	tpi-alpha-1	replica	v1.11.2	valkey/valkey-valkey-0-0-data	0.07 MiB of 1.9 GiB
2	valkey-valkey-0-2-0	tpi-alpha-1	replica	v1.11.2	valkey/valkey-valkey-0-2-data	0.06 MiB of 1.9 GiB
3	valkey-valkey-0-1-0	tpi-alpha-1	primary	v1.11.2	valkey/valkey-valkey-0-1-data	0.06 MiB of 1.9 GiB

Task 3 — cs-clickstack-mongodb, secondaries first, primary last (3 pod deletes, 6 volumes):

#	Pod delete	Node	stateStr at 00:01:36Z	IM version	Volumes moved (both from one restart)	FS used
4	cs-clickstack-mongodb-0	tpi-alpha-3	SECONDARY	v1.11.2 ×2	clickstack/data-volume-…-mongodb-0 + clickstack/logs-volume-…-mongodb-0	594.32 MiB + 67.36 MiB
5	cs-clickstack-mongodb-1	tpi-alpha-2	SECONDARY	v1.10.1 ×2	clickstack/data-volume-…-mongodb-1 + clickstack/logs-volume-…-mongodb-1	594.04 MiB + 67.19 MiB
6	cs-clickstack-mongodb-2	tpi-alpha-4	PRIMARY	v1.10.1 ×2	clickstack/data-volume-…-mongodb-2 + clickstack/logs-volume-…-mongodb-2	593.29 MiB + 67.10 MiB

`[VERIFIED: kubectl volumes/engines/instancemanagers + kubelet /stats/summary across tpi-alpha-1..4

• live role reads, 2026-08-29T23:54:07Z–2026-08-30T00:01:51Z]` — 6 pod deletes clear 9 tier-2a volumes. No pod identifier appears twice. Neither cluster’s primary is restarted before its own replicas or secondaries. Roles are re-read immediately before each delete regardless of this table: a valkey primary moves on its own, and a MongoDB primary is guaranteed to move when it is stepped down, so this table is the starting order and not the authority at step 3 or step 6.

Instance-manager mutation control, baseline. 19 instance-managers present at 2026-08-30T00:01:51Z, captured by name, image, node and creation timestamp; diffed against the same capture at the end of each task. [VERIFIED: kubectl get instancemanagers.longhorn.io]

Node scheduling state, baseline. All 8 nodes Ready, none SchedulingDisabled, at 2026-08-30T00:01:51Z. 02-02 twice left a node cordoned when a shell call timed out mid-fallback, so this is re-checked at the end of every task in this wave, not only at the end of the wave.

Task 2 — the valkey drain, replicas first and the primary last

Three single-replica StatefulSets behind one ValkeyCluster CR, all three pods on tpi-alpha-1, all three engines in one pre-v1.12.1 instance-manager. Restarted one at a time in the order Task 1 recorded, with the role re-read from valkeynodes.valkey.io immediately before each delete.

Step	Pod	ROLE at the moment of the delete	Role read at	T0 (delete)	Ready	Ready in	Node move	IM move
1	valkey-valkey-0-0-0	replica	00:07:13Z	00:07:26Z	00:08:15Z	49 s	tpi-alpha-1 → tpi-alpha-4	v1.11.2 → v1.12.1
2	valkey-valkey-0-2-0	replica	00:09:26Z	00:09:27Z	00:10:02Z	35 s	tpi-alpha-1 → tpi-alpha-4	v1.11.2 → v1.12.1
3	valkey-valkey-0-1-0	primary	00:10:44Z	00:10:46Z	00:11:23Z	37 s	tpi-alpha-1 → tpi-alpha-4	v1.11.2 → v1.12.1

[VERIFIED: kubectl, 2026-08-30T00:07:13Z–00:11:31Z] — no delete targeted a node whose ROLE read primary while another node’s ROLE read replica. The primary went last, and it was still valkey-0-1 at the moment it was deleted: restarting the two replicas did not move the primary, and the operator returned valkey-0-1 to primary after its own restart rather than electing a different node.

The gate between each pair of deletes, both halves recorded, never one:

Between	pod Ready True	valkeyclusters.valkey.io/valkey	Siblings at that moment
1 → 2	valkey-valkey-0-0-0 at 00:08:15Z	Ready / ClusterHealthy at 00:08:21Z (after 55 s of Reconciling / UpdatingNodes)	0-1 and 0-2 both 2/2 Running
2 → 3	valkey-valkey-0-2-0 at 00:10:02Z	Ready / ClusterHealthy at 00:10:11Z	0-0 and 0-1 both 2/2 Running
after 3	valkey-valkey-0-1-0 at 00:11:23Z	Ready / ClusterHealthy at 00:11:35Z (after one Reconciling / UpdatingNodes reading)	0-0 and 0-2 both 2/2 Running

At no point were two valkey pods simultaneously not Ready. Note that the cluster CR is not a redundant reading of the pod condition: after step 1 the pod was Ready at 00:08:15Z while the CR still read Reconciling / UpdatingNodes for a further six seconds. Gating on the pod alone would have allowed the second delete into a cluster the operator had not finished reconciling.

Positive control: the prescribed matcher CAN see valkey’s error format

02-02 established that a prescribed matcher must be measured against the workload’s real output before a zero from it is believed — it scored zero against 284 real NATS [ERR] lines. That control is re-run here, and this time it passes: valkey writes plain-English error text that the prescribed error|fatal|panic|corrupt matcher matches directly.

=== valkey namespace, 24 h to 2026-08-30T00:05Z ===
matched sample (all three pods):
  1:S 29 Aug 2026 03:35:50.258 # Error accepting a client connection:
      error:0A000126:SSL routines::unexpected eof while reading (addr= laddr=10.245.1.216:6379)
all-lines control, same window: valkey-0-0 499 / valkey-0-1 473 / valkey-0-2 466

[VERIFIED: default.otel_logs via cs-clickstack-clickhouse-clickhouse-0-0-0, 2026-08-30T00:05Z] — so a 0 in the after-column of a valkey row is a genuine zero, not the NATS-style blindness. The matched class is a chronic client-disconnect artefact (unexpected eof while reading, Connection reset by peer) that predates the drain and runs at roughly 5–15 lines per 30 minutes per pod; it is exactly what the per-pod baseline exists to absorb.

The three after-windows, read rather than counted

=== distinct matched classes, valkey, since each delete anchor, normalised ===
after step 1 (>= 00:07:26Z) : "# Error accepting a client connection: SSL ... unexpected eof"  x1  (0-0)
                              "# Error accepting a client connection: Connection reset by peer" x1  (0-0)
                              "# Error accepting a client connection: SSL ... unexpected eof"  x1  (0-1)
after step 2 (>= 00:09:27Z) : "# Error accepting a client connection: SSL ... unexpected eof"  x1  (0-0)
after step 3 (>= 00:10:46Z) : "# Error condition on socket for SYNC: Connection refused"       x2  (0-0)
                              "# Error condition on socket for SYNC: Connection refused"       x2  (0-2)
                              "# Error accepting a client connection: SSL ... unexpected eof"  x1  (0-2)

=== negative control, all three windows ===
corrupt|inconsistenc|Bad file format|Short read : 0, 0, 0

[VERIFIED: default.otel_logs, 2026-08-30T00:08Z, 00:10Z and 00:12Z]

There are exactly two classes across the whole drain. The first is the chronic client-disconnect class established as the baseline above. The second appears only after step 3 and only on the two surviving replicas: # Error condition on socket for SYNC: Connection refused is the replicas’ replication socket failing while the primary they replicate from is between pods. It is self-inflicted by our own restart, self-resolving, and it stopped as soon as valkey-0-1 was back — the cluster read ClusterHealthy twelve seconds later. Neither class is a storage fault, and the corruption-vocabulary negative control is zero in every window.

Rescue fields across the valkey restarts — read as a delta

ns/PVC	lastAutoSalvagedAt before → after	remountRequestedAt before → after	Advanced?
valkey/valkey-valkey-0-0-data	"" → ""	"" → ""	no
valkey/valkey-valkey-0-2-data	"" → ""	"" → ""	no
valkey/valkey-valkey-0-1-data	"" → ""	"" → ""	no

[VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io, read immediately before each delete and again after each reattach, 2026-08-30T00:06:21Z–00:11:31Z] — no field advanced on any of the three, and snapshotsError was empty on all three both before and after. These three sit in the minority of the estate whose remountRequestedAt is empty rather than carrying a historical node-restart date, so for them the delta reading and the emptiness reading happen to agree.

The shared instance-manager was reaped by Longhorn, not by us

instance-manager-5c94660e40a52187b9a427c2cb679808 (v1.11.2, tpi-alpha-1) held all three valkey engines at the start of this task. Deleting it would have taken the entire cluster in one action, which is the sharpest case for D-05 in the estate. It was never touched. Once its last engine left with step 3, Longhorn reaped it on its own:

=== IM inventory diff, 2026-08-30T00:01:51Z vs 00:12:20Z ===
< instance-manager-5c94660e40a52187b9a427c2cb679808 v1.11.2 tpi-alpha-1 2026-06-06T17:36:52Z
(one removal, no additions, no image or node change on any surviving object)

[VERIFIED: kubectl get instancemanagers.longhorn.io, diffed across the task window] — the single line of difference is the disappearance of the object whose engines we drained, which is the documented reaping behaviour (an instance-manager is reaped when both instanceEngines and instanceReplicas are empty). No instance-manager was deleted or patched by this task, and none was created: all three engines landed in instance-manager-3536df4b049a0ba362c2b953d0af899b, the v1.12.1 instance-manager on tpi-alpha-4 that already existed — 02-02 put renovate-ce’s two engines into it.

Correction: a StatefulSet readiness poll must key on the pod UID, not the pod name

The first delete’s readiness gate reported READY after 0s — which is false. A StatefulSet replacement reuses the pod name, so a poll of kubectl get pod <name> -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' issued immediately after delete --wait=false reads the terminating pod’s stale Ready=True and returns instantly. The gate was fixed before the second delete to require a pod whose .metadata.uid differs from the one captured before the delete and whose .metadata.deletionTimestamp is empty; steps 2 and 3 then reported honest 38-second waits, and the independently-read Ready condition lastTransitionTime (00:08:15Z, 00:10:02Z, 00:11:23Z) corroborates all three.

This matters beyond cosmetics: the whole point of the between-deletes gate is that the replacement is serving before the next member goes down, and a name-keyed poll cannot see the difference. It is the same defect class as the two matcher failures 02-02 recorded — a check that returns a passing answer without measuring the thing it claims to measure. Step 1’s actual recovery is not in doubt (Ready at 00:08:15Z, 49 s after the delete, on a new node with a new UID), and no second delete was issued until the cluster CR itself read ClusterHealthy. Waves 02-05 through 02-07 restart StatefulSets exclusively and must use the UID-keyed gate.

Gate movement attributable to Task 2

=== post-Task-2 LH-03 gate re-read, 2026-08-30T00:12:26Z ===
(a) engine processes outside v1.12.1 : 20   (was 23)
(a-inverted) control, inside v1.12.1 : 20   (was 17)

[VERIFIED: kubectl, 2026-08-30T00:12:26Z] — three engines moved, and both halves of the gate moved by exactly three in opposite directions.

Node scheduling state after Task 2: all 8 nodes Ready, none cordoned, re-checked at 00:12:20Z. Every one of the three cordon windows was closed by the drain helper’s own uncordon, and the helper traps EXIT/INT/TERM so a killed shell cannot leave tpi-alpha-1 unschedulable — the failure that happened twice in 02-02.

Task 3 — the ClickStack MongoDB drain, secondaries first and the primary last

A three-member mongodbcommunity replica set on an OrderedReady StatefulSet. Each pod holds two Longhorn volumes — a data-volume- and a logs-volume- — so each of the three deletes moved two engines and filled two verdict rows. No pod was deleted twice, and no per-volume loop was run.

Step	Pod	stateStr at the moment of the delete	Read at	T0 (delete)	Ready	Ready in	Node move	IM move	Volumes moved
1	cs-clickstack-mongodb-0	SECONDARY	00:15:00Z	00:15:02Z	00:16:18Z	76 s	tpi-alpha-3 → tpi-alpha-1	v1.11.2 → v1.12.1	data + logs
2	cs-clickstack-mongodb-1	SECONDARY	00:17:31Z	00:17:33Z	00:18:48Z	75 s	tpi-alpha-2 → tpi-alpha-3	v1.10.1 → v1.12.1	data + logs
3	cs-clickstack-mongodb-2	PRIMARY	00:19:24Z	00:19:26Z	00:20:41Z	75 s	tpi-alpha-4 → tpi-alpha-2	v1.10.1 → v1.12.1	data + logs

[VERIFIED: kubectl + rs.status(), 2026-08-30T00:15:00Z–00:21:07Z] — no delete targeted a member reading PRIMARY while another member read SECONDARY. Each role read was taken from a member that was not the target (steps 1 and 2 from cs-clickstack-mongodb-2, step 3 from cs-clickstack-mongodb-0), so the reading never came from the pod about to be destroyed.

The membership gate between each pair of deletes. Kubernetes readiness and replica-set membership are different facts, and the plan is explicit that the former does not stand in for the latter:

Between	pod Ready True	member back per a fresh rs.status()	Set state at that moment
1 → 2	cs-clickstack-mongodb-0 at 00:16:18Z	SECONDARY at 00:16:33Z (15 s after Ready)	0 SECONDARY, 1 SECONDARY, 2 PRIMARY
2 → 3	cs-clickstack-mongodb-1 at 00:18:48Z	SECONDARY at 00:19:07Z (19 s after Ready)	0 SECONDARY, 1 SECONDARY, 2 PRIMARY
after 3	cs-clickstack-mongodb-2 at 00:20:41Z	SECONDARY at 00:21:06Z (25 s after Ready)	0 PRIMARY, 1 SECONDARY, 2 SECONDARY

The gap between Ready and rejoin was 15–25 seconds on every step, so the distinction is not academic: a run gated on readiness alone would have issued the next delete while the previous member was still outside the set, and with three members that is the window in which the write majority is lost. The write majority held at 2 of 3 throughout.

Step 3 stepped the primary down, as expected — a primary is guaranteed to move when it is restarted, which is why the role is re-read before every step rather than once at the top of the wave. cs-clickstack-mongodb-0 was elected in its place and cs-clickstack-mongodb-2 rejoined as a secondary. The set ends with exactly one PRIMARY and two SECONDARY, all three health=1.

Two matchers, because the prescribed one is imprecise here in the opposite direction to NATS

For NATS the prescribed matcher was blind — it saw none of 284 real error lines. For MongoDB it is the reverse: it is over-sensitive. MongoDB writes structured JSON with an explicit severity field, and the prescribed substring matcher fires on the word error wherever it appears in a message body, including in informational text:

=== mongodb pods, 24 h to 2026-08-30T00:14Z ===
"s":"E" (error)   :      0
"s":"F" (fatal)   :      0
"s":"W" (warning) :  2,964
"s":"I" (info)    : 409,561
total             : 412,525

matched by the prescribed matcher, sample:
  {"s":"I","c":"NETWORK","id":4712102,"ctx":"ReplicaSetMonitor-TaskExecutor",
   "msg":"Host failed in replica set", ...}

[VERIFIED: default.otel_logs via cs-clickstack-clickhouse-clickhouse-0-0-0, 2026-08-30T00:14Z]

So every row carries both readings: the prescribed matcher for comparability with the other 34 rows in this record, and a MongoDB-aware matcher on the severity field itself, which is the reading the verdict rests on. The MongoDB-aware matcher reads 0 → 0 on all three steps — the replica set logged not one E or F line across the entire drain, including its own restarts.

The prescribed matcher’s after-counts, read rather than counted

=== distinct matched classes after step 1 (>= 00:15:02Z), by severity/component/msg ===
 96  I NETWORK  Host failed in replica set              (mongodb-1)
 91  I -        RSM received error response             (mongodb-1)
 78  I NETWORK  Host failed in replica set              (mongodb-2)
 73  I -        RSM received error response             (mongodb-2)
 25  I REPL_HB  Heartbeat failed after max retries      (mongodb-1)
 25  I REPL_HB  Heartbeat failed after max retries      (mongodb-2)
  4  W NETWORK  Failed to check socket connectivity     (mongodb-2)
  3  W NETWORK  Failed to check socket connectivity     (mongodb-1)
  3  I CONNPOOL Dropping all pooled connections         (mongodb-0)
  2  I NETWORK  Host failed in replica set              (mongodb-0)

[VERIFIED: default.otel_logs, 2026-08-30T00:16Z]

Every class is at severity I or W, and every one of them is the two surviving members noticing that the member we just deleted is gone. That is the correct behaviour of a replica set during a rolling restart, it is self-inflicted by our own delete, and it is self-resolving — the same shape as the NATS route-retry class 02-02 recorded, and for the same reason. This is why the row’s per-pod delta is read against a per-pod baseline and why the severity-field reading is the one the verdict rests on: on step 3 the restarted pod’s own prescribed count fell from 352 to 14 while its siblings’ counts rose, which is the signature of peer-noticing rather than of a fault in the restarted member.

The negative control, tightened after reading its hits

The first pass of the corruption-vocabulary control returned 6, which would have stopped the wave. Reading the six settled it:

4  I STORAGE WiredTiger message                     (mongodb-0)
1  I REPL    Initializing rollback ID               (mongodb-0)
1  I REPL    Searching for local Rollback ID document (mongodb-0)

[VERIFIED: default.otel_logs, 2026-08-30T00:17Z]

All six matched on the token rollback, and all six are routine startup bookkeeping: MongoDB initialises and reads its rollback-ID document on every start. rollback is not corruption vocabulary in MongoDB’s dialect — it is ordinary replication machinery — so it was dropped from the control and data loss added in its place. The tightened control (corrupt|inconsistenc|WT_TRY_SALVAGE|data loss) returns 0 in all three windows.

This is the third time in this phase that a criterion has had to be measured before it could be believed, and the first time the failure ran in the false-positive direction on a control rather than on the primary matcher. Recording the reading rather than the count is what separated them.

Rescue fields across the mongodb restarts — read as a delta

ns/PVC	lastAutoSalvagedAt before → after	remountRequestedAt before → after	Advanced?
clickstack/data-volume-cs-clickstack-mongodb-0	"" → ""	2026-08-07T13:53:30Z → 2026-08-07T13:53:30Z	no
clickstack/logs-volume-cs-clickstack-mongodb-0	"" → ""	2026-08-07T13:53:30Z → 2026-08-07T13:53:30Z	no
clickstack/data-volume-cs-clickstack-mongodb-1	"" → ""	"" → ""	no
clickstack/logs-volume-cs-clickstack-mongodb-1	"" → ""	"" → ""	no
clickstack/data-volume-cs-clickstack-mongodb-2	"" → ""	"" → ""	no
clickstack/logs-volume-cs-clickstack-mongodb-2	"" → ""	"" → ""	no

[VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io, read immediately before each delete and again after each reattach, 2026-08-30T00:15:01Z–00:21:07Z] — no field advanced on any of the six. mongodb-0’s pair carries a historical remountRequestedAt of 2026-08-07, byte-identical across the restart, which is the reading 02-01’s correction was written for: the criterion is that the field must not advance into the task window, not that it is empty. snapshotsError was empty on all six both before and after.

The Phase 3 overlap, recorded rather than rediscovered

Phase 3’s CS-02 climbs this replica set from MongoDB 5.0.32 to 8.0 across three hops. Each hop restarts these same three pods and therefore re-detaches these same six volumes, so this wave’s mongodb work will be redone at least three times over in Phase 3. Each of the six rows carries the note inline.

That repetition is deliberate, not an error. D-04 chose to drive the LH-03 gate to zero in this phase rather than block it on Phase 3, and Phase 3 benefits directly: it starts on an estate where no engine is two minors behind, so any instability observed during the version ladder cannot be confounded with instance-manager version skew. A later reader finding these volumes detached again in Phase 3 should read it as the planned consequence of D-04.

Four stale instance-managers reaped by Longhorn, none touched by us

=== IM inventory diff, 2026-08-30T00:01:51Z vs 00:21:30Z ===
< instance-manager-58b7a48389b11c2c7750c5940cb6802b v1.10.1 tpi-alpha-4 2025-12-06T02:21:50Z   (mongodb-2's pair)
< instance-manager-5c94660e40a52187b9a427c2cb679808 v1.11.2 tpi-alpha-1 2026-06-06T17:36:52Z   (all three valkey)
< instance-manager-9c18146a00a741ccff3e5d9983c2f788 v1.11.2 tpi-alpha-3 2026-06-06T17:36:46Z   (mongodb-0's pair)
< instance-manager-ddbf6cfd1f950d8cb36be16153997003 v1.10.1 tpi-alpha-2 2025-12-06T02:21:52Z   (mongodb-1's pair)

instance-managers CREATED after 00:01:51Z : 0

[VERIFIED: kubectl get instancemanagers.longhorn.io, diffed across the wave window, 2026-08-30T00:21:30Z]

Four removals, zero additions, and no image or node change on any surviving object. Each removal is an instance-manager whose engine set this wave emptied, disappearing on its own once both instanceEngines and instanceReplicas were empty. No instancemanagers.longhorn.io object was deleted or patched by either task, and every one of the nine engines landed in a v1.12.1 instance-manager that already existed. Two of the four reaped objects are the ones D-05 names as the worst-case deletes in the estate — 5c94660e… held the entire valkey cluster.

Gate movement attributable to Task 3

=== post-Task-3 LH-03 gate re-read, 2026-08-30T00:21:30Z ===
(a) engine processes outside v1.12.1 : 14   (was 20)
(a-inverted) control, inside v1.12.1 : 26   (was 20)

[VERIFIED: kubectl, 2026-08-30T00:21:30Z] — six engines moved, both halves of the gate moved by exactly six in opposite directions. Across the whole of 02-04 the gate went 23 → 14 and its control 17 → 26, both by exactly nine, which is the arithmetic a nine-volume drain should produce.

Estate gate (c) — volumes not attached + healthy on a v1.12.1 image — returned 0 rows against 40 volumes at 00:21:30Z, and all 8 nodes read Ready with none cordoned.

Wave 02-05 — start-of-wave re-read and the tier-2b work list

Tier 2b is the half of D-06 tier 2 that is the audit’s own instrument: the two cs-clickstack-clickhouse replicas and the three cs-clickstack-keeper Raft members. Every other verdict in the phase except postgres and vault was already on the record before a single ClickStack pod was touched here, exactly as the ordering decision in 02-04 and RESEARCH §9.2 require.

=== start-of-wave gate re-read: 2026-08-30T00:31:29Z ===
--- (a) engine processes still outside v1.12.1 ---
14
--- (a-inverted) CONTROL: engine processes INSIDE v1.12.1 IMs ---
26
--- (b) stale instance-managers ---
instance-manager-1b8c1c4d55fc05f3f1a9af947ff100a4 v1.10.1 tpi-beta-3 engines=1 replicas=0
instance-manager-27af93a4524a66b6e9b1bc3a6e0a6920 v1.11.2 tpi-alpha-2 engines=2 replicas=0
instance-manager-68fca8eecce9b1d85ba3f5986b1faae4 v1.11.2 tpi-beta-1 engines=3 replicas=0
instance-manager-996e594d45afc211165dc4b7bfca856b v1.11.2 tpi-beta-4 engines=4 replicas=0
instance-manager-9e303193f6af67a4962641022b0ed2ac v1.10.1 tpi-beta-2 engines=1 replicas=0
instance-manager-b48dac91b45fa0a6f4869005f61020b4 v1.11.2 tpi-alpha-4 engines=2 replicas=0
instance-manager-fc5f937c05ebc017992613b0f02d73ee v1.11.2 tpi-beta-3 engines=1 replicas=0
--- (c) volumes not attached+healthy on v1.12.1 image ---
--- (c) row count ---
0
--- total volumes ---
40

[VERIFIED: kubectl, 2026-08-30T00:31:29Z] — the inverted-predicate control returned 26, strictly greater than zero, so (a)’s reading of 14 is a measurement rather than a broken query.

Reconciliation against 02-04

Reading	UTC	(a) outside v1.12.1	control, inside v1.12.1
02-04 close of wave	2026-08-30T00:24:17Z	14	26
02-05 start of wave	2026-08-30T00:31:29Z	14	26

Drift since 02-04: zero, across 7 minutes. No engine migrated unaided, and none of the five volumes this wave owns had moved on its own, so no row changes to already-migrated on this reading.

Volume → instance-manager map for tier 2b, re-derived live

=== 2026-08-30T00:31:42Z — 40/40 volumes resolved, 14 engines to a pre-v1.12.1 IM ===
clickstack/…clickhouse-clickhouse-0-0-0  node=tpi-beta-1  attached/healthy/v1.12.1  IM=instance-manager-68fca8eecce9b1d85ba3f5986b1faae4 [v1.11.2]
clickstack/…clickhouse-clickhouse-0-1-0  node=tpi-beta-3  attached/healthy/v1.12.1  IM=instance-manager-fc5f937c05ebc017992613b0f02d73ee [v1.11.2]
clickstack/…keeper-keeper-0-0            node=tpi-beta-4  attached/healthy/v1.12.1  IM=instance-manager-996e594d45afc211165dc4b7bfca856b [v1.11.2]
clickstack/…keeper-keeper-1-0            node=tpi-beta-4  attached/healthy/v1.12.1  IM=instance-manager-996e594d45afc211165dc4b7bfca856b [v1.11.2]
clickstack/…keeper-keeper-2-0            node=tpi-beta-4  attached/healthy/v1.12.1  IM=instance-manager-996e594d45afc211165dc4b7bfca856b [v1.11.2]

[VERIFIED: kubectl volumes/engines/instancemanagers, 2026-08-30T00:31:42Z] — all five still outside v1.12.1 at work-list time.

The wave-shaping fact research did not record: the two ClickHouse volumes are SINGLE-replica

Read live before the first delete, and it changes the drain technique for this wave alone:

Volume	spec.numberOfReplicas	spec.dataLocality	actual size	replica lives on
…clickhouse-clickhouse-0-0-0	1	best-effort	142.21 GiB	tpi-beta-1 (the attached node)
…clickhouse-clickhouse-0-1-0	1	best-effort	137.98 GiB	tpi-beta-3 (the attached node)
…keeper-keeper-{0,1,2}-0	2	best-effort	0.34–0.38 GiB	tpi-beta-4 + one other

[VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io -o json, 2026-08-30T00:31:50Z] — every other volume in the estate carries 2 or 3 replicas; these two carry one. Global default-data-locality and replica-auto-balance both read best-effort [VERIFIED: settings.longhorn.io, 2026-08-30T00:32Z].

The consequence is specific and expensive. 02-04’s inherited technique is cordon the source node so the replacement lands elsewhere. For a single-replica volume whose one replica is on the source node, landing elsewhere makes best-effort data locality schedule a new local replica on the target node and rebuild ~140 GiB across the network, leaving the volume degraded for the duration — which is also the state the acceptance criterion forbids. The rest of the estate never exposed this because two- and three-replica volumes almost always already have a replica on the target, and the largest single-replica case before this wave was 0.4 GiB.

So this wave needed a way to detach and reattach without changing node. See the correction below for what was tried, what was measured, and what actually works.

Correction: a Longhorn volume does NOT detach while its StatefulSet pod is merely unschedulable

The first attempt was a detach-in-place: cordon every node the pod could schedule to, delete the pod so the replacement is Pending everywhere, wait for the volume to reach detached, then reopen only the home node so the pod returns to its own node and the engine is recreated in that node’s current (v1.12.1) instance-manager with zero data movement.

It does not work. Measured twice, in both directions.

=== attempt on …clickhouse-0-1-0, 2026-08-30T00:53:40Z–01:03:41Z ===
cordon tpi-alpha-4 tpi-beta-1 tpi-beta-2 tpi-beta-3 tpi-beta-4
DELETE issued at 2026-08-30T00:53:41Z
TIMEOUT: state=attached after 600s
eng while pod Pending: state=running IM=[instance-manager-fc5f937c05ebc017992613b0f02d73ee]

[VERIFIED: kubectl, 2026-08-30T00:53:41Z–01:03:41Z] — the pod sat Pending for a full ten minutes with no node assigned, and the volume stayed attached on tpi-beta-3 the whole time with its engine still running in the v1.11.2 instance-manager. Reopening the home node returned the pod to tpi-beta-3, it remounted the still-attached volume, and the restart migrated nothing.

The earlier attempt on …clickhouse-0-0-0 had looked like it might work — its detach was observed about 4m24s after the delete — but the timing is the tell: the detach landed at the same moment the abort path uncordoned the cluster and the pod was scheduled onto tpi-alpha-4. The detach was caused by the pod being placed on a different node, not by its being Pending.

Mechanism, consistent with 02-02’s finding rather than contradicting it. 02-02 established that the VolumeAttachment is per-node and survives a same-node replacement. This wave adds the other half: the attachment also survives an absent replacement. Kubernetes’ attach/detach controller does not tear down a VolumeAttachment merely because the workload is unscheduled, so Longhorn’s csi-attacher ticket stays satisfied and the engine process is never destroyed. A volume detaches when its attachment is needed on another node — nothing weaker triggers it.

The verb that works, for a single-replica volume that must end up at home: move the pod off its node, then move it straight back.

step 1  kubectl cordon <home>                  # replacement must land elsewhere
        kubectl -n <ns> delete pod <pod>       # detach + attach on the new node -> engine migrates
        kubectl uncordon <home>
        # wait for Ready and for the operator CR to read ready
step 2  kubectl cordon <every other eligible node>
        kubectl -n <ns> delete pod <pod>       # only home is schedulable -> pod goes home
        kubectl uncordon <all>

Step 1 is what performs the migration; step 2 is what returns the data-local placement and discards the part-built remote replica. Both steps use only kubectl delete pod and node cordons — the same two sanctioned verbs 02-02 established, with no instance-manager and no ArgoCD-owned object touched. Cost is one extra restart per single-replica volume. It applies only to the two ClickHouse volumes; every other volume in the estate has a second replica and needs step 1 alone.

Task 1 — the ClickHouse drain, one replica at a time

Step	Pod	Delete issued	Bound to	Ready	Elapsed	Engine IM before → after
1a	…clickhouse-0-0-0	00:42:43Z	tpi-alpha-4 (all-cordon attempt aborted)	00:47:56Z	313 s	68fca8ee… v1.11.2 → 3536df4b… v1.12.1
1b	…clickhouse-0-0-0	00:48:26Z	tpi-beta-1 (home)	00:50:22Z	116 s	→ 7e6aabe3… v1.12.1, tpi-beta-1
2a	…clickhouse-0-1-0	00:53:41Z	tpi-beta-3 (home, after 602 s Pending)	01:04:19Z	638 s	unchanged — the no-op detach-in-place attempt
2b	…clickhouse-0-1-0	01:04:59Z	tpi-alpha-4	01:06:27Z	88 s	fc5f937c… v1.11.2 → 3536df4b… v1.12.1
2c	…clickhouse-0-1-0	01:06:38Z	tpi-beta-3 (home)	01:07:14Z	36 s	→ 4e199918… v1.12.1, tpi-beta-3

[VERIFIED: kubectl pod/volume/engine/instancemanager reads at each boundary, 2026-08-30T00:42:10Z–01:07:30Z]

The between-deletes gate. clickhouseclusters.clickhouse.com/cs-clickstack-clickhouse read READY True / All shards are ready / 2 of 2 at 00:53:24Z — after every …0-0-0 restart and before the first …0-1-0 delete — and again at 01:07:30Z at the end. …0-0-0 was 1/1 Running at 00:53:24Z while …0-1-0 was still untouched, and …0-0-0 stayed 1/1 Running throughout all three …0-1-0 restarts. At no point were both ClickHouse pods simultaneously not Ready, and the PDB cs-clickstack-clickhouse-clickhouse-0 (minAvailable: 1, disruptionsAllowed: 1) was never the thing enforcing it — as 02-04 recorded for valkey, a PDB gates the Eviction API and not kubectl delete pod; the one-at-a-time discipline is the operator’s, not the cluster’s.

The reattach-and-start duration, which is the number 02-08’s runbook wants. The plan expected these to be the slowest reattaches in the phase because they are the largest volumes. They are not. A clean single restart of the 142 GiB volume reached Ready in 116 s and the 138 GiB volume in 36 s (home) and 88 s (away). For comparison 02-02 measured uptime-kuma at 105 s on a 15 MiB volume and renovate-ce at ~30 minutes on 19 GiB. Volume size does not predict reattach time in this estate — renovate-ce’s cost was an fsGroup chown over 1.57 M files, and ClickHouse’s own start is 5 s from process start to Ready for connections. The figure to carry into the runbook is that a large Longhorn volume is not by itself a reason to budget a long window.

=== …clickhouse-0-0-0 own startup, read from otel_logs via the surviving replica ===
00:50:16.691 <Information> Application: Starting ClickHouse 26.7.1.1315 (revision: 54512)
00:50:16.691 <Information> Application: Available RAM: 8.00 GiB; logical cores: 8; used cores: 4.
00:50:17.290 <Information> Application: Listening for replica communication (interserver): http://[::]:9009
00:50:21.697 <Information> TablesLoader: Parsed metadata of 40 tables in 2 databases in 0.734 sec
00:50:21.755 <Information> Application: Ready for connections.

[VERIFIED: default.otel_logs via clickhouse-client on …0-1-0, 2026-08-30T00:53Z] — 5 s from process start to serving, with nothing between Starting ClickHouse and Ready for connections that is not <Information> or <Debug>. That sequence is the workload half of the verdict.

Correction: the ClickHouse log matcher fails in a fourth direction — the audit query is its own false positive

02-02 recorded the prescribed matcher failing as a false positive on a CSS token and as a false negative on NATS; 02-04 recorded it firing on error inside MongoDB’s informational message text. ClickHouse adds a new failure mode: the audit query is itself logged, and the next audit query matches it.

=== 30-minute pre-wave baseline, 00:11:00Z–00:41:00Z ===
pod                                         prescribed_raw   prescribed_noecho
cs-clickstack-clickhouse-clickhouse-0-0-0               31                   0
cs-clickstack-clickhouse-clickhouse-0-1-0                2                   0

[VERIFIED: default.otel_logs, 2026-08-30T00:41Z] — all 31 hits on …0-0-0 and both on …0-1-0 are ClickHouse’s own <Debug> executeQuery: echo of the text of this record’s own matcher queries, which contain the literal strings error, fatal, panic and corrupt. The pod with 31 is the pod the queries were being run against. Run naively, the matcher measures the auditor.

Two matchers are therefore recorded on every ClickHouse and Keeper row:

• prescribed_raw — RESEARCH §5.4’s matcher exactly as written, kept for comparability.
• anchored_EFC — the severity token in its own structural position: extract(Body,'\} <([A-Za-z]+)>') IN ('Error','Fatal','Critical'). ClickHouse and Keeper share one line format, YYYY.MM.DD HH:MM:SS.ffffff [ tid ] {query_id} <Severity> Logger: message, so the severity is a field rather than a substring. Lines whose anchored severity is Debug, Trace or Test are excluded from every count as query echo.

Positive control, both workloads, before either zero was believed — the required check from 02-02 and 02-04:

=== 24-hour window, 2026-08-29T00:41Z–2026-08-30T00:41Z ===
pod                                         anchored_EFC   raw_LIKE   anchored_Warning   all_lines
cs-clickstack-clickhouse-clickhouse-0-0-0             11         14                  0   8,638,419
cs-clickstack-clickhouse-clickhouse-0-1-0              8          8                  0   8,635,010
cs-clickstack-keeper-keeper-0-0                        0          0                  5         515
cs-clickstack-keeper-keeper-1-0                        0          0                 10         524
cs-clickstack-keeper-keeper-2-0                        0          0                 40         535

[VERIFIED: default.otel_logs, 2026-08-30T00:39Z]

• ClickHouse passes directly: 11 and 8 genuine <Error> lines exist in the last 24 h (<Error> executeQuery: Code: 33. DB::Exception: Cannot read all data), so the matcher demonstrably matches this workload’s error format and a zero from it is a real zero. The 14-vs-11 gap on …0-0-0 is exactly the self-referential contamination above.
• Keeper has no error line in the window, so its zero cannot be self-certifying. The control that makes it readable is anchored_Warning — 5, 10 and 40 real <Warning> RaftInstance: … lines. The anchored matcher reaches Keeper’s log body and parses its severity field; Error/Fatal is empty because Keeper logged none, not because the matcher is blind to the format. This is the NATS lesson applied prospectively rather than after a false zero.

Correction: the corruption negative control fires on ClickHouse’s own documentation text

The tightened 02-04 control (corrupt|inconsistenc|data loss|checksum doesn) returned 4 on …clickhouse-0-0-0 across its restart window, which by the acceptance criterion stops the wave. Read rather than counted, all four are the same line:

=== all 4 hits, position of the matched token = 84202 in every one ===
00:48:05.983  Old: CREATE TABLE system.metric_log (…)
00:48:05.996  New: CREATE TABLE system.metric_log (…)
00:50:28.349  Old: CREATE TABLE system.metric_log (…)
00:50:28.350  New: CREATE TABLE system.metric_log (…)
… 4. Non-deterministic merge algorithm due to logical error in code.
   5. Data corruption in memory due to bug in code.
   6. Data corruption in memory due to hardware issue. …

[VERIFIED: default.otel_logs, positionCaseInsensitive per token, 2026-08-30T00:58Z] — the matched token is inside a column COMMENT in system.metric_log’s DDL, which ClickHouse echoes as an Old:/New: schema comparison on every server start. Two starts × two lines = four hits. It is ClickHouse quoting its own documentation of what a checksum mismatch could mean, not a report that anything happened.

The control was narrowed by one clause — Body NOT LIKE '%CREATE TABLE system.%' — chosen because it is surgical: it excludes the schema echo and nothing else, rather than blinding the control to a whole severity class. corrupt_tight reads 0 in every window of this wave, and the four raw hits are recorded above so the narrowing is auditable rather than a quiet loosening. This is the second control failure in the phase to run in the false-positive direction, and as in 02-04 reading the hits is what separated it from a finding.

The after-windows, read rather than counted

=== …0-0-0 restart window 00:42:43Z–00:52:00Z, queried FROM the surviving replica …0-1-0 ===
pod           anchored_EFC  prescribed_raw  prescribed_noecho  corrupt_raw  corrupt_tight  all_lines
…0-0-0            0 →  1        31 →   9          0 →   7        0 → 4        0 → 0    179,944 → 23,208
…0-1-0            0 →  4         2 →  11          0 →   6        0 → 0        0 → 0    179,973 → 55,689

=== …0-1-0 restart window 00:53:41Z–01:10:00Z, queried FROM the surviving replica …0-0-0 ===
…0-0-0            0 →  0         0 → 173          0 →  97        0 → 0        0 → 0      3,100 → 97,458
…0-1-0            0 →  1         2 →  13          0 →   9        0 → 4        0 → 0      4,421 → 39,132

[VERIFIED: default.otel_logs, 2026-08-30T00:51Z and 01:10Z] — the all_lines control is non-zero on both pods in both windows, which is what proves the query reaches the replacement pod and not only the pod it was reading before.

Every matched class was read:

Class	Where	n	What it is
<Information> HedgedConnectionsFactory: Connection failed at try №1	the surviving replica	64	the peer that was not restarted noticing the one that was
<Warning> DNSResolver: Cannot resolve host (…the deleted peer…)	the surviving replica	33 + 2	same — the headless-service record disappearing with the pod
<Error> …: Code: 210 NetException: Broken pipe … (NETWORK_ERROR)	the surviving replica	4	client sockets broken as ingest reconnected during the disruption
<Error> TCPHandler: Code: 394 … (QUERY_WAS_CANCELLED)	the restarted replica	1 + 1	an in-flight query cancelled by the shutdown that was issued
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log	the restarted replica	2 + 8	ClickHouse’s startup banner, matched on the word errors
Old:/New: CREATE TABLE system.metric_log	the restarted replica	4 + 4	the schema echo above; corrupt_tight excludes it

Not one matched line is attributable to storage. The Error-class total on the restarted pod is a single QUERY_WAS_CANCELLED per restart — an artifact of the delete we issued — and everything else sits on the surviving peer, which is the same peer-noticing signature 02-04 measured on MongoDB: the counts rise on the members that stayed up, not on the member that came back.

Rescue fields across the ClickHouse restarts — read as a delta

Volume	lastAutoSalvagedAt before → after	remountRequestedAt before → after	snapshotsError
…clickhouse-clickhouse-0-0-0	→	→	→
…clickhouse-clickhouse-0-1-0	→	→	→

[VERIFIED: kubectl volumes/engines, 00:42:10Z and 01:07:18Z] — both fields empty before and after on both volumes, so nothing advanced into the task window. Per 02-02’s correction this is read as a delta, not as an estate-wide emptiness test; 19 of 40 volumes carry a months-old remountRequestedAt and these two are not among them.

The replica churn the cordon windows caused, recorded rather than glossed

Cordoning nodes re-triggers Longhorn’s replica-auto-balance and data-locality evaluation, and on a single-replica 140 GiB volume that means a rebuild starts. Three such rebuilds were begun and all three were discarded unfinished once the volume was back on the node that already held its healthy replica:

00:47Z  pvc-d1086c25 (…0-0-0)  extra replica on tpi-alpha-4, healthyAt empty  -> discarded by 00:50Z
01:04Z  pvc-5fa64d0c (…0-1-0)  extra replica on tpi-alpha-1, healthyAt empty  -> discarded by 01:06Z
01:06Z  pvc-d1086c25 (…0-0-0)  extra replica on tpi-beta-3,  healthyAt empty  -> still building at 01:10Z

[VERIFIED: kubectl -n longhorn-system get replicas.longhorn.io, 00:47Z–01:10Z] — in every case the original replica kept its original healthyAt (2026-07-05T21:14:35Z and 2026-07-27T07:37:52Z), so no data was moved, rewritten or re-derived; the rebuilds were additive and were reaped. Both volumes read attached/healthy throughout. The one still in flight at the end of Task 1 is redundant by construction — numberOfReplicas is 1 and the volume is attached on the node that holds the healthy replica — and Longhorn will reap it the same way.

This is the cost of the cordon verb on a single-replica volume, and it belongs in 02-08’s runbook: prefer step 2 immediately after step 1 so the rebuild has the shortest possible window to run.

Gate movement attributable to Task 1

=== post-Task-1 LH-03 gate re-read, 2026-08-30T01:07:30Z ===
(a) engine processes outside v1.12.1 : 12   (was 14)
(a-inverted) control, inside v1.12.1 : 28   (was 26)

[VERIFIED: kubectl, 2026-08-30T01:07:30Z] — two engines moved and both halves moved by exactly two in opposite directions. instance-manager-fc5f937c05ebc017992613b0f02d73ee on tpi-beta-3, whose only engine was …clickhouse-0-1-0, left the inventory on its own once emptied; instance-manager-68fca8eecce9b… on tpi-beta-1 fell 3 → 2 engines. Zero instance-managers were created — the newest object in the inventory is dated 2026-08-24T14:58:29Z, five days before this wave — and none was deleted or patched by us. All 40 volumes read attached + healthy on the v1.12.1 image.

Node scheduling state after Task 1: all 8 nodes Ready, none cordoned, re-read at 01:07:30Z. Every cordon window was closed by the helper’s own uncordon, and the helper traps EXIT/INT/TERM — reopening the home node first so that even an aborted run sends the pod home rather than scattering it across the cluster.

Task 2 — the Keeper drain, strictly one at a time, followers before the leader

The pre-rotation gate, taken before the first delete so the rotation could not start from an already-degraded quorum:

=== 2026-08-30T01:14:03Z ===
keeperclusters.clickhouse.com/cs-clickstack-keeper  READY True  "Cluster is ready"  3/3
  Ready=True reason=ClusterReady since=2026-08-24T16:43:31Z
--- keeper mntr, all three members ---
cs-clickstack-keeper-keeper-0-0 : zk_server_state follower  zk_znode_count 33069
cs-clickstack-keeper-keeper-1-0 : zk_server_state follower  zk_znode_count 33085
cs-clickstack-keeper-keeper-2-0 : zk_server_state LEADER    zk_znode_count 33092  zk_followers 2  zk_synced_followers 2

[VERIFIED: kubectl + printf mntr | nc 127.0.0.1 2181 in each pod, 2026-08-30T01:14:03Z]

The Raft-role verb for this deployment. clickhouse-keeper-client is present in the image but the role is not exposed through it. The four-letter-word interface is, on the client port 2181 (not 9181, which this build does not listen on): printf mntr | timeout 5 nc 127.0.0.1 2181 returns zk_server_state on every member and, on the leader only, zk_followers and zk_synced_followers. The Prometheus endpoint on :9090 carries the same facts as ClickHouseAsyncMetrics_KeeperIsLeader / _KeeperSyncedFollowers and was used to cross-check the first reading. This is the Keeper analogue of 02-04’s rs.status() verb and it is recorded here so 02-08 does not have to rediscover it.

Step	Member	Role read at (from a NON-target witness)	Delete	Bound to	Pod Ready	KeeperCluster ready	Engine IM before → after
1	keeper-0-0	follower, 01:14:40Z (witnesses keeper-1, keeper-2)	01:14:41Z	tpi-alpha-1	+36 s	+138 s, 01:16:59Z	996e594d… v1.11.2 → 6abfc3c9… v1.12.1
2	keeper-1-0	follower, 01:17:23Z (witnesses keeper-0, keeper-2)	01:17:23Z	tpi-alpha-4	+32 s	+139 s, 01:19:42Z	996e594d… v1.11.2 → 3536df4b… v1.12.1
3	keeper-2-0	LEADER, 01:19:51Z (witnesses keeper-0, keeper-1)	01:19:52Z	tpi-alpha-4	+30 s	+137 s, 01:22:09Z	996e594d… v1.11.2 → 3536df4b… v1.12.1

[VERIFIED: kubectl + mntr at every boundary, 2026-08-30T01:14:03Z–01:22:30Z]

The leader went last, against a role re-read 1 second before the delete from the two members that were not the target. It stepped down as expected: at 01:22:09Z keeper-0-0 reads leader with zk_followers 2 / zk_synced_followers 2 and keeper-2-0 has rejoined as a follower.

Both gate halves were taken at every boundary, and the gap between them is the whole reason the CR gate exists. Pod Ready arrived at 30–36 s; keeperclusters.clickhouse.com did not read Cluster is ready until 137–139 s. That is a 101–107 second window in which Kubernetes called the member healthy and the operator did not yet call the quorum whole. 02-04 measured the same class of gap on MongoDB at 15–25 s; on a three-node Raft quorum it is four to seven times longer, and a run gated on pod readiness alone would have issued each next delete squarely inside it. At no point were two Keeper pods simultaneously not Ready — the sibling readiness listing at each of the three boundaries shows the other two 1/1 Running every time.

The PDB cs-clickstack-keeper-keeper (maxUnavailable: 1, disruptionsAllowed: 1) was never the enforcing mechanism, for the reason 02-04 established for valkey: a PDB gates the Eviction API, not kubectl delete pod. The one-at-a-time discipline here is entirely the operator’s.

The shared instance-manager was neither deleted nor patched, and its survival is expected

=== instance-manager-996e594d45afc211165dc4b7bfca856b, 2026-08-30T01:22:30Z ===
image=docker.io/longhornio/longhorn-instance-manager:v1.11.2  node=tpi-beta-4
created=2026-06-06T17:38:36Z   metadata.generation=1
engines=pvc-bc28579b-0140-4a1b-a418-3e8bcecda79d-e-0   ->   postgres/main-20

[VERIFIED: kubectl -n longhorn-system get instancemanagers.longhorn.io, 2026-08-30T01:22:30Z]

It went from 4 engines to 1. metadata.generation is still 1 and creationTimestamp is still 2026-06-06T17:38:36Z, so the object was neither replaced nor spec-modified at any point. Its one remaining engine is postgres/main-20, which 02-06 moves — so this instance-manager is expected to survive this wave and to be reaped by Longhorn during the next one, exactly as the plan predicted. Its continued presence in the third verify command’s output is the designed outcome, not a failure, and a later reader should not treat it as one.

This is the sharpest D-05 case in the estate and it was never touched. A single kubectl delete pod instance-manager-996e594d… at the start of this wave would have destroyed all three Keeper engines and a Postgres instance in one action — losing Raft quorum, taking ClickHouse down with it, and blinding the log half of every remaining verdict in this phase. The only verb used was kubectl delete pod against the workload, plus node cordons on tpi-beta-4, which carry no ArgoCD ownership.

Each member’s own recovery output — the workload half of the verdict

keeper-0  01:15:09.6  <Information> Application: Starting ClickHouse Keeper 26.7.5.10 (revision: 54516)
          01:15:10.8  KeeperServer: No config in log store, will use config from snapshot with log index 276156237
          01:15:14.7  RaftInstance: deny, req term 16, my term 16, req log idx 305958158, my log idx 305955825, receiving snapshot FALSE
          01:15:15.0  <Information> Application: Ready for connections.
keeper-1  01:17:50.4  Starting ClickHouse Keeper 26.7.5.10
          01:17:52.0  No config in log store, will use config from snapshot with log index 276156237
          01:17:52.2  req log idx 305967744, my log idx 305965408, receiving snapshot FALSE
          01:17:52.5  Ready for connections.
keeper-2  01:20:17.9  Starting ClickHouse Keeper 26.7.5.10
          01:20:20.6  KeeperStateMachine: Preprocessing 2 uncommitted log entries (305975303 - 305975305)
          01:20:20.6  KeeperStateMachine: Preprocessing done
          01:20:21.1  Ready for connections.

[VERIFIED: default.otel_logs via clickhouse-client, 2026-08-30T01:27Z]

This is the verdict input D-01/D-09 reserve for the workload itself, and it says more than a readiness probe can. Each member read its snapshot and log back off the reattached Longhorn volume, found its own log index only ~2,300 entries behind the leader, and caught up without requesting a snapshot transfer — receiving snapshot FALSE on both followers. keeper-2, the former leader, replayed 2 uncommitted entries from its own on-disk log and was serving 4 seconds later. A volume whose contents had not survived the detach would have forced a full snapshot transfer at best and refused to start at worst. No data-integrity check of our own was written, and none was needed.

The Keeper after-window, read rather than counted

=== 01:14:41Z–01:26:00Z, all three members, baseline from the 00:11:00Z–00:41:00Z wave window ===
pod        anchored_EFC  prescribed  corrupt_tight  anchored_Warning   all_lines
keeper-0-0     0 →  1      0 → 13        0 → 0          0 → 59          10 → 279
keeper-1-0     0 →  3      0 →  6        0 → 0          0 → 21           9 → 246
keeper-2-0     0 →  7      0 → 44        0 → 0          0 → 87          18 → 287

[VERIFIED: default.otel_logs, 2026-08-30T01:26Z] — the all_lines control rises on every member, which proves the query reaches all three replacement pods.

All 11 Error-class lines are one class, RaftInstance, and every one lands within 1.1 s of a delete this wave issued:

Delete instant	Error lines	Emitted by	Text
01:14:41Z (keeper-0)	4, at 01:14:42.225–.234	keeper-2 ×3, keeper-1 ×1	failed to read rpc header from socket ::ffff:10.245.0.209:… End of file
01:17:23Z (keeper-1)	4, at 01:17:24.049–.229	keeper-2 ×3, keeper-1 ×1	same, socket ::ffff:10.245.0.142:…; plus failed to accept a rpc connection due to error 125, Operation canceled
01:19:52Z (keeper-2)	3, at 01:19:52.576–.755	keeper-1, keeper-0, keeper-2	same, socket ::ffff:10.245.0.70:…; plus one Operation canceled

[VERIFIED: default.otel_logs, per-line timestamps, 2026-08-30T01:27Z]

The peer IPs are decisive: 10.245.0.209, 10.245.0.142 and 10.245.0.70 are the old pod IPs of keeper-0, keeper-1 and keeper-2 respectively, and each burst names exactly the member being deleted at that instant. These are the surviving members watching a peer’s TCP session close, plus the departing member closing its own listener — the Keeper dialect of the same peer-noticing signature 02-04 measured on MongoDB and 02-02 on NATS. Outside those three sub-second bursts the Error count over the whole 11-minute window is zero, and corrupt_tight is 0 → 0 on all three.

The 59/21/87 <Warning> lines are the chronic RaftInstance: … took long time (NNN ms) / skipped sending msg … class that the 24-hour positive control already showed running at 5/10/40 per day before this wave touched anything.

Rescue fields across the Keeper restarts — read as a delta

Volume	lastAutoSalvagedAt before → after	remountRequestedAt before → after	snapshotsError
…keeper-keeper-0-0	→	2026-07-03T18:05:37Z → 2026-07-03T18:05:37Z (byte-identical)	→
…keeper-keeper-1-0	→	→	→
…keeper-keeper-2-0	→	→	→

[VERIFIED: kubectl volumes/engines, 01:14:04Z–01:22:30Z] — keeper-0-0 is one of the 19 volumes carrying a months-old remountRequestedAt, and it did not advance into this task’s window. Per 02-02’s correction this field is read as a delta, never as an emptiness test; had it been read as an absolute the wave would have stopped on a value predating the phase by eight weeks.

The log half was NOT n/a — the instrument never went dark during its own drain

This wave was planned on the expectation that the log column might have to read n/a, because ClickHouse and Keeper are the query path the log half of every LH-01 verdict is read through. It did not. Cluster-wide default.otel_logs ingest was measured per minute across the entire wave:

=== rows per minute, 00:40Z–01:26Z, 46 consecutive minutes, cluster-wide ===
46 of 46 minutes present, no gap.  min 11,868  max 31,376
lowest minute 00:57Z (11,868) — during the …0-1-0 Pending window, and still 12k rows

[VERIFIED: default.otel_logs GROUP BY toStartOfMinute, 2026-08-30T01:26Z]

Both restart phases kept one replica serving throughout — …0-1-0 answered while …0-0-0 restarted, …0-0-0 answered while …0-1-0 restarted, and both answered throughout the Keeper rotation because quorum was never below 2 of 3. Every log cell in this wave’s five rows is therefore a real reading, not an n/a. The distinction matters in both directions: the record does not claim an n/a it did not need, and it does not record a pass on an input it did not read. The n/a-with-reason discipline remains the rule for any future wave whose instrument does go dark.

Final confirmation that the instrument is handed to 02-06 working:

SELECT count(), max(Timestamp) FROM default.otel_logs WHERE Timestamp > now() - INTERVAL 5 MINUTE
95209   2026-08-30 01:26:47.727549944

[VERIFIED: clickhouse-client on …0-1-0, 2026-08-30T01:27Z] — 95,209 rows in the trailing five minutes with the newest row seconds old.

Where the Keeper members now live, recorded because it changed

All three Keeper pods were on tpi-beta-4 before this wave; the cordon verb necessarily moved each one off it. They now sit on tpi-alpha-1 (keeper-0) and tpi-alpha-4 (keeper-1, keeper-2). That is an improvement on a three-of-three single-node concentration but it is still two of three on one node, and it is a side effect of the drain verb rather than a placement decision — nothing in this phase pins them, and the operator has not asked for an anti-affinity rule. Recorded so a later reader neither assumes it was designed nor mistakes it for drift.

Gate movement attributable to Task 2

=== post-Task-2 LH-03 gate re-read, 2026-08-30T01:22:46Z ===
(a) engine processes outside v1.12.1 : 9    (was 12)
(a-inverted) control, inside v1.12.1 : 31   (was 28)

[VERIFIED: kubectl, 2026-08-30T01:22:46Z] — three engines moved and both halves moved by exactly three in opposite directions. Across the whole wave the gate went 14 → 9 with its control 26 → 31, both by exactly five: the arithmetic a five-volume drain should produce, and itself a check that the two queries still partition the same set. All 40 volumes read attached + healthy on the v1.12.1 image, and (c) returns 0 rows.

Instance-manager mutation control for the wave: exactly one instance-manager left the inventory (fc5f937c…, emptied by Task 1 and reaped by Longhorn), zero were created — the newest object in the inventory is dated 2026-08-24T14:58:29Z, five days before this wave — and no survivor changed image, node or generation.

Node scheduling state after Task 2: all 8 nodes Ready, none cordoned, re-read at 01:22:46Z.

Wave 02-06 — start-of-wave re-read and the tier-3 postgres work list

Tier 3’s first half is the CNPG Cluster main — three instances, one of them the write primary, sitting upstream of keycloak-pooler-rw and therefore of the OIDC path this operator uses to reach ArgoCD and Vault. CONTEXT’s tier-3 list names main-16 and main-17; main-20 is stale too and is in this wave, as RESEARCH §3.3 already recorded.

=== start-of-wave gate re-read: 2026-08-30T01:36:47Z ===
--- (a) engine processes still outside v1.12.1 ---
9
--- (a-inverted) CONTROL: engine processes INSIDE v1.12.1 IMs ---
31
--- (b) stale instance-managers ---
instance-manager-1b8c1c4d55fc05f3f1a9af947ff100a4 v1.10.1 tpi-beta-3 engines=1 replicas=0
instance-manager-27af93a4524a66b6e9b1bc3a6e0a6920 v1.11.2 tpi-alpha-2 engines=2 replicas=0
instance-manager-68fca8eecce9b1d85ba3f5986b1faae4 v1.11.2 tpi-beta-1 engines=2 replicas=0
instance-manager-996e594d45afc211165dc4b7bfca856b v1.11.2 tpi-beta-4 engines=1 replicas=0
instance-manager-9e303193f6af67a4962641022b0ed2ac v1.10.1 tpi-beta-2 engines=1 replicas=0
instance-manager-b48dac91b45fa0a6f4869005f61020b4 v1.11.2 tpi-alpha-4 engines=2 replicas=0
--- (c) volumes not attached+healthy on v1.12.1 image ---
--- (c) row count ---
0
--- total volumes ---
40

[VERIFIED: kubectl, 2026-08-30T01:36:47Z] — the inverted-predicate control returned 31, strictly greater than zero, so (a)’s reading of 9 is a measurement and not a broken query.

Reconciliation against 02-05

Reading	UTC	(a) outside v1.12.1	control, inside v1.12.1	stale IM rows
02-05 close of wave	2026-08-30T01:22:30Z	9	31	7
02-06 start of wave	2026-08-30T01:36:47Z	9	31	6

Engine drift since 02-05: zero, across 14 minutes. The one difference is in the object inventory, not the engine count: instance-manager-fc5f937c05ebc017992613b0f02d73ee — emptied by 02-05’s first ClickHouse drain — has left the inventory, reaped by Longhorn between the two readings. That is the expected tail of 02-05’s own work, not new drift, and it moves no engine.

instance-manager-996e594d45afc211165dc4b7bfca856b is still present with engines=1, exactly as 02-05 predicted. Its lone survivor is postgres/main-20. Its disappearance later in this wave is the LH-03 reaping signal, not something to force. [VERIFIED: 2026-08-30T01:36:47Z]

Volume → instance-manager map for tier 3, re-derived live

=== 2026-08-30T01:36:59Z — 9 engines outside v1.12.1, all of them tier 3 ===
postgres/main-16    node=tpi-beta-3   attached/healthy/v1.12.1  IM=instance-manager-1b8c1c4d… [v1.10.1]  replicas=3  3.44 GiB
postgres/main-17    node=tpi-beta-2   attached/healthy/v1.12.1  IM=instance-manager-9e303193… [v1.10.1]  replicas=3  3.24 GiB
postgres/main-20    node=tpi-beta-4   attached/healthy/v1.12.1  IM=instance-manager-996e594d… [v1.11.2]  replicas=3  3.43 GiB
vault/data-vault-0  node=tpi-alpha-4  …  IM=instance-manager-b48dac91… [v1.11.2]  replicas=3
vault/audit-vault-0 node=tpi-alpha-4  …  IM=instance-manager-b48dac91… [v1.11.2]  replicas=3
vault/data-vault-1  node=tpi-beta-1   …  IM=instance-manager-68fca8ee… [v1.11.2]  replicas=2
vault/audit-vault-1 node=tpi-beta-1   …  IM=instance-manager-68fca8ee… [v1.11.2]  replicas=2
vault/data-vault-2  node=tpi-alpha-2  …  IM=instance-manager-27af93a4… [v1.11.2]  replicas=3
vault/audit-vault-2 node=tpi-alpha-2  …  IM=instance-manager-27af93a4… [v1.11.2]  replicas=3

[VERIFIED: kubectl volumes/engines/instancemanagers, 2026-08-30T01:36:59Z] — 3 postgres (this wave) + 6 vault (02-07) = the 9 the gate reports. All three postgres volumes carry numberOfReplicas: 3, so 02-05’s single-replica hazard does not apply here and the plain cordon verb is safe.

Live CNPG cluster state before anything was touched

2026-08-30T01:37:10Z
instances=3  currentPrimary=main-16  targetPrimary=main-16  readyInstances=3
phase="Cluster in healthy state"  timelineID=17  primaryUpdateStrategy=unsupervised
image=ghcr.io/cloudnative-pg/postgresql:18.1-standard-bullseye
healthy=[main-16, main-17, main-20]

main-16  uid=7da421db-…  node=tpi-beta-3  role=primary  Ready=True  restarts=2
main-17  uid=2862eed8-…  node=tpi-beta-2  role=replica  Ready=True  restarts=1
main-20  uid=bee9a8ba-…  node=tpi-beta-4  role=replica  Ready=True  restarts=0

poolers: keycloak-pooler-rw type=rw instances=2 ready=2
PDB:     main minAvailable=1 allowed=1 current=2/1 | main-primary minAvailable=1 allowed=0 current=1/1

[VERIFIED: kubectl -n postgres get cluster/pods/poolers/pdb, 2026-08-30T01:37:10Z] — roles match RESEARCH §4.3’s reading, taken live rather than inherited.

The backup window was checked, not assumed

argocd/app-configs/cnpg/backup.yaml:59 reads schedule: "0 0 */4 * * *" — six fields, seconds-leading, so it fires on the hour every four hours. The live object agrees and states its own boundaries, which is better evidence than parsing the expression:

main-backup-4h  schedule=0 0 */4 * * *  lastScheduleTime=2026-08-30T00:00:00Z  nextScheduleTime=2026-08-30T04:00:00Z

[VERIFIED: kubectl -n postgres get scheduledbackups, 2026-08-30T01:36:40Z] — the wave started at 01:36:47Z, 96 minutes after the last backup and 144 minutes before the next. Comfortably outside the window in both directions, and it finished long before 04:00Z.

The precondition was verified, not assumed

The plan’s precondition requires an ArgoCD and Vault session held before the wave, because Task 2’s switchover blips keycloak-pooler-rw and Keycloak is the OIDC provider for both. Both were checked read-only before the first delete:

Session	Evidence	Held since / until
ArgoCD	argocd account get-user-info → Logged In: true, sean@fuzzymagic.com, issuer https://argocd.fzymgc.house/api/dex	current
Vault	vault token lookup → display_name: oidc-…, meta.role: admin	issued 2026-08-29T15:56:42Z, expires 2026-08-30T09:56:42Z

[VERIFIED: 2026-08-30T01:36:30Z] — the Vault token was issued nine and a half hours before the wave and has eight hours left, so it does not depend on a login through the endpoint this wave disturbs.

The natural experiment that closed OQ-2 empirically, 30 minutes before the wave began

The plan closed OQ-2 from CloudNativePG’s 1.30 plugin documentation: the operator performs an in-place restart for primaries and deletes and recreates Pods for replicas, so a switchover demotes PostgreSQL without terminating the Pod and the Longhorn volume never detaches. That reading turned out to have been tested for us, unaided, by the operator itself during 02-05’s wave:

01:04:59Z  Cluster/main  SwitchingOver: Current primary is running on unschedulable node
                         tpi-beta-3, switching over from main-16 to main-17
01:05:23Z  Pod/main-16   Started: Container started        <- container restartCount 1 -> 2
01:06:39Z  Cluster/main  SwitchingOver: Current primary is running on unschedulable node
                         tpi-beta-2, switching over from main-17 to main-16
01:06:45Z  Pod/main-17   Started: Container started        <- container restartCount 0 -> 1

[VERIFIED: kubectl -n postgres get events, 2026-08-30T01:37:22Z]

Two full switchovers — out and back — and afterwards:

	main-16	main-17
Pod creationTimestamp	2026-06-16T15:12:31Z, unchanged	2026-06-16T15:07:01Z, unchanged
Pod uid	7da421db-…, unchanged	2862eed8-…, unchanged
container restartCount	1 → 2	0 → 1
engine instanceManagerName	instance-manager-1b8c1c4d… v1.10.1, unmoved	instance-manager-9e303193… v1.10.1, unmoved

[VERIFIED: kubectl -n postgres get pods -o json .metadata.uid/.creationTimestamp/.status.containerStatuses[0], 2026-08-30T01:37:14Z]

The Pods survived, the PVCs were never unmounted, and both engines stayed in their pre-v1.12.1 instance-managers across two switchovers. OQ-2’s answer is therefore not merely documented, it is measured on these exact volumes: a CNPG switchover moves the primary role and restarts the PostgreSQL process; it does not move the Longhorn engine. A wave that had treated kubectl cnpg promote as the detach would have recorded LH-03 as reached while main-16’s engine sat exactly where it started.

The cause of the two switchovers is 02-05’s cordon windows, and that is a side effect 02-05 did not record. The two-step move-away-then-home drain for …clickhouse-0-1-0 cordons tpi-beta-3 (step 1) and then everything-but-tpi-beta-3 including tpi-beta-2 (step 2). CNPG’s own reconciler treats “primary on an unschedulable node” as a reason to switch over, and did so both times, unprompted. Recorded here so a later reader attributes main-16’s and main-17’s restart counts correctly, and so 02-07 knows that cordoning a node that hosts the CNPG primary will move the write endpoint whether or not you asked it to. All eight nodes read unschedulable=false at 2026-08-30T01:37:33Z before this wave began, so no cordon residue was carried in.

Correction: the prescribed matcher is a line counter on postgres, because error is a schema field name

This is the fifth direction the prescribed error-class matcher has failed across five waves — after Mealie’s false positive, NATS’ false negative, MongoDB’s over-sensitivity and ClickHouse’s self-echo — and it is the most complete failure of the five.

CNPG wraps every PostgreSQL log line in a JSON envelope whose record object carries the field error_severity. The token error is therefore present in the schema of essentially every postgres log line, regardless of what the line says:

24-hour window, namespace postgres
positionCaseInsensitive(Body,'error') > 0                        2,669
Body LIKE '%error_severity%'                                     2,620
'error' present but NOT as the error_severity field name            49

[VERIFIED: default.otel_logs, 2026-08-30T01:38Z] — 98.2% of the prescribed matcher’s hits are the field name. Its per-pod 24-hour reading of 757 / 982 / 908 on main-16 / main-17 / main-20 is, to within a few percent, a count of how many lines each pod emitted.

Two anchored matchers replace it, each keyed on a severity token in its own structural position:

-- (i) the CNPG operator envelope's own level
Body LIKE '%"level":"error"%' OR Body LIKE '%"level":"fatal"%'
-- (ii) PostgreSQL's own severity vocabulary, inside the record
Body LIKE '%"error_severity":"ERROR"%' OR '%"error_severity":"FATAL"%' OR '%"error_severity":"PANIC"%'

Positive control, established before any zero was believed — both matchers reach the real format, so neither zero is a false negative:

pod	prescribed 24 h	cnpg-level 24 h	pg-severity 24 h	all lines 24 h
main-16	757	5	71	1,799
main-17	982	1	75	1,509
main-20	908	0	7	1,241

[VERIFIED: default.otel_logs, 2026-08-30T01:38Z]

The third control, available in advance for once: an unaided switchover of the same three pods

02-04 established that a steady-state baseline cannot contain a startup, so a delta must be checked against the same body classes on a prior incarnation. This wave gets that for free: the operator’s own 01:04:59Z–01:06:55Z switchover restarted the same PostgreSQL processes without any action of ours, and every one of the 153 pg-severity hits in the whole 24-hour window falls inside those two minutes.

class	severity	main-16	main-17	main-20	window
terminating connection due to administrator command	FATAL	50	64	—	01:05:16–01:06:41
the database system is starting up	FATAL	19	9	—	01:05:34–01:07:04
the database system is shutting down	FATAL	1	1	—	01:05:17 / 01:06:43
could not send end-of-streaming message to primary: SSL connection has been closed unexpectedly	FATAL	1	1	2	01:05:22–01:06:43
streaming replication receiver	FATAL	—	—	5	01:05:24–01:06:53
failed to run wal-archive command / switchover in progress, refusing archiving	cnpg error	3	1	—	01:05:28–01:06:55
Retention policy enforcement failed (chronic, daily, object-store conflict)	cnpg error	2	—	—	08:02Z, 00:02Z

[VERIFIED: default.otel_logs grouped by severity and message, 2026-08-30T01:38Z] — outside those two windows and the chronic daily retention line, both anchored matchers read zero across 24 hours. So the restart-attributable class profile was known before this wave issued a single delete, and any new class in an after-window would stand out immediately.

Correction: the corruption negative control fires on pg_controldata’s own field names

02-04 tightened the corruption control to corrupt|inconsistenc|data loss|checksum. On postgres it returns non-zero on every restart, and by the acceptance criterion that would stop the wave. Read rather than counted, every hit is the same thing:

offset 2322, both hits, identical:
  …ssing:              by value\nData page checksum version:           0\nDefault char data s…
  logger = "pg_controldata", level = "info"

[VERIFIED: position(Body,'checksum') and the surrounding 90 characters, 2026-08-30T01:43Z] — CNPG dumps the full pg_controldata output at both shutdown and startup, and that output contains the field name Data page checksum version. The control was matching a schema label, exactly as the prescribed matcher does with error_severity.

One clause narrows it — Body NOT LIKE '%"logger":"pg_controldata"%' — chosen because it excludes the control-data dump and nothing else, rather than blinding the control to a token class:

pod	corrupt_raw 24 h	corrupt_tight 24 h
main-16	4	0
main-17	8	0
main-20	0	0

[VERIFIED: 2026-08-30T01:43Z] — the raw hits and the matched token’s exact offset are recorded above so the narrowing is auditable rather than a quiet loosening.

Baseline windows, absolute UTC rather than relative

Relative now() - INTERVAL 30 MINUTE windows would have straddled the 01:05Z switchover and silently inflated the baseline. Fixed windows instead:

window	UTC	main-16 cnpg/pgsev/all	main-17 cnpg/pgsev/all	main-20 cnpg/pgsev/all
clean, pre-switchover	00:30:00–01:00:00	0 / 0 / 30	0 / 0 / 24	0 / 0 / 24
the unaided switchover (third control)	01:00:00–01:10:00	3 / 71 / 296	1 / 75 / 321	0 / 7 / 95
immediate pre-wave baseline	01:10:00–01:39:00	0 / 0 / 30	0 / 0 / 23	0 / 0 / 24

[VERIFIED: default.otel_logs, 2026-08-30T01:39Z] — the wave’s baseline on both anchored matchers is zero for all three instances, with a non-zero all-lines figure proving the query reaches each pod.

Rescue-field baseline, read as a delta

main-16  lastAutoSalvagedAt=""  remountRequestedAt="2026-05-12T01:25:34Z"  lastDegradedAt=""
main-17  lastAutoSalvagedAt=""  remountRequestedAt=""                      lastDegradedAt=""
main-20  lastAutoSalvagedAt=""  remountRequestedAt=""                      lastDegradedAt=""

[VERIFIED: 2026-08-30T01:39Z] — main-16 carries a remountRequestedAt predating this phase by three and a half months, consistent with 02-02’s correction that the field is not empty estate-wide. The criterion is that it must not advance into this wave’s window.

Task 1 — the two CNPG replicas, one at a time, primary untouched

Every gate below keys on .metadata.uid, never on the pod name: CNPG recreates the instance Pod under the same name, so a name-keyed readiness poll reads the old Terminating pod — still Ready=True — and passes in zero seconds (02-04’s correction).

#	instance	role at delete	delete	old uid → new uid	node	Ready	Cluster CR ready	engine IM
1a	main-17	replica	01:40:00Z	2862eed8… → 9da43de8… (3 s)	tpi-beta-2 → tpi-beta-2	True at 01:40:30Z, +30 s	main-16/main-16/3/healthy at 01:40:32Z, +32 s	9e303193… v1.10.1 — UNMOVED
1b	main-17	replica	01:41:20Z	9da43de8… → 757e57e6… (3 s)	tpi-beta-2 → tpi-alpha-4	True at 01:42:07Z, +47 s	main-16/main-16/3/healthy at 01:42:09Z, +49 s	9e303193… v1.10.1 → 3536df4b… v1.12.1
2	main-20	replica	01:43:57Z	bee9a8ba… → 671be9cb… (4 s)	tpi-beta-4 → tpi-beta-2	True at 01:44:45Z, +48 s	main-16/main-16/3/healthy at 01:44:45Z, +48 s	996e594d… v1.11.2 → 5f4ef201… v1.12.1

[VERIFIED: uid-gated drain log, 2026-08-30T01:40:00Z–01:44:45Z]

Step 1a is 02-02’s same-node non-migration, reproduced on CNPG. The plain delete returned main-17 to tpi-beta-2 in three seconds and the engine did not move — the CSI VolumeAttachment is keyed per node and a same-node replacement never releases it. 1b applied the cordon verb and it moved on the first attempt. main-20 was therefore driven cordon-first (02-04’s ordering), which is why it needed one delete rather than two.

Cordoning is safe here in a way it was not in 02-05: all three volumes carry three replicas, so there is no single-replica best-effort rebuild to force. It is not safe on the primary — see the natural experiment above, where a cordon on the primary’s node made CNPG switch over on its own. Only replica nodes were cordoned, and the source node was reopened the instant placement was decided (tpi-beta-2 uncordoned 3 s after the delete, tpi-beta-4 4 s after), with an unconditional uncordon on every exit path.

No two postgres pods were ever simultaneously not Ready. The sibling listing taken immediately before each delete reads all three instances Ready=True, and the Cluster CR returned readyInstances: 3 before the next delete was issued.

The workload half — read from the instance’s own recovery output and witnessed from the primary

main-17, 2026-08-30T01:41:59Z–01:42:07Z
  [LOG] database system was shut down in recovery at 2026-08-30 01:41:20 UTC
  [LOG] restored log file "00000011.history" from archive
  [LOG] entering standby mode
  [LOG] redo starts at 1CD/19002910
  [LOG] consistent recovery state reached at 1CD/190B1CC0
  [LOG] database system is ready to accept read-only connections
  [LOG] unexpected pageaddr 1CC/F7000000 in WAL segment 00000011000001CD0000001A …   <- end-of-archive boundary
  [LOG] started streaming WAL from primary at 1CD/1A000000 on timeline 17

main-20, 2026-08-30T01:44:34Z–01:44:37Z
  [LOG] database system was shut down in recovery at 2026-08-30 01:43:57 UTC
  [LOG] entering standby mode
  [LOG] redo starts at 1CD/1A0027C8
  [LOG] consistent recovery state reached at 1CD/1A0A2408
  [LOG] invalid resource manager ID 114 at 1CD/1A0A2408                              <- end-of-valid-WAL marker
  [LOG] database system is ready to accept read-only connections
  [LOG] started streaming WAL from primary at 1CD/1A000000 on timeline 17

[VERIFIED: kubectl -n postgres logs main-{17,20} -c postgres, 2026-08-30T01:42Z / 01:45Z] — both reached consistent recovery state and then started streaming, on timeline 17, the cluster’s current timeline. The two LOG-severity oddities are the ordinary end-of-recovered-WAL boundary markers PostgreSQL prints when it stops replaying an archived segment and switches to streaming; neither is an ERROR, FATAL or PANIC, and neither appears in the anchored counts.

Membership was read from a witness that is not the target — the primary’s own view:

2026-08-30T01:45Z   kubectl -n postgres exec main-16 -- psql -Atc 'SELECT … FROM pg_stat_replication'
main-17 | streaming | async | 00:00:00.004344
main-20 | streaming | async | 00:00:00.004615

[VERIFIED: from the PRIMARY, after both restarts] — both replicas back in streaming with replay lag of about 4 milliseconds. Pod Ready is not cluster membership; this is.

The filesystem half

kubelet /stats/summary, per PVC
tpi-alpha-4  main-17  main-17  2.20 GiB / 48.91 GiB  at 2026-08-30T01:43:40Z   (new node)
tpi-beta-2   main-20  main-20  2.22 GiB / 48.91 GiB  at 2026-08-30T01:45:45Z   (new node)
tpi-beta-3   main-16  main-16  2.22 GiB / 48.91 GiB  at 2026-08-30T01:45:14Z

in-pod /proc/mounts, explicit rw assertion
main-17  /dev/longhorn/pvc-b8815886-63b8-4db6-a72a-434ae667076a  ext4  rw,relatime
main-20  /dev/longhorn/pvc-bc28579b-0140-4a1b-a418-3e8bcecda79d  ext4  rw,relatime
main-16  /dev/longhorn/pvc-780c5703-1240-477c-a75e-62d48480c942  ext4  rw,relatime

[VERIFIED: 2026-08-30T01:41:57Z and 2026-08-30T01:45:45Z]

The after-windows, read rather than counted

window	pod	cnpg-level	pg-severity	corrupt_tight	prescribed	all lines
main-17 01:41:20Z → 01:45:35Z	main-17	0 → 1	0 → 18	0 → 0	0 → 38	0 → 157
main-20 01:43:57Z → 01:45:35Z	main-20	0 → 3	0 → 9	0 → 0	0 → 34	0 → 146
same window, untouched primary	main-16	0 → 0	0 → 0	0 → 0	0 → 2	5

Every hit read, not counted:

• main-17 cnpg-level, 1 line. 01:41:21.389Z "While terminating server" — 1.4 s after our own delete. Self-caused shutdown.
• main-20 cnpg-level, 3 lines. All at 01:43:58.09–.39Z, ~1 s after our own delete: "Error while getting cluster CA Client secret" and "Reconciler error" from the in-pod instance-manager controller losing its API client as the container tears down, then "While terminating server". Self-caused shutdown, all three.
• pg-severity, 27 lines across both. Three classes only: the database system is starting up (16 + 8), terminating walreceiver process due to administrator command (1 + 1), the database system is not yet accepting connections (1). All three appear in the 01:05Z unaided-switchover control above. Zero new body classes.
• corrupt_tight zero in both windows, with corrupt_raw non-zero and every raw hit located at offset 2322 in a pg_controldata dump.
• all-lines control non-zero in both windows, proving the query reaches the replacement pod and a zero would have been a real zero.

[VERIFIED: default.otel_logs, 2026-08-30T01:45:35Z]

Rescue fields across the two restarts — read as a delta

volume	lastAutoSalvagedAt before → after	remountRequestedAt before → after	lastDegradedAt before → after
main-17	"" → ""	"" → ""	"" → ""
main-20	"" → ""	"" → ""	"" → ""
main-16 (untouched)	"" → ""	2026-05-12T01:25:34Z → byte-identical	"" → ""

[VERIFIED: 2026-08-30T01:39Z and 2026-08-30T01:45:53Z] — nothing advanced into this task’s window.

Replica churn the cordon caused, recorded with its resolution

main-17’s replacement landed on tpi-alpha-4, where the volume had no replica, so best-effort data locality scheduled one. It appeared with healthyAt empty — the shape this repository’s prior-incident knowledge flags as dangerous when it is also failedAt: "", because such a replica holds its slot forever. Here it was a rebuild in progress, and it is recorded because the two states are indistinguishable at a single reading:

01:43:45Z  pvc-b8815886  4 replicas: tpi-beta-2, tpi-alpha-2, tpi-beta-1, tpi-alpha-4 healthyAt=""
01:45:53Z  pvc-b8815886  3 replicas: tpi-alpha-2, tpi-beta-1, tpi-alpha-4 healthyAt="2026-08-30T01:43:52Z"

[VERIFIED: kubectl -n longhorn-system get replicas.longhorn.io — the four-replica reading taken immediately before the main-20 delete, the three-replica reading at wave close] — the rebuild completed in about 2 minutes for 2.2 GiB and the excess tpi-beta-2 replica was dropped. The volume never read degraded. main-20 caused no churn at all: cordoning tpi-beta-4 placed it on tpi-beta-2, where a replica from 2026-08-23T17:28:24Z already lived, so nothing rebuilt. At task close all three volumes carry exactly three replicas, every one with a non-empty healthyAt and an empty failedAt.

The instance-manager reaping, observed and not forced

Two pre-v1.12.1 instance-managers left the inventory during this task, each within seconds of its last engine departing. Neither was deleted or patched:

instance-manager	version	node	held	outcome
instance-manager-9e303193…	v1.10.1	tpi-beta-2	postgres/main-17 only	NotFound at 01:42:20Z
instance-manager-996e594d…	v1.11.2	tpi-beta-4	postgres/main-20 only — 02-05’s lone survivor	NotFound at 01:44:55Z

[VERIFIED: kubectl -n longhorn-system get instancemanagers.longhorn.io <name>, both returning Error from server (NotFound), 2026-08-30T01:45:00Z]

This answers 02-05’s handoff. instance-manager-996e594d… went 4 engines → 1 during 02-05 without being touched, and its last engine was postgres/main-20. Moving that engine here emptied it and Longhorn reaped it on its own, exactly as D-05 requires. The instance-manager inventory totals 12 objects with the newest still dated 2026-08-24T14:58:29Z — zero created by this task, and no instancemanagers.longhorn.io object deleted or patched at any point in it.

Gate movement attributable to Task 1

Reading	UTC	outside v1.12.1	control, inside	stale IM rows
start of wave	01:36:47Z	9	31	6
after both replicas	01:44:55Z	7	33	4

[VERIFIED: kubectl, 2026-08-30T01:44:55Z] — down by exactly two, up by exactly two, with the two emptied instance-managers gone. The four remaining stale rows are 1b8c1c4d… holding main-16 alone (Task 2) and three vault instance-managers holding two engines each (02-07).

Task 2 — the switchover, then the gated detach of the demoted instance

The primary’s volume is the one case in this phase where the restart verb is not a pod delete. CloudNativePG’s own guidance is to move the primary role with a switchover before deleting a primary instance, because a raw delete races the operator’s failover. The switchover is the first half of the move; it is not the whole of it.

Pre-switchover reading, taken live:

2026-08-30T01:50:23Z
cluster:  main-16/main-16/3/Cluster in healthy state  timelineID=17
main-16   uid=7da421db-…  created=2026-06-16T15:12:31Z  node=tpi-beta-3   role=primary  restarts=2
main-17   uid=757e57e6-…  created=2026-08-30T01:41:22Z  node=tpi-alpha-4  role=replica  restarts=0
main-20   uid=671be9cb-…  created=2026-08-30T01:44:00Z  node=tpi-beta-2   role=replica  restarts=0
engine pvc-780c5703… : running in instance-manager-1b8c1c4d… [v1.10.1]
poolers:  keycloak-pooler-rw 2/2 Ready
OIDC:     https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration -> 200 in 0.061 s
baseline 01:45:35Z->01:50:23Z, all three instances: cnpg-level 0, pg-severity 0, corrupt_tight 0

[VERIFIED: kubectl + curl, 2026-08-30T01:50:23Z]

main-20 was chosen as the promotion target over main-17 because Task 1 landed it on a node that already held one of its replicas, so its storage path had had no rebuild activity at all, while main-17’s had rebuilt a replica minutes earlier. Both were eligible: the plan requires only that the target already sit in a v1.12.1 instance-manager, and both did.

The switchover, and its confirmation on the live Cluster object:

[01:50:53Z] kubectl cnpg promote -n postgres main main-20
            "Node main-20 in cluster main will be promoted"
[01:50:53Z] main-16/main-20/3/Switchover in progress
[01:50:59Z] main-16/main-20/2/Switchover in progress
[01:51:02Z] main-20/main-20/2/Switchover in progress
[01:51:18Z] main-20/main-20/3/Cluster in healthy state     <- CONFIRMED, 25 s

[VERIFIED: polled every 3 s against clusters.postgresql.cnpg.io/main, 01:50:53Z–01:51:18Z] — currentPrimary == targetPrimary == main-20 and the phase reads the healthy-cluster value. readyInstances dipped to 2 for nine seconds and returned to 3. The plugin is 1.30.0 against operator 1.29.1, and it needs -n postgres explicitly — without the namespace flag it looks in default and exits with cluster main not found.

The measurement that OQ-2 turns on, taken after the switchover and before any delete

2026-08-30T01:51:30Z — demoted instance main-16
  uid                     7da421db-8d12-4202-8daf-c9561c1afd3b   UNCHANGED
  creationTimestamp       2026-06-16T15:12:31Z                   UNCHANGED
  container restartCount  2 -> 3                                 restarted IN PLACE
  container startedAt     2026-08-30T01:50:57Z
  role                    primary -> replica
  Ready                   True
  engine instanceManager  instance-manager-1b8c1c4d…             UNMOVED
  that IM's image         longhornio/longhorn-instance-manager:v1.10.1

[VERIFIED: kubectl -n postgres get pods -o json and kubectl -n longhorn-system get engines.longhorn.io, 2026-08-30T01:51:30Z]

The Pod was not recreated and the engine did not move. The switchover demoted PostgreSQL by restarting the process inside the surviving Pod; the PVC was never unmounted, so Longhorn had no reason to relocate the engine. This is the same result the operator’s unaided 01:05Z switchovers produced, now reproduced deliberately on the wave’s own action. The extra pod delete was required, and the plan’s gate resolved to the “delete” branch rather than the no-op branch.

Also confirmed at the same reading: the timeline advanced 17 → 18, which is what makes this a real switchover rather than a role relabel, and both main-16 and main-17 were already streaming from main-20 with 4.4 ms and 4.0 ms replay lag read from the new primary. The OIDC discovery endpoint answered 200 in 0.026 s immediately after the switchover — the keycloak-pooler-rw blip did not surface as a failure at the login path.

The gated delete of the demoted instance

[01:51:46Z] role=replica (re-read, not assumed)  siblings: main-17 Ready/replica, main-20 Ready/primary
[01:51:46Z] cordon tpi-beta-3
[01:51:46Z] delete pod/main-16
[01:51:48Z] recreated uid=fce31f44-… (was 7da421db-…), placed on tpi-beta-4
[01:51:48Z] uncordon tpi-beta-3 — 2 s of cordon, placement already decided
[01:52:40Z] Ready True, +54 s
[01:52:43Z] cluster main-20/main-20/3/Cluster in healthy state
[01:52:43Z] engine im=instance-manager-24264c66… [v1.12.1]  (was instance-manager-1b8c1c4d… [v1.10.1])

[VERIFIED: uid-gated drain log, 2026-08-30T01:51:45Z–01:52:43Z]

The delete was issued only after the role was re-read as replica on the live pod, which is the state CNPG’s guidance asks for before deleting a primary instance. The primary pod was never deleted while it was the primary: at 01:51:46Z the primary was main-20 on tpi-beta-2, and the delete targeted main-16 on tpi-beta-3. Cordoning tpi-beta-3 was safe for the same reason — no primary was on it — and it is exactly the action that, taken while main-16 was still the primary, made CNPG switch over on its own at 01:04:59Z.

Recovery output, main-16’s own:

2026-08-30T01:52:31Z–01:52:34Z
  [LOG] database system was shut down in recovery at 2026-08-30 01:51:46 UTC
  [LOG] restored log file "00000012.history" from archive        <- the NEW timeline's history
  [LOG] entering standby mode
  [LOG] redo starts at 1CD/1C0001A8
  [LOG] consistent recovery state reached at 1CD/1C0D87B0
  [LOG] database system is ready to accept read-only connections
  [LOG] invalid record length at 1CD/1C0D87B0: expected at least 24, got 0   <- end-of-WAL marker
  [LOG] started streaming WAL from primary at 1CD/1C000000 on timeline 18

[VERIFIED: kubectl -n postgres logs main-16 -c postgres, 2026-08-30T01:52:50Z] — the demoted former primary picked up timeline 18 from the archive and rejoined as a standby of the instance that replaced it. Membership read from the new primary, not from the target:

2026-08-30T01:52:55Z  psql on main-20: SELECT … FROM pg_stat_replication
main-16 | streaming | async | 00:00:00.017408
main-17 | streaming | async | 00:00:00.005504

Filesystem half: /dev/longhorn/pvc-780c5703… ext4 rw,relatime in main-16’s /proc/mounts, and kubelet /stats/summary on tpi-beta-4 reporting 2.23 GiB of 48.91 GiB at 01:53:55Z — the new node.

The after-window, read rather than counted

window	pod	cnpg-level	pg-severity	corrupt_tight	prescribed	all lines
switchover + delete, 01:50:53Z → 01:53:33Z	main-16	0 → 1	0 → 94	0 → 0	0 → 142	0 → 382
	main-17	0 → 0	0 → 3	0 → 0	0 → 27	0 → 50
	main-20	0 → 1	0 → 1	0 → 0	0 → 16	0 → 45
	keycloak-pooler-rw ×2	0 → 0	0 → 0	0 → 0	0 → 2 each	0 → 14 each
delete only, 01:51:46Z → 02:02:00Z	main-16	1	9	0	—	151

The two cnpg-level hits, both read in full:

• 01:51:00.421Z main-20 — failed to run wal-archive command / switchover in progress, refusing archiving. The switchover class, present four times in the 01:05Z unaided control.
• 01:51:46.938Z main-16 — While terminating server, one second after our own delete.

The 98 pg-severity hits fall into six classes, every one of them in the 01:05Z unaided-switchover control: terminating connection due to administrator command (61), the database system is starting up (24), the database system is shutting down (8), streaming replication receiver (2), could not send end-of-streaming message to primary (2), terminating walreceiver process due to administrator command (1). Zero new body classes. corrupt_tight is zero everywhere, and the all-lines control is non-zero on all five pods, so every zero above is a real zero.

[VERIFIED: default.otel_logs, 2026-08-30T01:53:33Z and 2026-08-30T02:02:00Z]

Rescue fields across the switchover and the delete

volume	lastAutoSalvagedAt	remountRequestedAt	lastDegradedAt
main-16 before / after	"" / ""	2026-05-12T01:25:34Z / byte-identical	"" / ""
main-17 before / after	"" / ""	"" / ""	"" / ""
main-20 before / after	"" / ""	"" / ""	"" / ""

[VERIFIED: 2026-08-30T01:45:53Z and 2026-08-30T01:54:15Z] — main-16’s three-and-a-half-month-old remountRequestedAt did not advance across a switchover, a demotion and a pod delete.

The instance-manager reaping — the last v1.10.1 object in the estate

2026-08-30T01:52:58Z
instance-manager-1b8c1c4d55fc05f3f1a9af947ff100a4  ->  Error from server (NotFound)

[VERIFIED: kubectl -n longhorn-system get instancemanagers.longhorn.io <name>] — it held postgres/main-16 alone, and Longhorn reaped it within seconds of the engine leaving. It was never deleted or patched.

Three instance-managers were reaped across this wave — 9e303193…, 996e594d… and 1b8c1c4d… — and every one of them left on Longhorn’s initiative. The inventory went 12 → 11 objects with the newest still dated 2026-08-24T14:58:29Z, so zero were created, and no instancemanagers.longhorn.io object was deleted or patched at any point in either task.

With 1b8c1c4d… and 9e303193… gone, no v1.10.1 instance-manager remains anywhere in the estate. The three survivors are all v1.11.2 and all belong to vault.

Replica churn — one benign rebuild on the drained volume, and one collateral rebuild this wave caused elsewhere

main-16’s replacement landed on tpi-beta-4, where the volume had no replica, so a fourth was built and the excess dropped:

01:52:50Z  pvc-780c5703  4 replicas: tpi-beta-1, tpi-beta-3, tpi-alpha-2, tpi-beta-4 healthyAt=""
01:53:48Z  pvc-780c5703  4 replicas, 1 still empty          vol attached/healthy
01:54:04Z  pvc-780c5703  3 replicas: tpi-beta-3, tpi-alpha-2, tpi-beta-4 healthyAt="2026-08-30T01:53:55Z"

[VERIFIED: kubectl -n longhorn-system get replicas.longhorn.io, polled every 15 s] — ~2 minutes for 2.2 GiB, attached/healthy throughout, never degraded. At wave close all three postgres volumes carry exactly three replicas, each with a non-empty healthyAt and an empty failedAt.

The collateral finding, recorded rather than glossed. A repository-wide sweep for the dangerous replica shape — healthyAt: "" and failedAt: "" together, which holds a slot indefinitely because it is neither healthy nor failed — returned one hit at wave close, and it is not a postgres volume:

2026-08-30T01:54:15Z
replica pvc-5fa64d0c-…-r-2cc80b40   node=tpi-beta-2   created=2026-08-30T01:51:46Z
volume  clickstack/clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-1-0
        attached/healthy   spec.numberOfReplicas=1   currentNodeID=tpi-beta-3

Its creationTimestamp is 01:51:46Z — the exact second this task cordoned tpi-beta-3. That node holds the sole replica of one of 02-05’s two 138 GiB single-replica ClickHouse volumes, and replica-auto-balance: best-effort reacted to the node becoming unschedulable by starting a full rebuild on tpi-beta-2. The cordon lasted two seconds; the rebuild it triggered did not stop with it.

It is a rebuild in progress, not a stuck slot, and the discriminator is measured rather than assumed — engine.status.rebuildStatus.progress climbs monotonically:

01:54:41Z  5%    01:56:44Z 10%    01:59:07Z 15%    02:01:09Z 20%
01:55:22Z  7%    01:57:45Z 12%    02:00:08Z 18%    02:02:31Z 23%

[VERIFIED: polled every 20 s, 01:54:41Z–02:02:31Z] — about 2.3 %/min, so roughly 35 minutes to completion, after which Longhorn drops back to the volume’s declared single replica. The volume reads attached/healthy throughout and never degraded. A genuinely stuck empty/empty replica shows no rebuildStatus progress at all; that is the reading that separates the two, and it is the reading a future operator should take before treating this shape as the 27-day vault case.

Nothing was done about it, deliberately. The volume belongs to 02-05, the extra replica is above the declared count so Longhorn discards it unaided — measured three times in 02-05 — and deleting a rebuilding replica on a 138 GiB volume to tidy a record would be the more dangerous action.

The lesson for 02-07 and 02-08 is concrete: a cordon on this cluster is not a local action. Two seconds of tpi-beta-3 being unschedulable cost a 138 GiB rebuild on a volume in another namespace, and — earlier, during 02-05 — two unplanned switchovers of the production write primary. 02-08’s runbook should say so.

The instrument did not go dark during this wave either

Per-minute default.otel_logs ingest across the whole wave, 01:36:00Z–02:02:00Z: 26 of 26 consecutive minutes present, no gap, lowest minute 16,040 rows, 489,170 rows total. [VERIFIED: 2026-08-30T02:02Z] — so every log cell in all three rows is a real reading and no cell needed n/a.

Gate movement attributable to Task 2, and to the wave

Reading	UTC	outside v1.12.1	control, inside	stale IM rows
02-05 close	01:22:30Z	9	31	7
02-06 start of wave	01:36:47Z	9	31	6
after Task 1 (both replicas)	01:44:55Z	7	33	4
after Task 2 (the switchover half)	01:52:58Z	6	34	3

[VERIFIED: kubectl, 2026-08-30T01:52:58Z] — down by exactly three across the wave, up by exactly three, with three instance-managers reaped and none created. The six engines still outside v1.12.1 are the six vault engines, in three v1.11.2 instance-managers, and they are 02-07’s work. The estate gate (c) — volumes not attached+healthy on a v1.12.1 image — reads 0 of 40. No node is cordoned at wave close.

Tier 3’s postgres half is drained, using CloudNativePG’s own verbs throughout, and only vault remains.

Wave 02-07 — start-of-wave re-read, the unsealer pre-step, and the tier-3 vault work list

The last six engines in the estate are vault’s, and they are the only ones whose workload cannot be restarted and left to recover on its own. Vault here is Shamir-sealed with no in-cluster auto-unseal: what brings a restarted pod back is a container running on the Firewalla router, outside the cluster. D-06 puts this wave last for that reason. RESEARCH §4.3 found a second and sharper one, and this wave verified it against the live StatefulSet before touching anything.

=== start-of-wave gate re-read: 2026-08-30T03:21:33Z ===
--- (a) engine processes still outside v1.12.1 ---
6
--- (a-inverted) CONTROL: engine processes INSIDE v1.12.1 IMs ---
34
--- (b) instance-managers still on a pre-v1.12.1 image ---
instance-manager-27af93a4524a66b6e9b1bc3a6e0a6920 v1.11.2 tpi-alpha-2 engines=2 created=2026-06-06T17:36:48Z
instance-manager-68fca8eecce9b1d85ba3f5986b1faae4 v1.11.2 tpi-beta-1  engines=2 created=2026-06-06T17:37:13Z
instance-manager-b48dac91b45fa0a6f4869005f61020b4 v1.11.2 tpi-alpha-4 engines=2 created=2026-06-06T17:36:57Z
--- (c) volumes not attached+healthy on v1.12.1 image ---
--- (c) row count ---
0
--- total volumes ---
40
--- cordoned nodes ---
[]
--- rebuilds in flight ---
--- degraded volumes ---

[VERIFIED: kubectl, 2026-08-30T03:21:33Z] — the inverted-predicate control returned 34, strictly greater than zero, so (a)’s reading of 6 is a measurement and not a broken query. The instance-manager inventory holds 11 objects, 8 of them v1.12.1; the three above are the last pre-v1.12.1 objects in the estate and hold exactly the six vault engines, two each.

Reconciliation against 02-06

Reading	UTC	(a) outside v1.12.1	control, inside	stale IM rows
02-06 close of wave	2026-08-30T01:52:58Z	6	34	3
02-07 start of wave	2026-08-30T03:21:33Z	6	34	3

Engine drift since 02-06: zero, across 89 minutes, and the object inventory is unchanged too — no instance-manager was created or reaped in the gap. The estate that 02-06 handed over is the estate this wave starts from.

02-06’s handoff item, closed by one confirming read

02-06 left one open measurement: the 138 GiB collateral rebuild its two-second cordon of tpi-beta-3 started on clickstack/…clickhouse-0-1-0, a single-replica volume, still in flight at 28% when that wave closed. It resolved exactly as predicted:

2026-08-30T03:21:51Z
clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-1-0  attached/healthy  numberOfReplicas=1  node=tpi-beta-3
  replica pvc-5fa64d0c…-r-f074e0fb  node=tpi-beta-3  healthyAt=2026-07-27T07:37:52Z  failedAt=

[VERIFIED: kubectl -n longhorn-system get volumes/replicas.longhorn.io, 2026-08-30T03:21:51Z] — back to exactly one replica, on its home node, and it is the original replica with its pre-phase healthyAt byte-identical. Longhorn discarded the excess replica unaided once the rebuild finished, as 02-05 measured three times and 02-06 predicted. Estate-wide at the same instant: 0 rebuilds in flight, 0 degraded volumes, 0 cordoned nodes.

The unsealer pre-step — confirmed alive, not inferred

RESEARCH could not answer OQ-5 read-only from the cluster, and the plan forbids substituting “the pods are unsealed right now” as evidence: the pods being unsealed proves only that something unsealed them at some point, not that the thing that would unseal them after this wave’s deletes is running. Confirmation was taken on the Firewalla itself, over SSH, before any pod was touched.

2026-08-30T03:20:45Z   ssh router
  container:  vault-unseal | ghcr.io/lrstanley/vault-unseal:0.7.0 | running | Up 13 days
  inspect:    Status=running  StartedAt=2026-08-16T14:27:05.229712368Z
              Restarting=false  ExitCode=0  RestartPolicy=unless-stopped
  host:       up 19 days, load average 1.04

A container-status check alone would only show the process exists. The liveness claim is carried by its own log stream, which shows it polling all three endpoints right now rather than sitting in backoff. Sampled over the 30 minutes ending 03:21:04Z (long strings redacted at source, and the container never logs a key or a token — it logs the seal status it reads back):

{"fields":{"addr":"https://vault-0.fzymgc.house",…},"level":"info","timestamp":"2026-08-30T03:21:00.600222526Z","message":"running checks"}
{"fields":{"addr":"https://vault-0.fzymgc.house","status":{"type":"shamir","initialized":true,"sealed":false,
            "t":3,"n":5,"progress":0,"version":"1.21.2","recovery_seal":false,"storage_type":"raft"},…},
 "level":"info","timestamp":"2026-08-30T03:21:00.616071553Z","message":"seal status"}
{"fields":{"addr":"https://vault-2.fzymgc.house",…},"level":"info","timestamp":"2026-08-30T03:21:04.761705525Z","message":"running checks"}
…
--- log lines in the last 30 minutes ---
720

[VERIFIED: ssh router, sudo docker ps / inspect / logs --since 30m, 2026-08-30T03:20:45Z–03:21:04Z]

720 lines / 30 minutes = 24 lines per minute = 3 endpoints × 2 lines × 4 passes per minute, which is a 15-second cadence on every endpoint — the role’s vault_unseal_check_interval, not its vault_unseal_max_check_interval backoff. So the unsealer is not merely present: it is completing a successful check against each of vault-0, -1 and -2 every fifteen seconds, and it is reading t:3 n:5 — a 3-of-5 Shamir threshold — which is the shape it would need to unseal one. The recovery window a sealed pod faces is therefore ~15 s, not the 30-minute worst case, unless the unsealer starts failing and backs off.

Commands run on the Firewalla use the tools that box ships (grep, awk, sed); it has no rg. The repo’s rg-not-grep rule governs searches of this repository, not commands executed on a third-party appliance — an explicit CLAUDE.md exemption.

The probes, read from the live StatefulSet rather than inherited

2026-08-30T03:22:31Z  sts/vault
readinessProbe  GET https :8200 /v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204
                initialDelay=5s  period=5s  timeout=3s  failureThreshold=2
livenessProbe   GET https :8200 /v1/sys/health?standbyok=true
                initialDelay=120s period=5s  timeout=3s  failureThreshold=2
podAntiAffinity requiredDuringSchedulingIgnoredDuringExecution on kubernetes.io/hostname
updateStrategy  RollingUpdate      terminationGracePeriodSeconds 10
PDB             vault  maxUnavailable=1  allowedDisruptions=1

[VERIFIED: kubectl -n vault get sts vault -o json, 2026-08-30T03:22:31Z]

This is the trap, confirmed on the object rather than taken from research:

• The readiness probe carries sealedcode=204. A sealed Vault answers 204 on that URL, so Kubernetes marks the pod Ready. Readiness does not prove unsealed. Every gate in this wave is therefore sealed: false read from /v1/sys/health, never the pod’s Ready condition.
• The liveness probe has no sealedcode. A sealed Vault answers 503 there, so a pod that stays sealed fails liveness and is killed — after 120 s initial delay, then 2 × 5 s — roughly every 130 seconds. A climbing RESTARTS count on a vault pod is a symptom of sealing, not an unrelated crash. This is the most plausible account of vault-0’s restart count.
• rollout restart is forbidden here for a third reason on top of the two the plan gives: with RollingUpdate and ordinal-descending order it would reach vault-2, the active leader, first, and the readiness probe would let it walk straight on to -1 and -0.

Pre-wave seal state, leadership and restart counts

2026-08-30T03:19:38Z   curl /v1/sys/health?standbyok=true
vault-0 {"sealed":false,"standby":true, "initialized":true,"version":"1.21.2","cluster_name":"fzymgc-house"}
vault-1 {"sealed":false,"standby":true, "initialized":true,"version":"1.21.2","cluster_name":"fzymgc-house"}
vault-2 {"sealed":false,"standby":false,"initialized":true,"version":"1.21.2","cluster_name":"fzymgc-house"}

2026-08-30T03:19:38Z   kubectl -n vault get pods
NAME      UID         NODE          READY  RESTARTS  START
vault-0   fabcac0d-…  tpi-alpha-4   true   825       2026-06-16T15:08:26Z
vault-1   3239edac-…  tpi-beta-1    true    36       2026-07-03T18:05:44Z
vault-2   69ff4382-…  tpi-alpha-2   true    35       2026-06-16T15:06:48Z

[VERIFIED: curl + kubectl, 2026-08-30T03:19:38Z] — all three unsealed, exactly one standby: false. The ACTIVE LEADER is vault-2, which is also the highest ordinal. It is restarted last, and leadership is re-read immediately before every delete because it moves.

vault-0 carries 825 restarts, up from the 819 RESEARCH recorded. That is the pre-existing instability, and recording it here is what lets a later reader tell a restart this wave caused from one it did not. vault-1 and vault-2 sit at 36 and 35.

Cluster membership is read from Raft rather than from readiness, on the same principle 02-04 through 02-06 used for valkey, MongoDB, Keeper and CNPG — a witness that is not the target:

2026-08-30T03:23:05Z   vault operator raft list-peers
vault-0  vault-0.vault-internal:8201  voter=true  leader=false
vault-1  vault-1.vault-internal:8201  voter=true  leader=false
vault-2  vault-2.vault-internal:8201  voter=true  leader=true

[VERIFIED: vault operator raft list-peers, 2026-08-30T03:23:05Z] — three voters, quorum 2 of 3, so a strict one-at-a-time rotation never risks it. The operator session used for this read was issued 2026-08-29T15:56:42Z and expires 2026-08-30T09:56:42Z, so it does not depend on a login through the endpoint this wave disturbs. No token value was read, printed or copied at any point in this wave; nothing in it requires one.

The ExternalSecrets baseline, so refresh noise can be told from a volume problem

2026-08-30T03:23:05Z
ExternalSecrets:      84 total, 84 Ready, 0 not-Ready
ClusterSecretStore:   vault  Ready=True  reason=Valid

[VERIFIED: kubectl get externalsecrets -A / clustersecretstores, 2026-08-30T03:23:05Z] — ESO writes into Kubernetes Secret objects, so existing secrets survive a Vault blip and only refreshes inside the window can fail. With a clean 84/84 baseline, any not-Ready ExternalSecret appearing during an unseal window is attributable to the window and not to the volume, and the verdict row says which.

The matcher was measured before any zero was believed — the positive control

Five waves have now found the prescribed error matcher wrong in five different ways. On vault it is close but still not exact, and a positive control was established first so that a later zero is a reading rather than a broken query.

24 hours to 2026-08-30T03:24Z, ResourceAttributes['k8s.namespace.name'] = 'vault'
pod                      all_lines  prescribed  anchored [ERROR]  anchored [WARN]  corrupt_raw
vault-0                        375          44                29                8            0
vault-1                        112           1                 1                3            0
vault-2                      38962         241               241               57            0
vault-agent-injector-…         951           5                 5                1            0

The positive control passes: 29 / 1 / 241 real [ERROR] hits over 24 hours. A zero in a later window is therefore a measurement. Vault logs a bracketed level tag ([INFO], [WARN], [ERROR]), so the anchored matcher is position(Body,'[ERROR]')>0. The prescribed error substring over-counts on vault-0 by 15 of 44 — lines whose level is [INFO] or [WARN] but whose body carries an error="…" field — and matches exactly on vault-2. Both are reported per row.

The corruption negative control reads 0 raw across 24 hours on every pod and needed no tightening, unlike waves 02-04, 02-05 and 02-06.

The third control — a steady-state baseline cannot contain a startup, so the restart-attributable classes must be known in advance — is unusually complete here, because the whole 24-hour baseline is Raft peer churn, which is exactly the class our own restarts produce:

24 hours, [ERROR] body classes, timestamps and addresses normalised
vault-0  storage.raft.raft-net: failed to accept connection: error="Raft RPC layer closed"      12
vault-0  storage.raft.raft-net: failed to decode incoming command: error="transport shutdown"   10
vault-0  storage.raft.raft-net: failed to flush response: error="write tcp …: broken pipe"       7
vault-1  storage.raft.raft-net: failed to flush response: error="write tcp …: broken pipe"       1
vault-2  storage.raft: failed to heartbeat to: peer=vault-N.vault-internal:8201 backoff time=…  131
vault-2  storage.raft: failed to appendEntries to: peer="{Voter vault-N …}"                     101
vault-2  storage.raft: failed to pipeline appendEntries: peer="{Voter vault-N …}"                 9

[VERIFIED: default.otel_logs, 24 h to 2026-08-30T03:24Z] — every class is a peer-connectivity class already running in steady state, and the leader’s 241 are heartbeats to a peer that is already flapping. So the expectation for this wave is: hits, but zero new body classes.

Cordon blast radius, surveyed before it was needed — and one node ruled out

02-06 recorded that a cordon on this cluster is not a local action: two seconds of tpi-beta-3 unschedulable started a 138 GiB rebuild in another namespace. The global setting is what makes it so, and it was read rather than assumed:

2026-08-30T03:22:16Z  settings.longhorn.io
replica-auto-balance = best-effort        auto-salvage = true
replica-replenishment-wait-interval = 90  node-down-pod-deletion-policy = delete-both-statefulset-and-deployment-pod

Every volume in the estate carries spec.replicaAutoBalance: ignored, which means each one inherits best-effort. So the survey below was taken for all three vault source nodes before any fallback was considered:

Source node	vault pod	Volumes attached there	Single-replica volumes at risk	Cordon verdict
tpi-alpha-4	vault-0	12, all numberOfReplicas 2 or 3	none	costly — renovate-ce-cache-hi is 18.95 GiB
tpi-beta-1	vault-1	3, one of them clickstack/…clickhouse-0-0-0, numberOfReplicas: 1, sole replica pinned to this node, 123 GiB used	YES	FORBIDDEN
tpi-alpha-2	vault-2	7, all numberOfReplicas 2 or 3	none	costly but bounded

[VERIFIED: kubectl get pods --field-selector spec.nodeName / volumes + replicas.longhorn.io, 2026-08-30T03:22:16Z]

tpi-beta-1 must not be cordoned. It holds the sole replica of the 142 GiB single-replica ClickHouse volume, so cordoning it reproduces precisely the 02-04/02-05 hazard the phase has already paid for twice — a forced best-effort rebuild of ~123 GiB on a volume with no redundancy. The vault-1 step therefore has no cordon fallback available, and its verb must be the plain delete with a re-roll if the scheduler returns the pod to its own node.

Two facts make that acceptable here in a way it was not for the ClickHouse volumes. The vault volumes are tiny — 24 KiB used on each audit- and 41–53 MiB on each data- — so even an unwanted rebuild is seconds, not half an hour. And the pods carry no nodeSelector, no tolerations and no resource requests, on eight uncordoned untainted nodes, with only a requiredDuringScheduling pod-anti-affinity against the other two vault pods; the scheduler is otherwise free.

Tier-3 vault work list, keyed by owning pod

=== 2026-08-30T03:21:51Z — 6 engines outside v1.12.1, all of them vault ===
vault/data-vault-0   node=tpi-alpha-4  attached/healthy/v1.12.1  IM=instance-manager-b48dac91… [v1.11.2]  replicas=3
vault/audit-vault-0  node=tpi-alpha-4  attached/healthy/v1.12.1  IM=instance-manager-b48dac91… [v1.11.2]  replicas=3
vault/data-vault-1   node=tpi-beta-1   attached/healthy/v1.12.1  IM=instance-manager-68fca8ee… [v1.11.2]  replicas=2
vault/audit-vault-1  node=tpi-beta-1   attached/healthy/v1.12.1  IM=instance-manager-68fca8ee… [v1.11.2]  replicas=2
vault/data-vault-2   node=tpi-alpha-2  attached/healthy/v1.12.1  IM=instance-manager-27af93a4… [v1.11.2]  replicas=3
vault/audit-vault-2  node=tpi-alpha-2  attached/healthy/v1.12.1  IM=instance-manager-27af93a4… [v1.11.2]  replicas=3

Order	Pod	Role at wave start	Node	Volumes moved by that one delete	IM at start
1st	vault-0	standby	tpi-alpha-4	vault/data-vault-0, vault/audit-vault-0	b48dac91… v1.11.2
2nd	vault-1	standby	tpi-beta-1	vault/data-vault-1, vault/audit-vault-1	68fca8ee… v1.11.2
LAST	vault-2	ACTIVE LEADER	tpi-alpha-2	vault/data-vault-2, vault/audit-vault-2	27af93a4… v1.11.2

Three pod restarts clear six engines. Each vault pod holds both a data- and an audit- PVC, so the work is keyed by pod and the record by volume: delete once, gate once, fill both rows from that single restart. audit-vault-1 and data-vault-1 carry numberOfReplicas: 2 and the other four carry three. All ten replicas across the six volumes read a non-empty healthyAt and an empty failedAt at wave start — none is in the stuck-slot shape.

Task 2 — the vault drain, one pod at a time, standbys first and the leader last

Three deletes, six engines. Each vault pod holds both a data- and an audit- PVC, so the pod was deleted once, the gate taken once, and both volume rows filled from that single restart.

The gate between every step is sealed: false read from /v1/sys/health?standbyok=true. No step was gated on the pod’s Ready condition, and the reason is measured three times below rather than argued. kubectl rollout restart was not used at any point; every detach was kubectl -n vault delete pod <pod>. No instancemanagers.longhorn.io object was deleted or patched.

#	Pod	Delete UTC	uid before → after	node before → after	instance-manager before → after
1	vault-0 (standby)	03:27:42Z	fabcac0d… → 8ae2c0a6…	tpi-alpha-4 → tpi-beta-4	b48dac91… v1.11.2 → 24264c66… v1.12.1
2	vault-1 (standby)	03:45:05Z	3239edac… → 228fcbab…	tpi-beta-1 → tpi-beta-2	68fca8ee… v1.11.2 → 5f4ef201… v1.12.1
3	vault-2 (ACTIVE LEADER)	03:48:59Z	69ff4382… → c6ca51f8…	tpi-alpha-2 → tpi-alpha-1	27af93a4… v1.11.2 → 6abfc3c9… v1.12.1

[VERIFIED: kubectl -n vault get pod -o json + engines/instancemanagers.longhorn.io re-read per step, 2026-08-30T03:27:42Z–03:51:10Z] — every delete moved the pod to a different node on the first attempt, so no cordon fallback was needed anywhere in this wave and none was applied. Three deletes, three migrations, six engines.

A fresh three-endpoint leadership read was taken immediately before each delete, because leadership moves — including as a consequence of the previous restart:

03:27:33Z  before delete 1   vault-0 standby=true   vault-1 standby=true   vault-2 standby=FALSE
03:45:04Z  before delete 2   vault-0 standby=true   vault-1 standby=true   vault-2 standby=FALSE
03:48:59Z  before delete 3   vault-0 standby=true   vault-1 standby=true   vault-2 standby=FALSE
03:51:10Z  wave close        vault-0 standby=FALSE  vault-1 standby=true   vault-2 standby=true

Raft was read from vault operator raft list-peers at each boundary — a witness that is not the target, on the same principle 02-04 through 02-06 used for valkey, MongoDB, Keeper and CNPG. Three voters at every reading, so quorum held at 2 of 3 throughout and never dropped below it.

The sealedcode=204 trap, measured three times

This is the reason the plan forbids a readiness gate, and this wave produced the measurement rather than inheriting the claim. For each pod: the instant Kubernetes marked it Ready, and the seal state read from the API at and after that instant.

Pod	Ready=True at	still reading sealed: true at	first sealed: false	Ready while SEALED
vault-0	03:42:50Z	03:43:24Z	03:43:34Z	≥ 34 s
vault-1	03:46:36Z	03:47:18Z	03:47:29Z	≥ 42 s
vault-2	03:50:13Z	03:50:41Z	03:50:51Z	≥ 28 s

[VERIFIED: kubectl -n vault get pod -o json .status.conditions[type=Ready].lastTransitionTime, against the polled /v1/sys/health readings recorded in the step logs above]

A Ready-gated loop would have advanced to the next pod while the one it had just restarted was sealed — on all three pods, every time. The bound is a lower bound: Ready was true at the transition timestamp and the health poll’s cadence is 10 s, so the true overlap is at least the figure shown. Vault answers 204 on /v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204 while sealed, which is exactly what the readiness probe asks for, so the probe is behaving as configured — it is the gate built on it that is wrong.

The liveness probe’s half of the same story was also produced. vault-0’s container was started at ~03:40:33Z, was still sealed 130 seconds later, and was killed and restarted at 03:42:44Z — restartCount 0 → 1 on a pod that was three minutes old. That is initialDelaySeconds: 120 plus 2 × periodSeconds: 5 on a probe with no sealedcode, to the second. A climbing RESTARTS count on a vault pod is a symptom of a sealed pod, and it is the most plausible account of the 825 restarts vault-0 carried into this wave.

The unsealer did the work, and it was observed doing it

03:43:31.113Z  {"addr":"https://vault-0.fzymgc.house"}                         "running checks"
03:43:31.147Z  {"addr":"https://vault-0.fzymgc.house","progress":0,"token":1,"total":3}  "using unseal token"
03:43:31.149Z  {"addr":"https://vault-0.fzymgc.house"}                         "token successfully sent"
03:43:31.149Z  {"addr":"https://vault-0.fzymgc.house","progress":0,"token":2,"total":3}  "using unseal token"
03:43:31.151Z  {"addr":"https://vault-0.fzymgc.house"}                         "token successfully sent"
03:43:31.151Z  {"addr":"https://vault-0.fzymgc.house","progress":0,"token":3,"total":3}  "using unseal token"
03:43:31.280Z  {"addr":"https://vault-0.fzymgc.house"}                         "token successfully sent"
03:43:31.280Z  {"error":"(was sealed) https://vault-0.fzymgc.house now unsealed with tokens"}  "notify-error"
03:43:31.280Z  {"addr":"https://vault-0.fzymgc.house","delay":210000000000}     "delaying checks due to errors"

[VERIFIED: ssh router, sudo docker logs vault-unseal, long strings redacted at source, 2026-08-30T03:43:31Z] — the out-of-cluster container submitted 3 of 3 key shares and Vault came back. "token":N is a share index, "progress" is Vault’s own unseal progress counter, and no key material appears in the stream at any point; the container logs the seal status it reads back, not what it sends. It then set a 210-second backoff on that endpoint, which is why the next two pods were unsealed inside ~50 seconds each while vault-0 took three minutes: the unsealer had already been backing off on vault-0 through its twelve-minute absence.

This is the pre-step’s payoff. Had the container not been running, all three pods would have sat sealed behind a Ready-reporting probe, and the recovery would have been a manual unseal with the SOPS-held shares — an operator action, at 03:40 in the morning.

vault-0 stalled twelve minutes on a LUKS device vector was holding open

The first delete did not complete. vault-0 sat Pending on tpi-beta-4 from 03:27:42Z with a Multi-Attach error, and the CSI plugin on the source node said exactly why:

03:36:00.116Z  Volume pvc-fdf2a3c3-… closing active crypto device /dev/mapper/pvc-fdf2a3c3-…   NodeUnstageVolume
03:36:00.119Z  Closing LUKS device pvc-fdf2a3c3-…                                              crypto.CloseVolume
03:36:00.150Z  error  NodeUnstageVolume: failed after 10 attempts to refresh stale namespace directory:
               failed to execute: /usr/bin/nsenter [nsenter --mount=/host/proc/2434477/ns/mnt
               --ipc=/host/proc/2434477/ns/ipc cryptsetup luksClose pvc-fdf2a3c3-…],
               stderr "Device pvc-fdf2a3c3-… is still in use.": exit status 5

[VERIFIED: kubectl -n longhorn-system logs longhorn-csi-plugin-6hn2t -c longhorn-csi-plugin, tpi-alpha-4, 2026-08-30T03:36:00Z and again at 03:38:02Z] — the same failure on a two-minute retry loop. This is 02-02’s vector/LUKS interaction, which cost that wave fifteen minutes on nats-1. The remedy is one pod delete against vector on the source node, and it worked in 114 seconds:

03:38:24Z  kubectl -n vector delete pod vector-lglnc          (tpi-alpha-4, the SOURCE node)
03:40:18Z  VolumeAttachment pvc-fdf2a3c3-… tpi-alpha-4 GONE   <- detach finally completed
03:40:29Z  VolumeAttachment pvc-fdf2a3c3-… tpi-beta-4  attached=true
03:40:33Z  vault-0 container starts on tpi-beta-4

This wave sharpened 02-02’s finding. Each vault pod holds two Longhorn volumes and only one of them stalled:

/dev/longhorn/pvc-7ec86f2c-…  /vault/audit  ext4 rw,relatime     <- plain, detached at 03:27:57Z, 15 s
/dev/mapper/pvc-fdf2a3c3-…    /vault/data   ext4 rw,relatime     <- dm-crypt, stalled 12.6 minutes

The audit- volume detached and reattached cleanly inside the same delete that left the data- volume stuck. Both come from the same StorageClass name, but only data- is a live dm-crypt device at the time of unstage. It is the LUKS luksClose that vector’s /var/lib mount namespace blocks, not the detach in general — a plain Longhorn block device unstages straight through. The nsenter --mount=/host/proc/2434477/ns/mnt in the error names the pinning namespace directly.

The remaining two deletes therefore released vector on the source node pre-emptively, ~20 seconds after the vault pod delete, and neither stalled at all:

03:45:25Z  kubectl -n vector delete pod vector-hc8s2  (tpi-beta-1)   -> vault-1 running at 03:46:28Z
03:49:17Z  kubectl -n vector delete pod vector-5dxgk  (tpi-alpha-2)  -> vault-2 running at 03:50:05Z

The vector DaemonSet read 8 desired / 8 ready / 8 available at wave close.

tpi-beta-1 was not cordoned, and the volume that made that necessary is untouched

vault-1’s source node holds the sole replica of clickstack/…clickhouse-0-0-0 (numberOfReplicas: 1, 123 GiB used). With the global replica-auto-balance: best-effort, cordoning that node forces a full rebuild of it — the hazard 02-04 and 02-05 each paid for once and 02-06 triggered a third time from two seconds of unschedulable. The plain delete moved vault-1 on the first attempt, so the question never became live, and the volume was re-read afterwards to prove it:

03:47:45Z  clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-0-0
           attached/healthy  node=tpi-beta-1  numberOfReplicas=1
           cordoned=[]   rebuilds_in_flight=0

[VERIFIED: kubectl, 2026-08-30T03:47:45Z] — no node was cordoned at any point in this wave, so the wave has no cross-namespace collateral of its own to hand forward.

The leader step: sys/step-down is 403, so leadership was lost deliberately rather than moved

The intended verb was CNPG’s shape from 02-06 — move the role with the product’s own verb, confirm it, then delete the demoted member. Vault’s equivalent is sys/step-down, and this session cannot call it:

03:48:29Z  vault operator step-down   (VAULT_ADDR=https://vault-2.fzymgc.house)
           Error stepping down: PUT https://vault-2.fzymgc.house/v1/sys/step-down
           Code: 403.  * permission denied
03:48:37Z  re-read: vault-2 still standby=false, still Raft leader

[VERIFIED: vault operator step-down, 2026-08-30T03:48:29Z] — sys/step-down requires the sudo capability, and the admin policy this OIDC session carries does not grant it. Widening a Vault policy to tidy a drain is an architectural and security change, not a drain step, so it was not done; it is recorded here as standing knowledge for whoever next needs to move Vault leadership under change control.

Leadership was therefore lost deliberately rather than moved: the leader went last, after both other members had been re-read as sealed: false and confirmed as Raft voters, so the election had a healthy quorum of two waiting for it. It took 19 seconds or less:

03:48:59Z  delete vault-2 (standby=false, the leader)
03:49:05.225Z  vault-2 logs [ERROR] storage.raft: peer has newer term, stopping replication
03:49:18Z  vault-0 reads standby=FALSE  <- new leader, ≤19 s after the delete
03:51:10Z  raft list-peers: vault-0 leader=true, vault-1 voter, vault-2 voter

One consequence has to be stated plainly rather than glossed: at the moment of the third delete the target was reading standby: false. The task’s acceptance criteria ask both that the leader be restarted last and that no delete target read standby: false, and with sys/step-down returning 403 those two cannot both hold — the leader cannot be drained without deleting a standby: false pod. The plan’s <action> and <done> are unambiguous that the leader is restarted last, so that is what was done, and this paragraph is the record of the criterion that could not be satisfied literally and why. The intent behind it — never restart the active leader while other members are unverified — was satisfied: both standbys were drained, unsealed and confirmed as voters first.

The after-windows, read rather than counted

Pod	window	all lines	prescribed error	anchored [ERROR]	[WARN]	corruption control
vault-0	before 03:11:42–03:27:42	10	2	2	1	0
vault-0	after 03:27:42→	56	0	0	2	0
vault-1	before 03:29:05–03:45:05	3	0	0	0	0
vault-1	after 03:45:05→	42	3	3	2	0
vault-2	before 03:32:59–03:48:59	749	320	320	3	0
vault-2	after 03:48:59→	53	4	4	2	0

[VERIFIED: default.otel_logs, 2026-08-30T03:52Z] — the all-lines control moves in both directions across the three pods, which proves the query reaches each replacement pod rather than reading a dead stream. The corruption negative control reads 0 in every window, needing none of the tightening waves 02-04, 02-05 and 02-06 each had to derive.

vault-2’s before-window figure of 320 is not a baseline of anything: vault-2 was the leader while this wave had vault-0 down for twelve minutes, and 320 of those lines are its heartbeats to the peer we had deleted. It is our own drain, measured from the other side, and the drop to 4 afterwards is vault-2 no longer being leader. Stated rather than presented as an improvement.

Every after-window line was read. Nine classes across the three pods, and eight of the nine are already in the 24-hour pre-wave control — failed to heartbeat to, failed to appendEntries to, failed to pipeline appendEntries, failed to accept connection: Raft RPC layer closed, failed to decode incoming command: transport shutdown. Every one is peer connectivity, and every instance sits within seconds of one of our own three deletes.

One class is new, and it is the leadership handover itself:

03:49:05.225Z  vault-2  [ERROR] storage.raft: peer has newer term, stopping replication:
                        peer="{Voter vault-0 vault-0.vault-internal:8201}"          n=1

6.2 seconds after the leader delete, logged once, by the leader we deliberately took last, naming the peer that won the election. It is the expected signature of losing leadership deliberately and it appears nowhere in the 24-hour baseline because no leader election happened in that baseline. It is attributed, not waved past.

ExternalSecrets: the caveat did not become live

The plan asks that a vault row distinguish an elevated log delta caused by ExternalSecrets refresh failures during an unseal window from a genuine drain failure. It never had to, and that is a measurement rather than an omission:

Reading	UTC	ExternalSecrets Ready	ClusterSecretStore vault
pre-wave	03:23:05Z	84 / 84	Ready=True, Valid
after two restarts	03:47:45Z	84 / 84	—
wave close	03:52:07Z	84 / 84	Ready=True, Valid

[VERIFIED: kubectl get externalsecrets -A / clustersecretstores, three readings] — not one ExternalSecret left Ready across the whole wave, over 25+ namespaces. ESO writes into Kubernetes Secret objects, so cached secrets survive a Vault blip; the unseal windows here were 34–53 seconds each and the ClusterSecretStore reconciles against the vault.fzymgc.house service address, which had a live member throughout. No verdict row needed the distinction, and none claims it.

Rescue fields across the vault restarts — read as a DELTA

2026-08-30T03:51:22Z
audit-vault-0   remountRequestedAt=2026-05-12T13:05:06Z   lastAutoSalvagedAt=""
data-vault-0    remountRequestedAt=2026-05-12T13:05:06Z   lastAutoSalvagedAt=""
audit-vault-1   remountRequestedAt=""                     lastAutoSalvagedAt=""
data-vault-1    remountRequestedAt=""                     lastAutoSalvagedAt=""
audit-vault-2   remountRequestedAt=2026-05-12T12:02:01Z   lastAutoSalvagedAt=""
data-vault-2    remountRequestedAt=2026-05-12T12:02:01Z   lastAutoSalvagedAt=""

Four of the six carry a non-empty remountRequestedAt dated 2026-05-12, three and a half months before this phase — 02-02’s correction that the field is not empty estate-wide, holding on the last six volumes too. Read as the delta it must be: not one value advanced into this wave’s window, and none could have, since every one predates it by months. lastAutoSalvagedAt is empty on all six, consistent with 02-02’s finding that it is genuinely 0/40 estate-wide.

Replica churn, and every shape accounted for

Each of the three moves rebuilt one small replica onto the destination node and Longhorn retired the one on the source, so every volume held exactly its declared replica count at wave close and none went above it:

2026-08-30T03:51:22Z — all 16 replicas across the six volumes: healthyAt NON-EMPTY, failedAt EMPTY
audit-vault-0 (3)  tpi-beta-4  03:29:02Z   tpi-alpha-3 00:44:03Z   tpi-beta-3  2026-05-12T14:07:42Z
data-vault-0  (3)  tpi-beta-4  03:40:59Z   tpi-alpha-1 00:50:58Z   tpi-beta-3  2026-08-23T17:04:15Z
audit-vault-1 (2)  tpi-beta-2  03:46:33Z   tpi-alpha-3 00:51:53Z
data-vault-1  (2)  tpi-beta-2  03:46:59Z   tpi-alpha-1 00:44:28Z
audit-vault-2 (3)  tpi-alpha-1 03:50:46Z   tpi-beta-3  2026-08-15T07:25:52Z   tpi-alpha-2 2026-06-11T16:00:02Z
data-vault-2  (3)  tpi-alpha-1 03:51:04Z   tpi-beta-1  2026-08-18T07:21:18Z   tpi-alpha-2 2026-06-11T16:00:33Z

Not one replica is in the healthyAt: "" AND failedAt: "" shape — the stuck-slot trap that once held one of these very volumes on a single replica for 27 days. The three new replicas are 24 KiB and 41–53 MiB, so each reached healthyAt within seconds of its pod starting and none of them needed the rebuildStatus.progress-across-two-samples discriminator 02-06 had to derive. 0 rebuilds in flight and 0 degraded volumes estate-wide at wave close.

The instrument did not go dark during the wave

default.otel_logs, per minute, 03:19:00Z – 03:52:00Z
minutes present: 33 of 33      lowest minute: 15,044 rows

[VERIFIED: default.otel_logs, 2026-08-30T03:52Z] — no gap, so every log cell in the six rows below is a real reading and none needed n/a.

The last three pre-v1.12.1 instance-managers were reaped by Longhorn, not by us

2026-08-30T03:52:07Z
instance-manager-b48dac91b45fa0a6f4869005f61020b4   NotFound     (was v1.11.2, tpi-alpha-4, held vault-0's two engines)
instance-manager-68fca8eecce9b1d85ba3f5986b1faae4   NotFound     (was v1.11.2, tpi-beta-1,  held vault-1's two engines)
instance-manager-27af93a4524a66b6e9b1bc3a6e0a6920   NotFound     (was v1.11.2, tpi-alpha-2, held vault-2's two engines)

[VERIFIED: kubectl -n longhorn-system get instancemanagers.longhorn.io, 2026-08-30T03:52:07Z] — the inventory went 11 objects → 8, all eight of them v1.12.1, and the newest object is still dated 2026-08-24T14:58:29Z. Zero instance-managers created, zero deleted or patched by this wave; each disappeared on Longhorn’s own initiative once its last engine left, exactly as it did in 02-04, 02-05 and 02-06. There is now no pre-v1.12.1 instance-manager anywhere in the estate.

Gate movement attributable to Task 2, and to the wave

Reading	UTC	outside v1.12.1	control, inside	pre-v1.12.1 IM rows
02-06 close	01:52:58Z	6	34	3
02-07 start of wave	03:21:33Z	6	34	3
after vault-0	~03:44:22Z	4	36	2
after vault-1	~03:47:45Z	2	38	1
after vault-2 — wave and phase close	03:51:22Z	0	40	0

[VERIFIED: kubectl, 2026-08-30T03:51:22Z] — down by exactly six across the wave, up by exactly six, with three instance-managers reaped and none created. The estate gate (c) — volumes not attached + healthy on a v1.12.1 image — reads 0 of 40. The straggler set is empty on a live re-read, evaluated over every volume the vault namespace returns rather than against a fixed count. No node is cordoned. No rebuild is in flight. No volume is degraded. All three Vault endpoints read sealed: false and exactly one reads standby: false.

Every Longhorn engine in this estate now runs in a v1.12.1 instance-manager.

Wave 02-08 — the closing LH-03 gate, read two-sided, and the LH-01 record set-equal to the live estate

The phase’s whole value is in what it can prove afterwards. This wave proves it: the gate is read two-sided again, the instance-manager row list is read as the second and independent completion signal, and the verdict table is asserted set-equal to a live re-read in both directions rather than against any figure carried forward from an earlier wave.

Reconciliation against 02-07

Reading	UTC	(a) outside v1.12.1	control, inside	pre-v1.12.1 IM rows
02-07 close of wave	2026-08-30T03:51:22Z	0	40	0
02-08 start of wave	2026-08-30T04:05:35Z	0	40	0

Drift since 02-07: zero, across 14 minutes, in both the engine count and the object inventory.

The closing gate, verbatim

=== capture start: 2026-08-30T04:05:35Z ===
--- (a) engine processes still outside v1.12.1 ---
0
--- (a-inverted) CONTROL: engine processes INSIDE v1.12.1 IMs ---
40
--- (a-sum) total engine processes over ALL instance-managers, no version predicate ---
40
--- (b) stale instance-manager rows ---
--- (b) row count / (b-control) v1.12.1 rows / (b-total) all rows ---
0  8  8
--- (b) full inventory, all 8 ---
instance-manager-6abfc3c987e931e5e659a3ddbe768002 v1.12.1 tpi-alpha-1 engines=7  replicas=12 created=2026-08-24T14:58:17Z
instance-manager-e9ba934d9f24638826b55ad5a311c489 v1.12.1 tpi-alpha-2 engines=5  replicas=17 created=2026-08-24T14:58:06Z
instance-manager-88a77e8ce104cc546dcc35359557c2c5 v1.12.1 tpi-alpha-3 engines=4  replicas=12 created=2026-08-24T14:58:12Z
instance-manager-3536df4b049a0ba362c2b953d0af899b v1.12.1 tpi-alpha-4 engines=10 replicas=13 created=2026-08-24T14:57:53Z
instance-manager-7e6aabe3309d6872156f120555d3daa4 v1.12.1 tpi-beta-1  engines=1  replicas=4  created=2026-08-24T14:57:53Z
instance-manager-5f4ef20112275342d7f902e7c80e5f0f v1.12.1 tpi-beta-2  engines=6  replicas=11 created=2026-08-24T14:58:29Z
instance-manager-4e199918a77e61d48f64b343e853bd91 v1.12.1 tpi-beta-3  engines=1  replicas=9  created=2026-08-24T14:58:23Z
instance-manager-24264c6672978ac81edb62b293331e50 v1.12.1 tpi-beta-4  engines=6  replicas=8  created=2026-08-24T14:57:45Z
--- (c) volumes not attached+healthy on v1.12.1 ---
--- (c) row count / (c-control) rows that ARE / total volumes ---
0  40  40
--- cordoned nodes / degraded volumes / rebuilds in flight ---
0
0
0
=== capture end: 2026-08-30T04:05:36Z ===

[VERIFIED: kubectl -n longhorn-system get instancemanagers/volumes.longhorn.io + kubectl get nodes, 2026-08-30T04:05:35Z]

Why this zero is a measurement and not a broken query. Three independent properties are asserted on the same read, and a query that had stopped seeing engines would fail all three:

1. The inverted control is non-zero. (a) returns 0; the same traversal with the version predicate inverted returns 40. An empty primary result and a working query are only distinguishable by this half. RESEARCH P-9 documents the concrete way this goes wrong here — engines.longhorn.io .spec.instanceManagerName is empty on all 40 engines, so a gate built on it returns blanks that read exactly like a clean estate. .status.instanceManagerName is the populated field; it is non-empty on 40 of 40. [VERIFIED: kubectl get engines.longhorn.io, 2026-08-30T04:04:15Z — total 41, with .status.instanceManagerName 41, with .spec.instanceManagerName 0; 40 / 40 / 0 on the settled 40-volume re-read at 04:05:35Z]
2. The two halves sum to the live total. 0 + 40 = 40, and 40 is the count of engine processes across all instance-managers with no version predicate at all, and also the live volume count and the live engine count. The two predicates partition the set exactly; neither drops rows.
3. The query demonstrably moves. It is not a frozen constant — see the transient volume below, which the control counted as 41 at 04:04:15Z and 40 at 04:04:58Z. A query that can go up and come back down within 43 seconds is reading live state.

The denominator is not constant — a transient Velero volume, caught mid-flight

The first closing read was taken at 04:04:15Z and returned 41 volumes and 41 engines, not 40. The 41st was velero/agent-memory-pvc-backup-20260830040051-b8c4z, absent from the verdict table. A re-read 43 seconds later returned 40 and set-equality held exactly.

04:04:15Z  total volumes 41   control (engines inside v1.12.1) 41   live-not-in-table: velero/agent-memory-pvc-backup-20260830040051-b8c4z
04:04:58Z  total volumes 40   control                          40   live-not-in-table: (none)   table-not-in-live: (none)

Its provenance, read from Velero rather than guessed:

schedule  agent-memory-pvc-backup   0 4 * * *   lastBackup 2026-08-30T04:00:51Z
backup    agent-memory-pvc-backup-20260830040051   Completed   start 04:00:52Z   complete 04:04:52Z
dataupload agent-memory-pvc-backup-20260830040051-b8c4z  Completed  04:04:09Z → 04:04:41Z  sourcePVC=qdrant-data

[VERIFIED: kubectl -n velero get schedules/backups/datauploads.velero.io, 2026-08-30T04:05Z]

This is a standing property of the estate, not a one-off. Velero’s CSI data-mover clones the source PVC into a temporary PVC for the duration of an upload, and a Longhorn volume plus its engine exist for that window — here 04:00:52Z → 04:04:52Z, four minutes. Three Velero schedules and one Longhorn system-backup job produce such windows nightly. A set-equality assertion over volumes.longhorn.io run inside one of them sees a volume with no verdict row, and an engine count one higher than the workload estate has. It is recorded here because it is exactly the class of thing that makes a gate read wrong for a reason that has nothing to do with what the gate is measuring, and because the runbook’s pre-flight now has to say so.

The row is not added to the verdict table. It is not a workload volume, it did not exist during any wave of the drain, it never hosted a pre-v1.12.1 engine, and it no longer exists. Adding a row for an object with a four-minute lifetime would make the table unfalsifiable rather than complete.

LH-01 set-equality, both directions, against a live re-read

=== 2026-08-30T04:04:58Z ===
live volumes.longhorn.io, as ns/pvc      : 40
verdict-table rows, as ns/pvc            : 40
comm -23  live NOT in table              : (empty)
comm -13  table NOT in live              : (empty)
uniq -d   duplicate identifiers in table : (empty)

[VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io -o json | jq ns/pvc | sort, joined against the ## Per-volume verdictstable bycomm, 2026-08-30T04:04:58Z]

Every live volume has a row; every row has a live volume; no identifier appears twice. The table holds 40 data rows against 40 live volumes.

No row is left pending. rg -c 'pending' over the ## Per-volume verdicts table returns 0. Two occurrences of the word survive elsewhere in this file — at the 02-02 and 02-04 reconciliations, both in the sentence “no volume this phase left pending has moved on its own” — and both are narrative describing a past state, not placeholders. They are named here rather than reworded, so that the check stays capable of catching a real placeholder later.

Every n/a cell carries its reason inline. The only n/a cells in the table belong to karakeep/meilisearch-data, and all three read n/a — no restart performed. That volume had already migrated to a v1.12.1 instance-manager unaided before this phase began — it is the single engine the opening gate counted in its control at 2026-08-29T20:41:31Z — so no restart was owed to it and the three restart-derived inputs were never observable. Its verdict is recorded as already-migrated, not as pass: the distinction is that nothing was proven about it by this drain, and defaulting an unread input to pass is the failure this column exists to prevent. No other row carries an n/a, because query (c) returned 0 rows against 40 volumes and the kubelet /stats/summary sweep returned exactly 40 PVC rows across the eight nodes — one per volume, each naming a live owning pod — so no volume was detached or unowned at any audit instant.

The closing reconciliation — what this phase moved, and what moved on its own

The two numbers are kept separate deliberately. The pre-v1.12.1 set is measurably self-draining (CONTEXT records 31 → 30 unaided between 2026-08-24 and 2026-08-29, as karakeep/meilisearch-data migrated with no intervention), so a phase that reported its own progress as the whole movement would be claiming credit for the estate’s background drift.

Wave	Step	Engines moved	(a) after	control after
—	phase start, 2026-08-29T20:41:31Z	—	39	1
02-01	tracer, mealie/mealie-data	1	38	2
02-02	Task 2, tier-1 batch	12	26	14
02-02	Task 3, the three nats members	3	23	17
02-04	Task 2, the three valkey members	3	20	20
02-04	Task 3, the three MongoDB members	6	14	26
02-05	Task 1, the two ClickHouse replicas	2	12	28
02-05	Task 2, the three Keeper members	3	9	31
02-06	Task 1, the two CNPG replicas	2	7	33
02-06	Task 2, the switchover + demoted delete	1	6	34
02-07	Task 2, the three vault pods	6	0	40
02-08	— closing re-read, 04:05:35Z	0	0	40

Engines this phase moved: 39. Engines that migrated on their own between waves: 0.

The second number is a measurement, not an assumption. The reconciliation was re-run at the top of every wave and at the close of the phase — six inter-wave readings (02-01→02-02, 02-02→02-04, 02-04→02-05, 02-05→02-06, 02-06→02-07, 02-07→02-08), spanning 20:41:31Z on 2026-08-29 to 04:05:35Z on 2026-08-30 — and every one returned a drift of zero. Both halves of the gate move by exactly the number of volumes each step restarted, at every step, in opposite directions, which is itself a standing check that the two queries never stopped partitioning the same set.

Zero unaided drift over 7.4 hours is consistent with the 31 → 30 the estate managed over five days rather than in tension with it: spontaneous migration requires an unrelated pod to restart, and one event in five days is about 0.06/hour, so the expected count over this phase’s window is under half an engine. The phase inherited 39 of 40 engines outside v1.12.1 and moved all 39 itself.

Instance-manager mutation control for the wave, and for the phase

The (b) inventory above lists all eight surviving instance-managers with their creation timestamps. The newest is dated 2026-08-24T14:57:45Z–14:58:29Z — five days before the phase began. Zero instance-managers were created by any wave, and not one was deleted or patched by us at any point. Every one of the eleven pre-v1.12.1 objects the phase started with left the inventory on Longhorn’s own initiative, once its last engine had moved out:

Wave	Objects reaped by Longhorn	Deleted by us
02-01 → 02-02	0	0
02-02	0	0
02-04	5 (one shared, then four)	0
02-05	1 (fc5f937c…, reaped after the wave closed)	0
02-06	3	0
02-07	3	0
phase total	11 of 11	0

This wave issued only get verbs against instancemanagers.longhorn.io. The (b) row list is the second and independent LH-03 completion signal, and it reaching 0 rows is an observation about what Longhorn did, never an action we took. On an instance-manager still holding an engine, a delete kills every engine it hosts — in this estate one object held three ClickHouse Keeper engines and a Postgres instance simultaneously, so a single delete would have cost Keeper quorum and a Postgres instance together.

LH-04, closed on an observed firing rather than on a written schedule

02-03 landed the fstrim cron change and deliberately left LH-04 open, because a schedule that has been written down is not yet a schedule that has fired. The proof it deferred has now landed:

=== 2026-08-30T04:08:58Z ===
tf/cluster-bootstrap/longhorn.tf:218            cron = "0 20 * * *"
recurringjobs.longhorn.io/fstrim  .spec.cron  = 0 20 * * *
cronjob/fstrim  .spec.schedule                = 0 20 * * *
cronjob/fstrim  .spec.timeZone                = (unset)
cronjob/fstrim  .status.lastScheduleTime      = 2026-08-30T00:00:00Z
cronjob/fstrim  .status.lastSuccessfulTime    = 2026-08-30T00:02:05Z

[VERIFIED: rg over tf/cluster-bootstrap/longhorn.tf + kubectl -n longhorn-system get recurringjobs.longhorn.io,cronjob fstrim, 2026-08-30T04:08:58Z]

All three layers agree, the job fired on the new hour and completed successfully — a lastSuccessfulTime 125 seconds after the schedule, not merely a lastScheduleTime — and the CronJob still carries no spec.timeZone, which is what makes 0 20 land at 00:00Z: Longhorn’s generated CronJobs are evaluated in the node zone, America/New_York, and EDT is UTC−4.

The resulting nightly order, in UTC, with fstrim now first:

00:00Z  Longhorn fstrim                    (cron 0 20, node-local)   ← the change
01:05Z  Longhorn backup-snapshot-cleanup   (cron 5 21, node-local)
02:00Z  Velero   daily-backup              (cron 0 2,  UTC)
03:00Z  Velero   weekly-full-backup        (cron 0 3 * * 0, UTC)
04:00Z  Longhorn system-backup             (cron 0 0,  node-local)
04:00Z  Velero   agent-memory-pvc-backup   (cron 0 4,  UTC)
07:00Z  Longhorn daily-backup              (cron 0 3,  node-local)

fstrim now precedes every backup in the night — Velero’s included, which the originally decided 0 1 * * * would not have done — and 0 20 local is DST-safe: it lands at 00:00Z under EDT and 01:00Z under EST, ahead of backup-snapshot-cleanup at 01:05Z either way. LH-04 is closed.

Snapshot purge sweep, verbatim

The RESEARCH §6.2 sweep, run unmodified over the whole estate post-drain. All three discriminator parts are required together: (a) the shape — a markRemoved snapshot whose only child is volume-head; (b) both size tests — actualSize more than three times the filesystem’s used bytes and an absolute excess above 1 GiB, with the denominator read per PVC from the kubelet /api/v1/nodes/<node>/proxy/stats/summary endpoint; (c) persistence — the candidate snapshot’s status.creationTime predates the most recent backup run.

The population floor, taken on the same live read

A zero-SUSPECT result means nothing unless the query that produced it can see snapshots at all. A wrong namespace, an RBAC denial on the CRD, or a jq path that has stopped matching all produce output identical to a healthy estate. This is Task 1’s two-sided LH-03 gate applied to the LH-02 sweep, for the same reason.

=== LH-02 sweep, capture start: 2026-08-30T04:06:41Z ===
--- POPULATION FLOOR, same live read as the sweep ---
{
  "population_total_snapshots": 126,
  "population_markremoved": 33,
  "markremoved_leaf_candidates": 30,
  "volumes_covered_by_snapshots": 40
}
--- part (c) reference: Longhorn daily-backup CronJob lastScheduleTime ---
0 3 * * *  lastScheduleTime=2026-08-29T07:00:00Z
--- all four Longhorn recurring CronJobs, for context ---
backup-snapshot-cleanup   5 21 * * *   2026-08-30T01:05:00Z
daily-backup              0 3 * * *    2026-08-29T07:00:00Z
fstrim                    0 20 * * *   2026-08-30T00:00:00Z
system-backup             0 0 * * *    2026-08-30T04:00:00Z
--- kubelet /stats/summary sweep across all 8 nodes ---
PVC rows returned by the kubelet sweep:       40
candidate (markRemoved leaf) snapshots:       30

[VERIFIED: kubectl -n longhorn-system get snapshots.longhorn.io,cronjob + kubelet /api/v1/nodes/<node>/proxy/stats/summary across all 8 nodes, 2026-08-30T04:06:41Z]

Both floors clear, strictly: 126 snapshot rows > 0 and 33 markRemoved > 0, read live at 04:06:41Z, on the same kubectl get snapshots.longhorn.io -o json the sweep consumes. The kubelet half of the join returned 40 PVC rows for 40 volumes, so no candidate was silently dropped for want of a denominator.

Reconciled against the PRIOR figures in RESEARCH §6.3, taken at 2026-08-29T19:10Z:

Figure	RESEARCH, 2026-08-29T19:10Z	live, 2026-08-30T04:06:41Z	delta
total snapshots.longhorn.io rows	118	126	+8
markRemoved leaf candidates	20	30	+10
volumes covered by at least one snapshot	19 (leaf-carrying)	40 (any snapshot)	—

These are drift-prone and are not expected values. The only assertion made against the live population is > 0; neither figure is asserted to equal a research number. The +10 in leaf candidates is explained below and is a timing artifact of when the sweep was run, not a change in the estate’s health.

The sweep output, verbatim

--- SWEEP OUTPUT ---
normal    agent-memory/qdrant-data                                                 snap=system-backup-2e1e5c38a20141 ratio=6.92 excess_MiB=859.5 created=2026-08-30T04:02:07Z
normal    clickstack/clickhouse-storage-volume-cs-clickstack-keeper-keeper-0-0     snap=system-backup-7d2deb941abc4e ratio=2.83 excess_MiB=392.0 created=2026-08-30T04:00:10Z
normal    clickstack/clickhouse-storage-volume-cs-clickstack-keeper-keeper-1-0     snap=system-backup-e4a26774809e42 ratio=2.79 excess_MiB=348.6 created=2026-08-30T04:00:54Z
normal    clickstack/clickhouse-storage-volume-cs-clickstack-keeper-keeper-2-0     snap=system-backup-18b998103f2248 ratio=2.97 excess_MiB=361.2 created=2026-08-30T04:02:03Z
normal    clickstack/data-volume-cs-clickstack-mongodb-0                           snap=system-backup-c439aa6b52e541 ratio=0.99 excess_MiB=-8.3 created=2026-08-30T04:03:25Z
normal    clickstack/data-volume-cs-clickstack-mongodb-1                           snap=system-backup-fb16950dfc004a ratio=0.92 excess_MiB=-49.4 created=2026-08-30T04:00:20Z
normal    clickstack/data-volume-cs-clickstack-mongodb-2                           snap=system-backup-a897eb7fb62e40 ratio=0.97 excess_MiB=-16.7 created=2026-08-30T04:01:01Z
normal    clickstack/logs-volume-cs-clickstack-mongodb-0                           snap=system-backup-2adc9fb8d70a44 ratio=2.40 excess_MiB=98.7 created=2026-08-30T04:03:03Z
normal    clickstack/logs-volume-cs-clickstack-mongodb-1                           snap=system-backup-7587947fe08648 ratio=2.42 excess_MiB=100.0 created=2026-08-30T04:00:24Z
normal    clickstack/logs-volume-cs-clickstack-mongodb-2                           snap=system-backup-3fe7bafb41e04b ratio=2.42 excess_MiB=100.1 created=2026-08-30T04:00:13Z
normal    fovea/fovea-audit-js                                                     snap=system-backup-b855437a330a46 ratio=3.27 excess_MiB=149.2 created=2026-08-30T04:01:27Z
normal    fovea/qdrant-fovea-data                                                  snap=system-backup-122c8d4c613442 ratio=6.48 excess_MiB=828.3 created=2026-08-30T04:00:47Z
normal    headroom-agents/headroom-agents-workspace                                snap=system-backup-0d975ded79044b ratio=1.82 excess_MiB=49.0 created=2026-08-30T04:01:13Z
normal    headroom-apps/headroom-apps-workspace                                    snap=system-backup-bc5d6309ad9748 ratio=1.54 excess_MiB=49.1 created=2026-08-30T04:02:22Z
normal    karakeep/karakeep-data                                                   snap=system-backup-89bfe18b48c641 ratio=1.22 excess_MiB=1060.8 created=2026-08-30T04:02:25Z
normal    karakeep/meilisearch-data                                                snap=system-backup-7b979075abcf4e ratio=6.73 excess_MiB=477.7 created=2026-08-30T04:02:54Z
normal    mealie/mealie-data                                                       snap=system-backup-0050a560676342 ratio=3.27 excess_MiB=228.8 created=2026-08-30T04:00:20Z
normal    mosquitto/mosquitto-data                                                 snap=system-backup-9426789a9d064d ratio=2490.43 excess_MiB=68.1 created=2026-08-30T04:03:23Z
normal    postgres/main-16                                                         snap=system-backup-ea97cbf8174a49 ratio=1.50 excess_MiB=1133.1 created=2026-08-30T04:02:43Z
normal    postgres/main-17                                                         snap=system-backup-bd5d7eac946c48 ratio=1.50 excess_MiB=1132.8 created=2026-08-30T04:00:41Z
normal    postgres/main-20                                                         snap=system-backup-bedf77d3fbde46 ratio=1.49 excess_MiB=1111.5 created=2026-08-30T04:02:47Z
normal    renovate/renovate-ce-db                                                  snap=system-backup-977e7def2b7b4c ratio=4.43 excess_MiB=59.7 created=2026-08-30T04:02:18Z
normal    uptime-kuma/uptime-kuma-data                                             snap=system-backup-1f11df49689748 ratio=4.30 excess_MiB=51.4 created=2026-08-30T04:02:28Z
normal    vault/audit-vault-0                                                      snap=system-backup-1f44196bcd9c43 ratio=9739.50 excess_MiB=228.2 created=2026-08-30T04:01:07Z
normal    vault/audit-vault-1                                                      snap=system-backup-96211d5cbc3d42 ratio=9739.33 excess_MiB=228.2 created=2026-08-30T04:00:29Z
normal    vault/audit-vault-2                                                      snap=system-backup-f2852fd600de44 ratio=9739.50 excess_MiB=228.2 created=2026-08-30T04:03:13Z
normal    vault/data-vault-0                                                       snap=system-backup-5e1a481835524b ratio=5.78 excess_MiB=254.9 created=2026-08-30T04:03:17Z
normal    vault/data-vault-1                                                       snap=system-backup-a34cbbe5723549 ratio=11.89 excess_MiB=451.5 created=2026-08-30T04:00:57Z
normal    vault/data-vault-2                                                       snap=system-backup-b8f28a0e9f0d48 ratio=11.16 excess_MiB=472.0 created=2026-08-30T04:02:30Z
SUSPECT   octopus/qdrant-octopus-data                                              snap=system-backup-20ce98b7d62344 ratio=12.65 excess_MiB=7839.2 created=2026-08-30T04:02:36Z
--- SUSPECT count ---
       1
=== capture end: 2026-08-30T04:06:43Z ===

[VERIFIED: kubectl + kubelet /stats/summary, 2026-08-30T04:06:41Z–04:06:43Z]

Parts (a)+(b) alone return one row. The full three-part discriminator returns zero. The awk in RESEARCH §6.2 implements only (a) and (b); part (c) is evaluated separately, and it is what disposes of the single hit. This is reported in that order deliberately — softening it into “the sweep found nothing” would hide the most useful thing this sweep produced.

The one (a)+(b) hit, resolved on part (c) and then explained

octopus/qdrant-octopus-data clears the shape test and both size bars: ratio 12.65 against the > 3 threshold, excess 7,839 MiB against the > 1 GiB threshold. Part (c) fails, under both candidate reference clocks — which is what makes the disposal robust rather than a choice of denominator:

Reference clock	value	candidate creationTime	predates it?
daily-backup CronJob .status.lastScheduleTime (the reference RESEARCH §6.1 names)	2026-08-29T07:00:00Z	2026-08-30T04:02:36Z	no
system-backup CronJob .status.lastScheduleTime (the job that actually created it)	2026-08-30T04:00:00Z	2026-08-30T04:02:36Z	no

The candidate snapshot was 156 seconds old when the sweep read it.

The volume’s own chain, read directly, says where the 7.65 GiB of excess actually lives:

=== snapshot chain, pvc-e6d6b37e-… = octopus/qdrant-octopus-data, 2026-08-30T04:07:48Z ===
2026-08-23T19:00:04Z  snapshot-fb3c1849-…    markRemoved=false  size=5479MiB  children=snapshot-15e77410-…
2026-08-23T19:05:25Z  snapshot-15e77410-…    markRemoved=false  size=0MiB     children=system-backup-f790fec27ce54349
2026-08-26T04:00:52Z  system-backup-f790fec2 markRemoved=false  size=1231MiB  children=system-backup-bc9fcd91934349c0
2026-08-27T04:03:37Z  system-backup-bc9fcd91 markRemoved=false  size=2MiB     children=system-backup-70ce384b29b445ac
2026-08-29T04:00:55Z  system-backup-70ce384b markRemoved=false  size=1519MiB  children=daily-ba-17f092b8-…
2026-08-29T07:03:03Z  daily-ba-17f092b8-…    markRemoved=false  size=0MiB     children=system-backup-20ce98b7d6234415
2026-08-30T04:02:36Z  system-backup-20ce98b7 markRemoved=TRUE   size=278MiB   children=volume-head

[VERIFIED: kubectl -n longhorn-system get snapshots.longhorn.io, 2026-08-30T04:07:48Z]

The markRemoved snapshot holds 278 MiB of a 7,839 MiB excess — 3.5% of it. The other 96.5% is pinned by a seven-deep chain of live, retained, markRemoved=false snapshots reaching back to 2026-08-23. Nothing is stuck; the volume is simply carrying a week of retained restore points against a 673 MiB working set. Its verdict is normal.

This is a third discriminator failure mode, and RESEARCH had no case for it

RESEARCH §6.1 calibrated the composite against two false positives, one for each bar. This sweep found a third, which clears both bars and is still normal:

Volume	ratio	excess	shape (a)	classification	which bar fails, and why the verdict is normal
vault/audit-vault-0	9739.50	228.2 MiB	leaf present	normal	clears the ratio bar by 3,200×; fails the absolute bar by 4.5×. 228 MiB of retained system-backup chain against a 24 KiB filesystem. A pure-ratio sweep calls this catastrophic.
vault/audit-vault-1	9739.33	228.2 MiB	leaf present	normal	same
vault/audit-vault-2	9739.50	228.2 MiB	leaf present	normal	same
…clickhouse-clickhouse-0-0-0	1.31	39,979.2 MiB	no leaf	normal	39.0 GiB of excess — a pure-absolute-excess sweep calls this catastrophic. Fails on two independent grounds: the ratio bar (1.31 against > 3) and part (a) — it carries no markRemoved leaf at all. One day of ClickHouse part churn against a 128 GiB working set.
…clickhouse-clickhouse-0-1-0	1.23	29,259.9 MiB	no leaf	normal	same, 28.6 GiB
karakeep/karakeep-data	1.22	1,060.9 MiB	leaf present	normal	clears the absolute bar by 37 MiB; fails the ratio bar. A second, small-scale instance of the excess-only shape.
postgres/main-{16,17,20}	1.49–1.50	1,111–1,133 MiB	leaf present	normal	same shape as karakeep: over the absolute bar, well under the ratio bar.
…keeper-keeper-2-0	3.03	371.2 MiB	leaf present	normal	the LH-02 todo’s original volume. Clears the ratio bar; fails the absolute bar by 2.8×. Its two siblings sit at 2.83 and 2.79 in the same shape, so it is a family characteristic, not an outlier.
octopus/qdrant-octopus-data	12.65	7,839.2 MiB	leaf present	normal	clears BOTH bars. Disposed of by part (c) only.

Part (c) is therefore load-bearing, and this sweep is the first measurement that proves it. RESEARCH could argue it; it had no volume that cleared both size bars, so its own data could not distinguish a two-part discriminator from a three-part one. This one can.

The structural reason both size bars can be cleared by a healthy volume: the size tests measure the whole volume’s excess, not the candidate snapshot’s own size. A volume with a deep retained live chain and one fresh markRemoved leaf will clear them however the thresholds are tuned. The cheap additional reading that separates the two — and the one written into the runbook — is to compare the candidate snapshot’s own status.size against the excess it is supposed to explain. Here that ratio is 278 MiB against 7,839 MiB.

The coalescing evidence, re-measured post-drain and stronger than RESEARCH’s

RESEARCH’s positive evidence was that all 20 markRemoved leaves at 19:10Z had been created by that morning’s daily-backup run, so the prior cycle’s set was gone. The same property re-measured here is sharper, because it also rules out the alternative reading — that markRemoved snapshots had simply stopped being created.

=== snapshot population by creation day, and how many of each day are markRemoved ===
day          total   markRemoved
2026-06-02       1             0
2026-06-04       1             0
2026-08-18       4             0
2026-08-23       2             0
2026-08-24       4             0
2026-08-25       8             0
2026-08-26      10             0
2026-08-27       7             0
2026-08-28       9             0
2026-08-29      26             0
2026-08-30      54            33

=== markRemoved LEAF snapshots: creation-time span ===
oldest = 2026-08-30T04:00:10Z   newest = 2026-08-30T04:03:25Z   n = 30
markRemoved leaves older than the 04:00:00Z system-backup run: 0

[VERIFIED: kubectl -n longhorn-system get snapshots.longhorn.io, 2026-08-30T04:11:31Z]

Not one markRemoved snapshot in the estate predates 2026-08-30, and not one markRemoved leaf predates the system-backup run that fired at 04:00:00Z six minutes before the read — while 72 snapshots from 2026-08-29 and earlier survive, with zero markRemoved among them. The survivors prove the query sees old snapshots; their uniform markRemoved=false proves the removal marks that existed on prior cycles have been coalesced away rather than merely never applied. All 30 leaves were created inside a 3 minute 15 second window by one system-backup run.

That also explains the +10 against RESEARCH’s 20: the sweep was run six minutes after a snapshot-producing job, where RESEARCH ran twelve hours after one. A sweep timed just behind a system-backup or daily-backup run sees that run’s entire output as fresh candidates. It is a property of when the sweep is taken, not of the estate.

Purge status, and a correction to which object carries it

=== engines.longhorn.io .status.purgeStatus, 2026-08-30T04:08:40Z ===
total engines                       40
engines carrying a purgeStatus      40
replica-endpoint entries            86   all { state: complete, progress: 100, error: "", isPurging: false }
entries with progress != 100         0
entries with a non-empty error       0

=== replicas.longhorn.io .status.purgeStatus, same instant ===
total replicas                      86
replicas carrying a purgeStatus      0   -- the field is EMPTY on all 86

[VERIFIED: kubectl -n longhorn-system get engines.longhorn.io,replicas.longhorn.io, 2026-08-30T04:08:40Z]

purgeStatus lives on engines.longhorn.io, as a map keyed by replica endpoint — not on replicas.longhorn.io, where it is empty on all 86. RESEARCH §6.3’s “all 86 replicas report purgeStatus complete / 100 / no error” is correct in its figure and reproduces exactly, but the object it names is wrong: the 86 are the engine-side per-replica-endpoint entries. Reading the replica object instead returns an empty field on every row, which is the same trap as .spec.instanceManagerName in Task 1 — a wrong field path returning blanks that a reader can take for an answer. Recorded so the runbook’s query points at the right object.

No volume carries a TooManySnapshots condition. The deepest chain in the estate is 12, on …clickhouse-0-0-0, against a snapshot-max-count of 250 and a warning threshold of 100.

The governing settings, read and not changed

snapshot-max-count                                       250
snapshot-count-warning-threshold                         100
disable-snapshot-purge                                   false
auto-cleanup-system-generated-snapshot                   true
auto-cleanup-recurring-job-backup-snapshot               true
auto-cleanup-snapshot-after-on-demand-backup-completed   true
snapshot-data-integrity                                  {"v1":"fast-check","v2":"fast-check"}
remove-snapshots-during-filesystem-trim                  true

[VERIFIED: kubectl -n longhorn-system get settings.longhorn.io, 2026-08-30T04:07:50Z] — identical to RESEARCH §6.4. No settings.longhorn.io object was modified by this task, and no snapshot was created, marked removed or deleted. Every verb in this task is get.

Three mid-chain markRemoved snapshots — this phase’s own residue, named rather than excluded

Three of the 33 markRemoved snapshots are not leaves — their child is another snapshot, not volume-head — so they fail part (a) and never enter the sweep. They are named here rather than silently filtered out, because all three are large and all three are this drain’s own doing:

Volume	snapshot	created	own size	child	age at read
renovate/renovate-ce-cache-hi	09f79553-…	2026-08-30T00:10:55Z	21,880 MiB	e9cdd19e-…	4 h 0 m
…clickhouse-clickhouse-0-0-0	de4ca86a-…	2026-08-30T00:42:55Z	142,728 MiB	50aab241-…	3 h 28 m
…clickhouse-clickhouse-0-1-0	8df0ead3-…	2026-08-30T01:51:58Z	139,868 MiB	system-backup-71053f81…	2 h 19 m

[VERIFIED: kubectl -n longhorn-system get snapshots.longhorn.io at 2026-08-30T04:07:48Z and again at 04:11:31Z — all three present and unchanged across both reads]

Their creation times fall inside 02-05’s and 02-06’s restart windows: they are the pre-restart system snapshots the detach/reattach superseded. They fail part (a) and they also fail part (c) — all three postdate the daily-backup reference at 2026-08-29T07:00:00Z — so under the prescribed discriminator they are normal. The estate-wide evidence above supports that: no markRemoved snapshot survives a cycle boundary anywhere in the estate.

They are nonetheless carried to ## Open question as a named re-check rather than closed here. Two of them are ~140 GiB coalescing jobs queued on single-replica volumes, they are the largest such jobs the estate has ever had pending, and this phase created them. A cheap confirming read after the next backup-snapshot-cleanup cycle settles it; asserting now that they will coalesce would be a prediction dressed as a measurement.

LH-02, closed on the measurement

LH-02 is closed as understood, not reproducible at HEAD — the outcome the requirement explicitly permits. The closure rests on four things, not on a null result:

1. A measured zero, against a proven-non-empty population. The full three-part discriminator returns 0 SUSPECT across all 40 volumes, on a read that simultaneously returned 126 snapshot rows and 33 markRemoved rows, both strictly greater than zero, and a kubelet join that returned 40 denominators for 40 volumes.
2. A written-down discriminator, now standing knowledge in docs/operations/longhorn.md → ### Recognising a stuck snapshot purge: the markRemoved leaf shape is the normal post-backup state; the stuck signature is that shape plus both size tests plus persistence across a purge cycle; and, added from this sweep, the candidate snapshot’s own size must account for the excess, because the size tests measure the volume and not the snapshot.
3. A written-down workaround, also in the runbook: Longhorn’s own purge-snapshots action, then a volume detach and reattach. This phase performed that workaround across the entire estate — 39 of 40 volumes were detached and reattached during the drain — which is the LH-02 todo’s own untried remedy, exercised estate-wide as a side effect of work already being done.
4. Positive evidence that the purge path coalesces, re-measured post-drain: zero markRemoved snapshots older than one cycle, against 72 surviving older snapshots that carry none.

The volume the requirement was written against, …keeper-keeper-2-0 (pvc-316983f6…), reads actualSize 554 MiB against a 183 MiB filesystem — ratio 3.03, excess 371 MiB — down from the 7.8 GB in the original todo, and snapshot-7ea77aa1-… no longer exists.

Interpretation

LH-03 — settled. Every Longhorn engine process in this estate runs in a v1.12.1 instance-manager. The gate reads 0 outside v1.12.1 with its inverted control at 40, the two halves summing exactly to the live engine count, and the second and independent signal — the count of pre-v1.12.1 instancemanagers.longhorn.io rows — also reads 0. All eleven stale objects the phase started with were reaped by Longhorn on its own once emptied; not one was deleted or patched by us at any point, in any wave. The drain moved 39 engine processes across seven waves, and 0 migrated unaided in between, measured at six inter-wave reconciliations spanning 7.4 hours.

The mechanism the phase confirmed, and which the runbook now carries: an engine binary converges on its own after an upgrade, but an engine process moves only when its volume detaches and reattaches. Nothing about an upgrade detaches a volume, so a continuously-attached volume keeps its engine in a pre-upgrade instance-manager indefinitely while every health field in Longhorn reads current. That is why an estate can be entirely healthy and entirely stale at the same time, and why this was invisible until someone counted.

LH-01 — settled, with its boundary stated. All 40 volumes in the live estate carry a resolved verdict row, set-equal to a live re-read in both directions with no duplicate. Thirty-nine volumes were detached and reattached during the drain and every one came back attached, healthy, on a v1.12.1 engine image, with a read-write filesystem and a ready owning workload. The fortieth, karakeep/meilisearch-data, had migrated unaided before the phase began; its restart-derived inputs are recorded n/a — no restart performed and its verdict is already-migrated, not pass, because nothing was proven about it by this drain.

Not one volume showed data loss attributable to the 2026-07-27 tpi-beta-4 incident. Two volumes carried the kernel’s own EXT4-fs … potential data loss flag from that night — nats/nats-js-nats-1 explicitly, on inode 524325 under comm nats-server — and both were read end-to-end after their restart with NATS’ own JetStream recovery output as the workload half of the verdict; both recovered clean.

The rescue fields were read as a delta across each restart, not as an emptiness test, and that correction is itself a phase output. RESEARCH §1.2 recorded remountRequestedAt as set on 0 of 40; it is in fact set on 19 of 40, with values predating this phase by months, so a criterion phrased as “the field is empty” would have failed 19 healthy volumes on arrival and a criterion phrased as “the field is unchanged” is the one that means anything. Measured that way, no volume acquired a new or advanced remountRequestedAt across any restart in the phase. lastAutoSalvagedAt genuinely is set on 0 of 40, before and after. [VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io, 2026-08-30T04:13Z]

LH-02 — closed as understood, not reproducible at HEAD. The three-part discriminator returns 0 stuck volumes across the estate, on a read that simultaneously returned 126 snapshot rows and 33 markRemoved rows and joined 40 kubelet filesystem denominators to 40 volumes — so the zero is a measurement, not an empty query. Parts (a) and (b) alone returned one row, and disposing of it produced the most useful finding of the sweep: a volume can clear both size bars and still be entirely healthy, because the size bars measure the volume’s excess while the space is pinned by a live retained snapshot chain. Part (c), persistence, is what discriminates, and this is the first measurement in the phase that proves it — RESEARCH had no case that cleared both bars. The purge path demonstrably coalesces: not one markRemoved snapshot in the estate predates 2026-08-30, while 72 snapshots from earlier days survive carrying none between them.

The requirement’s own untried workaround — detach and reattach — was exercised across 39 of 40 volumes as a side effect of the drain, so if the original symptom were still live at HEAD this phase would have been the occasion to see it.

LH-04 — settled, on a firing rather than on a schedule. fstrim is codified, live in the RecurringJob, live in the derived CronJob, and it fired and completed at 2026-08-30T00:00:00Z / 00:02:05Z. That is 20:00 in the node zone, which puts the trim ahead of every backup in the night — Velero’s included, which the originally-decided 0 1 * * * would not have achieved — and it holds across the DST boundary.

What the phase established about method, beyond its four requirements. Six of the seven waves found a gate that read green because the instrument was blind, and the corrections are the durable output:

• A zero is only evidence if the query can produce a non-zero. Both the LH-03 gate and the LH-02 sweep are two-sided here for that reason, and in both cases the control moved.
• A generic severity matcher needs a positive control per application before any zero is believed. The prescribed matcher failed in six distinct ways across six workload families — a CSS colour token containing the word, a bracketed format it could not see at all, an over-sensitive one, a query that matched its own audit echo, a schema field name, and a level tag carried on non-error lines. Every one produced a plausible number.
• A steady-state baseline structurally cannot contain a startup, so a before/after window comparison needs a third control drawn from prior pod incarnations.
• A readiness gate keyed on a StatefulSet pod’s name is vacuous; key it on .metadata.uid.
• A cordon is not a local action. Two seconds of one node unschedulable started a 138 GiB rebuild on a single-replica volume in an unrelated namespace, and moved a production write primary twice.
• Read the populated field. .spec.instanceManagerName is empty on every engine and purgeStatus is empty on every replica; both return blanks that read as clean answers.

Open question

What this record does not settle. The dm-1 / vector attribution 02-01 left here is no longer among them — 02-02 verified it, and the entry is discharged.

1. The chronic nats invalid-client-ID error from the mosquitto bridge — a pre-existing defect

=== nats namespace, full 14-day retained window, to 2026-08-29T22:13Z ===
nats-0  n=16631  first=2026-08-15 22:16:27  last=2026-08-29 22:12:59
nats-1  n=16633  first=2026-08-15 22:16:04  last=2026-08-29 22:13:24
nats-2  n=16688  first=2026-08-15 22:14:27  last=2026-08-29 22:10:56

sample: [ERR] 10.245.6.189:35540 - mid:20935 - "mosquitto-….nats-bridge"
        - invalid character in client ID "mosquitto-….nats-bridge"

[VERIFIED: default.otel_logs, 2026-08-29T22:13Z]

The mosquitto bridge connects to NATS with a client ID containing a ., which NATS rejects. It repeats every few seconds on all three members, ~16,600 lines per pod, and first equals the start of the retained window — so it predates the window and therefore predates this phase entirely.

It is classified as a pre-existing defect and excluded from every nats verdict cell, by name, in the NATS-aware matcher. It is not a drain failure, and counting it as one would have failed all three nats rows on arrival. Two independent facts support the classification: it long predates the drain, and its rate is unchanged across it (nats-0 23 → 22, nats-1 21 → 15, nats-2 27 → 26 over matched before/after windows). It deserves its own todo — the fix is on the mosquitto bridge’s client-ID configuration, not on NATS or on storage — and it is filed here rather than fixed, because it is outside this phase’s four requirements.

2. renovate-ce’s fsGroup chown makes every restart a ~30-minute outage

Recorded in full under the Task 2 findings above. 1.57 M files chowned on every pod start because the pod spec sets an fsGroup without fsGroupChangePolicy: OnRootMismatch. Pre-existing, not incident residue, and a candidate todo.

3. vector pins LUKS devices open on every node, stalling encrypted-volume detach

Verified above, and it will recur in waves 02-04 through 02-07. The remedy is known and cheap (delete the vector pod on the source node), but the underlying interaction — a log-collection DaemonSet’s /var/lib hostPath holding the kubelet’s CSI mounts — is worth its own todo, because it makes every encrypted-volume migration in this cluster conditionally slow.

4. The two largest volumes in the estate are single-replica

Found by 02-05 while deriving its drain technique, and out of that wave’s scope to change. clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-0-0 (142.21 GiB) and …-0-1-0 (137.98 GiB) both carry spec.numberOfReplicas: 1. Every other volume in the estate carries 2 or 3. The global default-replica-count is {"v1":"2","v2":"2"}, so these two are below the cluster default rather than following it. [VERIFIED: kubectl -n longhorn-system get volumes.longhorn.io -o json, 2026-08-30T00:31:50Z]

Two consequences, both measured rather than argued:

1. No storage redundancy on the ingest path. A single lost replica takes the volume with it. ClickHouse’s own two-replica ReplicatedMergeTree layer means the data survives at the application level, so this is a degraded-not-lost condition — but it is the only place in the estate where Longhorn provides no redundancy at all.
2. It is what made this wave’s drain awkward. With one replica pinned to the attached node, best-effort data locality turns any node move into a ~140 GiB network rebuild. That is why 02-05 had to derive the move-away-then-home two-step and why it is the only wave in the phase that needed it.

Not changed here: raising numberOfReplicas would itself schedule two 140 GiB rebuilds, which is a deliberate capacity and IO decision for the operator and not a drain side effect. Whether it is intentional (a chart default accepted for a replicated application) or drift is unmeasured. Owed to whoever revisits ClickStack capacity; not owed to 02-08, which closes the drain runbook.

5. The log evidence in this record could only ever be taken FORWARD, never backward at the incident

default.otel_logs has a fourteen-day TTL, so the 2026-07-27 tpi-beta-4 incident is outside the queryable window and always was:

TTL toDateTime(Timestamp) + toIntervalDay(14)
oldest row : 2026-08-16 00:00:00
newest row : 2026-08-30 04:19:31
rows in 2026-07-26 .. 2026-07-29 : 0

[VERIFIED: default.otel_logs via cs-clickstack-clickhouse-clickhouse-0-0-0, 2026-08-30T04:19Z]

This is the honest boundary of what LH-01 could answer, and it belongs on the record rather than buried in a method note. Every log-half verdict in this file is a before/after delta across this phase’s own restart, taken in August 2026. None of it is a reading of the incident. What the verdicts establish is that each volume’s data reads back clean now, after being exercised end-to-end — which is the strongest statement available, and is not the same statement as “nothing was lost on 2026-07-27”.

Two things partly compensate and are recorded as such. The kernel’s own EXT4-fs … potential data loss flags from that night survive in the console ring capture (docs/operations/evidence/2026-08-23-tpi-beta-4-console-ring.md), so the set of affected devices is known even though the logs are not. And every volume carries a same-day Longhorn backup, so a restore point exists independently of what any log says.

Anyone wanting incident-window log evidence for a future event needs the retention raised beforehand; it cannot be recovered afterwards. Not changed here — a TTL increase on a 570-million-row table is a storage capacity decision, not a drain step.

6. Three large mid-chain markRemoved snapshots, this drain’s own residue

Named in ## Snapshot purge sweep, verbatim above. Two of them are ~140 GiB coalescing jobs queued on single-replica volumes, created inside 02-05’s and 02-06’s restart windows, still present and unchanged across two reads four minutes apart. They fail parts (a) and (c) of the discriminator, and the estate-wide evidence says no markRemoved snapshot survives a cycle boundary anywhere — so the expectation is that they coalesce on their own.

That expectation is a prediction, not a measurement, and they are the largest such jobs this estate has ever had pending. A confirming read after the next backup-snapshot-cleanup cycle settles it, and it is one query:

kubectl -n longhorn-system get snapshots.longhorn.io -o json | jq -r '
  [.items[] | select((.status.markRemoved // false) == true)
            | {t: .status.creationTime, n: .metadata.name, v: .spec.volume}]'

Any row dated earlier than the most recent snapshot-producing run is worth the full three-part discriminator from docs/operations/longhorn.md → ### Recognising a stuck snapshot purge.

7. Whatever periodically seals vault-0 is unknown — the phase found the mechanism, not the cause

vault-0 carried 825 container restarts into 02-07, up from 819 thirteen days earlier. That wave established the mechanism precisely: a sealed Vault fails the liveness probe, which carries no sealedcode, and the kubelet kills it on a initialDelaySeconds: 120 plus two periodSeconds: 5 cycle — reproduced to the second on a three-minute-old pod. The restarts are a symptom of sealing, not an independent crash.

What seals it in the first place is unresolved. It is not this drain’s to answer and no wave attempted it; the restart count is a durable signal that something recurring is happening to that member. This is the phase’s largest genuinely open item.

8. Two estate pods are unhealthy, and neither is this phase’s

Both predate the phase and both are logged in the phase’s deferred-items.md rather than fixed: agent-memory/engram-spine-scan-… failed on 2026-08-23, a week before the phase began, and temporal/temporal-worker-controller-manager-… has been Pending since 2026-05-12. Neither is a storage fault and neither was touched.

Discharged, and named so a reader looking for it finds the disposition

The console ring’s dm-1 … comm vector attribution — which 02-RESEARCH.md and 02-01 both carried as [ASSUMED], and which this record was expected to hand forward unresolved — was verified in 02-02 and is closed. It is recorded above under The vector hypothesis, upgraded from [ASSUMED] to verified: vector‘s mount namespace was caught holding the kubelet’s CSI mount entries for two LUKS devices belonging to other workloads, with Open count: 1 on each, NONE MOUNTED IN HOST NS, and the sole holder identified by a /proc/*/mounts scan across every namespace on the node. It reproduced on a second node, and one vector pod delete released it both times. vector does genuinely touch other workloads’ Longhorn volume mounts; there is still no Longhorn volume owned by vector, so there is nothing to audit under that name. The mechanism is now a standing entry in docs/operations/longhorn.md → ### Drain order.