Skip to content
fzymgc-house Cluster Docs

ClickStack scalable rebuild: implementation plan

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Rebuild ClickStack onto 1 shard x 2 ReplicatedMergeTree replicas (tables-only greenfield) with a hardened collector/Vector pipeline, per-signal retention, and RustFS-native backups.

Architecture: Three sequential GitOps PRs (values-only hardening → chart 3.0.1 + topology → cutover Job/guard/backup-exclusion), then a minutes-long cutover window that drops and re-creates the otel tables as Replicated via an idempotent PreSync bootstrap Job. MongoDB (dashboards/alerts) and all URLs survive untouched. Telemetry history is disposable by decision.

Tech Stack: ArgoCD (kustomize helmCharts inflation, --enable-helm), ClickStack helm chart 3.0.1 (HyperDX/collector 2.29.0), official ClickHouse operator (ClickHouseCluster/KeeperCluster, clickhouse.com/v1alpha1), ClickHouse 25.7, Longhorn, Vector, Velero, jj VCS, bd/beads.

Spec: docs/engineering/specs/2026-07-05-clickstack-scalable-rebuild-design.md (design bead hl-uj69, design-review READY round 5).

Model intent (for plan-to-beads, Rule 5): manifest/values authoring and scaffolding (Tasks 1-5, 8-10, 12-13, 17 Steps 2-4, 18 Step 2, 19) → model:sonnet. Merge-and-live-validate GATEs and destructive/irreversible operations — Task 6 (PR1 merge + effective-config/settings/throttle validation), Task 7 (orphaned-PVC deletion), Task 11 (Keeper quorum change validation), Task 14 (PreSync ordering + first-creation rehearsal), Task 15 (cutover window, point of telemetry no-return), Task 16 (validation suite incl. pod-kill), Task 17 Steps 1+5 (Vault/RustFS prep, restore rehearsal), Task 18 Step 3 (merge + watchdog kill-test) → model:opus, operator-driven. Task 18 Step 1 (uptime-kuma TF) → model:sonnet via the terraform skill.

Global conventions for every task:

  • Work in a jj workspace (NOT the default workspace — another session may own it): all VCS mutations from inside .worktrees/<name> need the Bash sandbox disabled (jj store lock lives in the main repo’s .jj/).
  • Commit via the commit-commands:commit skill. Push bookmark + gh pr create; merges happen via PR (branch protection).
  • Lint gates before every commit: yamllint -c .yamllint.yaml <changed yaml>; rumdl check <changed md>.
  • The clickstack Application is auto-synced: merge = rollout. Tasks marked GATE must fully pass before the next task starts.
  • ClickHouse queries for validation run via the clickhouse-ro MCP (mcp__clickhouse-ro__run_query) unless the query needs a specific pod — then kubectl exec read-only is acceptable.

Stage A — PR1: values-only pipeline hardening

Section titled “Stage A — PR1: values-only pipeline hardening”

Task 1: Workspace and bookmark for PR1

Section titled “Task 1: Workspace and bookmark for PR1”

Files: none (VCS setup)

  • Step 1: Create the workspace and bookmark
Terminal window
cd /Volumes/Code/github.com/fzymgc-house/selfhosted-cluster
jj --no-pager git fetch
jj workspace add .worktrees/clickstack-pr1 --name clickstack-pr1
cd .worktrees/clickstack-pr1
jj --no-pager new main@origin        # sandbox-disabled

Expected: Working copy (@) now at: <id> (empty) parented on current main.

Task 2: Custom collector config ConfigMap

Section titled “Task 2: Custom collector config ConfigMap”

Files:

  • Create: argocd/app-configs/clickstack/otel-collector-custom-config.yaml
  • Modify: argocd/app-configs/clickstack/kustomization.yaml (resources list, currently 9 entries ending otel-loadbalancer-service.yaml)
  • Step 1: Write the ConfigMap

Create argocd/app-configs/clickstack/otel-collector-custom-config.yaml:

---
# Locally-merged collector config (hl-uj69). The OpAMP supervisor
# composes: image bootstrap -> THIS FILE -> HyperDX remote config
# (koanf key-level merge; remote wins conflicting leaves, but
# service.extensions lists CONCATENATE across sources). Keys here are
# leaf-level ADDITIONS the remote config never sets, so they survive
# on collector 2.27 and 2.28+ alike.
#
# - file_storage/queue + sending_queue: disk-backed bounded export
#   queue — a ClickHouse write outage buffers to disk instead of
#   ballooning heap until OOM (2026-07-05 incident, 188 restarts).
# - async_insert false: async inserts bypass ReplicatedMergeTree's
#   idempotent block dedup; disabling restores non-duplicating
#   retry-after-timeout (spec Architecture depends on this).
# - batch.send_batch_max_size: caps single-request memory. No
#   HYPERDX_OTEL_BATCH_* env vars exist upstream — this MUST live
#   here, not in env.
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-custom-config
  namespace: clickstack
data:
  custom.config.yaml: |
    extensions:
      file_storage/queue:
        directory: /var/lib/otelcol/queue
        create_directory: true
    exporters:
      clickhouse:
        async_insert: false
        sending_queue:
          enabled: true
          sizer: items
          queue_size: 50000
          block_on_overflow: true
          storage: file_storage/queue
      clickhouse/rrweb:
        async_insert: false
        sending_queue:
          enabled: true
          sizer: items
          queue_size: 50000
          block_on_overflow: true
          storage: file_storage/queue
    processors:
      batch:
        send_batch_max_size: 10000
    service:
      extensions:
        - file_storage/queue
  • Step 2: Register it in the kustomization

In argocd/app-configs/clickstack/kustomization.yaml, append to resources::

  - otel-collector-custom-config.yaml
  • Step 3: Validate the kustomization builds
Terminal window
kubectl kustomize argocd/app-configs/clickstack | yq 'select(.kind=="ConfigMap" and .metadata.name=="otel-collector-custom-config") | .metadata.name'

Expected: otel-collector-custom-config.

  • Step 4: Lint + commit
Terminal window
yamllint -c .yamllint.yaml argocd/app-configs/clickstack/

Commit (commit skill): feat(clickstack): add locally-merged collector config — disk queue, async_insert off, batch cap [hl-uj69]

Task 3: Collector values — 2Gi, config plumbing, fluentd off

Section titled “Task 3: Collector values — 2Gi, config plumbing, fluentd off”

Files:

  • Modify: argocd/app-configs/clickstack-chart/values.yaml:188-199 (the otel-collector: block)

  • Step 1: Capture the render baseline (required by argocd/CLAUDE.md)

Terminal window
kubectl kustomize --enable-helm argocd/app-configs/clickstack-chart > /tmp/render-before.yaml
  • Step 2: Replace the otel-collector block

Replace lines 188-199 (keep the existing #219 comment block above at 175-187 unchanged — the image pin is dropped in PR2, not here) with:

otel-collector:
  enabled: true
  image:
    repository: docker.clickhouse.com/clickhouse/clickstack-otel-collector
    tag: "2.27.0"
  # 2Gi limit (hl-uj69): the image's baked-in memory_limiter
  # (limit_mib 1500, spike 512) is sized for a 2Gi container — on the
  # prior 1Gi limit its soft threshold sat ABOVE the cgroup OOM point
  # and could never engage (2026-07-05 incident: OOMKilled x188).
  # GOMEMLIMIT tracks the limit automatically (chart useGOMEMLIMIT).
  resources:
    requests:
      cpu: 200m
      memory: 1Gi
    limits:
      cpu: "1"
      memory: 2Gi
  # Locally-merged collector config: see the otel-collector-custom-config
  # ConfigMap (argocd/app-configs/clickstack/) for what and why.
  extraEnvs:
    - name: CUSTOM_OTELCOL_CONFIG_FILE
      value: /etc/otelcol-custom/custom.config.yaml
  extraVolumes:
    - name: otel-custom-config
      configMap:
        name: otel-collector-custom-config
    - name: otel-queue
      emptyDir:
        sizeLimit: 2Gi
  extraVolumeMounts:
    - name: otel-custom-config
      mountPath: /etc/otelcol-custom
      readOnly: true
    - name: otel-queue
      mountPath: /var/lib/otelcol/queue
  # fluentforward (24225) has no consumer in this cluster and is
  # unauthenticated upstream — close the ingest surface.
  ports:
    fluentd:
      enabled: false
  • Step 3: Render-diff — expect ONLY the collector Deployment delta
Terminal window
kubectl kustomize --enable-helm argocd/app-configs/clickstack-chart > /tmp/render-after.yaml
diff /tmp/render-before.yaml /tmp/render-after.yaml

Expected: changes confined to the cs-otel-collector Deployment (resources, env, volumes, volumeMounts) and its Service (fluentd port removed). Any ClickHouseCluster or HyperDX delta = wrong values path — stop and fix.

  • Step 4: Lint + commit

Commit: fix(clickstack): collector 2Gi + custom-config plumbing + fluentd off [hl-uj69]

Task 4: ClickHouse values — 8Gi, app-user cap, system-log TTLs

Section titled “Task 4: ClickHouse values — 8Gi, app-user cap, system-log TTLs”

Files:

  • Modify: argocd/app-configs/clickstack-chart/values.yaml:131-137 (containerTemplate resources) and the cluster.spec block (add settings: after shards: 1, near line 112)

  • Step 1: Update containerTemplate resources

Replace the resources: block under containerTemplate (currently requests 2/2Gi, limits 4/4Gi) with:

        resources:
          requests:
            cpu: "2"
            memory: 4Gi
          limits:
            cpu: "4"
            memory: 8Gi

Update the sizing comment above it: append a line # Memory 4Gi->8Gi (hl-uj69): 7d dashboard charts peak ~1.7GiB and died against the 3.6GiB server cap (0.9 x 4Gi) shared with ingest.

  • Step 2: Add the settings block (FULL PATH clickhouse.cluster.spec.settings.* — this chart silently drops values under wrong paths)

Insert directly after shards: 1 (same indent level as containerTemplate:):

      # settings merges with chart defaults (enableDatabaseSync,
      # extraConfig, extraUsersConfig users app/otelcollector). Helm
      # deep-merges maps, so we only state our deltas.
      settings:
        extraConfig:
          # Self-log TTLs (hl-uj69): system.text_log had grown to
          # 25.9GiB with NO TTL (40% of the disk-full incident). On a
          # definition change ClickHouse renames the old system log
          # table aside and creates a fresh one with the new TTL at
          # startup (extraConfig changes restart the pod); the renamed
          # _N tables can be dropped manually to reclaim space at once.
          text_log:
            ttl: "event_date + INTERVAL 7 DAY DELETE"
          query_log:
            ttl: "event_date + INTERVAL 7 DAY DELETE"
          part_log:
            ttl: "event_date + INTERVAL 7 DAY DELETE"
          trace_log:
            ttl: "event_date + INTERVAL 7 DAY DELETE"
          metric_log:
            ttl: "event_date + INTERVAL 7 DAY DELETE"
        extraUsersConfig:
          # Cap the HyperDX read path (hl-uj69): a runaway dashboard
          # query spills/fails ALONE at 2GiB instead of exhausting the
          # server-wide cap shared with ingest. ratio settings enable
          # spill-to-disk before the cap.
          profiles:
            hyperdx_reader:
              max_memory_usage: 2147483648
              max_bytes_ratio_before_external_group_by: 0.5
              max_bytes_ratio_before_external_sort: 0.5
          users:
            app:
              profile: hyperdx_reader
  • Step 3: Render-diff — expect ONLY the ClickHouseCluster CR delta
Terminal window
kubectl kustomize --enable-helm argocd/app-configs/clickstack-chart > /tmp/render-after2.yaml
diff /tmp/render-after.yaml /tmp/render-after2.yaml

Expected: ClickHouseCluster/cs-clickstack-clickhouse only — containerTemplate.resources and settings.extraConfig/settings.extraUsersConfig additions. CRITICAL check: the rendered CR must still contain the chart-default users.app.password and users.otelcollector entries (helm deep-merge preserved them). If users.app lost its password/grants, the merge clobbered the map — stop; move our profile assignment into a strategic-merge patch in kustomization.yaml instead.

  • Step 4: Lint + commit

Commit: fix(clickstack): CH 8Gi + hyperdx_reader 2GiB cap + system.* log TTLs [hl-uj69]

Task 5: Vector throttle — break the self-amplification loop

Section titled “Task 5: Vector throttle — break the self-amplification loop”

Files:

  • Modify: argocd/app-configs/monitoring-vector/vector.yaml:35-44 (transforms; insert throttle between source k8s_logs and remap k8s_logs_for_clickhouse)

  • Step 1: Confirm the only consumer of k8s_logs is the remap

Terminal window
rg -n 'inputs:' -A 2 argocd/app-configs/monitoring-vector/vector.yaml

Expected: exactly one inputs: [k8s_logs] (the k8s_logs_for_clickhouse remap) and one inputs: [k8s_logs_for_clickhouse] (the clickhouse_logs sink). If another consumer of k8s_logs exists, wire it through the throttle too.

  • Step 2: Insert the throttle transform

At the top of transforms: (before k8s_logs_for_clickhouse):

  # Per-container rate cap (hl-uj69): during the 2026-07-05 incident
  # clickhouse-server logged a ~20-frame stack trace PER failed insert;
  # this DS shipped that flood straight back into ClickHouse
  # (self-amplification). 6000 events / 60s (~100/s avg) per
  # namespace/container; excess is DROPPED (vector_events_discarded_total
  # tracks it). Legit workloads peak well under this.
  k8s_logs_throttled:
    type: throttle
    inputs:
      - k8s_logs
    key_field: "{{ kubernetes.pod_namespace }}/{{ kubernetes.container_name }}"
    threshold: 6000
    window_secs: 60

Then change the remap’s input:

  k8s_logs_for_clickhouse:
    type: remap
    inputs:
      - k8s_logs_throttled
  • Step 3: Lint + commit
Terminal window
yamllint -c .yamllint.yaml argocd/app-configs/monitoring-vector/vector.yaml

Commit: fix(vector): per-container throttle on k8s logs — break CH error-flood loop [hl-uj69]

Task 6: PR1 — open, merge, validate live (GATE)

Section titled “Task 6: PR1 — open, merge, validate live (GATE)”

Files: none (process)

  • Step 1: Push + PR
Terminal window
jj --no-pager bookmark create fix/clickstack-pipeline-hardening -r @-   # sandbox-disabled
jj git push -b fix/clickstack-pipeline-hardening                        # sandbox-disabled
gh pr create --repo fzymgc-house/selfhosted-cluster \
  --head fix/clickstack-pipeline-hardening --base main \
  --title "fix(clickstack): pipeline hardening — collector 2Gi+disk queue, CH 8Gi+read cap+system TTLs, Vector throttle [hl-uj69]" \
  --body "PR1 of 3 (spec: docs/engineering/specs/2026-07-05-clickstack-scalable-rebuild-design.md). Values-only; render-diff verified per argocd/CLAUDE.md."

USER GATE: request merge approval. After merge, ArgoCD auto-syncs (collector + CH pods restart).

  • Step 2: Validate collector effective config (spec Validation item 1)

Wait for cs-otel-collector pod Ready, then:

Terminal window
kubectl -n clickstack exec deploy/cs-otel-collector -- sh -c \
  'ls /var/lib/otelcol/queue && cat /etc/otelcol-custom/custom.config.yaml | head -5'

Expected: queue directory exists; config mounted. Then in the HyperDX UI → Team Settings → Collector (OpAMP effective config): confirm sending_queue (with storage: file_storage/queue), async_insert: false, send_batch_max_size: 10000 present in the effective config. If the keys are absent, the remote config clobbered them — capture the effective config to the bead and STOP (spec risk row “chart remote-config interaction”).

  • Step 3: Validate CH settings took effect
-- via clickhouse-ro MCP
SELECT name, value FROM system.settings WHERE name = 'max_memory_usage'
  SETTINGS use_query_cache = 0;  -- returns session default (unchanged)

Then verify the app user cap and TTLs:

SELECT value FROM system.settings_profile_elements WHERE profile_name = 'hyperdx_reader' AND setting_name = 'max_memory_usage';
-- expect: 2147483648
SELECT engine_full LIKE '%TTL%' FROM system.tables WHERE database='system' AND name='text_log';
-- expect: 1 (after the pod restart applied the config)
  • Step 4: Validate Vector throttle is live
Terminal window
kubectl -n vector logs daemonset/vector --tail 50 | rg -i 'throttle|error' | head -5

Expected: no config errors. Record on bead: bd note hl-uj69 "PR1 merged + validated: <PR url>".

Stage B — PR2: chart 3.0.1 + topology

Section titled “Stage B — PR2: chart 3.0.1 + topology”

Task 7: Delete the four orphaned 100Gi PVCs (operational prerequisite)

Section titled “Task 7: Delete the four orphaned 100Gi PVCs (operational prerequisite)”

Files: none (live-cluster operation; these PVCs are operator relics with no owning StatefulSet — NOT ArgoCD-managed, so this is not GitOps drift)

  • Step 1: Confirm they are orphans (no pods, no StatefulSets)
Terminal window
kubectl -n clickstack get pvc | rg 'clickhouse-storage-volume'
kubectl -n clickstack get sts | rg 'clickhouse'
kubectl -n clickstack describe pvc clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-1-0 | rg 'Used By'

Expected: 5 PVCs exist; only ...-0-0-0 has a StatefulSet/pod; Used By: <none> for 0-1-0, 0-2-0, 1-0-0, 2-0-0.

  • Step 2: USER GATE — confirm deletion, then delete

This is destructive (deletes the 2026-05 experiment volumes, ~400Gi Longhorn allocation). With explicit user approval:

Terminal window
kubectl -n clickstack delete pvc \
  clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-1-0 \
  clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-2-0 \
  clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-1-0-0 \
  clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-2-0-0
kubectl -n clickstack get pvc | rg 'clickhouse-storage-volume'

Expected: exactly one PVC remains (...-0-0-0, 200Gi). WHY THIS MUST PRECEDE PR2: the operator derives PVC names from shard/replica indices — the new replica would silently adopt the stale 0-1-0 volume (wrong StorageClass, old data).

Task 8: PR2 workspace + chart bump to 3.0.1

Section titled “Task 8: PR2 workspace + chart bump to 3.0.1”

Files:

  • Modify: argocd/app-configs/clickstack-chart/kustomization.yaml:16 (version: 3.0.0)

  • Modify: argocd/app-configs/clickstack-chart/values.yaml:175-192 (remove the image-pin comment block + image: override)

  • Step 1: New workspace clickstack-pr2 off updated main (same commands as Task 1, name clickstack-pr2)

  • Step 2: Capture render baseline (same as Task 3 Step 1)

  • Step 3: Bump the chart pin

In kustomization.yaml: version: 3.0.0 → version: 3.0.1.

  • Step 4: Drop the collector image pin

In values.yaml, delete the comment block at lines 175-187 (the #219 explanation) and the image: override under otel-collector: — chart 3.0.1 defaults the collector tag to 2.29.0 matching appVersion (verified via helm show values, 2026-07-05). Replace the deleted comment with:

# Chart 3.0.1 aligns the default collector tag (2.29.0) with
# appVersion — the #219 image pin is no longer needed.
  • Step 5: Capture the seed-schema diff (feeds Task 12’s DDL)
Terminal window
gh api "repos/hyperdxio/hyperdx/contents/docker/otel-collector/schema/seed?ref=@hyperdx/otel-collector@2.29.0" --jq '.[].name'

Expected: 10 files, including the version-gated alternates 00002_otel_logs_compat.sql and 00005_otel_traces_compat.sql (2.27.0 had 8; new in 2.29.0: 00005_otel_traces_compat.sql and the 00008 TimeSeries seed). The compat files define the SAME tables as their non-compat siblings — the Task 12 generator selects exactly one per table based on the pinned CH version. Save the list to the PR description.

  • Step 6: Render-diff (chart bump — REVIEW CAREFULLY)
Terminal window
kubectl kustomize --enable-helm argocd/app-configs/clickstack-chart > /tmp/render-301.yaml
yq ea -o=json '[.]|sort_by(.kind+"/"+.metadata.name)|map(sort_keys(..))' /tmp/render-before.yaml > /tmp/n.json
yq ea -o=json '[.]|sort_by(.kind+"/"+.metadata.name)|map(sort_keys(..))' /tmp/render-301.yaml  > /tmp/k.json
git diff --no-index /tmp/n.json /tmp/k.json | head -100

Expected: image tags 2.27.0→2.29.0 (app + collector), chart labels 3.0.0→3.0.1, plus whatever 3.0.1 changed upstream — read every hunk; anything surprising (renamed resources, releaseName drift) is a STOP.

  • Step 7: Lint + commit

Commit: chore(clickstack): chart 3.0.0->3.0.1, drop #219 collector image pin [hl-uj69]

Task 9: longhorn-single StorageClass

Section titled “Task 9: longhorn-single StorageClass”

Files:

  • Modify: argocd/app-configs/shared-resources/longhorn-storage-classes.yaml (currently only longhorn-retain)

  • Step 1: Append the new class

---
# Single-replica class for workloads that replicate at the APPLICATION
# layer (hl-uj69: ReplicatedMergeTree ClickHouse). Longhorn 3x under an
# app that keeps N full copies multiplies write amplification for no
# added durability. Node loss = volume loss = the app re-syncs the
# replica from its peers.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-single
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  numberOfReplicas: "1"
  dataLocality: best-effort
  dataEngine: v1
  • Step 2: Lint + commit

Commit: feat(storage): longhorn-single StorageClass for app-replicated workloads [hl-uj69]

Task 10: Keeper x3 + ClickHouse replicas 2 + storage class

Section titled “Task 10: Keeper x3 + ClickHouse replicas 2 + storage class”

Files:

  • Modify: argocd/app-configs/clickstack-chart/values.yaml — clickhouse.keeper.spec.replicas (line ~75), clickhouse.cluster.spec.replicas (line ~111), dataVolumeClaimSpec.storageClassName (line 171)

  • Step 1: Keeper 1 → 3

  keeper:
    spec:
      # 3-member ensemble (hl-uj69): a single Keeper is a
      # write-availability SPOF once tables are Replicated —
      # ReplicatedMergeTree goes read-only without Keeper quorum.
      replicas: 3
  • Step 2: Replicas 1 → 2 and update the topology comment

Replace the “Topology: 1 shard × 1 replica” comment block (lines ~85-110) with:

      # Topology: 1 shard x 2 replicas, ReplicatedMergeTree (hl-uj69).
      # Replication WITHOUT sharding: every replica holds ALL data, any
      # replica answers any query — no Distributed table, no HyperDX
      # support needed (its 2.28/2.29 Distributed work targets shards).
      # Tables are pre-created Replicated by the bootstrap Job (PR3);
      # the HyperDX migrator's plain-MergeTree seeds no-op via
      # IF NOT EXISTS. Guard: otelcollector profile sets
      # database_replicated_allow_only_replicated_engine=1 so a future
      # upstream seed table fails LOUDLY instead of silently splitting
      # data across replicas (spec: Grounding, split-brain hazard).

Set replicas: 2 (keep shards: 1).

  • Step 3: Storage class for CH data volumes

In dataVolumeClaimSpec: storageClassName: longhorn → storageClassName: longhorn-single. Add comment: # app-level replication; see longhorn-single class rationale. Affects NEW replica PVCs only; replica-0's PVC is recreated on this class during the cutover window.

  • Step 4: Render-diff, lint, commit

Render-diff expected: KeeperCluster replicas 3, ClickHouseCluster replicas 2 + storageClassName. Commit: feat(clickstack): 1x2 replicated topology — Keeper x3, CH replicas 2, longhorn-single [hl-uj69]

Task 11: PR2 — open, merge, validate live (GATE)

Section titled “Task 11: PR2 — open, merge, validate live (GATE)”

Files: none (process)

  • Step 1: Push bookmark feat/clickstack-replicated-topology, open PR (same pattern as Task 6 Step 1; PR title feat(clickstack): chart 3.0.1 + 1x2 replicated topology [hl-uj69], body notes Task 7 prerequisite completed). USER GATE: merge approval.

  • Step 2: Validate Keeper quorum (riskiest op in this PR)

Terminal window
kubectl -n clickstack get pods -l clickhouse.com/role=clickhouse-keeper
# CORRECTED during execution (hl-uj69.11): client port is 2181 (not
# 9181) and the keeper image has no `nc` — use clickhouse-keeper-client.
for p in 0 1 2; do kubectl -n clickstack exec cs-clickstack-keeper-keeper-$p-0 -- clickhouse-keeper-client -h localhost -p 2181 -q srvr | rg 'Mode|Connections'; done

Expected: 3 pods Running; exactly one leader with zk_synced_followers 2 (or two followers). NOT healthy → STOP, roll back Keeper to 1 replica via revert PR before anything else.

  • Step 3: Validate replica-1 arrived with schema, on the right class
Terminal window
kubectl -n clickstack get pvc clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-1-0 -o jsonpath='{.spec.storageClassName} {.status.capacity.storage}'

Expected: longhorn-single 200Gi.

-- per-pod check (NOT via Service): exec clickhouse-client on pod 0-1-0
SELECT count() FROM system.tables WHERE database = 'default';

Expected: same table count as replica 0-0-0 (Replicated DB metadata sync) but SELECT count() FROM default.otel_logs ≈ 0 on the new replica (plain MergeTree data does NOT replicate — expected split-data window; cutover today).

  • Step 4: Validate collector 2.29 kept the custom config

Repeat Task 6 Step 2 (effective config check) — 2.28+ moved processor lists local; our leaf additions must still be present. Record: bd note hl-uj69 "PR2 merged + validated: <PR url>; keeper quorum OK; replica-1 schema-only as expected".

Stage C — PR3 + cutover window

Section titled “Stage C — PR3 + cutover window”

Task 12: Bootstrap DDL — generation script + generated ConfigMap

Section titled “Task 12: Bootstrap DDL — generation script + generated ConfigMap”

Files:

  • Create: scripts/clickstack/generate-bootstrap-ddl.sh
  • Create: argocd/app-configs/clickstack/bootstrap-schema-configmap.yaml (generated, committed)
  • Modify: argocd/app-configs/clickstack/kustomization.yaml (add resource)
  • Step 1: New workspace clickstack-pr3 off updated main
  • Step 2: Write the generator

Create scripts/clickstack/generate-bootstrap-ddl.sh (mode 755):

#!/usr/bin/env bash
# Generate the Replicated bootstrap DDL for the ClickStack cutover
# (hl-uj69) from the PINNED upstream seed SQL, and wrap it in a
# ConfigMap. Re-run on every chart bump; commit the output.
#
# Transforms, per spec (docs/engineering/specs/2026-07-05-...):
#  - COMPAT SELECTION: upstream ships version-gated ALTERNATE
#    definitions of otel_logs/otel_traces (*_compat.sql, for
#    ClickHouse < 26.2 — no full-text-search indexes). The HyperDX
#    migrator picks at runtime via SELECT version(); we pick at
#    GENERATION time from the CH tag pinned in values.yaml. Getting
#    this wrong = DDL that fails on first REAL creation (the benign
#    IF-NOT-EXISTS sync would NOT catch it).
#  - ${DATABASE}   -> target db (arg 1, default "default")
#  - ${TABLES_TTL} -> per-signal interval (logs/traces/sessions 14d,
#                     metrics 30d)
#  - ENGINE = SummingMergeTree -> ReplicatedSummingMergeTree
#  - ENGINE = MergeTree        -> ReplicatedMergeTree
#    (NO engine arguments: in a Replicated database the defaults are
#     /clickhouse/tables/{uuid}/{shard} + {replica}; explicit paths are
#     rejected by database_replicated_allow_replicated_engine_arguments=0)
#  - 00008 (TimeSeries engine) is copied engine-UNTRANSFORMED —
#    TimeSeries does not round-trip through Replicated-database DDL;
#    documented exception in the spec (data splits across replicas;
#    PromQL unused).
#
# Usage: generate-bootstrap-ddl.sh            -> writes the ConfigMap
#        generate-bootstrap-ddl.sh <scratchdb> -> writes plain SQL to
#          /tmp/bootstrap-<scratchdb>.sql (first-creation rehearsal)
set -euo pipefail


TAG="@hyperdx/otel-collector@2.29.0"
DB="${1:-default}"
OUT="argocd/app-configs/clickstack/bootstrap-schema-configmap.yaml"
workdir=$(mktemp -d); trap 'rm -rf "$workdir"' EXIT


# ClickHouse version pinned in values.yaml decides compat vs non-compat.
# yq-scoped to the CH image (NOT a file-order-dependent grep — the
# collector image tag also matches a naive 'tag:' pattern).
CH_VER=$(yq '.clickhouse.cluster.spec.containerTemplate.image.tag' \
  argocd/app-configs/clickstack-chart/values.yaml | rg -o '^[0-9]+\.[0-9]+')
[ -n "$CH_VER" ] || { echo "ERROR: could not extract CH version from values.yaml" >&2; exit 1; }
if awk -v v="$CH_VER" 'BEGIN{split(v,a,"."); exit !(a[1]>26 || (a[1]==26 && a[2]>=2))}'; then
  SKIP_GLOB='_compat'         # >= 26.2: use full-text schema, skip compat
else
  SKIP_GLOB='00002_otel_logs.sql|00005_otel_traces.sql'  # < 26.2: use compat
fi
echo "CH ${CH_VER}: skipping seed files matching: ${SKIP_GLOB}"


ttl_for() {
  case "$1" in
    *otel_logs*|*logs_rollups*) echo "toIntervalDay(14)" ;;
    *otel_metrics*|*metrics_timeseries*) echo "toIntervalDay(30)" ;;
    *otel_traces*|*traces_rollups*) echo "toIntervalDay(14)" ;;
    *hyperdx_sessions*) echo "toIntervalDay(14)" ;;
    *) echo "toIntervalDay(30)" ;;
  esac
}


gh api "repos/hyperdxio/hyperdx/contents/docker/otel-collector/schema/seed?ref=${TAG}" \
  --jq '.[] | .name + " " + .download_url' |
while read -r name url; do
  curl -fsSL "$url" -o "$workdir/$name"
done


: > "$workdir/bootstrap.sql"
for f in "$workdir"/*.sql; do
  base=$(basename "$f")
  # The accumulator lives in $workdir too — without this guard the glob
  # feeds bootstrap.sql back into itself (sed reads while >> appends:
  # infinite loop). Bug found during execution (hl-uj69.12).
  [ "$base" = "bootstrap.sql" ] && continue
  if echo "$base" | rg -q "$SKIP_GLOB"; then
    echo "skip: $base (version-gated alternate)"; continue
  fi
  ttl=$(ttl_for "$base")
  if [[ "$base" == 00008_* ]]; then
    # TimeSeries exception: substitute macros only, keep engine.
    sed -e "s/\${DATABASE}/${DB}/g" -e "s/\${TABLES_TTL}/${ttl}/g" "$f"
  else
    sed -e "s/\${DATABASE}/${DB}/g" \
        -e "s/\${TABLES_TTL}/${ttl}/g" \
        -e "s/ENGINE = SummingMergeTree/ENGINE = ReplicatedSummingMergeTree/g" \
        -e "s/ENGINE = MergeTree/ENGINE = ReplicatedMergeTree/g" "$f"
  fi >> "$workdir/bootstrap.sql"
  printf '\n' >> "$workdir/bootstrap.sql"
done


# Guardrails: no unsubstituted macros; no plain MergeTree; and EXACTLY
# ONE definition each of otel_logs and otel_traces (compat selection
# worked — duplicate or zero definitions = generation bug, fail loudly).
! rg -q '\$\{' "$workdir/bootstrap.sql"
! rg -q 'ENGINE = (Summing)?MergeTree$' "$workdir/bootstrap.sql" || {
  echo "ERROR: untransformed MergeTree engine found" >&2; exit 1; }
for t in otel_logs otel_traces; do
  n=$(rg -c "CREATE TABLE IF NOT EXISTS ${DB}\.${t}\b" "$workdir/bootstrap.sql" || true)
  [ "$n" = "1" ] || { echo "ERROR: ${t} defined ${n} times (want 1)" >&2; exit 1; }
done


if [ "$DB" != "default" ]; then
  cp "$workdir/bootstrap.sql" "/tmp/bootstrap-${DB}.sql"
  echo "Wrote /tmp/bootstrap-${DB}.sql (rehearsal; not a ConfigMap)"
  exit 0
fi


{
  echo "---"
  echo "# GENERATED by scripts/clickstack/generate-bootstrap-ddl.sh — do not hand-edit."
  echo "# Source: hyperdxio/hyperdx ${TAG} docker/otel-collector/schema/seed/"
  echo "# Compat selection: CH ${CH_VER} (< 26.2 uses *_compat logs/traces schema)."
  kubectl create configmap clickstack-bootstrap-schema \
    --namespace clickstack \
    --from-file=bootstrap.sql="$workdir/bootstrap.sql" \
    --dry-run=client -o yaml
} > "$OUT"
echo "Wrote $OUT ($(rg -c 'CREATE TABLE' "$workdir/bootstrap.sql") CREATE TABLE statements)"
  • Step 3: Run it and inspect the output
Terminal window
scripts/clickstack/generate-bootstrap-ddl.sh
rg -n 'ENGINE = Replicated|ENGINE = TimeSeries|CREATE TABLE|TTL ' \
  argocd/app-configs/clickstack/bootstrap-schema-configmap.yaml | head -40

Expected: script prints CH 25.7: skipping seed files matching: 00002_otel_logs.sql|00005_otel_traces.sql (compat schemas selected — CH < 26.2 has no full-text-search indexes; the migrator does this selection at runtime, we do it at generation time); every base table ENGINE = ReplicatedMergeTree / ReplicatedSummingMergeTree (no args); 00008 keeps TimeSeries; TTLs toIntervalDay(14) on logs/traces/sessions, toIntervalDay(30) on metrics; everything IF NOT EXISTS; the uniqueness guardrail passed (exactly one definition each of otel_logs/otel_traces). Read the full generated SQL once end-to-end — you are the schema owner now.

  • Step 4: Register in kustomization, lint, commit

Append - bootstrap-schema-configmap.yaml to argocd/app-configs/clickstack/kustomization.yaml resources. Commit: feat(clickstack): generated Replicated bootstrap DDL from 2.29.0 seeds [hl-uj69]

Task 13: PreSync bootstrap Job + schema_admin user + guard + Velero policy

Section titled “Task 13: PreSync bootstrap Job + schema_admin user + guard + Velero policy”

Files:

  • Create: argocd/app-configs/clickstack/bootstrap-schema-job.yaml
  • Modify: argocd/app-configs/clickstack/kustomization.yaml (add resource)
  • Modify: argocd/app-configs/clickstack-chart/values.yaml (extraUsersConfig: schema_admin user + guard profile for otelcollector)
  • Create: argocd/app-configs/velero/resource-policies-configmap.yaml
  • Modify: argocd/app-configs/velero/backup-schedule.yaml (both Schedules: add resourcePolicy)
  • Modify: argocd/app-configs/velero/kustomization.yaml (add resource)
  • Step 0: Vault-source the schema_admin password (repo rule: MUST NOT commit secrets)

One-time operational step (with the user; kv patch preserves existing keys):

Terminal window
vault kv patch fzymgc-house/cluster/clickstack schema_admin_password="$(openssl rand -base64 24)"

Then in argocd/app-configs/clickstack/external-secret.yaml, following the existing Vault-sourced entries (admin_password, mcp_readwrite_password, …):

Add to spec.target.template.data:

        schema_admin_password: '{{ .schema_admin_password }}'

Add to spec.data:

    - secretKey: schema_admin_password
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: schema_admin_password

And in values.yaml, add to clickhouse.cluster.spec.containerTemplate (sibling of image:/resources:) so the operator can resolve the @from_env password directive below:

        env:
          - name: SCHEMA_ADMIN_PASSWORD
            valueFrom:
              secretKeyRef:
                name: clickstack-secret
                key: schema_admin_password
  • Step 1: Add schema_admin + the guard to extraUsersConfig

In the settings.extraUsersConfig block added in Task 4, extend to:

        extraUsersConfig:
          profiles:
            hyperdx_reader:
              max_memory_usage: 2147483648
              max_bytes_ratio_before_external_group_by: 0.5
              max_bytes_ratio_before_external_sort: 0.5
            # Fail-loud guard (hl-uj69): if a future chart bump ships a
            # NEW seed table our bootstrap DDL doesn't know, the
            # migrator's plain CREATE TABLE fails visibly (collector
            # crashloop) instead of silently creating a non-replicated
            # table with round-robin-split data. Remedy: re-run
            # scripts/clickstack/generate-bootstrap-ddl.sh, commit, sync.
            otel_writer:
              database_replicated_allow_only_replicated_engine: 1
          users:
            app:
              profile: hyperdx_reader
            otelcollector:
              profile: otel_writer
            # Bootstrap-DDL user: NOT guarded (must be able to create
            # the TimeSeries exception table). Password is
            # Vault-sourced via @from_env <- containerTemplate env <-
            # clickstack-secret (Step 0) — NOT a committed literal
            # (repo Security Constraint; also the operator's own
            # guidance for extraUsersConfig passwords). Network access
            # is gated by CiliumNetworkPolicy clickstack-default
            # (intra-namespace + vector/traefik/clickstack-operators
            # namespaces); the password is the real gate.
            schema_admin:
              # NOTE the LIST wrapper on password — the operator's
              # documented @from_env shape (configuration.mdx examples)
              # is `password: [ {'@from_env': VAR} ]`, not a flat map.
              password:
                - "@from_env": SCHEMA_ADMIN_PASSWORD
              profile: default
              # CREATE DATABASE (rehearsal db) + BACKUP (Task 17) need
              # *.* scope; INSERT is required by RESTORE (a separate
              # top-level privilege — CREATE does not imply it; without
              # it Task 17's restore rehearsal fails and would
              # masquerade as a RustFS incompatibility); SELECT/SHOW
              # for validation queries. The widest-privileged CH user —
              # called out in docs (Task 19).
              grants:
                - query: "GRANT CREATE, DROP, INSERT, SELECT, SHOW, BACKUP ON *.*"
  • Step 2: Write the PreSync Job

Create argocd/app-configs/clickstack/bootstrap-schema-job.yaml:

---
# Schema bootstrap (hl-uj69). PreSync is REQUIRED, not preference: the
# collector's migrator runs at POD START, so on any sync that rolls the
# collector this Job must complete BEFORE the new pod starts. PostSync
# would deadlock (guard crashloops the collector -> app never healthy
# -> PostSync never fires). NOTE: first PreSync hook in this repo (the
# three existing hooks are PostSync); ordering across this multi-source
# Application is validated on a benign sync before the cutover relies
# on it (spec: Validation).
apiVersion: batch/v1
kind: Job
metadata:
  name: clickstack-schema-bootstrap
  namespace: clickstack
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  ttlSecondsAfterFinished: 600
  backoffLimit: 4
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: bootstrap
          image: clickhouse/clickhouse-server:25.7-alpine
          command:
            - sh
            - -c
            - |
              exec clickhouse-client \
                --host cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc.cluster.local \
                --port 9000 \
                --user schema_admin \
                --password "$SCHEMA_ADMIN_PASSWORD" \
                --multiquery < /ddl/bootstrap.sql
          env:
            - name: SCHEMA_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: clickstack-secret
                  key: schema_admin_password
          volumeMounts:
            - name: ddl
              mountPath: /ddl
              readOnly: true
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi
      volumes:
        - name: ddl
          configMap:
            name: clickstack-bootstrap-schema

Append - bootstrap-schema-job.yaml to the clickstack kustomization resources.

  • Step 3: Velero resource policy — skip longhorn-single volumes

Create argocd/app-configs/velero/resource-policies-configmap.yaml:

---
# Volume policy (hl-uj69): skip backup of volumes on longhorn-single —
# today that is exactly the ClickHouse data volumes. CH telemetry is
# declared disposable; replication is its availability story; native
# BACKUP TO S3 (weekly, RustFS) covers the residual case. Keying on the
# StorageClass survives PVC recreation, unlike per-PVC labels.
apiVersion: v1
kind: ConfigMap
metadata:
  name: velero-resource-policies
  namespace: velero
data:
  policies.yaml: |
    version: v1
    volumePolicies:
      - conditions:
          storageClass:
            - longhorn-single
        action:
          type: skip

In backup-schedule.yaml, add to BOTH Schedules’ template: (sibling of snapshotMoveData):

    resourcePolicy:
      kind: configmap
      name: velero-resource-policies

Register the new file in argocd/app-configs/velero/kustomization.yaml.

  • Step 4: Render-diff (chart source), kustomize-build (clickstack + velero sources), lint, commit

Commit: feat(clickstack): PreSync schema-bootstrap Job, engine guard, Velero skip-policy for longhorn-single [hl-uj69]

Task 14: PR3 — open, merge, validate PreSync on a benign sync (GATE)

Section titled “Task 14: PR3 — open, merge, validate PreSync on a benign sync (GATE)”

Files: none (process)

  • Step 1: Push bookmark feat/clickstack-cutover-prep, open PR (title feat(clickstack): cutover prep — bootstrap Job, guard, backup exclusion [hl-uj69]). USER GATE: merge approval. NOTE: this merge is BENIGN — the Job’s IF NOT EXISTS DDL no-ops against the existing plain tables; the guard only constrains otelcollector CREATEs (it does none while tables exist); nothing is dropped yet.

  • Step 2: Validate PreSync ordering (first PreSync hook in this repo — do not assume)

Terminal window
kubectl -n clickstack get job clickstack-schema-bootstrap -o jsonpath='{.status.succeeded}'
kubectl -n argocd get app clickstack -o jsonpath='{.status.operationState.syncResult.resources[?(@.name=="clickstack-schema-bootstrap")].hookPhase}'

Expected: 1 and Succeeded, and in the ArgoCD UI the hook shows in the PreSync phase, completed BEFORE Sync-phase resources. If the hook ran but ordering is wrong/ambiguous, STOP — fallback per spec risk table: run the DDL manually in the window, demote Job to PostSync with documented caveat.

  • Step 3: Validate the guard is live without breaking ingest
SELECT value FROM system.settings_profile_elements
WHERE profile_name = 'otel_writer' AND setting_name = 'database_replicated_allow_only_replicated_engine';
-- expect: 1

Ingest still flowing (max(TimeUnix) staleness < 60s on otel_metrics_gauge).

  • Step 4: First-creation rehearsal in a scratch Replicated database (GATE — the benign sync only proved the IF-NOT-EXISTS no-op path; this proves the DDL actually CREATES on 25.7)
Terminal window
scripts/clickstack/generate-bootstrap-ddl.sh bootstrap_rehearsal
POD=cs-clickstack-clickhouse-clickhouse-0-0-0
PW=$(kubectl -n clickstack get secret clickstack-secret -o jsonpath='{.data.schema_admin_password}' | base64 -d)
kubectl -n clickstack exec $POD -- clickhouse-client \
  --user schema_admin --password "$PW" --query \
  "CREATE DATABASE bootstrap_rehearsal ENGINE = Replicated('/clickhouse/databases/bootstrap_rehearsal', '{shard}', '{replica}')"
kubectl -n clickstack exec -i $POD -- clickhouse-client \
  --user schema_admin --password "$PW" --multiquery \
  < /tmp/bootstrap-bootstrap_rehearsal.sql

Verify, then clean up:

SELECT engine, count() FROM system.tables WHERE database='bootstrap_rehearsal' GROUP BY engine;
-- expect Replicated* engines (+ the TimeSeries exception + MaterializedViews), zero errors above
DROP DATABASE bootstrap_rehearsal SYNC;

ANY DDL error here = the generator or compat selection is wrong — fix and re-merge BEFORE scheduling the cutover window. This is the last gate where a broken bootstrap costs nothing. Record: bd note hl-uj69 "PR3 merged; PreSync ordering + first-creation rehearsal VALIDATED".

Task 15: Execute the cutover (maintenance window, ~minutes)

Section titled “Task 15: Execute the cutover (maintenance window, ~minutes)”

Files: none (runbook; all manifests already merged)

USER GATE: schedule the window with the user. Everything below is destructive to telemetry history (approved by spec decision).

  • Step 1: Disable auto-sync (MANDATORY), then scale the collector to 0
Terminal window
argocd app set clickstack --sync-policy none
kubectl -n clickstack scale deploy/cs-otel-collector --replicas=0

Auto-sync MUST be off for the window: with selfHeal: true active, ArgoCD would restore the collector mid-drop against missing tables — recreating the exact error-flood failure class this design eliminates. Step 5 re-enables it (which also restores the collector).

  • Step 2: Recreate replica-0’s PVC on longhorn-single (MOVED EARLY — unwedges the operator)

AMENDED during execution (hl-uj69.11): operator v0.0.1 crashloops its reconcile trying to patch replica-0’s PVC storageClassName (immutable field), which STARVES all later reconcile steps — replica-1’s default database is still Atomic with no schema, and pending config restarts never fire. Deleting the pod+PVC unwedges it (the CREATE path is legal); this destroys the old telemetry a few minutes earlier than the original order — same window, same approved data loss.

Terminal window
kubectl -n clickstack delete pod cs-clickstack-clickhouse-clickhouse-0-0-0 --wait=false
kubectl -n clickstack delete pvc clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-0-0
# operator reconcile unwedges: recreates PVC (longhorn-single) + pod, applies
# pending users.d config, then runs databaseSync on BOTH replicas
kubectl -n clickstack get pvc clickhouse-storage-volume-cs-clickstack-clickhouse-clickhouse-0-0-0 -o jsonpath='{.spec.storageClassName}'

Expected: longhorn-single. Then WAIT for the operator to fully reconcile — this is the gate for everything after:

Terminal window
# CHC conditions all True again (Ready, SchemaInSync, ...)
kubectl -n clickstack get clickhousecluster cs-clickstack-clickhouse -o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}'
# BOTH replicas' default database MUST be Replicated engine before bootstrap:
for p in 0-0-0 0-1-0; do kubectl -n clickstack exec cs-clickstack-clickhouse-clickhouse-$p -- clickhouse-client --query "SELECT engine FROM system.databases WHERE name='default'"; done

Expected: Replicated on both. If either stays Atomic after reconcile settles, STOP — the bootstrap’s {uuid}-default ReplicatedMergeTree paths require the Replicated database engine to share table UUIDs across replicas.

  • Step 3: Drop any leftover plain tables (both replicas fresh/empty is the expected state)

Via clickhouse-client as schema_admin (fetch $PW from clickstack-secret as in Task 14 Step 4) on EACH pod, generate-and-review then execute:

SELECT 'DROP TABLE IF EXISTS default.' || name || ' SYNC;' FROM system.tables
WHERE database = 'default'
  AND (name LIKE 'otel_%' OR name LIKE 'hyperdx_%' OR name LIKE 'metrics_%');

After the early PVC swap both replicas should already be empty — this step is defensive (interim migrator/Vector table creation). Verify count 0 on BOTH pods before proceeding.

  • Step 4: Run the bootstrap Job
Terminal window
argocd app sync clickstack   # PreSync hook re-fires (BeforeHookCreation)
kubectl -n clickstack wait --for=condition=complete job/clickstack-schema-bootstrap --timeout=300s

Verify on BOTH pods: SELECT count() FROM system.replicas WHERE database='default' > 0 and equal; SELECT engine FROM system.tables WHERE database='default' AND name='otel_logs' → ReplicatedMergeTree.

  • Step 5: Restore the collector + auto-sync
Terminal window
argocd app set clickstack --sync-policy automated --auto-prune --self-heal
kubectl -n clickstack rollout status deploy/cs-otel-collector
  • Step 6: Confirm ingest on BOTH replicas — per-pod SELECT count() FROM default.otel_logs climbing on each within ~2 min. Record: bd note hl-uj69 "CUTOVER COMPLETE <timestamp>".

Task 16: Post-cutover validation suite (spec: Validation)

Section titled “Task 16: Post-cutover validation suite (spec: Validation)”

Files: none (verification)

  • Step 1: Replication health — SELECT database, table, is_readonly, queue_size, absolute_delay FROM system.replicas WHERE database='default': all rows is_readonly=0, queue_size≈0.
  • Step 2: Row-count convergence — per-pod counts on otel_logs/otel_metrics_gauge after 15 min differ by < the last flush interval (not 2x).
  • Step 3: Pod-kill test — kubectl -n clickstack delete pod cs-clickstack-clickhouse-clickhouse-0-1-0; dashboards + ingest continue (collector queue absorbs any blip); replica rejoins, system.replicas.queue_size drains to ~0.
  • Step 4: Dashboard smoke — render the 7d histogram-rate chart that OOM’d (2026-07-04 17:20 incident); expect success under the 2GiB hyperdx_reader cap. Also watch live tail for ~2 min: if rows visibly flicker as the LB alternates replicas, apply the spec’s contingency (sessionAffinity: ClientIP on the HyperDX-facing CH Service) as a follow-up PR.
  • Step 5: Alert smoke — one tile alert fires end-to-end to pushover (use the nats-slow alert’s test path or temporarily lower a threshold; restore after).
  • Step 6: Collector resilience re-test — SYSTEM STOP MERGES is not enough; instead scale CH pods’ Service away is invasive — accepted alternative: watch otelcol_exporter_queue_size during Step 3’s pod-kill; expect brief queue growth then drain, NO collector restart (kubectl -n clickstack get pod -l app.kubernetes.io/name=otel-collector -o jsonpath='{.items[0].status.containerStatuses[0].restartCount}' unchanged).
  • Step 7: Record all outcomes: bd note hl-uj69 "post-cutover validation: <summary>". Any failure → spec Risks table has the rollback (re-drop, plain bootstrap, replicas:1).

Stage D — post-cutover safety nets

Section titled “Stage D — post-cutover safety nets”

Task 17: Weekly CH backup to RustFS + restore rehearsal

Section titled “Task 17: Weekly CH backup to RustFS + restore rehearsal”

Files:

  • Create: argocd/app-configs/clickstack/backup-cronjob.yaml
  • Create: argocd/app-configs/clickstack/backup-secrets.yaml (ExternalSecret)
  • Modify: argocd/app-configs/clickstack/kustomization.yaml
  • Step 1: One-time RustFS + Vault prep (operational, with user)

On RustFS (s3.nas.fzymgc.house): create bucket clickstack-backups with a 28-day lifecycle expiry (mc ilm rule add --expire-days 28 <alias>/clickstack-backups) and a scoped access key. Store in Vault: vault kv put fzymgc-house/cluster/clickstack-backup access_key=<k> secret_key=<s>. Verify the external-secrets Vault policy covers fzymgc-house/cluster/* (check tf/vault/; if the policy enumerates paths, add this one via the terraform skill — Security Constraint: policies MUST be updated for new paths).

  • Step 2: ExternalSecret

argocd/app-configs/clickstack/backup-secrets.yaml, following the repo pattern (ClusterSecretStore vault):

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: clickstack-backup-s3
  namespace: clickstack
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: clickstack-backup-s3
    creationPolicy: Owner
  data:
    - secretKey: access_key
      remoteRef:
        key: fzymgc-house/cluster/clickstack-backup
        property: access_key
    - secretKey: secret_key
      remoteRef:
        key: fzymgc-house/cluster/clickstack-backup
        property: secret_key
  • Step 3: Backup CronJobs (weekly full Monday, daily incremental Tue–Sun; ISO-week names so incrementals self-locate their base)

argocd/app-configs/clickstack/backup-cronjob.yaml:

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: clickstack-backup-full
  namespace: clickstack
spec:
  schedule: "0 3 * * 1"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: clickhouse/clickhouse-server:25.7-alpine
              command:
                - sh
                - -c
                - |
                  set -eu
                  WEEK=$(date +%G-W%V)
                  exec clickhouse-client \
                    --host cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc.cluster.local \
                    --user schema_admin --password "$SCHEMA_ADMIN_PASSWORD" \
                    --query "BACKUP DATABASE default TO S3('https://s3.nas.fzymgc.house/clickstack-backups/${WEEK}-full', '$AWS_ACCESS_KEY_ID', '$AWS_SECRET_ACCESS_KEY')"
              env:
                - name: SCHEMA_ADMIN_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: clickstack-secret
                      key: schema_admin_password
                - name: AWS_ACCESS_KEY_ID
                  valueFrom: {secretKeyRef: {name: clickstack-backup-s3, key: access_key}}
                - name: AWS_SECRET_ACCESS_KEY
                  valueFrom: {secretKeyRef: {name: clickstack-backup-s3, key: secret_key}}
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: clickstack-backup-incr
  namespace: clickstack
spec:
  schedule: "0 3 * * 2-7"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: clickhouse/clickhouse-server:25.7-alpine
              command:
                - sh
                - -c
                - |
                  set -eu
                  WEEK=$(date +%G-W%V)
                  DAY=$(date +%u)
                  exec clickhouse-client \
                    --host cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc.cluster.local \
                    --user schema_admin --password "$SCHEMA_ADMIN_PASSWORD" \
                    --query "BACKUP DATABASE default TO S3('https://s3.nas.fzymgc.house/clickstack-backups/${WEEK}-inc-${DAY}', '$AWS_ACCESS_KEY_ID', '$AWS_SECRET_ACCESS_KEY') SETTINGS base_backup = S3('https://s3.nas.fzymgc.house/clickstack-backups/${WEEK}-full', '$AWS_ACCESS_KEY_ID', '$AWS_SECRET_ACCESS_KEY')"
              env:
                - name: SCHEMA_ADMIN_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: clickstack-secret
                      key: schema_admin_password
                - name: AWS_ACCESS_KEY_ID
                  valueFrom: {secretKeyRef: {name: clickstack-backup-s3, key: access_key}}
                - name: AWS_SECRET_ACCESS_KEY
                  valueFrom: {secretKeyRef: {name: clickstack-backup-s3, key: secret_key}}

Note: schema_admin’s grants (Task 13) already include BACKUP ON *.* — no grant change needed here.

  • Step 4: Register resources, lint, PR feat/clickstack-backups, merge (USER GATE)

  • Step 5: Rehearse a restore (acceptance gate for RustFS qualification)

Trigger the full job manually (kubectl -n clickstack create job --from=cronjob/clickstack-backup-full backup-rehearsal), wait for success, then via schema_admin — substituting $WEEK = the ISO week the job printed (date +%G-W%V) and the S3 creds read from the secret (kubectl -n clickstack get secret clickstack-backup-s3 -o jsonpath='{.data.access_key}' | base64 -d, same for secret_key):

RESTORE DATABASE default AS restore_rehearsal
FROM S3('https://s3.nas.fzymgc.house/clickstack-backups/$WEEK-full', '$ACCESS_KEY', '$SECRET_KEY');
SELECT count() FROM restore_rehearsal.otel_logs;  -- expect ~= default.otel_logs
DROP DATABASE restore_rehearsal SYNC;

If BACKUP or RESTORE fails on RustFS semantics: per spec fallback, drop CH backups entirely (revert this PR) and file a bead to re-evaluate with MinIO/Garage; the design’s availability story (replication) is unaffected.

Task 18: Watchdog — dead-man’s switch + engine-drift detector

Section titled “Task 18: Watchdog — dead-man’s switch + engine-drift detector”

Files:

  • Create: argocd/app-configs/clickstack/watchdog-cronjob.yaml
  • Modify: argocd/app-configs/clickstack/kustomization.yaml
  • Modify: tf/uptime-kuma/ (push monitor; via the fzymgc-house:terraform skill)

Design note (deviation from spec’s “ClickStack tile alert” for drift, rationale: an alert inside the stack cannot fire when the stack is dark — one external watchdog covers both cases): a CronJob every 5 min checks (a) ingest freshness and (b) plain-MergeTree drift, and pushes an uptime-kuma heartbeat ONLY if both pass; uptime-kuma (already pushover-wired) alerts on missed heartbeats.

  • Step 1: TF — uptime-kuma push monitor (terraform skill): monitor clickstack-watchdog, type push, heartbeat interval 300s, retries 2; write the push token to Vault fzymgc-house/cluster/clickstack-watchdog token=<push-token> (TF vault_kv_secret_v2, policy check as in Task 17).

  • Step 2: Watchdog CronJob + ExternalSecret

argocd/app-configs/clickstack/watchdog-cronjob.yaml (ExternalSecret for the token follows the Task 17 pattern, name clickstack-watchdog, property token):

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: clickstack-watchdog
  namespace: clickstack
spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 0
      activeDeadlineSeconds: 120
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: watchdog
              image: clickhouse/clickhouse-server:25.7-alpine
              command:
                - sh
                - -c
                - |
                  set -eu
                  CH="clickhouse-client --host cs-clickstack-clickhouse-clickhouse-headless.clickstack.svc.cluster.local --user app --password hyperdx"
                  STALE=$($CH --query "SELECT dateDiff('second', max(TimeUnix), now()) FROM default.otel_metrics_gauge WHERE TimeUnix > now() - INTERVAL 1 HOUR")
                  DRIFT=$($CH --query "SELECT count() FROM system.tables WHERE database='default' AND engine IN ('MergeTree','SummingMergeTree')")
                  echo "stale=${STALE}s drift=${DRIFT}"
                  [ "$STALE" -lt 300 ] && [ "$DRIFT" -eq 0 ] || exit 1
                  wget -q -O- "https://status.fzymgc.house/api/push/${PUSH_TOKEN}?status=up&msg=ok&ping=${STALE}"
              env:
                - name: PUSH_TOKEN
                  valueFrom: {secretKeyRef: {name: clickstack-watchdog, key: token}}

(uptime-kuma is status.fzymgc.house per docs/reference/services.md; the app user’s SELECT ON system.* grant covers the drift query — verified in the live CR.)

  • Step 3: Lint, PR feat/clickstack-watchdog (can share the Task 17 PR), merge (USER GATE), then verify: heartbeat green in uptime-kuma; kill test — suspend the CronJob (kubectl -n clickstack patch cronjob clickstack-watchdog -p '{"spec":{"suspend":true}}'), confirm pushover alert after ~2 missed beats, unsuspend.

Task 19: Docs, memory, bead closeout

Section titled “Task 19: Docs, memory, bead closeout”

Files:

  • Modify: docs/operations/clickstack.md (topology, cutover record, backup/restore runbook, watchdog, upgrade checklist)
  • Step 1: Update docs/operations/clickstack.md — add/replace sections:
    • Topology: 1 shard x 2 ReplicatedMergeTree replicas, Keeper x3, longhorn-single (1-replica) CH volumes; writes/reads via Services (collector→ClusterIP, Vector→headless); schema owned by clickstack-bootstrap-schema (generated by scripts/clickstack/generate-bootstrap-ddl.sh), NOT the HyperDX migrator.
    • Upgrade checklist: on every chart bump — (1) bump TAG in scripts/clickstack/generate-bootstrap-ddl.sh to the new collector version, (2) diff docker/otel-collector/schema/seed/ between tags, (3) re-run the generator and commit (it re-reads the pinned CH version for compat selection); a NEW seed table without regeneration = collector crashloop by design (guard) — the fix is regenerate+sync, not disabling the guard.
    • Backup/restore: weekly full + daily incremental to RustFS clickstack-backups (28d lifecycle); restore = RESTORE DATABASE ... AS <name> FROM S3(...); rehearsal date + result.
    • Watchdog: freshness+drift semantics, where the push monitor lives, what a pushover “clickstack-watchdog down” means (check collector, then CH, then drift query).
    • Retention: logs/traces/sessions 14d, metrics 30d (bootstrap DDL); system.* 7d (extraConfig).
    • Users: document the CH users and their blast radii — app (read, 2GiB cap; chart-literal password), otelcollector (write, engine-guarded; chart-literal password), schema_admin (CREATE/DROP/INSERT/SELECT/SHOW/BACKUP on *.* — the widest, INSERT required by RESTORE; Vault-sourced password via @from_env; used only by the bootstrap Job, backup CronJobs, and operators). Note the CiliumNetworkPolicy admits vector/traefik/clickstack-operators namespaces in addition to intra-namespace traffic — password auth, not the network, is the real gate.
  • Step 2: rumdl + PR (docs-only, docs/clickstack-rebuild-ops), merge.
  • Step 3: Store durable memory (engram, per curating-memory): topology + schema-ownership + upgrade-checklist facts; supersede the single-node topology facts in memory 871ba890/c93e8e07 where contradicted.
  • Step 4: Close beads — children of hl-uj69 epic as completed; re-scope or close hl-p7z (upstream PR no longer needed for our topology; keep open only if still pursuing upstream contribution).

Execution-order summary and gates

Section titled “Execution-order summary and gates”
OrderTask(s)Gate
11-5 (PR1 content)render-diff clean, lint
26merge + live validation (collector queue, CH settings, throttle)
37user-approved PVC deletion
48-10 (PR2 content)render-diff reviewed hunk-by-hunk
511merge + Keeper quorum + replica-1 schema
612-13 (PR3 content)generated DDL reviewed end-to-end
714merge + PreSync ordering proof + scratch-DB first-creation rehearsal
815scheduled window; step-by-step runbook
916full validation suite
1017-18backups + watchdog (restore rehearsal = RustFS gate)
1119docs + closeout