Vector + ClickStack — Step 0 Audit Outputs

Date run: 2026-05-11 Operator: Sean (audit run by Claude) Cluster: fzymgc-house Spec: 2026-05-11-vector-clickstack-migration-design.md Plan: docs/engineering/plans/2026-05-11-vector-clickstack-migration.md (PR0 Task 0.2)

---

Item 1: Conflicting CRDs / operators

Status: ✅ Clean — no pre-existing installations.

kubectl get crds | rg -i 'clickhouse|mongodb|opentelemetry' → (no matches)
kubectl get clickhousecluster,keepercluster,mongodb -A → server doesn't have resource type
kubectl get deploy -A | rg -i 'clickhouse-operator|mongodb-operator|opentelemetry-operator' → (no matches)

---

Item 2: Existing OTel deployment overlap

Status: ✅ Clean.

kubectl get pods -A | rg -i 'otel|opentelemetry' → (no matches)

---

Item 3: Chart version pin

Repo: https://clickhouse.github.io/ClickStack-helm-charts (ClickHouse org).

Two-chart pattern (per upstream README):

1. clickstack/clickstack-operators 1.0.0 — installs operators + CRDs (Altinity ClickHouse Operator + MongoDB Community Operator).
2. clickstack/clickstack 2.1.1 (app version 2.22.0) — installs the stack (ClickHouseCluster CR, KeeperCluster CR, MongoDBCommunity CR, HyperDX Deployment, OTel Collector Deployment).

Selected versions:

• clickstack-operators: 1.0.0
• clickstack: 2.1.1

Rationale: Latest GA per argocd/CLAUDE.md “Helm Chart Constraints” rule. Both pulled and rendered successfully on 2026-05-11.

Rendered kinds (clickstack-operators 1.0.0):

2 Deployment            (Altinity CH operator controller-manager, MongoDB community operator)
6 ClusterRole
5 ClusterRoleBinding
5 ServiceAccount
4 RoleBinding
4 Role
2 CustomResourceDefinition  (ClickHouseCluster + MongoDBCommunity — KeeperCluster comes from Altinity install)
1 Service

Rendered kinds (clickstack 2.1.1):

1 ClickHouseCluster     (Altinity CR; spec.replicas, spec.shards configurable)
1 KeeperCluster         (Altinity CR for ClickHouse Keeper, replaces ZK)
1 MongoDBCommunity      (MongoDB Operator CR; spec.members configurable)
2 Deployment            (cs-otel-collector, cs-clickstack-app/HyperDX)
2 Service               (otel-collector ClusterIP, app ClusterIP)
2 Secret
2 ConfigMap
1 ServiceAccount

Spec correction: spec said “3 operators (Altinity CH, MongoDB, OTel)”. Actual count: 2 operators — Altinity (manages both ClickHouse + Keeper from one operator) + MongoDB Community. OTel Collector is a plain Deployment, no operator.

---

Item 4: tf/authentik OIDC capability

Status: ✅ Already supports — PR1.5a NOT needed.

tf/authentik/grafana.tf, tf/authentik/kubernetes-oidc.tf, tf/authentik/karakeep.tf, etc. all use authentik_provider_oauth2. Follow the existing pattern when adding the HyperDX client in PR1.

---

Item 5: tf/grafana datasource + plugin capability

Status: ⚠️ PR1.5b NEEDED — tf/grafana has no grafana_data_source resource; only a data "grafana_data_source" data source (reads existing). No extraPlugins chart-values reference.

rg -l 'grafana_data_source' tf/grafana/
  → tf/grafana/dashboard_temporal.tf (uses `data "grafana_data_source"` to read the existing Prometheus datasource)

rg -l 'extraPlugins|grafana_plugin' tf/grafana/ argocd/cluster-app/templates/grafana.yaml argocd/app-configs/grafana/
  → (no matches)

Action for PR1.5b:

1. Add grafana-clickhouse-datasource plugin to Grafana chart’s plugin list (likely in argocd/cluster-app/templates/grafana.yaml chart valuesObject under plugins: or grafana.ini.plugins.allow_loading_unsigned_plugins).
2. Add a grafana_data_source resource to tf/grafana (probably as tf/grafana/datasources.tf per the plan).
3. Existing Grafana datasources (Prometheus, Loki) are likely chart-managed via datasources: values OR via grafana-operator GrafanaDatasource CRs — verify which before adding ClickHouse to make the new one consistent.

---

Item 6: Loki uncompressed ingest baseline

Status: ⚠️ TBD — capture via Grafana Explore at PR1 authoring time.

Audit attempted via in-cluster Prom HTTP API but failed (no curl/wget on the available Prom-stack pods). Recommended capture method at PR1 time:

• Open Grafana → Explore → Prometheus datasource
• Run query: avg_over_time(rate(loki_distributor_bytes_received_total[5m])[7d:5m])
• Read off bytes/sec → multiply by 86400 for GiB/day uncompressed

ClickHouse PVC sizing formula (per spec’s “Storage sizing baseline”): bytes_per_day × eval_window_days × 2 / 4 (headroom=2, compression=4)

Default to 100 GiB as PR1 placeholder if measurement is deferred — chart PVC supports online expansion via Longhorn if it turns out too small.

---

Item 7: Sync-wave -1 occupancy

Status: ⚠️ Conflict detected — recommend sync-wave 1 for ClickStack, not -1.

-100  system-upgrade
-10   shared-resources
 -9   reloader
 -8   metallb
 -5   cert-manager-config, k8s-oidc-rbac
 -1   kubernetes-replicator, monitoring-prometheus
  0   authentik, cnpg, grafana, grafana-operator, hcp-terraform-operator,
        monitoring-loki, router-hosts-operator (and others)
  1   most app workloads (alloy, cf-ssh-bastion, dolt, karakeep, mealie, mosquitto, nats, etc.)
  2   router-hosts-alerts
  5   grafana-mcp

Spec assumed: clickstack at -1 alongside monitoring-prometheus, with all dependencies (cert-manager, ESO, vault, authentik) at -2 or earlier.

Reality: authentik is at sync-wave 0, AFTER -1. ClickStack’s HyperDX OIDC client_secret is written to Vault via tf/authentik apply (a separate process); the clickstack-secret ExternalSecret references that Vault path. If ClickStack tries to sync before tf/authentik applies, the OIDC secret isn’t in Vault yet → ExternalSecret returns NotFound → HyperDX Pod CrashLoop.

Recommendation: Change ClickStack ArgoCD Application sync-wave from -1 to 1 (after authentik at 0, after all observability infra at 0). Or accept eventual-consistency CrashLoop until first tf/authentik apply lands the OIDC secret. Sync-wave 1 is the cleaner choice.

Update the spec/plan to reflect sync-wave 1 instead of -1.

---

Item 8: Cilium status + NetworkPolicy decision

Status: ✅ Cilium is fully installed (CRDs present, dated 2026-05-11). No follow-up bead needed.

kubectl get crds | rg -i 'cilium' →
  ciliumcidrgroups.cilium.io
  ciliumclusterwideenvoyconfigs.cilium.io
  ciliumclusterwidenetworkpolicies.cilium.io
  ciliumendpoints.cilium.io
  ciliumenvoyconfigs.cilium.io
  (and others)

Active Cilium PR3 migration beads (hl-51o and children) exist in beads but are for the broader Calico→Cilium migration, not for clickstack-specific NetworkPolicy migration.

NetworkPolicy variant for PR1: CiliumNetworkPolicy (Cilium is the cluster CNI).

---

Item 9: MongoDB replica decision

Status: ✅ HA path IS available (chart mongodb.spec.members is configurable).

# Default from chart values.yaml:
mongodb:
  enabled: true
  spec:
    members: 1            # ← configurable; set to 3 for HA
    type: ReplicaSet
    version: "5.0.32"

Decision: HA path — override mongodb.spec.members: 3 in PR1’s valuesObject. Add a PodDisruptionBudget minAvailable: 2 if the chart doesn’t already include one.

Storage: 5 Gi × 3 = 15 GiB. Verify chart’s mongodb.spec.statefulSet.spec.volumeClaimTemplates[0].spec.resources.requests.storage default; override to 5 Gi if larger.

---

Item 10: ScrapeConfig CRDs + ServiceMonitor enumeration

Status: ✅ No ScrapeConfig CRDs, no additionalScrapeConfigs in chart values. Total 24 SM/PM resources.

Full enumeration (24 total):

Namespace	Kind	Name
argocd	ServiceMonitor	argocd-application-controller
argocd	ServiceMonitor	argocd-applicationset-controller
argocd	ServiceMonitor	argocd-dex-server
argocd	ServiceMonitor	argocd-repo-server
argocd	ServiceMonitor	argocd-server
authentik	ServiceMonitor	ak-outpost-ldap
cert-manager	ServiceMonitor	cert-manager
dolt	ServiceMonitor	dolt
prometheus	ServiceMonitor	kps-kube-prometheus-stack-alertmanager
prometheus	ServiceMonitor	kps-kube-prometheus-stack-apiserver
prometheus	ServiceMonitor	kps-kube-prometheus-stack-coredns
prometheus	ServiceMonitor	kps-kube-prometheus-stack-kube-controller-manager
prometheus	ServiceMonitor	kps-kube-prometheus-stack-kube-etcd
prometheus	ServiceMonitor	kps-kube-prometheus-stack-kube-proxy
prometheus	ServiceMonitor	kps-kube-prometheus-stack-kube-scheduler
prometheus	ServiceMonitor	kps-kube-prometheus-stack-kubelet
prometheus	ServiceMonitor	kps-kube-prometheus-stack-operator
prometheus	ServiceMonitor	kps-kube-prometheus-stack-prometheus
prometheus	ServiceMonitor	kps-kube-state-metrics
prometheus	ServiceMonitor	kps-prometheus-node-exporter
temporal	ServiceMonitor	temporal
vault	ServiceMonitor	vault
velero	ServiceMonitor	velero
nats	PodMonitor	nats

Translation scope for PR4a: 24 targets. Most are kube-prometheus-stack defaults that Vector can re-cover via kubernetes_discovery with a tag transform asserting job=<name> parity. App-specific (dolt, temporal, vault, velero, nats, authentik LDAP) need per-target scrape blocks. ArgoCD’s 5 are well-documented exporter endpoints.

---

Item 11: ExternalSecrets canonical pattern

Status: ✅ ESO present. Surprise: vault-agent-injector is ALSO installed (not absent as spec assumed).

external-secrets/external-secrets-operator                   1/1 Running
external-secrets/external-secrets-operator-cert-controller   1/1 Running
external-secrets/external-secrets-operator-webhook           1/1 Running
vault/vault-agent-injector                                   1/1 Running   (340 days old)

Decision: Use ExternalSecret + kubernetes-replicator pattern as spec’d — matches the majority of cluster’s secret-distribution pattern (Grafana, Karakeep, Tandoor, etc. all use ESO). Vault-agent injector being present is fine; we just don’t use it for ClickStack.

Spec correction: “Step 0 audit verifies vault-agent injector is absent” — actually it’s present. Update spec wording to “verify vault-agent injector is not the canonical pattern for this cluster” (which it isn’t).

---

Item 12: MetalLB IP allocation

Status: ✅ Pool has 10 IPs, 4 in use, 6 free.

MetalLB pool addresses:
  192.168.20.145-192.168.20.149  (5 IPs)
  192.168.20.155-192.168.20.159  (5 IPs)

Current allocations:
  192.168.20.145  traefik/traefik
  192.168.20.146  loki/loki-gateway          ← router-host L2 reference
  192.168.20.147  dolt/dolt
  192.168.20.148  mosquitto/mosquitto

Free: 192.168.20.149, 192.168.20.155-159

Chosen OTel Collector IP: 192.168.20.149 — same subrange as Loki gateway (which router-hosts already reach), guaranteed L2 reachability.

L2 reachability: confirmed — same MetalLB pool, same subnet as existing Loki gateway router-hosts use today.

DNS: none required. Router-host ansible config references the MetalLB IP directly (matches existing Loki gateway access pattern). HyperDX UI hostname hyperdx.fzymgc.house is the only DNS addition, served via Traefik IngressRoute as spec’d.

---

Item 13: Loki kubernetes-events history + disruption baseline

Status: ✅ Loki has full 30-day retention.

Label correction: spec/plan said {source="kubernetes-events"} but actual label is {job="kubernetes-events"} (verified via Loki /api/v1/labels). Update spec/plan to use job= label.

Queries run:

sum(count_over_time({job="kubernetes-events"}[30d]))
  → 164,957 events over 30 days = ~5,500/day average

sum(count_over_time({job="kubernetes-events"} |~ "(?i)drain|upgrade|reboot" [30d]))
  → 932 disruption events over 30 days = ~31/day average

sum(count_over_time({job="kubernetes-events"} |~ "(?i)drain|upgrade|reboot" [7d]))
  → 246 disruption events over 7 days = ~35/day average

BASELINE_WINDOW for dim-3: 30d.

Baseline disruption rate: ~31 events/day (30d average). For the dim-3 single-replica-Mongo win criterion, normalize eval-window Mongo-down incidents to this rate.

Note: “drain|upgrade|reboot” pattern produces ~31/day, which is high — this reflects normal cluster churn (Longhorn snapshot deletes, kubelet pod recreations, system-upgrade-controller activity). Refine the pattern at decision time if it’s too inclusive — e.g., |~ "(?i)NodeDrain|NodeReboot|UpgradeStarted" for tighter matching.

---

Summary of audit-driven decisions

Decision	Value	Source
Operators chart pin	clickstack/clickstack-operators 1.0.0	Item 3
ClickStack chart pin	clickstack/clickstack 2.1.1 (app 2.22.0)	Item 3
Number of operators landing	2 (Altinity CH + MongoDB Community), not 3	Item 3
OTel Collector form	Plain Deployment, no operator	Item 3
MongoDB topology	HA path (members: 3 + PDB minAvailable 2)	Item 9
MongoDB PVC size per replica	5 GiB (default per chart, verify on render)	Item 9
ClickHouse PVC size	TBD — defer to PR1, default 100 GiB if Item 6 measurement still missing	Item 6
Shadow Prom retention	TBD — measure max for: at PR4a authoring time	Spec
MetalLB IP for OTel Collector	192.168.20.149	Item 12
NetworkPolicy variant	CiliumNetworkPolicy	Item 8
BASELINE_WINDOW (dim 3)	30d	Item 13
Baseline disruption rate	~31 events/day	Item 13
Loki events label	{job="kubernetes-events"} (spec said source=, was wrong)	Item 13
PR1.5a (tf/authentik)	skipped — module already supports OIDC clients	Item 4
PR1.5b (tf/grafana)	needed — no grafana_data_source resource exists	Item 5
Sync-wave for clickstack Application	1 (not -1; authentik dep at 0)	Item 7
Vault-agent injector availability	Present (but unused per cluster convention)	Item 11

Spec / plan updates surfaced by this audit

The audit revealed several spec/plan assumptions that need correction:

1. Operator count: spec says “3 operators (Altinity CH, MongoDB, OTel)”. Reality: 2 operators. Update spec’s risk table and architecture section.
2. Sync-wave: spec locks -1. Should change to 1 to satisfy the authentik dependency, OR explicitly accept eventual-consistency CrashLoop until tf/authentik apply lands the OIDC client secret. Recommendation: change to 1.
3. Loki events label: spec/plan query says {source="kubernetes-events"}. Actual label is {job="kubernetes-events"}. Update plan’s PR4a label-parity scripts.
4. vault-agent-injector: spec says it’s absent. Actually present, just not the canonical secret pattern for this cluster. Update spec wording.
5. Repo URL: spec/plan have a placeholder https://clickhouse.github.io/clickstack-helm-charts (lowercase). Actual: https://clickhouse.github.io/ClickStack-helm-charts (capital S). Update plan’s Task 0.2 commands.
6. Two-chart pattern: spec/plan assumed one umbrella chart. Actually two charts: install clickstack-operators first (waits for operators ready), then clickstack. PR1 needs to deploy BOTH — either as two ArgoCD Applications (cleaner, allows independent upgrades) or as a single Application with two sources entries.

The 6 corrections above can be applied as a single follow-up spec amendment commit OR rolled into PR1’s diff as inline corrections.