Vector + ClickStack Parallel-Run Migration Implementation Plan

⚠️ SUPERSEDED (2026-05-13): The parity-gate approach (shadow-Prom + alert-diff + PR4a/PR4b cutover) was abandoned during PR #1075’s review iterations. After confirming router-hosts is OTel-native (its internal/server/ metrics layer already uses native OTel SDK + otlpmetricgrpc), the architectural direction pivoted to: OTel Collector owns metric scraping, Vector is logs-only. The parity-gate infrastructure (shadow Prom, alert-diff CronJob) was deleted in PR #1075. Migration to OTel Collector is tracked in beads hl-3i9 (router-hosts endpoint config), hl-abk (Firewalla otelcol-contrib), hl-1mf (in-cluster scrape collector + OTLP gateway), and hl-ebc (cutover). The dual-ship + shadow-Prom mechanics described below were the original plan; ClickStack and Vector logs landed as designed, but the metrics path described here is no longer the target shape. Retained for historical context.

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace Alloy with Vector as the universal telemetry agent on k8s + router-hosts, stand up ClickStack (ClickHouse + HyperDX + MongoDB + OTel Collector) as a parallel logs+metrics backend, dual-ship to old (Loki/Prom) and new (ClickHouse) for 1-3 months of comparison, then a committed decision rule selects Path A/B/C in a Phase 2 spec.

Architecture: Vector replaces Alloy on both k8s DaemonSet and router-hosts. Vector dual-ships logs to Loki + OTLP to ClickStack OTel Collector. Vector takes over Prometheus’s scrape role, dual-shipping metrics to Prom remote_write + OTLP. Disk-buffered sinks with per-sink when_full policy preserve alert fidelity. PR4a deploys shadow-Prom + shadow-AM via ArgoCD for 24h label-parity gate before PR4b disables Prom’s native scrape.

Tech Stack: Vector (k8s Helm chart + ansible role), ClickStack umbrella Helm chart (bundles ClickHouse via Altinity operator, MongoDB via MongoDB operator, OTel Collector, HyperDX), kube-prometheus-stack (existing + shadow release), Loki (unchanged), Grafana + grafana-operator (unchanged), Vault + ESO + kubernetes-replicator for secrets, cert-manager + Vault PKI for TLS, Authentik for OIDC, ArgoCD for GitOps, lefthook + rumdl + cocogitto for repo gates.

Spec: docs/engineering/specs/2026-05-11-vector-clickstack-migration-design.md

Audit-output (authoritative for cluster-state values): docs/engineering/specs/2026-05-11-vector-clickstack-audit-output.md

File Structure

This table maps every file the plan creates or modifies across all PRs. Each file has one responsibility; files that change together live together.

Created (new)

Path	Purpose	PR
`docs/engineering/specs/2026-05-11-vector-clickstack-audit-output.md`	Step 0 audit outputs (chart pin, Mongo topology, MetalLB IP, Loki baseline, etc.)	Audit PR
`argocd/cluster-app/templates/clickstack-operators.yaml`	ArgoCD Application for ClickStack operators chart (sync-wave 0)	PR1
`argocd/cluster-app/templates/clickstack.yaml`	ArgoCD Application for ClickStack chart (sync-wave 1)	PR1
`argocd/app-configs/clickstack/kustomization.yaml`	Kustomize wrapper for ClickStack app-config	PR1
`argocd/app-configs/clickstack/external-secret.yaml`	ExternalSecret materializing Vault `clickstack/*` creds	PR1
`argocd/app-configs/clickstack/certificate-hyperdx.yaml`	cert-manager Certificate for HyperDX UI	PR1
`argocd/app-configs/clickstack/certificate-otel.yaml`	cert-manager Certificate for OTel Collector OTLP endpoint	PR1
`argocd/app-configs/clickstack/ingressroute.yaml`	Traefik IngressRoute for `hyperdx.fzymgc.house`	PR1
`argocd/app-configs/clickstack/networkpolicy.yaml`	CiliumNetworkPolicy (Cilium is the cluster CNI)	PR1
`tf/vault/policy-clickstack.tf`	Vault policy for ClickStack service accounts	PR1
`tf/grafana/datasource-clickhouse.tf` (or conditional PR1.5b)	Grafana ClickHouse datasource	PR1
`tf/authentik/clickstack.tf` (or conditional PR1.5a)	Authentik OIDC client + Vault KV write	PR1
`docs/operations/clickstack.md`	Runbook (backups, token rotation, Mongo-down)	PR1
`docs/operations/clickstack-comparison.md`	Eval doc skeleton (10 queries, divergence catalogue, decision criteria)	PR1
`argocd/app-configs/monitoring-vector/kustomization.yaml`	Kustomize wrapper for Vector k8s DaemonSet	PR2b
`argocd/app-configs/monitoring-vector/vector.yaml`	Vector `config.yaml` (kubernetes_logs + dual sinks)	PR2b
`argocd/app-configs/monitoring-vector/external-secret.yaml`	ExternalSecret for `otel-ingest-token` + replicator annotation	PR2b
`argocd/cluster-app/templates/monitoring-vector.yaml`	ArgoCD Application for Vector k8s	PR2b
`ansible/roles/vector/defaults/main.yml`	Vector role defaults	PR3
`ansible/roles/vector/handlers/main.yml`	Vector role handlers (systemd restart)	PR3
`ansible/roles/vector/meta/main.yml`	Vector role metadata	PR3
`ansible/roles/vector/tasks/main.yml`	Vector role tasks (install, configure, enable)	PR3
`ansible/roles/vector/templates/vector.yaml.j2`	Vector config template for router-hosts	PR3
`ansible/roles/vector/templates/vector.service.j2`	systemd unit template	PR3
`argocd/app-configs/monitoring-prometheus-shadow/kustomization.yaml`	Kustomize for shadow stack	PR4a
`argocd/app-configs/monitoring-prometheus-shadow/alert-diff-cronjob.yaml`	Alert-diff CronJob + RBAC	PR4a
`argocd/app-configs/monitoring-prometheus-shadow/promtool-job.yaml`	Promtool parity Job + RBAC	PR4a
`argocd/cluster-app/templates/monitoring-prometheus-shadow.yaml`	ArgoCD Application for shadow kps release	PR4a
`docs/operations/clickstack-vector-scrape-targets.md`	ServiceMonitor → Vector scrape-config translation table	PR4a

Modified (existing)

Path	Change	PR
`argocd/cluster-app/templates/monitoring-alloy.yaml`	Add `resources-finalizer.argocd.argoproj.io` to metadata.finalizers	PR2a
`argocd/cluster-app/templates/monitoring-alloy.yaml`	Delete file	PR2b
`argocd/cluster-app/templates/monitoring-alloy-ingest.yaml`	Delete file	PR2b
`argocd/app-configs/monitoring-alloy/` (entire dir)	Delete	PR2b
`ansible/roles/alloy/` (entire dir)	Delete	PR3
`ansible/inventory/*.yml` and playbook references to `alloy` role	Replace `alloy` → `vector`	PR3
`argocd/cluster-app/templates/monitoring-prometheus.yaml`	Disable native scrape: `serviceMonitorSelector` / `podMonitorSelector` match nothing; remove built-in scrape configs	PR4b
Various chart `valuesObject` blocks in `argocd/cluster-app/templates/*.yaml` (kube-prometheus-stack, cnpg, temporal-server, traefik, monitoring-loki)	`serviceMonitor.enabled: false`	PR4b
`mkdocs.yml`	Add nav entries for `operations/clickstack.md`, `clickstack-comparison.md`, `clickstack-vector-scrape-targets.md`	PR1 (first two), PR4a (third)
`docs/reference/services.md`	Add HyperDX entry	PR1
`docs/reference/secrets.md`	Add `fzymgc-house/cluster/clickstack` entry	PR1
`argocd/app-configs/velero/backup-schedule.yaml`	Verify `clickstack` is NOT in `excludedNamespaces` (no change expected; verify-only)	PR1

Pre-flight: worktree + lefthook

Step 1: Create worktree for the migration

Run: git worktree add .worktrees/feat-vector-clickstack -b feat/vector-clickstack origin/main

Step 2: Enter worktree and install hooks

Run: cd .worktrees/feat-vector-clickstack && lefthook install

Expected: lefthook v2.x install messages; pre-commit + commit-msg hooks installed.

Step 3: Install bd hooks if applicable

Run: [[ -f .beads/config.yaml ]] && bd hooks install --chain || echo "no beads config"

PR0: Step 0 Audit (separate PR, merges to main first)

Task 0.1: Create the audit-output document

Files:

Create: docs/engineering/specs/2026-05-11-vector-clickstack-audit-output.md
Step 1: Create the audit-output file with empty section structure

# Vector + ClickStack — Step 0 Audit Outputs

**Date run:** YYYY-MM-DD
**Operator:** Sean
**Cluster:** fzymgc-house
**Spec:** `2026-05-11-vector-clickstack-migration-design.md`

## Item 1: Conflicting CRDs / operators

(output of `kubectl get crds | rg -i 'clickhouse|mongodb|opentelemetry'`)

## Item 2: Existing OTel deployment overlap

(output of `kubectl get pods -A | rg -i 'otel|opentelemetry'`)

## Item 3: Chart version pin

ClickStack chart version selected: `<pin>`
Rationale: latest GA per argocd/CLAUDE.md "Helm Chart Constraints"
Pulled via: `helm pull oci://<repo>/clickstack --version <pin>`
Render `kind:` list (paste output below):

(kinds rendered by chart)

## Item 4: tf/authentik OIDC capability

Has `authentik_provider_oauth2` resource: yes / no
Decision: PR1.5a needed / skipped

## Item 5: tf/grafana datasource + plugin capability

Has `grafana_data_source` resource: yes / no
Plugin install mechanism: yes / no
Decision: PR1.5b needed / skipped

## Item 6: Loki uncompressed ingest baseline

Query: `avg_over_time(rate(loki_distributor_bytes_received_total[5m])[7d:5m])`
Result: ___ bytes/sec = ___ GiB/day uncompressed

## Item 7: Sync-wave -1 occupancy

(output of `rg -o 'argocd\.argoproj\.io/sync-wave: "(-?[0-9]+)"' argocd/cluster-app/templates/ -r '$1 $0' | sort -n | head -20`)

Conclusion: clean / conflict

## Item 8: Cilium-migration follow-up bead

Bead ID: ___ (or "filed as <new-id>")

## Item 9: MongoDB replica decision

Chart deploys Mongo via: MongoDBCommunity CR with members: 3 / single Deployment / other
Decision: HA path (members 3) / single-replica path

## Item 10: ScrapeConfig + additionalScrapeConfigs enumeration

(output of `kubectl get scrapeconfigs.monitoring.coreos.com,probes.monitoring.coreos.com -A`)
(output of `rg -n 'additionalScrapeConfigs' argocd/cluster-app/templates/monitoring-prometheus.yaml`)

Translation list seed: see PR4a annex

## Item 11: ExternalSecrets canonical pattern

(output of `kubectl get crds | rg -i 'external-secrets'`)
(output of `kubectl get deploy -A | rg -i 'vault-agent-injector|external-secrets'`)

Conclusion: ESO present, vault-agent-injector absent → ExternalSecret + kubernetes-replicator pattern confirmed

## Item 12: MetalLB IP allocation

(output of `kubectl get ipaddresspools.metallb.io -A -o yaml | rg -A 10 'addresses:'`)
(output of `kubectl get services -A -o jsonpath='{range .items[*]}{.status.loadBalancer.ingress[*].ip}{"\n"}{end}' | sort -u`)

Existing Loki gateway IP (router-host reference): ___
Chosen OTel Collector IP: ___
L2 reachability confirmed: yes / no

## Item 13: Loki kubernetes-events history

Query: `count_over_time({job="kubernetes-events"}[30d])`
Result: ___ events over 30d
Available window: ___ days
BASELINE_WINDOW for dim-3: <30d if available, else actual>

## Disruption baseline (single-replica Mongo only)

Query: `count_over_time({job="kubernetes-events"} |~ "(?i)node draining|Cordoned|Drained" [<BASELINE_WINDOW>])`
Result: ___ disruption events in ___ days
Rate: ___ per day

Step 2: Commit the empty audit-output skeleton

Run from worktree: git add docs/engineering/specs/2026-05-11-vector-clickstack-audit-output.md && git commit -m "docs(specs): capture Step 0 audit outputs for vector + ClickStack (skeleton)"

Expected: lefthook clean, commit-msg validates.

Task 0.2: Run audit and fill in outputs

Files:

Modify: docs/engineering/specs/2026-05-11-vector-clickstack-audit-output.md
Step 1: Run audit item 1 (CRDs / operators)

Run: kubectl get crds | rg -i 'clickhouse|mongodb|opentelemetry'

Paste output under “Item 1” in the audit-output doc. Expected: empty (no preexisting installs).

Step 2: Run audit item 2 (existing OTel deployments)

Run: kubectl get pods -A | rg -i 'otel|opentelemetry'

Paste under “Item 2”. Expected: empty.

Step 3: Run audit item 3 (chart version + render)

Run:

helm repo add clickstack https://clickhouse.github.io/clickstack-helm-charts || true
helm repo update
helm search repo clickstack/clickstack --versions | head -5

Pick latest GA. Pin in doc.

Then:

mkdir -p /tmp/clickstack-audit && cd /tmp/clickstack-audit
helm pull clickstack/clickstack --version <pinned>
helm template clickstack clickstack-<pinned>.tgz | rg '^kind:' | sort -u

Paste both outputs under “Item 3”.

Step 4: Run audit item 4 (tf/authentik OIDC capability)

Run: rg -l 'authentik_provider_oauth2' tf/authentik/

If output non-empty → PR1.5a skipped, record in doc. If empty → PR1.5a required, record in doc.

Step 5: Run audit item 5 (tf/grafana datasource capability)

Run:

rg -l 'grafana_data_source' tf/grafana/
rg -l 'extraPlugins|grafana_plugin' tf/grafana/ argocd/cluster-app/templates/grafana.yaml argocd/app-configs/grafana/

Record findings; decide PR1.5b skipped / required.

Step 6: Run audit item 6 (Loki ingest baseline)

Query Prometheus via Grafana Explore or curl:

PROM_URL=$(kubectl get svc -n prometheus kps-prometheus -o jsonpath='{.spec.clusterIP}'):9090
# From a debug pod or kubectl-proxy session:
curl -s "http://${PROM_URL}/api/v1/query?query=avg_over_time(rate(loki_distributor_bytes_received_total%5B5m%5D)%5B7d:5m%5D)" | jq

Record bytes/sec, compute GiB/day.

Step 7: Run audit item 7 (sync-wave check)

Run: rg -o 'argocd\.argoproj\.io/sync-wave: "(-?[0-9]+)"' argocd/cluster-app/templates/ -r '$1 $0' | sort -n | head -20

Paste. Manually verify: clickstack at -1 has no infra dependencies also at -1 (cert-manager, ESO, vault, kubernetes-replicator, authentik should all be at -2 or earlier).

Step 8: Run audit item 8 (Cilium follow-up bead)

Run: bd list --status=open 2>/dev/null | rg -i 'network.?policy|cilium.*policy'

If a relevant bead exists, record its ID. If not, file one: bd create --title="Migrate clickstack NetworkPolicy to CiliumNetworkPolicy post-Cilium-PR3" --description="Spec defers Cilium-vs-stdlib decision to PR1 start. If stdlib NetworkPolicy is chosen, migrate post-Cilium." --type=task --priority=3

Record bead ID.

Step 9: Run audit item 9 (MongoDB topology)

Run: helm template clickstack /tmp/clickstack-audit/clickstack-<pinned>.tgz | rg -A 10 'kind: MongoDB(Community)?|kind: Deployment.*mongo'

If CR with configurable members → HA path; if single Deployment → single-replica path. Record decision.

Step 10: Run audit item 10 (ScrapeConfig CRDs + additionalScrapeConfigs)

Run:

kubectl get scrapeconfigs.monitoring.coreos.com,probes.monitoring.coreos.com -A 2>/dev/null
rg -n 'additionalScrapeConfigs|additionalScrapeConfigsSecret' argocd/cluster-app/templates/monitoring-prometheus.yaml

Paste outputs.

Step 11: Run audit item 11 (ESO vs vault-agent-injector)

Run:

kubectl get crds | rg -i 'external-secrets'
kubectl get deploy -A | rg -i 'vault-agent-injector|external-secrets'

Confirm: ESO present, vault-agent-injector absent.

Step 12: Run audit item 12 (MetalLB pool + chosen IP)

Run:

kubectl get ipaddresspools.metallb.io -A -o yaml | rg -A 10 'addresses:'
kubectl get services -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}={.status.loadBalancer.ingress[*].ip}{"\n"}{end}' | sort -u
kubectl get svc -n loki loki-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Pick an unused IP in the same MetalLB pool as the Loki gateway IP. Record in doc.

Step 13: Run audit item 13 (Loki kubernetes-events history)

In Grafana Explore against Loki:

count_over_time({job="kubernetes-events"}[30d])

If result > 0 over 30d → BASELINE_WINDOW = 30d. Otherwise, query progressively shorter windows ([14d], [7d]) until non-zero; that’s the BASELINE_WINDOW.

Then run the disruption-baseline query:

count_over_time({job="kubernetes-events"} |~ "(?i)node draining|Cordoned|Drained" [<BASELINE_WINDOW>])

Record count + window in doc.

Step 14: Commit completed audit

Run: git add docs/engineering/specs/2026-05-11-vector-clickstack-audit-output.md && git commit -m "docs(specs): complete Step 0 audit for vector + ClickStack"

Step 15: Push branch and open PR

Run:

git push -u origin feat/vector-clickstack
gh pr create --title "docs(specs): Step 0 audit for vector + ClickStack migration" --body "$(cat <<'EOF'
## Summary
- Captures Step 0 audit outputs for the vector + ClickStack migration design
- All 13 audit items run; outputs documented for downstream PR reference
- Records pin (chart version), MongoDB replica decision, MetalLB IP, Loki baseline, ESO confirmation

## Test plan
- [ ] Verify all 13 items populated with command outputs
- [ ] Verify decisions recorded (PR1.5a need, PR1.5b need, Mongo HA path)
- [ ] Spec implementer can use this doc as PR1's input

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Step 16: Merge audit PR to main after review

Once approved, merge via GitHub UI or gh pr merge --squash. The audit doc now lives on main. Subsequent feature branches (PR1.5a/b, PR1, etc.) branch from this updated main.

PR1.5a: tf/authentik OIDC client support — SKIPPED

Audit Item 4 confirmed tf/authentik already supports authentik_provider_oauth2 (used by grafana.tf, kubernetes-oidc.tf, karakeep.tf, etc.). This PR is skipped. The HyperDX OIDC client is declared directly in tf/authentik/clickstack.tf via Task 1.4. The section below is preserved for historical context only — do not execute its tasks.

Task 1.5a.1: Create new branch from updated main

Step 1: Refresh main and create branch

Run from worktree:

git fetch origin
git checkout main && git pull
git checkout -b feat/tf-authentik-oidc-clients

Task 1.5a.2: Add authentik_provider_oauth2 resource scaffolding

Files:

Create: tf/authentik/oauth2_clients.tf
Modify: tf/authentik/variables.tf
Step 1: Add variable definition

Edit tf/authentik/variables.tf to append:

variable "oauth2_clients" {
  description = "OAuth2/OIDC clients managed by this module"
  type = map(object({
    redirect_uris   = list(string)
    scope_mapping_names = list(string)
    group_mapping_name  = optional(string)
  }))
  default = {}
}

Step 2: Create the resource file

Write tf/authentik/oauth2_clients.tf:

# OAuth2/OIDC clients (e.g., HyperDX, future services).
# Per-client secret is written to Vault by the consumer module.

resource "authentik_provider_oauth2" "client" {
  for_each = var.oauth2_clients

  name           = each.key
  client_id      = each.key
  authorization_flow = data.authentik_flow.default_authorization.id
  redirect_uris  = each.value.redirect_uris
  property_mappings = [
    for name in each.value.scope_mapping_names :
    data.authentik_scope_mapping.by_name[name].id
  ]
}

resource "authentik_application" "client" {
  for_each = var.oauth2_clients

  name              = each.key
  slug              = each.key
  protocol_provider = authentik_provider_oauth2.client[each.key].id
  group             = each.value.group_mapping_name != null ? data.authentik_group.by_name[each.value.group_mapping_name].pk : null
}

# Data sources for scope mapping name → id resolution
data "authentik_scope_mapping" "by_name" {
  for_each = toset(flatten([
    for c in values(var.oauth2_clients) : c.scope_mapping_names
  ]))
  name = each.key
}

data "authentik_group" "by_name" {
  for_each = toset(compact([
    for c in values(var.oauth2_clients) : c.group_mapping_name
  ]))
  name = each.key
}

data "authentik_flow" "default_authorization" {
  slug = "default-provider-authorization-implicit-consent"
}

Step 3: Add outputs

Append to tf/authentik/outputs.tf (create if missing):

output "oauth2_client_secrets" {
  description = "Map of client_id → client_secret for consumers to read"
  value       = { for k, v in authentik_provider_oauth2.client : k => v.client_secret }
  sensitive   = true
}

output "oauth2_client_ids" {
  description = "Map of client_id → client_id (for consistency)"
  value       = { for k, v in authentik_provider_oauth2.client : k => v.client_id }
}

Step 4: Run terraform validate

Run: terraform -chdir=tf/authentik init -upgrade && terraform -chdir=tf/authentik validate

Expected: Success! The configuration is valid.

Step 5: Commit

Run:

git add tf/authentik/
git commit -m "feat(tf/authentik): add OAuth2/OIDC client resource type

Adds authentik_provider_oauth2 + authentik_application resources keyed
by a new oauth2_clients map variable. Used by HyperDX in the upcoming
ClickStack migration; reusable for future OIDC consumers.

Exports oauth2_client_secrets and oauth2_client_ids as outputs for
consumer modules to wire into Vault writes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Step 6: Push and open PR

Run:

git push -u origin feat/tf-authentik-oidc-clients
gh pr create --title "feat(tf/authentik): add OAuth2/OIDC client resource type" --body "$(cat <<'EOF'
## Summary
- Adds `authentik_provider_oauth2` + `authentik_application` resources to tf/authentik
- Keyed by new `oauth2_clients` map variable
- Exports `oauth2_client_secrets` (sensitive) and `oauth2_client_ids` outputs

Required prerequisite for the vector + ClickStack migration (HyperDX OIDC client).

## Test plan
- [ ] `terraform validate` passes
- [ ] `terraform plan` is no-op (no clients defined yet)
- [ ] Consumer module (PR1 of vector+clickstack) can declare a client

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Step 7: Merge after review

PR1.5b: tf/grafana ClickHouse datasource support — REQUIRED

Audit Item 5 confirmed tf/grafana has no grafana_data_source resource (only a data source for reading existing datasources). This PR is required before PR1’s Grafana ClickHouse datasource wiring in Task 1.5 can succeed.

Task 1.5b.1: Branch and add datasource resource

Step 1: Branch from updated main

Run: git checkout main && git pull && git checkout -b feat/tf-grafana-clickhouse-datasource

Step 2: Add Grafana ClickHouse plugin to chart valuesObject

Edit argocd/cluster-app/templates/grafana.yaml — find the Helm valuesObject block; add the plugin to its plugins list:

        plugins:
          - grafana-clickhouse-datasource  # added for vector + ClickStack migration
          # ...existing plugins below...

(If no plugins: list exists, add it at the same indentation as grafana.ini: within the chart values.)

Step 3: Create datasource Terraform file

Write tf/grafana/datasources.tf:

# ClickHouse datasource for HyperDX-evaluation comparison panels.
# Endpoint resolved at apply time from the cluster.

variable "clickhouse_datasource_url" {
  description = "ClickHouse HTTP URL (e.g., https://clickhouse.clickstack.svc:8443)"
  type        = string
  default     = ""
}

variable "clickhouse_datasource_user" {
  description = "ClickHouse read-only user (provisioned by ClickStack chart)"
  type        = string
  default     = "default"
}

resource "grafana_data_source" "clickhouse" {
  count = var.clickhouse_datasource_url != "" ? 1 : 0

  type = "grafana-clickhouse-datasource"
  name = "ClickHouse"
  url  = var.clickhouse_datasource_url

  json_data_encoded = jsonencode({
    defaultDatabase = "default"
    protocol        = "http"
    secure          = true
    username        = var.clickhouse_datasource_user
  })

  secure_json_data_encoded = jsonencode({
    password = var.clickhouse_datasource_password
  })
}

variable "clickhouse_datasource_password" {
  description = "ClickHouse read-only password (from Vault)"
  type        = string
  sensitive   = true
  default     = ""
}

Step 4: Run terraform validate

Run: terraform -chdir=tf/grafana init -upgrade && terraform -chdir=tf/grafana validate

Expected: Success!

Step 5: Commit

git add argocd/cluster-app/templates/grafana.yaml tf/grafana/datasources.tf
git commit -m "feat(tf/grafana): add ClickHouse datasource + plugin support

Adds grafana-clickhouse-datasource plugin to the Grafana Helm chart's
plugins list and a grafana_data_source Terraform resource gated on a
non-empty clickhouse_datasource_url variable.

Required prerequisite for the vector + ClickStack migration
(comparison panels alongside HyperDX UI).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Step 6: Push, open PR, merge after review

git push -u origin feat/tf-grafana-clickhouse-datasource
gh pr create --title "feat(tf/grafana): add ClickHouse datasource + plugin support" --body "## Summary
- Adds grafana-clickhouse-datasource plugin to Grafana chart
- Adds grafana_data_source resource (gated on non-empty URL var)

Prerequisite for vector + ClickStack migration.

🤖 Generated with [Claude Code](https://claude.com/claude-code)"

PR1: Deploy ClickStack storage tier + comparison-runbook skeleton

Task 1.1: Branch from updated main

Step 1: Branch

Run: git checkout main && git pull && git checkout -b feat/clickstack-deploy

Task 1.2: Vault policy

Files:

Create: tf/vault/policy-clickstack.tf
Step 1: Write Vault policy + auth roles

# Vault policies + Kubernetes auth roles for ClickStack components.

resource "vault_policy" "clickstack_hyperdx" {
  name   = "clickstack-hyperdx"
  policy = <<-EOT
    path "fzymgc-house/cluster/clickstack/*" {
      capabilities = ["read"]
    }
  EOT
}

resource "vault_policy" "clickstack_otel_collector" {
  name   = "clickstack-otel-collector"
  policy = <<-EOT
    path "fzymgc-house/cluster/clickstack/otel_ingest_token" {
      capabilities = ["read"]
    }
    path "fzymgc-house/cluster/clickstack/admin_password" {
      capabilities = ["read"]
    }
  EOT
}

resource "vault_policy" "clickstack_clickhouse" {
  name   = "clickstack-clickhouse"
  policy = <<-EOT
    path "fzymgc-house/cluster/clickstack/admin_password" {
      capabilities = ["read"]
    }
  EOT
}

resource "vault_kubernetes_auth_backend_role" "clickstack_hyperdx" {
  backend                          = "kubernetes"
  role_name                        = "clickstack-hyperdx"
  bound_service_account_names      = ["clickstack-hyperdx"]
  bound_service_account_namespaces = ["clickstack"]
  token_policies                   = [vault_policy.clickstack_hyperdx.name]
  token_ttl                        = 3600
}

resource "vault_kubernetes_auth_backend_role" "clickstack_otel_collector" {
  backend                          = "kubernetes"
  role_name                        = "clickstack-otel-collector"
  bound_service_account_names      = ["clickstack-otel-collector"]
  bound_service_account_namespaces = ["clickstack"]
  token_policies                   = [vault_policy.clickstack_otel_collector.name]
  token_ttl                        = 3600
}

resource "vault_kubernetes_auth_backend_role" "clickstack_clickhouse" {
  backend                          = "kubernetes"
  role_name                        = "clickstack-clickhouse"
  bound_service_account_names      = ["clickstack-clickhouse"]
  bound_service_account_namespaces = ["clickstack"]
  token_policies                   = [vault_policy.clickstack_clickhouse.name]
  token_ttl                        = 3600
}

Step 2: Validate

Run: terraform -chdir=tf/vault validate

Expected: Success.

Step 3: Commit

git add tf/vault/policy-clickstack.tf
git commit -m "feat(tf/vault): add ClickStack policies + k8s auth roles"

Task 1.3: Bootstrap Vault secrets (manual, runbook step)

Step 1: Generate and store secrets via vault CLI

This is a one-time manual step (not in Terraform — secrets must be generated, not declared). Run from a workstation with vault token:

vault kv put fzymgc-house/cluster/clickstack \
  admin_password="$(openssl rand -base64 32)" \
  mongo_admin_password="$(openssl rand -base64 32)" \
  mongo_replicaset_keyfile="$(openssl rand -base64 756)" \
  hyperdx_bootstrap_secret="$(openssl rand -base64 32)" \
  hyperdx_oidc_client_secret="placeholder-overwritten-by-authentik" \
  otel_ingest_token="$(openssl rand -base64 48)"

Verify: vault kv get fzymgc-house/cluster/clickstack | rg created_time

Step 2: Document the bootstrap step in the runbook (task 1.6 will create the runbook file)

Task 1.4: Authentik OIDC client + Vault secret write

tf/authentik already supports authentik_provider_oauth2 (used by grafana.tf, kubernetes-oidc.tf, karakeep.tf). This task adds a new file modeled on tf/authentik/grafana.tf.

Files:

Create: tf/authentik/clickstack.tf
Step 1: Write the OIDC client + Vault KV write

Write tf/authentik/clickstack.tf:

# HyperDX OIDC integration
# Pattern mirrors tf/authentik/grafana.tf.

resource "authentik_group" "hyperdx_user" {
  name = "hyperdx-user"
}

resource "authentik_provider_oauth2" "hyperdx" {
  name      = "Provider for HyperDX"
  client_id = "clickstack-hyperdx"

  authorization_flow    = data.authentik_flow.default_provider_authorization_explicit_consent.id
  invalidation_flow     = data.authentik_flow.default_provider_invalidation_flow.id
  access_token_validity = "minutes=5"

  allowed_redirect_uris = [
    {
      matching_mode = "strict"
      url           = "https://hyperdx.fzymgc.house/auth/sso/callback"
    }
  ]

  property_mappings = [
    data.authentik_property_mapping_provider_scope.openid.id,
    data.authentik_property_mapping_provider_scope.email.id,
    data.authentik_property_mapping_provider_scope.profile.id,
  ]

  signing_key = data.authentik_certificate_key_pair.tls.id
}

resource "authentik_application" "hyperdx" {
  name              = "HyperDX"
  slug              = "hyperdx"
  protocol_provider = authentik_provider_oauth2.hyperdx.id
  meta_launch_url   = "https://hyperdx.fzymgc.house"
  meta_description  = "ClickStack observability UI (eval)"
  policy_engine_mode = "any"
}

resource "authentik_policy_binding" "hyperdx_access" {
  target = authentik_application.hyperdx.uuid
  group  = authentik_group.hyperdx_user.id
  order  = 0
}

# Read existing clickstack secrets (admin_password, mongo_*, otel_ingest_token)
# bootstrapped via vault CLI in Task 1.3 — merge new fields.
data "vault_kv_secret_v2" "clickstack_existing" {
  mount = "secret"
  name  = "fzymgc-house/cluster/clickstack"
}

resource "vault_kv_secret_v2" "clickstack" {
  mount = "secret"
  name  = "fzymgc-house/cluster/clickstack"
  data_json = jsonencode(merge(
    try(nonsensitive(data.vault_kv_secret_v2.clickstack_existing.data), {}),
    {
      hyperdx_oidc_client_id     = authentik_provider_oauth2.hyperdx.client_id
      hyperdx_oidc_client_secret = authentik_provider_oauth2.hyperdx.client_secret
    }
  ))
}

The data blocks (authentik_flow, authentik_property_mapping_provider_scope, authentik_certificate_key_pair) are already declared in tf/authentik per grafana.tf usage — no new data sources to add.

Step 2: Validate

Run: terraform -chdir=tf/authentik validate

Expected: Success!

Step 3: Commit

git add tf/authentik/clickstack.tf
git commit -m "feat(tf/authentik): add HyperDX OIDC client + Vault KV write"

Step 4: Apply before PR1 merges

The tf/authentik apply populates hyperdx_oidc_client_secret in Vault. PR1’s clickstack-secret ExternalSecret reads it. If PR1 merges before this apply, the ExternalSecret returns NotFound and HyperDX CrashLoops. Run via HCP Terraform OR locally with appropriate credentials:

terraform -chdir=tf/authentik apply
# Verify:
vault kv get fzymgc-house/cluster/clickstack | rg hyperdx_oidc_client_secret

Expected: secret has non-empty hyperdx_oidc_client_secret field.

Task 1.5: Grafana ClickHouse datasource variables

Files:

Modify: tf/grafana/datasources.tf (created in PR1.5b or already present)
Modify: tf/grafana/main.tf or wherever vars are wired
Step 1: Wire the datasource variables

Edit tf/grafana/datasources.tf (or the equivalent place where existing datasources are configured) to set:

# In the consumer block of the module that calls tf/grafana, set:
clickhouse_datasource_url      = "https://clickhouse.clickstack.svc.cluster.local:8443"
clickhouse_datasource_user     = "default"
clickhouse_datasource_password = data.vault_kv_secret_v2.clickstack.data["admin_password"]

If tf/grafana already had datasource resources without the new variables, add the variable wiring; the resource conditional ensures it’s a no-op until populated.

Step 2: Validate

Run: terraform -chdir=tf/grafana validate

Step 3: Commit

git add tf/grafana/
git commit -m "feat(tf/grafana): wire ClickHouse datasource for HyperDX comparison"

Task 1.6: ClickStack ArgoCD app-config files

Files:

Create: argocd/app-configs/clickstack/kustomization.yaml
Create: argocd/app-configs/clickstack/external-secret.yaml
Create: argocd/app-configs/clickstack/certificate-hyperdx.yaml
Create: argocd/app-configs/clickstack/certificate-otel.yaml
Create: argocd/app-configs/clickstack/ingressroute.yaml
Create: argocd/app-configs/clickstack/networkpolicy.yaml
Step 1: kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: clickstack

resources:
  - external-secret.yaml
  - certificate-hyperdx.yaml
  - certificate-otel.yaml
  - ingressroute.yaml
  - networkpolicy.yaml

Step 2: external-secret.yaml

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: clickstack-secret
  namespace: clickstack
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: clickstack-secret
    creationPolicy: Owner
  data:
    - secretKey: admin_password
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: admin_password
    - secretKey: mongo_admin_password
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: mongo_admin_password
    - secretKey: mongo_replicaset_keyfile
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: mongo_replicaset_keyfile
    - secretKey: hyperdx_bootstrap_secret
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: hyperdx_bootstrap_secret
    - secretKey: hyperdx_oidc_client_secret
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: hyperdx_oidc_client_secret
    - secretKey: otel_ingest_token
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: otel_ingest_token

Step 3: certificate-hyperdx.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hyperdx-tls
  namespace: clickstack
spec:
  secretName: hyperdx-tls
  duration: 2160h  # 90d
  renewBefore: 720h  # 30d
  issuerRef:
    name: vault-pki
    kind: ClusterIssuer
  commonName: hyperdx.fzymgc.house
  dnsNames:
    - hyperdx.fzymgc.house

(Verify vault-pki ClusterIssuer name matches what argocd/app-configs/grafana/certificate.yaml uses; adjust accordingly.)

Step 4: certificate-otel.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: otel-collector-tls
  namespace: clickstack
spec:
  secretName: otel-collector-tls
  duration: 2160h
  renewBefore: 720h
  issuerRef:
    name: vault-pki
    kind: ClusterIssuer
  commonName: otel-collector.clickstack.svc.cluster.local
  dnsNames:
    - otel-collector.clickstack.svc.cluster.local
    - otel-collector.clickstack.svc
    - "<MetalLB IP from audit item 12>"  # for router-host TLS validation

Replace <MetalLB IP from audit item 12> with the actual chosen IP from the audit doc.

Step 5: ingressroute.yaml

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: hyperdx
  namespace: clickstack
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`hyperdx.fzymgc.house`)
      kind: Rule
      services:
        - name: hyperdx
          port: 8080
  tls:
    secretName: hyperdx-tls

(Adjust service name/port to match what the ClickStack chart names HyperDX; verify via helm template.)

Step 6: networkpolicy.yaml (CiliumNetworkPolicy)

Cilium is the cluster CNI (audit Item 8). Ship a CiliumNetworkPolicy:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: clickstack-default
  namespace: clickstack
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vector
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: traefik
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: grafana
    # Router-hosts → OTel Collector via MetalLB LoadBalancer IP — Cilium
    # routes external traffic; allow ingress from world to OTLP port only.
    - fromEntities:
        - world
      toPorts:
        - ports:
            - port: "4317"
              protocol: TCP
  egress:
    - toEntities:
        - cluster
    - toEntities:
        - kube-apiserver
    - toEntities:
        - world  # for fetching Authentik OIDC discovery, Vault PKI, S3
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
            - port: "9200"  # MinIO/S3 if used for CH backup later
              protocol: TCP

Step 7: Commit app-config files

git add argocd/app-configs/clickstack/
git commit -m "feat(clickstack): add app-config resources (ExternalSecret, certs, ingress, netpol)"

Task 1.7: ClickStack ArgoCD Applications (two charts)

ClickStack ships as two Helm charts that must be installed in order:

clickstack/clickstack-operators 1.0.0 — installs Altinity ClickHouse Operator + MongoDB Community Operator + their CRDs.
clickstack/clickstack 2.1.1 (app 2.22.0) — installs the stack; creates ClickHouseCluster, KeeperCluster, and MongoDBCommunity CRs that consume the operators from chart 1.

PR1 ships two ArgoCD Applications: operators at sync-wave 0, clickstack at sync-wave 1.

Files:

Create: argocd/cluster-app/templates/clickstack-operators.yaml
Create: argocd/cluster-app/templates/clickstack.yaml
Step 1: Write the operators Application

Write argocd/cluster-app/templates/clickstack-operators.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: clickstack-operators
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: observability
  source:
    chart: clickstack-operators
    repoURL: https://clickhouse.github.io/ClickStack-helm-charts
    targetRevision: 1.0.0
    helm:
      releaseName: clickstack-operators
      # Operators chart uses sensible defaults; no overrides needed.
      valuesObject: {}
  destination:
    server: https://kubernetes.default.svc
    namespace: clickstack-operators
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true
      - RespectIgnoreDifferences=true

Step 2: Write the clickstack Application

Write argocd/cluster-app/templates/clickstack.yaml. The valuesObject schema below uses the chart’s actual key paths (verified via helm show values clickstack/clickstack --version 2.1.1):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: clickstack
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: observability
  sources:
    - chart: clickstack
      repoURL: https://clickhouse.github.io/ClickStack-helm-charts
      targetRevision: 2.1.1
      helm:
        releaseName: cs
        valuesObject:
          hyperdx:
            config:
              FRONTEND_URL: "https://hyperdx.fzymgc.house"
              OAUTH2_CLIENT_ID: "clickstack-hyperdx"
              OAUTH2_AUTH_URL: "https://authentik.fzymgc.house/application/o/authorize/"
              OAUTH2_TOKEN_URL: "https://authentik.fzymgc.house/application/o/token/"
              OAUTH2_USERINFO_URL: "https://authentik.fzymgc.house/application/o/userinfo/"
              OAUTH2_SCOPES: "openid email profile"
            deployment:
              useExistingConfigSecret: true
              existingConfigSecret: clickstack-secret
              replicas: 1
              resources:
                requests: { cpu: 100m, memory: 256Mi }
                limits:   { cpu: 500m, memory: 512Mi }
            service:
              type: ClusterIP
            ingress:
              enabled: false  # Traefik IngressRoute used instead
            podDisruptionBudget:
              enabled: false
          mongodb:
            enabled: true
            spec:
              members: 3
              type: ReplicaSet
              version: "5.0.32"
              security:
                authentication:
                  modes: ["SCRAM"]
              statefulSet:
                spec:
                  volumeClaimTemplates:
                    - metadata: { name: data-volume }
                      spec:
                        storageClassName: longhorn
                        accessModes: ["ReadWriteOnce"]
                        resources: { requests: { storage: 5Gi } }
                    - metadata: { name: logs-volume }
                      spec:
                        storageClassName: longhorn
                        accessModes: ["ReadWriteOnce"]
                        resources: { requests: { storage: 2Gi } }
            podDisruptionBudget:
              enabled: true
              minAvailable: 2
          clickhouse:
            enabled: true
            spec:
              replicas: 1
              shards: 1
              dataVolumeClaimSpec:
                accessModes: ["ReadWriteOnce"]
                storageClassName: longhorn
                resources:
                  requests:
                    storage: 100Gi  # placeholder; Longhorn online-expansion if undersized
              resources:
                requests: { cpu: 500m, memory: 2Gi }
                limits:   { cpu: 2, memory: 4Gi }
          keeper:
            spec:
              replicas: 1
              dataVolumeClaimSpec:
                accessModes: ["ReadWriteOnce"]
                storageClassName: longhorn
                resources: { requests: { storage: 10Gi } }
          otelCollector:
            enabled: true
            resources:
              requests: { cpu: 200m, memory: 512Mi }
              limits:   { cpu: 1, memory: 1Gi }
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/clickstack
  destination:
    server: https://kubernetes.default.svc
    namespace: clickstack
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    - kind: ExternalSecret
      group: external-secrets.io
      jqPathExpressions:
        - .spec.data[].remoteRef.conversionStrategy
        - .spec.data[].remoteRef.decodingStrategy
        - .spec.data[].remoteRef.metadataPolicy

Note: if helm show values reveals the OTel Collector Service is not exposable as LoadBalancer via chart values, ship a separate Service manifest at argocd/app-configs/clickstack/otel-loadbalancer-service.yaml with metallb.universe.tf/loadBalancerIPs: "192.168.20.149" selecting the OTel Collector pods by their chart-rendered labels.

Step 3: Run cluster-app helm-template validation gate

Per the cluster-app-helm-template-trap memory, validate the template renders before committing:

helm template argocd/cluster-app --name-template cluster-app --namespace argocd > /tmp/render.yaml

Expected: succeeds; both rendered Applications have no unexpanded {{ }} placeholders.

If you added any {{ }} syntax inside an Application’s valuesObject (e.g., for templated config), use raw-string passthrough: {{ \…` }}`.

Step 4: Commit both Applications

git add argocd/cluster-app/templates/clickstack-operators.yaml argocd/cluster-app/templates/clickstack.yaml
git commit -m "feat(clickstack): add ArgoCD Applications (operators wave 0, clickstack wave 1)"

Task 1.8: Runbook and comparison-runbook docs

Files:

Create: docs/operations/clickstack.md
Create: docs/operations/clickstack-comparison.md
Step 1: Write docs/operations/clickstack.md

# ClickStack Runbook

ClickStack (ClickHouse + HyperDX + MongoDB + OTel Collector) provides a
parallel logs+metrics backend for evaluation against the existing
Loki + Prometheus + Grafana stack.

## Endpoints

- HyperDX UI: https://hyperdx.fzymgc.house (Authentik OIDC)
- OTel Collector OTLP: gRPC on MetalLB IP `192.168.20.149:4317`,
  TLS-terminated, bearer-token-authenticated
- ClickHouse: cluster-internal `clickhouse.clickstack.svc:9000` (native)
  and `:8443` (HTTPS for Grafana datasource)

## Secrets

Vault path `fzymgc-house/cluster/clickstack`:

- `admin_password` — ClickHouse admin
- `mongo_admin_password` — MongoDB admin
- `mongo_replicaset_keyfile` — MongoDB internal replica auth (multi-replica only)
- `hyperdx_bootstrap_secret` — HyperDX session signing
- `hyperdx_oidc_client_secret` — written by `tf/authentik` from Authentik
- `otel_ingest_token` — bearer token for OTLP ingest

Bootstrap (one-time, before PR1 merge):

\`\`\`bash
vault kv put fzymgc-house/cluster/clickstack \\
  admin_password="$(openssl rand -base64 32)" \\
  mongo_admin_password="$(openssl rand -base64 32)" \\
  mongo_replicaset_keyfile="$(openssl rand -base64 756)" \\
  hyperdx_bootstrap_secret="$(openssl rand -base64 32)" \\
  hyperdx_oidc_client_secret="placeholder" \\
  otel_ingest_token="$(openssl rand -base64 48)"
\`\`\`

The `hyperdx_oidc_client_secret` placeholder is overwritten by
`tf/authentik` on first apply.

## OTel ingest bearer-token rotation

Routine refresh (15-minute ESO refresh):

\`\`\`bash
vault kv put fzymgc-house/cluster/clickstack/otel_ingest_token \\
  otel_ingest_token="$(openssl rand -base64 48)"
\`\`\`

Wait ≤15min for ESO refresh, then restart consumers:

\`\`\`bash
kubectl rollout restart daemonset/vector -n vector
kubectl rollout restart deployment/otel-collector -n clickstack
\`\`\`

Emergency expedited refresh:

\`\`\`bash
kubectl annotate externalsecret clickstack-secret -n clickstack \\
  force-sync=$(date +%s) --overwrite
kubectl annotate externalsecret otel-ingest-token -n vector \\
  force-sync=$(date +%s) --overwrite
\`\`\`

Then restart both consumers.

## Velero backups

The `clickstack` namespace is included in the default Velero schedule.
No per-resource backup hooks are required for 1×1 ClickHouse (MergeTree
is durable on disk; PVC snapshot is consistent).

MongoDB backup hooks (in `app-configs/clickstack/external-secret.yaml`
target Pod annotations or via Velero hook ConfigMap):

\`\`\`yaml
pre.hook.backup.velero.io/command: '["mongosh","--eval","db.fsyncLock()"]'
post.hook.backup.velero.io/command: '["mongosh","--eval","db.fsyncUnlock()"]'
\`\`\`

For 3-replica Mongo, hooks run on primary only. For single-replica, on
the sole pod.

## Mongo-down failure mode

HyperDX UI requires MongoDB for session state. Mongo unavailable →
HyperDX UI returns 5xx.

- HA path (3 replicas + PDB minAvailable 2): tolerates 1 replica loss.
  Investigate via `kubectl get pods -n clickstack -l app=mongodb` and
  `kubectl logs ...`.
- Single-replica path: Mongo restart = UI down. Restore from latest
  Velero snapshot or wait for pod to recover.

Emergency UI access if Mongo fully unrecoverable: redeploy HyperDX
pointing to a fresh empty Mongo (`kubectl exec` + `mongorestore` from
a Velero-extracted dump if available).

## Cert rotation pod-restart

ClickHouse reads its TLS cert at startup only. After cert-manager
rotates the cert (via the `Certificate` resource), restart CH:

\`\`\`bash
kubectl rollout restart statefulset/clickhouse -n clickstack
\`\`\`

OTel Collector similarly.

## Useful queries

- HyperDX (logs): see HyperDX UI search box, Lucene syntax
- ClickHouse direct: `kubectl exec -it -n clickstack <ch-pod> -- clickhouse-client`
  - `SELECT count(*) FROM otel_logs WHERE timestamp > now() - INTERVAL 1 HOUR`
  - `SELECT count(*) FROM otel_metrics_gauge WHERE timestamp > now() - INTERVAL 1 HOUR`

Step 2: Write docs/operations/clickstack-comparison.md

# ClickStack vs Loki+Prom Comparison

Evaluation runbook for the vector + ClickStack migration. Window:
1 month minimum / 3 months maximum from PR1 merge.

**Decision date:** TBD (set when PR1 merges)

## Known divergence classes (not data loss)

These differences are expected and do **not** count toward dimension 3's
"zero data-loss incidents":

- Log serialization: Loki LogQL line vs OTel LogRecord body/attributes
- Log transport: Loki HTTP push vs OTLP gRPC
- Metric semantics: Prom remote_write preserves data model exactly;
  OTLP path converts to OTel model (histogram re-bucketing, summary
  lossy mapping, staleness markers absent)
- Timestamp precision: Prom ms vs OTel ns
- Counter resets: Prom transmits resets; OTel cumulative conversion
  can mask/amplify

## Dimension scorecard (live during eval)

### Dimension 1: Query parity (target: all 10 queries expressible)

| # | Query | Loki/Prom expression | HyperDX expression | Equivalence |
|---|-------|---------------------|--------------------|-:|
| 1 | Last 1h pod crash count | `count_over_time({namespace="X"} \|= "CrashLoopBackOff"[1h])` | `kubernetes.namespace_name:X AND message:CrashLoopBackOff` | TBD |
| 2 | Cert expiring in 7d | `(probe_ssl_earliest_cert_expiry - time()) / 86400 < 7` | SQL: `SELECT ... FROM otel_metrics_gauge WHERE ...` | TBD |
| 3-10 | (to be filled in pre-evaluation) | | | TBD |

### Dimension 2: Investigation speed (target: HyperDX first-reach >50% weeks 3+, n≥10)

| Date | Question | Stack opened first | Stack that resolved | Notes |
|------|----------|-------------------|---------------------|-------|
| (filled in during eval) | | | | |

### Dimension 3: Operational cost

Resource consumption snapshots (weekly):

| Week | ClickStack CPU/mem/disk | Loki+Prom CPU/mem/disk | Ratio | Notes |
|------|-------------------------|------------------------|-------|-------|
| 1 | | | | baseline |

Mongo-down incidents: count and timestamps (per "Mongo-down incident"
definition in spec).

Disruption baseline (Loki LogQL):
\`\`\`logql
count_over_time({job="kubernetes-events"} |~ "(?i)node draining|Cordoned|Drained" [<BASELINE_WINDOW>])
\`\`\`
Result captured in audit doc; eval-window incidents rate-normalized
against this baseline.

### Dimension 4: IaC story

Attempted: (describe HyperDX dashboard-as-code attempt here)
Outcome: (pass/fail with rationale)

### Dimension i (informational): AI features

Auto-clustering value:

- (date, observation, did it surface something Grafana wouldn't?)

NL-to-query value:

- (date, observation)

Anomaly detection:

- (date, observation)

## Decision (filled in at end of window)

Hard dimensions won (out of 4 or 3 depending on dim-2 sample):

- Dim 1: WIN / LOSS / UNCLEAR
- Dim 2 (hard if n≥10): WIN / LOSS / INFORMATIONAL
- Dim 3: WIN / LOSS / UNCLEAR
- Dim 4: WIN / LOSS / UNCLEAR

Path selected: A / B / C
Rationale: ...
Next-step PR: ...

Step 3: Add nav entries to mkdocs.yml

Edit mkdocs.yml, find the nav: section, append (in the appropriate place under Operations:):

      - ClickStack: operations/clickstack.md
      - ClickStack Comparison: operations/clickstack-comparison.md

Step 4: Update docs/reference/services.md

Edit to add a HyperDX entry under the appropriate section (alongside Grafana). Match the existing entry format.

Step 5: Update docs/reference/secrets.md

Edit to add the fzymgc-house/cluster/clickstack row with its key list.

Step 6: Commit docs

git add docs/operations/clickstack.md docs/operations/clickstack-comparison.md docs/reference/ mkdocs.yml
git commit -m "docs(clickstack): runbook + comparison eval doc + reference updates"

Task 1.9: Pre-push validation

Step 1: Re-run helm-template validation gate

Run: helm template argocd/cluster-app --name-template cluster-app --namespace argocd > /tmp/render-final.yaml && rg '{{' /tmp/render-final.yaml | head -10

Expected: no unexpanded placeholders.

Step 2: Run mkdocs build to verify nav and docs

Run: uvx --with mkdocs-material mkdocs build --strict

Expected: clean build, no warnings.

Step 3: Run lefthook hooks against staged + previous commits

Run: lefthook run pre-commit --all-files

Expected: green across markdown lint/format, yamllint, etc.

Step 4: Push branch

Run: git push -u origin feat/clickstack-deploy

Step 5: Open PR1

gh pr create --title "feat(clickstack): deploy storage tier + comparison-runbook skeleton" --body "$(cat <<'EOF'
## Summary
- Adds two ArgoCD Applications: clickstack-operators (sync-wave 0) + clickstack (sync-wave 1)
- Brings in 2 operators (Altinity ClickHouse Operator manages CH+Keeper; MongoDB Community Operator); OTel Collector is a plain Deployment
- Authentik OIDC client + Vault KV write for OIDC secret (via tf/authentik/clickstack.tf — must be applied before merge)
- ExternalSecret-based bearer-token for OTLP ingest, propagated to clickstack ns via kubernetes-replicator
- MetalLB LoadBalancer for router-host OTLP ingest at 192.168.20.149 (matches Loki gateway L2)
- cert-manager certs for HyperDX UI + OTel Collector OTLP
- Ships clickstack-comparison.md skeleton (10-query stub, divergence catalogue, dimension scorecard)
- mkdocs nav + reference docs updated

## Audit input
- Audit-output doc at \`docs/engineering/specs/2026-05-11-vector-clickstack-audit-output.md\`
- Operators chart: \`clickstack/clickstack-operators 1.0.0\`
- ClickStack chart: \`clickstack/clickstack 2.1.1\` (app 2.22.0)
- MongoDB topology: HA (members: 3 + PDB minAvailable: 2)
- MetalLB IP: \`192.168.20.149\`
- ClickHouse PVC: 100 GiB placeholder (Longhorn online-expansion if observed usage > 60% in week 1)

## Pre-merge verification
- [ ] \`terraform -chdir=tf/authentik apply\` completed; \`vault kv get fzymgc-house/cluster/clickstack\` shows \`hyperdx_oidc_client_secret\` populated
- [ ] \`helm template argocd/cluster-app ...\` renders both Applications cleanly (no unexpanded {{ }})

## Test plan (post-merge)
- [ ] clickstack-operators Application syncs in ArgoCD (CRDs present, operator pods Ready)
- [ ] clickstack Application syncs after operators (ClickHouseCluster + KeeperCluster + MongoDBCommunity CRs reconciled)
- [ ] HyperDX UI reachable at https://hyperdx.fzymgc.house with Authentik OIDC
- [ ] OTel Collector LoadBalancer IP reachable: \`curl -k -H 'Authorization: Bearer \$TOKEN' https://192.168.20.149:4317\`
- [ ] ClickHouse ready: \`kubectl exec -n clickstack <ch-pod> -- clickhouse-client --query 'SELECT 1'\`
- [ ] MongoDB replicaset healthy: \`kubectl exec -n clickstack <mongo-0> -- mongosh --eval 'rs.status()'\` shows 3 members
- [ ] Velero backup of clickstack ns succeeds (manual trigger)
- [ ] mkdocs strict build passes

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Step 6: Wait for review; merge after approval

PR2a: Add finalizer to monitoring-alloy Application

Task 2a.1: Branch from updated main

Step 1: Branch

git checkout main && git pull
git checkout -b feat/argocd-finalizer-monitoring-alloy

Task 2a.2: Add finalizer to existing Application

Files:

Modify: argocd/cluster-app/templates/monitoring-alloy.yaml
Step 1: Add finalizer field to Application metadata

Edit argocd/cluster-app/templates/monitoring-alloy.yaml, locate the Application’s metadata block, and add:

metadata:
  name: monitoring-alloy
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
  finalizers:
    - resources-finalizer.argocd.argoproj.io

Step 2: Validate helm template renders

Run: helm template argocd/cluster-app --name-template cluster-app --namespace argocd | rg -A 10 'name: monitoring-alloy'

Expected: finalizer field present in rendered output.

Step 3: Commit

git add argocd/cluster-app/templates/monitoring-alloy.yaml
git commit -m "feat(argocd): add finalizer to monitoring-alloy Application

Adds resources-finalizer.argocd.argoproj.io so future deletion of
this Application cascades to its managed resources. Prerequisite for
the upcoming alloy → vector migration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Step 4: Push, open PR

git push -u origin feat/argocd-finalizer-monitoring-alloy
gh pr create --title "feat(argocd): add finalizer to monitoring-alloy Application" --body "## Summary
- Adds resources-finalizer.argocd.argoproj.io to monitoring-alloy Application
- Prerequisite for alloy → vector migration (PR2b will delete this Application)

## Verification (run before pushing PR2b)
\`\`\`bash
kubectl get application monitoring-alloy -n argocd -o jsonpath='{.metadata.finalizers}'
# Must contain: resources-finalizer.argocd.argoproj.io

kubectl get application monitoring-alloy -n argocd -o jsonpath='{.status.sync.status}'
# Must equal: Synced

kubectl get application monitoring-alloy -n argocd -o jsonpath='{.status.sync.comparedTo.source.targetRevision}'
# Must equal the post-merge commit SHA of this PR
\`\`\`

🤖 Generated with [Claude Code](https://claude.com/claude-code)"

Step 5: Wait for merge

Task 2a.3: Verification gate before PR2b

Step 1: Verify finalizer reconciled

Run:

kubectl get application monitoring-alloy -n argocd -o jsonpath='{.metadata.finalizers}'

Expected: contains resources-finalizer.argocd.argoproj.io.

Step 2: Verify sync status

Run: kubectl get application monitoring-alloy -n argocd -o jsonpath='{.status.sync.status}'

Expected: Synced.

Step 3: Verify ArgoCD has reconciled the post-merge commit

Run: kubectl get application monitoring-alloy -n argocd -o jsonpath='{.status.sync.comparedTo.source.targetRevision}'

Expected: a commit SHA on main ≥ PR2a’s merge commit.

If any check fails, do NOT push PR2b. Use argocd app sync monitoring-alloy --refresh to force a reconcile and re-run the checks.

PR2b: Vector k8s DaemonSet (replaces monitoring-alloy)

Task 2b.1: Branch

Step 1: Branch from updated main

git checkout main && git pull
git checkout -b feat/vector-k8s-daemonset

Task 2b.2: Vector ConfigMap (`vector.yaml`)

Files:

Create: argocd/app-configs/monitoring-vector/vector.yaml
Step 1: Write Vector config

# Vector configuration for k8s DaemonSet.
# Sources: kubernetes_logs, host_metrics, internal_metrics.
# Sinks: loki (existing), opentelemetry (ClickStack via OTLP).

data_dir: "/vector-data-dir"

api:
  enabled: true
  address: 0.0.0.0:8686

sources:
  # K8s pod logs
  k8s_logs:
    type: kubernetes_logs
    # No exclusion (matches Alloy's prior coverage)

  # Host node metrics
  host_metrics:
    type: host_metrics
    collectors:
      - cpu
      - disk
      - filesystem
      - load
      - host
      - memory
      - network

  # Vector internal metrics (so we can scrape Vector itself via Prom)
  vector_self:
    type: internal_metrics

transforms:
  # Normalize log labels for Loki (extract pod/namespace/container as labels)
  k8s_logs_for_loki:
    type: remap
    inputs:
      - k8s_logs
    source: |
      .namespace = .kubernetes.pod_namespace
      .pod = .kubernetes.pod_name
      .container = .kubernetes.container_name
      .node = .kubernetes.pod_node_name

sinks:
  # Existing Loki ship path (preserves current observability)
  loki:
    type: loki
    inputs:
      - k8s_logs_for_loki
    endpoint: "http://loki-gateway.loki.svc.cluster.local:80"
    encoding:
      codec: json
    labels:
      namespace: "{{ namespace }}"
      pod: "{{ pod }}"
      container: "{{ container }}"
      node: "{{ node }}"
    buffer:
      type: disk
      max_size: 1073741824  # 1Gi
      when_full: block

  # New OTLP path to ClickStack OTel Collector — logs
  otlp_logs:
    type: opentelemetry
    inputs:
      - k8s_logs  # raw, not the relabeled-for-loki version
    protocol:
      type: http
      uri: "https://otel-collector.clickstack.svc.cluster.local:4318/v1/logs"
      tls:
        ca_file: /etc/ssl/certs/ca-certificates.crt
        verify_certificate: true
      auth:
        strategy: bearer
        token: "${OTEL_INGEST_TOKEN}"
    buffer:
      type: disk
      max_size: 1073741824
      when_full: drop_newest

  # Prometheus remote_write to existing Prom (preserves alerting)
  prom_remote_write:
    type: prometheus_remote_write
    inputs:
      - host_metrics
      - vector_self
    endpoint: "http://kps-prometheus.prometheus.svc.cluster.local:9090/api/v1/write"
    buffer:
      type: disk
      max_size: 2147483648  # 2Gi
      when_full: block

  # New OTLP path — metrics
  otlp_metrics:
    type: opentelemetry
    inputs:
      - host_metrics
      - vector_self
    protocol:
      type: http
      uri: "https://otel-collector.clickstack.svc.cluster.local:4318/v1/metrics"
      tls:
        verify_certificate: true
      auth:
        strategy: bearer
        token: "${OTEL_INGEST_TOKEN}"
    buffer:
      type: disk
      max_size: 1073741824
      when_full: drop_newest

Note: This is Vector’s TOML-like YAML config. Adjust paths/keys based on the Vector Helm chart’s expected schema (some charts expect a flat ConfigMap key, others nested values). Verify via helm show values grafana/vector --version <selected>.

Step 2: Create kustomization.yaml

Write argocd/app-configs/monitoring-vector/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: vector

resources:
  - external-secret.yaml

configMapGenerator:
  - name: vector-config
    files:
      - vector.yaml
    options:
      disableNameSuffixHash: true

Step 3: Create ExternalSecret with replicator annotation

Write argocd/app-configs/monitoring-vector/external-secret.yaml:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: otel-ingest-token
  namespace: vector
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: otel-ingest-token
    creationPolicy: Owner
    template:
      metadata:
        annotations:
          replicator.v1.mittwald.de/replicate-to: clickstack/otel-ingest-token
  data:
    - secretKey: OTEL_INGEST_TOKEN
      remoteRef:
        key: fzymgc-house/cluster/clickstack
        property: otel_ingest_token

The replicator.v1.mittwald.de/replicate-to annotation tells kubernetes-replicator to mirror the Secret into the clickstack namespace where the OTel Collector consumes it.

Step 4: Commit app-config

git add argocd/app-configs/monitoring-vector/
git commit -m "feat(vector): k8s app-config (vector.yaml + ExternalSecret with replicator)"

Task 2b.3: Vector ArgoCD Application

Files:

Create: argocd/cluster-app/templates/monitoring-vector.yaml
Step 1: Write the Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: monitoring-vector
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: observability
  sources:
    - chart: vector
      repoURL: https://helm.vector.dev
      targetRevision: 0.52.0  # vector/vector chart, app 0.55.0-distroless-libc (audit-pinned)
      helm:
        releaseName: vector
        valuesObject:
          role: Agent  # DaemonSet
          rbac:
            create: true
          serviceAccount:
            create: true
            name: vector
          existingConfigMaps:
            - vector-config
          dataDir: /vector-data-dir
          podMonitor:
            enabled: false  # we're disabling ServiceMonitor pattern in PR4
          resources:
            requests:
              cpu: 200m
              memory: 512Mi
            limits:
              cpu: 1
              memory: 1Gi
          extraVolumes:
            - name: data
              emptyDir: {}
            - name: buffer
              emptyDir:
                sizeLimit: 5Gi  # disk buffer for sinks
          extraVolumeMounts:
            - name: data
              mountPath: /vector-data-dir
            - name: buffer
              mountPath: /vector-buffer
          env:
            - name: OTEL_INGEST_TOKEN
              valueFrom:
                secretKeyRef:
                  name: otel-ingest-token
                  key: OTEL_INGEST_TOKEN
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/monitoring-vector
  destination:
    server: https://kubernetes.default.svc
    namespace: vector
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    - kind: ExternalSecret
      group: external-secrets.io
      jqPathExpressions:
        - .spec.data[].remoteRef.conversionStrategy
        - .spec.data[].remoteRef.decodingStrategy
        - .spec.data[].remoteRef.metadataPolicy

Vector chart pinned at 0.52.0 per audit (Item-equivalent at audit time).

Step 2: Validate helm-template

Run: helm template argocd/cluster-app --name-template cluster-app --namespace argocd | rg -A 5 'name: monitoring-vector'

Expected: no unexpanded {{ }}.

Step 3: Commit

git add argocd/cluster-app/templates/monitoring-vector.yaml
git commit -m "feat(vector): add k8s DaemonSet ArgoCD Application"

Task 2b.4: Delete monitoring-alloy

Files:

Delete: argocd/cluster-app/templates/monitoring-alloy.yaml
Delete: argocd/cluster-app/templates/monitoring-alloy-ingest.yaml
Delete: argocd/app-configs/monitoring-alloy/ (entire dir)
Step 1: Delete the alloy Applications and app-config

git rm argocd/cluster-app/templates/monitoring-alloy.yaml
git rm argocd/cluster-app/templates/monitoring-alloy-ingest.yaml
git rm -r argocd/app-configs/monitoring-alloy/

Step 2: Add velero exclusion entry

Edit argocd/app-configs/velero/backup-schedule.yaml: locate the alloy namespace in excludedNamespaces, and replace alloy → vector (vector is also stateless telemetry).

# In the Telemetry category:
- vector  # was: alloy

Step 3: Commit deletion

git add argocd/cluster-app/templates/ argocd/app-configs/ argocd/app-configs/velero/
git commit -m "feat(vector): remove monitoring-alloy Application + app-config

PR2a previously added resources-finalizer to monitoring-alloy. With
this PR merged, ArgoCD's prune+finalizer cascade deletes the alloy
DaemonSet, ConfigMap, ExternalSecret, Ingress, and namespace.

Velero exclusion swapped alloy → vector (both stateless telemetry)."

Task 2b.5: Push, open PR, monitor sync

Step 1: Run pre-push checks

helm template argocd/cluster-app --name-template cluster-app --namespace argocd > /tmp/render.yaml
rg '{{' /tmp/render.yaml | head
lefthook run pre-commit --all-files

Expected: clean.

Step 2: Push

Run: git push -u origin feat/vector-k8s-daemonset

Step 3: Open PR

gh pr create --title "feat(vector): k8s DaemonSet replaces monitoring-alloy" --body "$(cat <<'EOF'
## Summary
- Adds Vector DaemonSet (k8s telemetry agent) via Helm chart
- ConfigMap-generated vector.yaml with dual-ship: Loki + OTLP-to-ClickStack
- ExternalSecret for OTEL_INGEST_TOKEN with kubernetes-replicator annotation propagating to clickstack ns
- Disk-buffered sinks: block on Prom remote_write (alert fidelity), drop_newest on OTLP (eval data)
- Deletes monitoring-alloy Application (finalizer added in PR2a cascades the cleanup)
- Swaps `alloy` → `vector` in Velero excludedNamespaces

## Prereqs
- PR1 merged (ClickStack OTel Collector exists at the OTLP endpoint)
- PR2a merged AND verification gate green (finalizer reconciled onto live alloy Application)

## Test plan
- [ ] After merge + ArgoCD sync: `kubectl get ns vector` exists, vector DaemonSet ready on all nodes
- [ ] `kubectl get ns alloy` returns NotFound (finalizer cascade cleaned it up)
- [ ] Loki receives logs from Vector: query Grafana Explore for any recent pod log → present
- [ ] ClickHouse receives logs: `kubectl exec -n clickstack <ch-pod> -- clickhouse-client --query "SELECT count(*) FROM otel_logs WHERE timestamp > now() - INTERVAL 5 MINUTE"` returns > 0
- [ ] Vector buffer not back-pressured: `vector_buffer_byte_size` metric < 80% of `vector_buffer_max_byte_size`
- [ ] AlertManager fires the same alerts it was firing pre-PR2b (nothing new silenced)

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Step 4: Merge after review; monitor sync

Use Monitor tool with kubectl wait patterns to follow ArgoCD reconciliation; emit one line per state change for:

monitoring-alloy Application deletion (and namespace gone)
monitoring-vector Application Synced+Healthy
DaemonSet pods Ready on all nodes
First otel_logs row in ClickHouse

PR3: Vector ansible role for router-hosts

Task 3.1: Branch

Step 1: Branch

git checkout main && git pull
git checkout -b feat/vector-router-hosts

Task 3.2: Create ansible/roles/vector skeleton

Files:

Create: ansible/roles/vector/{defaults,handlers,meta,tasks,templates}/
Step 1: Create directory structure

Run: mkdir -p ansible/roles/vector/{defaults,handlers,meta,tasks,templates}

Step 2: meta/main.yml

---
galaxy_info:
  author: Sean
  description: Vector telemetry agent for router-hosts
  license: MIT
  min_ansible_version: "2.12"
  platforms:
    - name: Debian
      versions:
        - bookworm
dependencies: []

Step 3: defaults/main.yml

---
vector_version: "0.43.0"  # pin to current GA at PR3 authoring time; verify via `apt-cache madison vector` or releases
vector_apt_repo_url: "https://repositories.timber.io/public/vector/deb/debian"
vector_apt_signing_key_url: "https://repositories.timber.io/public/vector/gpg.E952ED0F2BCAEACEC11A8C2FE6BB3C57A03A36DA.key"

vector_config_dir: /etc/vector
vector_data_dir: /var/lib/vector
vector_buffer_dir: /var/lib/vector/buffer

# OTel Collector endpoint (MetalLB IP from audit item 12)
vector_otel_endpoint: "https://192.168.20.149:4317"  # MetalLB IP from audit Item 12
vector_otel_token: "{{ lookup('community.hashi_vault.hashi_vault', 'secret=fzymgc-house/cluster/clickstack:otel_ingest_token') }}"

# Loki endpoint (existing)
vector_loki_endpoint: "http://<LOKI-GATEWAY-IP>:80"  # from existing inventory

Step 4: handlers/main.yml

---
- name: Restart vector
  ansible.builtin.systemd:
    name: vector
    state: restarted
    daemon_reload: true

Step 5: tasks/main.yml

---
- name: Add Vector APT signing key
  ansible.builtin.apt_key:
    url: "{{ vector_apt_signing_key_url }}"
    state: present

- name: Add Vector APT repository
  ansible.builtin.apt_repository:
    repo: "deb {{ vector_apt_repo_url }} bookworm main"
    state: present
    filename: vector

- name: Install Vector
  ansible.builtin.apt:
    name: "vector={{ vector_version }}*"
    state: present
    update_cache: true

- name: Ensure config + data + buffer dirs exist
  ansible.builtin.file:
    path: "{{ item }}"
    state: directory
    owner: vector
    group: vector
    mode: "0755"
  loop:
    - "{{ vector_config_dir }}"
    - "{{ vector_data_dir }}"
    - "{{ vector_buffer_dir }}"

- name: Render Vector config
  ansible.builtin.template:
    src: vector.yaml.j2
    dest: "{{ vector_config_dir }}/vector.yaml"
    owner: vector
    group: vector
    mode: "0640"
  notify: Restart vector

- name: Render systemd override (env vars for OTel token)
  ansible.builtin.template:
    src: vector.service.j2
    dest: /etc/systemd/system/vector.service.d/override.conf
    owner: root
    group: root
    mode: "0644"
  notify: Restart vector

- name: Ensure systemd directory for override
  ansible.builtin.file:
    path: /etc/systemd/system/vector.service.d
    state: directory
    owner: root
    group: root
    mode: "0755"

- name: Enable and start vector
  ansible.builtin.systemd:
    name: vector
    enabled: true
    state: started
    daemon_reload: true

Step 6: templates/vector.yaml.j2

data_dir: "{{ vector_data_dir }}"

api:
  enabled: true
  address: 127.0.0.1:8686

sources:
  journald:
    type: journald
    current_boot_only: false

  host_metrics:
    type: host_metrics
    collectors:
      - cpu
      - disk
      - filesystem
      - load
      - host
      - memory
      - network

  vector_self:
    type: internal_metrics

transforms:
  journald_for_loki:
    type: remap
    inputs:
      - journald
    source: |
      .host = "{{ inventory_hostname }}"

sinks:
  loki:
    type: loki
    inputs:
      - journald_for_loki
    endpoint: "{{ vector_loki_endpoint }}"
    encoding:
      codec: json
    labels:
      host: "{{ '{{ host }}' }}"
      unit: "{{ '{{ _SYSTEMD_UNIT }}' }}"
    buffer:
      type: disk
      max_size: 1073741824
      when_full: block

  otlp_logs:
    type: opentelemetry
    inputs:
      - journald
    protocol:
      type: http
      uri: "{{ vector_otel_endpoint }}/v1/logs"
      tls:
        verify_certificate: true
      auth:
        strategy: bearer
        token: "${OTEL_INGEST_TOKEN}"
    buffer:
      type: disk
      max_size: 1073741824
      when_full: drop_newest

  otlp_metrics:
    type: opentelemetry
    inputs:
      - host_metrics
      - vector_self
    protocol:
      type: http
      uri: "{{ vector_otel_endpoint }}/v1/metrics"
      tls:
        verify_certificate: true
      auth:
        strategy: bearer
        token: "${OTEL_INGEST_TOKEN}"
    buffer:
      type: disk
      max_size: 1073741824
      when_full: drop_newest

Note the {{ '{{ ... }}' }} escaping for Vector-template-language literals nested inside Jinja.

Step 7: templates/vector.service.j2

[Service]
Environment=OTEL_INGEST_TOKEN={{ vector_otel_token }}

Step 8: Run ansible-lint

Run: ansible-lint ansible/roles/vector/

Expected: clean.

Step 9: Commit role

git add ansible/roles/vector/
git commit -m "feat(ansible): add vector role for router-hosts"

Task 3.3: Wire role into playbook + remove alloy role

Files:

Modify: ansible/playbooks/<router-hosts-playbook>.yml (locate via rg -l 'roles:.*alloy' ansible/)
Delete: ansible/roles/alloy/
Step 1: Find the playbook reference

Run: rg -l "role:.*alloy|roles:.*alloy" ansible/

Step 2: Replace alloy with vector in the playbook

In the matched playbook, edit:

roles:
  - vector  # was: alloy

Step 3: Delete the alloy role

Run: git rm -r ansible/roles/alloy/

Step 4: Commit

git add ansible/ -A
git commit -m "feat(ansible): swap alloy role → vector role in router-hosts playbook"

Task 3.4: Dry-run + smoke test on one router-host

Step 1: Run ansible in check mode

ansible-playbook -i ansible/inventory/router-hosts.yml ansible/playbooks/<playbook>.yml --check --diff --limit <single-test-host>

Expected: clean diff showing alloy uninstall + vector install.

Step 2: Apply to one host

ansible-playbook -i ansible/inventory/router-hosts.yml ansible/playbooks/<playbook>.yml --limit <single-test-host>

Step 3: Verify vector is running

ansible <single-test-host> -m shell -a "systemctl status vector --no-pager"

Expected: active (running).

Step 4: Verify dual-ship is working

ansible <single-test-host> -m shell -a "journalctl -u vector --since '5 min ago' | tail -50"

Look for connection success messages to both Loki and OTel endpoints; no auth errors.

Step 5: Verify logs appearing in ClickHouse

kubectl exec -n clickstack <ch-pod> -- clickhouse-client --query \
  "SELECT count(*) FROM otel_logs WHERE timestamp > now() - INTERVAL 5 MINUTE AND ServiceName LIKE '%router%'"

Expected: > 0.

Task 3.5: Push, open PR, roll out to remaining hosts

Step 1: Push

Run: git push -u origin feat/vector-router-hosts

Step 2: Open PR

gh pr create --title "feat(vector): router-hosts ansible role (replaces alloy)" --body "## Summary
- Adds ansible/roles/vector for router-hosts telemetry
- journald + host_metrics → dual-ship to Loki + ClickStack OTel Collector
- Disk-buffered sinks, bearer-token auth via Vault lookup
- Deletes ansible/roles/alloy; playbook updated

## Test plan
- [x] Dry-run on one host (smoke-tested pre-merge)
- [x] Vector running on test host
- [x] Logs appearing in both Loki and ClickHouse from test host
- [ ] After merge: roll out to remaining router-hosts via \`ansible-playbook ... --limit !<test-host>\`

🤖 Generated with [Claude Code](https://claude.com/claude-code)"

Step 3: After merge, roll out to remaining hosts

ansible-playbook -i ansible/inventory/router-hosts.yml ansible/playbooks/<playbook>.yml --limit '!<test-host>'

PR4a: Vector scrape configs + shadow-Prom parity gate

Task 4a.1: Branch

Step 1: Branch

git checkout main && git pull
git checkout -b feat/vector-scrape-shadow-prom

Task 4a.2: Author the ServiceMonitor translation list

Files:

Create: docs/operations/clickstack-vector-scrape-targets.md
Step 1: Enumerate cluster scrape targets

kubectl get servicemonitor,podmonitor,probe -A -o json > /tmp/scrape-discovery.json
kubectl get scrapeconfigs.monitoring.coreos.com -A -o json 2>/dev/null > /tmp/scrape-configs.json
rg -n 'additionalScrapeConfigs' argocd/cluster-app/templates/monitoring-prometheus.yaml > /tmp/additional-scrapes.txt

Step 2: Write the translation doc

# Vector Scrape-Target Translation

Enumeration of every Prometheus-Operator-discovered scrape target,
translated to Vector `prometheus_scrape` source blocks for the
parallel-run migration's PR4a/PR4b.

Source: cluster snapshot at <ISO-8601-TIMESTAMP>.

## ServiceMonitors

| Namespace | Name | Selector | Endpoint(s) | jobLabel | Vector scrape job equivalent |
|-----------|------|----------|-------------|----------|-----------------------------|
| (filled from /tmp/scrape-discovery.json) | | | | | |

## PodMonitors

(same table format)

## ScrapeConfigs

(if any from /tmp/scrape-configs.json)

## additionalScrapeConfigs

(if any from /tmp/additional-scrapes.txt; expand from referenced Secret/ConfigMap)

## Vector scrape configuration

The translations above are realized as `prometheus_scrape` sources +
`kubernetes_discovery` filters in Vector's config. See
`argocd/app-configs/monitoring-vector/vector.yaml` for the full
generated config; this doc is the audit trail for which targets are
covered.

## Coverage gates

Pre-PR4b checklist:

- [ ] Every row in the tables above has a corresponding Vector scrape job
- [ ] `job=<name>` label values match what existing PrometheusRules use
  (verified via `promtool-parity` Job output)
- [ ] Long-`for:` rules verified via label-set comparison (separate
  Job output)

Fill in tables from JSON outputs.

Step 3: Add mkdocs nav entry

Edit mkdocs.yml to add:

      - ClickStack Scrape Translation: operations/clickstack-vector-scrape-targets.md

Step 4: Commit doc

git add docs/operations/clickstack-vector-scrape-targets.md mkdocs.yml
git commit -m "docs(clickstack): enumerate ServiceMonitor → Vector scrape translation"

Task 4a.3: Extend Vector config with prometheus_scrape sources

Files:

Modify: argocd/app-configs/monitoring-vector/vector.yaml
Step 1: Add prometheus_scrape sources for each translated target

Append to the sources: section of vector.yaml:

  # Translated from kube-prometheus-stack ServiceMonitors.
  # Each scrape job emits an explicit `job=<name>` label to match
  # existing PrometheusRule expectations.

  scrape_kubelet:
    type: prometheus_scrape
    endpoints:
      - "https://${KUBERNETES_NODE_IP}:10250/metrics"
    tls:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    auth:
      strategy: bearer
      token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    scrape_interval_secs: 30

  scrape_kube_state_metrics:
    type: prometheus_scrape
    endpoints:
      - "http://kube-state-metrics.kube-system.svc.cluster.local:8080/metrics"
    scrape_interval_secs: 30

  scrape_node_exporter:
    type: prometheus_scrape
    endpoints:
      - "http://kps-node-exporter.prometheus.svc.cluster.local:9100/metrics"
    scrape_interval_secs: 30

  # ... add one per translated target from the translation doc ...

For each, add a corresponding VRL transform that adds the job= label:

transforms:
  # ... existing transforms ...

  label_kubelet:
    type: remap
    inputs:
      - scrape_kubelet
    source: |
      .tags.job = "kubelet"

  label_kube_state_metrics:
    type: remap
    inputs:
      - scrape_kube_state_metrics
    source: |
      .tags.job = "kube-state-metrics"

  # ... one per scrape source ...

Update sink inputs to consume the labelled transforms instead of the raw scrape sources:

sinks:
  prom_remote_write:
    inputs:
      - host_metrics
      - vector_self
      - label_kubelet
      - label_kube_state_metrics
      - label_node_exporter
      # ... add all labeled transforms ...
    # ... rest unchanged ...

  otlp_metrics:
    inputs:
      - host_metrics
      - vector_self
      - label_kubelet
      - label_kube_state_metrics
      - label_node_exporter
      # ... same list ...
    # ... rest unchanged ...

Step 2: Verify ConfigMap regenerates cleanly

Run from worktree root: kubectl kustomize argocd/app-configs/monitoring-vector | rg -A 5 'kind: ConfigMap'

Expected: ConfigMap rendered with the new sources/transforms.

Step 3: Commit

git add argocd/app-configs/monitoring-vector/vector.yaml
git commit -m "feat(vector): add prometheus_scrape sources with job= label parity"

Task 4a.4: Shadow-Prom + shadow-AM + diff CronJob + promtool Job

Files:

Create: argocd/cluster-app/templates/monitoring-prometheus-shadow.yaml
Create: argocd/app-configs/monitoring-prometheus-shadow/kustomization.yaml
Create: argocd/app-configs/monitoring-prometheus-shadow/alert-diff-cronjob.yaml
Create: argocd/app-configs/monitoring-prometheus-shadow/promtool-job.yaml
Step 1: Application (shadow kube-prometheus-stack release)

argocd/cluster-app/templates/monitoring-prometheus-shadow.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: monitoring-prometheus-shadow
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: observability
  sources:
    - chart: kube-prometheus-stack
      repoURL: https://prometheus-community.github.io/helm-charts
      targetRevision: 84.5.0  # match primary kps version
      helm:
        releaseName: kps-shadow
        valuesObject:
          crds:
            enabled: false  # CRDs come from primary kps install
          prometheusOperator:
            enabled: false  # reuse existing operator
          defaultRules:
            create: false
          grafana:
            enabled: false
          prometheus-node-exporter:
            enabled: false  # primary's data is fine
          kube-state-metrics:
            enabled: false  # same
          prometheus:
            prometheusSpec:
              replicas: 1
              retention: <SHADOW-RETENTION>  # 4h/48h based on audit max-for:
              enableRemoteWriteReceiver: true
              # Empty selectors: shadow-prom doesn't scrape anything itself
              serviceMonitorSelector:
                matchLabels:
                  shadow-prom-only: "true"
              podMonitorSelector:
                matchLabels:
                  shadow-prom-only: "true"
              probeSelector:
                matchLabels:
                  shadow-prom-only: "true"
              storageSpec: {}  # emptyDir if retention ≤4h; PVC if ≥48h
              # If PVC needed:
              # storageSpec:
              #   volumeClaimTemplate:
              #     spec:
              #       storageClassName: longhorn
              #       resources:
              #         requests:
              #           storage: 10Gi
              resources:
                requests: { cpu: 500m, memory: 1Gi }
                limits:   { cpu: 2, memory: 2Gi }
          alertmanager:
            enabled: true
            alertmanagerSpec:
              replicas: 1
              storage: {}  # ephemeral
              resources:
                requests: { cpu: 100m, memory: 256Mi }
                limits:   { cpu: 500m, memory: 512Mi }
            config:
              global:
                resolve_timeout: 5m
              route:
                receiver: "null"
              receivers:
                - name: "null"
                  # No webhook configs — drops all alerts silently
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/monitoring-prometheus-shadow
  destination:
    server: https://kubernetes.default.svc
    namespace: prometheus
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true

Replace <SHADOW-RETENTION> per audit max-for: finding (4h / 48h).

Step 2: kustomization.yaml

argocd/app-configs/monitoring-prometheus-shadow/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: prometheus

resources:
  - alert-diff-cronjob.yaml
  - promtool-job.yaml

Step 3: alert-diff CronJob

argocd/app-configs/monitoring-prometheus-shadow/alert-diff-cronjob.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: alert-diff
  namespace: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: alert-diff
  namespace: prometheus
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["create", "patch", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: alert-diff
  namespace: prometheus
subjects:
  - kind: ServiceAccount
    name: alert-diff
    namespace: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: alert-diff
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: alert-diff
  namespace: prometheus
spec:
  schedule: "*/5 * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          serviceAccountName: alert-diff
          restartPolicy: OnFailure
          containers:
            - name: diff
              image: alpine/k8s:1.29.0  # has kubectl + jq + curl
              command:
                - sh
                - -c
                - |
                  set -eo pipefail
                  TS=$(date +%Y%m%d-%H%M%S)
                  PRIMARY=$(curl -s "http://kps-alertmanager.prometheus.svc:9093/api/v2/alerts?active=true&silenced=false" | jq -S '[.[] | {labels: .labels}] | sort_by(.labels)')
                  SHADOW=$(curl -s "http://kps-shadow-alertmanager.prometheus.svc:9093/api/v2/alerts?active=true&silenced=false" | jq -S '[.[] | {labels: .labels}] | sort_by(.labels)')
                  # Compute differences
                  DIFF_ONLY_PRIMARY=$(echo "$PRIMARY $SHADOW" | jq -s '.[0] - .[1]')
                  DIFF_ONLY_SHADOW=$(echo "$PRIMARY $SHADOW" | jq -s '.[1] - .[0]')
                  # Write per-run ConfigMap
                  cat <<EOF | kubectl apply -f -
                  apiVersion: v1
                  kind: ConfigMap
                  metadata:
                    name: alert-diff-results-$TS
                    namespace: prometheus
                    labels:
                      app: alert-diff
                  data:
                    only-in-primary.json: |
                  $(echo "$DIFF_ONLY_PRIMARY" | sed 's/^/      /')
                    only-in-shadow.json: |
                  $(echo "$DIFF_ONLY_SHADOW" | sed 's/^/      /')
                  EOF

Step 4: promtool parity Job

argocd/app-configs/monitoring-prometheus-shadow/promtool-job.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtool-parity
  namespace: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: promtool-parity
  namespace: prometheus
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["create", "patch", "get"]
  - apiGroups: ["monitoring.coreos.com"]
    resources: ["prometheusrules"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: promtool-parity
  namespace: prometheus
subjects:
  - kind: ServiceAccount
    name: promtool-parity
    namespace: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: promtool-parity
---
apiVersion: batch/v1
kind: Job
metadata:
  name: promtool-parity-initial
  namespace: prometheus
spec:
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      serviceAccountName: promtool-parity
      restartPolicy: OnFailure
      containers:
        - name: promtool
          image: quay.io/prometheus/prometheus:v3.11.3  # matches primary kps Prometheus version; bundles promtool
          command:
            - sh
            - -c
            - |
              # Pseudocode: evaluate each rule's expr against primary and shadow,
              # compare /api/v1/series for the rule's metric matchers, output
              # divergences. Full implementation requires per-rule iteration
              # via kubectl + jq + promtool query commands.
              # This is the entry point; full script lives in a ConfigMap mount.
              /scripts/promtool-parity.sh
          volumeMounts:
            - name: scripts
              mountPath: /scripts
      volumes:
        - name: scripts
          configMap:
            name: promtool-parity-scripts
            defaultMode: 0755
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtool-parity-scripts
  namespace: prometheus
data:
  promtool-parity.sh: |
    #!/bin/sh
    set -eo pipefail
    PRIMARY="http://kps-prometheus.prometheus.svc:9090"
    SHADOW="http://kps-shadow-prometheus.prometheus.svc:9090"

    # For each PrometheusRule:
    #   - extract expr + for duration
    #   - if for <= shadow-retention: query both, compare result vectors
    #   - if for > shadow-retention: query /api/v1/series on both, compare label-set
    # Write findings to ConfigMap promtool-parity-results.

    # Simplified placeholder; the full script enumerates rules and compares.
    # For this initial Job, output a summary indicating "manual review required".
    cat <<'YAML' | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: promtool-parity-results
      namespace: prometheus
    data:
      summary.txt: |
        promtool parity check ran at $(date -u +%Y-%m-%dT%H:%M:%SZ).
        Manual review of comparison required.
        See alert-diff-results-* ConfigMaps for live alert divergence.
    YAML

Note: the promtool-parity script is necessarily a sketch because the per-rule comparison logic is non-trivial. The Job runs, emits a ConfigMap, and the implementer reviews alert-diff-results-* ConfigMaps as the primary parity signal. The promtool Job is a secondary verification — extend its script as needed when the spec is operationalized.

Step 5: Validate helm template

Run: helm template argocd/cluster-app --name-template cluster-app --namespace argocd | rg -A 3 'name: monitoring-prometheus-shadow'

Expected: Application present.

Step 6: Commit

git add argocd/app-configs/monitoring-prometheus-shadow/ argocd/cluster-app/templates/monitoring-prometheus-shadow.yaml
git commit -m "feat(prometheus): shadow-Prom + shadow-AM + alert-diff CronJob + promtool Job"

Task 4a.5: Push, open PR, observe 24h parity gate

Step 1: Push

git push -u origin feat/vector-scrape-shadow-prom

Step 2: Open PR

gh pr create --title "feat(vector): scrape configs + shadow-Prom 24h parity gate" --body "## Summary
- Vector takes over metrics scraping with explicit job= label parity
- Shadow Prometheus + shadow AlertManager deployed via ArgoCD Application
- Alert-diff CronJob (every 5min for 24h, per-run dated ConfigMaps)
- Promtool parity Job (manual extension for full per-rule check)
- Translation doc enumerates every ServiceMonitor + PodMonitor + Probe + ScrapeConfig + additionalScrapeConfigs

## Pre-PR4b gate
Before PR4b can be pushed, observe the parity gate green for a contiguous 24h:
- alert-diff-results-* ConfigMaps: \`only-in-primary.json\` is empty (no false negatives)
- promtool-parity-results ConfigMap: no label-set divergences for long-for: rules
- Shadow Prom ingesting Vector remote_write: \`kubectl exec -n prometheus kps-shadow-prometheus-0 -- wget -q -O- http://localhost:9090/api/v1/targets\` shows non-empty

## Test plan
- [ ] Vector pods restart cleanly after ConfigMap regeneration
- [ ] Shadow Prom Synced+Healthy
- [ ] CronJob first run produces non-empty ConfigMap
- [ ] After 24h: review alert-diff-results-* ConfigMaps; \`only-in-primary.json\` is consistently empty

🤖 Generated with [Claude Code](https://claude.com/claude-code)"

Step 3: Merge after review
Step 4: Use Monitor tool to observe 24h gate

Set up a Monitor loop polling every 30 minutes:

# Until: only-in-primary divergence stays at 0 for 24 contiguous hours
TS_START=$(date +%s)
until [[ $(date +%s) -gt $((TS_START + 86400)) ]]; do
  for cm in $(kubectl get cm -n prometheus -l app=alert-diff -o name | tail -3); do
    DIFF=$(kubectl get $cm -n prometheus -o jsonpath='{.data.only-in-primary\.json}')
    echo "[$(date +%H:%M)] $cm: $(echo $DIFF | jq 'length')"
  done
  sleep 1800  # 30min
done
echo "24h gate complete. Review final ConfigMaps + proceed to PR4b."

Document divergences in PR4a’s PR conversation as evidence.

PR4b: Disable Prom native scrape + chart ServiceMonitors

Task 4b.1: Branch (after 24h gate green)

Step 1: Branch

git checkout main && git pull
git checkout -b feat/disable-prom-scrape

Task 4b.2: Disable Prom native scrape

Files:

Modify: argocd/cluster-app/templates/monitoring-prometheus.yaml
Step 1: Set selectors to match nothing

Edit monitoring-prometheus.yaml to add inside prometheus.prometheusSpec:

              serviceMonitorSelectorNilUsesHelmValues: false
              podMonitorSelectorNilUsesHelmValues: false
              probeSelectorNilUsesHelmValues: false
              # Set selectors to a label that no resource carries → matches nothing
              serviceMonitorSelector:
                matchLabels:
                  prom-scrape-disabled: "true"
              podMonitorSelector:
                matchLabels:
                  prom-scrape-disabled: "true"
              probeSelector:
                matchLabels:
                  prom-scrape-disabled: "true"

(If selectors already exist for the release: kps label, replace with the disabled-label form above.)

Step 2: Commit

git add argocd/cluster-app/templates/monitoring-prometheus.yaml
git commit -m "feat(prometheus): disable native scrape (Vector takes over)

Sets serviceMonitorSelector/podMonitorSelector/probeSelector to match
a label no resource carries. Prom continues to ingest via remote_write
from Vector; AlertManager rules fire from the remote_write data.

Pre-merge: 24h shadow-Prom parity gate observed green (see PR4a).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>"

Task 4b.3: Disable chart-installed ServiceMonitors

Files:

Modify: various argocd/cluster-app/templates/*.yaml and argocd/app-configs/*/ containing chart values
Step 1: Sweep for serviceMonitor.enabled keys

rg -l 'serviceMonitor:' argocd/cluster-app/templates/ argocd/app-configs/

Step 2: For each match, disable

For each file found, add or modify:

serviceMonitor:
  enabled: false

Likely candidates: cnpg, temporal-server, traefik, monitoring-loki (Loki self-monitoring), and kube-prometheus-stack components (node-exporter, kube-state-metrics).

Step 3: Verify all SMs are off

After sync, run: kubectl get servicemonitor -A | wc -l

Expected: only the shadow-Prom stack’s monitors (if any) and any handcrafted ones remain.

Step 4: Delete the shadow-Prom Application

Edit argocd/cluster-app/templates/monitoring-prometheus-shadow.yaml: delete the file entirely.

git rm argocd/cluster-app/templates/monitoring-prometheus-shadow.yaml
git rm -r argocd/app-configs/monitoring-prometheus-shadow/

(The resources-finalizer cascades the cleanup.)

Step 5: Commit

git add argocd/
git commit -m "feat(charts): disable chart-installed ServiceMonitors + remove shadow stack

After 24h parity gate green, Vector is the canonical scraper.
Chart-installed ServiceMonitors are now dead CRDs and disabled at the
chart values layer. Shadow-Prom + shadow-AM + diff/promtool jobs
deleted via Application removal (finalizer cascades cleanup)."

Task 4b.4: Push, open PR, monitor cutover

Step 1: Push

git push -u origin feat/disable-prom-scrape

Step 2: Open PR

gh pr create --title "feat(prometheus): disable native scrape + chart ServiceMonitors" --body "## Summary
- Sets Prom selectors to match nothing → Prom is remote_write-only TSDB
- Disables chart-installed ServiceMonitors across kps, cnpg, temporal, traefik, loki
- Removes shadow-Prom Application (finalizer cascades cleanup)

## Pre-merge evidence
- 24h shadow-Prom parity gate green (PR4a observation window)
- alert-diff only-in-primary.json consistently empty

## Test plan
- [ ] AlertManager continues firing the same alerts (no silenced rules)
- [ ] Prom \`up{}\` only shows targets Vector has remote-written (kubelet, kube-state-metrics, node-exporter, etc. — matching the translation list)
- [ ] kubectl get ns prometheus-shadow returns NotFound (Application cascade succeeded)

🤖 Generated with [Claude Code](https://claude.com/claude-code)"

Step 3: Monitor cutover via Monitor tool

Poll every 5 min until AlertManager firing-set is identical to pre-merge and shadow-Prom resources are gone.

PR5: Comparison-runbook eval-window updates (lived-in)

Drop into PR5 each time a meaningful data point is captured during the evaluation window. Multiple smaller commits to the comparison doc; not one big PR.

Task 5.1: Schedule decision date

Step 1: Set decision date in clickstack-comparison.md

When PR1 merges (or PR4b merges, whichever is the eval-start anchor), edit docs/operations/clickstack-comparison.md:

**Decision date:** 2026-MM-DD (PR1-merge + 30 days, 2026-MM-DD hard cap)

Commit + push as a docs PR.

Task 5.2: Weekly resource-cost snapshot

Step 1: Capture weekly resource consumption

Every week during eval, run:

kubectl top pods -n clickstack
kubectl top pods -n loki
kubectl top pods -n prometheus
kubectl get pvc -A | grep -E 'clickstack|loki|prometheus'

Update the Dimension 3 table in clickstack-comparison.md with the week’s numbers.

Step 2: Commit each weekly update

git checkout -b docs/eval-week-N
# edit clickstack-comparison.md
git commit -m "docs(clickstack): eval week N resource cost snapshot"
git push && gh pr create --title "docs(clickstack): eval week N update" --body "Weekly resource snapshot"

Task 5.3: Investigation journaling

Step 1: Log each investigation in real-time

When an operational question takes ≥3 min to answer, after answering, append to the Dimension 2 table:

| 2026-MM-DD HH:MM | Why is the foo pod restarting? | HyperDX | Grafana+Loki | HyperDX clustering surfaced the issue immediately but resolving needed Loki's structured-metadata query |

Batch commits weekly.

Task 5.4: Decision-time evaluation

Step 1: After 1 month minimum, evaluate per-dimension

Edit clickstack-comparison.md Decision section:

Hard dimensions won:

- Dim 1: WIN (10/10 queries expressed)
- Dim 2: <WIN/LOSS/INFORMATIONAL> (n=___)
- Dim 3: <WIN/LOSS> (resource ratio ___, Mongo incidents ___)
- Dim 4: <WIN/LOSS>

Path selected: <A/B/C>
Rationale: ...
Next-step PR (Phase 2): ...

Step 2: If extension needed, document and continue

If 3 of 4 (or 2 of 3 under n<10) dimensions win, extend to 3-month cap. Document the extension and continue weekly snapshots.

Step 3: Final commit + PR

git checkout -b docs/clickstack-decision
# Finalize doc
git commit -m "docs(clickstack): evaluation-window decision — Path <X>"
git push && gh pr create --title "docs(clickstack): evaluation decision (Path <X>)" --body "..."

This PR is the trigger for the Phase 2 spec (or the Phase 2 Path B revert PR per the spec’s “Path B revert recipe”).

Final cross-cutting checks (run before declaring migration complete)

Step 1: mkdocs build --strict succeeds (Run: uvx --with mkdocs-material mkdocs build --strict)
Step 2: helm template argocd/cluster-app ... renders clean (no unexpanded {{ }})
Step 3: All ArgoCD Applications Synced+Healthy (kubectl get applications -n argocd)
Step 4: Velero backup of clickstack namespace succeeds (manual trigger)
Step 5: No orphaned alloy artifacts (kubectl get ns alloy → NotFound; ls ansible/roles/ → no alloy dir)
Step 6: Beads issues closed for PR0-PR5
Step 7: Cilium follow-up bead updated (NetworkPolicy → CiliumNetworkPolicy migration tracked, if standard NP was shipped in PR1)

Self-review summary

This plan covers all 7 PRs in the spec (audit, 1, optional 1.5a/b, 2a, 2b, 3, 4a, 4b, plus the lived-in 5). Spec sections mapped:

Goals (spec lines 36-72): PR1 (storage tier + comparison runbook), PR2b + PR3 (Vector replaces Alloy), PR4a (Vector scraping with label parity).
Architecture / data flow (spec lines 109-141): PR1, PR2b, PR3, PR4a configs.
Known divergence classes (spec lines 143-170): documented in PR1’s clickstack-comparison.md skeleton.
PR sequence (spec lines 369-388): matches plan PR sections.
Evaluation plan (spec lines 419-455): PR5 (lived-in updates), Task 5.4 (decision-time).
Path B revert recipe (spec lines 459-528): referenced from spec; plan does not pre-execute (it’s conditional on eval outcome).
Cross-cutting compliance (spec lines 466-744): Vault policy (Task 1.2), certificates (Task 1.6 step 3-4), OIDC (Task 1.4), bearer-token via ExternalSecret + replicator (Task 2b.2 step 3), Vector RBAC (delegated to chart default in Task 2b.3), shadow-Prom topology (Task 4a.4 step 1), audit-output doc (Task 0.1).
Step 0 audit (spec lines 853-981): PR0 fully covers.
Risks (spec lines 985-1003): Mitigations referenced inline (disk buffer config, Mongo replica branching, baseline measurement, etc.).

Vector + ClickStack Parallel-Run Migration Implementation Plan

File Structure

Created (new)

Modified (existing)

Pre-flight: worktree + lefthook

PR0: Step 0 Audit (separate PR, merges to main first)

Task 0.1: Create the audit-output document

Task 0.2: Run audit and fill in outputs

PR1.5a: tf/authentik OIDC client support — SKIPPED

Task 1.5a.1: Create new branch from updated main

Task 1.5a.2: Add authentik_provider_oauth2 resource scaffolding

PR1.5b: tf/grafana ClickHouse datasource support — REQUIRED

Task 1.5b.1: Branch and add datasource resource

PR1: Deploy ClickStack storage tier + comparison-runbook skeleton

Task 1.1: Branch from updated main

Task 1.2: Vault policy

Task 1.3: Bootstrap Vault secrets (manual, runbook step)

Task 1.4: Authentik OIDC client + Vault secret write

Task 1.5: Grafana ClickHouse datasource variables

Task 1.6: ClickStack ArgoCD app-config files

Task 1.7: ClickStack ArgoCD Applications (two charts)

Task 1.8: Runbook and comparison-runbook docs

Task 1.9: Pre-push validation

PR2a: Add finalizer to monitoring-alloy Application

Task 2a.1: Branch from updated main

Task 2a.2: Add finalizer to existing Application

Task 2a.3: Verification gate before PR2b

PR2b: Vector k8s DaemonSet (replaces monitoring-alloy)

Task 2b.1: Branch

Task 2b.2: Vector ConfigMap (vector.yaml)

Task 2b.3: Vector ArgoCD Application

Task 2b.4: Delete monitoring-alloy

Task 2b.5: Push, open PR, monitor sync

PR3: Vector ansible role for router-hosts

Task 3.1: Branch

Task 3.2: Create ansible/roles/vector skeleton

Task 3.3: Wire role into playbook + remove alloy role

Task 3.4: Dry-run + smoke test on one router-host

Task 3.5: Push, open PR, roll out to remaining hosts

PR4a: Vector scrape configs + shadow-Prom parity gate

Task 4a.1: Branch

Task 4a.2: Author the ServiceMonitor translation list

Task 4a.3: Extend Vector config with prometheus_scrape sources

Task 4a.4: Shadow-Prom + shadow-AM + diff CronJob + promtool Job

Task 4a.5: Push, open PR, observe 24h parity gate

PR4b: Disable Prom native scrape + chart ServiceMonitors

Task 4b.1: Branch (after 24h gate green)

Task 4b.2: Disable Prom native scrape

Task 4b.3: Disable chart-installed ServiceMonitors

Task 4b.4: Push, open PR, monitor cutover

PR5: Comparison-runbook eval-window updates (lived-in)

Task 5.1: Schedule decision date

Task 5.2: Weekly resource-cost snapshot

Task 5.3: Investigation journaling

Task 5.4: Decision-time evaluation

Final cross-cutting checks (run before declaring migration complete)

Self-review summary

Task 2b.2: Vector ConfigMap (`vector.yaml`)