Skip to content

Design: ClickStack user lockdown

See openspec/changes/clickstack-user-lockdown/proposal.md, section “Why”, for the exposure. The facts below shape the approach.

The ClickStack chart (version 3.4.0, inflated through kustomize helmCharts) renders a ClickHouseCluster object for the official ClickHouse operator (clickhouse.com/v1alpha1). The operator writes the main users file of each replica, users.yaml, and the repository does not control it. On the live replica cs-clickstack-clickhouse-clickhouse-0-0 that file reads:

users:
default:
profile: default
quota: default
no_password:
grants:
- query: "GRANT ALL ON *.* WITH GRANT OPTION"
operator:
password_sha256_hex: ...
grants:
- query: "GRANT ALL ON *.*"

The operator also writes etc-clickhouse-client-config-yaml with user: default, so clickhouse-client inside a pod connects as default over the loopback with no password. The operator itself connects as operator.

Everything under clickhouse.cluster.spec.settings.extraUsersConfig in argocd/app-configs/clickstack-chart/values.yaml lands verbatim in a second file, 99-extra-users-config.yaml, which ClickHouse merges over users.yaml. The five users that this repository added (mcp_readonly, mcp_readwrite, dns_control_reader, schema_admin, and the profile overrides of app and otelcollector) already go through that file, and three of them take their password from the pod environment with @from_env, fed by containerTemplate.env entries that read clickstack-secret.

The chart sets the app and otelcollector passwords from hyperdx.secrets.*, which this deployment replaces with clickstack-secret through useExistingConfigSecret. The ExternalSecret in argocd/app-configs/clickstack/external-secret.yaml renders that Secret with creationPolicy: Merge, and it writes the two passwords as literals, once each in CLICKHOUSE_PASSWORD and CLICKHOUSE_APP_PASSWORD, and once more inside connections.json. Vector carries the otelcollector password as a second literal in argocd/app-configs/monitoring-vector/external-secret.yaml.

Four consumers log in with those two passwords:

Consumer User Reads the password from
HyperDX app (cs-clickstack-app) app the connection stored in MongoDB, seeded once from connections.json
ClickStack collector (cs-otel-collector) otelcollector clickstack-secret, CLICKHOUSE_PASSWORD, through envFrom
Watchdog CronJob app clickstack-secret, CLICKHOUSE_APP_PASSWORD
Vector DaemonSet otelcollector Secret otel-ingest-token, key CLICKHOUSE_PASSWORD, as a file

HyperDX reads connections.json only when the team has no connection yet (packages/api/src/setupDefaults.ts, setupTeamDefaults). After that, the password lives in MongoDB, and a restart does not update it.

Traefik forwards ch.fzymgc.house to port 8123 and ch-native.fzymgc.house to port 9000 of the headless Service. The Cilium policy clickstack-default allows every port from the traefik namespace. docs/reference/network.md places the Traefik address 192.168.250.145 in the Lab static block, which the tailnet advertises. The uptime-kuma monitor reads https://ch.fzymgc.house/ping, which needs no credentials.

Issue #2412 records that the clickstack Application combines RespectIgnoreDifferences=true with jqPathExpressions over .spec.data[] of the ExternalSecret. ArgoCD cannot add an array item through that normalizer, so a new spec.data entry never reaches the cluster. This change adds two entries.

Goals:

  • Refuse the default user from every address but the loopback of its own pod.
  • Keep the in-pod clickhouse-client path, so no runbook command changes.
  • Take the app and otelcollector passwords from Vault, with no literal left in argocd/.
  • Give the operator a rotation procedure that works for any ClickHouse user.
  • Make a new key on clickstack-secret reach the cluster on sync.

Non-Goals:

  • A password for default. See decision 1.
  • Removing the Traefik routes to ClickHouse. Operators use ch-native.fzymgc.house from a workstation, and the watchdog monitor reads /ping through Traefik.
  • The MongoDB password and the app grants (see openspec/changes/clickstack-user-lockdown/proposal.md, section “What Changes”).
  • A port rule in the Cilium policy for the traefik namespace. It would not change what a LAN host can reach, because Traefik only forwards the two ClickHouse ports.

1. Restrict default by network, not by password

Section titled “1. Restrict default by network, not by password”

extraUsersConfig.users.default.networks.ip gets the two loopback addresses, ::1 and 127.0.0.1. ClickHouse then answers every other source with Code: 516 ... default: Authentication failed, whatever credentials the request carries.

Alternatives:

  • A password on default through @from_env. The operator’s users.yaml carries no_password:, and ClickHouse refuses to start a user that names two authentication methods. Removing the key needs no_password: {"@remove": "1"} in the merged file, and a wrong merge stops the server. The network rule adds one key and conflicts with nothing.
  • Removing the default user. The operator’s client configuration names it, and the operator can recreate it at any reconcile.

The grants of default stay as the operator writes them. A process inside the pod keeps full access, which needs kubectl exec and so Kubernetes RBAC.

2. The passwords come from Vault through the same path as the MCP users

Section titled “2. The passwords come from Vault through the same path as the MCP users”

Two new keys at fzymgc-house/cluster/clickstack: app_password and otelcollector_password. The clickstack ExternalSecret gains two spec.data entries and renders CLICKHOUSE_PASSWORD, CLICKHOUSE_APP_PASSWORD and the password in connections.json from them. In values.yaml, users.app.password and users.otelcollector.password become @from_env lists, and containerTemplate.env gains APP_PASSWORD from clickstack-secret/CLICKHOUSE_APP_PASSWORD and OTELCOLLECTOR_PASSWORD from clickstack-secret/CLICKHOUSE_PASSWORD. This is the pattern that mcp_readonly, mcp_readwrite, dns_control_reader and schema_admin already use.

The Vector ExternalSecret otel-ingest-token replaces its one spec.data entry, otel_ingest_api_key, with otelcollector_password, and renders CLICKHOUSE_PASSWORD from it. Vector never read otel_ingest_api_key. The entry existed only because ESO rejects an ExternalSecret with no data. The monitoring-vector Application ignores .spec.data[] fields in the diff only, without RespectIgnoreDifferences, so the new entry syncs.

Alternative: one Secret for every consumer. Vector runs in its own namespace and reads the password as a file, so it keeps its own ExternalSecret.

3. The plumbing lands with the current values, and the rotation is an operator procedure

Section titled “3. The plumbing lands with the current values, and the rotation is an operator procedure”

The operator seeds the two Vault keys with the current values, hyperdx and otelcollectorpass, before pull request 3 merges. The sync then changes no password, and a render diff shows only the source of each value. Every consumer restarts once on that sync: HyperDX and the collector through Reloader, Vector through the Reloader auto annotation that its DaemonSet already carries, and the ClickHouse replicas through the operator, because the pod template gained two env entries.

The rotation is a runbook procedure in docs/operations/clickstack.md, run once per user after the merge:

  1. printf '%s' "$(openssl rand -base64 32)" | vault kv patch secret/fzymgc-house/cluster/clickstack <key>=-
  2. Wait for the ExternalSecrets, or annotate them with force-sync, and read SecretSynced.
  3. Delete the ClickHouse pods one at a time, 0-1-0 then 0-0-0, and wait for 1/1 between them. The operator recreates each pod, and the new process reads the new value.
  4. Reloader restarts HyperDX, the collector and Vector on the Secret change.
  5. For app only: in HyperDX, Team Settings, Connections, set the password of Local ClickHouse to the new value. HyperDX keeps the connection in MongoDB and does not read connections.json again.
  6. Run the acceptance commands of the proposal.

Alternative: rotate in the same pull request. The window between the ClickHouse restart and the consumer restarts is then as long as the ESO refresh interval, 15 minutes, with no person watching. The two-step shape keeps every restart inside one operator session.

4. The #2412 fix lands first, as its own pull request

Section titled “4. The #2412 fix lands first, as its own pull request”

Each remoteRef in argocd/app-configs/clickstack/external-secret.yaml states conversionStrategy: Default, decodingStrategy: None and metadataPolicy: None, which are the values the ExternalSecret CRD defaults to. The three jqPathExpressions leave the clickstack Application. RespectIgnoreDifferences=true stays, because the /data rule on the Secret still needs it. The runbook section “Unstick a sync that ignores array fields” in docs/operations/argocd.md gains the correction from the issue comment of 2026-09-27: a server-side apply as argocd-controller also removes the argocd.argoproj.io/tracking-id annotation, so the recovery command must name a different field manager.

  • [The operator does not merge a default entry from extraUsersConfig] → The render task reads the ClickHouseCluster object from kubectl kustomize --enable-helm, and the acceptance reads system.users for default after the sync. If host_ip still reads ::/0, revert the pull request and open the question with the operator’s user handling.
  • [A ClickHouse restart pauses ingest] → The collector holds rows in a 2Gi disk queue and Vector in a 5Gi disk buffer. The replicas restart one at a time, and the other replica serves reads meanwhile.
  • [A mismatched password during the rotation drops rows] → The procedure restarts ClickHouse before the consumers, and Reloader restarts the consumers within seconds of the Secret change. The queues above cover the gap. The spec scenario “Rows arrive after the rotation” measures the outcome.
  • [HyperDX keeps the old app password in MongoDB] → Step 5 of the procedure. Until it runs, HyperDX dashboards and alerts read nothing, and the HyperDX MCP tools fail.
  • [The Vault keys are missing when the sync runs] → ESO reports SecretSyncedError and the ClickHouse pods stop in CreateContainerConfigError, the failure that #2412 records. The seed is an operator task that must finish before the merge of pull request 3.
  • [A LAN tool connected as default through Traefik] → No manifest in argocd/, no role in ansible/ and no Terraform workspace names default as a ClickHouse user. The uptime-kuma monitor reads /ping, which needs no user. A person who used default from a workstation switches to schema_admin.

Pull requests land in the order the proposal lists. Each is one sync, and each has a rollback:

Step Live effect Rollback
PR 1, #2412 fix ArgoCD applies the full spec.data of the ExternalSecret. No pod restarts. git revert the merge commit.
PR 2, loopback rule The operator rewrites 99-extra-users-config.yaml, and ClickHouse reloads the users file without a restart. git revert the merge commit. The reload opens default again within one reconcile.
PR 3, Vault plumbing Both ClickHouse replicas restart one at a time. HyperDX, the collector and Vector restart once. No password changes. git revert the merge commit. The Vault keys stay and harm nothing.
Rotation, operator One restart per ClickHouse replica per user, and one HyperDX connection edit. vault kv patch the key back to its last value, then run the procedure again.

A copy of this design goes to docs/engineering/specs/2026-10-11-clickstack-user-lockdown-design.md, because the rotation procedure is a new operational pattern.