Design: ClickStack user lockdown
Context
Section titled “Context”See openspec/changes/clickstack-user-lockdown/proposal.md, section “Why”, for the exposure.
The facts below shape the approach.
The ClickStack chart (version 3.4.0, inflated through kustomize helmCharts) renders a
ClickHouseCluster object for the official ClickHouse operator (clickhouse.com/v1alpha1). The
operator writes the main users file of each replica, users.yaml, and the repository does not
control it. On the live replica cs-clickstack-clickhouse-clickhouse-0-0 that file reads:
users: default: profile: default quota: default no_password: grants: - query: "GRANT ALL ON *.* WITH GRANT OPTION" operator: password_sha256_hex: ... grants: - query: "GRANT ALL ON *.*"The operator also writes etc-clickhouse-client-config-yaml with user: default, so
clickhouse-client inside a pod connects as default over the loopback with no password.
The operator itself connects as operator.
Everything under clickhouse.cluster.spec.settings.extraUsersConfig in
argocd/app-configs/clickstack-chart/values.yaml lands verbatim in a second file,
99-extra-users-config.yaml, which ClickHouse merges over users.yaml. The five users that this
repository added (mcp_readonly, mcp_readwrite, dns_control_reader, schema_admin, and the
profile overrides of app and otelcollector) already go through that file, and three of them
take their password from the pod environment with @from_env, fed by containerTemplate.env
entries that read clickstack-secret.
The chart sets the app and otelcollector passwords from hyperdx.secrets.*, which this
deployment replaces with clickstack-secret through useExistingConfigSecret. The ExternalSecret
in argocd/app-configs/clickstack/external-secret.yaml renders that Secret with creationPolicy: Merge, and it writes the two passwords as literals, once each in CLICKHOUSE_PASSWORD and
CLICKHOUSE_APP_PASSWORD, and once more inside connections.json. Vector carries the
otelcollector password as a second literal in
argocd/app-configs/monitoring-vector/external-secret.yaml.
Four consumers log in with those two passwords:
| Consumer | User | Reads the password from |
|---|---|---|
HyperDX app (cs-clickstack-app) |
app |
the connection stored in MongoDB, seeded once from connections.json |
ClickStack collector (cs-otel-collector) |
otelcollector |
clickstack-secret, CLICKHOUSE_PASSWORD, through envFrom |
| Watchdog CronJob | app |
clickstack-secret, CLICKHOUSE_APP_PASSWORD |
| Vector DaemonSet | otelcollector |
Secret otel-ingest-token, key CLICKHOUSE_PASSWORD, as a file |
HyperDX reads connections.json only when the team has no connection yet
(packages/api/src/setupDefaults.ts, setupTeamDefaults). After that, the password lives in
MongoDB, and a restart does not update it.
Traefik forwards ch.fzymgc.house to port 8123 and ch-native.fzymgc.house to port 9000 of the
headless Service. The Cilium policy clickstack-default allows every port from the traefik
namespace. docs/reference/network.md places the Traefik address 192.168.250.145 in the Lab
static block, which the tailnet advertises. The uptime-kuma monitor reads
https://ch.fzymgc.house/ping, which needs no credentials.
Issue #2412 records that the clickstack Application combines RespectIgnoreDifferences=true
with jqPathExpressions over .spec.data[] of the ExternalSecret. ArgoCD cannot add an array
item through that normalizer, so a new spec.data entry never reaches the cluster. This change
adds two entries.
Goals / Non-Goals
Section titled “Goals / Non-Goals”Goals:
- Refuse the
defaultuser from every address but the loopback of its own pod. - Keep the in-pod
clickhouse-clientpath, so no runbook command changes. - Take the
appandotelcollectorpasswords from Vault, with no literal left inargocd/. - Give the operator a rotation procedure that works for any ClickHouse user.
- Make a new key on
clickstack-secretreach the cluster on sync.
Non-Goals:
- A password for
default. See decision 1. - Removing the Traefik routes to ClickHouse. Operators use
ch-native.fzymgc.housefrom a workstation, and the watchdog monitor reads/pingthrough Traefik. - The MongoDB password and the
appgrants (seeopenspec/changes/clickstack-user-lockdown/proposal.md, section “What Changes”). - A port rule in the Cilium policy for the
traefiknamespace. It would not change what a LAN host can reach, because Traefik only forwards the two ClickHouse ports.
Decisions
Section titled “Decisions”1. Restrict default by network, not by password
Section titled “1. Restrict default by network, not by password”extraUsersConfig.users.default.networks.ip gets the two loopback addresses, ::1 and
127.0.0.1. ClickHouse then answers every other source with Code: 516 ... default: Authentication failed, whatever credentials the request carries.
Alternatives:
- A password on
defaultthrough@from_env. The operator’susers.yamlcarriesno_password:, and ClickHouse refuses to start a user that names two authentication methods. Removing the key needsno_password: {"@remove": "1"}in the merged file, and a wrong merge stops the server. The network rule adds one key and conflicts with nothing. - Removing the
defaultuser. The operator’s client configuration names it, and the operator can recreate it at any reconcile.
The grants of default stay as the operator writes them. A process inside the pod keeps full
access, which needs kubectl exec and so Kubernetes RBAC.
2. The passwords come from Vault through the same path as the MCP users
Section titled “2. The passwords come from Vault through the same path as the MCP users”Two new keys at fzymgc-house/cluster/clickstack: app_password and otelcollector_password.
The clickstack ExternalSecret gains two spec.data entries and renders CLICKHOUSE_PASSWORD,
CLICKHOUSE_APP_PASSWORD and the password in connections.json from them. In values.yaml,
users.app.password and users.otelcollector.password become @from_env lists, and
containerTemplate.env gains APP_PASSWORD from clickstack-secret/CLICKHOUSE_APP_PASSWORD
and OTELCOLLECTOR_PASSWORD from clickstack-secret/CLICKHOUSE_PASSWORD. This is the pattern
that mcp_readonly, mcp_readwrite, dns_control_reader and schema_admin already use.
The Vector ExternalSecret otel-ingest-token replaces its one spec.data entry,
otel_ingest_api_key, with otelcollector_password, and renders CLICKHOUSE_PASSWORD from it.
Vector never read otel_ingest_api_key. The entry existed only because ESO rejects an
ExternalSecret with no data. The monitoring-vector
Application ignores .spec.data[] fields in the diff only, without RespectIgnoreDifferences,
so the new entry syncs.
Alternative: one Secret for every consumer. Vector runs in its own namespace and reads the password as a file, so it keeps its own ExternalSecret.
3. The plumbing lands with the current values, and the rotation is an operator procedure
Section titled “3. The plumbing lands with the current values, and the rotation is an operator procedure”The operator seeds the two Vault keys with the current values, hyperdx and
otelcollectorpass, before pull request 3 merges. The sync then changes no password, and a
render diff shows only the source of each value. Every consumer restarts once on that sync:
HyperDX and the collector through Reloader, Vector through the Reloader auto annotation that
its DaemonSet already carries, and the ClickHouse replicas through the operator, because the pod
template gained two env entries.
The rotation is a runbook procedure in docs/operations/clickstack.md, run once per user after
the merge:
printf '%s' "$(openssl rand -base64 32)" | vault kv patch secret/fzymgc-house/cluster/clickstack <key>=-- Wait for the ExternalSecrets, or annotate them with
force-sync, and readSecretSynced. - Delete the ClickHouse pods one at a time,
0-1-0then0-0-0, and wait for1/1between them. The operator recreates each pod, and the new process reads the new value. - Reloader restarts HyperDX, the collector and Vector on the Secret change.
- For
apponly: in HyperDX, Team Settings, Connections, set the password ofLocal ClickHouseto the new value. HyperDX keeps the connection in MongoDB and does not readconnections.jsonagain. - Run the acceptance commands of the proposal.
Alternative: rotate in the same pull request. The window between the ClickHouse restart and the consumer restarts is then as long as the ESO refresh interval, 15 minutes, with no person watching. The two-step shape keeps every restart inside one operator session.
4. The #2412 fix lands first, as its own pull request
Section titled “4. The #2412 fix lands first, as its own pull request”Each remoteRef in argocd/app-configs/clickstack/external-secret.yaml states
conversionStrategy: Default, decodingStrategy: None and metadataPolicy: None, which are the
values the ExternalSecret CRD defaults to. The three jqPathExpressions leave the clickstack
Application. RespectIgnoreDifferences=true stays, because the /data rule on the Secret still
needs it. The runbook section “Unstick a sync that ignores array fields” in
docs/operations/argocd.md gains the correction from the issue comment of 2026-09-27: a
server-side apply as argocd-controller also removes the argocd.argoproj.io/tracking-id
annotation, so the recovery command must name a different field manager.
Risks / Trade-offs
Section titled “Risks / Trade-offs”- [The operator does not merge a
defaultentry fromextraUsersConfig] → The render task reads theClickHouseClusterobject fromkubectl kustomize --enable-helm, and the acceptance readssystem.usersfordefaultafter the sync. Ifhost_ipstill reads::/0, revert the pull request and open the question with the operator’s user handling. - [A ClickHouse restart pauses ingest] → The collector holds rows in a 2Gi disk queue and Vector in a 5Gi disk buffer. The replicas restart one at a time, and the other replica serves reads meanwhile.
- [A mismatched password during the rotation drops rows] → The procedure restarts ClickHouse before the consumers, and Reloader restarts the consumers within seconds of the Secret change. The queues above cover the gap. The spec scenario “Rows arrive after the rotation” measures the outcome.
- [HyperDX keeps the old
apppassword in MongoDB] → Step 5 of the procedure. Until it runs, HyperDX dashboards and alerts read nothing, and the HyperDX MCP tools fail. - [The Vault keys are missing when the sync runs] → ESO reports
SecretSyncedErrorand the ClickHouse pods stop inCreateContainerConfigError, the failure that #2412 records. The seed is an operator task that must finish before the merge of pull request 3. - [A LAN tool connected as
defaultthrough Traefik] → No manifest inargocd/, no role inansible/and no Terraform workspace namesdefaultas a ClickHouse user. The uptime-kuma monitor reads/ping, which needs no user. A person who useddefaultfrom a workstation switches toschema_admin.
Migration Plan
Section titled “Migration Plan”Pull requests land in the order the proposal lists. Each is one sync, and each has a rollback:
| Step | Live effect | Rollback |
|---|---|---|
| PR 1, #2412 fix | ArgoCD applies the full spec.data of the ExternalSecret. No pod restarts. |
git revert the merge commit. |
| PR 2, loopback rule | The operator rewrites 99-extra-users-config.yaml, and ClickHouse reloads the users file without a restart. |
git revert the merge commit. The reload opens default again within one reconcile. |
| PR 3, Vault plumbing | Both ClickHouse replicas restart one at a time. HyperDX, the collector and Vector restart once. No password changes. | git revert the merge commit. The Vault keys stay and harm nothing. |
| Rotation, operator | One restart per ClickHouse replica per user, and one HyperDX connection edit. | vault kv patch the key back to its last value, then run the procedure again. |
A copy of this design goes to docs/engineering/specs/2026-10-11-clickstack-user-lockdown-design.md,
because the rotation procedure is a new operational pattern.