HyperDX dnsConfig GitOps Fix (retire the hl-qjh3.12 stopgap)

Design bead: hl-qjh3.14 · Date: 2026-06-30 · Author: Sean Brandt (+ Claude)

Predecessor: hl-qjh3.12 (CLOSED, PR #1492) shipped the interim this spec retires. Parent epic: hl-qjh3 (Prometheus decommission).

1. Context

hl-qjh3.12 root-caused a musl/Alpine DNS trap in the HyperDX image: Node fetch/undici uses musl getaddrinfo, which short-circuits on the cluster-appended fzymgc.house search domain (it returns NODATA for api.pushover.net.fzymgc.house, and musl stops the search there rather than falling through to the absolute name, as glibc would). The result: every HyperDX alert webhook POST to Pushover failed ENOTFOUND, while the c-ares-based ClickHouse/Mongo drivers worked. The fix is a pod dnsConfig with options ndots:1, which makes a multi-dot name like api.pushover.net resolve absolute-first and skip the search list.

The ClickStack chart (3.0.0, the latest published version) has no dnsConfig passthrough: the HyperDX Deployment template (templates/hyperdx/deployment.yaml in the published 3.0.0 tarball — upstream main has since refactored these into an hdx-oss-v2 subchart at charts/hdx-oss-v2/templates/hyperdx-deployment.yaml) exposes only nodeSelector / tolerations / topologySpreadConstraints / priorityClassName, and values.yaml has no dnsConfig key. So hl-qjh3.12 could not fix it through chart values and shipped a two-part interim:

A kubectl patch adding dnsConfig.options ndots=1 to the live cs-clickstack-app Deployment (out of Git — live-only state).
An ignoreDifferences entry on /spec/template/spec/dnsConfig in the clickstack ArgoCD Application, so selfHeal would not strip the manual field.

2. Problem

The interim is a GitOps anti-pattern flagged by the PR #1492 security review (hl-n5ju.1): the ignoreDifferences entry is a bidirectional blind spot. ArgoCD will neither re-apply the field if a Deployment recreation drops it (chart upgrade, prune-recreate, manual delete — ArgoCD still reports Synced while alerting silently breaks), nor detect / self-heal a malicious dnsConfig mutation (injected nameservers or search domains). The field must move into Git and the ignore must be deleted.

3. Decision

Mechanism A (operator-chosen, 2026-06-30): enable kustomize Helm-chart inflation in ArgoCD (--enable-helm) and convert the clickstack Application’s chart source to a kustomize helmCharts: source carrying a strategic-merge dnsConfig patch. The chart stays pinned upstream (no vendoring); the field lives in Git as a kustomize patch.

Alternatives considered and rejected:

B — upstream chart PR adding dnsConfig passthrough (idiomatic; repo has upstream-PR precedent). Rejected as the primary path because it is gated on external maintainer merge + release + a targetRevision bump, leaving the security blind spot live for weeks. May still be pursued later as an independent upstream contribution; it does not block this fix.
C — vendor/fork the chart in-repo. Rejected: clickstack is an umbrella chart with mongodb + clickhouse + otel-collector subcharts, so vendoring imports a large surface and an ongoing drift-tracking burden for a one-field fix.

4. Grounding (live + docs, 2026-06-30)

Patch target is a single object. cronjob-check-alerts is not deployed (the chart ships templates/hyperdx/cronjob-check-alerts.yaml — task-checkAlerts.yaml in the hdx-oss-v2 subchart on upstream main — but it is gated off by tasks.enabled which defaults false, and is confirmed absent live); the ALERT-TASK webhook runs only in the cs-clickstack-app Deployment. A kustomize patch targeting the absent CronJob would fail kustomize build with “no matches for target”, so the patch set is Deployment-only.
Multi-source Application. clickstack.yaml has source[0] = the Helm chart (with a ~200-line inline valuesObject) and source[1] = a git path (argocd/app-configs/clickstack, the ExternalSecret). The two sources emit different resources (chart → clickstack-secret Secret; git path → ExternalSecret), so there is no collision; ArgoCD’s “last source wins on collision” rule is not triggered, but source order must be preserved.
ArgoCD support confirmed (deepwiki argoproj/argo-cd): a multi-source Application supports a kustomize-helmCharts source alongside a plain-git source; --enable-helm is set via kustomize.buildOptions: --enable-helm in argocd-cm; kustomize inflation runs helm template (no hooks / no lookup) — and this chart is hook-free and CRD-free (its operator CRDs live in the separate clickstack-operators app), so the render is faithful.
No existing precedent. There is no helmCharts: usage anywhere in argocd/ or tf/ today, and argocd.tf configs.cm has no kustomize.buildOptions. This change introduces a new, reusable repo pattern.
Name preservation is load-bearing. releaseName must stay cs (it is the prefix on every resource name, e.g. cs-clickstack-app) and namespace must stay clickstack, or the conversion mass-renames/relocates resources and triggers a full recreate of ClickHouse / MongoDB / HyperDX.

5. Design

5.1 Two-PR sequence (hard ordering dependency)

--enable-helm must be live in argocd-repo-server before the Application is restructured, or kustomize build rejects the helmCharts: field and the app degrades.

PR #1 — Terraform (tf/cluster-bootstrap/argocd.tf): add "kustomize.buildOptions" = "--enable-helm" to the ArgoCD Helm-release configs.cm map. tf/cluster-bootstrap (main-cluster-bootstrap) is a Local-execution, Manual-trigger workspace — it bootstraps the HCP TF Operator itself, so it cannot use the agent and is not VCS/merge-triggered (unlike the Agent workspaces tf/keycloak / tf/vault / etc.; see docs/operations/hcp-terraform.md and tf/CLAUDE.md). Merging the PR does not apply it; after merge an operator runs it locally:
Terminal window
```
export VAULT_TOKEN=...   # operator's cluster-bootstrap local-apply token (see note below)
terraform -chdir=tf/cluster-bootstrap apply
```
Vault auth note: tf/cluster-bootstrap is excluded from the per-workspace OIDC/Vault-policy infrastructure (tf/vault/jwt-hcp-terraform.tf) and has no terraform-cluster-bootstrap-admin policy — its vault provider auths via the VAULT_TOKEN env var (terraform.tf:20-22). The terraform-WORKSPACE-admin snippet in docs/operations/hcp-terraform.md is a placeholder template, not a literal policy; use the operator’s standard cluster-bootstrap local-apply token.

Gate (MUST pass before PR #2 merges; operator-confirmed, not merge-inferred):
Terminal window
```
kubectl -n argocd get cm argocd-cm -o jsonpath='{.data.kustomize\.buildOptions}'  # → --enable-helm
kubectl -n argocd rollout restart deploy/argocd-repo-server   # force a clean reload of the option
```
PR #2 — GitOps (this repo): restructure the Application source, add the patch, and remove the interim. Merged only after the PR #1 gate passes.

5.2 New kustomize directory (PR #2)

Add argocd/app-configs/clickstack-chart/:

values.yaml — the current valuesObject lifted verbatim out of the Application (kept as a file, not inlined, to preserve its in-line rationale comments and give a clean diff surface).

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
  - name: clickstack
    repo: https://clickhouse.github.io/ClickStack-helm-charts
    version: 3.0.0
    releaseName: cs          # MUST match — prefix of every resource name
    namespace: clickstack    # MUST match — chart resources land here
    valuesFile: values.yaml
patches:
  - target:
      kind: Deployment
      name: cs-clickstack-app
    patch: |-
      - op: add
        path: /spec/template/spec/dnsConfig
        value:
          options:
            - name: ndots
              value: "1"

(Patch form — JSON6902 vs strategic-merge — finalized at plan time against what the inflated manifest accepts; the strategic-merge equivalent is acceptable if cleaner.)

5.3 Application restructure (PR #2)

In argocd/cluster-app/templates/clickstack.yaml, replace source[0] (the chart: Helm source with its inline valuesObject) with a git source pointing at the new directory:

sources:
  - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
    targetRevision: HEAD
    path: argocd/app-configs/clickstack-chart
  - repoURL: https://github.com/fzymgc-house/selfhosted-cluster   # unchanged
    targetRevision: HEAD
    path: argocd/app-configs/clickstack

Preserve source order and everything below sources: (destination, syncPolicy).

5.4 Interim removal (PR #2)

Delete only the Deployment/cs-clickstack-app → /spec/template/spec/dnsConfig entry from ignoreDifferences.
Keep the ExternalSecret strategy entries and the clickstack-secret /data entry, and all syncOptions (RespectIgnoreDifferences, ServerSideApply, CreateNamespace).
No manual unpatch needed: the kustomize-rendered Deployment now carries an identical dnsConfig, so ArgoCD reconciles the field in place (same content → no rollout churn) and owns it going forward.

5.5 Render-fidelity safety gate (the linchpin)

Because inflation runs helm template (not install), there is a residual risk of .Capabilities / api-version render differences for the umbrella chart. Before merging PR #2, diff the current native-Helm render against the new kustomize build --enable-helm output and require the only delta to be the added dnsConfig.

Run the diff against the lifted values.yaml (§5.2), not a hand-retyped copy: extract the inline valuesObject (today at argocd/cluster-app/templates/clickstack.yaml, the valuesObject: block) into argocd/app-configs/clickstack-chart/values.yaml first, diff, and only then delete the inline block from the Application — so the gate validates the exact file that ships.

# baseline: today's render, from the lifted values.yaml (same content as the inline valuesObject)
helm template cs clickstack --repo https://clickhouse.github.io/ClickStack-helm-charts \
  --version 3.0.0 -n clickstack -f argocd/app-configs/clickstack-chart/values.yaml > /tmp/native.yaml
# candidate: the kustomize path
kustomize build --enable-helm argocd/app-configs/clickstack-chart > /tmp/kustomize.yaml
dyff between /tmp/native.yaml /tmp/kustomize.yaml   # expect: only +dnsConfig on the Deployment

If the diff shows anything beyond dnsConfig, add kubeVersion / apiVersions to the helmCharts entry to match the cluster, or pin the resolved difference, until the diff is clean.

6. Risks & rollback

Risk	Mitigation
`--enable-helm` not live when PR #2 syncs → `kustomize build` fails, app degrades	PR #1 gate (verify flag live) before PR #2 merge; ordering is the whole point of the split
Inflation render differs from native helm (Capabilities/api-versions)	§5.5 render-diff gate is a hard pre-merge check; `kubeVersion`/`apiVersions` knobs available
`releaseName`/`namespace` drift → mass resource rename/recreate	Both pinned explicitly in `helmCharts`; the render-diff gate would surface any name change immediately
Rollback	Revert PR #2 (Application returns to native-helm source); the live manual patch is re-applied as the documented interim until re-attempt. PR #1 (`--enable-helm`) is inert on its own and can stay.

--enable-helm is cluster-wide but opt-in per kustomization (only dirs with helmCharts: use it) and does not expand the trust boundary — repo write access already implies arbitrary deploy. Net security posture improves: the bidirectional dnsConfig blind spot is removed.

7. Out of scope

The cluster-wide fzymgc.house search-domain root cause (tracked separately under the internal zone / DNS epics hl-zh3w, hl-620y). This spec only neutralizes its effect on HyperDX.
An upstream chart dnsConfig passthrough PR (option B) — may be pursued independently later; it does not block this fix.
Any change to the clickstack-operators, clickstack-alerts, or monitoring-vector apps.

8. Acceptance

dnsConfig.options ndots:1 on cs-clickstack-app originates from Git (kustomize patch), not a live kubectl patch.
The Deployment/cs-clickstack-app dnsConfig ignoreDifferences entry is gone; the other ignoreDifferences entries and all syncOptions are intact.
clickstack Application is Synced + Healthy after the restructure, with the render-diff gate showing only the dnsConfig delta.
A HyperDX alert webhook fire-test still reaches Pushover (DNS resolution intact).
Docs updated for the new operational pattern (root CLAUDE.md: “MUST update docs with every PR that changes operational patterns”): argocd/CLAUDE.md documents the kustomize-helmCharts source convention (it currently describes only plain-Helm-source + ExternalSecret apps), and the --enable-helm argocd-cm setting is noted where ArgoCD config is described (docs/operations/hcp-terraform.md and/or docs/operations/clickstack.md). This is in scope for PR #2 (TF-config note may ride with PR #1).