HyperDX dnsConfig GitOps Fix Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Move the HyperDX dnsConfig: {options: [{name: ndots, value: "1"}]} from a live-only kubectl patch into Git via ArgoCD kustomize-helmCharts inflation, and delete the bidirectional ignoreDifferences blind spot shipped by hl-qjh3.12.

Architecture: Two ordered PRs. PR #1 enables kustomize.buildOptions: --enable-helm on ArgoCD (Terraform, tf/cluster-bootstrap — a Local/Manual workspace, operator-applied). PR #2 converts the clickstack Application’s chart source into a kustomize directory that inflates the pinned 3.0.0 chart via helmCharts: and applies a strategic-merge dnsConfig patch to the single cs-clickstack-app Deployment, then removes the interim. A byte-level render diff (helm template vs kustomize build --enable-helm) gates PR #2.

Tech Stack: ArgoCD (multi-source Application), Kustomize helmCharts generator, Helm (ClickStack umbrella chart 3.0.0), Terraform (HCP TF Local workspace), dyff for render diffing.

Spec: docs/engineering/specs/2026-06-30-hyperdx-dnsconfig-gitops-design.md

Ordering invariant: PR #1’s terraform apply MUST complete and its gate MUST pass before PR #2 merges. tf/cluster-bootstrap is not merge-triggered (Local/Manual workspace) — merging PR #1 does nothing until an operator runs terraform apply locally.

---

Implementation tasks

Task 1: Enable kustomize Helm inflation on ArgoCD (PR #1)

Files:

• Modify: tf/cluster-bootstrap/argocd.tf (the configs.cm map, opens at line 166)

This is infrastructure config applied through a Local Terraform workspace — there is no unit test; the verification is the live argocd-cm gate in Step 5 (TDD config exception).

• Step 1: Add the build option to configs.cm

In tf/cluster-bootstrap/argocd.tf, inside the cm = { ... } map (opens at line 166, alongside url, "dex.config"), add:

        "kustomize.buildOptions" = "--enable-helm"

Place it as a sibling key of url within the cm map (ordering within the map is irrelevant to Terraform; keep it near url for readability).

• Step 2: Validate the Terraform

Run: terraform -chdir=tf/cluster-bootstrap fmt && terraform -chdir=tf/cluster-bootstrap validate Expected: Success! The configuration is valid. (init may be required first: terraform -chdir=tf/cluster-bootstrap init -backend=false for a validate-only check.)

• Step 3: Commit

feat(argocd): enable kustomize --enable-helm build option [hl-qjh3.14]

(Co-Authored-By byline per repo convention.)

• Step 4: Open PR #1, merge, then operator-apply locally

tf/cluster-bootstrap (main-cluster-bootstrap) is a Local-execution / Manual-trigger workspace — it bootstraps the HCP TF Operator itself and cannot use the agent, so merging the PR does not apply it. After merge, an operator runs the apply locally.

Vault auth (important — do not invent a policy): the vault provider authenticates via the VAULT_TOKEN env var (tf/cluster-bootstrap/terraform.tf:20-22). tf/cluster-bootstrap is intentionally excluded from the per-workspace OIDC/Vault-policy infrastructure (tf/vault/jwt-hcp-terraform.tf) and has no dedicated terraform-cluster-bootstrap-admin policy (the defined terraform-*-admin policies live in tf/vault/policy-terraform-workspaces.tf and do not include cluster-bootstrap). The terraform-WORKSPACE-admin snippet in docs/operations/hcp-terraform.md “Local-Only Workspaces” is a placeholder template, not a literal policy name. Use the operator’s standard cluster-bootstrap local-apply credential — a VAULT_TOKEN with read access to the KV paths the workspace consumes (the argocd config + ICA1 CA secrets it reads via data "vault_kv_secret_v2").

export VAULT_TOKEN=...   # operator's cluster-bootstrap local-apply token (see note above)
terraform -chdir=tf/cluster-bootstrap apply

Expected: plan shows the argocd Helm release updating argocd-cm with the new kustomize.buildOptions key; apply succeeds.

• Step 5: Gate — verify the flag is live (MUST pass before Task 2’s PR merges)

kubectl -n argocd get cm argocd-cm -o jsonpath='{.data.kustomize\.buildOptions}'; echo
kubectl -n argocd rollout restart deploy/argocd-repo-server
kubectl -n argocd rollout status deploy/argocd-repo-server

Expected: the jsonpath prints --enable-helm; repo-server rolls out cleanly. Only then proceed to merge Task 2’s PR.

---

Task 2: Restructure clickstack to kustomize-helmCharts + dnsConfig patch + remove interim (PR #2)

Files:

• Create: argocd/app-configs/clickstack-chart/values.yaml
• Create: argocd/app-configs/clickstack-chart/kustomization.yaml
• Modify: argocd/cluster-app/templates/clickstack.yaml (replace source[0] lines 13-215; remove the Deployment/dnsConfig ignoreDifferences entry plus its preceding comment block — lines ~254-271, from the # cs-clickstack-app dnsConfig: INTERIM stopgap comment through the - /spec/template/spec/dnsConfig jsonPointer)
• Modify: argocd/CLAUDE.md (document the kustomize-helmCharts convention)

No unit test; the verification is the render-diff gate (Step 4) — the design’s linchpin safety control.

• Step 1: Lift the inline valuesObject into values.yaml

Copy the body of the valuesObject: block (argocd/cluster-app/templates/clickstack.yaml, lines 19-215 — everything indented under valuesObject: at line 18) into a new file argocd/app-configs/clickstack-chart/values.yaml, de-indented to column 0 (the block is currently indented 10 spaces under valuesObject:; strip those 10 leading spaces so hyperdx:, mongodb:, clickhouse:, otel-collector: become top-level keys). Preserve all inline comments verbatim.

The first lines of values.yaml should read:

hyperdx:
  config:
    FRONTEND_URL: "https://hyperdx.fzymgc.house"
    # ... (rest of the lifted block, de-indented)

• Step 2: Create kustomization.yaml

Create argocd/app-configs/clickstack-chart/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: clickstack
    repo: https://clickhouse.github.io/ClickStack-helm-charts
    version: 3.0.0
    releaseName: cs          # MUST match — prefix of every resource name (cs-clickstack-app)
    namespace: clickstack    # MUST match — chart resources land here
    valuesFile: values.yaml

patches:
  - target:
      kind: Deployment
      name: cs-clickstack-app
    patch: |-
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: cs-clickstack-app
      spec:
        template:
          spec:
            dnsConfig:
              options:
                - name: ndots
                  value: "1"

(Strategic-merge form shown; if kustomize build rejects it for the inflated manifest, fall back to the JSON6902 op: add form from spec §5.2.)

• Step 3: Lint the new YAML

Run: yamllint argocd/app-configs/clickstack-chart/ Expected: no errors (match repo .yamllint.yaml rules).

• Step 4: Render-diff safety gate (the linchpin — run BEFORE editing the Application)

Tooling prerequisites (operator workstation — not currently listed in the repo’s docs/getting-started/environment.md; install if which helm kustomize dyff shows any missing): helm, a standalone kustomize ≥ v5 (needed for --enable-helm), and dyff. Fallbacks if a tool is unavailable: kubectl kustomize --enable-helm <dir> substitutes for standalone kustomize (kubectl embeds kustomize); git diff --no-index a b (or diff -u a b) substitutes for dyff.

cd argocd/app-configs/clickstack-chart
# baseline: native helm render from the SAME lifted values.yaml
helm template cs clickstack --repo https://clickhouse.github.io/ClickStack-helm-charts \
  --version 3.0.0 -n clickstack -f values.yaml > /tmp/native.yaml
# candidate: kustomize inflation + patch (or: kubectl kustomize --enable-helm . )
kustomize build --enable-helm . > /tmp/kustomize.yaml
dyff between /tmp/native.yaml /tmp/kustomize.yaml   # or: git diff --no-index /tmp/native.yaml /tmp/kustomize.yaml

Expected: the only delta is + spec.template.spec.dnsConfig on the cs-clickstack-app Deployment. If anything else differs (resource renames, dropped fields, Capabilities/api-version drift), STOP — add kubeVersion / apiVersions to the helmCharts entry to match the cluster (kubectl version), re-run, and do not proceed until the diff is clean. A name change here means releaseName/namespace is wrong.

• Step 5: Repoint source[0] of the Application to the kustomize directory

In argocd/cluster-app/templates/clickstack.yaml, replace the entire source[0] chart block (lines 13-215, from - chart: clickstack through the end of the valuesObject: block) with a git source. The sources: list becomes:

  sources:
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/clickstack-chart
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/clickstack

Leave source[1] (the existing app-configs/clickstack ExternalSecret path) and everything below sources: (destination, syncPolicy) unchanged. Preserve source order.

• Step 6: Remove the interim ignoreDifferences Deployment/dnsConfig entry

In the same file, delete only the Deployment dnsConfig ignoreDifferences entry and its preceding comment block (the # cs-clickstack-app dnsConfig: INTERIM stopgap ... / # HAZARD: ... comments plus:

    - kind: Deployment
      name: cs-clickstack-app
      namespace: clickstack
      jsonPointers:
        - /spec/template/spec/dnsConfig

). Keep the ExternalSecret entry, the clickstack-secret /data entry, and the entire syncPolicy.syncOptions block (ServerSideApply, CreateNamespace, RespectIgnoreDifferences).

• Step 7: Render the cluster-app chart to confirm the Application manifest is still valid

Run: helm template argocd/cluster-app | yq 'select(.metadata.name == "clickstack")' Expected: the clickstack Application now has two git sources (no chart:/valuesObject:), and ignoreDifferences lists exactly the ExternalSecret + clickstack-secret /data entries (no Deployment/dnsConfig entry).

• Step 8: Document the new pattern in argocd/CLAUDE.md

Add a short subsection (near the existing kustomize note ~line 75) describing the kustomize-helmCharts source convention: that argocd/app-configs/clickstack-chart inflates a remote Helm chart via kustomize (helmCharts: + valuesFile) to apply pod-spec fields the chart does not expose (dnsConfig), enabled cluster-wide by kustomize.buildOptions: --enable-helm in argocd-cm (set in tf/cluster-bootstrap/argocd.tf). Note the render-diff gate as the safety check when changing chart version or values.

• Step 9: Lint markdown + commit

Run: rumdl check argocd/CLAUDE.md docs/engineering/plans/2026-06-30-hyperdx-dnsconfig-gitops.md Expected: no issues.

Commit:

feat(clickstack): GitOps dnsConfig via kustomize-helmCharts; retire hl-qjh3.12 interim [hl-qjh3.14]

• Step 10: Open PR #2 (do NOT merge until Task 1 Step 5 gate has passed)

PR body must state the dependency: “Depends on PR #1 (--enable-helm) being merged AND terraform apply’d AND the argocd-cm gate confirmed live. Merging before that degrades the clickstack Application.” Include the clean dyff output from Step 4 as evidence.

---

Task 3: Post-merge verification + interim retirement confirmation

Files: none (live verification)

• Step 1: Confirm the Application synced cleanly

kubectl -n argocd get application clickstack -o jsonpath='{.status.sync.status}/{.status.health.status}'; echo

Expected: Synced/Healthy. If Unknown/Failed with a kustomize-build error mentioning helmCharts, the --enable-helm flag did not take effect on repo-server — re-run Task 1 Step 5.

• Step 2: Confirm dnsConfig is now Git-sourced (not the manual patch)

kubectl -n clickstack get deploy cs-clickstack-app -o jsonpath='{.spec.template.spec.dnsConfig}'; echo
kubectl -n clickstack get deploy cs-clickstack-app -o jsonpath='{.metadata.annotations.argocd\.argoproj\.io/tracking-id}'; echo

Expected: dnsConfig shows {"options":[{"name":"ndots","value":"1"}]}; the deployment is ArgoCD-tracked. Because the synced field is identical to the manual patch, there should be no new rollout (confirm kubectl -n clickstack rollout history deploy/cs-clickstack-app did not add a revision attributable to this change beyond the sync).

• Step 3: Confirm the ignoreDifferences blind spot is gone

kubectl -n argocd get application clickstack -o jsonpath='{range .spec.ignoreDifferences[*]}{.kind}{" "}{.name}{"\n"}{end}'

Expected: lists the ExternalSecret and Secret/clickstack-secret entries only — no Deployment/cs-clickstack-app.

• Step 4: Fire-test webhook delivery to Pushover (DNS intact)

Per memory 10c120e0 (live HyperDX alert fire-test): nudge a db.alerts threshold via mongosh (MONGO_URI from secret cs-clickstack-mongodb-hyperdx-hyperdx, exec in cs-clickstack-mongodb-0 -c mongod) so a current metric crosses, wait one ~5m checkAlerts cycle, then confirm in otel_logs an ALERT-TASK Triggering webhook alarm! with zero Failed to send / fetch failed, and a Pushover notification arrives. Revert the threshold.

Expected: webhook reaches Pushover (getaddrinfo(api.pushover.net) resolves) — proving the Git-sourced dnsConfig preserves the fix.

• Step 5: Update the bead

bd note hl-qjh3.14 with the verification evidence; the interim (manual patch + ignoreDifferences) is now fully retired. Leave in_progress for merge-time closure.

---

Verification summary

Gate	Command / check	Pass criterion
PR #1 live	kubectl -n argocd get cm argocd-cm -o jsonpath='{.data.kustomize\.buildOptions}'	--enable-helm
Render fidelity	dyff between /tmp/native.yaml /tmp/kustomize.yaml	only +dnsConfig on the Deployment
App health	kubectl -n argocd get application clickstack	Synced/Healthy
dnsConfig Git-sourced	kubectl -n clickstack get deploy cs-clickstack-app -o jsonpath='{...dnsConfig}'	ndots:1 present
Blind spot gone	...ignoreDifferences[*]	no Deployment entry
Webhook works	fire-test → otel_logs ALERT-TASK	reaches Pushover, no fetch-failed