Vault KV-v2 deprecation migration — implementation plan

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Eliminate every deprecated data "vault_kv_secret_v2" read that has a legal replacement, leaving only the four documented exception data sources, with zero behavior change to running workloads.

Architecture: Five mechanisms, chosen per value sensitivity (see the spec): ephemeral reads into provider configs; split TF-owned cluster/<app>/oidc Vault paths written with plain data_json; the non-deprecated vault_pki_secret_backend_issuer data source for public CA material; ephemeral reads into *_wo write-only sinks for genuine secrets; documented exceptions where no legal sink exists.

Tech Stack: Terraform ~>1.15 on HCP TFC (Agent + Local workspaces), hashicorp/vault ~>5.9, hashicorp/kubernetes >=3.2.1, hashicorp/tfe ~>0.78, keycloak/keycloak ~>5.8, cloudflare/cloudflare ~>5.19, ExternalSecrets Operator manifests under argocd/.

Spec: docs/engineering/specs/2026-07-03-vault-kv-deprecation-migration-design.md — read it first. Bead: hl-106s.

Ground rules for every task

This repo is jj-first (colocated jj+git). Never run mutating git commands. Every task is its own PR cut from fresh main:

jj git fetch
jj new main@origin
# ... edit files ...
jj commit -m "<conventional commit message>"
jj bookmark create <branch-name> -r @-
jj git push -b <branch-name> --allow-new
gh pr create --head <branch-name> --title "<title>" --body "<body>"

Tasks 1→7 are strictly ordered; do not start a task until the previous task’s PR has applied (TFC run green for Agent workspaces; operator terraform apply completed for Local ones — Tasks 4 and 5).
terraform validate needs TFC credentials/init; the authoritative validation gate is the PR’s TFC speculative plan. Locally run only terraform -chdir=<dir> fmt -check -recursive (expected: exit 0, no output).
Commit messages reference the bead: suffix [hl-106s].
Deprecation-warning acceptance test (run per workspace in the TFC plan UI): after Task 7, the only remaining “Deprecated” warnings repo-wide come from the four exception data sources, all in two workspaces: keycloak (tf/keycloak) and cloudflare_access, discord_webhook, hcp_terraform_hmac (all three in tf/cloudflare). tf/hcp-terraform’s own hmac read migrates to token_wo in Task 5 and must be warning-free afterward.

Task 1: Vault policy grants (tf/vault — Agent workspace, auto-applies on merge)

Files:

Modify: tf/vault/policy-keycloak.tf (per-app grant list inside the vault_policy.keycloak heredoc, and a new PKI issuer grant)
Modify: tf/vault/policy-terraform-workspaces.tf (cloudflare-admin heredoc ~lines 224-250; vault_policy.terraform_hcp_terraform_local heredoc ~lines 301-320)

Both edits land inside existing vault_policy heredocs — no new resources.

Step 1: Add /oidc subpath grants for the six split apps to terraform-keycloak-admin

In tf/vault/policy-keycloak.tf, inside the policy = <<-EOT heredoc, directly after each app’s existing bare-path pair (mealie, miniflux, karakeep, clickstack, engram-ui, cloudflare-access — argocd already has cluster/argocd/* grants at lines ~72-75, skip it), add the matching /oidc pair. Example for mealie (repeat the same shape for the other five apps, changing only the app name):

    path "secret/data/fzymgc-house/cluster/mealie/oidc" {
      capabilities = ["create", "read", "update"]
    }
    path "secret/metadata/fzymgc-house/cluster/mealie/oidc" {
      capabilities = ["create", "read", "update"]
    }

Rationale comment to add once above the new block group:

    # TF-owned OIDC subpaths (hl-106s split): tf/keycloak writes ONLY its own
    # fields here with plain data_json; the bare cluster/<app> paths above
    # revert to operator-only. The bare-path grants above have no trailing
    # wildcard, so they match only that exact path and do NOT cover these
    # subpaths (unlike cluster/argocd/*, whose glob already covers /oidc).

Step 2: Add PKI issuer read grant to terraform-keycloak-admin

Same heredoc, at the end before EOT:

    # Read the ICA1 issuer (certificate + ca_chain) from the live PKI mount —
    # replaces the deprecated KV read of infrastructure/pki/fzymgc-ica1-ca
    # for public CA material (hl-106s Plank C).
    path "fzymgc-house/v1/ica1/v1/issuer/*" {
      capabilities = ["read"]
    }

Step 3: Add cloudflare-access/oidc read grants to terraform-cloudflare-admin

In tf/vault/policy-terraform-workspaces.tf, next to the existing bare-path grants (path "secret/data/fzymgc-house/cluster/cloudflare-access" and its metadata twin), add:

# Keycloak-written OIDC creds moved to the TF-owned /oidc subpath (hl-106s);
# bare-path grant above does not cover the subpath.
path "secret/data/fzymgc-house/cluster/cloudflare-access/oidc" {
  capabilities = ["read"]
}
path "secret/metadata/fzymgc-house/cluster/cloudflare-access/oidc" {
  capabilities = ["read"]
}

Step 4: Add PKI issuer read grant to terraform-hcp-terraform-local

In the vault_policy.terraform_hcp_terraform_local heredoc, before EOT:

# Read the ICA1 issuer from the live PKI mount for TFC_VAULT_ENCODED_CACERT
# (hl-106s Plank C; replaces the deprecated KV read of fzymgc-ica1-ca).
path "fzymgc-house/v1/ica1/v1/issuer/*" {
  capabilities = ["read"]
}

Step 5: Format check

Run: terraform -chdir=tf/vault fmt -check -recursive Expected: exit 0, no output.

Step 6: Commit, push, PR

jj commit -m "feat(vault): grant oidc-subpath and ICA1 issuer reads for kv-deprecation migration [hl-106s]"
jj bookmark create feat/hl-106s-vault-grants -r @-
jj git push -b feat/hl-106s-vault-grants --allow-new
gh pr create --head feat/hl-106s-vault-grants --title "feat(vault): grants for vault_kv_secret_v2 deprecation migration [hl-106s]" --body "Step 1 of the hl-106s rollout (see docs/engineering/specs/2026-07-03-vault-kv-deprecation-migration-design.md). Adds: /oidc subpath CRUD for six split apps (argocd already covered by cluster/argocd/*), cloudflare-admin read on cloudflare-access/oidc, ICA1 PKI issuer reads for keycloak + hcp-terraform-local."

Gate: TFC speculative plan on the PR shows only vault_policy.keycloak, cloudflare-admin, and terraform_hcp_terraform_local in-place updates. Merge; confirm the main-cluster-vault apply is green before starting Task 2 (later plans 403 otherwise).

Task 2: tf/keycloak — split paths, ephemeral provider config, PKI CA read (Agent workspace)

Files:

Modify: tf/keycloak/vault.tf (add ephemeral + PKI issuer data source; delete fzymgc_ica1_ca KV data source; repoint oidc_discovery_ca_pem)
Modify: tf/keycloak/terraform.tf (provider keycloak credentials)
Modify: tf/keycloak/{mealie,argocd,miniflux,karakeep,clickstack,engram_ui,cloudflare_access}.tf (split-path rewrite ×7)
Modify: tf/keycloak/github_idp.tf, tf/keycloak/main.tf (exception comments only)

The biggest task: seven split-path rewrites plus the ephemeral/PKI conversions.

Step 1: Pre-flight byte-diff gate (Plank C) — operator shell with Vault admin token

vault read -format=json fzymgc-house/v1/ica1/v1/issuer/default \
  | jq -r '.data.ca_chain | join("\n")' > /tmp/pki_chain.pem
vault read -format=json secret/data/fzymgc-house/infrastructure/pki/fzymgc-ica1-ca \
  | jq -r '.data.data.fullchain' > /tmp/kv_fullchain.pem
diff /tmp/pki_chain.pem /tmp/kv_fullchain.pem && echo MATCH

Expected: MATCH (a trailing-newline-only diff is acceptable). Any other difference: STOP the Plank C portions (Steps 3, and Task 4/5 PKI repoints), keep the KV reads for CA material, and investigate — do not improvise. The split-path and ephemeral work below is unaffected either way.

Step 2: Ephemeral read for the keycloak provider credentials

In tf/keycloak/vault.tf, directly under the existing data "vault_kv_secret_v2" "keycloak" block (KEEP that data source — it still feeds the GitHub IdP and realm SMTP exception reads; ephemeral resources live in a different address namespace so the same name does not collide):

# Ephemeral twin of the data source above, used ONLY for provider credentials
# (ephemeral values are legal in provider config). The data source remains for
# the Plank E exception consumers (github_idp.tf, main.tf smtp block) until
# the keycloak provider ships write-only variants (hl-106s).
ephemeral "vault_kv_secret_v2" "keycloak" {
  mount = "secret"
  name  = "fzymgc-house/cluster/keycloak"
}

In tf/keycloak/terraform.tf, change the provider "keycloak" block credentials:

provider "keycloak" {
  url       = "https://id.fzymgc.house"
  client_id = "admin-cli"
  username  = ephemeral.vault_kv_secret_v2.keycloak.data["terraform_username"]
  password  = ephemeral.vault_kv_secret_v2.keycloak.data["terraform_password"]
}

Also update the stale comment above the provider block (it currently says the data source is defined in vault.tf — extend it to mention the ephemeral twin).

Step 3: Repoint oidc_discovery_ca_pem to the PKI issuer

In tf/keycloak/vault.tf, replace the whole data "vault_kv_secret_v2" "fzymgc_ica1_ca" block (and its comment) with:

# ICA1 chain read live from the PKI mount (not deprecated, no state secret) —
# replaces the deprecated KV read of infrastructure/pki/fzymgc-ica1-ca.
# Same chain content; verified byte-identical to the KV fullchain (hl-106s).
data "vault_pki_secret_backend_issuer" "ica1" {
  backend    = "fzymgc-house/v1/ica1/v1"
  issuer_ref = "default"
}

Change line ~110:

  oidc_discovery_ca_pem = join("\n", data.vault_pki_secret_backend_issuer.ica1.ca_chain)

Step 4: Split-path rewrite — mealie (pattern for all seven)

In tf/keycloak/mealie.tf: delete the data "vault_kv_secret_v2" "mealie_existing" block entirely, and replace the resource "vault_kv_secret_v2" "mealie" block with:

# TF-owned OIDC secret at the /oidc subpath (hl-106s split). Plain data_json —
# NOT deprecated, fully diffable. The bare cluster/mealie path is operator-only
# now; ExternalSecrets repoint per-property (separate argocd/ PR).
resource "vault_kv_secret_v2" "mealie_oidc" {
  mount = "secret"
  name  = "fzymgc-house/cluster/mealie/oidc"

  data_json = jsonencode({
    oidc_client_id      = keycloak_openid_client.mealie.client_id
    oidc_client_secret  = keycloak_openid_client.mealie.client_secret
    oidc_auth_enabled   = "true"
    oidc_signup_enabled = "true"
    oidc_user_group     = keycloak_group.mealie_users.name
    oidc_admin_group    = keycloak_group.mealie_admins.name
  })

  custom_metadata {
    max_versions = 5
    data = {
      managed_by  = "terraform"
      application = "mealie"
    }
  }
}

# Forget the old co-owned resource WITHOUT destroying the operator-seeded
# path (pattern from the Authentik teardown, PR #1476). Plan MUST show
# "will not be destroyed" — never a destroy.
removed {
  from = vault_kv_secret_v2.mealie

  lifecycle {
    destroy = false
  }
}

Step 5: Split-path rewrite — argocd (different field names, written out in full)

In tf/keycloak/argocd.tf: delete data "vault_kv_secret_v2" "argocd_existing", replace resource "vault_kv_secret_v2" "argocd" with:

# TF-owned OIDC secret (hl-106s split). NOTE the field names are
# keycloak_client_id/keycloak_client_secret (not oidc_*) — consumed by
# tf/cluster-bootstrap's Dex connector secret, not an ExternalSecret.
resource "vault_kv_secret_v2" "argocd_oidc" {
  mount = "secret"
  name  = "fzymgc-house/cluster/argocd/oidc"

  data_json = jsonencode({
    keycloak_client_id     = keycloak_openid_client.argocd.client_id
    keycloak_client_secret = keycloak_openid_client.argocd.client_secret
  })

  custom_metadata {
    max_versions = 5
    data = {
      managed_by  = "terraform"
      application = "argocd"
    }
  }
}

removed {
  from = vault_kv_secret_v2.argocd

  lifecycle {
    destroy = false
  }
}

Step 6: Split-path rewrite — remaining five apps

Apply the identical transformation as Step 4 (new resource label = <old>_oidc; new path = old path + /oidc; custom_metadata copied verbatim from the old resource; removed block targets the old label; delete the _existing data source):

File	Old resource / data source	New `data_json` fields (verbatim from the old merge block)
`miniflux.tf`	`vault_kv_secret_v2.miniflux` / `data...miniflux_existing`	`oidc_client_id`, `oidc_client_secret` from `keycloak_openid_client.miniflux`
`karakeep.tf`	`vault_kv_secret_v2.karakeep` / `data...karakeep_existing`	`oauth_wellknown_url = "https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration"`, `oauth_provider_name = "Keycloak"`, `oidc_client_id`, `oidc_client_secret` from `keycloak_openid_client.karakeep`
`clickstack.tf`	`vault_kv_secret_v2.clickstack` / `data...clickstack_existing`	`hyperdx_oidc_client_id`, `hyperdx_oidc_client_secret` from `keycloak_openid_client.clickstack`
`engram_ui.tf`	`vault_kv_secret_v2.engram_ui` / `data...engram_ui_existing`	`oidc_client_id`, `oidc_client_secret` from `keycloak_openid_client.engram_ui`
`cloudflare_access.tf`	`vault_kv_secret_v2.cloudflare_access` / `data...cloudflare_access_existing`	`oidc_client_id`, `oidc_client_secret` from `keycloak_openid_client.cloudflare_access`

Each file gets its own removed { from = vault_kv_secret_v2.<old-label> lifecycle { destroy = false } } block.

Step 7: Exception comments (Plank E)

In tf/keycloak/github_idp.tf, above the client_id/client_secret lines:

  # DEPRECATION EXCEPTION (hl-106s): reads the deprecated vault_kv_secret_v2
  # data source. The github-specific IdP resource has no client_secret_wo
  # (the generic keycloak_oidc_identity_provider does, but GitHub is
  # OAuth2-not-OIDC so it is not a substitute). Revisit when
  # keycloak/terraform-provider-keycloak#1545-adjacent work lands.

In tf/keycloak/main.tf, inside the smtp_server block comment (extend the existing hl-zih3 comment):

  # DEPRECATION EXCEPTION (hl-106s): `from` and the cf_email_token password
  # read the deprecated data source; keycloak_realm has no password_wo.

Step 8: Format check

Run: terraform -chdir=tf/keycloak fmt -check -recursive Expected: exit 0, no output.

Step 9: Commit, push, PR

jj commit -m "feat(keycloak): split TF-owned oidc paths, ephemeral provider creds, PKI CA read [hl-106s]"
jj bookmark create feat/hl-106s-keycloak-split -r @-
jj git push -b feat/hl-106s-keycloak-split --allow-new
gh pr create --head feat/hl-106s-keycloak-split --title "feat(keycloak): vault kv deprecation migration — split paths + ephemeral reads [hl-106s]" --body "Step 2 of hl-106s (spec: docs/engineering/specs/2026-07-03-vault-kv-deprecation-migration-design.md). 7 apps split to cluster/<app>/oidc with plain data_json; old co-owned resources leave state via removed+destroy=false (plan MUST show no destroys); provider creds via ephemeral read; oidc_discovery_ca_pem via PKI issuer; exception comments on github_idp + smtp."

Step 10: Speculative-plan gate (BLOCKING — review in the TFC UI before merge)

Verify ALL of:

7 × vault_kv_secret_v2.<app>_oidc create.
7 × old vault_kv_secret_v2.<app> listed as removed-from-state with “will not be destroyed” wording. Zero destroy actions anywhere.
vault_jwt_auth_backend update in-place only on oidc_discovery_ca_pem iff the byte-diff in Step 1 was newline-only; if the field shows a content diff beyond whitespace, STOP and re-run Step 1.
Deprecation warnings for this workspace reduced to only data.vault_kv_secret_v2.keycloak usages.

Merge; confirm the main-cluster-keycloak apply is green. Post-apply spot-check (operator):

vault read -format=json secret/data/fzymgc-house/cluster/mealie/oidc | jq '.data.data | keys'

Expected: the six mealie oidc keys.

Task 3: ExternalSecret repoints (argocd/ — applied by ArgoCD sync)

Files:

Modify: argocd/app-configs/mealie/secrets.yaml
Modify: argocd/app-configs/miniflux/secrets.yaml
Modify: argocd/app-configs/karakeep/secrets.yaml
Modify: argocd/app-configs/clickstack/external-secret.yaml
Modify: argocd/app-configs/agent-memory/external-secret-engram-ui.yaml

Covers five of the seven split apps. argocd has no remoteRefs (Task 4 handles it); cloudflare-access is consumed by Terraform (Task 6).

Step 1: Repoint only the TF-written properties

For each entry whose property: is in the list below, change its key: fzymgc-house/cluster/<app> to key: fzymgc-house/cluster/<app>/oidc. Leave every other entry untouched (those fields are operator-seeded and stay on the bare path).

File	Properties to repoint
`mealie/secrets.yaml`	`oidc_auth_enabled`, `oidc_client_id`, `oidc_client_secret`, `oidc_signup_enabled`, `oidc_user_group`, `oidc_admin_group` (6 of 9 entries; `base_url`, `allow_signup`, `openai_api_key` stay)
`miniflux/secrets.yaml`	`oidc_client_id`, `oidc_client_secret` (2 of 4; `admin_username`, `admin_password` stay)
`karakeep/secrets.yaml`	`oauth_wellknown_url`, `oauth_provider_name`, `oidc_client_id`, `oidc_client_secret` (4 of 9)
`clickstack/external-secret.yaml`	`hyperdx_oidc_client_secret` only (1 of 8 — the client_id is not ES-consumed)
`agent-memory/external-secret-engram-ui.yaml`	`oidc_client_secret` only (1 of 2; `ui_cookie_key` stays)

Step 2: Lint

Run: yamllint argocd/app-configs/mealie/secrets.yaml argocd/app-configs/miniflux/secrets.yaml argocd/app-configs/karakeep/secrets.yaml argocd/app-configs/clickstack/external-secret.yaml argocd/app-configs/agent-memory/external-secret-engram-ui.yaml Expected: exit 0.

Step 3: Commit, push, PR

jj commit -m "fix(secrets): repoint oidc remoteRefs to TF-owned /oidc subpaths [hl-106s]"
jj bookmark create feat/hl-106s-es-repoint -r @-
jj git push -b feat/hl-106s-es-repoint --allow-new
gh pr create --head feat/hl-106s-es-repoint --title "fix(secrets): repoint oidc remoteRefs to cluster/<app>/oidc [hl-106s]" --body "Step 3 of hl-106s. Repoints ONLY the TF-written properties for 5 apps; operator-seeded fields stay on the bare paths. Requires Task-2 apply (paths exist). Secret content is identical, so no pod restarts expected."

Step 4: Post-sync verification (read-only)

After ArgoCD syncs, check every repointed ExternalSecret reports SecretSynced:

Run: kubectl get externalsecret -A -o wide | rg -i 'mealie|miniflux|karakeep|clickstack|engram' Expected: Ready=True, reason SecretSynced on each. Any SecretSyncedError mentioning a 403 or missing key → verify Task 1 applied and the Task-2 spot-check keys exist.

Task 4: tf/cluster-bootstrap — PKI CA reads + write-only k8s secrets (Local workspace — MANUAL APPLY)

Files:

Create: tf/cluster-bootstrap/pki.tf
Modify: tf/cluster-bootstrap/external-secrets.tf (~lines 24-50)
Modify: tf/cluster-bootstrap/argocd.tf (data source ~line 10; secrets ~lines 14-80; Dex rootCAs ~line 196)
Modify: tf/cluster-bootstrap/cert-manager.tf (~lines 10-14, 68-82)
Modify: tf/cluster-bootstrap/longhorn.tf (~lines 10-20, 55-86)
Modify: tf/cluster-bootstrap/versions.tf (kubernetes pin)

Local-execution workspace: merge does not apply; Step 9 is the operator apply.

Step 1: Shared PKI issuer data source

Create tf/cluster-bootstrap/pki.tf:

# ICA1 issuer read live from the PKI mount (hl-106s Plank C). Shared by the
# ESO ca.crt secret (external-secrets.tf) and the Dex rootCAs helm value
# (argocd.tf). Public material — a regular, diffable data source is correct
# here; it is NOT the deprecated KV data source.
data "vault_pki_secret_backend_issuer" "ica1" {
  backend    = "fzymgc-house/v1/ica1/v1"
  issuer_ref = "default"
}

Step 2: Repoint the two public-CA consumers

external-secrets.tf: delete the data "vault_kv_secret_v2" "vault_ca_chain" block; in kubernetes_secret_v1.vault_ca_chain change:

  data = {
    "ca.crt" = join("\n", data.vault_pki_secret_backend_issuer.ica1.ca_chain)
  }

argocd.tf line ~196 (Dex rootCAs, inside the helm values heredoc):

                rootCAs:
                  - ${base64encode(join("\n", data.vault_pki_secret_backend_issuer.ica1.ca_chain))}

(Keep the existing explanatory comment block about Dex APPEND semantics — only the interpolation changes.)

Step 3: cert-manager ICA1 pair → ephemeral + data_wo

cert-manager.tf: replace data "vault_kv_secret_v2" "fzymgc_ica1_ca" with:

# Ephemeral read: tls.key is the ICA1 PRIVATE KEY — must not persist in state.
# Feeds only the write-only data_wo below (hl-106s Plank D).
ephemeral "vault_kv_secret_v2" "fzymgc_ica1_ca" {
  mount = "secret"
  name  = "fzymgc-house/infrastructure/pki/fzymgc-ica1-ca"
}

In kubernetes_secret_v1.fzymgc_house_ica1_key replace the data attribute with:

  # Write-only: rotate by bumping data_wo_revision (see docs/operations/vault.md).
  data_wo = {
    "tls.crt" = ephemeral.vault_kv_secret_v2.fzymgc_ica1_ca.data["cert"]
    "tls.key" = ephemeral.vault_kv_secret_v2.fzymgc_ica1_ca.data["cleartext_key"]
  }
  data_wo_revision = 1

Step 4: longhorn secrets → ephemeral + data_wo

longhorn.tf: replace the two data sources:

ephemeral "vault_kv_secret_v2" "longhorn_crypto_key" {
  mount = "secret"
  name  = "fzymgc-house/cluster/longhorn/crypto-key"
}

ephemeral "vault_kv_secret_v2" "longhorn_cloudflare_r2" {
  mount = "secret"
  name  = "fzymgc-house/cluster/longhorn/cloudflare-r2"
}

Paths verified against longhorn.tf:9-17 — identical to the deleted data sources’ name values.

In kubernetes_secret_v1.longhorn_crypto_config and kubernetes_secret_v1.longhorn_backup_creds, rename data → data_wo (all fields move wholesale, constants included), swap data.vault_kv_secret_v2.X.data[...] → ephemeral.vault_kv_secret_v2.X.data[...], and add data_wo_revision = 1 to each.

Step 5: argocd_config split → two ephemerals + data_wo

argocd.tf: replace data "vault_kv_secret_v2" "argocd_config" (~line 10) with:

# Split reads (hl-106s): github_app_* stays operator-seeded on the bare path;
# keycloak creds are TF-written by tf/keycloak to the /oidc subpath.
ephemeral "vault_kv_secret_v2" "argocd_base" {
  mount = "secret"
  name  = "fzymgc-house/cluster/argocd"
}

ephemeral "vault_kv_secret_v2" "argocd_oidc" {
  mount = "secret"
  name  = "fzymgc-house/cluster/argocd/oidc"
}

Convert the three consuming secrets (argocd_oidc_secret, argocd_selfhosted_repo, argocd_temporal_workers_repo): data → data_wo + data_wo_revision = 1, with references:

  # argocd_oidc_secret
  data_wo = {
    "keycloak.client_id"     = ephemeral.vault_kv_secret_v2.argocd_oidc.data["keycloak_client_id"]
    "keycloak.client_secret" = ephemeral.vault_kv_secret_v2.argocd_oidc.data["keycloak_client_secret"]
  }
  data_wo_revision = 1

  # both repo secrets (url/type constants move wholesale)
  data_wo = {
    type                    = "git"
    url                     = "https://github.com/fzymgc-house/selfhosted-cluster" # temporal-workers URL in the second secret
    githubAppID             = ephemeral.vault_kv_secret_v2.argocd_base.data["github_app_app_id"]
    githubAppInstallationID = ephemeral.vault_kv_secret_v2.argocd_base.data["github_app_installation_id"]
    githubAppPrivateKey     = base64decode(ephemeral.vault_kv_secret_v2.argocd_base.data["github_app_private_key"])
  }
  data_wo_revision = 1

The Dex connector values (clientID: $argocd-oidc-secret:keycloak.client_id at ~line 184) reference the k8s secret by name and need no change.

Step 6: kubernetes provider pin floor

versions.tf: change the kubernetes entry to:

    kubernetes = {
      source = "hashicorp/kubernetes"
      # >= 3.2.1: fixes the missing-resource-identity-after-create error for
      # secrets using write-only data_wo (hl-106s).
      version = ">= 3.2.1, < 4.0.0"
    }

Step 7: Format check

Run: terraform -chdir=tf/cluster-bootstrap fmt -check -recursive Expected: exit 0, no output.

Step 8: Commit, push, PR

jj commit -m "feat(cluster-bootstrap): PKI issuer CA reads + write-only k8s secret data [hl-106s]"
jj bookmark create feat/hl-106s-cluster-bootstrap -r @-
jj git push -b feat/hl-106s-cluster-bootstrap --allow-new
gh pr create --head feat/hl-106s-cluster-bootstrap --title "feat(cluster-bootstrap): vault kv deprecation migration [hl-106s]" --body "Step 4 of hl-106s. LOCAL-EXECUTION workspace: merging does NOT apply — operator must terraform apply manually (docs/operations/cluster-bootstrap.md). Closes the argocd frozen-creds window opened by step 2, so apply promptly after merge."

Step 9: MANUAL LOCAL APPLY (operator) + verification

Record the pre-migration secret hashes FIRST:

for s in "cert-manager fzymgc-house-ica1-key" "longhorn-system longhorn-crypto-config" "longhorn-system longhorn-backup-creds" "argocd argocd-oidc-secret" "argocd argocd-selfhosted-repo" "argocd argocd-temporal-workers-repo" "external-secrets vault-ca-chain"; do set -- $s; kubectl get secret -n $1 $2 -o json | jq -S '.data' | sha256sum; done > /tmp/pre_hashes.txt

Then, per docs/operations/cluster-bootstrap.md:

terraform -chdir=tf/cluster-bootstrap plan -out=tfplan

Plan gate: expect in-place updates on the seven secrets (data → data_wo shape change), the ESO ca.crt and Dex rootCAs values unchanged-or-whitespace-only, and no destroys. Then:

terraform -chdir=tf/cluster-bootstrap apply tfplan

Re-run the hash loop into /tmp/post_hashes.txt and diff /tmp/pre_hashes.txt /tmp/post_hashes.txt. Expected: identical. If any secret came back empty or changed: bump that resource’s data_wo_revision to 2, re-plan/apply (re-sends the write-only payload), and re-verify.

Task 5: tf/hcp-terraform — PKI CACERT + write-only notifications (Local workspace — MANUAL APPLY)

Files:

Modify: tf/hcp-terraform/data.tf (~lines 8-14)
Modify: tf/hcp-terraform/notifications.tf (~lines 11-36)
Modify: tf/hcp-terraform/workspace-variables.tf (~line 98)

Local-execution workspace: merge does not apply; Step 4 covers the operator apply.

Step 1: CACERT via PKI issuer

data.tf: replace data "vault_kv_secret_v2" "ca_certificate" with:

# ICA1 chain read live from the PKI mount (hl-106s Plank C) — replaces the
# deprecated KV read. Public material; regular data source keeps diffability.
data "vault_pki_secret_backend_issuer" "ica1" {
  backend    = "fzymgc-house/v1/ica1/v1"
  issuer_ref = "default"
}

workspace-variables.tf (tfe_variable.vault_encoded_cacert):

  value = base64encode(join("\n", data.vault_pki_secret_backend_issuer.ica1.ca_chain))

Step 2: Notification url/token → ephemeral + write-only auto-hash mode

notifications.tf: replace the two data sources with:

ephemeral "vault_kv_secret_v2" "hcp_terraform_worker" {
  mount = "secret"
  name  = "fzymgc-house/infrastructure/cloudflare/hcp-terraform-worker"
}

ephemeral "vault_kv_secret_v2" "hcp_terraform_hmac" {
  mount = "secret"
  name  = "fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac"
}

In tfe_notification_configuration.discord, replace url and token:

  # Write-only with AUTO-MANAGED updates: *_wo_version deliberately omitted —
  # the provider stores a sha256 of the value in private state and re-sends
  # on change (tfe >= 0.76.2). Do NOT add the _wo_version attributes.
  url_wo   = ephemeral.vault_kv_secret_v2.hcp_terraform_worker.data["url"]
  token_wo = ephemeral.vault_kv_secret_v2.hcp_terraform_hmac.data["token"]

Step 3: Format check

Run: terraform -chdir=tf/hcp-terraform fmt -check -recursive Expected: exit 0, no output.

Step 4: Commit, push, PR, manual apply

jj commit -m "feat(hcp-terraform): PKI CACERT + write-only notification url/token [hl-106s]"
jj bookmark create feat/hl-106s-hcp-terraform -r @-
jj git push -b feat/hl-106s-hcp-terraform --allow-new
gh pr create --head feat/hl-106s-hcp-terraform --title "feat(hcp-terraform): vault kv deprecation migration [hl-106s]" --body "Step 5 of hl-106s. LOCAL-EXECUTION workspace — operator applies manually per docs/operations/hcp-terraform.md."

After merge (operator, per docs/operations/hcp-terraform.md): terraform -chdir=tf/hcp-terraform plan -out=tfplan — expect in-place updates on every tfe_notification_configuration.discord[*] (url/token → write-only) and on tfe_variable.vault_encoded_cacert[*] only if the chain differs by whitespace; no destroys. Then terraform -chdir=tf/hcp-terraform apply tfplan.

Verification: trigger any TFC run (Task 6’s PR works) and confirm the Discord notification still arrives — proves url_wo/token_wo landed intact.

Task 6: tf/cloudflare — ephemeral provider token + exception annotations (Agent workspace)

Files:

Modify: tf/cloudflare/terraform.tf (~lines 4-30)
Modify: tf/cloudflare/access.tf (~lines 31-58)
Modify: tf/cloudflare/workers.tf (~lines 12-24, 51-62)

One clean conversion plus the two documented exception annotations.

Step 1: Provider token via ephemeral read

terraform.tf: replace data "vault_kv_secret_v2" "cloudflare_bootstrap_token" with an ephemeral block (keep the existing two-token-pattern comment):

ephemeral "vault_kv_secret_v2" "cloudflare_bootstrap_token" {
  mount = "secret"
  name  = "fzymgc-house/infrastructure/cloudflare/bootstrap-token"
}

provider "cloudflare" {
  api_token = ephemeral.vault_kv_secret_v2.cloudflare_bootstrap_token.data["token"]
}

Before committing, confirm nothing else references the old data source: Run: rg -n 'data\.vault_kv_secret_v2\.cloudflare_bootstrap_token' tf/cloudflare/ Expected: no matches outside comments (r2.tf mentions it in a comment only — update that comment to say ephemeral.vault_kv_secret_v2...).

Step 2: Repoint the access read + exception comments

access.tf: change the data source (fix the stale “Authentik” comment while there):

# Keycloak OIDC credentials for Cloudflare Access, TF-written by
# tf/keycloak/cloudflare_access.tf to the TF-owned /oidc subpath (hl-106s).
# DEPRECATION EXCEPTION: stays on the deprecated data source — the cloudflare
# provider has no write-only attrs (v5.6.0 shipped one on this very field and
# reverted it; see provider issues #5702/#5748, FR #6964).
data "vault_kv_secret_v2" "cloudflare_access" {
  mount = "secret"
  name  = "fzymgc-house/cluster/cloudflare-access/oidc"
}

workers.tf: above the two data sources, add:

# DEPRECATION EXCEPTION (hl-106s): worker secret_text bindings have no
# write-only variant in cloudflare v5 (the v4 cloudflare_worker_secret
# resource was dropped). These two reads stay on the deprecated data source.

Step 3: Format check, commit, PR

Run: terraform -chdir=tf/cloudflare fmt -check -recursive — expected: exit 0.

jj commit -m "feat(cloudflare): ephemeral provider token, repoint access read to /oidc [hl-106s]"
jj bookmark create feat/hl-106s-cloudflare -r @-
jj git push -b feat/hl-106s-cloudflare --allow-new
gh pr create --head feat/hl-106s-cloudflare --title "feat(cloudflare): vault kv deprecation migration [hl-106s]" --body "Step 6 of hl-106s. Provider token via ephemeral read; access IdP read repointed to cluster/cloudflare-access/oidc (needs Task-1 grant + Task-2 write); worker binding reads annotated as documented exceptions."

Gate: speculative plan shows cloudflare_zero_trust_access_identity_provider.keycloak with no changes (same values, new path) or an in-place no-op; remaining warnings only from the three exception data sources.

Task 7: tf/uptime-kuma — provider pin bump + ephemeral provider creds (Agent workspace)

Files:

Modify: tf/uptime-kuma/versions.tf (vault pin)
Modify: tf/uptime-kuma/vault.tf (data → ephemeral)
Modify: tf/uptime-kuma/main.tf (provider block refs)

Smallest task: pin bump plus a provider-config conversion.

Step 1: Bump the vault provider floor

versions.tf:

    vault = {
      source = "hashicorp/vault"
      # >= 5.9 for the ephemeral vault_kv_secret_v2 resource (hl-106s);
      # matches the pin every other workspace already uses.
      version = "~> 5.9"
    }

Step 2: Ephemeral provider creds

vault.tf: replace the data "vault_kv_secret_v2" "uptime_kuma" block (keep the header comment describing the secret structure):

ephemeral "vault_kv_secret_v2" "uptime_kuma" {
  mount = "secret"
  name  = "fzymgc-house/cluster/uptime-kuma"
}

main.tf provider block:

provider "uptimekuma" {
  endpoint = var.uptime_kuma_url
  username = ephemeral.vault_kv_secret_v2.uptime_kuma.data["username"]
  password = ephemeral.vault_kv_secret_v2.uptime_kuma.data["password"]
  # (keep the existing per_attempt_timeout/max_retries/timeout block verbatim)
}

Step 3: Format check, commit, PR

Run: terraform -chdir=tf/uptime-kuma fmt -check -recursive — expected: exit 0.

jj commit -m "feat(uptime-kuma): vault provider 5.9 + ephemeral provider creds [hl-106s]"
jj bookmark create feat/hl-106s-uptime-kuma -r @-
jj git push -b feat/hl-106s-uptime-kuma --allow-new
gh pr create --head feat/hl-106s-uptime-kuma --title "feat(uptime-kuma): vault kv deprecation migration [hl-106s]" --body "Step 7 of hl-106s. Vault pin ~>5.6 → ~>5.9; provider creds via ephemeral read; zero deprecation warnings remain in this workspace."

Gate: speculative plan shows provider upgrade + zero resource changes and zero deprecation warnings.

Task 8: Stale-field cleanup + docs + agent guidance (manual ops + docs PR)

Files:

Modify: docs/reference/secrets.md (path-layout section)
Modify: docs/operations/vault.md (new runbook section)
Modify: tf/CLAUDE.md (Security Constraints — replace the stale data-source rule)
Modify: argocd/CLAUDE.md (ExternalSecret Pattern — path-split convention)

Run the cleanup only after Tasks 3, 4, and 6 are applied and verified. The CLAUDE.md updates describe the migrated end state, which is why they land here and not earlier — publishing “never add KV data-source reads” while earlier tasks are still in flight would misdirect agents mid-migration.

Step 1: Remove stale oidc fields from the operator paths (operator shell, Vault admin token)

For each app, read → strip the TF-written fields (per-app lists = exactly the fields in Task 2 Step 4/5) → write back. KV versioning (max_versions=5) gives rollback. Example for mealie:

P=secret/data/fzymgc-house/cluster/mealie
vault read -format=json $P | jq '.data.data' > /tmp/mealie.json
jq 'del(.oidc_client_id, .oidc_client_secret, .oidc_auth_enabled, .oidc_signup_enabled, .oidc_user_group, .oidc_admin_group)' /tmp/mealie.json > /tmp/mealie_clean.json
vault write $P data=@/tmp/mealie_clean.json
vault read -format=json $P | jq '.data.data | keys'   # expect: only operator fields
rm /tmp/mealie.json /tmp/mealie_clean.json

Repeat for: argocd (keycloak_client_id, keycloak_client_secret), miniflux, karakeep (+oauth_wellknown_url, oauth_provider_name), clickstack (hyperdx_oidc_*), engram-ui, cloudflare-access (oidc_*). Do this ONLY after Tasks 3, 4, and 6 are applied and verified — the stale copies are the rollback path until then.

Step 2: Update docs/reference/secrets.md

In the cluster-secrets section, add:

### TF-owned OIDC subpaths (hl-106s)

Keycloak-provisioned OIDC credentials live at `secret/fzymgc-house/cluster/<app>/oidc`
(written by `tf/keycloak` with plain `data_json` — fully diffable). The bare
`cluster/<app>` paths are **operator-only**: Terraform never writes them, so
`vault kv put`-style whole-version replaces by operators can no longer clobber
TF-written fields (and vice versa). Apps: mealie, argocd, miniflux, karakeep,
clickstack, engram-ui, cloudflare-access.

Step 3: Add the write-only rotation runbook to docs/operations/vault.md

## Rotating secrets consumed via write-only (`*_wo`) attributes

Some Terraform sinks read Vault secrets **ephemerally** and write them through
write-only attributes (`kubernetes_secret_v1.data_wo`,
`tfe_notification_configuration.token_wo`, …). Terraform cannot diff
write-only content, so **rotating the Vault secret does not propagate by
itself** (exception: the tfe notification url/token use the provider's
auto-hash mode and re-send automatically).

After rotating such a secret in Vault:

1. Find the consuming resource (grep the workspace for `data_wo` /
   `ephemeral "vault_kv_secret_v2"`).
2. Increment its `data_wo_revision` by exactly 1 (helm's `set_wo_revision`
   must strictly increase; kubernetes accepts any change).
3. Plan/apply the workspace (`main-cluster-bootstrap` is Local execution —
   apply manually).
4. Verify the in-cluster value changed (`kubectl get secret ... -o json | jq -S '.data' | sha256sum`).

Step 4: Update tf/CLAUDE.md — replace the stale data-source rule

In the ## Security Constraints section, replace the line - **SHOULD** use Vault data sources instead of variable inputs where possible with:

- **SHOULD** source secret values from Vault, but **MUST NOT** add new
  `data "vault_kv_secret_v2"` reads — the data source is deprecated
  (hl-106s; see ADR hl-ujwi). Route by sensitivity:
  - Provider credentials → `ephemeral "vault_kv_secret_v2"` referenced in the
    provider block
  - Secrets flowing into resources → ephemeral read + write-only argument
    (`data_wo`/`value_wo`/`token_wo` + its revision counter; rotation runbook:
    `../docs/operations/vault.md`). If the downstream provider has no `_wo`
    variant, keep the data source and add a
    `DEPRECATION EXCEPTION (hl-106s)` comment naming the upstream gap
  - Public CA/chain material → `data "vault_pki_secret_backend_issuer"` on
    the ICA1 mount (`fzymgc-house/v1/ica1/v1`) — public certs are not secrets
  - Terraform-generated app credentials → write to a TF-owned
    `cluster/<app>/oidc` subpath with plain `data_json` (ADR hl-u103);
    **MUST NOT** co-write operator-seeded bare `cluster/<app>` paths
- **MUST** use `removed { lifecycle { destroy = false } }` when dropping a
  `vault_kv_secret_v2` resource from a co-owned path — the plan must show
  *forget*, never *destroy*
- New `cluster/<app>/...` subpaths need paired `secret/data/` +
  `secret/metadata/` grants in `tf/vault` — a bare-path grant has no wildcard
  and does not cover subpaths

Step 5: Update argocd/CLAUDE.md — ExternalSecret path-split convention

In the ## ExternalSecret Pattern section, after the example manifest, add:

**Vault path split (hl-106s, ADR hl-u103):** Keycloak-provisioned OIDC fields
live at `fzymgc-house/cluster/<app>/oidc` (TF-owned); operator-seeded fields
live at the bare `fzymgc-house/cluster/<app>` path. Point each `remoteRef.key`
at the path that owns that field — never move operator-seeded properties to
`/oidc`, and never point TF-written `oidc_*` properties at the bare path.

Step 6: Lint, commit, PR

Run: rumdl check docs/reference/secrets.md docs/operations/vault.md tf/CLAUDE.md argocd/CLAUDE.md — expected: no issues.

jj commit -m "docs(vault): oidc subpath layout, rotation runbook, agent guidance [hl-106s]"
jj bookmark create docs/hl-106s-runbooks -r @-
jj git push -b docs/hl-106s-runbooks --allow-new
gh pr create --head docs/hl-106s-runbooks --title "docs: vault kv migration layout, runbook + CLAUDE.md guidance [hl-106s]" --body "Step 8 of hl-106s. Human docs (secrets layout, write-only rotation runbook) plus agent guidance: tf/CLAUDE.md loses the stale 'use Vault data sources' rule in favor of the per-sensitivity routing table; argocd/CLAUDE.md documents the bare-path vs /oidc remoteRef convention."

Task 9: Upstream issue drafts (review-gated — do NOT file without operator approval)

Files:

Create: /tmp drafts only (nothing committed; issue text must contain no internal hostnames, paths, or secret names)
Step 1: Draft the keycloak provider issue

Write to /tmp/keycloak-issue.md:

Title: Write-only variants for realm SMTP password and social IdP client secrets

The generic keycloak_oidc_identity_provider and keycloak_openid_client gained
client_secret_wo / client_secret_wo_version, which works great with Terraform
1.11+ ephemeral values. Two adjacent gaps still force secrets into state:

- keycloak_realm's smtp_server.auth.password has no password_wo, so realm SMTP
  credentials can't be sourced from an ephemeral secret store read.
- The provider-specific social IdP resources (keycloak_oidc_github_identity_provider,
  ..._google_identity_provider) only have plain client_secret. The generic OIDC
  resource isn't a substitute for these since the social providers aren't
  standard OIDC.

Would you accept PRs extending the existing client_secret_wo pattern to these
resources? Related: #1545.

Step 2: Draft the cloudflare provider issue

Write to /tmp/cloudflare-issue.md:

Title: Write-only attribute support for secret-bearing fields (Access IdP client_secret, Workers secret_text bindings)

Terraform 1.11+ write-only arguments let ephemeral values (e.g. Vault reads)
flow into resources without persisting secrets in state. The provider currently
has no write-only attributes: zero_trust_access_identity_provider
config.client_secret and workers script/version secret_text binding text are
plain Sensitive attributes.

I understand 5.6.0 briefly made config.client_secret write-only and it was
reverted (#5702, #5748) due to state-upgrade and OpenTofu issues — is a
_wo-suffixed companion attribute (the pattern hashicorp providers use, avoiding
the in-place breaking change) something you'd consider? Related: #6964.

Step 3: Operator review gate

Present both drafts to the operator verbatim. Only after explicit approval, file with gh issue create --repo keycloak/terraform-provider-keycloak ... / --repo cloudflare/terraform-provider-cloudflare ..., then bd note hl-106s "upstream issues: <urls>". If the operator edits, file the edited text.

Completion checklist

All seven TFC/local applies green; diff pre/post secret hashes identical (Task 4).
Repo-wide plan warnings: only the four exception data sources (keycloak, cloudflare_access, discord_webhook, hcp_terraform_hmac).
ExternalSecrets all SecretSynced (Task 3); ArgoCD SSO login works (Dex creds via /oidc); Discord TFC notifications still arrive (Task 5).
Stale fields removed from operator paths (Task 8); docs + CLAUDE.md agent guidance merged.
Upstream drafts reviewed by operator; filed or explicitly declined (Task 9).
bd close the per-task beads; final bd note hl-106s summarizing outcomes.