Vault KV-v2 deprecation migration: ephemeral reads, write-only sinks, PKI-served CA

• Date: 2026-07-03
• Bead: hl-106s
• Status: Approved (revised Approach 1)

Problem

The Vault provider (pinned ~> 5.9, everywhere except uptime-kuma at ~> 5.6) deprecates the KV-v2 read paths: the data "vault_kv_secret_v2" data source wholesale, and the .data readback attribute on the vault_kv_secret_v2 resource. Every plan now emits Deprecated Resource / Deprecated value used warnings. The replacement is the ephemeral resource ephemeral "vault_kv_secret_v2", whose values are legal only in provider configuration, write-only (*_wo) arguments, and other ephemeral contexts.

The write-side data_json is not deprecated. Warnings on data_json = jsonencode(merge(...)) expressions come from the deprecated .data reads nested inside them.

Footprint (recounted after rebasing onto main post-#1554/#1555): 23 data-source definitions (22 in scope — one sits in the decommissioning tf/authentik), ~45 reference sites, across tf/{keycloak,cloudflare,cluster-bootstrap,hcp-terraform,authentik,uptime-kuma}. PR #1554 (Cloudflare Email Sending) already deleted the former tf/keycloak/smtp.tf data source and hardcoded the non-secret SMTP fields.

Goal (operator decision): zero warnings, restructuring where needed, with narrowly documented exceptions only where the provider ecosystem offers no legal sink.

Verified provider facts

All verified against tagged provider source, not registry-rendered docs (registry renders from main — the docs-from-main trap).

Provider	Tag	Write-only support relevant to us
hashicorp/vault	v5.10.1	ephemeral vault_kv_secret_v2 (mount, mount_id, name, version → exports data plus data_json and timestamp/metadata attributes, but no current-version number — the load-bearing gap for Plank D revisions); resource data_json_wo + data_json_wo_version; data "vault_pki_secret_backend_issuer" NOT deprecated (exports certificate, ca_chain); data "vault_generic_secret" NOT deprecated (noted, not used)
hashicorp/kubernetes	v3.2.1	kubernetes_secret_v1: data_wo/binary_data_wo + *_wo_revision (since 2.36.0; 3.2.1 fixes a create-identity bug — floor the pin at 3.2.1)
hashicorp/helm	v3.2.0	helm_release: set_wo + set_wo_revision (name/value/type entries only; raw-YAML values is not write-only)
hashicorp/tfe	v0.78.0	tfe_variable: value_wo + mandatory value_wo_version (manual bump; pattern solid since v0.75.0); tfe_notification_configuration: token_wo/url_wo with auto-managed hash mode when *_wo_version is omitted — provider stores a SHA-256 in private state and auto-bumps (verified in tagged source internal/provider/resource_tfe_notification_configuration.go @ v0.78.0, computeWOHash; feature landed v0.76.2, PR #2028)
keycloak/keycloak	v5.8.0	client_secret_wo on keycloak_openid_client and the generic keycloak_oidc_identity_provider only (verified in tagged source: provider/resource_keycloak_oidc_identity_provider.go:58-72 and provider/resource_keycloak_openid_client.go:75-88 @ v5.8.0, both WriteOnly: true — note deepwiki’s index of this repo lags and misses these; verify via the tagged files). No password_wo on realm smtp_server; no _wo on keycloak_oidc_github_identity_provider (and the generic resource is not a substitute — GitHub is OAuth2, not OIDC). Upstream issue keycloak/terraform-provider-keycloak#1545 tracks extending write-only support
cloudflare/cloudflare	v5.21.1	None. Tagged source shows zero framework write-only attributes; the dedicated worker-secret resource (v4’s cloudflare_worker_secret) is not usable at this tag — dropped in the v5 move to secret_text bindings, only test scaffolding remains in-tree. Upstream issues #5702/#5748 report v5.6.0 briefly shipping write-only on the Access IdP client_secret and reverting it (single-sourced from those issues; consistent with the v5.21.1 schema). Open FR #6964 (api_token)

Per-row evidence paths (tag-qualified file/line references) are recorded as grounding/* notes on bead hl-106s. The ephemeral resource’s version argument is listed in the v5.10.1 tagged doc (website/docs/ephemeral-resources/kv_secret_v2.html.md); no design site pins a version.

Design

Values are sorted by their true sensitivity; each class gets a different mechanism. Ephemeral resources do not replace data sources 1:1.

Plank A — provider configuration → ephemeral reads

Provider config accepts ephemeral values today. Convert the reads feeding:

• tf/keycloak/terraform.tf — keycloak provider username/password
• tf/cloudflare/terraform.tf — cloudflare provider api_token
• tf/uptime-kuma/main.tf — uptimekuma provider username/password (bump the vault provider pin ~> 5.6 → ~> 5.9 in the same PR)

mount_id is not needed (mount is the static secret engine; plan-time reads are fine, matching current data-source behavior).

Shared-source rule: converting a provider-config consumer does not remove the underlying data source while other consumers remain. Concretely, data.vault_kv_secret_v2.keycloak (tf/keycloak/vault.tf) feeds the provider config (this plank) and the GitHub IdP secret and the realm SMTP block’s from/cf_email_token fields (all Plank E). Add a new ephemeral "vault_kv_secret_v2" "keycloak" for the provider config and retain the same-named data source for the Plank E consumers — data sources and ephemeral resources live in different address namespaces, so the shared name does not collide. The data source is deleted only when its last consumer migrates (i.e., when upstream write-only support lands).

Plank B — read-modify-write merges → split TF-owned paths

Seven tf/keycloak sites (mealie, argocd, miniflux, karakeep, clickstack, engram_ui, cloudflare_access) currently do data_json = jsonencode(merge(data...._existing.data, { oidc_* })) on a path co-owned with operator-seeded fields.

Restructure: TF writes only its own fields to a new TF-owned path secret/fzymgc-house/cluster/<app>/oidc with plain, diffable data_json (not deprecated). The operator-seeded path becomes operator-only. The _existing data sources are deleted. No merge, no read, no data_json_wo_version bookkeeping, and the whole-version-replace clobber risk class disappears structurally.

Mechanics:

• custom_metadata (managed_by/application) moves to the new resources.
• State safety (critical): the old vault_kv_secret_v2.<app> resources must leave state via removed { from = ...; lifecycle { destroy = false } } blocks. A plain deletion destroys the shared path including operator-seeded fields (established pattern from the Authentik teardown, PR #1476). Every plan must show remove/forget, never destroy, before merge.
• ExternalSecrets use per-property remoteRefs; the oidc_* entries repoint key: to the new path. ESO’s reader policy is wildcarded (secret/data/*) — no ESO policy change. Filenames vary — most apps use argocd/app-configs/<app>/secrets.yaml, but clickstack uses external-secret.yaml and engram_ui lives at argocd/app-configs/agent-memory/external-secret-engram-ui.yaml; locate repoint targets by grepping for the Vault path, not by filename.
• The tf-keycloak workspace policy needs data CRUD + metadata CRUDL grants on the new paths (owned by tf/vault, must apply first). Note the ACL glob gotcha: cluster/<app>/* does not match the bare cluster/<app> — the new nested paths are covered by /* globs, but verify per-app grants.
• Cross-workspace coupling (cloudflare-access): tf/cloudflare/access.tf reads the path written by tf/keycloak/cloudflare_access.tf; its data source name repoints to cluster/cloudflare-access/oidc. That read itself remains a documented exception (see Plank E).
• Cross-workspace coupling (argocd): argocd is the sole Plank B app with no oidc_* remoteRefs — its Keycloak client creds are consumed by tf/cluster-bootstrap/argocd.tf (data.vault_kv_secret_v2.argocd_config → the Dex connector’s Kubernetes secret), a different TFC workspace. Rollout step 3 therefore does not apply to argocd; instead rollout step 4 splits argocd_config into two ephemeral reads: base cluster/argocd for the operator-seeded github_app_* fields, and cluster/argocd/oidc for keycloak_client_id/keycloak_client_secret — both feeding data_wo sinks (Plank D). Until step 4 applies, Dex keeps working off the frozen base-path copy, but a Keycloak client-secret rotation in that window would silently not propagate — run steps 2→4 promptly (step 4 is a manual local apply, see Rollout), and note that post-split, argocd client-secret rotations require the Plank D revision bump in cluster-bootstrap.
• Stale oidc_* fields left on the operator paths are harmless; clean up manually after cutover (documented in the rollout, not automated).

Plank C — public CA material → PKI issuer data source

ICA1 is hosted as a live Vault PKI mount (fzymgc-house/v1/ica1/v1). Public chain material moves from KV reads to data "vault_pki_secret_backend_issuer" (backend, issuer_ref), which is not deprecated, reads live on every plan (no staleness), and — being a normal data source — legally feeds regular attributes with full diffability. Public certs are not secrets; state persistence is a non-issue.

Sites:

• tf/keycloak/vault.tf — vault_jwt_auth_backend.oidc_discovery_ca_pem
• tf/cluster-bootstrap/external-secrets.tf — ESO ca.crt secret
• tf/cluster-bootstrap/argocd.tf — Dex rootCAs (helm values; raw-YAML is not write-only, but the value is public so a regular data source is correct). Shared-source coupling: this reference currently resolves the same data.vault_kv_secret_v2.fzymgc_ica1_ca declaration that cert-manager.tf consumes for the cert+key pair. Plank D converts that declaration to an ephemeral resource, whose values are illegal here — so argocd.tf must repoint to a newly, separately declared data "vault_pki_secret_backend_issuer" in the same PR (rollout step 4 bundles both files; do the repoint first or terraform validate fails).
• tf/hcp-terraform — TFC_VAULT_ENCODED_CACERT variable and the data.tf CA read

Notes:

• ca_chain exports a list of PEMs — join("\n", ...) where the KV fullchain string is expected.
• Implementation gate: before switching each site, byte-compare the PKI-served chain against the KV fullchain/cert fields (root inclusion differs between mounts depending on how the issuer was imported). Any mismatch stops that site’s migration pending investigation.
• Issuer-read grants on the ICA1 mount (added in tf/vault) are needed for terraform-keycloak-admin and terraform-hcp-terraform-local only. tf/cluster-bootstrap runs Local execution with the operator’s VAULT_TOKEN (the admin policy already carries full PKI read) — no policy change for it.
• tf/cluster-bootstrap/cert-manager.tf needs the cert and private key as a pair; it does not use the PKI source — one ephemeral KV read feeds both fields into data_wo (Plank D). Vault PKI never exposes CA private keys.

Plank D — genuinely-secret sinks with write-only support

• kubernetes_secret_v1 (cluster-bootstrap: cert-manager ICA1 pair, longhorn crypto config + R2 creds, argocd OIDC creds + GitHub App credentials): ephemeral read → data_wo + data_wo_revision. Floor the kubernetes pin at >= 3.2.1.
• tfe_variable (any future secret values; TFC_VAULT_ENCODED_CACERT itself moves to Plank C): value_wo + value_wo_version.
• tfe_notification_configuration: token_wo with token_wo_version omitted — auto-managed hash mode detects changes without manual bumps. The url value is also Vault-read today; once that read becomes ephemeral it cannot feed the plain url attribute, so it moves to url_wo (auto-managed as well).
• Revision convention: start at 1; bump manually on rotation. The ephemeral KV resource exports no current-version output, so revisions cannot be derived from Vault metadata. Rotation runbook lives in docs/operations/vault.md (update in the same PR as the first _wo sink; note helm’s trigger requires strict increase, kubernetes accepts any change).

Non-secret fields co-located in these k8s secrets (e.g. longhorn CRYPTO_KEY_CIPHER, R2 AWS_ENDPOINTS) move wholesale into data_wo with their secret siblings — splitting them buys nothing.

Plank E — documented exceptions (no legal sink today)

Four deprecated data sources remain (keycloak, cloudflare_access, discord_webhook, hcp_terraform_hmac), serving four sink groups, each annotated with a comment naming the upstream gap:

Site	Blocker
tf/keycloak/main.tf realm smtp_server (from + cf_email_token password, post-#1554)	keycloak provider: no password_wo (issue #1545 adjacent)
tf/keycloak/github_idp.tf client_secret	keycloak provider: no _wo on the GitHub-specific IdP resource; generic OIDC resource is not a valid substitute for OAuth2-only GitHub
tf/cloudflare/access.tf Access IdP client_secret	cloudflare provider: write-only shipped in 5.6.0 and reverted (#5702)
tf/cloudflare/workers.tf two secret_text bindings	cloudflare provider: no write-only; no dedicated worker-secret resource in v5 (v4’s cloudflare_worker_secret was dropped)

Surface-shrink: already done by PR #1554 — SMTP host/port/username are hardcoded non-secret values and the former smtp data source is deleted. The remaining SMTP reads (from, cf_email_token) live on the keycloak secret, which persists regardless for the GitHub IdP exception; nothing further to shrink.

data "vault_generic_secret" is un-deprecated at v5.10.1 and could silence these warnings; rejected — identical read-into-state semantics, likely to be deprecated next, dodges rather than migrates. Recorded as a fallback lever only.

Upstream: file issues only (no PRs for now) for the keycloak password_wo/GitHub-IdP gap and the cloudflare write-only gaps. Issue text must contain no internal specifics (no hostnames, paths, secret names), be written in a natural human voice, and be reviewed by the operator before filing. Tracked as a follow-up bead.

Out of scope

• tf/authentik — workspace is being decommissioned (hl-5g05); its reads and writes are left untouched and disappear with the workspace.
• Plain data_json writes without embedded reads (e.g. tf/keycloak/octopus.tf, tf/vault/secrets-cloudflare.tf, tf/cloudflare/{r2,workers,tunnel}.tf) — not deprecated, no change.
• Upstream PR contributions (issues only, per operator decision).

Rollout

Each numbered step is an independent PR. Apply semantics differ by step: steps 1, 2, 6, 7 target Agent-execution TFC workspaces and apply automatically on merge to main; step 3 is applied by ArgoCD sync; steps 4 and 5 target Local-execution workspaces (main-cluster-bootstrap, hcp-terraform) where merging alone changes nothing — an operator must run terraform apply locally after merge (per docs/operations/cluster-bootstrap.md / docs/operations/hcp-terraform.md).

1. tf/vault — policy grants: tf-keycloak CRUD on cluster/<app>/oidc paths; terraform-cloudflare-admin read (data + metadata) on cluster/cloudflare-access/oidc (its existing grant is the bare path only, which does not cover the nested subpath — same ACL-glob rule as Plank B); issuer-read grants on the ICA1 PKI mount for terraform-keycloak-admin and terraform-hcp-terraform-local (tf/cluster-bootstrap needs none — Local execution under the operator’s admin token, which already has full PKI read). Must apply before steps 2–6 (403-at-plan race otherwise; speculative plans of later PRs will 403 until this applies).
2. tf/keycloak — Plank B split paths + removed blocks; Plank A provider config; Plank C oidc_discovery_ca_pem; Plank E exception comments. Plan review gate: every old vault secret shows remove/forget, never destroy.
3. argocd/ — repoint oidc_* remoteRefs to cluster/<app>/oidc (after step 2 applies, so the new paths exist before ESO refreshes). Covers five of the seven Plank B apps; argocd has no remoteRefs (its repoint happens in step 4 — see Plank B’s argocd coupling bullet) and cloudflare-access is consumed by a Terraform data source, repointed in step 6.
4. tf/cluster-bootstrap — Plank C (ESO ca.crt, Dex rootCAs via its own PKI issuer data source) + Plank D (data_wo conversions), including the argocd_config split: two ephemeral reads (base path for github_app_*, cluster/argocd/oidc for the Keycloak client creds); kubernetes pin floor 3.2.1. Local execution: the argocd frozen-creds window stays open until the operator’s manual terraform apply — schedule it immediately after step 2’s TFC apply, not just after the merge.
5. tf/hcp-terraform — Plank C (CACERT via PKI source) + Plank D (notification token_wo/url_wo auto-managed). Local execution — manual operator apply after merge.
6. tf/cloudflare — repoint the access read to cluster/cloudflare-access/oidc; Plank A provider token; Plank E exception comments.
7. tf/uptime-kuma — vault pin bump + Plank A provider config.
8. Cleanup + docs — remove stale oidc_* fields from operator paths (manual, via raw-endpoint patch per existing runbook); update docs/reference/secrets.md (new path layout) and docs/operations/vault.md (revision-bump rotation runbook); file the sanitized upstream issues after operator review.

Risks and mitigations

• Operator-path destruction — mitigated by removed+destroy = false blocks and a mandatory plan-output check (step 2 gate).
• ESO repoint race — new paths are written by step 2 before step 3 repoints; old fields remain during the overlap, so consumers never see a missing key.
• PKI chain mismatch vs KV fullchain — byte-diff gate per site before switching (Plank C).
• _wo rotation staleness — revisions are manual by design; the runbook documents the bump. tfe notification token avoids this via auto-hash mode. Split-path Plank B avoids it entirely for the highest-churn secrets (Keycloak client secrets remain fully diffable).
• 403 apply-ordering — step 1 applies first; later speculative plans may 403 until it does (known TFC behavior in this repo; re-run after step 1 applies).

Verification

• Per-PR: TFC speculative plan shows zero deprecation warnings for migrated sites; only the four Plank E exception data sources’ warnings remain repo-wide at the end.
• Step 2: plan shows remove/forget (not destroy) for old secrets; post-apply, operator-path data hash unchanged; new <app>/oidc paths populated.
• Step 3: ESO SecretSynced on all repointed ExternalSecrets; consuming pods unchanged (secret content identical — restart-latent class, so no pod churn expected).
• Step 4: post-apply, in-cluster secret values byte-identical to pre-migration (compare data hashes as in PR #1476); plan output confirms the Dex connector’s Keycloak creds now source from cluster/argocd/oidc (two ephemeral read addresses visible) — guards against the silent frozen-base-path failure mode.