HCP Terraform Operations
Every Terraform module under tf/ runs in an HCP Terraform workspace. A pull request produces a
speculative plan, and a merge to main applies. Most workspaces run on an agent inside the
cluster and authenticate to Vault with a per-run JWT. This page states what exists, where each
part is defined, and the procedures that touch it.
What exists
Section titled “What exists”| Property | Value | Defined in |
|---|---|---|
| Organization | fzymgc-house |
tf/hcp-terraform/terraform.tf |
| Project | main-cluster |
tf/hcp-terraform/project.tf |
| Agent pool | fzymgc-house-k8s, one agent, in the hcp-terraform namespace |
argocd/app-configs/hcp-terraform-operator/agent-pool.yaml |
| Operator | The HCP Terraform Operator Helm chart | argocd/cluster-app/templates/hcp-terraform-operator.yaml |
| Operator API token | The api_token field of secret/fzymgc-house/cluster/hcp-terraform, through an ExternalSecret |
argocd/app-configs/hcp-terraform-operator/external-secrets.yaml |
| Workspaces | local.all_workspaces |
tf/hcp-terraform/workspaces.tf |
The operator holds the HCP API token and mints the agent tokens named under agentTokens in
the AgentPool. The agent runs one replica, so runs execute one at a time and never race on
state. An init container installs the internal CA so the agent trusts vault.fzymgc.house.
Workspaces
Section titled “Workspaces”Every workspace in the map is VCS-connected to main, has auto_apply, speculative_enabled
and file_triggers_enabled set, and pins the Terraform version in workspaces.tf. A run starts
only when a merge touches the workspace’s working directory, or one of its trigger_prefixes.
| Workspace | Directory | Execution | Vault run role |
|---|---|---|---|
main-cluster-vault |
tf/vault |
agent | tfc-vault |
main-cluster-keycloak |
tf/keycloak |
agent | tfc-keycloak |
main-cluster-cloudflare |
tf/cloudflare |
agent | tfc-cloudflare |
main-cluster-core-services |
tf/core-services |
agent | tfc-core-services |
main-cluster-tailscale |
tf/tailscale, also triggered by tailscale/ |
agent | tfc-tailscale |
main-cluster-uptime-kuma |
tf/uptime-kuma |
agent | tfc-uptime-kuma |
main-cluster-unifi |
tf/unifi |
agent | tfc-unifi |
main-cluster-litellm |
tf/litellm |
agent | tfc-litellm |
main-cluster-backblaze |
tf/backblaze |
agent | tfc-backblaze |
main-cluster-rustfs |
tf/rustfs |
agent | tfc-rustfs |
main-cluster-bootstrap |
tf/cluster-bootstrap |
local | none, VAULT_TOKEN |
hcp-terraform |
tf/hcp-terraform |
local | none, VAULT_TOKEN |
Two workspaces run locally because each manages something the agent depends on.
main-cluster-bootstrap deploys the operator, and hcp-terraform manages the agent pool and
the workspaces themselves. The hcp-terraform workspace is not VCS-connected, so a change to
the map lands only when an operator applies it.
Four workspaces need agent execution for reachability, not only for credentials. The
litellm and rustfs providers call addresses that only a run inside the
cluster or on the LAN reaches, and vault and core-services read the in-cluster kubeconfig.
The comments on each entry in workspaces.tf state the reason.
The main-cluster-cloudflare workspace reads secret/fzymgc-house/cluster/cloudflare-access/oidc,
which main-cluster-keycloak writes. A pull request that touches both can fail the consumer’s
plan until the producer applies. Merge, and re-run the consumer if it fails after the merge.
Vault dynamic credentials
Section titled “Vault dynamic credentials”Each agent workspace authenticates to Vault with a signed JWT from HCP Terraform. HCP Terraform
exchanges the JWT for a Vault token before the run and writes the token to a file. The Vault
provider reads it through auth_login_token_file, as tf/core-services/vault.tf shows.
| Variable | Value | Defined in |
|---|---|---|
TFC_VAULT_PROVIDER_AUTH |
true |
tf/hcp-terraform/workspace-variables.tf |
TFC_VAULT_ADDR |
https://vault.fzymgc.house |
the same file |
TFC_VAULT_AUTH_PATH |
jwt-hcp-terraform |
the same file |
TFC_VAULT_RUN_ROLE |
tfc-<second tag of the workspace> |
the same file |
TFC_VAULT_ENCODED_CACERT |
The ICA1 chain, read from the PKI mount | the same file |
The JWT backend and one role per workspace are in tf/vault/jwt-hcp-terraform.tf. Each role
binds the sub claim to its own workspace and grants one policy from
tf/vault/policy-terraform-workspaces.tf. A workspace entry without a JWT role authenticates
to nothing, and its first plan fails at its first Vault read.
The main-cluster-vault and main-cluster-core-services workspaces also carry empty
kubeconfig_path and kubeconfig_context variables, so the Kubernetes provider uses the
agent’s in-cluster identity.
Drift detection and guarded apply
Section titled “Drift detection and guarded apply”The Temporal terraform worker in argocd/cluster-app/templates/temporal-workers.yaml runs a
scheduled plan on the workspaces listed under workspaces and posts the result to Discord
#system-notifications. The plan runs each day at 06:00, from the drift_detection schedule
of the chart. A plan with changes posts Approve and Reject buttons, and the run holds the queue of
the workspace for up to four hours, until a person decides or the approval times out. Approve
applies the plan, so the files win. For main-cluster-unifi, a change in the plan is usually a
UI change: choose Reject, and adopt the change in tf/unifi through a pull request.
The same worker holds a guarded apply of tf/hcp-terraform: every plan needs approval, and no
rule auto-applies. Its Vault role is terraform-hcp-bootstrap in
tf/vault/policy-temporal.tf.
Run notifications
Section titled “Run notifications”A Cloudflare Worker turns HCP Terraform run webhooks into Discord embeds.
| Component | Defined in |
|---|---|
| Worker code | cloudflare/workers/hcp-terraform-discord/worker.js |
| Worker deployment and bindings | tf/cloudflare/workers.tf |
| Notification configuration | tf/hcp-terraform/notifications.tf, the URL and HMAC token as write-only values |
| Discord webhook URL | secret/fzymgc-house/infrastructure/cloudflare/discord-webhook |
| HMAC token | secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac, generated by tf/vault |
Workflow
Section titled “Workflow”Standard changes
Section titled “Standard changes”- Create a feature branch.
- Edit the module under
tf/. - Open a pull request. HCP Terraform posts a speculative plan for every affected workspace.
- Read each plan.
- Merge. Each affected workspace applies in commit order.
main-cluster-unifiis the exception: its run waits until a person confirms it in the HCP UI (auto_apply = falseintf/hcp-terraform/workspaces.tf). Before you confirm, make sure that the plan changes only what the pull request changed. A change to any other object is drift from the UI. Discard the run, adopt the drift intf/unifi, and merge that first.
Do not start a run through the API or an MCP tool. Read-only calls (list_runs,
get_run_details, get_workspace_details) are fine. A terraform plan through the module’s
cloud block is also fine, because a speculative plan creates no apply. The rule forbids
reaching past the merge path to change infrastructure.
Provider version bumps
Section titled “Provider version bumps”A change to a module’s versions.tf and lock file changes the providers HCP installs, and a
minor bump can change a plan. Because every workspace auto-applies, read each affected
speculative plan before you merge. If a plan shows a change you did not intend, pin the prior
version or fix the drift first.
Regenerate the lock file for the agent platform and for local use:
terraform -chdir=tf/<module> providers lock \ -platform=linux_amd64 -platform=darwin_arm64Manual runs
Section titled “Manual runs”The two local workspaces run from a workstation with a Vault token:
vault login -method=oidcterraform -chdir=tf/<module> planterraform -chdir=tf/<module> applyThe hcp-terraform module also runs under the terraform-hcp-terraform-local policy in
tf/vault/policy-terraform-workspaces.tf, which reads only the notification secrets and the
issuer chain.
For an agent workspace that needs a run without a merge, use the HCP Terraform UI: open the workspace, start a new run, and choose plan and apply.
Procedures
Section titled “Procedures”Add a workspace
Section titled “Add a workspace”-
Add a
{ dir, tags }entry tolocal.all_workspacesintf/hcp-terraform/workspaces.tf. The second tag names the Vault run role. Addtrigger_prefixesif the module reads a file outside its directory. Addexecution_mode = "local"only if the module cannot run on the agent. -
Add the matching
tfc-<tag>role totf/vault/jwt-hcp-terraform.tfand its policy totf/vault/policy-terraform-workspaces.tf, and merge that first. Themain-cluster-vaultworkspace applies it. -
Get a real plan of the new module against the live system before the workspace exists. A new workspace queues a run on the latest
maincommit as soon as it is created, and applies it. -
Apply
tf/hcp-terraformfrom a workstation, as under Manual runs. -
Make sure that the workspace exists with its variables:
Terminal window terraform -chdir=tf/hcp-terraform state list | rg '<workspace>'
Rotate the operator’s API token
Section titled “Rotate the operator’s API token”The operator authenticates to HCP Terraform with the api_token field of
secret/fzymgc-house/cluster/hcp-terraform. To rotate it:
-
Create a new token in HCP Terraform for the account the operator uses, and revoke the old one after step 3.
-
Write it to Vault:
Terminal window vault kv patch secret/fzymgc-house/cluster/hcp-terraform api_token=<new token> -
Wait for the ExternalSecret refresh, or restart the operator:
Terminal window kubectl -n hcp-terraform rollout restart deployment/hcp-terraform-operatorkubectl -n hcp-terraform get agentpool fzymgc-house-agents
The operator mints the agent tokens itself from the agentTokens list in the AgentPool. To
rotate an agent token, rename its entry there and let ArgoCD sync. The operator revokes the old
token and creates the new one.
Merge past a blocked required check
Section titled “Merge past a blocked required check”main requires two check contexts: pr-gates and Octopus Review. Both are declared in
tf/core-services/github.tf and live in the GitHub ruleset the same file adopts. Octopus Review
runs on this cluster, so it cannot go green while the cluster, LiteLLM or the model provider is
down, or when a review fails on its own error. When a check will not go green and a change has
to land, there are two sanctioned paths and one forbidden one. Path 1 is narrower, needs no
undo, and GitHub records it.
Path 1, bypass the single pull request. This is the default.
tf/core-services/github.tf carries two bypass_actors blocks: one OrganizationAdmin, one
RepositoryRole admin, each at bypass_mode = "pull_request". The grant is the union of those
two role memberships. It affects exactly one pull request, it leaves the gate intact for every
other pull request, and GitHub writes an audit entry for the bypass. The move is the merge. The
bypass applies through role membership, so there is no flag to pass, no ruleset field to change,
and nothing to put back:
gh pr merge <number> --squashNo ruleset mutation happens on this path. creation, deletion and non_fast_forward stay
enforced against direct pushes, because pull_request mode scopes the bypass to the
pull-request merge path.
Path 2, drop a single required context. This is the escalation.
Use this only when a context is broken for every pull request, because the workflow cannot start
or the job name no longer matches the required string. Its cost: it unblocks everyone, including
pull requests nobody vetted, and nothing reports it until the next drift detection run on
main-cluster-core-services. Read the ruleset id from tf/core-services/github.tf.
# 1. Read the WHOLE ruleset object, unfiltered. Keep this copy untouched: it is both the# input to the edit and the reference the restore compares against.gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> > /tmp/ruleset-original.json
# 2. Remove exactly ONE entry from the required_status_checks context array, selected by its# context string. Every other rule and every other field is carried through untouched.jq --arg ctx pr-gates ' (.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks) |= map(select(.context != $ctx))' /tmp/ruleset-original.json > /tmp/ruleset-dropped.json
# 3. PUT the WHOLE edited body back, from the file. Never assemble this body by hand.gh api -X PUT repos/fzymgc-house/selfhosted-cluster/rulesets/<id> \ --input /tmp/ruleset-dropped.json > /dev/null
# 4. Read back what is still required.gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> \ --jq '.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks[].context'The read-modify-write is a correctness requirement. PUT /repos/{owner}/{repo}/rulesets/{id}
has no partial-update form. It replaces the entire object, so a hand-typed body that carries
only required_status_checks deletes the pull_request, creation, deletion and
non_fast_forward rules with it. Feeding the whole object back through --input is what stops
that. The round trip also preserves require_extra_approval_for_unattributed_changes, which the
Terraform provider does not model and omits from its own writes.
Path 2, restore. Run the same read-modify-write in the opposite direction, then prove that the required set came back:
# 1. Read the whole object again.gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> > /tmp/ruleset-current.json
# 2. Add the context back. Same shape as the drop, opposite operation. An app check needs# its integration_id back too: for `Octopus Review`, add `integration_id: 3985029` to the# object, as tf/core-services/github.tf declares it.jq --arg ctx pr-gates ' (.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks) |= (. + [{context: $ctx}] | unique_by(.context))' /tmp/ruleset-current.json > /tmp/ruleset-restored.json
# 3. PUT the whole restored body.gh api -X PUT repos/fzymgc-house/selfhosted-cluster/rulesets/<id> \ --input /tmp/ruleset-restored.json > /dev/null
# 4. Side A: what the live ruleset requires now.gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> \ --jq '.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks[].context' \ | sort > /tmp/live-contexts.txtset -euo pipefail
# 5. Side B: what tf/core-services/github.tf commits. Read it from the HCL rather than# retyping it, so there is no second copy of the expected set to drift. This procedure# runs from /tmp, so every repository path below is anchored on REPO_ROOT.: "${REPO_ROOT:?point REPO_ROOT at your selfhosted-cluster checkout before running this block}"LIVE_CONTEXTS="${LIVE_CONTEXTS:-/tmp/live-contexts.txt}"EXPECTED_CONTEXTS="${EXPECTED_CONTEXTS:-/tmp/expected-contexts.txt}"GITHUB_TF="$REPO_ROOT/tf/core-services/github.tf"
{ rg -o -e '^[[:space:]]*context[[:space:]]*=[[:space:]]*"[^"]+"' "$GITHUB_TF" || true; } \ | { rg -o -e '"[^"]+"$' || true; } | tr -d '"' | sort > "$EXPECTED_CONTEXTS"
# 5b. Cardinality cross-check. Both numbers are read from that same file, so nothing here# hard-codes the size of the required set, and a partially disabled block cannot pass# off a short expectation as the whole one.ctx_n="$( { rg -o -e '^[[:space:]]*context[[:space:]]*=[[:space:]]*"[^"]+"' "$GITHUB_TF" || true; } | wc -l | tr -d ' ')"blk_n="$( { rg -o -e '^[[:space:]]*required_check[[:space:]]*\{' "$GITHUB_TF" || true; } | wc -l | tr -d ' ')"live_n=0if [ -s "$LIVE_CONTEXTS" ]; then live_n="$(wc -l < "$LIVE_CONTEXTS" | tr -d ' ')"fi
# 6. Assert, BEFORE comparing. Each of these ABORTS: nothing printed below a failure here# may be read as success.test "$live_n" -gt 0 || { printf 'FAIL: the live required set is EMPTY. main is unprotected. Nothing printed below this line is success.\n' >&2 exit 1}test "$ctx_n" -gt 0 || { printf 'FAIL: no committed context found in %s. Either its required_status_checks block is commented out, or REPO_ROOT points somewhere else.\n' "$GITHUB_TF" >&2 exit 1}test "$ctx_n" = "$blk_n" || { printf 'FAIL: %s declares %s context(s) across %s required_check block(s). Part of the block is disabled, so the expectation is short.\n' "$GITHUB_TF" "$ctx_n" "$blk_n" >&2 exit 1}
# 7. Set equality, both sides sorted. Reachable only after all three assertions passed.diff -u "$EXPECTED_CONTEXTS" "$LIVE_CONTEXTS" || { printf 'FAIL: the live required set is not the committed set. The restore did not land.\n' >&2 exit 1}printf 'RESTORED: required set equals the committed set (%s contexts)\n' "$ctx_n"# 8. And the whole rules array, not only the context list. The object you restored must be# the object you started from.diff -u <(jq -S '.rules' /tmp/ruleset-original.json) \ <(gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> --jq '.rules' | jq -S '.')Step 7 is a sorted diff rather than a presence test, because a diff of two empty files is
green, and an empty required set is the one state this check most needs to catch. The three
assertions in step 6 abort, so a guard that prints a failure and carries on cannot print the
RESTORED: line over an unprotected main. The cardinality check in step 5b catches the
narrower version of the same trap: a partially disabled block in the HCL yields a short
expectation, and a short live set matches it. The procedure runs from /tmp, which is why
every repository path is anchored on REPO_ROOT.
The next main-cluster-core-services apply also restores the set from the HCL. The manual
restore exists so that the gap is minutes rather than the time until the next merge.
The forbidden reflex: deleting the required_status_checks rule.
Do not remove the rule object from the ruleset. That removes the gate for every pull request indefinitely, and nothing reports that it is gone. Dropping to zero contexts and deleting the rule are different operations, and only the first is recoverable by inspection. Path 2 delivers the narrow version of the same relief, restores in four commands, and leaves a trace the drift detector finds.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Check | Fix |
|---|---|---|
| A plan fails on a pull request | The plan output in the pull request and the run log in HCP Terraform | Fix the module and push |
| An apply fails after a merge | The run log, then the resource state | Fix and re-run the workspace from the UI. A state problem needs a hand-run from a workstation |
| A run stays queued | kubectl -n hcp-terraform get pods, then kubectl -n hcp-terraform get externalsecret,agentpool |
The agent pod, the operator and its token. vault kv get -field=api_token secret/fzymgc-house/cluster/hcp-terraform must return a value |
role not found at the Vault read |
vault read auth/jwt-hcp-terraform/role/tfc-<tag> |
Add the role to tf/vault/jwt-hcp-terraform.tf |
claim not in bound_claims |
vault read -field=bound_claims auth/jwt-hcp-terraform/role/tfc-<tag> |
The sub claim must name the workspace exactly |
permission denied at a Vault read |
vault policy read <policy> |
Add the path to the workspace’s policy in tf/vault/policy-terraform-workspaces.tf |
| The Discord embed does not arrive | The Worker logs in Cloudflare, then tf/hcp-terraform/notifications.tf |
Rotate the HMAC token or the webhook URL and bump the notification, see Vault |
See Also
Section titled “See Also”- Vault Operations, the auth methods and the rotation procedure
- Cloudflare Operations, the Worker
- GitHub Tokens, the
fzymgc-terraformApp that writes the ruleset tf/CLAUDE.md, the workspace table and the GitOps rule