Skip to content

HCP Terraform Operations

Every Terraform module under tf/ runs in an HCP Terraform workspace. A pull request produces a speculative plan, and a merge to main applies. Most workspaces run on an agent inside the cluster and authenticate to Vault with a per-run JWT. This page states what exists, where each part is defined, and the procedures that touch it.

Property Value Defined in
Organization fzymgc-house tf/hcp-terraform/terraform.tf
Project main-cluster tf/hcp-terraform/project.tf
Agent pool fzymgc-house-k8s, one agent, in the hcp-terraform namespace argocd/app-configs/hcp-terraform-operator/agent-pool.yaml
Operator The HCP Terraform Operator Helm chart argocd/cluster-app/templates/hcp-terraform-operator.yaml
Operator API token The api_token field of secret/fzymgc-house/cluster/hcp-terraform, through an ExternalSecret argocd/app-configs/hcp-terraform-operator/external-secrets.yaml
Workspaces local.all_workspaces tf/hcp-terraform/workspaces.tf

The operator holds the HCP API token and mints the agent tokens named under agentTokens in the AgentPool. The agent runs one replica, so runs execute one at a time and never race on state. An init container installs the internal CA so the agent trusts vault.fzymgc.house.

Every workspace in the map is VCS-connected to main, has auto_apply, speculative_enabled and file_triggers_enabled set, and pins the Terraform version in workspaces.tf. A run starts only when a merge touches the workspace’s working directory, or one of its trigger_prefixes.

Workspace Directory Execution Vault run role
main-cluster-vault tf/vault agent tfc-vault
main-cluster-keycloak tf/keycloak agent tfc-keycloak
main-cluster-cloudflare tf/cloudflare agent tfc-cloudflare
main-cluster-core-services tf/core-services agent tfc-core-services
main-cluster-tailscale tf/tailscale, also triggered by tailscale/ agent tfc-tailscale
main-cluster-uptime-kuma tf/uptime-kuma agent tfc-uptime-kuma
main-cluster-unifi tf/unifi agent tfc-unifi
main-cluster-litellm tf/litellm agent tfc-litellm
main-cluster-backblaze tf/backblaze agent tfc-backblaze
main-cluster-rustfs tf/rustfs agent tfc-rustfs
main-cluster-bootstrap tf/cluster-bootstrap local none, VAULT_TOKEN
hcp-terraform tf/hcp-terraform local none, VAULT_TOKEN

Two workspaces run locally because each manages something the agent depends on. main-cluster-bootstrap deploys the operator, and hcp-terraform manages the agent pool and the workspaces themselves. The hcp-terraform workspace is not VCS-connected, so a change to the map lands only when an operator applies it.

Four workspaces need agent execution for reachability, not only for credentials. The litellm and rustfs providers call addresses that only a run inside the cluster or on the LAN reaches, and vault and core-services read the in-cluster kubeconfig. The comments on each entry in workspaces.tf state the reason.

The main-cluster-cloudflare workspace reads secret/fzymgc-house/cluster/cloudflare-access/oidc, which main-cluster-keycloak writes. A pull request that touches both can fail the consumer’s plan until the producer applies. Merge, and re-run the consumer if it fails after the merge.

Each agent workspace authenticates to Vault with a signed JWT from HCP Terraform. HCP Terraform exchanges the JWT for a Vault token before the run and writes the token to a file. The Vault provider reads it through auth_login_token_file, as tf/core-services/vault.tf shows.

Variable Value Defined in
TFC_VAULT_PROVIDER_AUTH true tf/hcp-terraform/workspace-variables.tf
TFC_VAULT_ADDR https://vault.fzymgc.house the same file
TFC_VAULT_AUTH_PATH jwt-hcp-terraform the same file
TFC_VAULT_RUN_ROLE tfc-<second tag of the workspace> the same file
TFC_VAULT_ENCODED_CACERT The ICA1 chain, read from the PKI mount the same file

The JWT backend and one role per workspace are in tf/vault/jwt-hcp-terraform.tf. Each role binds the sub claim to its own workspace and grants one policy from tf/vault/policy-terraform-workspaces.tf. A workspace entry without a JWT role authenticates to nothing, and its first plan fails at its first Vault read.

The main-cluster-vault and main-cluster-core-services workspaces also carry empty kubeconfig_path and kubeconfig_context variables, so the Kubernetes provider uses the agent’s in-cluster identity.

The Temporal terraform worker in argocd/cluster-app/templates/temporal-workers.yaml runs a scheduled plan on the workspaces listed under workspaces and posts the result to Discord #system-notifications. The plan runs each day at 06:00, from the drift_detection schedule of the chart. A plan with changes posts Approve and Reject buttons, and the run holds the queue of the workspace for up to four hours, until a person decides or the approval times out. Approve applies the plan, so the files win. For main-cluster-unifi, a change in the plan is usually a UI change: choose Reject, and adopt the change in tf/unifi through a pull request.

The same worker holds a guarded apply of tf/hcp-terraform: every plan needs approval, and no rule auto-applies. Its Vault role is terraform-hcp-bootstrap in tf/vault/policy-temporal.tf.

A Cloudflare Worker turns HCP Terraform run webhooks into Discord embeds.

Component Defined in
Worker code cloudflare/workers/hcp-terraform-discord/worker.js
Worker deployment and bindings tf/cloudflare/workers.tf
Notification configuration tf/hcp-terraform/notifications.tf, the URL and HMAC token as write-only values
Discord webhook URL secret/fzymgc-house/infrastructure/cloudflare/discord-webhook
HMAC token secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac, generated by tf/vault
  1. Create a feature branch.
  2. Edit the module under tf/.
  3. Open a pull request. HCP Terraform posts a speculative plan for every affected workspace.
  4. Read each plan.
  5. Merge. Each affected workspace applies in commit order. main-cluster-unifi is the exception: its run waits until a person confirms it in the HCP UI (auto_apply = false in tf/hcp-terraform/workspaces.tf). Before you confirm, make sure that the plan changes only what the pull request changed. A change to any other object is drift from the UI. Discard the run, adopt the drift in tf/unifi, and merge that first.

Do not start a run through the API or an MCP tool. Read-only calls (list_runs, get_run_details, get_workspace_details) are fine. A terraform plan through the module’s cloud block is also fine, because a speculative plan creates no apply. The rule forbids reaching past the merge path to change infrastructure.

A change to a module’s versions.tf and lock file changes the providers HCP installs, and a minor bump can change a plan. Because every workspace auto-applies, read each affected speculative plan before you merge. If a plan shows a change you did not intend, pin the prior version or fix the drift first.

Regenerate the lock file for the agent platform and for local use:

Terminal window
terraform -chdir=tf/<module> providers lock \
-platform=linux_amd64 -platform=darwin_arm64

The two local workspaces run from a workstation with a Vault token:

Terminal window
vault login -method=oidc
terraform -chdir=tf/<module> plan
terraform -chdir=tf/<module> apply

The hcp-terraform module also runs under the terraform-hcp-terraform-local policy in tf/vault/policy-terraform-workspaces.tf, which reads only the notification secrets and the issuer chain.

For an agent workspace that needs a run without a merge, use the HCP Terraform UI: open the workspace, start a new run, and choose plan and apply.

  1. Add a { dir, tags } entry to local.all_workspaces in tf/hcp-terraform/workspaces.tf. The second tag names the Vault run role. Add trigger_prefixes if the module reads a file outside its directory. Add execution_mode = "local" only if the module cannot run on the agent.

  2. Add the matching tfc-<tag> role to tf/vault/jwt-hcp-terraform.tf and its policy to tf/vault/policy-terraform-workspaces.tf, and merge that first. The main-cluster-vault workspace applies it.

  3. Get a real plan of the new module against the live system before the workspace exists. A new workspace queues a run on the latest main commit as soon as it is created, and applies it.

  4. Apply tf/hcp-terraform from a workstation, as under Manual runs.

  5. Make sure that the workspace exists with its variables:

    Terminal window
    terraform -chdir=tf/hcp-terraform state list | rg '<workspace>'

The operator authenticates to HCP Terraform with the api_token field of secret/fzymgc-house/cluster/hcp-terraform. To rotate it:

  1. Create a new token in HCP Terraform for the account the operator uses, and revoke the old one after step 3.

  2. Write it to Vault:

    Terminal window
    vault kv patch secret/fzymgc-house/cluster/hcp-terraform api_token=<new token>
  3. Wait for the ExternalSecret refresh, or restart the operator:

    Terminal window
    kubectl -n hcp-terraform rollout restart deployment/hcp-terraform-operator
    kubectl -n hcp-terraform get agentpool fzymgc-house-agents

The operator mints the agent tokens itself from the agentTokens list in the AgentPool. To rotate an agent token, rename its entry there and let ArgoCD sync. The operator revokes the old token and creates the new one.

main requires two check contexts: pr-gates and Octopus Review. Both are declared in tf/core-services/github.tf and live in the GitHub ruleset the same file adopts. Octopus Review runs on this cluster, so it cannot go green while the cluster, LiteLLM or the model provider is down, or when a review fails on its own error. When a check will not go green and a change has to land, there are two sanctioned paths and one forbidden one. Path 1 is narrower, needs no undo, and GitHub records it.

Path 1, bypass the single pull request. This is the default.

tf/core-services/github.tf carries two bypass_actors blocks: one OrganizationAdmin, one RepositoryRole admin, each at bypass_mode = "pull_request". The grant is the union of those two role memberships. It affects exactly one pull request, it leaves the gate intact for every other pull request, and GitHub writes an audit entry for the bypass. The move is the merge. The bypass applies through role membership, so there is no flag to pass, no ruleset field to change, and nothing to put back:

Terminal window
gh pr merge <number> --squash

No ruleset mutation happens on this path. creation, deletion and non_fast_forward stay enforced against direct pushes, because pull_request mode scopes the bypass to the pull-request merge path.

Path 2, drop a single required context. This is the escalation.

Use this only when a context is broken for every pull request, because the workflow cannot start or the job name no longer matches the required string. Its cost: it unblocks everyone, including pull requests nobody vetted, and nothing reports it until the next drift detection run on main-cluster-core-services. Read the ruleset id from tf/core-services/github.tf.

Terminal window
# 1. Read the WHOLE ruleset object, unfiltered. Keep this copy untouched: it is both the
# input to the edit and the reference the restore compares against.
gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> > /tmp/ruleset-original.json
# 2. Remove exactly ONE entry from the required_status_checks context array, selected by its
# context string. Every other rule and every other field is carried through untouched.
jq --arg ctx pr-gates '
(.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks)
|= map(select(.context != $ctx))
' /tmp/ruleset-original.json > /tmp/ruleset-dropped.json
# 3. PUT the WHOLE edited body back, from the file. Never assemble this body by hand.
gh api -X PUT repos/fzymgc-house/selfhosted-cluster/rulesets/<id> \
--input /tmp/ruleset-dropped.json > /dev/null
# 4. Read back what is still required.
gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> \
--jq '.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks[].context'

The read-modify-write is a correctness requirement. PUT /repos/{owner}/{repo}/rulesets/{id} has no partial-update form. It replaces the entire object, so a hand-typed body that carries only required_status_checks deletes the pull_request, creation, deletion and non_fast_forward rules with it. Feeding the whole object back through --input is what stops that. The round trip also preserves require_extra_approval_for_unattributed_changes, which the Terraform provider does not model and omits from its own writes.

Path 2, restore. Run the same read-modify-write in the opposite direction, then prove that the required set came back:

Terminal window
# 1. Read the whole object again.
gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> > /tmp/ruleset-current.json
# 2. Add the context back. Same shape as the drop, opposite operation. An app check needs
# its integration_id back too: for `Octopus Review`, add `integration_id: 3985029` to the
# object, as tf/core-services/github.tf declares it.
jq --arg ctx pr-gates '
(.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks)
|= (. + [{context: $ctx}] | unique_by(.context))
' /tmp/ruleset-current.json > /tmp/ruleset-restored.json
# 3. PUT the whole restored body.
gh api -X PUT repos/fzymgc-house/selfhosted-cluster/rulesets/<id> \
--input /tmp/ruleset-restored.json > /dev/null
# 4. Side A: what the live ruleset requires now.
gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> \
--jq '.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks[].context' \
| sort > /tmp/live-contexts.txt
Terminal window
set -euo pipefail
# 5. Side B: what tf/core-services/github.tf commits. Read it from the HCL rather than
# retyping it, so there is no second copy of the expected set to drift. This procedure
# runs from /tmp, so every repository path below is anchored on REPO_ROOT.
: "${REPO_ROOT:?point REPO_ROOT at your selfhosted-cluster checkout before running this block}"
LIVE_CONTEXTS="${LIVE_CONTEXTS:-/tmp/live-contexts.txt}"
EXPECTED_CONTEXTS="${EXPECTED_CONTEXTS:-/tmp/expected-contexts.txt}"
GITHUB_TF="$REPO_ROOT/tf/core-services/github.tf"
{ rg -o -e '^[[:space:]]*context[[:space:]]*=[[:space:]]*"[^"]+"' "$GITHUB_TF" || true; } \
| { rg -o -e '"[^"]+"$' || true; } | tr -d '"' | sort > "$EXPECTED_CONTEXTS"
# 5b. Cardinality cross-check. Both numbers are read from that same file, so nothing here
# hard-codes the size of the required set, and a partially disabled block cannot pass
# off a short expectation as the whole one.
ctx_n="$( { rg -o -e '^[[:space:]]*context[[:space:]]*=[[:space:]]*"[^"]+"' "$GITHUB_TF" || true; } | wc -l | tr -d ' ')"
blk_n="$( { rg -o -e '^[[:space:]]*required_check[[:space:]]*\{' "$GITHUB_TF" || true; } | wc -l | tr -d ' ')"
live_n=0
if [ -s "$LIVE_CONTEXTS" ]; then
live_n="$(wc -l < "$LIVE_CONTEXTS" | tr -d ' ')"
fi
# 6. Assert, BEFORE comparing. Each of these ABORTS: nothing printed below a failure here
# may be read as success.
test "$live_n" -gt 0 || {
printf 'FAIL: the live required set is EMPTY. main is unprotected. Nothing printed below this line is success.\n' >&2
exit 1
}
test "$ctx_n" -gt 0 || {
printf 'FAIL: no committed context found in %s. Either its required_status_checks block is commented out, or REPO_ROOT points somewhere else.\n' "$GITHUB_TF" >&2
exit 1
}
test "$ctx_n" = "$blk_n" || {
printf 'FAIL: %s declares %s context(s) across %s required_check block(s). Part of the block is disabled, so the expectation is short.\n' "$GITHUB_TF" "$ctx_n" "$blk_n" >&2
exit 1
}
# 7. Set equality, both sides sorted. Reachable only after all three assertions passed.
diff -u "$EXPECTED_CONTEXTS" "$LIVE_CONTEXTS" || {
printf 'FAIL: the live required set is not the committed set. The restore did not land.\n' >&2
exit 1
}
printf 'RESTORED: required set equals the committed set (%s contexts)\n' "$ctx_n"
Terminal window
# 8. And the whole rules array, not only the context list. The object you restored must be
# the object you started from.
diff -u <(jq -S '.rules' /tmp/ruleset-original.json) \
<(gh api repos/fzymgc-house/selfhosted-cluster/rulesets/<id> --jq '.rules' | jq -S '.')

Step 7 is a sorted diff rather than a presence test, because a diff of two empty files is green, and an empty required set is the one state this check most needs to catch. The three assertions in step 6 abort, so a guard that prints a failure and carries on cannot print the RESTORED: line over an unprotected main. The cardinality check in step 5b catches the narrower version of the same trap: a partially disabled block in the HCL yields a short expectation, and a short live set matches it. The procedure runs from /tmp, which is why every repository path is anchored on REPO_ROOT.

The next main-cluster-core-services apply also restores the set from the HCL. The manual restore exists so that the gap is minutes rather than the time until the next merge.

The forbidden reflex: deleting the required_status_checks rule.

Do not remove the rule object from the ruleset. That removes the gate for every pull request indefinitely, and nothing reports that it is gone. Dropping to zero contexts and deleting the rule are different operations, and only the first is recoverable by inspection. Path 2 delivers the narrow version of the same relief, restores in four commands, and leaves a trace the drift detector finds.

Symptom Check Fix
A plan fails on a pull request The plan output in the pull request and the run log in HCP Terraform Fix the module and push
An apply fails after a merge The run log, then the resource state Fix and re-run the workspace from the UI. A state problem needs a hand-run from a workstation
A run stays queued kubectl -n hcp-terraform get pods, then kubectl -n hcp-terraform get externalsecret,agentpool The agent pod, the operator and its token. vault kv get -field=api_token secret/fzymgc-house/cluster/hcp-terraform must return a value
role not found at the Vault read vault read auth/jwt-hcp-terraform/role/tfc-<tag> Add the role to tf/vault/jwt-hcp-terraform.tf
claim not in bound_claims vault read -field=bound_claims auth/jwt-hcp-terraform/role/tfc-<tag> The sub claim must name the workspace exactly
permission denied at a Vault read vault policy read <policy> Add the path to the workspace’s policy in tf/vault/policy-terraform-workspaces.tf
The Discord embed does not arrive The Worker logs in Cloudflare, then tf/hcp-terraform/notifications.tf Rotate the HMAC token or the webhook URL and bump the notification, see Vault