Skip to content
fzymgc-house Cluster Docs

HCP Terraform Operations

Operational guide for Terraform execution via HCP Terraform self-hosted agents.

Quick Reference

Section titled “Quick Reference”
Property Value
Organization fzymgc-house
VCS Connection GitHub
Agent Namespace hcp-terraform
Agent Pool fzymgc-house-k8s
Terraform Module tf/hcp-terraform/

Architecture

Section titled “Architecture”
GitHub PR -> HCP Terraform -> Agent Pod -> Dynamic Credentials -> Terraform Apply
                |                              (JWT -> Vault Token)
        Cloudflare Worker -> Discord

Workspaces

Section titled “Workspaces”
HCP TFC Workspace Directory Purpose Trigger
main-cluster-vault tf/vault Vault configuration, policies, auth PR merge
main-cluster-grafana tf/grafana Grafana dashboards and config PR merge
main-cluster-cloudflare tf/cloudflare DNS, tunnel, Workers, Access PR merge
main-cluster-core-services tf/core-services Core K8s service configuration PR merge + daily scheduled drift detection
main-cluster-bootstrap tf/cluster-bootstrap Initial cluster infrastructure Manual
hcp-terraform tf/hcp-terraform Self-managed workspace configuration Manual

Note: When using MCP Terraform tools or the TFC API, use the full workspace name.

Drift detection enrolment is configured in argocd/cluster-app/templates/temporal-workers.yaml under the terraform worker’s workspaces list, which notifies Discord #system-notifications.

Local-Only Workspaces

Section titled “Local-Only Workspaces”

Two workspaces cannot use the HCP TF agent and must run locally:

HCP TFC Workspace Reason
main-cluster-bootstrap Deploys the HCP Terraform Operator itself
hcp-terraform Manages the agent pool configuration

Both have circular dependencies - they manage infrastructure that the agent depends on.

Solution: Run these workspaces locally with VAULT_TOKEN:

Terminal window
export VAULT_TOKEN=$(vault token create -field=token -policy=terraform-WORKSPACE-admin)
terraform -chdir=tf/WORKSPACE apply

Workspace Dependencies

Section titled “Workspace Dependencies”

Some workspaces depend on secrets created by other workspaces:

Consumer Workspace Producer Workspace Secret Path Notes
main-cluster-cloudflare main-cluster-keycloak secret/fzymgc-house/cluster/cloudflare-access OIDC credentials for Cloudflare Access
main-cluster-grafana main-cluster-keycloak secret/fzymgc-house/cluster/grafana OIDC credentials for Grafana SSO

Impact on PRs: When a PR touches both producer and consumer workspaces, the consumer’s plan may fail until the producer’s changes are merged and applied.

Resolution: Merge the PR - workspaces will apply in commit order. If main-cluster-cloudflare fails post-merge, re-run it after main-cluster-keycloak completes.

Workflow

Section titled “Workflow”

Standard Changes

Section titled “Standard Changes”
  1. Create feature branch
  2. Edit Terraform files in tf/
  3. Create PR - triggers speculative plan
  4. Review plan output in PR comments
  5. Merge PR - triggers apply

Provider Version Bumps

Section titled “Provider Version Bumps”

Updating provider version constraints in a module’s versions.tf (and regenerating .terraform.lock.hcl) changes the providers HCP installs and can introduce schema or behavior changes. Because most workspaces have auto_apply = true, review each affected workspace’s speculative plan on the PR before merging — especially production-facing modules (tf/cloudflare DNS, tf/keycloak SSO, tf/grafana dashboards), where a minor bump can cause drift that applies on merge. If a plan shows unexpected changes, hold the merge and pin to the prior version or address the drift first.

Regenerate lock files for both the HCP agent platform and local use:

Terminal window
terraform -chdir=tf/<module> providers lock \
  -platform=linux_amd64 -platform=darwin_arm64

Manual Runs

Section titled “Manual Runs”

For workspaces requiring manual trigger:

  1. Log into HCP Terraform
  2. Navigate to workspace
  3. Click “Start new run”
  4. Select “Plan and apply”

Vault Authentication

Section titled “Vault Authentication”

Workspaces authenticate to Vault using Dynamic Provider Credentials:

HCP TF Run Start
    |
    v
HCP TF generates signed JWT (workload identity)
    |
    v
HCP TF exchanges JWT for Vault token (internally)
    |
    v
Writes token to file, injects tfc_vault_dynamic_credentials variable
    |
    v
Vault provider reads token via auth_login_token_file

Environment Variables

Section titled “Environment Variables”
Environment Variable Value Purpose
TFC_VAULT_PROVIDER_AUTH true Enable dynamic credentials
TFC_VAULT_ADDR https://vault.fzymgc.house Vault server address
TFC_VAULT_AUTH_PATH jwt-hcp-terraform JWT auth backend path
TFC_VAULT_RUN_ROLE tfc-<workspace> Per-workspace Vault role
TFC_VAULT_ENCODED_CACERT Base64-encoded CA chain Verify Vault’s TLS certificate

Vault Resources

Section titled “Vault Resources”
  • JWT auth backend: jwt-hcp-terraform
  • Per-workspace roles: tfc-vault, tfc-keycloak, tfc-grafana, tfc-cloudflare, tfc-core-services
  • Policies: Grant least-privilege access per workspace

Agent Deployment

Section titled “Agent Deployment”

The HCP Terraform Operator manages agent pods:

  • Namespace: hcp-terraform
  • Operator: HashiCorp HCP Terraform Operator (Helm)
  • Agent Pool CRD: fzymgc-house-agents
  • Token: Stored in Vault at secret/fzymgc-house/cluster/hcp-terraform

Discord Notifications

Section titled “Discord Notifications”

Cloudflare Worker transforms HCP Terraform webhooks to Discord embeds.

Component Location
Worker code cloudflare/workers/hcp-terraform-discord/
Secret management tf/cloudflare/workers.tf
Webhook URL Vault: secret/fzymgc-house/infrastructure/cloudflare/discord-webhook

Common Operations

Section titled “Common Operations”

View Run Status

Section titled “View Run Status”

Check PR comments for plan output or view in HCP dashboard.

Cancel Run

Section titled “Cancel Run”

In HCP dashboard, navigate to run and click “Cancel”.

Agent Token Rotation

Section titled “Agent Token Rotation”

The agent token (tfe_agent_token) doesn’t auto-rotate. To rotate manually:

Terminal window
# 1. Taint the token resource to force recreation
terraform -chdir=tf/hcp-terraform taint tfe_agent_token.k8s


# 2. Apply to generate new token
terraform -chdir=tf/hcp-terraform apply


# 3. Update Vault secret with new token
vault kv put secret/fzymgc-house/cluster/hcp-terraform \
  agent_token="$(terraform -chdir=tf/hcp-terraform output -raw agent_token)"


# 4. ExternalSecret will sync automatically, restart agent if needed
kubectl -n hcp-terraform rollout restart deployment/fzymgc-house-agents

Break-Glass: Merging Past a Blocked Required Check

Section titled “Break-Glass: Merging Past a Blocked Required Check”

main requires the pr-gates check context. It is declared in tf/core-services/github.tf and lives in GitHub ruleset 10553802. When it will not go green and a change has to land anyway, there are two sanctioned paths and one forbidden one. Work down the list — path 1 is narrower, needs no undo, and GitHub records it.

Path 1 — bypass the single pull request. This is the default.

tf/core-services/github.tf carries two bypass_actors blocks: one OrganizationAdmin, one RepositoryRole admin, each at bypass_mode = "pull_request". The grant is the union of those two role memberships, so whichever role you still hold at 2am is the one that works.

This path is first for three reasons: it affects exactly one pull request, it leaves the gate intact for every other pull request, and GitHub writes an audit entry for the bypass. A bypassed gate that is visible is the whole point of the mechanism.

The move is just the merge. The bypass applies because of role membership, so there is no flag to pass, no ruleset field to change, and nothing to put back afterwards:

Terminal window
gh pr merge <number> --squash

No ruleset mutation happens on this path. creation, deletion and non_fast_forward stay fully enforced against direct pushes, because pull_request mode scopes the bypass to the pull-request merge path and to nothing else.

Path 2 — drop a single required context. This is the escalation.

Use this only when a context is structurally broken for every pull request — the workflow cannot start, or the job name no longer matches the required string — so bypassing per pull request does not scale. Its cost, stated plainly: it unblocks everyone, including pull requests nobody vetted, and it is invisible until the daily drift detection run on main-cluster-core-services next reports.

Terminal window
# 1. Read the WHOLE ruleset object, unfiltered. Keep this copy untouched: it is both the
#    input to the edit and the reference the restore compares against.
gh api repos/fzymgc-house/selfhosted-cluster/rulesets/10553802 > /tmp/ruleset-original.json


# 2. Remove exactly ONE entry from the required_status_checks context array, selected by its
#    context string. Every other rule and every other field is carried through untouched.
jq --arg ctx pr-gates '
  (.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks)
    |= map(select(.context != $ctx))
' /tmp/ruleset-original.json > /tmp/ruleset-dropped.json


# 3. PUT the WHOLE edited body back, from the file. Never assemble this body by hand.
gh api -X PUT repos/fzymgc-house/selfhosted-cluster/rulesets/10553802 \
  --input /tmp/ruleset-dropped.json > /dev/null


# 4. Read back what is still required.
gh api repos/fzymgc-house/selfhosted-cluster/rulesets/10553802 \
  --jq '.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks[].context'

The read-modify-write is a correctness requirement, not a style preference. PUT /repos/{owner}/{repo}/rulesets/{id} has no partial-update form: it replaces the entire object, so a hand-typed body carrying only required_status_checks silently deletes the pull_request, creation, deletion and non_fast_forward rules along with it. Feeding the whole object back through --input is what stops that. The round trip also preserves require_extra_approval_for_unattributed_changes, which the Terraform provider does not model and therefore omits from its own writes — so in that one respect the jq path is safer than an apply.

Path 2 — restore. Run the same read-modify-write in the opposite direction, then prove the required set came back:

Terminal window
# 1. Read the whole object again.
gh api repos/fzymgc-house/selfhosted-cluster/rulesets/10553802 > /tmp/ruleset-current.json


# 2. Add the context back. Same shape as the drop, opposite operation.
jq --arg ctx pr-gates '
  (.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks)
    |= (. + [{context: $ctx}] | unique_by(.context))
' /tmp/ruleset-current.json > /tmp/ruleset-restored.json


# 3. PUT the whole restored body.
gh api -X PUT repos/fzymgc-house/selfhosted-cluster/rulesets/10553802 \
  --input /tmp/ruleset-restored.json > /dev/null


# 4. Side A — what the live ruleset requires now.
gh api repos/fzymgc-house/selfhosted-cluster/rulesets/10553802 \
  --jq '.rules[] | select(.type == "required_status_checks") | .parameters.required_status_checks[].context' \
  | sort > /tmp/live-contexts.txt
Terminal window
set -euo pipefail


# 5. Side B — what tf/core-services/github.tf commits. Read it from the HCL rather than
#    retyping it, so there is no second copy of the expected set to drift. This procedure
#    runs from /tmp, so every repository path below is anchored on REPO_ROOT.
: "${REPO_ROOT:?point REPO_ROOT at your selfhosted-cluster checkout before running this block}"
LIVE_CONTEXTS="${LIVE_CONTEXTS:-/tmp/live-contexts.txt}"
EXPECTED_CONTEXTS="${EXPECTED_CONTEXTS:-/tmp/expected-contexts.txt}"
GITHUB_TF="$REPO_ROOT/tf/core-services/github.tf"


{ rg -o -e '^[[:space:]]*context[[:space:]]*=[[:space:]]*"[^"]+"' "$GITHUB_TF" || true; } \
  | { rg -o -e '"[^"]+"$' || true; } | tr -d '"' | sort > "$EXPECTED_CONTEXTS"


# 5b. Cardinality cross-check. Both numbers are read from that same file, so nothing here
#     hard-codes the size of the required set, and a partially disabled block cannot pass
#     off a short expectation as the whole one.
ctx_n="$( { rg -o -e '^[[:space:]]*context[[:space:]]*=[[:space:]]*"[^"]+"' "$GITHUB_TF" || true; } | wc -l | tr -d ' ')"
blk_n="$( { rg -o -e '^[[:space:]]*required_check[[:space:]]*\{' "$GITHUB_TF" || true; } | wc -l | tr -d ' ')"
live_n=0
if [ -s "$LIVE_CONTEXTS" ]; then
  live_n="$(wc -l < "$LIVE_CONTEXTS" | tr -d ' ')"
fi


# 6. Assert, BEFORE comparing. Each of these ABORTS: nothing printed below a failure here
#    may be read as success.
test "$live_n" -gt 0 || {
  printf 'FAIL: the live required set is EMPTY — main is unprotected. Nothing printed below this line is success.\n' >&2
  exit 1
}
test "$ctx_n" -gt 0 || {
  printf 'FAIL: no committed context found in %s — either its required_status_checks block is commented out, or REPO_ROOT points somewhere else.\n' "$GITHUB_TF" >&2
  exit 1
}
test "$ctx_n" = "$blk_n" || {
  printf 'FAIL: %s declares %s context(s) across %s required_check block(s) — part of the block is disabled, so the expectation is short.\n' "$GITHUB_TF" "$ctx_n" "$blk_n" >&2
  exit 1
}


# 7. Set equality, both sides sorted. Reachable only after all three assertions passed.
diff -u "$EXPECTED_CONTEXTS" "$LIVE_CONTEXTS" || {
  printf 'FAIL: the live required set is not the committed set — the restore did not land.\n' >&2
  exit 1
}
printf 'RESTORED: required set equals the committed set (%s contexts)\n' "$ctx_n"
Terminal window
# 8. And the whole rules array, not only the context list — the object you restored must be
#    the object you started from.
diff -u <(jq -S '.rules' /tmp/ruleset-original.json) \
  <(gh api repos/fzymgc-house/selfhosted-cluster/rulesets/10553802 --jq '.rules' | jq -S '.')

The comparison in step 7 is a sorted diff rather than a presence test because a diff of two empty files is green, and an empty required set is exactly this repository’s pre-enforcement value — so an unasserted comparison reports success on the one state it most needs to catch. The three assertions in step 6 each abort. That is a change made on 2026-09-02: the phase-4 verification reproduced the previous form, whose guards printed a failure line and carried on, printing the RESTORED: success line with both context files empty and exiting 0 against a completely unprotected main. The step-5b cardinality cross-check exists for the narrower version of the same trap: a partially disabled required_status_checks block in the HCL yields a short expectation, and a short live set would match it.

The procedure runs from /tmp, which is why every repository path is anchored on REPO_ROOT: a repo-relative read from /tmp was one of the two ways both sides of the comparison reached empty.

The next main-cluster-core-services apply also restores the set from the HCL. The manual restore exists so the gap is minutes rather than however long it is until the next merge.

The forbidden reflex: deleting the required_status_checks rule.

Do not remove the rule object from the ruleset. That is the move made on 2026-08-06, and it removed the gate for every pull request indefinitely with nothing reporting that it was gone. Dropping to zero contexts and deleting the rule are different operations, and only the first is recoverable by inspection.

Both paths above exist so this one is never the obvious option. If a situation seems to demand it, path 2 delivers the narrow version of the same relief, restores in four commands, and leaves a trace the drift detector can find.

Troubleshooting

Section titled “Troubleshooting”

Plan Failures

Section titled “Plan Failures”
  1. Check error message in PR comment
  2. Review Terraform logs in HCP
  3. Fix code and push update

Apply Failures

Section titled “Apply Failures”
  1. Review apply logs
  2. Check resource state
  3. May need manual intervention for state issues

Agent Not Connecting

Section titled “Agent Not Connecting”
  1. Check agent pod status: kubectl -n hcp-terraform get pods
  2. Check ExternalSecret: kubectl -n hcp-terraform get externalsecret
  3. Verify Vault secret exists: vault kv get secret/fzymgc-house/cluster/hcp-terraform

Dynamic Credentials / OIDC Failures

Section titled “Dynamic Credentials / OIDC Failures”
Error Cause Fix
role not found Missing JWT role in Vault Create role: vault write auth/jwt-hcp-terraform/role/tfc-WORKSPACE ...
claim not in bound_claims Workspace name mismatch Check bound_claims_value matches HCP TF workspace name
token expired JWT past validity window Verify clocks are synced; tokens valid 5 minutes
permission denied Policy missing capabilities Check policy grants access to required paths

Debug OIDC authentication:

Terminal window
# Verify JWT auth backend exists
vault auth list | grep jwt-hcp-terraform


# Check role configuration
vault read auth/jwt-hcp-terraform/role/tfc-WORKSPACE


# Verify bound claims (must match HCP TF workspace exactly)
vault read -field=bound_claims auth/jwt-hcp-terraform/role/tfc-WORKSPACE


# Test policy access
vault policy read terraform-WORKSPACE-admin

Migration History

Section titled “Migration History”

HCP Terraform replaced Windmill for Terraform execution (December 2025).

Document Location
Design 2025-12-26-hcp-terraform-migration-design.md (pruned from the docs site; see VCS history for docs/plans/archive/migrations/)
Implementation 2025-12-26-hcp-terraform-migration-implementation.md (pruned; see VCS history)

See Also

Section titled “See Also”