HCP Terraform Operations¶

Operational guide for Terraform execution via HCP Terraform self-hosted agents.

Quick Reference¶

Property	Value
Organization	`fzymgc-house`
VCS Connection	GitHub
Agent Namespace	`hcp-terraform`
Agent Pool	`fzymgc-house-k8s`
Terraform Module	`tf/hcp-terraform/`

Architecture¶

GitHub PR -> HCP Terraform -> Agent Pod -> Dynamic Credentials -> Terraform Apply
                |                              (JWT -> Vault Token)
        Cloudflare Worker -> Discord

Workspaces¶

Workspace	Directory	Purpose	Trigger
vault	tf/vault	Vault configuration, policies, auth	PR merge
authentik	tf/authentik	Authentik SSO configuration	PR merge
grafana	tf/grafana	Grafana dashboards and config	PR merge
cloudflare	tf/cloudflare	DNS, tunnel, Workers configuration	PR merge
core-services	tf/core-services	Core K8s service configuration	PR merge
cluster-bootstrap	tf/cluster-bootstrap	Initial cluster infrastructure	Manual
hcp-terraform	tf/hcp-terraform	Self-managed workspace configuration	Manual

Local-Only Workspaces¶

Two workspaces cannot use the HCP TF agent and must run locally:

Workspace	Reason
`cluster-bootstrap`	Deploys the HCP Terraform Operator itself
`hcp-terraform`	Manages the agent pool configuration

Both have circular dependencies - they manage infrastructure that the agent depends on.

Solution: Run these workspaces locally with VAULT_TOKEN:

export VAULT_TOKEN=$(vault token create -field=token -policy=terraform-WORKSPACE-admin)
terraform -chdir=tf/WORKSPACE apply

Workflow¶

Standard Changes¶

Create feature branch
Edit Terraform files in tf/
Create PR - triggers speculative plan
Review plan output in PR comments
Merge PR - triggers apply

Manual Runs¶

For workspaces requiring manual trigger:

Log into HCP Terraform
Navigate to workspace
Click "Start new run"
Select "Plan and apply"

Vault Authentication¶

Workspaces authenticate to Vault using Dynamic Provider Credentials:

HCP TF Run Start
    |
    v
HCP TF generates signed JWT (workload identity)
    |
    v
HCP TF exchanges JWT for Vault token (internally)
    |
    v
Writes token to file, injects tfc_vault_dynamic_credentials variable
    |
    v
Vault provider reads token via auth_login_token_file

Environment Variables¶

Environment Variable	Value	Purpose
`TFC_VAULT_PROVIDER_AUTH`	`true`	Enable dynamic credentials
`TFC_VAULT_ADDR`	`https://vault.fzymgc.house`	Vault server address
`TFC_VAULT_AUTH_PATH`	`jwt-hcp-terraform`	JWT auth backend path
`TFC_VAULT_RUN_ROLE`	`tfc-<workspace>`	Per-workspace Vault role
`TFC_VAULT_ENCODED_CACERT`	Base64-encoded CA chain	Verify Vault's TLS certificate

Vault Resources¶

JWT auth backend: jwt-hcp-terraform
Per-workspace roles: tfc-vault, tfc-authentik, tfc-grafana, tfc-cloudflare, tfc-core-services
Policies: Grant least-privilege access per workspace

Agent Deployment¶

The HCP Terraform Operator manages agent pods:

Namespace: hcp-terraform
Operator: HashiCorp HCP Terraform Operator (Helm)
Agent Pool CRD: fzymgc-house-agents
Token: Stored in Vault at secret/fzymgc-house/cluster/hcp-terraform

Discord Notifications¶

Cloudflare Worker transforms HCP Terraform webhooks to Discord embeds.

Component	Location
Worker code	`cloudflare/workers/hcp-terraform-discord/`
Secret management	`tf/cloudflare/workers.tf`
Webhook URL	Vault: `secret/fzymgc-house/infrastructure/cloudflare/discord-webhook`

Common Operations¶

View Run Status¶

Check PR comments for plan output or view in HCP dashboard.

Cancel Run¶

In HCP dashboard, navigate to run and click "Cancel".

Agent Token Rotation¶

The agent token (tfe_agent_token) doesn't auto-rotate. To rotate manually:

# 1. Taint the token resource to force recreation
terraform -chdir=tf/hcp-terraform taint tfe_agent_token.k8s

# 2. Apply to generate new token
terraform -chdir=tf/hcp-terraform apply

# 3. Update Vault secret with new token
vault kv put secret/fzymgc-house/cluster/hcp-terraform \
  agent_token="$(terraform -chdir=tf/hcp-terraform output -raw agent_token)"

# 4. ExternalSecret will sync automatically, restart agent if needed
kubectl -n hcp-terraform rollout restart deployment/fzymgc-house-agents

Troubleshooting¶

Plan Failures¶

Check error message in PR comment
Review Terraform logs in HCP
Fix code and push update

Apply Failures¶

Review apply logs
Check resource state
May need manual intervention for state issues

Agent Not Connecting¶

Check agent pod status: kubectl -n hcp-terraform get pods
Check ExternalSecret: kubectl -n hcp-terraform get externalsecret
Verify Vault secret exists: vault kv get secret/fzymgc-house/cluster/hcp-terraform

Dynamic Credentials / OIDC Failures¶

Error	Cause	Fix
`role not found`	Missing JWT role in Vault	Create role: `vault write auth/jwt-hcp-terraform/role/tfc-WORKSPACE ...`
`claim not in bound_claims`	Workspace name mismatch	Check `bound_claims_value` matches HCP TF workspace name
`token expired`	JWT past validity window	Verify clocks are synced; tokens valid 5 minutes
`permission denied`	Policy missing capabilities	Check policy grants access to required paths

Debug OIDC authentication:

# Verify JWT auth backend exists
vault auth list | grep jwt-hcp-terraform

# Check role configuration
vault read auth/jwt-hcp-terraform/role/tfc-WORKSPACE

# Verify bound claims (must match HCP TF workspace exactly)
vault read -field=bound_claims auth/jwt-hcp-terraform/role/tfc-WORKSPACE

# Test policy access
vault policy read terraform-WORKSPACE-admin

Rollback to Windmill¶

If HCP Terraform fails post-migration:

Scale down operator: kubectl -n hcp-terraform scale deployment hcp-terraform-operator --replicas=0
Re-enable Windmill flows in windmill/f/terraform/
Remove HCP TF workspace variables in HCP TF UI (restore local execution)

Migration History¶

HCP Terraform replaced Windmill for Terraform execution (December 2025).

Document	Location
Design	`docs/plans/archive/migrations/2025-12-26-hcp-terraform-migration-design.md`
Implementation	`docs/plans/archive/migrations/2025-12-26-hcp-terraform-migration-implementation.md`