HCP Terraform Operations

Operational guide for Terraform execution via HCP Terraform self-hosted agents.

Quick Reference

Property	Value
Organization	fzymgc-house
VCS Connection	GitHub
Agent Namespace	hcp-terraform
Agent Pool	fzymgc-house-k8s
Terraform Module	tf/hcp-terraform/

Architecture

GitHub PR -> HCP Terraform -> Agent Pod -> Dynamic Credentials -> Terraform Apply
                |                              (JWT -> Vault Token)
        Cloudflare Worker -> Discord

Workspaces

HCP TFC Workspace	Directory	Purpose	Trigger
main-cluster-vault	tf/vault	Vault configuration, policies, auth	PR merge
main-cluster-grafana	tf/grafana	Grafana dashboards and config	PR merge
main-cluster-cloudflare	tf/cloudflare	DNS, tunnel, Workers, Access	PR merge
main-cluster-core-services	tf/core-services	Core K8s service configuration	PR merge
main-cluster-router-hosts	tf/router-hosts	DNS host entries via router-hosts	PR merge
main-cluster-bootstrap	tf/cluster-bootstrap	Initial cluster infrastructure	Manual
hcp-terraform	tf/hcp-terraform	Self-managed workspace configuration	Manual

Note: When using MCP Terraform tools or the TFC API, use the full workspace name.

Local-Only Workspaces

Two workspaces cannot use the HCP TF agent and must run locally:

HCP TFC Workspace	Reason
main-cluster-bootstrap	Deploys the HCP Terraform Operator itself
hcp-terraform	Manages the agent pool configuration

Both have circular dependencies - they manage infrastructure that the agent depends on.

Solution: Run these workspaces locally with VAULT_TOKEN:

export VAULT_TOKEN=$(vault token create -field=token -policy=terraform-WORKSPACE-admin)
terraform -chdir=tf/WORKSPACE apply

Workspace Dependencies

Some workspaces depend on secrets created by other workspaces:

Consumer Workspace	Producer Workspace	Secret Path	Notes
main-cluster-cloudflare	main-cluster-keycloak	secret/fzymgc-house/cluster/cloudflare-access	OIDC credentials for Cloudflare Access
main-cluster-grafana	main-cluster-keycloak	secret/fzymgc-house/cluster/grafana	OIDC credentials for Grafana SSO

Impact on PRs: When a PR touches both producer and consumer workspaces, the consumer’s plan may fail until the producer’s changes are merged and applied.

Resolution: Merge the PR - workspaces will apply in commit order. If main-cluster-cloudflare fails post-merge, re-run it after main-cluster-keycloak completes.

Workflow

Standard Changes

1. Create feature branch
2. Edit Terraform files in tf/
3. Create PR - triggers speculative plan
4. Review plan output in PR comments
5. Merge PR - triggers apply

Provider Version Bumps

Updating provider version constraints in a module’s versions.tf (and regenerating .terraform.lock.hcl) changes the providers HCP installs and can introduce schema or behavior changes. Because most workspaces have auto_apply = true, review each affected workspace’s speculative plan on the PR before merging — especially production-facing modules (tf/cloudflare DNS, tf/keycloak SSO, tf/grafana dashboards), where a minor bump can cause drift that applies on merge. If a plan shows unexpected changes, hold the merge and pin to the prior version or address the drift first.

Regenerate lock files for both the HCP agent platform and local use:

terraform -chdir=tf/<module> providers lock \
  -platform=linux_amd64 -platform=darwin_arm64

Manual Runs

For workspaces requiring manual trigger:

1. Log into HCP Terraform
2. Navigate to workspace
3. Click “Start new run”
4. Select “Plan and apply”

Vault Authentication

Workspaces authenticate to Vault using Dynamic Provider Credentials:

HCP TF Run Start
    |
    v
HCP TF generates signed JWT (workload identity)
    |
    v
HCP TF exchanges JWT for Vault token (internally)
    |
    v
Writes token to file, injects tfc_vault_dynamic_credentials variable
    |
    v
Vault provider reads token via auth_login_token_file

Environment Variables

Environment Variable	Value	Purpose
TFC_VAULT_PROVIDER_AUTH	true	Enable dynamic credentials
TFC_VAULT_ADDR	https://vault.fzymgc.house	Vault server address
TFC_VAULT_AUTH_PATH	jwt-hcp-terraform	JWT auth backend path
TFC_VAULT_RUN_ROLE	tfc-<workspace>	Per-workspace Vault role
TFC_VAULT_ENCODED_CACERT	Base64-encoded CA chain	Verify Vault’s TLS certificate

Vault Resources

• JWT auth backend: jwt-hcp-terraform
• Per-workspace roles: tfc-vault, tfc-keycloak, tfc-grafana, tfc-cloudflare, tfc-core-services
• Policies: Grant least-privilege access per workspace

Agent Deployment

The HCP Terraform Operator manages agent pods:

• Namespace: hcp-terraform
• Operator: HashiCorp HCP Terraform Operator (Helm)
• Agent Pool CRD: fzymgc-house-agents
• Token: Stored in Vault at secret/fzymgc-house/cluster/hcp-terraform

Discord Notifications

Cloudflare Worker transforms HCP Terraform webhooks to Discord embeds.

Component	Location
Worker code	cloudflare/workers/hcp-terraform-discord/
Secret management	tf/cloudflare/workers.tf
Webhook URL	Vault: secret/fzymgc-house/infrastructure/cloudflare/discord-webhook

Common Operations

View Run Status

Check PR comments for plan output or view in HCP dashboard.

Cancel Run

In HCP dashboard, navigate to run and click “Cancel”.

Agent Token Rotation

The agent token (tfe_agent_token) doesn’t auto-rotate. To rotate manually:

# 1. Taint the token resource to force recreation
terraform -chdir=tf/hcp-terraform taint tfe_agent_token.k8s

# 2. Apply to generate new token
terraform -chdir=tf/hcp-terraform apply

# 3. Update Vault secret with new token
vault kv put secret/fzymgc-house/cluster/hcp-terraform \
  agent_token="$(terraform -chdir=tf/hcp-terraform output -raw agent_token)"

# 4. ExternalSecret will sync automatically, restart agent if needed
kubectl -n hcp-terraform rollout restart deployment/fzymgc-house-agents

Router-Hosts Certificate Renewal

The main-cluster-router-hosts workspace requires mTLS certificates to authenticate to the router-hosts gRPC server. These certificates have a 90-day TTL and must be renewed before expiration.

Architecture:

1. hcp-terraform workspace issues certs from Vault PKI
2. Certs are stored as sensitive TFC workspace variables
3. router-hosts workspace reads certs at plan/apply time

Manual Renewal:

# Run hcp-terraform locally to reissue certs
export VAULT_TOKEN=$(vault token create -field=token -policy=terraform-hcp-terraform-admin)
terraform -chdir=tf/hcp-terraform apply

# This updates TFC variables automatically - no restart needed
# Next router-hosts run will use new certs

Certificate Location:

• Issuance: tf/hcp-terraform/router-hosts-cert.tf
• TFC Variables: router_hosts_client_cert, router_hosts_client_key, router_hosts_ca_cert
• Consumer: tf/router-hosts/terraform.tf (provider config)

Troubleshooting

Plan Failures

1. Check error message in PR comment
2. Review Terraform logs in HCP
3. Fix code and push update

Apply Failures

1. Review apply logs
2. Check resource state
3. May need manual intervention for state issues

Agent Not Connecting

1. Check agent pod status: kubectl -n hcp-terraform get pods
2. Check ExternalSecret: kubectl -n hcp-terraform get externalsecret
3. Verify Vault secret exists: vault kv get secret/fzymgc-house/cluster/hcp-terraform

Dynamic Credentials / OIDC Failures

Error	Cause	Fix
role not found	Missing JWT role in Vault	Create role: vault write auth/jwt-hcp-terraform/role/tfc-WORKSPACE ...
claim not in bound_claims	Workspace name mismatch	Check bound_claims_value matches HCP TF workspace name
token expired	JWT past validity window	Verify clocks are synced; tokens valid 5 minutes
permission denied	Policy missing capabilities	Check policy grants access to required paths

Debug OIDC authentication:

# Verify JWT auth backend exists
vault auth list | grep jwt-hcp-terraform

# Check role configuration
vault read auth/jwt-hcp-terraform/role/tfc-WORKSPACE

# Verify bound claims (must match HCP TF workspace exactly)
vault read -field=bound_claims auth/jwt-hcp-terraform/role/tfc-WORKSPACE

# Test policy access
vault policy read terraform-WORKSPACE-admin

Migration History

HCP Terraform replaced Windmill for Terraform execution (December 2025).

Document	Location
Design	2025-12-26-hcp-terraform-migration-design.md (pruned from the docs site; see VCS history for docs/plans/archive/migrations/)
Implementation	2025-12-26-hcp-terraform-migration-implementation.md (pruned; see VCS history)

See Also

• Vault Operations - Dynamic credentials setup
• Cloudflare Operations - Discord notifications
• Agent config: argocd/app-configs/hcp-terraform-operator/