Vault Operations

Operational guide for HashiCorp Vault secrets management in the fzymgc-house cluster.

Quick Reference

Property	Value
URL	`https://vault.fzymgc.house`
Auth Methods	OIDC (Keycloak), Token
Storage Backend	Integrated Storage (Raft)
Terraform Module	`tf/vault/`
Helper Script	`./scripts/vault-helper.sh`

Secret Structure

All infrastructure secrets stored under secret/fzymgc-house/:

Path	Purpose
`infrastructure/bmc/tpi-alpha`	TuringPi Alpha BMC credentials
`infrastructure/bmc/tpi-beta`	TuringPi Beta BMC credentials
`infrastructure/cloudflare/api-token`	Cloudflare API token
`cluster/keycloak`	Keycloak Terraform/bootstrap admin credentials
`cluster/hcp-terraform`	HCP Terraform agent token

Authentication

export VAULT_ADDR=https://vault.fzymgc.house

# OIDC login (browser-based)
vault login -method=oidc

# Token login
vault login

Common Operations

Read a Secret

vault kv get secret/fzymgc-house/cluster/keycloak

# Get single field
vault kv get -field=terraform_password secret/fzymgc-house/cluster/keycloak

List Secrets

vault kv list secret/fzymgc-house/cluster/

Write a Secret

vault kv put secret/fzymgc-house/cluster/example key=value

Helper Script Operations

# Check connectivity
./scripts/vault-helper.sh status

# List infrastructure secrets
./scripts/vault-helper.sh list

# Get specific secret
./scripts/vault-helper.sh get bmc/tpi-alpha

# Get single field
./scripts/vault-helper.sh get bmc/tpi-alpha password

Integration Examples

Ansible Integration

# Vault lookup in group_vars
tpi_bmc_password: "{{ lookup('community.hashi_vault.vault_kv2_get', 'infrastructure/bmc/tpi-alpha', engine_mount_point='secret/fzymgc-house').secret.password }}"

cloudflare_api_token: "{{ lookup('community.hashi_vault.vault_kv2_get', 'infrastructure/cloudflare/api-token', engine_mount_point='secret/fzymgc-house').secret.token }}"

For Ansible modules that access Vault directly, ensure the CA bundle is set so Python requests can validate TLS. Configure vault_ca_cert_bundle in ansible/inventory/group_vars/all.yml or set VAULT_CACERT.

Requires community.hashi_vault collection (installed via requirements).

Terraform Integration

data "vault_kv_secret_v2" "cloudflare" {
  mount = "secret/fzymgc-house"
  name  = "infrastructure/cloudflare/api-token"
}

provider "cloudflare" {
  api_token = data.vault_kv_secret_v2.cloudflare.data["token"]
}

Secret Rotation

Manual Rotation

Generate new credential
Update Vault secret
Restart affected pods to pick up changes

Automatic Rotation

Some secrets support automatic rotation via Vault policies.

Rotating secrets consumed via write-only (`*_wo`) attributes

Some Terraform sinks read Vault secrets ephemerally and write them through write-only attributes (kubernetes_secret_v1.data_wo, tfe_notification_configuration.token_wo, …; hl-106s, ADR hl-ujwi). Terraform cannot diff write-only content, so rotating the Vault secret does not propagate by itself (exception: the tfe notification url/token use the provider’s auto-hash mode and re-send automatically).

After rotating such a secret in Vault:

Find the consuming resource (grep the workspace for data_wo / ephemeral "vault_kv_secret_v2").
Increment its data_wo_revision by exactly 1 (helm’s set_wo_revision must strictly increase; kubernetes accepts any change).
Plan/apply the workspace (main-cluster-bootstrap is Local execution — apply manually).
Verify the in-cluster value changed (kubectl get secret ... -o json | jq -S '.data' | sha256sum).

PKI Certificate Rotation

Kubernetes Workloads (VaultDynamicSecret)

Kubernetes workloads using VaultDynamicSecret automatically rotate PKI certificates:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: router-hosts-client-cert
spec:
  mount: fzymgc-house/v1/ica1/v1
  path: issue/router-hosts-client
  destination:
    name: router-hosts-client-cert
    create: true
  renewalPercent: 67

The operator renews certificates at 67% of TTL automatically.

CLI Client Certificates

For CLI tools (like router-hosts), certificates must be manually renewed:

# Check certificate expiry
openssl x509 -in ~/.config/router-hosts/tls.crt -text -noout | grep -A2 Validity

# Re-issue certificate (Bash/Zsh)
CERT_DATA=$(vault write -format=json fzymgc-house/v1/ica1/v1/issue/router-hosts-client \
  common_name="cli-$(whoami)" ttl=720h)
echo "$CERT_DATA" | jq -r '.data.certificate' > ~/.config/router-hosts/tls.crt
echo "$CERT_DATA" | jq -r '.data.private_key' > ~/.config/router-hosts/tls.key
echo "$CERT_DATA" | jq -r '.data.issuing_ca' > ~/.config/router-hosts/ca.crt

Available PKI Roles

Role	Mount Path	Purpose	Default TTL
`router-hosts-client`	`fzymgc-house/v1/ica1/v1`	gRPC client auth	720h
`router-hosts-server`	`fzymgc-house/v1/ica1/v1`	gRPC server cert	720h

Required Policy

Developers need infrastructure-developer policy. Defined in tf/vault/policy-infrastructure-developer.hcl:

path "secret/data/fzymgc-house/infrastructure/*" {
  capabilities = ["read", "list"]
}
path "secret/metadata/fzymgc-house/infrastructure/*" {
  capabilities = ["list"]
}
path "secret/data/fzymgc-house/*" {
  capabilities = ["read", "list"]
}
path "secret/metadata/fzymgc-house/*" {
  capabilities = ["list"]
}

Troubleshooting

Vault Sealed

If Vault is sealed, unseal keys are required. Contact cluster admin.

Permission Denied

Check your Vault policy assignments in Keycloak groups.

Connectivity Issues

curl -s https://vault.fzymgc.house/v1/sys/health
vault token lookup

Ansible Secret Issues

ansible-galaxy collection list | grep hashi_vault
ansible localhost -m debug -a "msg={{ lookup('community.hashi_vault.vault_kv2_get', 'infrastructure/bmc/tpi-alpha', engine_mount_point='secret/fzymgc-house').secret.password }}"

Terraform Issues

vault token lookup
terraform console
> data.vault_kv_secret_v2.keycloak.data