Vault Operations

Operational guide for HashiCorp Vault secrets management in the fzymgc-house cluster.

Quick Reference

Property	Value
URL	`https://vault.fzymgc.house`
Auth Methods	OIDC (Keycloak), Token
Storage Backend	Integrated Storage (Raft)
Terraform Module	`tf/vault/`
Helper Script	`./scripts/vault-helper.sh`

Secret Structure

All infrastructure secrets stored under secret/fzymgc-house/:

Path	Purpose
`infrastructure/bmc/tpi-alpha`	TuringPi Alpha BMC credentials
`infrastructure/bmc/tpi-beta`	TuringPi Beta BMC credentials
`infrastructure/cloudflare/api-token`	Cloudflare API token
`cluster/keycloak`	Keycloak Terraform/bootstrap admin credentials
`cluster/hcp-terraform`	HCP Terraform agent token
`hermes/jarvis`	The `jarvis` hermes agent profile on seattle (Mac mini). Read by Vault Agent through the `hermes-jarvis` AppRole. The operator creates the path with `vault kv put`

Authentication

export VAULT_ADDR=https://vault.fzymgc.house

# OIDC login (browser-based)
vault login -method=oidc

# Token login
vault login

Common Operations

Read a Secret

vault kv get secret/fzymgc-house/cluster/keycloak

# Get single field
vault kv get -field=terraform_password secret/fzymgc-house/cluster/keycloak

List Secrets

vault kv list secret/fzymgc-house/cluster/

Write a Secret

vault kv put secret/fzymgc-house/cluster/example key=value

Revoke a token you do not hold

Sometimes a token has to be neutralised that you cannot log in as — one left behind in a Kubernetes Secret, or one issued to a controller that no longer runs. Work through the accessor, never the token value. An accessor is a non-credential handle: vault token lookup -accessor and vault token revoke -accessor both operate on it, it cannot be used to authenticate, and it is the only handle safe to write into a runbook, a ticket or an evidence file. Record accessors; never record values.

export VAULT_ADDR=https://vault.fzymgc.house

# The identity needs a policy permitting auth/token/lookup-accessor and revoke-accessor.
vault login -method=oidc

# Read the value exactly once, straight into a variable. Never echo, log or `-o yaml` it.
TOKEN=$(kubectl -n <namespace> get secret <secret> -o jsonpath='{.data.<key>}' | base64 -d)

# -format=json includes .data.id — the token VALUE — which the default table output omits.
# The jq projection below is a control that strips it, not cosmetics: drop the projection
# and you have printed the credential.
vault token lookup -format=json "$TOKEN" \
  | jq '.data | {accessor, display_name, policies, entity_id, path, expire_time}'
unset TOKEN

# Irreversible. The accessor is enough; the value is never needed again.
vault token revoke -accessor <accessor>

# Proof the revoke took: this MUST fail with `Code: 400`. Measured live 2026-09-07 against
# this cluster, the whole of what was observed:
#   Code: 400.  (URL: POST https://vault.fzymgc.house/v1/auth/token/lookup-accessor)
# Only the status code is asserted, because only the status code was observed. Do not add an
# expected message body here until someone has actually seen one.
#
# MIND THE ENDPOINT — the two lookup forms are different endpoints with different failure
# modes, and their wordings are NOT interchangeable:
#   -accessor   -> POST auth/token/lookup-accessor  -> unknown accessor gives `Code: 400`
#   by value    -> POST auth/token/lookup           -> dead token gives `Code: 403 ... bad token`
# The `bad token` string belongs to the by-value path used in the pre-flight above. It has
# never been observed on the accessor path; do not expect it here.
vault token lookup -accessor <accessor>

# Pair that 400 with a control that MUST succeed, or it proves only that lookup is broken.
# Bare `vault token lookup` looks up your own token — self is the default and there is no
# -self flag; passing one is rejected as an undefined flag.
vault token lookup -format=json | jq '.data | {display_name, policies, path}'

If the pre-flight lookup — the by-value one, auth/token/lookup — already returns Code: 403 ... bad token, the token was expired or revoked before you arrived. Record that outcome as “already dead — nothing to revoke” rather than reporting a revocation that never happened, and confirm it two-sided the same way: the same command against a token known to be valid must return 200 on the same endpoint, or the 403 is a statement about your own policy rather than about the subject token.

Helper Script Operations

# Check connectivity
./scripts/vault-helper.sh status

# List infrastructure secrets
./scripts/vault-helper.sh list

# Get specific secret
./scripts/vault-helper.sh get bmc/tpi-alpha

# Get single field
./scripts/vault-helper.sh get bmc/tpi-alpha password

Integration Examples

Ansible Integration

# Vault lookup in group_vars
tpi_bmc_password: "{{ lookup('community.hashi_vault.vault_kv2_get', 'infrastructure/bmc/tpi-alpha', engine_mount_point='secret/fzymgc-house').secret.password }}"

cloudflare_api_token: "{{ lookup('community.hashi_vault.vault_kv2_get', 'infrastructure/cloudflare/api-token', engine_mount_point='secret/fzymgc-house').secret.token }}"

For Ansible modules that access Vault directly, ensure the CA bundle is set so Python requests can validate TLS. Configure vault_ca_cert_bundle in ansible/inventory/group_vars/all.yml or set VAULT_CACERT.

Requires community.hashi_vault collection (installed via requirements).

Terraform Integration

data "vault_kv_secret_v2" "cloudflare" {
  mount = "secret/fzymgc-house"
  name  = "infrastructure/cloudflare/api-token"
}

provider "cloudflare" {
  api_token = data.vault_kv_secret_v2.cloudflare.data["token"]
}

Secret Rotation

Manual Rotation

Generate new credential
Update Vault secret
Restart affected pods to pick up changes

Automatic Rotation

Some secrets support automatic rotation via Vault policies.

Rotating secrets consumed via write-only (`*_wo`) attributes

Some Terraform sinks read Vault secrets ephemerally and write them through write-only attributes (kubernetes_secret_v1.data_wo, tfe_notification_configuration.token_wo, …; hl-106s, ADR hl-ujwi). Terraform cannot diff write-only content, so rotating the Vault secret does not propagate by itself (exception: the tfe notification url/token use the provider’s auto-hash mode and re-send automatically).

After rotating such a secret in Vault:

Find the consuming resource (grep the workspace for data_wo / ephemeral "vault_kv_secret_v2").
Increment its data_wo_revision by exactly 1 (helm’s set_wo_revision must strictly increase; kubernetes accepts any change).
Plan/apply the workspace (main-cluster-bootstrap is Local execution — apply manually).
Verify the in-cluster value changed (kubectl get secret ... -o json | jq -S '.data' | sha256sum).

PKI Certificate Rotation

Available PKI Roles

Certificates are issued from these roles in one of two ways, per role: by a Vault Agent sidecar on the host that holds the AppRole, which renews the certificate near expiry, or by Ansible from the control node at converge, which re-issues it on the next converge once it is missing or near expiry; the role’s tasks under ansible/roles/ name which.

Role	Mount Path	Purpose	Default TTL
`kea-ha`	`fzymgc-house/v1/ica1/v1`	Kea HA peer mTLS between the two resolver nodes (server and client flags)	720h
`otel-collector-dns-client`	`fzymgc-house/v1/ica1/v1`	Client auth for the resolver nodes’ otel-collector to `otel-gateway.fzymgc.house`	720h
`otel-collector-nas-client`	`fzymgc-house/v1/ica1/v1`	Client auth for the NAS otel-collector to `otel-gateway.fzymgc.house`	720h
`otel-collector-firewalla-client`	`fzymgc-house/v1/ica1/v1`	Client auth for the Firewalla otel-collector to `otel-gateway.fzymgc.house`	720h
`nas-rustfs-s3-server`	`fzymgc-house/v1/ica1/v1`	Server certificate for the NAS RustFS direct S3 endpoint	720h
`otel-collector-nas-support-client`	`fzymgc-house/v1/ica1/v1`	Client auth for the nas-support otel-collector to `otel-gateway.fzymgc.house`	720h

Required Policy

Developers need infrastructure-developer policy. Defined in tf/vault/policy-infrastructure-developer.hcl:

path "secret/data/fzymgc-house/infrastructure/*" {
  capabilities = ["read", "list"]
}
path "secret/metadata/fzymgc-house/infrastructure/*" {
  capabilities = ["list"]
}
path "secret/data/fzymgc-house/*" {
  capabilities = ["read", "list"]
}
path "secret/metadata/fzymgc-house/*" {
  capabilities = ["list"]
}

Troubleshooting

Vault Sealed

If Vault is sealed, unseal keys are required. Contact cluster admin.

Vault auto-unseals from outside the cluster, so a sealed pod usually recovers on its own. A vault-unseal container runs on the Firewalla router — outside Kubernetes, deployed by ansible/roles/router-vault-unseal — watching all three vault-N.fzymgc.house endpoints. When it finds one sealed it submits the key shares and the pod comes back without anyone touching it. It polls every 15s and backs off after an error toward a maximum of 30m — those are vault_unseal_check_interval and vault_unseal_max_check_interval in ansible/roles/router-vault-unseal/defaults/main.yml, which remains the source of truth if the role changes them. 30m is the ceiling to wait against. A pod the unsealer has just recovered can sit unchecked for that long before its next look, so a sealed pod that has not come back within the ceiling means the unsealer — not Vault — is the thing to investigate. Do not budget against a typical backoff; a measured one is a dated reading and belongs in docs/operations/evidence/. docs/engineering/specs/2026-01-18-router-vault-unseal-design.md is the mechanism. Before doing anything manual, check whether the unsealer is alive and working — it is the thing that performs the recovery, and it is not visible from inside the cluster:

ssh router 'sudo docker ps --filter name=vault-unseal \
  --format "{{.Names}}|{{.Image}}|{{.State}}|{{.Status}}"'
# and confirm it is polling rather than merely running:
ssh router 'sudo docker logs --since 5m vault-unseal 2>&1 | tail -20'

The Firewalla has no rg; use the grep/awk/sed it ships if you need to filter on the box.

A sealed Vault still reports Ready. Never use kubectl get pod to decide whether Vault is usable. The StatefulSet’s readiness probe hits /v1/sys/health with sealedcode=204, so a sealed Vault answers 204 on exactly the URL the probe asks for and Kubernetes marks the pod Ready. Readiness proves the process is listening; it does not prove Vault is unsealed. Any automation that waits on condition=Ready before moving to the next member will walk the whole StatefulSet into a sealed state and report success. Read the seal state from the API instead:

for n in 0 1 2; do
  printf '%s ' "vault-$n"
  curl -s "https://vault-$n.fzymgc.house/v1/sys/health?standbyok=true" \
    | jq -c '{sealed, standby, initialized}'
done

All three should read sealed: false, and exactly one should read standby: false — that one is the active leader.

A climbing RESTARTS count on a vault pod is a symptom of sealing, not an unrelated crash. The liveness probe hits the same health endpoint without sealedcode, so a sealed Vault fails it and the kubelet kills the container. With the probe’s initial delay and failure threshold that repeats on a roughly two-minute cycle for as long as the pod stays sealed. If a vault pod is accumulating restarts, read sealed from /v1/sys/health before investigating a crash.

Restarting vault pods. Do not use kubectl rollout restart on the vault StatefulSet. Its update order is ordinal-descending, so it reaches the highest-ordinal pod first — which is frequently the active leader — and combined with the readiness behaviour above, nothing stops it proceeding through a sealed cluster. Delete pods individually with kubectl -n vault delete pod <pod>, standby members first and the active leader last, re-reading leadership from all three endpoints before each step because it moves. Wait for sealed: false on the pod you just restarted before touching the next one. Raft holds quorum at two of three, so a strict one-at-a-time rotation is safe.

Permission Denied

Check your Vault policy assignments in Keycloak groups.

Connectivity Issues

curl -s https://vault.fzymgc.house/v1/sys/health
vault token lookup

Ansible Secret Issues

ansible-galaxy collection list | grep hashi_vault
ansible localhost -m debug -a "msg={{ lookup('community.hashi_vault.vault_kv2_get', 'infrastructure/bmc/tpi-alpha', engine_mount_point='secret/fzymgc-house').secret.password }}"

Terraform Issues

vault token lookup
terraform console
> data.vault_kv_secret_v2.keycloak.data