Vault Operations
Vault holds every secret the estate reads: Ansible lookups, Terraform providers, Kubernetes ExternalSecrets, Vault Agent on the NAS and the Mac mini, and the PKI that signs internal certificates. This page states what exists, where each part is defined, and the procedures that touch it. The secrets reference lists every path and who reads it.
What exists
Section titled “What exists”The cluster
Section titled “The cluster”| Property | Value | Defined in |
|---|---|---|
| URL | https://vault.fzymgc.house |
argocd/app-configs/vault/ingress-route.yaml |
| Members | vault-0, vault-1 and vault-2, each at https://vault-<n>.fzymgc.house |
the same IngressRoute, one HostSNI route per member |
| Version | The server.image.tag pin in argocd/cluster-app/templates/vault.yaml |
the chart default is not used |
| Storage | Integrated Raft, three replicas, data on longhorn-encrypted, audit log on longhorn |
argocd/cluster-app/templates/vault.yaml |
| TLS | The vault-api-tls certificate, reloaded on change |
argocd/app-configs/vault/vault-certificate.yaml |
| Terraform | tf/vault/, applied by the main-cluster-vault workspace |
HCP Terraform |
| Helper script | scripts/vault-helper.sh |
reads and writes under secret/fzymgc-house/ |
Auth methods
Section titled “Auth methods”| Method | Path | Who uses it | Defined in |
|---|---|---|---|
| OIDC | oidc |
People, through Keycloak. The reader and admin roles map Keycloak groups to the external identity groups of the same names |
tf/keycloak/vault.tf and tf/vault/groups-and-roles.tf |
| Token | token |
Scripts and the local Terraform workspaces | built in |
| Kubernetes | kubernetes |
In-cluster workloads, one role per consumer | tf/vault/k8s.tf and tf/vault/k8s-*.tf |
| AppRole | approle |
Hosts outside the cluster: GitHub Actions and the otel collector on the NAS | tf/vault/approle-*.tf |
| JWT | jwt-hcp-terraform |
HCP Terraform runs, one role per workspace | tf/vault/jwt-hcp-terraform.tf |
Secrets
Section titled “Secrets”The KV v2 mount is secret. Every estate secret sits under secret/fzymgc-house/. The
infrastructure/ subtree holds credentials that Ansible and Terraform read, and the cluster/
subtree holds credentials that in-cluster workloads read. Each path has a policy under
tf/vault/policy-*.tf, and the policy is bound to the consumer’s Kubernetes role, AppRole or JWT
role.
People who run Ansible or Terraform hold the infrastructure-developer policy through the
admin group. Its source is tf/vault/policy-infrastructure-developer.hcl:
path "secret/data/fzymgc-house/infrastructure/*" { capabilities = ["read", "list"]}path "secret/metadata/fzymgc-house/infrastructure/*" { capabilities = ["list"]}path "secret/data/fzymgc-house/*" { capabilities = ["read", "list"]}path "secret/metadata/fzymgc-house/*" { capabilities = ["list"]}The internal issuing CA is the mount fzymgc-house/v1/ica1/v1. Each role issues one kind of
certificate with a 720 h TTL. A certificate renews in one of two ways, named by the consumer’s
role under ansible/roles/: a Vault Agent on the host renews it near expiry, or Ansible
re-issues it at the next converge once it is missing or near expiry.
| Role | Purpose | Defined in |
|---|---|---|
kea-ha |
Kea HA peer mTLS between the two resolver nodes | tf/vault/pki-kea-ha.tf |
otel-collector-dns-client |
Client auth for the resolver nodes’ otel collector | tf/vault/pki-otel-collector-dns.tf |
otel-collector-nas-client |
Client auth for the NAS otel collector | tf/vault/pki-otel-collector-nas.tf |
otel-collector-nas-support-client |
Client auth for the nas-support otel collector | tf/vault/pki-otel-collector-nas-support.tf |
nas-rustfs-s3-server |
Server certificate for the NAS RustFS direct S3 endpoint | tf/vault/pki-nas-rustfs.tf |
A second mount, fzymgc-house/v1/kea-ha/v1, holds the Kea HA sub-CA that ICA1 signs. Its role
kea-ha issues the Kea HA certificate of the Vault Agents on the resolvers. The name constraints
of the sub-CA permit only the HA addresses and names of the two resolvers. It is defined in
tf/vault/pki-kea-ha-issuer.tf, and Host certificates
describes it.
The unsealer
Section titled “The unsealer”Vault auto-unseals from outside the cluster. An unsealer watches the three members. When it finds one sealed, it submits the key shares and the member comes back. Each unsealer holds all three shares, so one unsealer alone restores a member.
Two unsealers run:
| Host | Form | Role | Configuration |
|---|---|---|---|
heimdall (192.168.40.20) |
Release binary under the systemd unit vault-unseal, user vault-unseal |
ansible/roles/vault-unseal |
/etc/vault-unseal/vault-unseal.yaml |
NAS container vault-unseal (vault-unseal-nas.fzymgc.house, 192.168.40.34) |
Release binary under the systemd unit vault-unseal, user vault-unseal |
ansible/roles/vault-unseal |
/etc/vault-unseal/vault-unseal.yaml |
The role reads the key shares from the SOPS file ansible/secrets/vault-unseal.sops.yaml at
converge. The poll interval and the backoff ceiling are vault_unseal_check_interval and
vault_unseal_max_check_interval in the role defaults. The ceiling is the longest a recovered
member can wait before its next check. The role sets it to one minute, so a restarted member
unseals within about 90 seconds of Running. The unit reads the member names through the host
resolver, and the converge fails if the host cannot resolve them.
Procedures
Section titled “Procedures”Log in
Section titled “Log in”export VAULT_ADDR=https://vault.fzymgc.house
vault login -method=oidc # a person, through Keycloakvault login # a tokenThe helper script wraps the common reads and writes under secret/fzymgc-house/:
./scripts/vault-helper.sh status./scripts/vault-helper.sh list./scripts/vault-helper.sh get bmc/tpi-alpha./scripts/vault-helper.sh get bmc/tpi-alpha passwordRead, list and write a secret
Section titled “Read, list and write a secret”vault kv get secret/fzymgc-house/cluster/keycloakvault kv get -field=terraform_password secret/fzymgc-house/cluster/keycloakvault kv list secret/fzymgc-house/cluster/vault kv put secret/fzymgc-house/cluster/example key=valueAdd a secret path
Section titled “Add a secret path”-
Write the value with
vault kv put, or let the Terraform workspace that owns it write it. -
Add a policy for the path under
tf/vault/policy-<consumer>.tf, and bind it to the consumer’s Kubernetes role, AppRole or JWT role in the same change. -
Add the path to the secrets reference with its keys and its readers.
-
After the
main-cluster-vaultworkspace applies, make sure that the consumer can read it:Terminal window vault policy read <policy>vault kv get secret/fzymgc-house/<path>
Widen the policy of the tf/vault workspace
Section titled “Widen the policy of the tf/vault workspace”The policy terraform-vault-admin refuses changes to itself, and main-cluster-vault applies on
merge. If a pull request adds a grant to that policy, the apply fails with 403 unless you write
the new policy by hand first:
-
Check out the branch of the pull request.
-
With a token that can write policies, write the policy from the branch:
Terminal window awk '/resource "vault_policy" "terraform_vault_admin"/{f=1} f && /<<EOT/{p=1; next} p && /^EOT$/{exit} p' \tf/vault/policy-terraform-workspaces.tf | vault policy write terraform-vault-admin - -
Run
terraform planintf/vault. Make sure that the plan shows no change tovault_policy.terraform_vault_admin. -
Merge the pull request.
Revoke a token you do not hold
Section titled “Revoke a token you do not hold”A token that sits in a Kubernetes Secret, or one issued to a controller that no longer runs,
is revoked through its accessor. The accessor is not a credential. vault token lookup -accessor
and vault token revoke -accessor both accept it, it cannot authenticate, and it is the only
handle that is safe to write into a runbook or a ticket. Record accessors, never values.
export VAULT_ADDR=https://vault.fzymgc.house
# The identity needs a policy that permits auth/token/lookup-accessor and revoke-accessor.vault login -method=oidc
# Read the value once, straight into a variable. Never echo, log or `-o yaml` it.TOKEN=$(kubectl -n <namespace> get secret <secret> -o jsonpath='{.data.<key>}' | base64 -d)
# -format=json includes .data.id, the token value, which the table output omits.# The jq projection strips it. Drop the projection and you have printed the credential.vault token lookup -format=json "$TOKEN" \ | jq '.data | {accessor, display_name, policies, entity_id, path, expire_time}'unset TOKEN
# Irreversible. The accessor is enough, and the value is never needed again.vault token revoke -accessor <accessor>
# Proof: the lookup by accessor must fail with `Code: 400`.vault token lookup -accessor <accessor>
# Control: a lookup that must succeed, or the 400 proves only that lookup is broken.# Bare `vault token lookup` looks up your own token. There is no -self flag.vault token lookup -format=json | jq '.data | {display_name, policies, path}'The two lookup forms are different endpoints with different failures. The accessor form
(auth/token/lookup-accessor) answers Code: 400 for an unknown accessor. The by-value form
(auth/token/lookup) answers Code: 403 ... bad token for a dead token. If the by-value
pre-flight already answers 403, the token was dead before you arrived. Record that outcome as
“already dead, nothing to revoke”. Then run the same by-value lookup against a token you know is
valid. If that one also fails, the 403 is a statement about your own policy, not about the
subject token.
Secret Rotation
Section titled “Secret Rotation”Rotate a secret that a pod reads through an ExternalSecret:
-
Generate the new credential at its source.
-
Write it to the Vault path with
vault kv putorvault kv patch. -
Wait for the ExternalSecret refresh, or restart the consumer’s pods.
-
Make sure that the in-cluster value changed:
Terminal window kubectl -n <namespace> get secret <secret> -o json | jq -S '.data' | sha256sum
Some Terraform sinks read a Vault secret ephemerally and write it through a write-only
attribute, such as kubernetes_secret_v1.data_wo in tf/cluster-bootstrap/ and
tfe_notification_configuration.url_wo and token_wo in tf/hcp-terraform/. Terraform cannot
diff write-only content, so a rotated Vault value does not propagate on its own. The tfe
notification is the exception: the provider hashes the value and re-sends it. For every other
sink:
-
Find the consuming resource:
Terminal window rg -n 'data_wo|ephemeral "vault_kv_secret_v2"' tf -
Increase its
data_wo_revisionby one. -
Plan and apply the workspace.
main-cluster-bootstrapruns locally, so apply it by hand. -
Make sure that the in-cluster value changed, with the
sha256sumread above.
The design is the KV reads ADR.
Recover a sealed member
Section titled “Recover a sealed member”Before any manual step, make sure that at least one unsealer is alive and polling. The unsealers
perform the recovery, and they are not visible from inside the cluster. Neither host has rg, so
filter on the box with grep.
Read heimdall at its literal address, because four identities answer to the name heimdall:
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \ -a 'systemctl is-active vault-unseal; journalctl -u vault-unseal --since -5min --no-pager | tail -20'Read the NAS container:
scripts/nas-adhoc.sh vault-unseal-nas -b -m ansible.builtin.shell \ -a 'systemctl is-active vault-unseal; journalctl -u vault-unseal --since -5min --no-pager | tail -20'Read the seal state from the API, never from kubectl get pod:
for n in 0 1 2; do printf '%s ' "vault-$n" curl -s "https://vault-$n.fzymgc.house/v1/sys/health?standbyok=true" \ | jq -c '{sealed, standby, initialized}'doneAll three must read sealed: false, and exactly one reads standby: false. That one is the
active leader.
If a member stays sealed past the unsealer’s backoff ceiling, the unsealer is the thing to
investigate, not Vault. Read vault_unseal_max_check_interval in the role defaults for the
ceiling.
Restart Vault pods
Section titled “Restart Vault pods”Do not use kubectl rollout restart on the StatefulSet. Its update order is ordinal-descending,
so it reaches the highest ordinal first, which is often the active leader, and nothing stops it
from walking through a sealed cluster. Delete pods one at a time with
kubectl -n vault delete pod <pod>, standby members first and the active leader last. Re-read
leadership from all three endpoints before each step, because it moves. Wait for
sealed: false on the pod you restarted before you touch the next one. Raft holds quorum at two
of three, so a strict one-at-a-time rotation is safe.
Constraints
Section titled “Constraints”- A sealed Vault reports Ready. The readiness probe asks
/v1/sys/healthwithsealedcode=204, so a sealed member answers204on that exact URL and Kubernetes marks the pod Ready. Readiness proves the process listens. It does not prove the member is unsealed. Any automation that waits oncondition=Readywalks the StatefulSet into a sealed state and reports success. - A climbing
RESTARTScount on a Vault pod is a symptom of sealing. The liveness probe asks the same endpoint withoutsealedcode, so a sealed member fails it and the kubelet restarts the container on the probe’s cycle. Readsealedfrom the API before you investigate a crash. - The member names resolve only at the resolver pair. Each unsealer uses the resolver of its host.
- The unsealer refuses a configuration file with a mode wider than
0600. On both hosts, the file is0600and owned byvault-unseal, and the converge reads it as that user and asnobody.
Integration
Section titled “Integration”Ansible reads a secret with the community.hashi_vault lookup, as in
ansible/inventory/group_vars/tpi_alpha_hosts.yml:
tpi_bmc_password: "{{ lookup('community.hashi_vault.vault_kv2_get', 'infrastructure/bmc/tpi-alpha', engine_mount_point='secret/fzymgc-house').secret.password }}"The lookup validates TLS through vault_ca_cert_bundle in
ansible/inventory/group_vars/all.yml, or through VAULT_CACERT.
Terraform reads a secret with the vault_kv_secret_v2 data source, or ephemerally where the
value must stay out of state. The pattern is in tf/cloudflare/terraform.tf:
data "vault_kv_secret_v2" "cloudflare" { mount = "secret" name = "infrastructure/cloudflare/api-token"}Troubleshooting
Section titled “Troubleshooting”| Symptom | Check | Fix |
|---|---|---|
permission denied |
vault token lookup shows your policies |
Your Keycloak group maps to the reader or admin role in tf/keycloak/vault.tf. Ask for the group, or add the path to the policy the consumer holds |
| No connection | curl -s https://vault.fzymgc.house/v1/sys/health |
The IngressRoute in argocd/app-configs/vault/ and the seal state above |
| An Ansible lookup fails | ansible-galaxy collection list | grep hashi_vault, then the lookup as a debug task |
The collection, VAULT_ADDR, VAULT_TOKEN and the CA bundle |
| A Terraform read fails | vault token lookup, then terraform console and the data source |
The workspace’s JWT role and policy in tf/vault/, see HCP Terraform |
See Also
Section titled “See Also”- Secrets Reference
- HCP Terraform Operations, the dynamic credentials
docs/engineering/specs/2026-01-18-router-vault-unseal-design.md, the design of the unsealer that ran on the Firewalladocs/engineering/specs/2026-09-29-vault-unseal-two-hosts-design.md, the move to heimdall and the NAS