Skip to content

Vault Operations

Vault holds every secret the estate reads: Ansible lookups, Terraform providers, Kubernetes ExternalSecrets, Vault Agent on the NAS and the Mac mini, and the PKI that signs internal certificates. This page states what exists, where each part is defined, and the procedures that touch it. The secrets reference lists every path and who reads it.

Property Value Defined in
URL https://vault.fzymgc.house argocd/app-configs/vault/ingress-route.yaml
Members vault-0, vault-1 and vault-2, each at https://vault-<n>.fzymgc.house the same IngressRoute, one HostSNI route per member
Version The server.image.tag pin in argocd/cluster-app/templates/vault.yaml the chart default is not used
Storage Integrated Raft, three replicas, data on longhorn-encrypted, audit log on longhorn argocd/cluster-app/templates/vault.yaml
TLS The vault-api-tls certificate, reloaded on change argocd/app-configs/vault/vault-certificate.yaml
Terraform tf/vault/, applied by the main-cluster-vault workspace HCP Terraform
Helper script scripts/vault-helper.sh reads and writes under secret/fzymgc-house/
Method Path Who uses it Defined in
OIDC oidc People, through Keycloak. The reader and admin roles map Keycloak groups to the external identity groups of the same names tf/keycloak/vault.tf and tf/vault/groups-and-roles.tf
Token token Scripts and the local Terraform workspaces built in
Kubernetes kubernetes In-cluster workloads, one role per consumer tf/vault/k8s.tf and tf/vault/k8s-*.tf
AppRole approle Hosts outside the cluster: GitHub Actions and the otel collector on the NAS tf/vault/approle-*.tf
JWT jwt-hcp-terraform HCP Terraform runs, one role per workspace tf/vault/jwt-hcp-terraform.tf

The KV v2 mount is secret. Every estate secret sits under secret/fzymgc-house/. The infrastructure/ subtree holds credentials that Ansible and Terraform read, and the cluster/ subtree holds credentials that in-cluster workloads read. Each path has a policy under tf/vault/policy-*.tf, and the policy is bound to the consumer’s Kubernetes role, AppRole or JWT role.

People who run Ansible or Terraform hold the infrastructure-developer policy through the admin group. Its source is tf/vault/policy-infrastructure-developer.hcl:

path "secret/data/fzymgc-house/infrastructure/*" {
capabilities = ["read", "list"]
}
path "secret/metadata/fzymgc-house/infrastructure/*" {
capabilities = ["list"]
}
path "secret/data/fzymgc-house/*" {
capabilities = ["read", "list"]
}
path "secret/metadata/fzymgc-house/*" {
capabilities = ["list"]
}

The internal issuing CA is the mount fzymgc-house/v1/ica1/v1. Each role issues one kind of certificate with a 720 h TTL. A certificate renews in one of two ways, named by the consumer’s role under ansible/roles/: a Vault Agent on the host renews it near expiry, or Ansible re-issues it at the next converge once it is missing or near expiry.

Role Purpose Defined in
kea-ha Kea HA peer mTLS between the two resolver nodes tf/vault/pki-kea-ha.tf
otel-collector-dns-client Client auth for the resolver nodes’ otel collector tf/vault/pki-otel-collector-dns.tf
otel-collector-nas-client Client auth for the NAS otel collector tf/vault/pki-otel-collector-nas.tf
otel-collector-nas-support-client Client auth for the nas-support otel collector tf/vault/pki-otel-collector-nas-support.tf
nas-rustfs-s3-server Server certificate for the NAS RustFS direct S3 endpoint tf/vault/pki-nas-rustfs.tf

A second mount, fzymgc-house/v1/kea-ha/v1, holds the Kea HA sub-CA that ICA1 signs. Its role kea-ha issues the Kea HA certificate of the Vault Agents on the resolvers. The name constraints of the sub-CA permit only the HA addresses and names of the two resolvers. It is defined in tf/vault/pki-kea-ha-issuer.tf, and Host certificates describes it.

Vault auto-unseals from outside the cluster. An unsealer watches the three members. When it finds one sealed, it submits the key shares and the member comes back. Each unsealer holds all three shares, so one unsealer alone restores a member.

Two unsealers run:

Host Form Role Configuration
heimdall (192.168.40.20) Release binary under the systemd unit vault-unseal, user vault-unseal ansible/roles/vault-unseal /etc/vault-unseal/vault-unseal.yaml
NAS container vault-unseal (vault-unseal-nas.fzymgc.house, 192.168.40.34) Release binary under the systemd unit vault-unseal, user vault-unseal ansible/roles/vault-unseal /etc/vault-unseal/vault-unseal.yaml

The role reads the key shares from the SOPS file ansible/secrets/vault-unseal.sops.yaml at converge. The poll interval and the backoff ceiling are vault_unseal_check_interval and vault_unseal_max_check_interval in the role defaults. The ceiling is the longest a recovered member can wait before its next check. The role sets it to one minute, so a restarted member unseals within about 90 seconds of Running. The unit reads the member names through the host resolver, and the converge fails if the host cannot resolve them.

Terminal window
export VAULT_ADDR=https://vault.fzymgc.house
vault login -method=oidc # a person, through Keycloak
vault login # a token

The helper script wraps the common reads and writes under secret/fzymgc-house/:

Terminal window
./scripts/vault-helper.sh status
./scripts/vault-helper.sh list
./scripts/vault-helper.sh get bmc/tpi-alpha
./scripts/vault-helper.sh get bmc/tpi-alpha password
Terminal window
vault kv get secret/fzymgc-house/cluster/keycloak
vault kv get -field=terraform_password secret/fzymgc-house/cluster/keycloak
vault kv list secret/fzymgc-house/cluster/
vault kv put secret/fzymgc-house/cluster/example key=value
  1. Write the value with vault kv put, or let the Terraform workspace that owns it write it.

  2. Add a policy for the path under tf/vault/policy-<consumer>.tf, and bind it to the consumer’s Kubernetes role, AppRole or JWT role in the same change.

  3. Add the path to the secrets reference with its keys and its readers.

  4. After the main-cluster-vault workspace applies, make sure that the consumer can read it:

    Terminal window
    vault policy read <policy>
    vault kv get secret/fzymgc-house/<path>

Widen the policy of the tf/vault workspace

Section titled “Widen the policy of the tf/vault workspace”

The policy terraform-vault-admin refuses changes to itself, and main-cluster-vault applies on merge. If a pull request adds a grant to that policy, the apply fails with 403 unless you write the new policy by hand first:

  1. Check out the branch of the pull request.

  2. With a token that can write policies, write the policy from the branch:

    Terminal window
    awk '/resource "vault_policy" "terraform_vault_admin"/{f=1} f && /<<EOT/{p=1; next} p && /^EOT$/{exit} p' \
    tf/vault/policy-terraform-workspaces.tf | vault policy write terraform-vault-admin -
  3. Run terraform plan in tf/vault. Make sure that the plan shows no change to vault_policy.terraform_vault_admin.

  4. Merge the pull request.

A token that sits in a Kubernetes Secret, or one issued to a controller that no longer runs, is revoked through its accessor. The accessor is not a credential. vault token lookup -accessor and vault token revoke -accessor both accept it, it cannot authenticate, and it is the only handle that is safe to write into a runbook or a ticket. Record accessors, never values.

Terminal window
export VAULT_ADDR=https://vault.fzymgc.house
# The identity needs a policy that permits auth/token/lookup-accessor and revoke-accessor.
vault login -method=oidc
# Read the value once, straight into a variable. Never echo, log or `-o yaml` it.
TOKEN=$(kubectl -n <namespace> get secret <secret> -o jsonpath='{.data.<key>}' | base64 -d)
# -format=json includes .data.id, the token value, which the table output omits.
# The jq projection strips it. Drop the projection and you have printed the credential.
vault token lookup -format=json "$TOKEN" \
| jq '.data | {accessor, display_name, policies, entity_id, path, expire_time}'
unset TOKEN
# Irreversible. The accessor is enough, and the value is never needed again.
vault token revoke -accessor <accessor>
# Proof: the lookup by accessor must fail with `Code: 400`.
vault token lookup -accessor <accessor>
# Control: a lookup that must succeed, or the 400 proves only that lookup is broken.
# Bare `vault token lookup` looks up your own token. There is no -self flag.
vault token lookup -format=json | jq '.data | {display_name, policies, path}'

The two lookup forms are different endpoints with different failures. The accessor form (auth/token/lookup-accessor) answers Code: 400 for an unknown accessor. The by-value form (auth/token/lookup) answers Code: 403 ... bad token for a dead token. If the by-value pre-flight already answers 403, the token was dead before you arrived. Record that outcome as “already dead, nothing to revoke”. Then run the same by-value lookup against a token you know is valid. If that one also fails, the 403 is a statement about your own policy, not about the subject token.

Rotate a secret that a pod reads through an ExternalSecret:

  1. Generate the new credential at its source.

  2. Write it to the Vault path with vault kv put or vault kv patch.

  3. Wait for the ExternalSecret refresh, or restart the consumer’s pods.

  4. Make sure that the in-cluster value changed:

    Terminal window
    kubectl -n <namespace> get secret <secret> -o json | jq -S '.data' | sha256sum

Some Terraform sinks read a Vault secret ephemerally and write it through a write-only attribute, such as kubernetes_secret_v1.data_wo in tf/cluster-bootstrap/ and tfe_notification_configuration.url_wo and token_wo in tf/hcp-terraform/. Terraform cannot diff write-only content, so a rotated Vault value does not propagate on its own. The tfe notification is the exception: the provider hashes the value and re-sends it. For every other sink:

  1. Find the consuming resource:

    Terminal window
    rg -n 'data_wo|ephemeral "vault_kv_secret_v2"' tf
  2. Increase its data_wo_revision by one.

  3. Plan and apply the workspace. main-cluster-bootstrap runs locally, so apply it by hand.

  4. Make sure that the in-cluster value changed, with the sha256sum read above.

The design is the KV reads ADR.

Before any manual step, make sure that at least one unsealer is alive and polling. The unsealers perform the recovery, and they are not visible from inside the cluster. Neither host has rg, so filter on the box with grep.

Read heimdall at its literal address, because four identities answer to the name heimdall:

Terminal window
scripts/heimdall-adhoc.sh heimdall -b -m ansible.builtin.shell \
-a 'systemctl is-active vault-unseal; journalctl -u vault-unseal --since -5min --no-pager | tail -20'

Read the NAS container:

Terminal window
scripts/nas-adhoc.sh vault-unseal-nas -b -m ansible.builtin.shell \
-a 'systemctl is-active vault-unseal; journalctl -u vault-unseal --since -5min --no-pager | tail -20'

Read the seal state from the API, never from kubectl get pod:

Terminal window
for n in 0 1 2; do
printf '%s ' "vault-$n"
curl -s "https://vault-$n.fzymgc.house/v1/sys/health?standbyok=true" \
| jq -c '{sealed, standby, initialized}'
done

All three must read sealed: false, and exactly one reads standby: false. That one is the active leader.

If a member stays sealed past the unsealer’s backoff ceiling, the unsealer is the thing to investigate, not Vault. Read vault_unseal_max_check_interval in the role defaults for the ceiling.

Do not use kubectl rollout restart on the StatefulSet. Its update order is ordinal-descending, so it reaches the highest ordinal first, which is often the active leader, and nothing stops it from walking through a sealed cluster. Delete pods one at a time with kubectl -n vault delete pod <pod>, standby members first and the active leader last. Re-read leadership from all three endpoints before each step, because it moves. Wait for sealed: false on the pod you restarted before you touch the next one. Raft holds quorum at two of three, so a strict one-at-a-time rotation is safe.

  • A sealed Vault reports Ready. The readiness probe asks /v1/sys/health with sealedcode=204, so a sealed member answers 204 on that exact URL and Kubernetes marks the pod Ready. Readiness proves the process listens. It does not prove the member is unsealed. Any automation that waits on condition=Ready walks the StatefulSet into a sealed state and reports success.
  • A climbing RESTARTS count on a Vault pod is a symptom of sealing. The liveness probe asks the same endpoint without sealedcode, so a sealed member fails it and the kubelet restarts the container on the probe’s cycle. Read sealed from the API before you investigate a crash.
  • The member names resolve only at the resolver pair. Each unsealer uses the resolver of its host.
  • The unsealer refuses a configuration file with a mode wider than 0600. On both hosts, the file is 0600 and owned by vault-unseal, and the converge reads it as that user and as nobody.

Ansible reads a secret with the community.hashi_vault lookup, as in ansible/inventory/group_vars/tpi_alpha_hosts.yml:

tpi_bmc_password: "{{ lookup('community.hashi_vault.vault_kv2_get', 'infrastructure/bmc/tpi-alpha', engine_mount_point='secret/fzymgc-house').secret.password }}"

The lookup validates TLS through vault_ca_cert_bundle in ansible/inventory/group_vars/all.yml, or through VAULT_CACERT.

Terraform reads a secret with the vault_kv_secret_v2 data source, or ephemerally where the value must stay out of state. The pattern is in tf/cloudflare/terraform.tf:

data "vault_kv_secret_v2" "cloudflare" {
mount = "secret"
name = "infrastructure/cloudflare/api-token"
}
Symptom Check Fix
permission denied vault token lookup shows your policies Your Keycloak group maps to the reader or admin role in tf/keycloak/vault.tf. Ask for the group, or add the path to the policy the consumer holds
No connection curl -s https://vault.fzymgc.house/v1/sys/health The IngressRoute in argocd/app-configs/vault/ and the seal state above
An Ansible lookup fails ansible-galaxy collection list | grep hashi_vault, then the lookup as a debug task The collection, VAULT_ADDR, VAULT_TOKEN and the CA bundle
A Terraform read fails vault token lookup, then terraform console and the data source The workspace’s JWT role and policy in tf/vault/, see HCP Terraform
  • Secrets Reference
  • HCP Terraform Operations, the dynamic credentials
  • docs/engineering/specs/2026-01-18-router-vault-unseal-design.md, the design of the unsealer that ran on the Firewalla
  • docs/engineering/specs/2026-09-29-vault-unseal-two-hosts-design.md, the move to heimdall and the NAS