Secrets & Vault Paths
Reference for the Vault paths this repository declares, who reads each one, and the policies that
grant them. Vault is at https://vault.fzymgc.house. The KV v2 mount is secret, and every path
sits under fzymgc-house/. The PKI mounts are fzymgc-house/v1/ica1/v1 (ICA1) and
fzymgc-house/v1/kea-ha/v1 (the Kea HA sub-CA). Every policy is in
tf/vault/. See Vault Operations for the procedures.
Path structure
Section titled “Path structure”secret/└── fzymgc-house/ ├── cluster/ # Secrets that cluster workloads read through ExternalSecrets │ ├── <app> # Operator-written keys for one app │ ├── <app>/oidc # Keycloak client credentials, written by tf/keycloak │ ├── litellm-keys/ # One LiteLLM credential per path │ ├── postgres/ # CNPG users and backup credentials │ ├── pushover/ # The shared user key and one app token per domain │ ├── hl-assets/ # The hl-assets.dev credentials, written by tf/cloudflare │ └── ghcr/ # The shared GHCR pull credential ├── infrastructure/ # Credentials that Terraform, Ansible and scripts read │ ├── cloudflare/ # Cloudflare tokens and the HCP Terraform notification pair │ ├── dns/ # Resolver pair credentials │ ├── nas/ # NAS credentials │ ├── bmc/ # Turing Pi BMC credentials │ └── pki/ # The ICA1 CA chain └── hermes/ # The last path of the retired hermes agent fleetcluster/ is the only prefix the cluster can read. The vault ClusterSecretStore serves every
namespace, and its policy (external-secrets-operator) is an allow-list: secret/data/fzymgc-house/cluster/*,
the exact path infrastructure/pki/fzymgc-ica1-ca, and the PKI ca_chain. A secret for a cluster
workload goes under cluster/<app> and needs no policy change. A secret the cluster must never
see goes anywhere else and is unreachable from an ExternalSecret.
Keycloak-provisioned OIDC credentials live at cluster/<app>/oidc, written by tf/keycloak. The
bare cluster/<app> path is operator-written and Terraform never writes it, so a vault kv put
on one path cannot clobber the other.
Cluster secrets
Section titled “Cluster secrets”Keys are the properties the ExternalSecrets read. A path with no keys listed is read whole.
| Path | Read by | Keys |
|---|---|---|
cluster/agent-memory |
the agent-memory ExternalSecret and Jobs |
service_auth_static_tokens |
cluster/agent-memory/b2, cluster/clickstack/b2, cluster/clickstack/dns-archive-b2, cluster/longhorn/b2, cluster/postgres/b2, cluster/velero/b2 |
the writer of each path, against the B2 bucket fzymgc-k8s-backups: the DNS query archive for cluster/clickstack/dns-archive-b2, and a backup writer for each other path; written by tf/backblaze/cluster_backups.tf |
access_key_id, secret_access_key, bucket, prefix, region, endpoint |
cluster/argocd, cluster/argocd/oidc |
tf/cluster-bootstrap/argocd.tf; the oidc path is written by tf/keycloak/argocd.tf |
|
cluster/argocd-image-updater |
argocd/app-configs/argocd-image-updater, seeded by the operator (Argo CD Image Updater) |
github_app_id, github_app_installation_id, github_app_private_key |
cluster/cert-manager |
argocd/app-configs/cert-manager-config |
cloudflare_api_token |
cluster/clickstack |
the clickstack ExternalSecret, the otel-ingest-token ExternalSecret of Vector, clickhouse-mcp, dns-control, every OTLP producer’s headers Secret, and the three external collector roles |
admin_password, mongo_admin_password, mongo_replicaset_keyfile, hyperdx_bootstrap_secret, otel_ingest_api_key, mcp_readonly_password, mcp_readwrite_password, schema_admin_password, dns_control_reader_password, app_password, otelcollector_password, dns_query_archiver_password |
cluster/clickstack/oidc |
the clickstack ExternalSecret; written by tf/keycloak/clickstack.tf |
hyperdx_oidc_client_secret |
cluster/clickstack-watchdog |
the watchdog CronJob; written by tf/uptime-kuma/clickstack-watchdog.tf |
token |
cluster/cloudflare-access, cluster/cloudflare-access/oidc |
tf/cloudflare/access.tf; the oidc path is written by tf/keycloak/cloudflare_access.tf |
|
cluster/cloudflared/tunnels/fzymgc-house-main |
argocd/app-configs/cloudflared-main; written by tf/cloudflare/tunnel.tf |
tunnel_token |
cluster/discord/cluster-notifications-app |
the Temporal workers | |
cluster/dns-agent-triage/oidc |
the triage agent at run time, written by tf/keycloak/dns_agent_triage.tf |
oidc_client_id, oidc_client_secret |
cluster/dns-control |
the dns-control ExternalSecret, seeded by the operator |
mcp_static_key_hashes (one name:sha256hex line for each static key), mcp_smoke_key (the plaintext of the smoke key, never projected into the cluster), nats_creds (the credentials file of the NATS user dns-control in the account DNS. The Deployment mounts it at /etc/dns-control/nats/dns-control.creds with mode 0400. The operator writes it with the procedure in NATS Operations), unifi_username and unifi_password (the UniFi OS local account dns-control-observer with the read-only Network role. The observed AAAA sweep reads the client table of the UniFi controller with it), unifi_writer_username and unifi_writer_password (the UniFi OS local account dns-control-writer with the role Network Admin. dns-control writes the alias and the note of a UniFi client with it. The operator adds both with vault kv patch, because vault kv put replaces every field of the path) |
cluster/dns-control/oidc |
the dns-control ExternalSecret; written by tf/keycloak/dns_control.tf |
oidc_client_id, oidc_client_secret |
cluster/dolt |
argocd/app-configs/dolt |
root_password, agent_user, agent_password, beads_user, beads_password |
cluster/engram-ui, cluster/engram-ui/oidc |
argocd/app-configs/agent-memory; the oidc path is written by tf/keycloak/engram_ui.tf |
ui_cookie_key; oidc_client_secret |
cluster/external-dns/oidc |
the dns-control webhook sidecar of external-dns; written by tf/keycloak/dns_control_external_dns.tf |
oidc_client_id, oidc_client_secret |
cluster/firewalla-mcp |
argocd/app-configs/firewalla-mcp |
msp_id, token, box_id |
cluster/ghcr/pull-secret |
every namespace that pulls a private ghcr.io/fzymgc-house/* image, the NAS collector roles, and Argo CD Image Updater |
username, password |
cluster/github |
the ARC runners and scripts/sync-vault-to-windmill-vars.sh |
|
cluster/goreleaser |
no workload. The operator copies the key to the Actions secret GORELEASER_KEY, which .github/workflows/release.yml reads (App Releases) |
license_key (the GoReleaser Pro license) |
cluster/hcp-terraform |
the HCP Terraform operator and the Temporal workers; tf/hcp-terraform/agent_pool.tf |
|
cluster/hl-assets/r2 |
argocd/app-configs/hl-assets; written by tf/cloudflare/r2.tf |
access_key_id, secret_access_key, endpoint, public_bucket, private_bucket, custom_domain |
cluster/hl-assets/salt |
argocd/app-configs/hl-assets and the hl-assets-operational-items workflow; written by tf/cloudflare/r2.tf with prevent_destroy |
salt |
cluster/hl-assets/access-token |
scripts/share.fish; written by tf/cloudflare/access.tf |
|
cluster/hl-assets/monitor-token |
tf/uptime-kuma/vault.tf; written by tf/cloudflare/access.tf |
|
cluster/hl-assets/token-expiry-monitor |
the arc-runner-jobs policy; written by tf/cloudflare/token_expiry_monitor.tf |
token, account_id, target_token_id |
cluster/karakeep, cluster/karakeep/oidc |
argocd/app-configs/karakeep; the oidc path is written by tf/keycloak/karakeep.tf |
nextauth_secret, nextauth_url, meili_master_key, browser_token, openai_api_key; oidc_client_id, oidc_client_secret, oauth_provider_name, oauth_wellknown_url |
cluster/keycloak |
argocd/app-configs/keycloak and tf/keycloak/vault.tf |
admin_username, admin_password |
cluster/life-agent |
argocd/app-configs/life-agent, seeded by the operator |
discord_bot_token, discord_allowed_users, discord_allowed_channels, ssh_client_private_key, ssh_client_public_key, ssh_host_private_key, ssh_host_public_key, engram_token |
cluster/life-agent/github-app |
the ExternalSecrets life-agent-github-app and life-agent-gateway, seeded by the operator (Life agent) |
app_id, installation_id, private_key, webhook_secret |
cluster/litellm, cluster/litellm/oidc |
argocd/app-configs/litellm, clickhouse-mcp and firewalla-mcp; the oidc path is written by tf/keycloak/litellm.tf |
openrouter_api_key, gemini_api_key, context7_api_key, exa_api_key, fal_api_key, firecrawl_api_key, clickhouse_mcp_auth_token, firewalla_mcp_auth_token; oidc_client_id, oidc_client_secret |
cluster/litellm/openrouter-zdr |
the litellm-secrets ExternalSecret in argocd/app-configs/litellm; written by tf/litellm/openrouter.tf |
api_key |
cluster/litellm-keys/<consumer> |
one ExternalSecret per consumer (agent-memory, karakeep, life-agent, mealie, octopus), master for the proxy, and the main-cluster-litellm workspace. The triage agent reads dns-agent-triage from Vault at run time, and no ExternalSecret projects it |
api_key, and master_key on master |
cluster/litellm-valkey |
argocd/app-configs/litellm |
password |
cluster/longhorn/crypto-key |
tf/cluster-bootstrap/longhorn.tf |
password |
cluster/mealie, cluster/mealie/oidc |
argocd/app-configs/mealie; the oidc path is written by tf/keycloak/mealie.tf |
base_url, allow_signup, openai_api_key; oidc_* |
cluster/miniflux, cluster/miniflux/oidc |
argocd/app-configs/miniflux; the oidc path is written by tf/keycloak/miniflux.tf |
admin_username, admin_password; oidc_client_id, oidc_client_secret |
cluster/mosquitto |
argocd/app-configs/mosquitto and nats |
passwd, bridge_username, bridge_password |
cluster/nats |
argocd/app-configs/nats |
the operator, SYS, SERVICES, IOT and DNS account seeds, public keys and JWTs. The ExternalSecret does not map operator_seed |
cluster/oauth2-proxy/oidc |
argocd/app-configs/oauth2-proxy; written by tf/keycloak/oauth2_proxy.tf |
client_id, client_secret, cookie_secret |
cluster/octopus, cluster/octopus-oidc |
argocd/app-configs/octopus; the oidc path is written by tf/keycloak/octopus.tf |
github_app_id, github_app_private_key, github_webhook_secret, github_state_secret, github_client_id, github_client_secret, better_auth_secret, octopus_data_key, admin_emails, openai_api_key; oidc_client_id, oidc_client_secret |
cluster/postgres/users/<db>-<user> |
the CNPG user manifests and each app’s ExternalSecret | username, password |
cluster/postgres/users/main-dns-control |
the ExternalSecret main-dns-control-credentials for the CNPG managed role dns_control, and the ExternalSecret dns-control-db, which renders the database URL of dns-control with sslmode=verify-full. The operator seeds the path |
username (dns_control), password |
cluster/pushover |
the alerts bootstrap, the NAS Kopia roles and the hl-assets-operational-items workflow |
user_key |
cluster/pushover/app/cluster-infra |
the same consumers | token |
cluster/pushover/app/cluster-edge |
tf/uptime-kuma |
token |
cluster/renovate |
argocd/app-configs/renovate |
github_app_id, github_app_private_key, github_webhook_secret, mend_license_key, server_api_secret |
cluster/temporal/github |
the Temporal workers | |
cluster/unifi-mcp |
the unifi-mcp-secrets ExternalSecret, written by the operator as a copy of infrastructure/unifi/readonly (the ExternalSecrets policy reads only cluster/*). A rotation of automation-readonly writes both paths |
username, password |
cluster/uptime-kuma |
tf/uptime-kuma/vault.tf |
|
cluster/valkey |
argocd/app-configs/valkey and oauth2-proxy |
password |
cluster/vault/oidc |
written by tf/keycloak/vault.tf |
|
cluster/windmill |
scripts/create-windmill-workspace.sh and its siblings |
Infrastructure secrets
Section titled “Infrastructure secrets”| Path | Read by | Keys |
|---|---|---|
infrastructure/backblaze/terraform |
tf/backblaze |
|
infrastructure/bmc/tpi-alpha, infrastructure/bmc/tpi-beta |
ansible/inventory/group_vars/tpi_alpha_hosts.yml and tpi_beta_hosts.yml |
username, password |
infrastructure/calgary/automation-ssh |
scripts/calgary-playbook.sh, for the Ansible login fzymgc on calgary; seeded by the operator |
private_key, public_key |
infrastructure/cloudflare/api-token |
ansible/inventory/group_vars/tp_cluster_nodes.yml |
|
infrastructure/cloudflare/bootstrap-token |
tf/cloudflare |
token |
infrastructure/cloudflare/discord-webhook |
tf/cloudflare/workers.tf |
url |
infrastructure/cloudflare/hcp-terraform-hmac |
tf/cloudflare/workers.tf and tf/hcp-terraform/notifications.tf; written by tf/vault/secrets-cloudflare.tf |
secret |
infrastructure/cloudflare/hcp-terraform-worker |
tf/cloudflare/workers.tf and tf/hcp-terraform/notifications.tf |
token |
infrastructure/dns/automation-ssh |
scripts/dns-playbook.sh, scripts/dns-adhoc.sh, scripts/dns-vip-holder.py |
private_key |
infrastructure/dns/kea-d2-tsig |
the kea and pdns roles |
|
infrastructure/dns/kea-ha |
the kea role |
|
infrastructure/dns/pdns-api |
the pdns role, the dns-control ExternalSecret |
api_key |
infrastructure/dns/tailscale-auth |
the dns-tailscale role; written by tf/tailscale/oauth_clients.tf |
|
infrastructure/github-terraform |
tf/core-services/vault.tf |
|
infrastructure/heimdall/automation-ssh |
the heimdall-common role and the heimdall scripts |
|
infrastructure/home-assistant/acme |
the operator, who enters the token in the Let’s Encrypt app of Home Assistant; written by tf/cloudflare/ha_acme.tf |
cloudflare_api_token |
infrastructure/home-assistant/backup-key |
the operator, for a restore of a Home Assistant backup; seeded by the operator | key |
infrastructure/home-assistant/tesla-fleet |
the operator, for a restore of /config/tesla_fleet.key in Home Assistant; seeded by the operator. The public key is cloudflare/workers/tesla-fleet-key/com.tesla.3p.public-key.pem |
private_key, domain |
infrastructure/k8s/tailscale-auth |
the k8s-tailscale role; written by tf/tailscale/oauth_clients.tf |
|
infrastructure/nas/automation-ssh |
the nas-common role and scripts/nas-playbook.sh |
|
infrastructure/nas/ares-tailscale-auth, infrastructure/nas/tailscale-auth |
the nas-aresmush and nas-tailscale roles; written by tf/tailscale/oauth_clients.tf |
|
infrastructure/nas/cloudflare-email |
the nas-mail role; written by tf/cloudflare/nas_email.tf |
|
infrastructure/nas/forgejo, infrastructure/nas/forgejo/oidc |
the nas-forgejo role; the oidc path is written by tf/keycloak/forgejo.tf |
|
infrastructure/nas/kopia-b2 |
the nas-kopia-backup role |
b2_key_id, b2_application_key, repo_password |
infrastructure/nas/precipice |
the nas-aresmush role |
|
infrastructure/nas/rustfs-admin |
the nas-rustfs role and tf/rustfs |
|
infrastructure/nas/rustfs-s3 |
the nas-rustfs and lightningstream roles |
|
infrastructure/nodes/cloudflare-email |
the tp2-bootstrap-node role; written by tf/cloudflare/nodes_email.tf |
|
infrastructure/openrouter/terraform |
tf/litellm, an ephemeral read into the openrouter provider; seeded by hand |
management_key |
infrastructure/pki/fzymgc-ica1-ca |
tf/cluster-bootstrap/cert-manager.tf, the HCP Terraform operator and the otel-scraper CA ExternalSecrets |
fullchain |
infrastructure/tailscale/bootstrap-oauth |
tf/tailscale/terraform.tf |
|
infrastructure/terraform-registry/signing-key |
tf/hcp-terraform/registry.tf |
|
infrastructure/unifi/acme |
the operator, who enters the token in UniFi OS for the console certificate; written by tf/cloudflare/unifi_acme.tf |
cloudflare_api_token |
infrastructure/unifi/admin |
tf/unifi |
|
infrastructure/unifi/wlan |
tf/unifi, an ephemeral read into the write-only passphrase_wo of each unifi_wlan; written by the operator with vault kv patch |
mainwireless, iot, guest, telework |
Hermes agent secrets
Section titled “Hermes agent secrets”The hermes agent fleet (seanb4t/hermes-fleet) is retired. Its profile jarvis on seattle no longer
exists, and no AppRole reads the hermes/ prefix. One path remains until the backup bucket of that
profile is destroyed:
| Path | Read by |
|---|---|
hermes/jarvis-kopia |
nothing; written by tf/backblaze, and removed with that workspace |
Vault policies
Section titled “Vault policies”Every policy is a vault_policy resource under tf/vault/policy-*.tf. The table names the
policies that grant KV paths and who binds them. Read the file for the exact grants.
| Policy | Grants | Bound to |
|---|---|---|
external-secrets-operator |
cluster/* read, the ICA1 CA path, the PKI ca_chain |
the external-secrets Kubernetes auth role behind the vault ClusterSecretStore |
clickstack-hyperdx, clickstack-otel-collector, clickstack-clickhouse |
cluster/clickstack |
Kubernetes auth roles of the same names |
dolt, karakeep, mealie, nats |
the app’s own path, plus postgres/users/main-mealie for mealie |
Kubernetes auth roles |
arc-runners |
cluster/github |
the ARC runner Kubernetes auth role |
arc-runner-jobs |
cluster/hl-assets/token-expiry-monitor, cluster/pushover, cluster/pushover/app/cluster-infra, cluster/hl-assets/salt metadata |
the ARC scheduled-job Kubernetes auth role |
github-actions |
cluster/github, cluster/windmill |
the github_actions AppRole |
temporal-worker, temporal-worker-terraform |
cluster/temporal/*, cluster/discord/*, cluster/github/*, cluster/cloudflare/r2/*, cluster/workers/*, cluster/hcp-terraform |
Kubernetes auth roles |
terraform-keycloak-admin |
cluster/keycloak, the bare and oidc paths of every Keycloak-provisioned app, cluster/argocd/*, cluster/vault/*, infrastructure/nas/forgejo/oidc, the ICA1 CA |
the main-cluster-keycloak workspace through jwt-hcp-terraform |
terraform-vault-admin, terraform-uptime-kuma-admin, terraform-cloudflare-admin, terraform-core-services-admin, terraform-hcp-terraform-local, terraform-tailscale-admin, terraform-unifi, terraform-rustfs, terraform-litellm, terraform-backblaze |
one workspace each; each grant is an exact path or one prefix | the matching HCP Terraform workspace through jwt-hcp-terraform |
otel-collector-nas-support, otel-collector-dns |
the host’s PKI issue and sign paths, the CA chain, and cluster/clickstack. The Vault Agents issue these certificates, and no converge uses the issue and sign paths |
the resolver and nas-support deploy paths |
kea-ha, kea-d2-tsig |
the matching infrastructure/dns/* path, and the CA chain of ICA1 |
the resolver pair |
dns-node-worker-sandiego, dns-node-worker-portland |
create and update on the issue path of the PKI role of the same node, plus auth/token/renew-self and auth/token/lookup-self. No sign path, no path of the other node and no KV path. Defined in tf/vault/policy-dns-node-worker.tf |
the AppRole of the same name, which Vault Agent in dns-node-worker-pki.service on that Pi uses |
nas-kopia, nas-rustfs |
infrastructure/nas/kopia-b2 (nas-kopia), and the CA chain of ICA1 (nas-rustfs) |
the NAS roles |
cert-agent-sandiego, cert-agent-portland, cert-agent-calgary, cert-agent-nas-support, cert-agent-nas-otelcol, cert-agent-nas-rustfs |
create and update on the issue path of each PKI role of the agent’s host or NAS app, plus auth/token/renew-self and auth/token/lookup-self. No sign path and no KV path. The two resolver policies share otel-collector-dns-client on ICA1 and kea-ha on the Kea HA mount. Defined in tf/vault/policy-cert-agents.tf |
the AppRole of the same name, which the Vault Agent of that host or app uses |
external-dns |
infrastructure/dns/pdns-api |
the dns-control ExternalSecret, through the external-secrets Kubernetes auth role, which tf/vault/k8s-external-secrets.tf binds to this policy as well as to external-secrets-operator |
cert-manager, pki-fzymgc-house, vault-agent, default, reader |
PKI paths | cert-manager and every token |
infrastructure-developer |
secret/data/fzymgc-house/* read |
human operators |
admin, policy-admin, identity-admin, kubernetes-config-admin, oidc-config-admin, ldap-config-admin, token-create, vault-snapshot-read |
Vault’s own configuration | administrators and the snapshot job |
Auth methods
Section titled “Auth methods”| Method | Mount | Purpose | Defined in |
|---|---|---|---|
| Kubernetes | kubernetes/ |
service account auth for the roles above | tf/vault/k8s.tf |
| OIDC | oidc/ |
human SSO through Keycloak | tf/keycloak/vault.tf |
| AppRole | approle/ |
github_actions, dns-node-worker-sandiego, dns-node-worker-portland, and the six cert-agent-* roles |
tf/vault/approle-*.tf |
| JWT | jwt-hcp-terraform/ |
HCP Terraform workspaces | tf/vault/jwt-hcp-terraform.tf |
The AppRoles of the node workers and of the certificate agents bind no CIDR. A host reaches Vault through the Traefik TLS passthrough route, and Vault reads no PROXY protocol header. Vault therefore sees the address of a Traefik pod and never the address of the host, so a CIDR bound to the host refuses every login. The secret ids do not expire. To rotate the secret id of a node worker, follow The node worker role. To rotate the secret id of a certificate agent, remove its secret id file on the host and run the role that deploys the agent.
PKI roles on the ICA1 mount
Section titled “PKI roles on the ICA1 mount”Each role issues one identity on fzymgc-house/v1/ica1/v1, so revocation and audit stay per
host. Every role uses EC P-256 keys.
| PKI role | Allowed domain | Use | Consumer |
|---|---|---|---|
otel-collector-nas-client |
otelcol-nas.fzymgc.house |
client auth | the nas-otel-collector role on the NAS |
otel-collector-nas-support-client |
otelcol-nas-support.fzymgc.house |
client auth | the nas-support-otel-collector role on 192.168.40.32 |
otel-collector-dns-client |
otelcol-dns.fzymgc.house |
client auth | the Vault Agents of the resolver pair, through cert-agent-sandiego and cert-agent-portland, for the dns-otel-collector role |
nas-rustfs-s3-server |
s3-direct.nas.fzymgc.house |
server auth | the nas-rustfs role |
dns-node-worker-sandiego |
sandiego.dns-node-worker.fzymgc.house, and the one URI SAN spiffe://fzymgc.house/dns-node-worker/sandiego |
client auth, TTL 30 days | the node worker on sandiego, through Vault Agent in dns-node-worker-pki.service. Defined in tf/vault/pki-dns-node-worker.tf |
dns-node-worker-portland |
portland.dns-node-worker.fzymgc.house, and the one URI SAN spiffe://fzymgc.house/dns-node-worker/portland |
client auth, TTL 30 days | the node worker on portland, the same way |
The four collector clients authenticate to otel-gateway.fzymgc.house, which requires and
verifies a client certificate. All four collectors read the same ingest key from
cluster/clickstack, property otel_ingest_api_key.
The Kea HA mount
Section titled “The Kea HA mount”tf/vault/pki-kea-ha-issuer.tf holds the mount fzymgc-house/v1/kea-ha/v1. Its only issuer is
the sub-CA fzymgc-house Kea HA CA v1 (EC P-384), which ICA1 signs. The sub-CA has pathlen:0,
and its name constraints permit only 192.168.40.6/32, 192.168.40.7/32,
kea-ha-sandiego.fzymgc.house and kea-ha-portland.fzymgc.house. It expires on 2027-12-31.
| PKI role | Allowed domain | Use | Consumer |
|---|---|---|---|
kea-ha |
kea-ha-sandiego.fzymgc.house, kea-ha-portland.fzymgc.house, with IP SANs |
server and client auth, TTL 30 days | the Vault Agents of the resolvers, through cert-agent-sandiego and cert-agent-portland |
The policy terraform-vault-admin grants the mount, the paths under it and the ICA1 path
root/sign-intermediate, so the main-cluster-vault workspace can create and replace the sub-CA.
Kubernetes integration
Section titled “Kubernetes integration”tf/cluster-bootstrap/external-secrets.tf creates the ClusterSecretStore vault. It uses the
Kubernetes auth mount with the external-secrets role, and every ExternalSecret references it:
apiVersion: external-secrets.io/v1kind: ExternalSecretmetadata: name: keycloak-admin namespace: keycloakspec: secretStoreRef: name: vault kind: ClusterSecretStore target: name: keycloak-admin-secret data: - secretKey: admin-password remoteRef: key: fzymgc-house/cluster/keycloak property: admin_passwordAdd a new secret
Section titled “Add a new secret”-
Write it to Vault:
Terminal window export VAULT_ADDR=https://vault.fzymgc.housevault login -method=oidcvault kv put secret/fzymgc-house/cluster/new-service api_key="value" secret_key="value"vault kv get secret/fzymgc-house/cluster/new-service -
Add an ExternalSecret in the app’s kustomization that reads
fzymgc-house/cluster/new-servicewith oneremoteRefper key. -
If the path is under
cluster/, no policy changes. If it is anywhere else, add avault_policyintf/vault/that grantssecret/data/<path>andsecret/metadata/<path>, bind it to the reader, and add the path to this page. -
After the sync, make sure that the Secret exists:
Terminal window kubectl get externalsecret -n new-servicekubectl get secret new-service-secrets -n new-service
Naming conventions
Section titled “Naming conventions”| Type | Pattern | Example |
|---|---|---|
| Service secrets | secret/fzymgc-house/cluster/<service> |
secret/fzymgc-house/cluster/keycloak |
| Keycloak client credentials | secret/fzymgc-house/cluster/<service>/oidc |
secret/fzymgc-house/cluster/mealie/oidc |
| Database users | secret/fzymgc-house/cluster/postgres/users/<db>-<user> |
secret/fzymgc-house/cluster/postgres/users/main-mealie |
| Infrastructure | secret/fzymgc-house/infrastructure/<provider>/<name> |
secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token |
| Per-node secrets | secret/fzymgc-house/infrastructure/<type>/<node> |
secret/fzymgc-house/infrastructure/bmc/tpi-alpha |
Rotate a secret
Section titled “Rotate a secret”# Replace one key without touching the othersvault kv patch secret/fzymgc-house/cluster/service key=new-value
# ESO refreshes on the ExternalSecret's refreshInterval. To force it:kubectl annotate externalsecret <name> -n <namespace> force-sync=$(date +%s) --overwrite
# Restart the consumer if Reloader does not watch its Secretkubectl rollout restart deployment/service -n namespaceA Terraform-written path is rotated by Terraform, never by hand. Read the Terraform file the table names, change it there, and apply.
Troubleshooting
Section titled “Troubleshooting”vault kv get secret/fzymgc-house/cluster/service # the path existskubectl get externalsecret -n namespace # Ready and SecretSyncedkubectl describe externalsecret name -n namespace # the sync error, if anykubectl get secret name -n namespace -o yaml # the synced Secret| Symptom | Cause | Fix |
|---|---|---|
SecretSyncedError |
wrong path, wrong property name, or a path outside cluster/ |
make sure that the path and property match Vault, or move the path under cluster/ |
| Secret not updating | refreshInterval not elapsed |
wait, or force the sync as above |
| Permission denied from Vault | the policy does not grant the path | add the grant in tf/vault/ and apply |