Secrets & Vault Paths

Reference for secrets management and Vault path organization.

Vault Path Structure

secret/
└── fzymgc-house/
    ├── cluster/           # Kubernetes service secrets
    │   ├── keycloak       # Keycloak admin credentials
    │   ├── argocd/        # ArgoCD configuration
    │   ├── grafana        # Grafana admin/OIDC
    │   ├── vault/         # Vault configuration
    │   ├── temporal/      # Temporal worker secrets
    │   ├── discord/       # Discord bot credentials
    │   ├── workers/       # Temporal workflow secrets (HMAC keys, etc.)
    │   ├── mealie         # Mealie secrets
    │   ├── nats           # NATS NKey credentials
    │   ├── mosquitto      # Mosquitto MQTT credentials
    │   ├── github         # GitHub tokens
    │   ├── cloudflared/   # Tunnel credentials
    │   ├── tailscale/     # Tailscale operator OAuth
    │   ├── alloy          # Alloy external collector auth
    │   ├── clickstack     # ClickStack (ClickHouse + HyperDX + MongoDB + OTel)
    │   ├── merlin         # Merlin (OpenClaw) gateway credentials
    │   ├── dolt           # Dolt SQL server credentials
    │   └── postgres/      # Database credentials
    │       └── users/     # Per-app DB users
    ├── infrastructure/    # Infrastructure credentials
    │   ├── cloudflare/    # Cloudflare API tokens
    │   ├── hcp/           # HCP Terraform credentials
    │   └── bmc/           # BMC credentials per node
    ├── hermes/            # Per-profile secrets for the hermes agent fleet
    │   └── jarvis         # The jarvis profile on seattle
    └── applications/      # Application-specific secrets

cluster/ is the only prefix the cluster can read. The vault ClusterSecretStore serves every namespace, and its policy (external-secrets-operator) is an allow-list: secret/data/fzymgc-house/cluster/* plus the exact CA chain infrastructure/pki/fzymgc-ica1-ca (and infrastructure/dns/pdns-api through external-dns). A secret for a cluster workload goes under cluster/<app> and needs no policy change; a secret the cluster must never see goes anywhere else and is unreachable from an ExternalSecret by default.

Cluster Secrets

Path	Purpose	Keys
secret/fzymgc-house/cluster/smtp	Shared Mailgun SMTP transport — DEPRECATED, being retired. Keycloak migrated to Cloudflare Email Sending (hl-zih3); Authentik (its last consumer) has been fully decommissioned, so this path is pending final deletion.	host, port, username, password, use_tls
secret/fzymgc-house/cluster/keycloak	Keycloak bootstrap admin + realm admin (Terraform provider auth) + outbound email (Cloudflare Email Sending, hl-zih3) + GitHub social login (hl-d0md)	admin_username, admin_password (Keycloak bootstrap admin, consumed by the keycloak deployment ExternalSecret), terraform_username, terraform_password (admin-cli auth for the keycloak Terraform provider), email_from (per-app sender on the onboarded domain — auth@hl.fzymgc.house), cf_email_token (Cloudflare API token with “Email Sending: Edit” — used as the SMTP submission password; manually seeded, see docs/operations/outbound-email.md), github_client_id, github_client_secret (GitHub OAuth App — manually created, see docs/engineering/specs/2026-07-02-keycloak-email-github-login-design.md). Note: the old email_password key was a stale, unreferenced remnant and is removed with hl-zih3 (the Mailgun password lived on cluster/smtp, never here).
secret/fzymgc-house/cluster/octopus	Octopus self-hosted PR review	github_app_id, github_app_private_key, github_webhook_secret, github_state_secret, github_client_id, github_client_secret, better_auth_secret, octopus_data_key, admin_emails, openai_api_key (a LiteLLM virtual key since Phase 3)
secret/fzymgc-house/cluster/fovea	fovea AI code review (hl-3w5i)	openai_api_key (a LiteLLM virtual key since Phase 3). qdrant_api_key (Qdrant auth for fovea audit auditQdrantKey + qdrant-fovea QDRANT__SERVICE__API_KEY; hl-uh07.10).
secret/fzymgc-house/cluster/litellm	LiteLLM proxy — master key, OpenRouter keys, and the seven upstream/MCP credentials moved from the retired agentgateway path in Phase 5 (D-74; read by the litellm-secrets, firewalla-mcp-secrets and clickhouse-mcp-auth ExternalSecrets). The cluster/litellm/oidc subpath is Terraform-written (tf/keycloak/litellm.tf, hl-106s) and holds the OIDC client pair.	master_key, openrouter_api_key, openrouter_passthrough_api_key, claude_code_api_key, gemini_api_key, context7_api_key, exa_api_key, firecrawl_api_key, fal_api_key, clickhouse_mcp_auth_token, firewalla_mcp_auth_token
secret/fzymgc-house/cluster/engram-ui	engram Connect web-UI (operator-seeded; OIDC creds moved to engram-ui/oidc, written by tf/keycloak/engram_ui.tf — hl-106s)	ui_cookie_key
secret/fzymgc-house/cluster/argocd	ArgoCD config	admin_password, github_token, oidc_secret, webhook.github.secret
secret/fzymgc-house/cluster/grafana	Grafana secrets	admin_password, oidc_client_id, oidc_client_secret
secret/fzymgc-house/cluster/vault/*	Vault config	oidc_client_id, oidc_client_secret
secret/fzymgc-house/cluster/temporal/*	Temporal worker secrets	Various per-workflow
secret/fzymgc-house/cluster/temporal/github	GitHub App for repo cloning	app_id, installation_id, private_key
secret/fzymgc-house/cluster/workers/drift-detection	Drift detection workflow	discord-webhook-hmac-key
secret/fzymgc-house/cluster/discord/bot	Discord bot	token
secret/fzymgc-house/cluster/mealie	Mealie config	oidc_client_id, oidc_client_secret
secret/fzymgc-house/cluster/nats	NATS NKey auth	operator_jwt, operator_public, sys_account_seed, sys_account_public, sys_account_jwt, services_account_seed, services_account_public, services_account_jwt, iot_account_seed, iot_account_public, iot_account_jwt
secret/fzymgc-house/cluster/mosquitto	Mosquitto MQTT auth	passwd (pre-hashed output from mosquitto_passwd), bridge_username (any non-empty), bridge_password (IOT user bearer JWT for NATS MQTT)
secret/fzymgc-house/cluster/github	GitHub integration	app_id, app_private_key, webhook_secret
secret/fzymgc-house/cluster/cloudflared/*	Tunnel creds	tunnel_token, tunnel_id
secret/fzymgc-house/cluster/tailscale/oauth	Tailscale operator OAuth	clientId, clientSecret
secret/fzymgc-house/cluster/alloy	Alloy external collector auth	external_token
secret/fzymgc-house/cluster/clickstack	ClickStack secrets	admin_password, mongo_admin_password, mongo_replicaset_keyfile, hyperdx_bootstrap_secret, hyperdx_oidc_client_id, hyperdx_oidc_client_secret (written by tf/keycloak/clickstack.tf), otel_ingest_api_key (HyperDX team key — the real OTLP ingest bearer), otel_ingest_token (legacy/vestigial — does not authenticate OTLP). Reused by headroom: both headroom-apps and headroom-agents consume otel_ingest_api_key from this same path via their headroom-otlp-headers ExternalSecret — no new Vault path is created for headroom. Reused by hl-assets (Phase 4): the uploader projects the same property as Secret hl-assets-otlp-headers and injects it as OTEL_EXPORTER_OTLP_HEADERS, because cs-otel-collector rejects unauthenticated OTLP — opening egress without a token moves the failure from “dropped by policy” to “401 discarded by the collector”, both of which are silent from the sender’s side. Present the raw token with no Bearer scheme prefix: a scheme mismatch is rejected with a message that echoes the presented token into sender logs, so treat it as credential exposure rather than a configuration typo.
secret/fzymgc-house/cluster/firewalla-mcp	Firewalla MSP MCP server (hl-vbvu.1)	msp_id (<domain>.firewalla.net), token (MSP personal access token), box_id. The bearer the server validates (firewalla_mcp_auth_token) is LiteLLM’s property on cluster/litellm.
secret/fzymgc-house/cluster/merlin	Merlin (OpenClaw) gateway	gateway-token, claude-ai-session-key, gh-token, synthetic-api-key, openai-api-key, elevenlabs-api-key, openrouter-api-key
secret/fzymgc-house/cluster/dolt	Dolt SQL server credentials	root_password, agent_user, agent_password, beads_user, beads_password
secret/fzymgc-house/cluster/postgres/users/*	DB users	username, password
secret/fzymgc-house/cluster/pushover	Pushover shared user key	user_key
secret/fzymgc-house/cluster/pushover/app/cluster-infra	Pushover API token — infra domain. Second consumer added Phase 6: the hl-assets operational items scheduled workflow reads this path as the arc-runner-jobs Vault role to page on a tripped operational check.	token
secret/fzymgc-house/cluster/pushover/app/cluster-data	Pushover API token — data domain	token
secret/fzymgc-house/cluster/pushover/app/cluster-apps	Pushover API token — apps domain	token
secret/fzymgc-house/cluster/pushover/app/cluster-edge	Pushover API token — edge domain. Consumed since Phase 6 by the Uptime Kuma monitors, including the two hl-assets monitors (D-84/D-85).	token
secret/fzymgc-house/cluster/hl-assets/r2	hl-assets.dev public asset sharing — the R2 workload credential. Minted and written by tf/cloudflare/r2.tf (cloudflare_account_token, scoped through its resources map to both buckets). This one credential reads and writes both buckets — R2 offers no write-only permission group; the accepted residual is recorded in the split read/write ADR and bounded, with the no-List* bound discharged 2026-08-02, in the credential-isolation ADR. Projected as Secret hl-assets-r2 in namespace hl-assets by argocd/app-configs/hl-assets/external-secret.yaml — six discrete entries, never a bulk extraction.	access_key_id, secret_access_key, public_bucket, private_bucket, endpoint, custom_domain
secret/fzymgc-house/cluster/hl-assets/salt	hl-assets.dev content-addressing salt. Generated by random_password in tf/cloudflare/r2.tf with prevent_destroy and lifecycle { ignore_changes } on its data. NEVER ROTATE. Rotation does not error — it silently stops deduplication. The public object key is HMAC(salt, the bytes actually stored), so a new salt derives every subsequent upload into a different key space: the same photo lands twice under two URLs, and everyone already holding the old links cannot be told, because there is no channel. There is no signal of any kind that this has happened. Projected as its own Secret hl-assets-salt (argocd/app-configs/hl-assets/external-secret-salt.yaml) rather than as a seventh key on the R2 object, because that object’s six field names are asserted as a multiset against tf/cloudflare/r2.tf. The rule is also written into that manifest, into the Vault object’s custom_metadata, and into the uploader-service ADR; Phase 6 tracks it as a standing operational item. The rule is now MEASURED as well as written: op-salt-version compares two pinned values off this path’s KV metadata on every scheduled run — current_version against 1, and created_time against the recorded creation timestamp — so a rotation that happens anyway is noticed rather than merely forbidden. Both are needed: vault kv metadata delete followed by vault kv put restarts current_version at 1, so a destroy-and-recreate is invisible to the counter alone, and it is the route prevent_destroy cannot stop because it happens out of band. The check reads metadata only and cannot read the secret — the runner’s policy grants secret/metadata/…/salt and deliberately omits the paired data stanza, so that is a property of the credential rather than of the script.	salt
secret/fzymgc-house/cluster/hl-assets/access-token	hl-assets.dev Cloudflare Access service-token pair set for the Phase 5 drop page at drop.fzymgc.net, written by the same Terraform apply that mints the tokens (tf/cloudflare/access.tf). Three client classes, one pair each: share.fish and the documented curl one-liner read their pairs from Vault at run time; the iOS pair is additionally provisioned into the Shortcut from 1Password at setup. No values recorded.	ios_client_id, ios_client_secret, cli_client_id, cli_client_secret, curl_client_id, curl_client_secret
secret/fzymgc-house/cluster/hl-assets/monitor-token	hl-assets.dev Cloudflare Access service-token pair for the uptime monitor (D-85) — a fourth client class alongside the three in access-token, deliberately on its own path. Written by tf/cloudflare/access.tf (HCP workspace main-cluster-cloudflare) in the same apply that mints the token. Read by tf/uptime-kuma (main-cluster-uptime-kuma), which the terraform-uptime-kuma-admin policy grants read-only on this path and nothing else — so a scheduled health check never holds an uploader credential. Apply order: main-cluster-vault (the grant) → main-cluster-cloudflare (this write) → main-cluster-uptime-kuma (the read). Standing hazard: client_secret is readable only at creation, so a destroy-and-recreate rotates the monitor’s credential and the monitor reads DOWN until the uptime-kuma workspace re-applies — a false outage on a canary. Re-apply the consumer in the same change. Kept out of access-token on purpose: that object’s field set is asserted by size (authingress.EXPECTED_CLIENT_COUNT), and a fourth pair there would keep the check green while deleting the property it stands for.	client_id, client_secret
secret/fzymgc-house/cluster/ghcr/pull-secret	Shared GHCR image-pull credential for private ghcr.io/fzymgc-house/* packages. Each consuming namespace projects it as a kubernetes.io/dockerconfigjson Secret named ghcr-pull-secret through its own ExternalSecret; no new Vault path or grant is created per consumer. Consumed by agent-memory, firewalla-mcp, fovea, headroom-agents, headroom-apps, hl-assets (added Phase 4), octopus, temporal-workers, and shared-resources (the Helm registry config).	username, password (a read:packages PAT)

Token-expiry monitoring credential

secret/fzymgc-house/cluster/hl-assets/token-expiry-monitor is written by tf/cloudflare/token_expiry_monitor.tf. Its fields are token, account_id, and target_token_id. The token has only Account API Tokens Read on this account; the target is the bootstrap token, whose value is never copied here.

The arc-runner-jobs policy grants read-only access to this exact data path and its metadata. The Cloudflare Terraform workspace writes it through its existing paired cluster/hl-assets/* grants. This generated credential is held in Terraform state and Vault, like the Email Sending tokens; do not export its value or print plan JSON. See operational checks for deployment and bootstrap-replacement handling.

TF-owned OIDC subpaths (hl-106s)

Keycloak-provisioned OIDC credentials live at secret/fzymgc-house/cluster/<app>/oidc (written by tf/keycloak with plain data_json — fully diffable). The bare cluster/<app> paths are operator-only: Terraform never writes them, so vault kv put-style whole-version replaces by operators can no longer clobber TF-written fields (and vice versa). Apps: mealie, argocd, miniflux, karakeep, clickstack, engram-ui, cloudflare-access. See ADR hl-u103 and docs/engineering/specs/2026-07-03-vault-kv-deprecation-migration-design.md.

Infrastructure Secrets

Path	Purpose	Keys
secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token	Terraform token	token
secret/fzymgc-house/infrastructure/cloudflare/discord-webhook	Notifications	url
secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac	Webhook auth	secret
secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-worker	Worker auth	token
secret/fzymgc-house/infrastructure/hcp	HCP credentials	client_id, client_secret
secret/fzymgc-house/infrastructure/bmc/*	Node BMC creds	username, password
secret/fzymgc-house/infrastructure/k8s/tailscale-auth	k8s subnet-router Tailscale OAuth (GH-1711). Written by tf/tailscale; read control-node-side by the k8s-tailscale Ansible role on fresh deploys. See docs/operations/tailscale-subnet-routers.md.	client_id, client_secret
secret/fzymgc-house/infrastructure/seattle/tailscale-auth	seattle (Mac mini) hermes node Tailscale OAuth (requirement NET-02 of seanb4t/hermes-fleet). Written by tf/tailscale. The operator reads it once, under Sean’s own Vault identity, for the tailscale up --advertise-tags=tag:hermes enrollment. The hermes-jarvis AppRole cannot read it.	client_id, client_secret
secret/fzymgc-house/infrastructure/router/kopia-r2	Router Kopia backup to R2	username (R2 access key), password (R2 secret key), repo_password (Kopia encryption)
secret/fzymgc-house/infrastructure/nas/kopia-b2	NAS Kopia backup to Backblaze B2	b2_key_id, b2_application_key (B2 S3 creds), repo_password (Kopia encryption; also held offline in 1Password), b2_region, b2_bucket
secret/fzymgc-house/infrastructure/nas/cloudflare-email	NAS outbound email (Cloudflare Email Sending, hl-xz9j) — the TrueNAS native alert transport (SMART/scrub/pool-health). Minted + written by tf/cloudflare/nas_email.tf; read by the nas-mail Ansible role.	cf_email_token (per-app Cloudflare API token with “Email Sending: Edit”, used as the SMTP submission password), email_from (nas@hl.fzymgc.house — onboarded-domain sender). See docs/operations/outbound-email.md.
secret/fzymgc-house/infrastructure/nodes/cloudflare-email	Cluster-nodes outbound email (Cloudflare Email Sending, hl-zejv.2) — the Turing Pi postfix system-mail relay (unattended-upgrades/cron). Minted + written by tf/cloudflare/nodes_email.tf; read control-node-side by the tp2-bootstrap-node role.	cf_email_token (per-app Cloudflare API token with “Email Sending: Edit”, the SMTP submission password), email_from (nodes@hl.fzymgc.house — onboarded-domain sender). See docs/operations/outbound-email.md.
secret/fzymgc-house/infrastructure/router/dhcpv6	Router DHCPv6 DUID	duid (colon-separated hex bytes for IPv6 prefix persistence)
secret/fzymgc-house/infrastructure/dns/automation-ssh	DNS resolver automation SSH key (mount secret) for the fzymgc login on sandiego/portland. Read by scripts/dns-playbook.sh (field private_key), which materializes it to a 0600 temp file and forces IdentitiesOnly — the SSH agent is deliberately not used. Deliberately distinct from infrastructure/heimdall/automation-ssh (D-02): heimdall is the out-of-band recovery host for the resolver cutover, so shared key material would couple the escape hatch to the failure it insures against.	private_key, public_key (the public half is retained so the authorized_keys authorization can be re-installed after a node rebuild)
secret/fzymgc-house/infrastructure/unifi/readonly	UniFi controller API access for automation and agents: local account automation-readonly with the Network: View Only role, so the controller itself rejects writes. Use https://unifi.fzymgc.house (the bare IP fails TLS). The write-capable credential is the separate unifi/admin path, never extra keys here. The human admin login stays in 1Password.	username, password
secret/fzymgc-house/infrastructure/unifi/admin	UniFi controller credential for the main-cluster-unifi workspace: local account automation-admin with Network: Full Management, so tf/unifi can apply changes to the switching fabric. Granted narrowly by terraform-unifi (ephemeral read; never in state). It sits outside cluster/, so no ExternalSecret can read it; admin / infrastructure-developer (humans) can. See UniFi.	username, password
secret/fzymgc-house/infrastructure/terraform-registry/signing-key	GPG key that signs releases of the providers this organization publishes to the HCP Terraform private registry (first: app.terraform.io/fzymgc-house/unifi). RSA-4096, sign-only, no expiry; retire it by registering a new key, not by letting it lapse. The public half is committed at tf/hcp-terraform/files/terraform-provider-signing.asc and registered by tfe_registry_gpg_key; never read this path from Terraform, because a data source would copy the private key into state. It sits outside cluster/, so no ExternalSecret can read it; admin / infrastructure-developer read it for releases.	private_key (ASCII-armored), passphrase, public_key, fingerprint, key_id (the registry’s 16-hex key-id), revocation_certificate
secret/fzymgc-house/infrastructure/github-terraform	fzymgc-terraform GitHub App credentials — Terraform provider auth for github_repository_ruleset.main_branch_protection in tf/core-services. Read ephemerally at plan time (ephemeral "vault_kv_secret_v2"); Terraform never writes this path. Kept outside cluster/, so the cluster’s External Secrets store cannot read it. Manually seeded by the operator — App creation, single-repository installation, private-key generation and the vault kv put are the numbered runbook in docs/operations/github-tokens.md. Do not confuse it with secret/fzymgc-house/cluster/github (Cluster Secrets): that is the Actions-runner GitHub integration, a different identity with a different blast radius, and it carries different key names (app_id, app_private_key, webhook_secret). Rotating the wrong one of the two breaks the other. The issuer key is client_id, not app_id: the provider passes app_auth.id straight through as the JWT iss claim with no numeric coercion, and GitHub now takes the App’s Client ID there. The key is named for what it holds so a future rotation does not “correct” it back to the numeric App ID.	client_id, installation_id, private_key

Hermes Agent Secrets

One path per agent profile in the hermes fleet (seanb4t/hermes-fleet). Each node runs one profile, and a profile reads its own path and nothing else.

Path	Purpose	Keys
secret/fzymgc-house/hermes/jarvis	The jarvis agent profile on seattle (Mac mini). Vault Agent runs on the node as the macOS user jarvis, logs in with the hermes-jarvis AppRole, and renders the profile .env from a template. Terraform creates no value here. The operator writes it once.	litellm_api_key (the LiteLLM virtual key for the jarvis model lane), engram_token (agent memory service auth)

The path does not exist until that first write. Create it with vault kv put, which creates the path. vault kv patch merges into an existing path and fails on an absent one.

# Check first, because put replaces every key at the path.
vault kv get secret/fzymgc-house/hermes/jarvis || \
  vault kv put secret/fzymgc-house/hermes/jarvis litellm_api_key=... engram_token=...

It sits outside cluster/, so no ExternalSecret can read it. Only the hermes-jarvis AppRole and the human policies reach it.

Vault Policies

Policy to Secret Path Mapping

Policy	Paths	Consumers
external-secrets-operator	secret/data/fzymgc-house/cluster/* (read) and the exact secret/data/fzymgc-house/infrastructure/pki/fzymgc-ica1-ca (read). An allow-list: anything outside cluster/ is unreachable from an ExternalSecret	External Secrets Operator (vault ClusterSecretStore, every namespace)
terraform-unifi	secret/data/fzymgc-house/infrastructure/unifi/admin (read), one exact path	main-cluster-unifi HCP workspace (tf/vault/policy-terraform-workspaces.tf), for the UniFi credential read ephemerally
hermes-jarvis	secret/data/fzymgc-house/hermes/jarvis and secret/metadata/fzymgc-house/hermes/jarvis (read), plus token self-management. Two exact paths, no wildcard. It deliberately cannot read infrastructure/mac-mini/tailscale-auth, so a compromised agent profile cannot mint tailnet nodes	Vault Agent on the Mac mini, as the macOS user jarvis, through the hermes-jarvis AppRole (tf/vault/approle-hermes-jarvis.tf)
arc-runners	secret/data/fzymgc-house/cluster/github	GitHub Actions runners
temporal-worker	secret/data/fzymgc-house/cluster/temporal/*, secret/data/fzymgc-house/cluster/discord/*, secret/data/fzymgc-house/cluster/github/*, secret/data/fzymgc-house/cluster/cloudflare/r2/*, secret/data/fzymgc-house/cluster/workers/*	Temporal workers
temporal-worker-terraform	secret/data/fzymgc-house/cluster/hcp-terraform, secret/data/fzymgc-house/cluster/discord/cluster-notifications-app, secret/data/fzymgc-house/cluster/workers/drift-detection, secret/data/fzymgc-house/cluster/temporal/github	Terraform drift detection & GuardedApply workflow
terraform-hcp-terraform-local	secret/data/fzymgc-house/infrastructure/cloudflare/hcp-terraform-worker, secret/data/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac, secret/data/fzymgc-house/infrastructure/pki/fzymgc-ica1-ca	GuardedApplyWorkflow hcp-terraform module
terraform-cloudflare-admin	secret/data/fzymgc-house/infrastructure/cloudflare/* (bootstrap token, discord webhook, HMAC, worker URL), secret/data/fzymgc-house/cluster/cloudflared/*, secret/data/fzymgc-house/cluster/cloudflare-access + /oidc, secret/data/fzymgc-house/infrastructure/nas/cloudflare-email, secret/data/fzymgc-house/infrastructure/nodes/cloudflare-email, and the paired secret/data/fzymgc-house/cluster/hl-assets/* + secret/metadata/fzymgc-house/cluster/hl-assets/* wildcard grants (a bare-path grant carries no wildcard and would not cover the r2 / salt / access-token subpaths)	main-cluster-cloudflare HCP workspace (tf/vault/policy-terraform-workspaces.tf)
terraform-core-services-admin	secret/data/fzymgc-house/infrastructure/github-terraform (read) + secret/metadata/fzymgc-house/infrastructure/github-terraform (read, list) — two exact paths, no wildcards, no write capability of any kind. The paired metadata grant is a deliberate departure from terraform-dns-records-admin, which declined it (operator ruling 2026-08-29, argued in the HCL comment); do not harmonise them in either direction. Not the only reader of that path — admin and infrastructure-developer reach it through broad globs, and external-secrets-operator because it sits under cluster/; it moves to infrastructure/github-terraform, see #1980.	main-cluster-core-services HCP workspace (tf/vault/policy-terraform-workspaces.tf), for the fzymgc-terraform App credential read ephemerally at plan time
alloy-agent	secret/data/fzymgc-house/cluster/alloy	Firewalla Alloy collector
cert-manager	pki/fzymgc-house/*	cert-manager PKI
mealie	secret/data/fzymgc-house/cluster/mealie, secret/data/fzymgc-house/cluster/postgres/users/main-mealie	Mealie app
nats	secret/data/fzymgc-house/cluster/nats	NATS server
merlin	secret/data/fzymgc-house/cluster/merlin	Merlin (OpenClaw) gateway service
github-actions	secret/data/fzymgc-house/cluster/github	CI/CD workflows
hcp-terraform	Multiple cluster paths	Terraform workspaces
infrastructure-developer	secret/data/fzymgc-house/infrastructure/*, secret/data/fzymgc-house/* (read)	Human operators
nas-kopia	secret/data/fzymgc-house/infrastructure/nas/kopia-b2 (read)	NAS Kopia→B2 backup role (interior sandbox)
otel-collector-nas	fzymgc-house/v1/ica1/v1/issue/otel-collector-nas-client, fzymgc-house/v1/ica1/v1/sign/otel-collector-nas-client, fzymgc-house/v1/ica1/v1/cert/ca, fzymgc-house/v1/ica1/v1/ca_chain, secret/data/fzymgc-house/cluster/clickstack (read)	.200 TrueNAS nas-otel-collector (control-node cert issuance + ingest token)
otel-collector-nas-support	fzymgc-house/v1/ica1/v1/issue/otel-collector-nas-support-client, fzymgc-house/v1/ica1/v1/sign/otel-collector-nas-support-client, fzymgc-house/v1/ica1/v1/cert/ca, fzymgc-house/v1/ica1/v1/ca_chain, secret/data/fzymgc-house/cluster/clickstack (read, reused — no new subpath)	.202 nas-support nas-support-otel-collector (control-node cert issuance + ingest token)
otel-collector-dns	fzymgc-house/v1/ica1/v1/issue/otel-collector-dns-client, fzymgc-house/v1/ica1/v1/sign/otel-collector-dns-client, fzymgc-house/v1/ica1/v1/cert/ca, fzymgc-house/v1/ica1/v1/ca_chain, secret/data/fzymgc-house/cluster/clickstack (read, reused — no new subpath)	.40.6/.40.7 resolvers dns-otel-collector (control-node cert issuance + ingest token)
admin	secret/* (full)	Administrators

Auth Methods

Method	Mount	Purpose
Kubernetes	kubernetes/	Service account auth
OIDC	oidc/	Human SSO via Keycloak
AppRole	approle/	Automation/CI
JWT	jwt/	HCP Terraform

PKI mTLS Client Roles (otel collectors)

Per-host clientAuth identities on the ICA1 mount (fzymgc-house/v1/ica1/v1), issued from the control node at deploy time and used to authenticate OTel collectors to otel-gateway.fzymgc.house (RequireAndVerifyClientCert). Each host gets a distinct role + CN for clean revocation/audit.

PKI role	Cert CN	Mount	EKU	Key	TTL	Consumer
otel-collector-nas-client	otelcol-nas.fzymgc.house	fzymgc-house/v1/ica1/v1	clientAuth	ECDSA P-256	720h	.200 TrueNAS nas-otel-collector
otel-collector-nas-support-client	otelcol-nas-support.fzymgc.house	fzymgc-house/v1/ica1/v1	clientAuth	ECDSA P-256	720h	.202 nas-support nas-support-otel-collector
otel-collector-dns-client	otelcol-dns.fzymgc.house	fzymgc-house/v1/ica1/v1	clientAuth	ECDSA P-256	720h	.40.6/.40.7 resolvers dns-otel-collector

All three collectors read the same ClickStack ingest token (secret/data/fzymgc-house/cluster/clickstack, property otel_ingest_api_key) — the .202 and resolver policies reuse this existing path rather than creating a new secret subpath.

The resolver rows landed 2026-08-13 (03-REVIEW WR-05). tf/vault/policy-otel-collector-dns.tf and tf/vault/pki-otel-collector-dns.tf shipped with phase 03 and this reference was not updated with them, so an operator auditing who may read the ClickStack ingest token enumerated this table, found two consumers, and missed the third — the one on the estate’s DNS serving plane. otel-collector-firewalla is a Vault policy that is still absent from this table; it is outside this phase’s changed set and is recorded here rather than silently left, so the omission is visible to whoever audits next.

Kubernetes Integration

ClusterSecretStore

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: "https://vault-internal.vault:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"

ExternalSecret Example

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: grafana-admin
  namespace: grafana
spec:
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: grafana-admin-secret
  data:
    - secretKey: admin-password
      remoteRef:
        key: fzymgc-house/cluster/grafana
        property: admin_password

Adding New Secrets

1. Create Secret in Vault

# Login to Vault
export VAULT_ADDR=https://vault.fzymgc.house
vault login -method=oidc

# Create secret
vault kv put secret/fzymgc-house/cluster/new-service \
  api_key="value" \
  secret_key="value"

# Verify
vault kv get secret/fzymgc-house/cluster/new-service

2. Create ExternalSecret

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: new-service-secrets
  namespace: new-service
spec:
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: new-service-secrets
  data:
    - secretKey: API_KEY
      remoteRef:
        key: fzymgc-house/cluster/new-service
        property: api_key
    - secretKey: SECRET_KEY
      remoteRef:
        key: fzymgc-house/cluster/new-service
        property: secret_key

3. Update Policy (if new path)

If using a new path pattern, update the relevant policy in tf/vault/:

resource "vault_policy" "new-service" {
  name   = "new-service"
  policy = <<EOT
# Allow service to read its secrets from Vault
path "secret/data/fzymgc-house/cluster/new-service" {
  capabilities = ["read"]
}

# Required for ExternalSecrets to verify secret existence
path "secret/metadata/fzymgc-house/cluster/new-service" {
  capabilities = ["read"]
}
EOT
}

Secret Naming Conventions

Type	Pattern	Example
Service secrets	secret/fzymgc-house/cluster/<service>	secret/fzymgc-house/cluster/grafana
Sub-service secrets	secret/fzymgc-house/cluster/<service>/<component>	secret/fzymgc-house/cluster/argocd/github
Database users	secret/fzymgc-house/cluster/postgres/users/<db>-<user>	secret/fzymgc-house/cluster/postgres/users/main-mealie
Infrastructure	secret/fzymgc-house/infrastructure/<provider>	secret/fzymgc-house/infrastructure/cloudflare
Per-node secrets	secret/fzymgc-house/infrastructure/<type>/<node>	secret/fzymgc-house/infrastructure/bmc/tpi-alpha-1

Rotation Procedures

Manual Rotation

# Update secret value
vault kv put secret/fzymgc-house/cluster/service key=new-value

# Restart dependent pods
kubectl rollout restart deployment/service -n namespace

Automated Rotation

ExternalSecrets refreshes secrets based on refreshInterval:

spec:
  refreshInterval: 1h  # Check for updates hourly

Troubleshooting

Verify Secret Access

# Check Vault path exists
vault kv get secret/fzymgc-house/cluster/service

# Check ExternalSecret status
kubectl get externalsecret -n namespace
kubectl describe externalsecret name -n namespace

# Check synced Kubernetes secret
kubectl get secret name -n namespace -o yaml

Common Issues

Issue	Cause	Solution
SecretSyncedError	Wrong path or missing permissions	Verify Vault path and policy
Secret not updating	refreshInterval not elapsed	Wait or trigger manual refresh
Permission denied	Missing policy binding	Update Kubernetes auth role

See Vault Operations for detailed operational procedures.