Secrets & Vault Paths

Reference for secrets management and Vault path organization.

Vault Path Structure

secret/
└── fzymgc-house/
    ├── cluster/           # Kubernetes service secrets
    │   ├── keycloak       # Keycloak admin credentials
    │   ├── argocd/        # ArgoCD configuration
    │   ├── grafana        # Grafana admin/OIDC
    │   ├── vault/         # Vault configuration
    │   ├── temporal/      # Temporal worker secrets
    │   ├── discord/       # Discord bot credentials
    │   ├── workers/       # Temporal workflow secrets (HMAC keys, etc.)
    │   ├── mealie         # Mealie secrets
    │   ├── nats           # NATS NKey credentials
    │   ├── mosquitto      # Mosquitto MQTT credentials
    │   ├── github         # GitHub tokens
    │   ├── cloudflared/   # Tunnel credentials
    │   ├── tailscale/     # Tailscale operator OAuth
    │   ├── alloy          # Alloy external collector auth
    │   ├── clickstack     # ClickStack (ClickHouse + HyperDX + MongoDB + OTel)
    │   ├── merlin         # Merlin (OpenClaw) gateway credentials
    │   ├── dolt           # Dolt SQL server credentials
    │   └── postgres/      # Database credentials
    │       └── users/     # Per-app DB users
    ├── infrastructure/    # Infrastructure credentials
    │   ├── cloudflare/    # Cloudflare API tokens
    │   ├── hcp/           # HCP Terraform credentials
    │   └── bmc/           # BMC credentials per node
    └── applications/      # Application-specific secrets

Cluster Secrets

Path	Purpose	Keys
secret/fzymgc-house/cluster/smtp	Shared Mailgun SMTP transport — DEPRECATED, being retired. Keycloak migrated to Cloudflare Email Sending (hl-zih3); Authentik (its last consumer) has been fully decommissioned, so this path is pending final deletion.	host, port, username, password, use_tls
secret/fzymgc-house/cluster/keycloak	Keycloak bootstrap admin + realm admin (Terraform provider auth) + outbound email (Cloudflare Email Sending, hl-zih3) + GitHub social login (hl-d0md)	admin_username, admin_password (Keycloak bootstrap admin, consumed by the keycloak deployment ExternalSecret), terraform_username, terraform_password (admin-cli auth for the keycloak Terraform provider), email_from (per-app sender on the onboarded domain — auth@hl.fzymgc.house), cf_email_token (Cloudflare API token with “Email Sending: Edit” — used as the SMTP submission password; manually seeded, see docs/operations/outbound-email.md), github_client_id, github_client_secret (GitHub OAuth App — manually created, see docs/engineering/specs/2026-07-02-keycloak-email-github-login-design.md). Note: the old email_password key was a stale, unreferenced remnant and is removed with hl-zih3 (the Mailgun password lived on cluster/smtp, never here).
secret/fzymgc-house/cluster/octopus	Octopus self-hosted PR review	github_app_id, github_app_private_key, github_webhook_secret, github_state_secret, github_client_id, github_client_secret, better_auth_secret, octopus_data_key, admin_emails, openai_api_key (agentgateway VK)
secret/fzymgc-house/cluster/fovea	fovea AI code review (hl-3w5i)	openai_api_key (agentgateway VK — the vk_fovea client copy). Also adds vk_fovea to the cluster/agentgateway accept-list path. qdrant_api_key (Qdrant auth for fovea audit auditQdrantKey + qdrant-fovea QDRANT__SERVICE__API_KEY; hl-uh07.10).
secret/fzymgc-house/cluster/engram-ui	engram Connect web-UI (operator-seeded; OIDC creds moved to engram-ui/oidc, written by tf/keycloak/engram_ui.tf — hl-106s)	ui_cookie_key
secret/fzymgc-house/cluster/argocd	ArgoCD config	admin_password, github_token, oidc_secret, webhook.github.secret
secret/fzymgc-house/cluster/grafana	Grafana secrets	admin_password, oidc_client_id, oidc_client_secret
secret/fzymgc-house/cluster/vault/*	Vault config	oidc_client_id, oidc_client_secret
secret/fzymgc-house/cluster/temporal/*	Temporal worker secrets	Various per-workflow
secret/fzymgc-house/cluster/temporal/github	GitHub App for repo cloning	app_id, installation_id, private_key
secret/fzymgc-house/cluster/workers/drift-detection	Drift detection workflow	discord-webhook-hmac-key
secret/fzymgc-house/cluster/discord/bot	Discord bot	token
secret/fzymgc-house/cluster/mealie	Mealie config	oidc_client_id, oidc_client_secret
secret/fzymgc-house/cluster/nats	NATS NKey auth	operator_jwt, operator_public, sys_account_seed, sys_account_public, sys_account_jwt, services_account_seed, services_account_public, services_account_jwt, iot_account_seed, iot_account_public, iot_account_jwt
secret/fzymgc-house/cluster/mosquitto	Mosquitto MQTT auth	passwd (pre-hashed output from mosquitto_passwd), bridge_username (any non-empty), bridge_password (IOT user bearer JWT for NATS MQTT)
secret/fzymgc-house/cluster/github	GitHub integration	app_id, app_private_key, webhook_secret
secret/fzymgc-house/cluster/cloudflared/*	Tunnel creds	tunnel_token, tunnel_id
secret/fzymgc-house/cluster/tailscale/oauth	Tailscale operator OAuth	clientId, clientSecret
secret/fzymgc-house/cluster/alloy	Alloy external collector auth	external_token
secret/fzymgc-house/cluster/clickstack	ClickStack secrets	admin_password, mongo_admin_password, mongo_replicaset_keyfile, hyperdx_bootstrap_secret, hyperdx_oidc_client_id, hyperdx_oidc_client_secret (written by tf/keycloak/clickstack.tf), otel_ingest_api_key (HyperDX team key — the real OTLP ingest bearer), otel_ingest_token (legacy/vestigial — does not authenticate OTLP). Reused by headroom: both headroom-apps and headroom-agents consume otel_ingest_api_key from this same path via their headroom-otlp-headers ExternalSecret — no new Vault path is created for headroom.
secret/fzymgc-house/cluster/firewalla-mcp	Firewalla MSP MCP server (hl-vbvu.1)	msp_id (<domain>.firewalla.net), token (MSP personal access token), box_id. Also adds vk_firewalla + firewalla_mcp_auth_token to the cluster/agentgateway path.
secret/fzymgc-house/cluster/merlin	Merlin (OpenClaw) gateway	gateway-token, claude-ai-session-key, gh-token, synthetic-api-key, openai-api-key, elevenlabs-api-key, openrouter-api-key
secret/fzymgc-house/cluster/dolt	Dolt SQL server credentials	root_password, agent_user, agent_password, beads_user, beads_password
secret/fzymgc-house/cluster/postgres/users/*	DB users	username, password
secret/fzymgc-house/cluster/pushover	Pushover shared user key	user_key
secret/fzymgc-house/cluster/pushover/app/cluster-infra	Pushover API token — infra domain	token
secret/fzymgc-house/cluster/pushover/app/cluster-data	Pushover API token — data domain	token
secret/fzymgc-house/cluster/pushover/app/cluster-apps	Pushover API token — apps domain	token
secret/fzymgc-house/cluster/pushover/app/cluster-edge	Pushover API token — edge domain	token

TF-owned OIDC subpaths (hl-106s)

Keycloak-provisioned OIDC credentials live at secret/fzymgc-house/cluster/<app>/oidc (written by tf/keycloak with plain data_json — fully diffable). The bare cluster/<app> paths are operator-only: Terraform never writes them, so vault kv put-style whole-version replaces by operators can no longer clobber TF-written fields (and vice versa). Apps: mealie, argocd, miniflux, karakeep, clickstack, engram-ui, cloudflare-access. See ADR hl-u103 and docs/engineering/specs/2026-07-03-vault-kv-deprecation-migration-design.md.

Infrastructure Secrets

Path	Purpose	Keys
secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token	Terraform token	token
secret/fzymgc-house/infrastructure/cloudflare/discord-webhook	Notifications	url
secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac	Webhook auth	secret
secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-worker	Worker auth	token
secret/fzymgc-house/infrastructure/hcp	HCP credentials	client_id, client_secret
secret/fzymgc-house/infrastructure/bmc/*	Node BMC creds	username, password
secret/fzymgc-house/infrastructure/router/kopia-r2	Router Kopia backup to R2	username (R2 access key), password (R2 secret key), repo_password (Kopia encryption)
secret/fzymgc-house/infrastructure/nas/kopia-b2	NAS Kopia backup to Backblaze B2	b2_key_id, b2_application_key (B2 S3 creds), repo_password (Kopia encryption; also held offline in 1Password), b2_region, b2_bucket
secret/fzymgc-house/infrastructure/nas/cloudflare-email	NAS outbound email (Cloudflare Email Sending, hl-xz9j) — the TrueNAS native alert transport (SMART/scrub/pool-health). Minted + written by tf/cloudflare/nas_email.tf; read by the nas-mail Ansible role.	cf_email_token (per-app Cloudflare API token with “Email Sending: Edit”, used as the SMTP submission password), email_from (nas@hl.fzymgc.house — onboarded-domain sender). See docs/operations/outbound-email.md.
secret/fzymgc-house/infrastructure/nodes/cloudflare-email	Cluster-nodes outbound email (Cloudflare Email Sending, hl-zejv.2) — the Turing Pi postfix system-mail relay (unattended-upgrades/cron). Minted + written by tf/cloudflare/nodes_email.tf; read control-node-side by the tp2-bootstrap-node role.	cf_email_token (per-app Cloudflare API token with “Email Sending: Edit”, the SMTP submission password), email_from (nodes@hl.fzymgc.house — onboarded-domain sender). See docs/operations/outbound-email.md.
secret/fzymgc-house/infrastructure/router/dhcpv6	Router DHCPv6 DUID	duid (colon-separated hex bytes for IPv6 prefix persistence)

Vault Policies

Policy to Secret Path Mapping

Policy	Paths	Consumers
external-secrets-operator	secret/data/* (read)	External Secrets Operator
fzymgc-cluster-secret-reader	secret/data/* (read)	General cluster read access
arc-runners	secret/data/fzymgc-house/cluster/github	GitHub Actions runners
temporal-worker	secret/data/fzymgc-house/cluster/temporal/*, secret/data/fzymgc-house/cluster/discord/*, secret/data/fzymgc-house/cluster/github/*, secret/data/fzymgc-house/cluster/cloudflare/r2/*, secret/data/fzymgc-house/cluster/workers/*	Temporal workers
temporal-worker-terraform	secret/data/fzymgc-house/cluster/hcp-terraform, secret/data/fzymgc-house/cluster/discord/cluster-notifications-app, secret/data/fzymgc-house/cluster/workers/drift-detection, secret/data/fzymgc-house/cluster/temporal/github	Terraform drift detection & GuardedApply workflow
terraform-router-hosts-admin	fzymgc-house/v1/ica1/v1/issue/router-hosts-client, fzymgc-house/v1/ica1/v1/cert/ca, fzymgc-house/v1/ica1/v1/ca_chain, sys/mounts/fzymgc-house/v1/ica1/v1	GuardedApplyWorkflow PKI cert renewal
terraform-hcp-terraform-local	secret/data/fzymgc-house/infrastructure/cloudflare/hcp-terraform-worker, secret/data/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac, secret/data/fzymgc-house/infrastructure/pki/fzymgc-ica1-ca	GuardedApplyWorkflow hcp-terraform module
alloy-agent	secret/data/fzymgc-house/cluster/alloy	Firewalla Alloy collector
cert-manager	pki/fzymgc-house/*	cert-manager PKI
mealie	secret/data/fzymgc-house/cluster/mealie, secret/data/fzymgc-house/cluster/postgres/users/main-mealie	Mealie app
nats	secret/data/fzymgc-house/cluster/nats	NATS server
merlin	secret/data/fzymgc-house/cluster/merlin	Merlin (OpenClaw) gateway service
github-actions	secret/data/fzymgc-house/cluster/github	CI/CD workflows
hcp-terraform	Multiple cluster paths	Terraform workspaces
infrastructure-developer	secret/data/fzymgc-house/infrastructure/*, secret/data/fzymgc-house/* (read)	Human operators
nas-kopia	secret/data/fzymgc-house/infrastructure/nas/kopia-b2 (read)	NAS Kopia→B2 backup role (interior sandbox)
otel-collector-nas	fzymgc-house/v1/ica1/v1/issue/otel-collector-nas-client, fzymgc-house/v1/ica1/v1/sign/otel-collector-nas-client, fzymgc-house/v1/ica1/v1/cert/ca, fzymgc-house/v1/ica1/v1/ca_chain, secret/data/fzymgc-house/cluster/clickstack (read)	.200 TrueNAS nas-otel-collector (control-node cert issuance + ingest token)
otel-collector-nas-support	fzymgc-house/v1/ica1/v1/issue/otel-collector-nas-support-client, fzymgc-house/v1/ica1/v1/sign/otel-collector-nas-support-client, fzymgc-house/v1/ica1/v1/cert/ca, fzymgc-house/v1/ica1/v1/ca_chain, secret/data/fzymgc-house/cluster/clickstack (read, reused — no new subpath)	.202 nas-support nas-support-otel-collector (control-node cert issuance + ingest token)
admin	secret/* (full)	Administrators

Auth Methods

Method	Mount	Purpose
Kubernetes	kubernetes/	Service account auth
OIDC	oidc/	Human SSO via Keycloak
AppRole	approle/	Automation/CI
JWT	jwt/	HCP Terraform

PKI mTLS Client Roles (otel collectors)

Per-host clientAuth identities on the ICA1 mount (fzymgc-house/v1/ica1/v1), issued from the control node at deploy time and used to authenticate OTel collectors to otel-gateway.fzymgc.house (RequireAndVerifyClientCert). Each host gets a distinct role + CN for clean revocation/audit.

PKI role	Cert CN	Mount	EKU	Key	TTL	Consumer
otel-collector-nas-client	otelcol-nas.fzymgc.house	fzymgc-house/v1/ica1/v1	clientAuth	ECDSA P-256	720h	.200 TrueNAS nas-otel-collector
otel-collector-nas-support-client	otelcol-nas-support.fzymgc.house	fzymgc-house/v1/ica1/v1	clientAuth	ECDSA P-256	720h	.202 nas-support nas-support-otel-collector

Both collectors read the same ClickStack ingest token (secret/data/fzymgc-house/cluster/clickstack, property otel_ingest_api_key) — the .202 policy reuses this existing path rather than creating a new secret subpath.

Kubernetes Integration

ClusterSecretStore

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: "https://vault-internal.vault:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"

ExternalSecret Example

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: grafana-admin
  namespace: grafana
spec:
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: grafana-admin-secret
  data:
    - secretKey: admin-password
      remoteRef:
        key: fzymgc-house/cluster/grafana
        property: admin_password

Adding New Secrets

1. Create Secret in Vault

# Login to Vault
export VAULT_ADDR=https://vault.fzymgc.house
vault login -method=oidc

# Create secret
vault kv put secret/fzymgc-house/cluster/new-service \
  api_key="value" \
  secret_key="value"

# Verify
vault kv get secret/fzymgc-house/cluster/new-service

2. Create ExternalSecret

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: new-service-secrets
  namespace: new-service
spec:
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: new-service-secrets
  data:
    - secretKey: API_KEY
      remoteRef:
        key: fzymgc-house/cluster/new-service
        property: api_key
    - secretKey: SECRET_KEY
      remoteRef:
        key: fzymgc-house/cluster/new-service
        property: secret_key

3. Update Policy (if new path)

If using a new path pattern, update the relevant policy in tf/vault/:

resource "vault_policy" "new-service" {
  name   = "new-service"
  policy = <<EOT
# Allow service to read its secrets from Vault
path "secret/data/fzymgc-house/cluster/new-service" {
  capabilities = ["read"]
}

# Required for ExternalSecrets to verify secret existence
path "secret/metadata/fzymgc-house/cluster/new-service" {
  capabilities = ["read"]
}
EOT
}

Secret Naming Conventions

Type	Pattern	Example
Service secrets	secret/fzymgc-house/cluster/<service>	secret/fzymgc-house/cluster/grafana
Sub-service secrets	secret/fzymgc-house/cluster/<service>/<component>	secret/fzymgc-house/cluster/argocd/github
Database users	secret/fzymgc-house/cluster/postgres/users/<db>-<user>	secret/fzymgc-house/cluster/postgres/users/main-mealie
Infrastructure	secret/fzymgc-house/infrastructure/<provider>	secret/fzymgc-house/infrastructure/cloudflare
Per-node secrets	secret/fzymgc-house/infrastructure/<type>/<node>	secret/fzymgc-house/infrastructure/bmc/tpi-alpha-1

Rotation Procedures

Manual Rotation

# Update secret value
vault kv put secret/fzymgc-house/cluster/service key=new-value

# Restart dependent pods
kubectl rollout restart deployment/service -n namespace

Automated Rotation

ExternalSecrets refreshes secrets based on refreshInterval:

spec:
  refreshInterval: 1h  # Check for updates hourly

Troubleshooting

Verify Secret Access

# Check Vault path exists
vault kv get secret/fzymgc-house/cluster/service

# Check ExternalSecret status
kubectl get externalsecret -n namespace
kubectl describe externalsecret name -n namespace

# Check synced Kubernetes secret
kubectl get secret name -n namespace -o yaml

Common Issues

Issue	Cause	Solution
SecretSyncedError	Wrong path or missing permissions	Verify Vault path and policy
Secret not updating	refreshInterval not elapsed	Wait or trigger manual refresh
Permission denied	Missing policy binding	Update Kubernetes auth role

See Vault Operations for detailed operational procedures.