Secrets & Vault Paths¶

Reference for secrets management and Vault path organization.

Vault Path Structure¶

secret/
└── fzymgc-house/
    ├── cluster/           # Kubernetes service secrets
    │   ├── authentik      # Authentik admin credentials
    │   ├── argocd/        # ArgoCD configuration
    │   ├── grafana        # Grafana admin/OIDC
    │   ├── vault/         # Vault configuration
    │   ├── temporal/      # Temporal worker secrets
    │   ├── discord/       # Discord bot credentials
    │   ├── workers/       # Temporal workflow secrets (HMAC keys, etc.)
    │   ├── mealie         # Mealie secrets
    │   ├── nats           # NATS NKey credentials
    │   ├── mosquitto      # Mosquitto MQTT credentials
    │   ├── github         # GitHub tokens
    │   ├── cloudflared/   # Tunnel credentials
    │   ├── tailscale/     # Tailscale operator OAuth
    │   ├── alloy          # Alloy external collector auth
    │   ├── merlin         # Merlin (OpenClaw) gateway credentials
    │   ├── dolt           # Dolt SQL server credentials
    │   └── postgres/      # Database credentials
    │       └── users/     # Per-app DB users
    ├── infrastructure/    # Infrastructure credentials
    │   ├── cloudflare/    # Cloudflare API tokens
    │   ├── hcp/           # HCP Terraform credentials
    │   └── bmc/           # BMC credentials per node
    └── applications/      # Application-specific secrets

Cluster Secrets¶

Path	Purpose	Keys
`secret/fzymgc-house/cluster/authentik`	Authentik admin	`terraform_token`, `secret_key`, `bootstrap_password`, `email_host`, `email_port`, `email_username`, `email_password`, `email_use_tls`, `email_from`
`secret/fzymgc-house/cluster/argocd`	ArgoCD config	`admin_password`, `github_token`, `oidc_secret`, `webhook.github.secret`
`secret/fzymgc-house/cluster/grafana`	Grafana secrets	`admin_password`, `oidc_client_id`, `oidc_client_secret`
`secret/fzymgc-house/cluster/vault/*`	Vault config	`oidc_client_id`, `oidc_client_secret`
`secret/fzymgc-house/cluster/temporal/*`	Temporal worker secrets	Various per-workflow
`secret/fzymgc-house/cluster/temporal/github`	GitHub App for repo cloning	`app_id`, `installation_id`, `private_key`
`secret/fzymgc-house/cluster/workers/drift-detection`	Drift detection workflow	`discord-webhook-hmac-key`
`secret/fzymgc-house/cluster/discord/bot`	Discord bot	`token`
`secret/fzymgc-house/cluster/mealie`	Mealie config	`oidc_client_id`, `oidc_client_secret`
`secret/fzymgc-house/cluster/nats`	NATS NKey auth	`operator_jwt`, `operator_public`, `sys_account_seed`, `sys_account_public`, `sys_account_jwt`, `services_account_seed`, `services_account_public`, `services_account_jwt`, `iot_account_seed`, `iot_account_public`, `iot_account_jwt`
`secret/fzymgc-house/cluster/mosquitto`	Mosquitto MQTT auth	`passwd` (pre-hashed output from `mosquitto_passwd`), `bridge_username` (any non-empty), `bridge_password` (IOT user bearer JWT for NATS MQTT)
`secret/fzymgc-house/cluster/github`	GitHub integration	`app_id`, `app_private_key`, `webhook_secret`
`secret/fzymgc-house/cluster/cloudflared/*`	Tunnel creds	`tunnel_token`, `tunnel_id`
`secret/fzymgc-house/cluster/tailscale/oauth`	Tailscale operator OAuth	`clientId`, `clientSecret`
`secret/fzymgc-house/cluster/alloy`	Alloy external collector auth	`external_token`
`secret/fzymgc-house/cluster/merlin`	Merlin (OpenClaw) gateway	`gateway-token`, `claude-ai-session-key`, `gh-token`, `synthetic-api-key`, `openai-api-key`, `elevenlabs-api-key`, `openrouter-api-key`
`secret/fzymgc-house/cluster/dolt`	Dolt SQL server credentials	`root_password`, `agent_user`, `agent_password`, `beads_user`, `beads_password`
`secret/fzymgc-house/cluster/postgres/users/*`	DB users	`username`, `password`

Infrastructure Secrets¶

Path	Purpose	Keys
`secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token`	Terraform token	`token`
`secret/fzymgc-house/infrastructure/cloudflare/discord-webhook`	Notifications	`url`
`secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac`	Webhook auth	`secret`
`secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-worker`	Worker auth	`token`
`secret/fzymgc-house/infrastructure/hcp`	HCP credentials	`client_id`, `client_secret`
`secret/fzymgc-house/infrastructure/bmc/*`	Node BMC creds	`username`, `password`
`secret/fzymgc-house/infrastructure/router/kopia-r2`	Router Kopia backup to R2	`username` (R2 access key), `password` (R2 secret key), `repo_password` (Kopia encryption)
`secret/fzymgc-house/infrastructure/router/dhcpv6`	Router DHCPv6 DUID	`duid` (colon-separated hex bytes for IPv6 prefix persistence)

Vault Policies¶

Policy to Secret Path Mapping¶

Policy	Paths	Consumers
`external-secrets-operator`	`secret/data/*` (read)	External Secrets Operator
`fzymgc-cluster-secret-reader`	`secret/data/*` (read)	General cluster read access
`arc-runners`	`secret/data/fzymgc-house/cluster/github`	GitHub Actions runners
`temporal-worker`	`secret/data/fzymgc-house/cluster/temporal/`, `secret/data/fzymgc-house/cluster/discord/`, `secret/data/fzymgc-house/cluster/github/`, `secret/data/fzymgc-house/cluster/cloudflare/r2/`, `secret/data/fzymgc-house/cluster/workers/*`	Temporal workers
`temporal-worker-terraform`	`secret/data/fzymgc-house/cluster/hcp-terraform`, `secret/data/fzymgc-house/cluster/discord/cluster-notifications-app`, `secret/data/fzymgc-house/cluster/workers/drift-detection`, `secret/data/fzymgc-house/cluster/temporal/github`	Terraform drift detection & GuardedApply workflow
`terraform-router-hosts-admin`	`fzymgc-house/v1/ica1/v1/issue/router-hosts-client`, `fzymgc-house/v1/ica1/v1/cert/ca`, `fzymgc-house/v1/ica1/v1/ca_chain`, `sys/mounts/fzymgc-house/v1/ica1/v1`	GuardedApplyWorkflow PKI cert renewal
`terraform-hcp-terraform-local`	`secret/data/fzymgc-house/infrastructure/cloudflare/hcp-terraform-worker`, `secret/data/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac`, `secret/data/fzymgc-house/infrastructure/pki/fzymgc-ica1-ca`	GuardedApplyWorkflow hcp-terraform module
`alloy-agent`	`secret/data/fzymgc-house/cluster/alloy`	Firewalla Alloy collector
`cert-manager`	`pki/fzymgc-house/*`	cert-manager PKI
`mealie`	`secret/data/fzymgc-house/cluster/mealie`, `secret/data/fzymgc-house/cluster/postgres/users/main-mealie`	Mealie app
`nats`	`secret/data/fzymgc-house/cluster/nats`	NATS server
`merlin`	`secret/data/fzymgc-house/cluster/merlin`	Merlin (OpenClaw) gateway service
`github-actions`	`secret/data/fzymgc-house/cluster/github`	CI/CD workflows
`hcp-terraform`	Multiple cluster paths	Terraform workspaces
`infrastructure-developer`	`secret/data/fzymgc-house/infrastructure/`, `secret/data/fzymgc-house/` (read)	Human operators
`admin`	`secret/*` (full)	Administrators

Auth Methods¶

Method	Mount	Purpose
Kubernetes	`kubernetes/`	Service account auth
OIDC	`oidc/`	Human SSO via Authentik
AppRole	`approle/`	Automation/CI
JWT	`jwt/`	HCP Terraform

Kubernetes Integration¶

ClusterSecretStore¶

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: "https://vault-internal.vault:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"

ExternalSecret Example¶

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: grafana-admin
  namespace: grafana
spec:
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: grafana-admin-secret
  data:
    - secretKey: admin-password
      remoteRef:
        key: fzymgc-house/cluster/grafana
        property: admin_password

Adding New Secrets¶

1. Create Secret in Vault¶

# Login to Vault
export VAULT_ADDR=https://vault.fzymgc.house
vault login -method=oidc

# Create secret
vault kv put secret/fzymgc-house/cluster/new-service \
  api_key="value" \
  secret_key="value"

# Verify
vault kv get secret/fzymgc-house/cluster/new-service

2. Create ExternalSecret¶

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: new-service-secrets
  namespace: new-service
spec:
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: new-service-secrets
  data:
    - secretKey: API_KEY
      remoteRef:
        key: fzymgc-house/cluster/new-service
        property: api_key
    - secretKey: SECRET_KEY
      remoteRef:
        key: fzymgc-house/cluster/new-service
        property: secret_key

3. Update Policy (if new path)¶

If using a new path pattern, update the relevant policy in tf/vault/:

resource "vault_policy" "new-service" {
  name   = "new-service"
  policy = <<EOT
# Allow service to read its secrets from Vault
path "secret/data/fzymgc-house/cluster/new-service" {
  capabilities = ["read"]
}

# Required for ExternalSecrets to verify secret existence
path "secret/metadata/fzymgc-house/cluster/new-service" {
  capabilities = ["read"]
}
EOT
}

Secret Naming Conventions¶

Type	Pattern	Example
Service secrets	`secret/fzymgc-house/cluster/<service>`	`secret/fzymgc-house/cluster/grafana`
Sub-service secrets	`secret/fzymgc-house/cluster/<service>/<component>`	`secret/fzymgc-house/cluster/argocd/github`
Database users	`secret/fzymgc-house/cluster/postgres/users/<db>-<user>`	`secret/fzymgc-house/cluster/postgres/users/main-mealie`
Infrastructure	`secret/fzymgc-house/infrastructure/<provider>`	`secret/fzymgc-house/infrastructure/cloudflare`
Per-node secrets	`secret/fzymgc-house/infrastructure/<type>/<node>`	`secret/fzymgc-house/infrastructure/bmc/tpi-alpha-1`

Rotation Procedures¶

Manual Rotation¶

# Update secret value
vault kv put secret/fzymgc-house/cluster/service key=new-value

# Restart dependent pods
kubectl rollout restart deployment/service -n namespace

Automated Rotation¶

ExternalSecrets refreshes secrets based on refreshInterval:

spec:
  refreshInterval: 1h  # Check for updates hourly

Troubleshooting¶

Verify Secret Access¶

# Check Vault path exists
vault kv get secret/fzymgc-house/cluster/service

# Check ExternalSecret status
kubectl get externalsecret -n namespace
kubectl describe externalsecret name -n namespace

# Check synced Kubernetes secret
kubectl get secret name -n namespace -o yaml

Common Issues¶

Issue	Cause	Solution
`SecretSyncedError`	Wrong path or missing permissions	Verify Vault path and policy
Secret not updating	`refreshInterval` not elapsed	Wait or trigger manual refresh
Permission denied	Missing policy binding	Update Kubernetes auth role

See Vault Operations for detailed operational procedures.