Secrets & Vault Paths¶

Reference for secrets management and Vault path organization.

Vault Path Structure¶

secret/
└── fzymgc-house/
    ├── cluster/           # Kubernetes service secrets
    │   ├── authentik      # Authentik admin credentials
    │   ├── argocd/        # ArgoCD configuration
    │   ├── grafana        # Grafana admin/OIDC
    │   ├── vault/         # Vault configuration
    │   ├── windmill       # Windmill tokens
    │   ├── mealie         # Mealie secrets
    │   ├── github         # GitHub tokens
    │   ├── cloudflared/   # Tunnel credentials
    │   └── postgres/      # Database credentials
    │       └── users/     # Per-app DB users
    ├── infrastructure/    # Infrastructure credentials
    │   ├── cloudflare/    # Cloudflare API tokens
    │   ├── hcp/           # HCP Terraform credentials
    │   └── bmc/           # BMC credentials per node
    └── applications/      # Application-specific secrets

Cluster Secrets¶

Path	Purpose	Keys
`secret/fzymgc-house/cluster/authentik`	Authentik admin	`terraform_token`, `secret_key`, `bootstrap_password`, `email_host`, `email_port`, `email_username`, `email_password`, `email_use_tls`, `email_from`
`secret/fzymgc-house/cluster/argocd/*`	ArgoCD config	`admin_password`, `github_token`, `oidc_secret`
`secret/fzymgc-house/cluster/grafana`	Grafana secrets	`admin_password`, `oidc_client_id`, `oidc_client_secret`
`secret/fzymgc-house/cluster/vault/*`	Vault config	`oidc_client_id`, `oidc_client_secret`
`secret/fzymgc-house/cluster/windmill`	Windmill secrets	`github_token`, `webhook_secret`
`secret/fzymgc-house/cluster/mealie`	Mealie config	`oidc_client_id`, `oidc_client_secret`
`secret/fzymgc-house/cluster/github`	GitHub integration	`app_id`, `app_private_key`, `webhook_secret`
`secret/fzymgc-house/cluster/cloudflared/*`	Tunnel creds	`tunnel_token`, `tunnel_id`
`secret/fzymgc-house/cluster/postgres/users/*`	DB users	`username`, `password`

Infrastructure Secrets¶

Path	Purpose	Keys
`secret/fzymgc-house/infrastructure/cloudflare/bootstrap-token`	Terraform token	`token`
`secret/fzymgc-house/infrastructure/cloudflare/discord-webhook`	Notifications	`url`
`secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-hmac`	Webhook auth	`secret`
`secret/fzymgc-house/infrastructure/cloudflare/hcp-terraform-worker`	Worker auth	`token`
`secret/fzymgc-house/infrastructure/hcp`	HCP credentials	`client_id`, `client_secret`
`secret/fzymgc-house/infrastructure/bmc/*`	Node BMC creds	`username`, `password`

Vault Policies¶

Policy to Secret Path Mapping¶

Policy	Paths	Consumers
`external-secrets-operator`	`secret/data/*` (read)	External Secrets Operator
`fzymgc-cluster-secret-reader`	`secret/data/*` (read)	General cluster read access
`arc-runners`	`secret/data/fzymgc-house/cluster/github`	GitHub Actions runners
`cert-manager`	`pki/fzymgc-house/*`	cert-manager PKI
`mealie`	`secret/data/fzymgc-house/cluster/mealie`, `secret/data/fzymgc-house/cluster/postgres/users/main-mealie`	Mealie app
`github-actions`	`secret/data/fzymgc-house/cluster/windmill`, `secret/data/fzymgc-house/cluster/github`	CI/CD workflows
`hcp-terraform`	Multiple cluster paths	Terraform workspaces
`infrastructure-developer`	`secret/data/fzymgc-house/infrastructure/`, `secret/data/fzymgc-house/` (read)	Human operators
`admin`	`secret/*` (full)	Administrators

Auth Methods¶

Method	Mount	Purpose
Kubernetes	`kubernetes/`	Service account auth
OIDC	`oidc/`	Human SSO via Authentik
AppRole	`approle/`	Automation/CI
JWT	`jwt/`	HCP Terraform

Kubernetes Integration¶

ClusterSecretStore¶

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: "https://vault-internal.vault:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"

ExternalSecret Example¶

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: grafana-admin
  namespace: grafana
spec:
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: grafana-admin-secret
  data:
    - secretKey: admin-password
      remoteRef:
        key: fzymgc-house/cluster/grafana
        property: admin_password

Adding New Secrets¶

1. Create Secret in Vault¶

# Login to Vault
export VAULT_ADDR=https://vault.fzymgc.house
vault login -method=oidc

# Create secret
vault kv put secret/fzymgc-house/cluster/new-service \
  api_key="value" \
  secret_key="value"

# Verify
vault kv get secret/fzymgc-house/cluster/new-service

2. Create ExternalSecret¶

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: new-service-secrets
  namespace: new-service
spec:
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: new-service-secrets
  data:
    - secretKey: API_KEY
      remoteRef:
        key: fzymgc-house/cluster/new-service
        property: api_key
    - secretKey: SECRET_KEY
      remoteRef:
        key: fzymgc-house/cluster/new-service
        property: secret_key

3. Update Policy (if new path)¶

If using a new path pattern, update the relevant policy in tf/vault/:

resource "vault_policy" "new-service" {
  name   = "new-service"
  policy = <<EOT
path "secret/data/fzymgc-house/cluster/new-service" {
  capabilities = ["read"]
}
path "secret/metadata/fzymgc-house/cluster/new-service" {
  capabilities = ["read", "list"]
}
EOT
}

Secret Naming Conventions¶

Type	Pattern	Example
Service secrets	`secret/fzymgc-house/cluster/<service>`	`secret/fzymgc-house/cluster/grafana`
Sub-service secrets	`secret/fzymgc-house/cluster/<service>/<component>`	`secret/fzymgc-house/cluster/argocd/github`
Database users	`secret/fzymgc-house/cluster/postgres/users/<db>-<user>`	`secret/fzymgc-house/cluster/postgres/users/main-mealie`
Infrastructure	`secret/fzymgc-house/infrastructure/<provider>`	`secret/fzymgc-house/infrastructure/cloudflare`
Per-node secrets	`secret/fzymgc-house/infrastructure/<type>/<node>`	`secret/fzymgc-house/infrastructure/bmc/tpi-alpha-1`

Rotation Procedures¶

Manual Rotation¶

# Update secret value
vault kv put secret/fzymgc-house/cluster/service key=new-value

# Restart dependent pods
kubectl rollout restart deployment/service -n namespace

Automated Rotation¶

ExternalSecrets refreshes secrets based on refreshInterval:

spec:
  refreshInterval: 1h  # Check for updates hourly

Troubleshooting¶

Verify Secret Access¶

# Check Vault path exists
vault kv get secret/fzymgc-house/cluster/service

# Check ExternalSecret status
kubectl get externalsecret -n namespace
kubectl describe externalsecret name -n namespace

# Check synced Kubernetes secret
kubectl get secret name -n namespace -o yaml

Common Issues¶

Issue	Cause	Solution
`SecretSyncedError`	Wrong path or missing permissions	Verify Vault path and policy
Secret not updating	`refreshInterval` not elapsed	Wait or trigger manual refresh
Permission denied	Missing policy binding	Update Kubernetes auth role

See Vault Operations for detailed operational procedures.