oauth2-proxy server-side session store (Valkey Sentinel HA) — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Move oauth2-proxy off its default cookie session store to a TLS-secured, Sentinel-HA Valkey server-side store, collapsing the ~4.7 KB _oauth2_proxy cookie to a ~150 B ticket.

Architecture: A new operator-managed Valkey (primary + 2 replicas + Sentinel quorum, AOF, TLS via vault-issuer/ICA1) is deployed as its own ArgoCD app; oauth2-proxy is reconfigured to use it via the chart-native sessionStorage.redis (Sentinel client, rediss://), with the redis/sentinel passwords sourced from Vault→ExternalSecret (never in git). GitOps throughout; each piece is an independently-reversible PR.

Tech Stack: Kubernetes, ArgoCD (app-of-apps in argocd/cluster-app/templates/ + argocd/app-configs/), Helm, OT-Container-Kit redis-operator (leading candidate — redis.redis.opstreelabs.in/v1beta2), Valkey, cert-manager (vault-issuer ClusterIssuer / ICA1), External Secrets Operator (ClusterSecretStore vault), Longhorn, oauth2-proxy chart 10.7.0.

Spec: docs/engineering/specs/2026-07-05-oauth2-proxy-session-store-design.md · Bead: hl-bqx2

Model hints (for bead materialization, Rule 5): Task 2 = model:haiku (mechanical YAML mirrored from existing templates). Tasks 1, 3, 4, 5, 6 = model:sonnet (spike + fallback judgment, operator/CRD wiring, HA-verification gate, the cutover, and E2E verification).

Prerequisites (one-time, before Task 2)

P1: Seed the Valkey passwords into Vault (out-of-band; secrets never go in git)

Run (requires an authenticated vault session — e.g. after vault login -method=oidc role=admin):

vault kv put fzymgc-house/cluster/valkey \
  redis-password="$(openssl rand -base64 32)" \
  redis-sentinel-password="$(openssl rand -base64 32)"

Expected: Success! Data written to: fzymgc-house/cluster/valkey.

P2: Confirm ESO can read the new path

Run: vault policy read external-secrets 2>/dev/null | rg -i "cluster/(\*|valkey|data)" || echo "CHECK POLICY" Expected: a path "…/cluster/*" (or explicit cluster/valkey) read grant. If absent, add the grant in tf/vault and apply before proceeding (per argocd/CLAUDE.md: “MUST update Vault policies when adding new secret paths”).

Task 1: Operator + TLS-with-Sentinel spike (GATE)

This task is a validation spike, not a committed change. Its output is a decision recorded on hl-bqx2 and a set of proven manifests that Task 3 productionizes. Do it in a throwaway namespace against the live cluster; commit nothing here.

Files:

Scratch only (a temp dir): /tmp/valkey-spike/* — not committed.
Step 1: Install the candidate operator into a scratch namespace

kubectl create namespace valkey-spike
helm repo add ot-helm https://ot-container-kit.github.io/helm-charts && helm repo update ot-helm
helm install redis-operator ot-helm/redis-operator -n valkey-spike --wait

Expected: operator pod Running; CRDs redisreplications.redis.redis.opstreelabs.in and redissentinels.redis.redis.opstreelabs.in present (kubectl get crd | rg opstreelabs).

Step 2: Mint a throwaway TLS cert for the spike

kubectl -n valkey-spike apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: valkey-tls
  namespace: valkey-spike
spec:
  secretName: valkey-tls
  issuerRef:
    name: vault-issuer
    kind: ClusterIssuer
  dnsNames:
    - valkey-spike.valkey-spike.svc.cluster.local
    - valkey-spike-headless.valkey-spike.svc.cluster.local
    - valkey-spike-sentinel.valkey-spike.svc.cluster.local
    - "*.valkey-spike-headless.valkey-spike.svc.cluster.local"
  usages: [server auth, client auth]
EOF

Expected: kubectl -n valkey-spike get secret valkey-tls shows keys ca.crt, tls.crt, tls.key.

Step 3: Deploy Valkey (RedisReplication + embedded Sentinel + TLS) using the Valkey image

kubectl -n valkey-spike apply -f - <<'EOF'
apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: RedisReplication
metadata:
  name: valkey-spike
  namespace: valkey-spike
spec:
  clusterSize: 3
  kubernetesConfig:
    image: valkey/valkey:8-alpine   # SPIKE: prove the valkey image works under this operator
    imagePullPolicy: IfNotPresent
  TLS:
    ca: ca.crt
    cert: tls.crt
    key: tls.key
    secret:
      secretName: valkey-tls
  sentinel:
    size: 3
    image: valkey/valkey:8-alpine   # SPIKE: Valkey ships sentinel in the main image, not a separate -sentinel image
EOF

Step 4: Run the three acceptance criteria

Run each; ALL must pass to choose this operator:

# (a) Valkey image runs under the operator
kubectl -n valkey-spike get pods -l app.kubernetes.io/name=valkey-spike -o wide   # 3 redis + 3 sentinel Running
# (b) TLS: connect over TLS and PING
kubectl -n valkey-spike exec valkey-spike-0 -- sh -c \
  'valkey-cli --tls --cacert /etc/redis/tls/ca.crt -a "$(cat /etc/redis/auth/password 2>/dev/null)" ping'   # PONG
# (c) Sentinel auto-failover: delete the primary, confirm promotion
kubectl -n valkey-spike delete pod "$(kubectl -n valkey-spike get pod -l redis-role=master -o jsonpath='{.items[0].metadata.name}')"
sleep 20
kubectl -n valkey-spike get pods -L redis-role   # a different pod now labelled master
# (d) CRITICAL: capture the master-group NAME Sentinel registers — oauth2-proxy's masterName MUST equal it
kubectl -n valkey-spike exec valkey-spike-sentinel-0 -- valkey-cli -p 26379 sentinel masters | sed -n '1,3p'

Expected: (a) all pods Running, (b) PONG over TLS, (c) a new master promoted within the failover window, (d) the Sentinel master-group name is recorded — OT operator v0.25.0 registers mymaster, which MUST equal sessionStorage.redis.sentinel.masterName in Task 5.

Step 5: Record the decision and tear down
If all three pass → decision: OT-Container-Kit redis-operator + Valkey; capture the working RedisReplication YAML for Task 3.
If TLS-with-Sentinel or the Valkey image fails → fallback: standalone Valkey + AOF + TLS via the official valkey-io/valkey-helm chart (drops auto-failover; see the spec’s Fallback section). Task 3 then deploys that instead; Tasks 2/4/5/6 are unchanged.

bd note hl-bqx2 "SPIKE RESULT: <operator chosen | fallback taken>; TLS-with-Sentinel <pass|fail>; valkey image <pass|fail>. Working manifest captured for Task 3."
kubectl delete namespace valkey-spike

Expected: bd note recorded; namespace deleted.

SPIKE-GATE BRANCH. Tasks 3 below is written for the OT-operator + Valkey path (Step-5 “all pass”). If the spike took the fallback, replace Task 3’s manifests with the official valkey-io/valkey-helm standalone+AOF+TLS deployment and swap Task 5’s clientType: sentinel for clientType: standalone (single connectionUrl). Tasks 2, 4, 5 (CA mount), and 6 are unchanged either way.

Task 2: Valkey namespace, password ExternalSecret, and TLS certificate

Files:

Create: argocd/app-configs/valkey/namespace.yaml
Create: argocd/app-configs/valkey/externalsecret.yaml
Create: argocd/app-configs/valkey/certificate.yaml
Create: argocd/app-configs/valkey/kustomization.yaml
Step 1: Write the namespace

argocd/app-configs/valkey/namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: valkey

Step 2: Write the password ExternalSecret (follows argocd/app-configs/keycloak/secrets.yaml)

argocd/app-configs/valkey/externalsecret.yaml:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: valkey-auth
  namespace: valkey
spec:
  refreshPolicy: Periodic
  refreshInterval: 5m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: valkey-auth
    creationPolicy: Owner
  data:
    - secretKey: redis-password
      remoteRef:
        key: fzymgc-house/cluster/valkey
        property: redis-password
    - secretKey: redis-sentinel-password
      remoteRef:
        key: fzymgc-house/cluster/valkey
        property: redis-sentinel-password

Step 3: Write the TLS Certificate (follows argocd/app-configs/keycloak/certificate.yaml)

argocd/app-configs/valkey/certificate.yaml — SANs cover the Service FQDNs oauth2-proxy/Sentinel will address; finalize the exact set from the spike’s captured manifest:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: valkey-tls
  namespace: valkey
spec:
  secretName: valkey-tls
  issuerRef:
    name: vault-issuer
    kind: ClusterIssuer
  dnsNames:
    - valkey.valkey.svc.cluster.local
    - valkey-headless.valkey.svc.cluster.local
    - valkey-sentinel.valkey.svc.cluster.local
    - "*.valkey-headless.valkey.svc.cluster.local"
  usages: [server auth, client auth]

Step 4: Write the kustomization

argocd/app-configs/valkey/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - externalsecret.yaml
  - certificate.yaml

Step 5: Validate the render

Run: kubectl kustomize argocd/app-configs/valkey Expected: three objects render (Namespace, ExternalSecret, Certificate); no errors.

Step 6: Lint

Run: yamllint argocd/app-configs/valkey/ Expected: no errors.

Step 7: Commit

Commit using VCS-appropriate commands per references/vcs-preamble.md. Message: feat(valkey): namespace, password ExternalSecret, and vault-issuer TLS cert [hl-bqx2].

Task 3: Valkey HA app (operator + RedisReplication) — OT-operator path

Files:

Create: argocd/cluster-app/templates/valkey.yaml (ArgoCD Application: redis-operator chart + the argocd/app-configs/valkey path)
Modify: argocd/app-configs/valkey/kustomization.yaml (add the RedisReplication)
Create: argocd/app-configs/valkey/redis-replication.yaml
Step 1: Add the ArgoCD Application (two sources: operator chart + the app-configs path), following the shape of argocd/cluster-app/templates/oauth2-proxy.yaml

argocd/cluster-app/templates/valkey.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: valkey
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # sync operator+Valkey ahead of default-wave apps; the real cutover gate is Task 4 (manual HA verification) before Task 5 merges
spec:
  project: core-services
  sources:
    - chart: redis-operator
      repoURL: https://ot-container-kit.github.io/helm-charts
      targetRevision: "0.25.0"   # latest stable as of 2026-07-05; re-verify with `helm search repo ot-helm/redis-operator --versions | head`
      helm:
        releaseName: redis-operator
    - repoURL: https://github.com/fzymgc-house/selfhosted-cluster
      targetRevision: HEAD
      path: argocd/app-configs/valkey
  destination:
    server: https://kubernetes.default.svc
    namespace: valkey
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [ServerSideApply=true, CreateNamespace=true]
  ignoreDifferences:
    - kind: ExternalSecret
      group: external-secrets.io
      jqPathExpressions:
        - .spec.data[].remoteRef.conversionStrategy
        - .spec.data[].remoteRef.decodingStrategy
        - .spec.data[].remoteRef.metadataPolicy

Step 2: Add the RedisReplication (from the spike’s captured, working manifest — the sketch below is the grounded starting point)

argocd/app-configs/valkey/redis-replication.yaml:

apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: RedisReplication
metadata:
  name: valkey
  namespace: valkey
spec:
  clusterSize: 3
  kubernetesConfig:
    image: valkey/valkey:8-alpine
    imagePullPolicy: IfNotPresent
    redisSecret:
      name: valkey-auth
      key: redis-password
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: longhorn
        accessModes: [ReadWriteOnce]
        resources: { requests: { storage: 2Gi } }
  redisConfig:
    additionalRedisConfig: |
      appendonly yes
  TLS:
    ca: ca.crt
    cert: tls.crt
    key: tls.key
    secret:
      secretName: valkey-tls
  sentinel:
    size: 3
    image: valkey/valkey:8-alpine

Step 3: Register it in the kustomization

Modify argocd/app-configs/valkey/kustomization.yaml resources: — add - redis-replication.yaml.

Step 4: Validate the render

Run: kubectl kustomize argocd/app-configs/valkey | rg "kind: RedisReplication" Expected: the RedisReplication renders. (CRD-schema validation happens at ArgoCD sync.)

Step 5: Commit, then let ArgoCD sync

Commit: feat(valkey): HA RedisReplication (Valkey + Sentinel + TLS + AOF) via redis-operator [hl-bqx2]. After merge, ArgoCD syncs the valkey app.

Task 4: Verify Valkey HA (failover + TLS + persistence) — pre-cutover gate

Files: none (live verification).

Step 1: Pods + quorum

Run: kubectl -n valkey get pods -o wide and kubectl -n valkey get redisreplication,redissentinel Expected: 3 Valkey + 3 Sentinel pods Running/Ready; the CR reports healthy.

Then confirm the Sentinel master-group name matches Task 5’s config (guards against the mymaster mismatch): Run: kubectl -n valkey exec valkey-sentinel-0 -- valkey-cli -p 26379 sentinel masters | sed -n '1,3p' Expected: the group name is mymaster (== sessionStorage.redis.sentinel.masterName). If it differs, fix Task 5’s masterName before cutover.

Step 2: TLS connectivity

Run: kubectl -n valkey exec valkey-0 -- sh -c 'valkey-cli --tls --cacert /etc/redis/tls/ca.crt -a "$(cat /etc/redis/auth/password)" ping' Expected: PONG.

Step 3: Failover

Run: delete the master pod (as in Task 1 Step 4c), wait, re-check roles. Expected: Sentinel promotes a replica; a new master within the failover window.

Step 4: AOF persistence

Run: valkey-cli … set spikekey 1, then kubectl -n valkey delete pod valkey-0, wait Ready, then valkey-cli … get spikekey. Expected: 1 survives the restart.

Step 5: Record the gate

Run: bd note hl-bqx2 "Valkey HA verified: quorum OK, TLS PONG, failover OK, AOF survives restart. Safe to cut over oauth2-proxy."

Task 5: Cut oauth2-proxy over to the Valkey session store

Files:

Modify: argocd/cluster-app/templates/oauth2-proxy.yaml
Modify: argocd/app-configs/oauth2-proxy/secrets.yaml (Step 3 — mirror the valkey-auth ExternalSecret into the oauth2-proxy namespace)
Step 1: Add the ICA1 CA volume mount at a dedicated path (NOT the existing /etc/ssl/oauth2-proxy OIDC bundle mount)

In argocd/cluster-app/templates/oauth2-proxy.yaml, under helm.valuesObject, add to extraVolumes and extraVolumeMounts:

          extraVolumes:
            - name: ca-bundle            # existing — unchanged
              configMap:
                name: oauth2-proxy-ca-bundle
            - name: valkey-ca            # NEW
              secret:
                secretName: fzymgc-ica1-ca   # mittwald-replicated to every namespace
          extraVolumeMounts:
            - name: ca-bundle            # existing — unchanged
              mountPath: /etc/ssl/oauth2-proxy
              readOnly: true
            - name: valkey-ca            # NEW — separate path, no collision
              mountPath: /etc/ssl/valkey
              readOnly: true

Step 2: Set the Redis Sentinel session store via the chart’s native sessionStorage (keeps the password out of git)

Add to helm.valuesObject:

          sessionStorage:
            type: redis
            redis:
              existingSecret: valkey-auth        # ExternalSecret from Task 2 (must exist in oauth2-proxy ns — see Step 3)
              passwordKey: redis-password
              clientType: sentinel               # REQUIRED precondition for the sentinel.* block
              sentinel:
                masterName: mymaster              # OT-operator embeds Sentinel with a HARDCODED master group "mymaster" (v0.25.0 redisreplication_controller.go) — NOT the CR name. Verify via Task 1 Step 4(d) / Task 4 Step 1.
                passwordKey: redis-sentinel-password
                connectionUrls:
                  - rediss://valkey-sentinel.valkey.svc.cluster.local:26379

Step 3: Make the password secret available in the oauth2-proxy namespace

The valkey-auth secret from Task 2 lives in the valkey namespace; oauth2-proxy needs it in oauth2-proxy. Add a second ExternalSecret (same Vault path) in argocd/app-configs/oauth2-proxy/secrets.yaml targeting valkey-auth in the oauth2-proxy namespace (mirror Task 2 Step 2, namespace: oauth2-proxy). Commit it with this task.

Step 4: Point oauth2-proxy’s TLS trust at the mounted ICA1 CA

In the configFile, add: redis_ca_path = "/etc/ssl/valkey/ca.crt" (do NOT add redis_insecure_skip_tls_verify). Confirm the chart’s sessionStorage values and the configFile do not both set the redis store (use sessionStorage as the single source of truth; leave redis lines out of configFile except redis_ca_path, which has no sessionStorage equivalent).

Step 5: Verify no secret is in git

Run: rg -i "redis[-_](password|sentinel).*[:=].{6,}" argocd/ | rg -v "passwordKey|secretKey|property|existingSecret" || echo "CLEAN" Expected: CLEAN (only key names, never values).

Step 6: Validate render + lint

Run: helm template oauth2-proxy oauth2-proxy/oauth2-proxy --version 10.7.0 -f <(yq '.spec.sources[0].helm.valuesObject' argocd/cluster-app/templates/oauth2-proxy.yaml) 2>&1 | rg "SESSION_STORE_TYPE|REDIS_SENTINEL" and yamllint argocd/cluster-app/templates/oauth2-proxy.yaml Expected: rendered env shows OAUTH2_PROXY_SESSION_STORE_TYPE=redis and the sentinel connection/password env; lint clean.

Step 7: Commit

Commit: feat(oauth2-proxy): use Valkey Sentinel session store over TLS (ticket cookie) [hl-bqx2]. After merge, ArgoCD rolls oauth2-proxy — this invalidates existing cookie sessions (one-time re-login for all protected apps; expected).

Task 6: End-to-end verification and close

Files: none (live verification).

Step 1: oauth2-proxy healthy against Valkey

Run: kubectl -n oauth2-proxy logs deploy/oauth2-proxy | rg -i "redis|sentinel|error" | tail Expected: connects to the sentinel; no redis/TLS errors.

Step 2: Cookie is now a ticket — log into an oauth2-proxy-protected app (e.g. longhorn.fzymgc.house), then in DevTools → Application → Cookies:

Expected: a single _oauth2_proxy cookie ≤ ~200 B; the _1 chunk is gone.

Step 3: TLS end-to-end — confirm from Step 1 logs the connection is rediss:// and ICA1-verified (no insecure_skip_tls_verify), and that a protected-app login completes.
Step 4: Header headroom on Keycloak — with the oauth2-proxy cookie shrunk, re-run the header check from hl-g3sv:

Run: curl -sS -o /dev/null -w "%{http_code}\n" "https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration" Expected: 200 (and the ~4.7 KB cookie no longer rides along to id.fzymgc.house).

Step 5: Close the bead

Run: bd close hl-bqx2 --reason="oauth2-proxy on Valkey Sentinel session store over TLS; _oauth2_proxy cookie is now a ~150 B ticket; failover + AOF + TLS verified; no secret in git."

Notes for the implementer

Order matters: P1/P2 → Task 1 (gate) → 2 → 3 → 4 (gate) → 5 → 6. Do not cut over oauth2-proxy (Task 5) until Task 4 passes.
Reversibility: each task is its own PR; Task 5 is revertible (drop back to sessionStorage.type: cookie).
Spike-gate: if Task 1 took the fallback, adjust Task 3 (standalone Valkey chart) and Task 5 Step 2 (clientType: standalone, single connectionUrl) per the spec’s Fallback section; everything else holds.
Longhorn/Velero: the valkey namespace is stateful → backed up by default under the exclude-only Velero policy; no action needed.
Monitoring (required fast-follow, file a child bead): because Valkey now gates auth for every oauth2-proxy-protected app (per-request forwardAuth), wire alerts before considering this hardened — Valkey up, Sentinel quorum healthy, oauth2-proxy redis error-rate, and valkey-tls cert near-expiry — via the existing Grafana/Prometheus stack (use the homelab:grafana skill). Not a blocker for the store cutover (Tasks 1–6), but do not close the epic without it.