oauth2-proxy server-side session store (Valkey Sentinel HA)

• Bead: hl-bqx2
• Status: Design (approved 2026-07-05; design-review round 2)
• Related: hl-g3sv (Keycloak HTTP/2 max-header-list-size fix — the outage this cookie bloat helped trigger; already resolved and verified)

Problem

oauth2-proxy runs as a forward-auth service (Traefik forwardAuth middlewares) protecting several apps (temporal, longhorn-dashboard, hubble-ui, agentgateway, traefik-dashboard). It uses the chart default cookie session store, which embeds the full OAuth tokens (id / access / refresh — the id_token is inflated by the groups claim) into the session cookie. Measured live 2026-07-05, that cookie is ~4.7 KB, chunked as _oauth2_proxy_0 (3936 B) + _oauth2_proxy_1 (750 B), and is scoped to the parent domain .fzymgc.house (cookie_domains=[".fzymgc.house"]). So the browser sends it to every *.fzymgc.house host — including id.fzymgc.house (Keycloak, the IdP), which does not use oauth2-proxy at all.

That dead-weight cookie was the dominant contributor to the request headers exceeding Keycloak’s HTTP/2 SETTINGS_MAX_HEADER_LIST_SIZE (8192), producing the id.fzymgc.house login 500/431 (hl-g3sv). hl-g3sv raised Keycloak’s ceiling to 32 KB, which fixes the symptom — this design removes the cause of the bloat, cluster-wide.

Goal

Move oauth2-proxy to a server-side session store backed by Valkey with Sentinel-based HA and TLS, collapsing the _oauth2_proxy cookie from ~4.7 KB to a ~150-byte ticket (cookie name + ticket ID + secret; the encrypted session lives in Valkey).

This satisfies all four drivers identified during design:

1. Header-limit defense — small cookies never trip Keycloak (32 KB), Cloudflare’s public path (Cloudflare enforces per-header and total request-header limits — figures unverified here; treated as supporting rationale, not a load-bearing requirement), or any stricter app proxy.
2. Bandwidth/latency — stop shipping ~4.7 KB on every request to every *.fzymgc.house app.
3. Architectural correctness — server-side sessions are the oauth2-proxy best practice; the cookie store (full tokens in the browser) is a known foot-gun.
4. Security posture — access / id / refresh tokens leave the browser entirely; smaller blast radius if a cookie leaks.

Non-goals

• Ingress / service-mesh migration (Istio/Envoy). Tracked separately. Grounding established this is orthogonal: Envoy’s native oauth2 filter is also cookie-based (id/refresh tokens in cookies, no server-side store), and Istio’s RequestAuthentication only validates JWTs — interactive OIDC login still runs through oauth2-proxy as ext_authz. Cookie size is a function of the session-storage strategy, not the edge proxy. This design is deliberately mesh-neutral so the work is never wasted.
• Narrowing cookie_domains. Keep .fzymgc.house so cross-app SSO is preserved. A parent-domain cookie cannot exclude a single subdomain anyway; shrinking the cookie is the fix, not re-scoping it.
• Reducing Keycloak’s own cookies. They total ~2 KB and Keycloak 26 already drops the duplicate *_LEGACY cookies by default (a SameSite-cookie change shipped in the 26.0 line; the claim is verified against Keycloak 26 behavior, the exact upstream issue number is not cited to avoid an unverified reference). Not in play.

Architecture

browser ──(ticket cookie ~150 B)──▶ Traefik forwardAuth ──▶ oauth2-proxy
                                     (per-request auth call)   │  sessionStorage.type=redis
                                                               │  redis sentinel, rediss:// (TLS)
                                                               ▼
                                       Valkey HA (primary + replicas + Sentinel quorum),
                                       operator-managed, AOF persistence, TLS everywhere

• oauth2-proxy’s role is unchanged — a standalone forward-auth service. It works as Traefik forwardAuth today and drops in as Envoy ext_authz in any future mesh, keeping this mesh-neutral.
• cookie_domains=[".fzymgc.house"] stays; only the cookie contents change (full session → ticket).
• cookie_secret stays — it now encrypts the server-side session record rather than the whole payload.

Components

1. Valkey HA (new)

• Topology: 1 primary + 2 replicas, plus a Sentinel quorum (3) for automated failover.
• Persistence: AOF (appendonly yes) on a Longhorn PVC, so sessions survive pod restart/reschedule, not only failover.
• Namespace: dedicated valkey.
• Hardening: a PodDisruptionBudget sized to preserve Sentinel quorum, resource requests/limits, Prometheus metrics exporter, backed up per the exclude-only Velero policy (stateful → backed up). The PDB is a settled decision; only the concrete numeric thresholds are a plan-level detail.

2. Operator (selection via spike — first implementation task)

• Leading candidate: OT-Container-Kit redis-operator — Sentinel CRDs (RedisReplication + RedisSentinel), on OperatorHub, GitOps-friendly. Valkey support is on its roadmap, not a certified feature — today it works only via an arbitrary image: override, which is exactly why operator selection is gated behind a spike rather than assumed.
• Watch: the official valkey-io operator is not GA yet (community estimate: months out); revisit when it ships. hyperspike/valkey-operator is Valkey-native but Cluster-oriented, not Sentinel.
• Spike acceptance criteria: the chosen path MUST support (a) the Valkey image, (b) Sentinel auto-failover, and (c) TLS together with Sentinel (see TLS risk below). If none cleanly does all three, take the fallback in the next section (do not silently drop TLS or the HA intent).

3. Secrets & certificates

• Valkey ACL password, oauth2-proxy redis_password, and redis_sentinel_password: Vault → ExternalSecret into a Kubernetes Secret. oauth2-proxy consumes it natively via the chart’s sessionStorage.redis.existingSecret (+ passwordKey) — the password is never written into the git-committed configFile (satisfies “MUST NOT commit secrets to Git”). Verified against the oauth2-proxy chart 10.7.0 values.yaml (sessionStorage.redis.{existingSecret,passwordKey} exist).
• Valkey server certificate: cert-manager Certificate issued by the vault-issuer (ICA1), SANs covering the primary / replica / Sentinel Service FQDNs (and any per-pod names the operator requires).
• CA trust: oauth2-proxy trusts ICA1 via redis_ca_path, mounting the cluster-replicated fzymgc-ica1-ca (ca.crt, mittwald replicator replicate-to: "*") at a dedicated mount path /etc/ssl/valkey — a separate extraVolumeMount, NOT the existing /etc/ssl/oauth2-proxy OIDC CA-bundle mount (two volumes cannot share one mount path). So redis_ca_path=/etc/ssl/valkey/ca.crt.

4. oauth2-proxy configuration

Change argocd/cluster-app/templates/oauth2-proxy.yaml:

• Set the session store via the chart’s sessionStorage values (not the legacy configFile):
  • sessionStorage.type: redis
  • sessionStorage.redis.existingSecret: <vault-owned secret> (+ passwordKey) for the redis and sentinel passwords
  • Sentinel settings: use the chart’s sessionStorage.redis sentinel keys (sentinel.masterName, sentinel.connectionUrls over rediss://…:26379, sentinel password) — exact key names confirmed against chart 10.7.0 during the plan phase.
  • keep the bundled redis sub-chart disabled (we use the external operator-managed Valkey).
• Add a second extraVolume/extraVolumeMount for the ICA1 CA at /etc/ssl/valkey, and set redis_ca_path=/etc/ssl/valkey/ca.crt (do not use redis_insecure_skip_tls_verify).
• OIDC / forward-auth settings and the existing /etc/ssl/oauth2-proxy provider CA bundle are unchanged; cookie_secret, cookie_domains, cookie_expire/cookie_refresh unchanged unless tuning is warranted.

TLS (required)

TLS is a hard requirement, not a follow-up:

• Valkey presents a vault-issuer (ICA1) cert for client↔server, and TLS is enabled for replica↔primary replication and Sentinel communication as well.
• oauth2-proxy connects over rediss:// and verifies against the ICA1 CA (redis_ca_path).
• Risk (spike gate): Sentinel + TLS is historically fiddly — Sentinels must announce TLS ports and the client must speak TLS to the Sentinels, not just the data nodes. The chosen operator’s ability to do TLS-with-Sentinel cleanly is an explicit acceptance criterion of the operator spike.

Fallback (if TLS-with-Sentinel is not cleanly achievable)

If the operator spike cannot deliver TLS-with-Sentinel, fall back to standalone Valkey + AOF + TLS (official valkey-io chart), which keeps TLS and session durability but drops automatic failover. Definition of done for this branch, so it is not left ambiguous:

• The Sentinel-failover acceptance test (below) is replaced by: (a) an AOF-survives-restart test (restart/reschedule the Valkey pod; active sessions persist, no forced re-login), and (b) a documented manual-failover runbook.
• Auto-failover is explicitly deferred (a follow-up bead), adopted when the official Valkey chart ships Sentinel (#22) or the chosen operator’s TLS-with-Sentinel matures — not silently dropped.

Data flow & failover

1. Login: oauth2-proxy completes the Keycloak OIDC flow, writes the encrypted session to the Valkey primary, and returns a ticket cookie.
2. Every subsequent request: Traefik forwardAuth calls oauth2-proxy per request; oauth2-proxy validates the ticket and looks up the session in Valkey. (This per-request coupling is why a Valkey outage affects in-progress sessions, not just new logins — see failure modes.)
3. Failover: primary failure → Sentinel promotes a replica → oauth2-proxy rediscovers the new primary via Sentinel. No user re-login on failover (the point of Sentinel).

Error handling & failure modes

• Valkey / Sentinel quorum loss ⇒ oauth2-proxy cannot read/write sessions. Because forwardAuth runs per request, this does not merely block new logins — it locks out already-authenticated, mid-session users across all oauth2-proxy-protected apps (temporal, longhorn, hubble, agentgateway, traefik-dashboard) for the duration of the outage. This is the inherent cost of server-side sessions and materially raises the availability bar for Valkey. Keycloak (id.fzymgc.house) is unaffected — it does not use oauth2-proxy.
• SLO implication: Valkey’s effective availability target must be ≥ the aggregate availability expected of every protected app’s auth path. Alerting must page on Valkey/Sentinel quorum health, not just “is a pod up.”
• Mitigations: the HA topology (this design’s purpose), AOF persistence, PDB to protect quorum, and monitoring/alerts on: Valkey up, Sentinel quorum healthy, oauth2-proxy Redis error rate, and certificate near-expiry.
• Cert expiry: cert-manager auto-renews the ICA1-issued cert; alert on near-expiry.

Rollout (GitOps / ArgoCD sync-waves)

1. Deploy the operator + Valkey HA + certificates + secrets in an earlier sync-wave. Verify quorum and a manual failover test before touching oauth2-proxy.
2. Flip oauth2-proxy to the Redis store.
   • One-time cost: existing cookie sessions are invalidated at cutover, so every user of every oauth2-proxy-protected app re-logs-in once. Acceptable and communicated; fully reversible by reverting the oauth2-proxy config.

Mesh-awareness

oauth2-proxy stays a standalone auth service (Traefik forwardAuth now, Envoy ext_authz later); the Valkey Service is a stable, mesh-reusable endpoint. Nothing here blocks a future Istio/Envoy migration — that migration would keep oauth2-proxy + Valkey for OIDC login regardless.

Verification / acceptance criteria

• The _oauth2_proxy cookie is a single ticket ≤ ~200 B (measured via the same DevTools/curl method that found the bloat); the _1 chunk disappears.
• Sentinel failover (primary path): kill the Valkey primary; active sessions persist and users are not forced to re-login. (In the fallback branch, replaced by the AOF-survives-restart test + manual-failover runbook, per the Fallback section.)
• An oauth2-proxy-protected app (e.g., longhorn-dashboard) completes login end-to-end with TLS to Valkey (rediss://, ICA1-verified, no insecure_skip_tls_verify), and no redis password appears in any git-committed file.
• Header-size headroom on id.fzymgc.house re-confirmed (the oauth2-proxy cookie no longer rides along at ~4.7 KB).

Open items / first tasks

1. Operator selection spike — validate OT-Container-Kit redis-operator for Valkey image, Sentinel, and TLS-with-Sentinel; compare the official Valkey operator’s timeline. Record the decision (and any fallback taken) on hl-bqx2.
2. TLS-with-Sentinel validation — the known-fiddly bit; gates primary-path vs the fallback branch.
3. Namespace/naming, PVC sizing, resource requests/limits, and the concrete PDB threshold values.
4. Monitoring/alert wiring (Valkey up, Sentinel quorum, oauth2-proxy redis errors, cert expiry).
5. Confirm the exact chart 10.7.0 sessionStorage.redis sentinel keys during plan authoring — including the sessionStorage.redis.clientType: sentinel precondition (the sentinel.* fields only take effect when it is set), and the separate sessionStorage.redis.sentinel.existingSecret/passwordKey pair (which defaults to the main redis secret if unset — decide whether to set it explicitly).

Estimated shape

Bounded and incremental: (a) operator, (b) Valkey HA app + certs/secrets, (c) oauth2-proxy config flip — a few focused, individually-reversible PRs. Not a mesh migration.