Keycloak Migration Phase 2 — OIDC App Cutover Design

Design bead: hl-5g05 (epic) · Date: 2026-06-29 · Author: Sean Brandt (+ Claude)

Parent design: 2026-06-28-keycloak-migration-design.md (§6 phase table, §9 testing). This spec details Phase 2 only.

1. Context

Phase 0+1 (epic hl-jb6q, closed) delivered the Keycloak foundation and the MCP authorization-server slice: realm fzymgc, issuer https://id.fzymgc.house/realms/fzymgc, the tf/keycloak module (HCP workspace main-cluster-keycloak, provider keycloak/keycloak ~> 5.0), the mcp-public public client, and real RFC 7591 DCR. ADR hl-amad supersedes hl-xktj.

Phase 2 migrates the remaining Authentik OIDC applications to Keycloak. Authentik runs in parallel through Phase 5 — every cutover here is a reversible config flip with a known-good Authentik fallback. Authentik is not removed in this phase.

2. Scope

The parent spec §2 inventoried 13 OIDC providers. Two are already handled and two are being decommissioned, so Phase 2’s new cutover work is 9 web SSO apps + kubernetes-oidc (the RBAC tail):

App	Phase 2 status	Reason
argocd, vault, clickstack, karakeep, mealie, miniflux, octopus, engram-ui, cloudflare-access	Migrate	9 web SSO apps
kubernetes-oidc	Migrate — LAST	k3s RBAC; structured-auth dual-issuer (§8)
mcp-public	Done	Migrated in Phase 1 (tf/keycloak/mcp_public.tf)
grafana	Retire — do NOT migrate	Grafana-Labs sunset → ClickStack (epic hl-nlxl)
tandoor	Retire — do NOT migrate	Decommission tracked in hl-nknn (was undocumented)

Building a Keycloak client for an app mid-decommission is wasted work; grafana and tandoor keep authenticating against Authentik until their retirement beads land.

2.1 Settled decisions

Decision	Choice	Rationale
Group→token model	Groups-claim (shared groups client scope), roles-ready	All 5 authorizing apps read a groups claim; zero read roles. Mirrors Authentik 1:1, lowest cutover blast radius.
Roles	Deferred to Phase 4	Parent spec §6 sequences groups→roles into Phase 4. Phase-2 groups become the role-assignment containers — additive, not throwaway.
Delivery	Foundation PR + one PR per app	Smallest blast radius; clean per-app rollback; matches the per-app exit gate.
Cutover order	Risk-ascending	Prove the pattern on login-only leaves before touching lockout-capable apps.
kubernetes-oidc client	Public / PKCE (kubelogin)	Mirrors the existing Authentik kubernetes client (loopback + oob redirects); no secret to distribute.

3. Architecture

tf/keycloak is the source of truth for realm/clients/scopes/groups/protocol-mappers (parent spec §3). The module mirrors tf/authentik’s structure: one .tf per app.

3.1 Vault-policy prerequisite (PR 0 — lands before the Foundation PR)

The main-cluster-keycloak workspace authenticates to Vault with the terraform-keycloak-admin policy (tf/vault/policy-keycloak.tf), which today grants read-only on cluster/keycloak + the keycloak postgres path. Every per-app vault_kv_secret_v2 write in §3.3 targets a cluster/<app> path not in that policy, so it would 403 at apply (HCP agent), not at validate. This is a hard prerequisite, not an open item:

• Per-app secret writes: expand tf/vault/policy-keycloak.tf with create/update/read (data and metadata) on the paths the Keycloak workspace will own — cluster/{miniflux, octopus-oidc, karakeep, clickstack, cloudflare-access, engram-ui, mealie, argocd/*, vault/*} — mirroring the grants terraform-authentik-admin already carries (tf/vault/policy-terraform-workspaces.tf:81-193).
• Vault dual-mount (§7.3): add auth/oidc-keycloak/* and sys/auth/oidc-keycloak capabilities so the dual JWT backend can be mounted. (The existing authentik policy covers auth/oidc/* + sys/auth/oidc — the single mount, not a second one.)

tf/vault runs via the main-cluster-vault HCP Agent workspace (VCS-triggered on merge, like all Agent workspaces), so PR 0 is a PR whose main-cluster-vault run must succeed before PR 1 is merged.

3.2 Shared scaffolding (the Foundation PR)

File	Purpose
tf/keycloak/groups.tf	Replaces the placeholder with the per-app group taxonomy (§4). Nested groups where a tier hierarchy exists.
tf/keycloak/client_scopes.tf (new)	One optional groups client scope carrying a keycloak_openid_group_membership_protocol_mapper (claim_name = "groups", full_path = false, added to access + id + userinfo). Attached as a default scope only to clients that gate on groups.
tf/keycloak/users.tf (exists)	data.keycloak_user.operator (username sean). Operator group membership added via keycloak_user_groups (non-exhaustive, keyed on .id) so Phase-4-managed membership is not clobbered.

A single shared groups scope (rather than a mapper per client) is the context7-confirmed idiom: the mapper lives on the scope, and any client that needs group gating gets the scope as a default.

3.3 Per-app unit (one <app>.tf, one PR)

Each app PR contains:

1. keycloak_openid_client — access_type = "CONFIDENTIAL" for the 9 web apps, "PUBLIC" + pkce_code_challenge_method = "S256" for kubernetes. valid_redirect_uris and web_origins from the inventory (§6). Standard openid/email/profile scopes; the groups scope where the app gates on groups; a keycloak_openid_audience_protocol_mapper where the app validates aud.
2. The app’s group(s) (§4) + operator membership.
3. Write client_id/client_secret to Vault via vault_kv_secret_v2 using the existing merge pattern (read existing → merge OIDC keys) at the same path the app already consumes (§6).
4. Repoint the issuer to https://id.fzymgc.house/realms/fzymgc in the app’s manifest / tf / ansible (§6 lists each location).
5. Verify (§7), then drop the Authentik provider in a separate follow-up merge.

4. Group taxonomy & claim model

The taxonomy mirrors the existing Authentik groups 1:1 — no new group concepts. The authorization chain is unchanged from today: user → group membership → groups claim → app maps group name to access/role.

App	Groups	Shape	Authentik source
vault	vault-users, vault-admin	Nested (vault-admin ⊂ vault-users)	tf/authentik/vault.tf:16-24
mealie	mealie-users, mealie-admins	Nested (mealie-admins ⊂ mealie-users)	tf/authentik/mealie.tf:9-17
argocd	argocd-user, argocd-admin, cluster-admin	Flat	tf/authentik/argocd.tf:5-15
clickstack	hyperdx-user	Flat	tf/authentik/clickstack.tf
karakeep	karakeep-users	Flat (exists; authz unused today)	tf/authentik/groups.tf:9
kubernetes	k8s-admins, k8s-developers, k8s-viewers	Flat	tf/authentik/kubernetes-groups.tf
miniflux, octopus, cloudflare-access, engram-ui	—	None	login-only / email claim

4.1 Nested groups retire the Authentik workaround

Authentik nesting is broken (goauthentik/authentik#15159): child membership does not propagate, so vault.tf:147-153 (and grafana) double-bind the admin group directly as a workaround. Keycloak’s nested groups propagate correctly — a member of vault-admin is automatically counted in vault-users, so the token carries groups: ["vault-users","vault-admin"]. We model the nesting once via keycloak_group.parent_id and delete the workaround.

4.2 Apps without groups

miniflux, octopus, cloudflare-access do no per-group authorization today — any user who can authenticate via the IdP is admitted. There is nothing to mirror; the client needs no groups scope. engram-ui is special: it authorizes on the email claim (record ownership, engram 0.8.0 ownerClaim = email), not groups — see §6 / hl-w5kl.

5. Secret & issuer-repoint flow

Keycloak client (TF) ──client_id/secret──▶ Vault kv (vault_kv_secret_v2, merge)
                                              fzymgc-house/cluster/<app>
        ▲                                          │
        │ HCP applies tf/keycloak on merge         ▼
   id.fzymgc.house/realms/fzymgc          ExternalSecret (ClusterSecretStore) ──▶ app env/config
        ▲                                          │
        └──────── issuer repoint (per-app, §6) ────┘

• Client credentials: new client → new client_id/secret; written to the same Vault path the app’s ExternalSecret already reads (no ExternalSecret rewiring beyond a refresh). The vault_kv_secret_v2 merge pattern preserves co-located non-OIDC keys.
• Issuer: most apps hardcode a discovery issuer auth.fzymgc.house/application/o/<app>/; Phase 2 repoints each to the realm issuer. Locations vary (§6) — app manifest, TF local, or Ansible var. Exception: clickstack/HyperDX hardcodes three separate OAuth2 endpoint URLs against authentik.fzymgc.house (authorize/token/userinfo, clickstack.yaml:23-25), not a single discovery issuer — its repoint replaces all three, so a generic “swap the issuer” helper does not apply to it.

6. Per-app inventory (grounded)

App	Client	Redirect URIs	Vault key	Issuer repoint location	Notes
miniflux	confidential	https://miniflux.fzymgc.house/oauth2/oidc/callback	cluster/miniflux	argocd/app-configs/miniflux/deployment.yaml:86	login-only
octopus	confidential	https://octopus.fzymgc.house/api/auth/oauth2/callback/authentik	cluster/octopus-oidc	argocd/app-configs/octopus/deployment.yaml:123	callback path literal authentik is an app-side provider name; keep or rename app-side
karakeep	confidential	https://karakeep.fzymgc.house/api/auth/callback/custom	cluster/karakeep	tf/authentik/karakeep.tf:70 → argocd secret (well-known URL)	NextAuth “custom” provider; uses discovery
clickstack	confidential	https://hyperdx.fzymgc.house/auth/sso/callback	cluster/clickstack	argocd/cluster-app/templates/clickstack.yaml:23-25 (3 URLs)	HyperDX; login-only
cloudflare-access	confidential	https://.*\.cloudflareaccess\.com/cdn-cgi/access/callback (regex)	cluster/cloudflare-access	Cloudflare dashboard / tf/cloudflare (external SaaS, outside argocd)	IdP config lives in CF Access, not in-cluster
engram-ui	confidential	https://engram.fzymgc.house/auth/callback	cluster/engram-ui	OIDC discovery; tracked in agentgateway/mcp-engram.yaml	authorizes on email; durable fix for hl-w5kl owner-claim P0
mealie	confidential	https://mealie.fzymgc.house/login, …/login?direct=1	cluster/mealie (merge)	argocd/app-configs/mealie/deployment.yaml:89	groups via env OIDC_USER_GROUP/OIDC_ADMIN_GROUP (name match)
argocd	confidential	https://argocd.fzymgc.house/api/dex/callback, https://localhost:8085/auth/callback	cluster/argocd (fields authentik_client_id / authentik_client_secret)	tf/cluster-bootstrap/argocd.tf Dex connector (issuer line ~177; clientID/secret refs authentik.client_id/authentik.client_secret lines ~178-179)	groups via Dex claimMapping {groups:groups} + argocd-rbac-cm policy.csv. NB: the Dex secret argocd-oidc-secret reads cluster/argocd (with authentik_* field names), not the cluster/argocd/oidc that tf/authentik writes — the consumed path + authentik.* field-name refs both need updating during cutover
vault	confidential	…/oidc/callback, …/ui/vault/auth/oidc/oidc/callback, http://localhost:8250/oidc/callback	cluster/vault/oidc	new vault_jwt_auth_backend pointing at the realm (dual-mount, §7)	lockout-capable; groups_claim = "groups"
kubernetes	public/PKCE	http://localhost:8000, http://localhost:18000, urn:ietf:wg:oauth:2.0:oob	n/a (kubelogin)	ansible/roles/k3s-common/defaults/main.yml:58	lockout-capable; structured-auth (§8); LAST

7. Cutover order, delivery & rollback

7.1 Waves (risk-ascending)

Wave	Apps	Why this tier
1	miniflux, octopus, karakeep, clickstack	login-only, no authz claim — prove the client/issuer/secret pattern
2	cloudflare-access, engram-ui	edge (external config) + email-claim owner (unblocks hl-w5kl)
3	mealie	first group-claim authz (nested groups + env name-match)
4	argocd, vault	infra-critical / lockout-capable
5	kubernetes-oidc	k3s RBAC tail; structured-auth dual-issuer (§8)

7.2 Delivery

PR 0 (Vault policy, §3.1): expand terraform-keycloak-admin (merge → main-cluster-vault Agent run) — must land before PR 1, or every per-app secret write 403s at apply.

PR 1 (Foundation): groups.tf taxonomy, the shared groups client scope + membership mapper, and operator group memberships. Seeding membership here is a prerequisite for the per-app group-claim smoke tests (§9).

PRs 2..N (one per app): the per-app unit (§3.3), in wave order. Each app’s Authentik-provider removal is a separate follow-up PR, taken only after the app’s exit gate is green — so rollback within the cutover PR is “flip issuer/creds back,” and Authentik remains a live fallback until the removal PR.

7.3 Rollback & lockout mitigation

• General: parallel-run — rollback = do not cut over (revert the issuer/creds flip). Authentik provider dropped only post-green.
• Vault (lockout-capable): add a second vault_jwt_auth_backend (e.g. mount oidc-keycloak) pointing at the realm, alongside the existing Authentik oidc mount. Both live simultaneously; cut the default role over only after the Keycloak mount is verified. Never break the only path back into Vault.
• k8s (lockout-capable): the k3s AuthenticationConfiguration carries two JWT authenticators (Authentik + Keycloak issuers) trusting both transitionally, applied via the --authentication-config server flag + restart (Ansible config-file edit, not kubectl apply). No atomic apiserver flip.

7.4 Terraform hygiene

• Validate locally before PR: terraform init -backend=false && terraform validate.
• Never commit .terraform.lock.hcl (platform-specific; breaks HCP linux agents).
• Most app config is native TF. Realm components with no provider resource use the kcadm bootstrap-Job pattern (argocd/app-configs/keycloak/dcr-trusted-hosts-job.yaml); the quay.io/keycloak/keycloak:26.6.4 image has no awk/sed/cut/python3/curl — bash built-ins only. No new kcadm Jobs are anticipated for standard clients.

8. kubernetes-oidc (the RBAC tail)

Done last and separately because it is the one cutover that is not an in-cluster config flip:

• Client: keycloak_openid_client kubernetes, access_type = "PUBLIC", PKCE S256, redirects http://localhost:8000, http://localhost:18000, urn:ietf:wg:oauth:2.0:oob — mirrors the kubelogin flow.
• Groups: k8s-admins, k8s-developers, k8s-viewers; emitted in the groups claim; consumed by the existing ClusterRoleBindings (argocd/app-configs/k8s-oidc-rbac/).
• k3s auth: structured AuthenticationConfiguration (v1beta1) with a slice of two JWT authenticators (Authentik issuer + realm issuer), groups_claim: groups, applied via the --authentication-config server flag + a controlled server restart through Ansible (k3s-common). Trust is transitional — both issuers valid until Authentik is retired in Phase 5. Dual-issuer safety lives in the content of the config (two authenticator entries), not in a gradual flag migration.
• Flag removal is atomic, not additive: Kubernetes 1.30+ forbids --authentication-config coexisting with the legacy --oidc-* flags. The current template (ansible/roles/k3s-common/templates/k3s-config.yaml.j2:24-36) sets oidc-issuer-url/oidc-client-id/oidc-groups-claim etc. — those must be removed in the same restart that adds --authentication-config. The Ansible change is a template rewrite, not a one-line flag add.
• Exit gate: kubectl with a Keycloak-issued token resolves to the correct RBAC group under the dual-issuer config; the Authentik token path still works throughout.

9. Testing strategy

Per-app exit gate (parent spec §6/§9):

1. SSO login against Keycloak succeeds (redirect → realm → callback → app session).
2. Group-claim smoke test: the operator, placed in the app’s group (Foundation PR), resolves to the expected in-app authorization — e.g. vault-admin → Vault admin policy; mealie-admins → Mealie admin; argocd-admin → ArgoCD role:admin.
3. Token-claim assertion: inspect the issued token and confirm the groups claim carries the expected names (incl. the parent for nested groups).
4. Fallback intact: the Authentik login path for the same app still works (until its removal PR).
5. k8s (§8): kubectl with a Keycloak token maps to the right ClusterRole under the dual-issuer structured-auth config.

10. Out of scope / deferred

• Roles overlay (groups→realm/client roles) — Phase 4.
• Forward-auth apps (agentgateway, hubble-ui, temporal, uptime-kuma) — Phase 3 (oauth2-proxy).
• grafana / tandoor decommission — hl-nlxl / hl-nknn.
• engram-ui owner-claim memory continuity (old Authentik sub → new Keycloak identity) — hl-w5kl (upstream engram fix; do not solve locally).
• Authentik teardown — Phase 5 (gated on hl-ney2 backup + realm export).

11. Open items for the plan

1. Vault dual-mount details — the second vault_jwt_auth_backend (mount oidc-keycloak) is owned by the keycloak workspace (tf/keycloak/vault.tf, parallel to how tf/authentik/vault.tf owns the Authentik mount); this requires adding vault resources to tf/keycloak/vault.tf (which today holds only a data source — the vault provider block already exists in tf/keycloak/terraform.tf) and the auth/oidc-keycloak/* + sys/auth/oidc-keycloak grants from PR 0 (§3.1). Remaining detail for the plan: exact role/bound_audiences config and the default_role cutover sequencing.
2. argocd Dex repoint — issuer lives in tf/cluster-bootstrap (local execution), not an Agent-mode HCP workspace; sequence the Dex connector change with the Keycloak client landing.
3. cloudflare-access — IdP config is in the Cloudflare dashboard / tf/cloudflare, not argocd; confirm whether this is Terraform-managed or a console step.
4. octopus callback path — the redirect literal …/callback/authentik is an app-side provider name; decide keep-vs-rename app-side during cutover.
5. k8s structured-auth file — confirm the k3s AuthenticationConfiguration schema and the Ansible restart choreography against the running cluster version (incl. the atomic --oidc-* flag removal noted in §8).