Keycloak Migration Phase 2 — OIDC App Cutover Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Migrate 9 Authentik OIDC web applications + the kubernetes OIDC client from Authentik to the Keycloak fzymgc realm, preserving the groups-claim authorization contract, with every app cutover individually reversible.

Architecture: Per-app Terraform keycloak_openid_client (+ groups + shared groups client-scope) in tf/keycloak (HCP main-cluster-keycloak, provider keycloak/keycloak ~> 5.0); credentials flow to Vault and on to each app via its existing ExternalSecret; each app’s issuer is repointed to https://id.fzymgc.house/realms/fzymgc. Authentik runs in parallel — each app’s Authentik provider is dropped only after its Keycloak SSO smoke test is green.

Tech Stack: Terraform (keycloak/keycloak, hashicorp/vault providers), HCP Terraform (Agent workspaces), Vault KV v2 + ExternalSecrets, ArgoCD GitOps, jj VCS.

Design spec: docs/engineering/specs/2026-06-29-keycloak-phase2-oidc-apps-design.md · Design bead: hl-5g05

Conventions used by every task

VCS (jj-first)

Per the repo’s jj-first policy (root CLAUDE.md → “Branch Protection” / “Worktree workflow”; the jj:jujutsu skill): one jj change per task, --no-pager/--git on diffs, rebase onto main@origin before pushing, feature branch + PR (branch protection blocks direct main). After a PR squash-merges: jj git fetch && jj rebase -s <root> -o main@origin --skip-emptied && jj bookmark delete <bm>, then jj new main@origin before the next task.

Terraform validation gate (the “test”)

Infra has no unit-test TDD; the pre-merge gate is:

cd tf/keycloak    # or tf/vault / tf/cluster-bootstrap per task
terraform init -backend=false && terraform validate
terraform fmt -check -recursive

Expected: Success! The configuration is valid. and no fmt diff. Never commit .terraform.lock.hcl (platform-specific; breaks HCP linux agents) — confirm it is untracked before every commit.

Per-app SSO smoke test (the real acceptance gate)

For each app cutover, after the HCP apply + issuer repoint + ExternalSecret refresh:

# 1. SSO login (browser): drive the app's login URL, confirm redirect to id.fzymgc.house
agent-browser open "https://<app>.fzymgc.house"
agent-browser snapshot -i        # click the OIDC/SSO login control
# confirm the address bar lands on https://id.fzymgc.house/realms/fzymgc/protocol/openid-connect/auth...
# complete login as operator (sean); confirm callback returns to the app authenticated

# 2. Group-claim assertion (for group-gated apps): inspect the issued token's groups claim
#    via the app's own behavior (e.g. Vault admin capabilities, ArgoCD admin UI, Mealie admin),
#    OR decode the access token (browser devtools / app debug) and assert groups=[...expected...].

For the bearer/admin read of realm state, use the public LE endpoint (no internal truststore): id.fzymgc.house/realms/master admin-cli direct grant with creds from k8s secret keycloak/keycloak-bootstrap-admin (see engram memory 7920109e). Exit gate per app: SSO green and the operator resolves to the expected in-app authorization.

Access-gating semantics (READ BEFORE TASK 6 / clickstack, karakeep)

Authentik policy_engine_mode="any" + group binding gates who can log in at all. Keycloak does not gate client access by group out of the box. Apps that read the groups claim self-gate (vault → policy, argocd → RBAC default-deny, mealie → admin/user). Login-only apps that relied purely on the Authentik access policy (clickstack hyperdx-user, karakeep karakeep-users) become reachable by any realm user under Keycloak. Decision (per design bead hl-5g05): ACCEPTED for the ~3-user all-trusted homelab; Keycloak authorization-services per-client gating is deferred. Each such task still creates the group and emits it in the claim, so a future authz-services pass (or an app that starts reading the claim) has the data.

The canonical per-app client recipe

Every standard web-app task creates one file tf/keycloak/<app>.tf from this concrete template, substituting the row from the Per-app parameter table below. <groups_block> and the default_scopes groups entry are present only for group-gated apps.

# tf/keycloak/<app>.tf — Keycloak OIDC client for <App> (Phase 2, hl-5g05)

resource "keycloak_openid_client" "<app>" {
  realm_id  = keycloak_realm.fzymgc.id
  client_id = "<client_id>"
  name      = "<App>"
  enabled   = true

  access_type           = "CONFIDENTIAL"
  standard_flow_enabled = true

  valid_redirect_uris = [<redirect_uris>]
  web_origins         = [<web_origins>]   # usually the app origin; "+" to mirror redirect hosts
}

# Read existing co-located secret, merge in the OIDC creds (preserves non-OIDC keys).
data "vault_kv_secret_v2" "<app>_existing" {
  mount = "secret"
  name  = "<vault_path>"
}

resource "vault_kv_secret_v2" "<app>" {
  mount = "secret"
  name  = "<vault_path>"

  data_json = jsonencode(merge(
    data.vault_kv_secret_v2.<app>_existing.data,
    {
      <oidc_id_key>     = keycloak_openid_client.<app>.client_id
      <oidc_secret_key> = keycloak_openid_client.<app>.client_secret
    }
  ))

  custom_metadata {
    max_versions = 5
    data = { managed_by = "terraform", application = "<app>" }
  }
}

Group-gated apps additionally include <groups_block>:

# nested example (vault/mealie); flat apps omit parent_id
resource "keycloak_group" "<app>_users"  { realm_id = keycloak_realm.fzymgc.id  name = "<app>-users" }
resource "keycloak_group" "<app>_admin"  { realm_id = keycloak_realm.fzymgc.id  name = "<app>-admin"  parent_id = keycloak_group.<app>_users.id }

# operator membership (non-exhaustive — does not clobber Phase-4 membership)
resource "keycloak_user_groups" "operator_<app>" {
  realm_id   = keycloak_realm.fzymgc.id
  user_id    = data.keycloak_user.operator.id
  exhaustive = false
  group_ids  = [keycloak_group.<app>_admin.id]
}

# attach the shared groups scope so the token carries the groups claim
resource "keycloak_openid_client_default_scopes" "<app>" {
  realm_id       = keycloak_realm.fzymgc.id
  client_id      = keycloak_openid_client.<app>.id
  default_scopes = ["profile", "email", "roles", "web-origins", "acr", "basic", keycloak_openid_client_scope.groups.name]
}

Per-app parameter table (concrete)

App (task)	`client_id`	Redirect URIs	Vault path	`oidc_id_key` / `oidc_secret_key`	Group-gated	Issuer repoint location
miniflux (T3)	`miniflux`	`https://miniflux.fzymgc.house/oauth2/oidc/callback`	`fzymgc-house/cluster/miniflux`	`oauth2_client_id` / `oauth2_client_secret`	no	`argocd/app-configs/miniflux/deployment.yaml:86`
octopus (T4)	`octopus`	`https://octopus.fzymgc.house/api/auth/oauth2/callback/authentik`	`fzymgc-house/cluster/octopus-oidc`	`oidc_client_id` / `oidc_client_secret`	no	`argocd/app-configs/octopus/deployment.yaml:123`
karakeep (T5)	`karakeep`	`https://karakeep.fzymgc.house/api/auth/callback/custom`	`fzymgc-house/cluster/karakeep`	`oidc_client_id` / `oidc_client_secret`	claim-only (`karakeep-users`)	`tf/authentik/karakeep.tf:70` well-known → argocd secret
clickstack (T6)	`clickstack-hyperdx`	`https://hyperdx.fzymgc.house/auth/sso/callback`	`fzymgc-house/cluster/clickstack`	`oidc_client_id` / `oidc_client_secret`	claim-only (`hyperdx-user`)	`argocd/cluster-app/templates/clickstack.yaml:23-25` (3 URLs)
engram-ui (T8)	`engram-ui`	`https://engram.fzymgc.house/auth/callback`	`fzymgc-house/cluster/engram-ui`	`oidc_client_id` / `oidc_client_secret`	no (`email` claim)	OIDC discovery (`agentgateway/mcp-engram.yaml`)
mealie (T9)	`mealie`	`https://mealie.fzymgc.house/login`, `https://mealie.fzymgc.house/login?direct=1`	`fzymgc-house/cluster/mealie`	`oidc_client_id` / `oidc_client_secret`	nested (`mealie-users`→`mealie-admins`)	`argocd/app-configs/mealie/deployment.yaml:89`

Note: the octopus redirect literal …/callback/authentik is the app’s Better-Auth provider id; keep the provider id authentik (just repointed) so the callback URL is unchanged. The karakeep well-known URL is written by tf/authentik; in Keycloak it is written by the new tf/keycloak/karakeep.tf (the app reads it via discovery). cloudflare-access (T7), argocd (T10), vault (T11), kubernetes (T12) are bespoke — see their tasks.

Implementation tasks

Task 1: PR 0 — expand the Keycloak Vault policy (prerequisite)

Files:

Modify: tf/vault/policy-keycloak.tf

Without this, every per-app vault_kv_secret_v2 write 403s at apply on the main-cluster-vault Agent run. Mirror the paths terraform-authentik-admin already grants (tf/vault/policy-terraform-workspaces.tf:81-193).

Step 1: Add CRUD grants for every per-app path + the dual-mount auth path

Replace the policy = <<-EOT … EOT body of vault_policy.keycloak so it keeps the existing two read grants and adds a paired data+metadata grant for each of cluster/{miniflux, octopus-oidc, karakeep, clickstack, cloudflare-access, engram-ui, mealie}, the wildcards cluster/argocd/* and cluster/vault/*, and the bare base path cluster/argocd (see the glob note below):

    path "secret/data/fzymgc-house/cluster/miniflux" {
      capabilities = ["create", "read", "update"]
    }
    path "secret/metadata/fzymgc-house/cluster/miniflux" {
      capabilities = ["create", "read", "update"]
    }
    # … repeat the data+metadata pair for: octopus-oidc, karakeep, clickstack,
    #    cloudflare-access, engram-ui, mealie, argocd/*, vault/* …

    # argocd writes to the BASE path cluster/argocd (Task 10 merges keycloak_* keys
    # alongside the co-located admin creds). A Vault glob `cluster/argocd/*` does NOT
    # match the bare `cluster/argocd`, so it needs its own explicit paired grant:
    path "secret/data/fzymgc-house/cluster/argocd" {
      capabilities = ["create", "read", "update"]
    }
    path "secret/metadata/fzymgc-house/cluster/argocd" {
      capabilities = ["create", "read", "update"]
    }

    # Vault provider state-refresh of the dual JWT backend (Task 11) reads mount info.
    # Mirror terraform-authentik-admin (policy-terraform-workspaces.tf:99-102).
    path "sys/mounts/auth/*" {
      capabilities = ["read"]
    }

    # Vault dual-mount (§7.3 of the spec) — lets tf/keycloak mount a 2nd OIDC backend
    path "auth/oidc-keycloak/*" {
      capabilities = ["create", "read", "update", "delete", "list", "sudo"]
    }
    path "sys/auth/oidc-keycloak" {
      capabilities = ["create", "read", "update", "delete", "sudo"]
    }

(Use the exact cluster/* path set from tf/vault/policy-terraform-workspaces.tf lines 118-190 as the authoritative template, plus the bare cluster/argocd base path and sys/mounts/auth/* read shown above — the latter two are required by Tasks 10 and 11 respectively and are not in that line range.)

Step 2: Validate

Run: cd tf/vault && terraform init -backend=false && terraform validate && terraform fmt -check Expected: Success! The configuration is valid., no fmt diff.

Step 3: Commit + open PR + merge

jj describe -m "feat(vault): grant terraform-keycloak-admin per-app + dual-mount paths [hl-5g05]"
jj bookmark create feat/keycloak-vault-policy -r @
jj git push -b feat/keycloak-vault-policy
gh pr create --head feat/keycloak-vault-policy --title "feat(vault): keycloak workspace per-app secret + dual-mount grants [hl-5g05]" --body "PR 0 prerequisite for Keycloak Phase 2 (hl-5g05). Without it every per-app vault_kv_secret_v2 write 403s at apply."

Step 4: Confirm the main-cluster-vault Agent run succeeds

Watch the HCP run (read-only): fzymgc-house:terraform skill or HCP UI → main-cluster-vault. Expected: apply green, policy terraform-keycloak-admin now lists the new paths. This run must be green before Task 2 merges.

Task 2: PR 1 — Foundation (group taxonomy + shared `groups` scope + operator memberships)

Files:

Modify: tf/keycloak/groups.tf (replace placeholder)
Create: tf/keycloak/client_scopes.tf

This lands the shared scaffolding every group-gated app depends on. Per-app group resources live in each app’s own <app>.tf (Tasks 9-12); this task creates only the shared groups scope + mapper. (Operator memberships are created alongside each app’s groups in its task, via keycloak_user_groups.)

Step 1: Write the shared groups client scope + membership mapper

Create tf/keycloak/client_scopes.tf:

# Shared optional client scope that emits a `groups` claim (group names, not full path).
# Attached as a DEFAULT scope only to clients that gate on groups (vault, mealie, argocd,
# clickstack, karakeep, kubernetes) via keycloak_openid_client_default_scopes in each app file.

resource "keycloak_openid_client_scope" "groups" {
  realm_id               = keycloak_realm.fzymgc.id
  name                   = "groups"
  description            = "Group memberships as a `groups` claim (Phase 2, hl-5g05)"
  include_in_token_scope = true
}

resource "keycloak_openid_group_membership_protocol_mapper" "groups" {
  realm_id        = keycloak_realm.fzymgc.id
  client_scope_id = keycloak_openid_client_scope.groups.id
  name            = "groups"

  claim_name = "groups"
  full_path  = false

  add_to_id_token     = true
  add_to_access_token = true
  add_to_userinfo     = true
}

Step 2: Replace the tf/keycloak/groups.tf placeholder with a header comment only

The file currently is a placeholder comment. Leave it as a documented anchor (per-app groups live in <app>.tf):

# Per-app Keycloak groups are defined in each app's own file (<app>.tf), mirroring
# tf/authentik. The shared `groups` client scope + membership mapper live in
# client_scopes.tf. This file is intentionally a placeholder anchor.

Step 3: Validate

Run: cd tf/keycloak && terraform init -backend=false && terraform validate && terraform fmt -check Expected: Success! The configuration is valid.

Step 4: Commit + PR + merge + confirm HCP run

jj describe -m "feat(keycloak): shared groups client-scope + membership mapper [hl-5g05]"
jj bookmark create feat/keycloak-foundation -r @
jj git push -b feat/keycloak-foundation
gh pr create --head feat/keycloak-foundation --title "feat(keycloak): Phase 2 foundation — groups client-scope [hl-5g05]" --body "PR 1 (Foundation) for Keycloak Phase 2 (hl-5g05). Shared groups scope; per-app clients follow."

Confirm the main-cluster-keycloak HCP run is green (scope groups exists in the realm).

Files:

Create: tf/keycloak/miniflux.tf
Modify: argocd/app-configs/miniflux/deployment.yaml:86 (issuer)
Step 1: Write tf/keycloak/miniflux.tf using the canonical recipe with the miniflux row: client_id="miniflux", redirect ["https://miniflux.fzymgc.house/oauth2/oidc/callback"], web_origins=["https://miniflux.fzymgc.house"], vault path fzymgc-house/cluster/miniflux, keys oauth2_client_id/oauth2_client_secret, no groups block.
Step 2: Validate — cd tf/keycloak && terraform init -backend=false && terraform validate && terraform fmt -check → Success!
Step 3: Commit + PR + merge the TF, confirm the main-cluster-keycloak run creates the miniflux client and writes cluster/miniflux.
Step 4: Repoint the app issuer. In argocd/app-configs/miniflux/deployment.yaml change the OAuth2 OIDC discovery URL from https://auth.fzymgc.house/application/o/miniflux/ to https://id.fzymgc.house/realms/fzymgc. Confirm the ExternalSecret for miniflux still maps oauth2_client_id/oauth2_client_secret → the app’s OAUTH2_CLIENT_ID/OAUTH2_CLIENT_SECRET env. Commit + PR + merge; let ArgoCD sync.
Step 5: Smoke test — drive https://miniflux.fzymgc.house, click OIDC login, confirm redirect to id.fzymgc.house/realms/fzymgc, complete login, land authenticated. (No group assertion — login-only.)
Step 6: Drop the Authentik provider (follow-up PR). Remove authentik_provider_oauth2.miniflux + its app + vault write from tf/authentik/miniflux.tf (leave the rest of the file). Validate tf/authentik, PR, merge. Only after Step 5 is green.

Files:

Create: tf/keycloak/octopus.tf
Modify: argocd/app-configs/octopus/deployment.yaml:123 (issuer)
Step 1: Write tf/keycloak/octopus.tf — recipe with client_id="octopus", redirect ["https://octopus.fzymgc.house/api/auth/oauth2/callback/authentik"] (keep the authentik provider-id segment; the app’s Better-Auth provider stays named authentik, repointed), web_origins=["https://octopus.fzymgc.house"], vault fzymgc-house/cluster/octopus-oidc, keys oidc_client_id/oidc_client_secret, no groups.
Step 2: Validate → Success!
Step 3: Commit + PR + merge TF, confirm client + cluster/octopus-oidc write.
Step 4: Repoint issuer in argocd/app-configs/octopus/deployment.yaml:123 — the value is a discovery URL https://auth.fzymgc.house/application/o/octopus/.well-known/openid-configuration; change it to https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration. Keep the Better-Auth provider id authentik. Commit + PR + merge; ArgoCD sync.
Step 5: Smoke test — SSO login at https://octopus.fzymgc.house → redirect to realm → authenticated.
Step 6: Drop Authentik provider (tf/authentik/octopus.tf) — follow-up PR, after Step 5 green.

Files:

Create: tf/keycloak/karakeep.tf
Modify: argocd/app-configs/karakeep/secrets.yaml (well-known/issuer source)
Step 1: Write tf/keycloak/karakeep.tf — recipe with client_id="karakeep", redirect ["https://karakeep.fzymgc.house/api/auth/callback/custom"], web_origins=["https://karakeep.fzymgc.house"], vault fzymgc-house/cluster/karakeep, keys oidc_client_id/oidc_client_secret. Add the karakeep-users group (flat, claim-only per the Access-gating note) + operator membership + the default_scopes groups attachment so the claim is emitted. Also write the well-known URL key the app consumes (mirror tf/authentik/karakeep.tf:70, value https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration) into the same Vault path.
Step 2: Validate → Success!
Step 3: Commit + PR + merge TF, confirm client + group + cluster/karakeep write (incl. the well-known key).
Step 4: Repoint — the app reads the well-known/issuer from the Vault key via its ExternalSecret (argocd/app-configs/karakeep/secrets.yaml); confirm it now resolves to the realm. No deployment-manifest issuer hardcode to change. Sync.
Step 5: Smoke test — SSO login at https://karakeep.fzymgc.house (NextAuth “custom” provider) → realm → authenticated. Note: karakeep is now reachable by any realm user (Access-gating note); verify login works, access-gating change is accepted.
Step 6: Drop Authentik provider (tf/authentik/karakeep.tf) — follow-up PR, after Step 5 green.

Files:

Create: tf/keycloak/clickstack.tf
Modify: argocd/cluster-app/templates/clickstack.yaml:23-25 (three endpoint URLs)
Step 1: Write tf/keycloak/clickstack.tf — recipe with client_id="clickstack-hyperdx", redirect ["https://hyperdx.fzymgc.house/auth/sso/callback"], web_origins=["https://hyperdx.fzymgc.house"], vault fzymgc-house/cluster/clickstack, keys oidc_client_id/oidc_client_secret. Add hyperdx-user group (flat, claim-only) + operator membership + default_scopes groups attachment.
Step 2: Validate → Success!
Step 3: Commit + PR + merge TF, confirm client + group + cluster/clickstack write.
Step 4: Repoint the THREE endpoint URLs. HyperDX hardcodes authorize/token/userinfo (not a discovery issuer). In argocd/cluster-app/templates/clickstack.yaml:23-25 replace:
- authorize → https://id.fzymgc.house/realms/fzymgc/protocol/openid-connect/auth
- token → https://id.fzymgc.house/realms/fzymgc/protocol/openid-connect/token
- userinfo → https://id.fzymgc.house/realms/fzymgc/protocol/openid-connect/userinfo
Commit + PR + merge; ArgoCD sync.
Step 5: Smoke test — SSO login at https://hyperdx.fzymgc.house → realm → authenticated. Access-gating change accepted (any realm user).
Step 6: Drop Authentik provider (tf/authentik/clickstack.tf) — follow-up PR, after Step 5 green.

Task 7: cloudflare-access (Wave 2 — edge, external config)

Files:

Create: tf/keycloak/cloudflare_access.tf
External: Cloudflare Zero Trust dashboard (or tf/cloudflare) — the IdP-consumer side
Step 1: Write tf/keycloak/cloudflare_access.tf — keycloak_openid_client client_id="cloudflare-access", CONFIDENTIAL, valid_redirect_uris = ["https://*.cloudflareaccess.com/cdn-cgi/access/callback"] (Keycloak wildcard; Cloudflare provides the team-domain callback), web_origins=["+"], write creds to fzymgc-house/cluster/cloudflare-access keys oidc_client_id/oidc_client_secret (direct write — dedicated path). No groups.
Step 2: Validate → Success!
Step 3: Commit + PR + merge TF, confirm client + cluster/cloudflare-access write.
Step 4: Repoint the Cloudflare Access IdP. In the Cloudflare Zero Trust dashboard (Settings → Authentication → Login methods → the OIDC IdP), update issuer/auth/token/certs URLs to the realm, and client id/secret to the new Keycloak values (from Vault). If tf/cloudflare manages this, update there instead. This is outside the cluster — confirm whether IaC or console (spec §11.3).
Step 5: Smoke test — access a Cloudflare-Access-protected hostname; confirm the IdP redirect goes to id.fzymgc.house/realms/fzymgc and login succeeds.
Step 6: Drop Authentik provider (tf/authentik/cloudflare-access.tf) — follow-up PR, after Step 5 green.

Task 8: engram-ui (Wave 2 — `email`-claim owner, hl-w5kl-adjacent)

Files:

Create: tf/keycloak/engram_ui.tf
Modify: argocd/cluster-app/templates/agent-memory.yaml (ExternalSecret/issuer for the engram UI lane)
Step 1: Write tf/keycloak/engram_ui.tf — recipe with client_id="engram-ui", redirect ["https://engram.fzymgc.house/auth/callback"], web_origins=["https://engram.fzymgc.house"], vault fzymgc-house/cluster/engram-ui, keys oidc_client_id/oidc_client_secret. No groups — engram authorizes on the email claim (ownerClaim=email). Ensure the client emits a verified email: the standard email scope carries email + email_verified; the operator account must have email set and email_verified=true (engram is fail-closed, memory a8f93cdf).
Step 2: Validate → Success!
Step 3: Commit + PR + merge TF, confirm client + cluster/engram-ui write.
Step 4: Repoint the engram UI lane to the realm issuer (ENGRAM_UI_ISSUER / discovery in agent-memory.yaml) and the new client id/secret. Cross-check engram memory 43e8bf68: this migrates the UI lane off Authentik onto Keycloak — the durable fix referenced by hl-w5kl. Do NOT touch the engram bearer/MCP lane (already Keycloak). Commit + PR + merge; sync.
Step 5: Smoke test — log into https://engram.fzymgc.house; confirm realm redirect + that the console shows the operator’s records (i.e. email claim asserted + email_verified:true; if empty/401, the operator account is missing verified email — fix in Keycloak, do not change ownerClaim).
Step 6: Drop Authentik provider (tf/authentik/engram_ui.tf) — follow-up PR, after Step 5 green. Coordinate with hl-w5kl before removing the Authentik engram-ui app.

Task 9: mealie (Wave 3 — nested group, env name-match)

Files:

Create: tf/keycloak/mealie.tf
Modify: argocd/app-configs/mealie/deployment.yaml:89 (issuer)
Step 1: Write tf/keycloak/mealie.tf — recipe with client_id="mealie", redirects ["https://mealie.fzymgc.house/login", "https://mealie.fzymgc.house/login?direct=1"], web_origins=["https://mealie.fzymgc.house"], vault fzymgc-house/cluster/mealie. Nested groups mealie-users → mealie-admins (parent_id). Vault merge must also (re)write oidc_user_group="mealie-users", oidc_admin_group="mealie-admins", oidc_client_id, oidc_client_secret, oidc_auth_enabled="true", oidc_signup_enabled="true" (mirror tf/authentik/mealie.tf:77-87). Add operator membership (to mealie-admins) + default_scopes groups attachment.
Step 2: Validate → Success!
Step 3: Commit + PR + merge TF, confirm client + nested groups + cluster/mealie merge write.
Step 4: Repoint issuer argocd/app-configs/mealie/deployment.yaml:89 — the value is a discovery URL https://auth.fzymgc.house/application/o/mealie/.well-known/openid-configuration; change it to https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration. The app matches OIDC_ADMIN_GROUP/OIDC_USER_GROUP env (names unchanged) against the groups claim. Commit + PR + merge; sync.
Step 5: Smoke test — SSO login; confirm operator (in mealie-admins) gets Mealie admin; decode token and assert groups contains both mealie-users and mealie-admins (nested-group propagation works).
Step 6: Drop Authentik provider (tf/authentik/mealie.tf) — follow-up PR, after Step 5 green.

Task 10: argocd (Wave 4 — infra-critical, Dex connector)

Files:

Create: tf/keycloak/argocd.tf
Modify: tf/cluster-bootstrap/argocd.tf (Dex connector issuer + secret field refs; local workspace)

ArgoCD is special: its OIDC client is a Dex connector configured in tf/cluster-bootstrap (local execution), reading a TF-managed k8s secret argocd-oidc-secret from Vault cluster/argocd fields authentik_client_id/authentik_client_secret.

Step 1: Write tf/keycloak/argocd.tf — keycloak_openid_client client_id="argocd", CONFIDENTIAL, redirects ["https://argocd.fzymgc.house/api/dex/callback", "https://localhost:8085/auth/callback"], web_origins=["https://argocd.fzymgc.house"]. Flat groups argocd-user, argocd-admin, cluster-admin + operator membership (to argocd-admin) + default_scopes groups attachment. Write creds to fzymgc-house/cluster/argocd with new keys keycloak_client_id/keycloak_client_secret (merge — preserves the existing authentik_* keys until cutover).
Step 2: Validate tf/keycloak → Success!. Commit + PR + merge; confirm client + groups + cluster/argocd keys written.
Step 3: Repoint Dex in tf/cluster-bootstrap/argocd.tf: change the Dex issuer (line ~177) to https://id.fzymgc.house/realms/fzymgc; change the argocd-oidc-secret source fields and the clientID/clientSecret refs from authentik.client_id/authentik.client_secret to keycloak.client_id/keycloak.client_secret (the k8s secret built from the new Vault keys). Keep insecureEnableGroups: true and the argocd-rbac-cm policy.csv group→role mappings unchanged (group names identical). Also revisit rootCAs (line ~181): it currently pins the internal ICA chain (fzymgc_ica1_ca_fullchain), but id.fzymgc.house is served the Let’s Encrypt cert on the frontend hop (argocd/app-configs/keycloak/ingress-route.yaml keycloak-public-tls), not the ICA cert. Dex must verify against the system/LE pool — remove the rootCAs pin (or confirm Dex’s Go TLS appends rootCAs to the system pool rather than replacing it). Step 5’s parallel-connector safety net catches a TLS failure before SSO breaks.

Correction (hl-9lcj): removing the rootCAs pin was wrong. It assumed Dex reaches id.fzymgc.house via the public LE cert, but in-cluster the host is split-horizoned to keycloak-internal’s internal ICA1 cert (coredns-custom.yaml, hl-rs8m), so Dex (system-roots-only) failed SSO with x509: certificate signed by unknown authority. Dex v2.44’s rootCAs appends to the system pool (it does not replace), so the pin was always safe against the LE cert. The pin was restored in argocd.tf — do not remove it again on the “would replace the pool” premise.
Step 4: Validate tf/cluster-bootstrap (terraform init -backend=false && terraform validate). This is a local-execution workspace → operator terraform apply (not VCS-triggered). Apply, confirm Dex restarts with the new connector.
Step 5: Smoke test — log into ArgoCD via SSO; confirm realm redirect; confirm operator (in argocd-admin) gets role:admin (RBAC g, argocd-admin, role:admin). Keep the Authentik connector path available until green (Dex can hold both — verify before removing).
Step 6: Drop Authentik provider (tf/authentik/argocd.tf) + remove the now-unused authentik_* keys from cluster/argocd — follow-up PR, after Step 5 green.

Task 11: vault (Wave 4 — lockout-capable, dual JWT mount)

Files:

Modify: tf/keycloak/vault.tf (add the dual JWT backend + groups; currently a data source)
Modify: argocd/app-configs/.../vault UI config if any (issuer is in the auth backend, not the app)

Vault is lockout-capable: never break the only auth path back in. Strategy: add a SECOND vault_jwt_auth_backend (mount oidc-keycloak) pointing at the realm, alongside the existing Authentik oidc mount, then cut the default over.

Step 1: Add the Keycloak client + groups in tf/keycloak/vault.tf — append keycloak_openid_client client_id="vault", CONFIDENTIAL, redirects ["https://vault.fzymgc.house/oidc/callback", "https://vault.fzymgc.house/ui/vault/auth/oidc/oidc/callback", "http://localhost:8250/oidc/callback"], web_origins=["https://vault.fzymgc.house"]; nested groups vault-users→vault-admin + operator membership (to vault-admin) + default_scopes groups attachment; write creds to fzymgc-house/cluster/vault/oidc keys client_id/client_secret (direct write).
Step 2: Add the dual vault_jwt_auth_backend oidc-keycloak in tf/keycloak/vault.tf (provider vault already present): mount path oidc-keycloak, type oidc, oidc_discovery_url = "https://id.fzymgc.house/realms/fzymgc", with oidc_client_id = keycloak_openid_client.vault.client_id and oidc_client_secret = keycloak_openid_client.vault.client_secret (direct TF attribute references — same file, gives an implicit dependency; do NOT round-trip through a data "vault_kv_secret_v2", which would create a write-then-read cycle). Add reader/admin roles mirroring tf/authentik/vault.tf:54-90 (groups_claim="groups", bound_audiences=["vault"], oidc_scopes=["openid","email","profile","groups"], the same redirect URIs, default_role="reader"). Do not modify the existing Authentik oidc mount.
Step 3: Validate → Success!. Commit + PR + merge; confirm the main-cluster-keycloak run creates the client, groups, the oidc-keycloak mount + roles, and the cluster/vault/oidc write. The Authentik oidc mount is untouched and still works.
Step 4: Smoke test the new mount WITHOUT removing the old. vault login -method=oidc -path=oidc-keycloak role=admin (browser) → realm redirect → confirm the operator (in vault-admin) receives the admin token policy. Confirm vault login -method=oidc -path=oidc (Authentik) STILL works.
Step 5: Cut the default + UI default only after Step 4 green: make oidc-keycloak the Vault UI default auth method; optionally keep oidc (Authentik) mounted as fallback through Phase 5.
Step 6: Drop the Authentik provider + oidc mount — follow-up PR (remove authentik_provider_oauth2.vault + the Authentik vault_jwt_auth_backend.oidc from tf/authentik/vault.tf), only after the Keycloak mount is confirmed and you have a non-OIDC break-glass (root/token) verified.

Task 12: kubernetes-oidc (Wave 5 — public client now; k3s structured-auth deferred)

Files:

Create: tf/keycloak/kubernetes.tf (the Keycloak public client — concrete now)
Follow-up (separate plan/bead): ansible/roles/k3s-common/ structured AuthenticationConfiguration

The k3s structured-auth rollout is a different subsystem (Ansible + apiserver restart + lockout risk) with open schema questions (spec §11.5). This task lands the Keycloak-side client now; it files a dedicated bead for the k3s rollout to be planned against the live cluster at wave-5 time.

Step 1: Write tf/keycloak/kubernetes.tf — keycloak_openid_client client_id="kubernetes", access_type="PUBLIC", standard_flow_enabled=true, pkce_code_challenge_method="S256", valid_redirect_uris=["http://localhost:8000", "http://localhost:18000", "urn:ietf:wg:oauth:2.0:oob"]. Flat groups k8s-admins, k8s-developers, k8s-viewers + operator membership (to k8s-admins) + default_scopes groups attachment. No Vault secret (public client, kubelogin).
Step 2: Validate → Success!. Commit + PR + merge; confirm the public kubernetes client + the three k8s groups exist.
Step 3: Smoke test the client in isolation — run kubelogin/kubectl oidc-login get-token against https://id.fzymgc.house/realms/fzymgc with client_id=kubernetes; confirm a token is issued and its groups claim contains k8s-admins. (Do not wire it into k3s yet.)
Step 4: File the k3s structured-auth rollout bead. Create a child of hl-5g05 for the dual-issuer AuthenticationConfiguration work (Ansible k3s-common template rewrite removing --oidc-* flags atomically with --authentication-config, two JWT authenticators trusting Authentik + Keycloak, controlled server restart, kubectl-token RBAC verification). It is authored as its own plan against the live cluster version (spec §8, §11.5). This task completes when the Keycloak client is verified (Step 3) and the bead is filed.

Notes for the implementer

Order is a hard dependency chain for the shared pieces: Task 1 (vault policy) → Task 2 (foundation scope) → Tasks 3-12. Within waves, apps are independent and may proceed in parallel, but respect wave order (1→5) for risk.
Reversibility: every app’s Authentik provider stays live until its Step 5 smoke test is green; Step 6 (drop Authentik) is always a separate, later PR.
tf/keycloak and tf/vault are Agent workspaces (VCS-triggered on merge). tf/cluster-bootstrap is Local (operator terraform apply). Plan PRs accordingly.
Out of scope here: roles overlay (Phase 4), forward-auth apps (Phase 3), grafana/tandoor decommission (hl-nlxl/hl-nknn), the k3s structured-auth rollout (filed in Task 12, Step 4).

Keycloak Migration Phase 2 — OIDC App Cutover Implementation Plan

Conventions used by every task

VCS (jj-first)

Terraform validation gate (the “test”)

Per-app SSO smoke test (the real acceptance gate)

Access-gating semantics (READ BEFORE TASK 6 / clickstack, karakeep)

The canonical per-app client recipe

Per-app parameter table (concrete)

Implementation tasks

Task 1: PR 0 — expand the Keycloak Vault policy (prerequisite)

Task 2: PR 1 — Foundation (group taxonomy + shared groups scope + operator memberships)

Task 3: miniflux (Wave 1 — login-only)

Task 4: octopus (Wave 1 — login-only)

Task 5: karakeep (Wave 1 — login-only, claim-only group)

Task 6: clickstack / HyperDX (Wave 1 — login-only, claim-only group, 3-URL issuer)

Task 7: cloudflare-access (Wave 2 — edge, external config)

Task 8: engram-ui (Wave 2 — email-claim owner, hl-w5kl-adjacent)

Task 9: mealie (Wave 3 — nested group, env name-match)

Task 10: argocd (Wave 4 — infra-critical, Dex connector)

Task 11: vault (Wave 4 — lockout-capable, dual JWT mount)

Task 12: kubernetes-oidc (Wave 5 — public client now; k3s structured-auth deferred)

Notes for the implementer

Task 2: PR 1 — Foundation (group taxonomy + shared `groups` scope + operator memberships)

Task 8: engram-ui (Wave 2 — `email`-claim owner, hl-w5kl-adjacent)