Design: Migrate IdP from Authentik to Keycloak

Design bead: hl-jb6q Date: 2026-06-28 Status: Draft (pending design-reviewer gate) Author: Sean Brandt (with Claude) Supersedes: ADR hl-23m (mock-DCR short-circuit / “no IdP migration”) — see §1.

1. Framing: a re-evaluation of ADR hl-23m

This is a genuine evaluation, not a foregone migration. ADR hl-23m (2026-06) explicitly considered and rejected migrating the cluster IdP to a DCR-native provider (Keycloak/Zitadel), choosing instead a mock OAuth Dynamic Client Registration (RFC 7591) short-circuit inside agentgateway. Two premises behind that rejection have since collapsed:

Mock-DCR is structurally unusable against Authentik. The in-flight reversal (hl-0sr.26, spec 2026-06-28-mcp-dcr-honesty-design.md) found via a source read of agentgateway v1.3.0-alpha.1 (crates/agentgateway/src/mcp/auth.rs) that without an mcp.provider the gateway never serves /.well-known/oauth-authorization-server or the /register route, so the clientId short-circuit is unreachable; and the McpIDP provider enum is Auth0 | Keycloak | Okta only — no Authentik adapter exists or is planned. Live probes confirmed 404s on every registration path.
“DCR is too thin a reason” no longer holds. Real DCR on the agentgateway MCP path is now a primary driver, and the agentgateway Keycloak adapter does real RFC 7591 DCR-proxy today on shipped GA software (v1.3.1, 2026-06-22).

The NAS LDAP outpost cited in hl-23m as migration-blocking cost is confirmed unused and is dropped from scope entirely (see §8).

Decision criteria (operator-weighted)

Decisive, in priority order:

Time-to-real-DCR — how fast we get honest, spec-compliant RFC 7591 DCR.
Minimal operational surface — prefer one IdP, not two; minimize ongoing upgrade/maintenance toil.
Avoid upstream-dependency risk — minimize bets on other people’s roadmaps/merges.

Explicitly not decisive: preserving the Authentik investment (the 18 apps, flows, policies). Re-homing them is an acceptable cost.

Options & scoring

Criterion	A. Migrate to Keycloak	B. Stay + contribute agentgateway Authentik adapter	C. Status quo (static `mcp-public`)	D. Hybrid (Keycloak = MCP-only AS, permanent)
Time-to-real-DCR	✅ Now (shipped GA)	❌ Gated on Authentik 2026.8 (#8751) and an unwritten Rust adapter	❌ Never (mock-DCR reverted; static ≠ DCR)	✅ Now
Min operational surface	✅ One IdP end-state	✅ One IdP	✅ One IdP	❌ Two IdPs forever
Avoid upstream-dependency risk	✅ Only shipped software	❌ Two unshipped bets	✅ No new deps	⚠️ + dual-IdP coordination

Decision: A — full migration to Keycloak, executed via a phased D-style transition (Keycloak stood up as the MCP authorization server first, then the remaining apps migrate, Authentik decommissioned last). This captures D’s speed and de-risking while reaching A’s single-IdP end state — D’s “two IdPs” becomes a temporary migration state, not the design.

B is eliminated by criterion 3 (the only option requiring unshipped software). C is eliminated by criterion 1 (it structurally cannot deliver real DCR). D-permanent is eliminated by criterion 2.

2. Current Authentik footprint (grounded inventory)

Deployment: official authentik Helm chart (v2026.5.3) on CNPG Postgres (authentik-pooler-rw) + Redis, secrets via ExternalSecrets. Config managed in tf/authentik/* via goauthentik/authentik provider (pinned ~> 2025.12).

18 applications, three provider types:

Provider type	Count	Apps	Keycloak path
OAuth2 / OIDC	13	argocd, grafana, vault, clickstack, karakeep, mealie, miniflux, octopus, tandoor, engram-ui, cloudflare-access, kubernetes-oidc, mcp-public	Direct — Keycloak clients
Proxy / forward-auth (`forward_single`)	4	agentgateway, hubble-ui, temporal, uptime-kuma	oauth2-proxy + Traefik `forwardAuth` (§5)
LDAP outpost	1	nas	Dropped — unused (§8)

Additional Authentik machinery requiring parity: a custom enrollment flow with email invitation + Python expression policies (send_invitation_email.py, match_sensitive_audit.py); 5 event rules + notification transports (audit alerting); groups→policy-binding RBAC; and the MCP PKCE issuer auth.fzymgc.house/application/o/mcp-public/ that engram/agentgateway validate.

3. Target architecture

Keycloak via the official Keycloak Operator (kind: Keycloak), single realm fzymgc, instances: 3, ArgoCD-managed, proxy.headers: xforwarded.
Dedicated CNPG cluster cnpg-keycloak (mirrors Authentik’s CNPG pattern); DB creds and admin secret via ExternalSecrets from Vault; db-tls-mode: verify-server against the CNPG CA.
tf/keycloak Terraform module using keycloak/terraform-provider-keycloak, mirroring the tf/authentik structure (one .tf per app), driven by a new HCP Terraform workspace. Terraform is the source of truth for realm/clients/scopes/groups/protocol-mappers.
oauth2-proxy (a confidential Keycloak OIDC client) wired as a Traefik forwardAuth middleware, replacing the Authentik forward_single outpost for the 4 auth-less apps.
agentgateway mcpAuthentication repointed at Keycloak: provider: Keycloak, issuer = realm URL, jwks = realms/fzymgc/protocol/openid-connect/certs, audiences, resourceMetadata, no clientId (real DCR-proxy, not the mock short-circuit).

Hostname / issuer strategy

Keycloak’s issuer URL is embedded in every token, so it is fixed once, up front to avoid churn at decommission: Keycloak is published at id.fzymgc.house, realm issuer https://id.fzymgc.house/realms/fzymgc, throughout the parallel run. Authentik retains auth.fzymgc.house until Phase 5. At decommission, auth.fzymgc.house is retired, not repointed (a repoint would not change already-issued Keycloak token issuers and adds no value).

4. DCR data flow (the forcing function, end state)

MCP client → agentgateway (.well-known PRM + AS-metadata proxy)
          → agentgateway proxies POST /register ──anonymous──▶
              Keycloak /realms/fzymgc/clients-registrations/openid-connect
          ◀── real, UNIQUE client_id/secret ──────────────────
              (Keycloak Trusted-Hosts policy permits the agentgateway source)
client → PKCE authorize/token @ Keycloak realm → JWT
       → agentgateway Strict JWT validation (iss / jwks / aud)
       → forward → engram re-validates (defense in depth)

The agentgateway Keycloak adapter proxies the anonymous /register (no initial-access-token) and rewrites registration_endpoint in the AS metadata to the gateway’s own /client-registration (a documented workaround for Keycloak’s lack of CORS on client registration).

Security trade-off (must be documented in the implementing ADR): Keycloak disables anonymous DCR by default — the Trusted Hosts client-registration policy whitelists no hosts, “effectively disabling anonymous client registration.” We must explicitly trust the agentgateway-proxied registration source — the trusted entry is the agentgateway Service ClusterIP (stable), not a pod IP (pods reschedule). Mitigation: the /register route is only reachable through the gateway, which owns PRM and enforces Strict JWT on every other path; engram re-validates as defense in depth.

5. Forward-auth replacement

The 4 proxy apps run Authentik forward_single mode — they have no native auth and rely entirely on the Authentik outpost. Replacement: deploy oauth2-proxy as an OIDC client of Keycloak, exposed as a Traefik forwardAuth middleware; swap each app’s Traefik middleware from the Authentik outpost to oauth2-proxy. This is the closest 1:1 to forward_single and the standard OSS pattern (Keycloak ships no embedded proxy). Group/role gating is enforced via oauth2-proxy’s --allowed-group against Keycloak group claims, mirroring the current policy_engine_mode = "any" group bindings.

6. Migration phases (parallel-run; each reversible)

Phase	Scope	Exit gate
0 — Foundation	CNPG `cnpg-keycloak`; Keycloak Operator; `kind: Keycloak`; realm `fzymgc`; `tf/keycloak` skeleton + HCP workspace; `id.fzymgc.house` ingress + cert	Keycloak admin reachable; TF plan/apply clean
1 — MCP AS slice ⭐	`mcp-public` client in Keycloak; repoint agentgateway `mcpAuthentication` (`provider: Keycloak`); configure Trusted-Hosts for the DCR proxy; engram re-validates Keycloak issuer	Two registrations yield distinct `client_id`s (the real-DCR property mock/static both fail); full Claude MCP DCR + PKCE + tool call works. hl-23m superseded here.
2 — OIDC apps (13)	Per app: TF client + protocol-mappers + groups; repoint app OIDC (issuer/client-id/secret via ExternalSecret) to Keycloak; verify SSO; drop Authentik app. k8s OIDC RBAC last, using Kubernetes structured authentication config — an `AuthenticationConfiguration` with a slice of JWT authenticators (v1beta1, default-on since 1.30), applied on k3s via the `--authentication-config` server flag (a k3s config-file edit + server restart, not a `kubectl apply`) — to trust both issuers transitionally	Per app: SSO + group-claim smoke test green; no apiserver hard-flip
3 — Forward-auth (4)	Deploy oauth2-proxy; swap Traefik middleware Authentik→oauth2-proxy on agentgateway, hubble-ui, temporal, uptime-kuma; drop Authentik proxy providers + outpost	Unauth→Keycloak→app redirect + group gating + logout work
4 — Parity	Keycloak realm registration flow + email (replaces enrollment flow + `send_invitation_email.py`); Keycloak event listener → existing alerting/ClickStack (replaces 5 event rules + `match_sensitive_audit.py`); groups→realm/client roles; ~3 users + groups recreated	New-user registration works; audit events visible in ClickStack
5 — Decommission	Remove `tf/authentik`, Authentik Helm app + CNPG + Redis + secrets + DNS; write superseding ADR; update docs + engram memory; retire `auth.fzymgc.house`	Gated on `hl-ney2` (CNPG→B2 backup) validated for `cnpg-keycloak` + a realm export taken before any Authentik removal; Authentik fully removed; docs/memory/bd updated

Phase order is deliberate: Phase 1 front-loads the only urgent capability (real DCR) on the lowest-blast-radius surface and proves Keycloak end-to-end before any SSO app is touched. If Phase 1 disappoints, the sunk cost is one CNPG cluster and one client.

7. Error handling, rollback & recovery

Parallel run: Authentik stays fully live through Phases 0–4. Every app cutover is a config/DNS switch with a known-good Authentik fallback; roll back by not cutting over.
k8s OIDC: structured authentication config validates both issuers during transition — no atomic apiserver flag flip.
Phase 1 fallback: the current reverted state (static mcp-public against Authentik) — acceptable because it is where the cluster sits today.
Backup/restore: Keycloak has no Authentik-blueprint equivalent. Recovery story is Terraform (config) + CNPG Postgres backup (data), riding the in-flight Kopia→B2 work (hl-ney2). A realm export is taken before Phase 5.

8. Out of scope (YAGNI)

NAS LDAP outpost — confirmed unused; dropped entirely, not migrated.
SAML — none in use.
Multi-realm / Keycloak Organizations — single realm fzymgc.
Custom Keycloak SPIs / Java extensions — none; Python expression policies are replaced by built-in Keycloak flows + event listeners, not ported as code.

9. Testing strategy

Per-phase exit gates (§6) are the acceptance criteria. Cross-cutting:

Phase 1 real-DCR assertion: two POST /register calls return distinct client_ids — the precise property that mock-DCR and static-client both fail.
Token-claim matrix: for each migrated app, assert the expected group/role claims map to the correct in-app authorization.
Forward-auth: unauthenticated → Keycloak redirect → callback → app; group-denied user is rejected; logout clears session.
k8s OIDC: kubectl with a Keycloak-issued token resolves to the correct RBAC group under the structured-auth config.
Backup/restore drill: restore cnpg-keycloak from backup + terraform apply reproduces the realm.

10. Open items for the plan / implementing ADRs

Trusted-Hosts DCR policy posture — exact source-trust config for the agentgateway proxy path (use the agentgateway Service ClusterIP, not a pod IP); document the security trade-off in an ADR.
Issuer cutover coordination — the engram/mcp-gw client config and any stored issuer references (incl. engram memory records) must be re-pointed when Phase 1 lands.
oauth2-proxy topology — one shared instance vs per-app; cookie/session store (Redis reuse?).
Superseding ADR — record the hl-23m re-evaluation outcome and the anonymous-DCR security decision as first-class ADRs at implementation time.