Authentik Decommission Design (post-Keycloak)

Authentik Decommission Design (post-Keycloak)

Design bead	hl-0ahs
Status	Draft (rev 4) — design-reviewer READY (r3); §9 decisions finalized
Date	2026-07-03
Supersedes/completes	epic hl-5g05 (Keycloak Phase 2), ADR hl-12i8 (IdP migration)
Related beads	hl-k8kp (NAS LDAP), hl-iawe (SMTP/Mailgun), hl-nknn (Tandoor retire — hard dependency), hl-cp6h (cert data-source), hl-0sr.29 (real MCP DCR — superseded by hl-ll5p), hl-qjh3 (ClickStack alerting)

Changelog

• rev 4: converted §9 from open questions to firm decisions — security-alert replacement committed (ClickStack/Grafana, P6-blocking bead); traefik/longhorn cutover + auth.fzymgc.house teardown specified. No open questions remain.
• rev 3 (design-review round 2): forward-auth changed to a single shared oauth2-proxy with per-route ?allowed_groups= overrides (per-app least-privilege authorization without per-app infra); corrected the MCP DCR status (real Keycloak DCR is already live, ADR hl-ll5p); fixed §2 standalone-file count and the traefik/longhorn Certificate composition.
• rev 2 (design-review round 1): added MCP OAuth + Tandoor OIDC as explicit inventory areas with teardown gates; corrected the “all OIDC migrated” claim, middleware file layout, and identity-federation grounding.

1. Context & motivation

The Keycloak migration (epic hl-5g05) moved the OIDC web-SSO apps and Kubernetes structured-auth off Authentik. k3s no longer trusts Authentik, and those apps authenticate against the fzymgc Keycloak realm. What remains is the long tail: Authentik is still running and still load-bearing for a set of non-OIDC capabilities Keycloak does not provide, plus two live OIDC providers that were not part of the web-SSO migration (MCP OAuth and Tandoor).

This design defines how to remove Authentik completely — the Helm release, its CNPG database, DNS, Vault paths, the tf/authentik Terraform module, and the main-cluster-authentik HCP Terraform workspace — without losing any capability still in use or stranding the cluster in a broken auth state mid-teardown, and to update all documentation, memory, and CLAUDE.md routing accordingly.

2. Current-state inventory

Authentik still owns nine areas. “Migrated” means a live Keycloak equivalent exists and consumers already use it; “leftover” means the Keycloak side is live and only Authentik-side deletion remains.

#	Capability	Live tf/authentik state	Keycloak side	Disposition
1	OIDC web SSO (argocd, clickstack, mealie, miniflux, karakeep, engram_ui, cloudflare-access, octopus)	6 files removed{}-only; mealie.tf/argocd.tf also hold retained groups	✅ live, consumers cut over	Delete group-free files; groups → teardown (§6.6)
2	Forward-auth / proxy — authentik_provider_proxy ×4 + embedded outpost	Live (agentgateway, temporal, hubble; uptime-kuma launcher-only)	❌ none (Keycloak can’t forward-auth)	Replace (§4), then tear down
3	MCP OAuth — authentik_provider_oauth2.mcp_public (PKCE)	Live	✅ leftover — Keycloak DCR + static mcp-public live; engram + agentgateway already on .../realms/fzymgc	Verify issuer, then delete (§6.3)
4	Tandoor OIDC — authentik_provider_oauth2.tandoor + app + Vault write	Live	❌ none (being retired, not migrated)	Gate on hl-nknn (§6.4)
5	NAS LDAP — authentik_provider_ldap + LDAP outpost + Prometheus scrape	Live	❌ none	Confirm unused, remove (hl-k8kp)
6	Users & groups — 3 authentik_user, ~22 authentik_group	Live	🟡 groups partial; users federated (GitHub IDP)	Delete redundant copy (§6.6)
7	Enrollment / invitation — flow + email + invitation stages (+1 event rule/transport)	Live	🟡 Keycloak registration off by design	Drop (unused)
8	Event notifications — 5 event_rule + 1 transport → security-alerts email	Live	❌ different model	Replace via observability (§6.8)
9	Deployment + infra — Helm 2026.5.3, CNPG db-authentik + pooler + shared postgres-cluster.yaml managed authentik role + cnpg/kustomization.yaml, ingress, cert, whoami, auth.fzymgc.house (CF tunnel/DNS via var.authentik_services), R2 media (r2.tf), Vault (cluster/authentik, terraform_authentik_admin policy, tfc_authentik jwt role), tf/uptime-kuma monitor, TFC workspace	Live	n/a	Tear down last

The crux: forward-auth blast radius

Authentik’s embedded outpost provides Traefik forwardAuth for 5 internal dashboards (all verified pointing at authentik-server.authentik.svc/outpost.goauthentik.io/auth/traefik):

Host	Auth middleware location	tf/authentik provider
agentgateway.fzymgc.house	argocd/app-configs/agentgateway/modern-auth.yaml (standalone)	agentgateway.tf
temporal.fzymgc.house	argocd/app-configs/temporal-server/ingress.yaml (inline Middleware)	temporal.tf
hubble.fzymgc.house	argocd/app-configs/hubble-ui/traefik-middleware.yaml (standalone)	hubble-ui.tf
Traefik dashboard	argocd/app-configs/service-ingresses/traefik-dashboard.yaml (Middleware+IngressRoute; reuses wildcard-fzymgc-house-tls)	none (untracked)
Longhorn dashboard	argocd/app-configs/service-ingresses/longhorn-dashboard.yaml (combined Middleware+Certificate+IngressRoute)	none (untracked)
status.fzymgc.net (uptime-kuma)	—	uptime-kuma.tf (launcher-only — auth already on Cloudflare Access)

Only 2 of 5 have a standalone middleware file (agentgateway, hubble); temporal (inline) and traefik/longhorn (combined) need partial edits, not file deletes. uptime-kuma is a free win. Keycloak is an OIDC IdP only, so these 5 need an oauth2-proxy shim before the pod dies.

3. Goals / non-goals

Goals

• Remove every Authentik resource, deployment, and its TFC workspace.
• Preserve per-app, least-privilege access control on the 5 forward-auth dashboards.
• Preserve security-event visibility (replace, not silently drop).
• Leave docs/memory/CLAUDE.md with no stale Authentik references.

Non-goals

• Building a Keycloak self-registration flow (deferred; registration_allowed = false by design).
• Changing the MCP auth model. Real cluster-internal Keycloak RFC-7591 DCR is already live (ADR hl-ll5p, 2026-07-02): agentgateway proxies anonymous /register to Keycloak; the static mcp-public client is only the discovery anchor/fallback. So Authentik removal does not affect MCP auth — the Authentik mcp_public_oauth.tf is a pure leftover. hl-0sr.29 (re-evaluate Authentik-native DCR) is moot / superseded by hl-ll5p; this design neither depends on nor changes DCR.
• Grafana (hl-nlxl) sunset — tracked separately.
• Resolving the operator’s pre-existing tf/keycloak working-copy divergence.

4. Key decision — shared oauth2-proxy with per-route group gating (least-privilege)

Chosen approach: a single shared oauth2-proxy instance; each dashboard’s Traefik forwardAuth call carries a per-route ?allowed_groups=<app>-users override — delivering per-app least-privilege authorization without per-app infrastructure. (Supersedes rev-1 coarse-gate and rev-2 per-instance designs.)

Why shared + per-route override

oauth2-proxy’s /oauth2/auth endpoint — the target of Traefik forwardAuth — accepts a documented per-request allowed_groups query parameter. One instance authenticates every dashboard, but each IngressRoute’s forwardAuth address gates on that dashboard’s own group. That yields per-app authorization (each dashboard reachable only by its group) with one Deployment, one Keycloak client, and one CA bundle. (Verify the allowed_groups query-param support against the pinned oauth2-proxy chart version before the plan locks it in.)

Trade-off accepted: a shared session/cookie + single client secret is a coarser isolation boundary than per-app instances — compromising the shared proxy or its client affects all dashboards. For homelab admin surfaces that is an acceptable exchange for the drop in moving parts; if per-app credential isolation is later wanted, split into per-app clients without changing the authorization model.

Component & data flow

Browser → Traefik (dashboard IngressRoute)
            │  middleware: oauth-errors      (401/403 → /oauth2/sign_in?rd)
            │  middleware: oauth-auth-<app>   (forwardAuth → oauth2-proxy
            │              /oauth2/auth?allowed_groups=<app>-users)
            ▼
        oauth2-proxy (shared)  ──OIDC code flow──▶  Keycloak (fzymgc realm)
            │   --provider=keycloak-oidc                client: oauth2-proxy
            │   groups claim in session                 groups: <app>-users (flat, direct membership)
            ▼
        cookie (domain=.fzymgc.house) → SSO across dashboards; per-route group gate

Each dashboard gets its own oauth-auth-<app> Traefik Middleware differing only in the ?allowed_groups=<app>-users query string; a shared oauth-errors middleware handles the 401/403 → sign-in redirect.

Keycloak side (tf/keycloak/)

• Per-app groups (new — these do not exist yet; the 12 existing keycloak_group mirror the OIDC apps): one flat <app>-users group each for agentgateway, temporal, hubble, traefik-dashboard, longhorn. Flat, not nested: oauth2-proxy gates on a single ?allowed_groups=<app>-users, and the shared mapper’s full_path=false means the groups claim carries only direct memberships — a nested -admin child would not satisfy the -users gate. Bind each human via keycloak_user_groups (mirrors the OIDC apps).
• One confidential client oauth2-proxy: redirect URIs for each dashboard host callback + the oauth2-proxy callback; secret to Vault cluster/oauth2-proxy/oidc; the shared groups client-scope attached so the session carries group membership for the per-route checks.

Deployment (argocd/app-configs/oauth2-proxy/)

One Deployment: --provider=keycloak-oidc, --oidc-issuer-url=https://id.fzymgc.house/realms/fzymgc, --reverse-proxy=true, --set-xauthrequest=true, --cookie-domain=.fzymgc.house, --whitelist-domain=.fzymgc.house, client + cookie secrets via ExternalSecret. Authorization is enforced by the per-route middleware query string, not a global --allowed-group.

• Split-horizon CA trust (known trap): id.fzymgc.house serves the internal ICA1 chain in-cluster. oauth2-proxy is Go → SSL_CERT_FILE-replaces-roots case → mount a combined ICA1 + Mozilla CA bundle via caFiles + useSystemTrustStore (verify exact flag vs pinned chart). Mirrors the miniflux/engram ICA1 fixes (hl-9lcj, hl-rs8m, hl-23vy). Restart-latent — get it right at creation.

Traefik middleware swap

Per §2 crux table, edit each of the 5 IngressRoutes to replace the Authentik forward-auth middleware with oauth-auth-<app> (+ the shared oauth-errors): standalone-file deletes for hubble + agentgateway; partial edits for temporal (inline ingress.yaml) and traefik/longhorn (combined service-ingresses/*.yaml). Delete argocd/app-configs/authentik/traefik-middleware.yaml.

Cutover safety

Cut one dashboard first (recommend traefik-dashboard or hubble), verify login

• that a non-member of <app>-users is denied (the query-param gate works), then roll the rest. Delete the Authentik proxy providers/apps only after all 5 are confirmed on oauth2-proxy.

5. Decommission sequencing

Dependency-ordered — identity and the pod die last.

Phase	Workstream	Gate to proceed
P0	Free wins: delete uptime-kuma proxy + the 7 OIDC removed{} files	main-cluster-authentik plan shows only removals/forgets
P1	Keycloak: create 5 per-app groups + the oauth2-proxy client; deploy the shared oauth2-proxy (+ CA bundle)	Instance healthy; test login + group-deny via ?allowed_groups= on a scratch route
P2	Cut 5 dashboards over one at a time (per-route middleware); delete Authentik proxy providers/apps	All 5 authenticate via oauth2-proxy; per-route group gate verified
P3	MCP OAuth leftover: confirm no consumer references the Authentik mcp-public issuer, then delete mcp_public_oauth.tf	grep/telemetry shows only .../realms/fzymgc; agentgateway + engram healthy
P4	Tandoor: gated on hl-nknn — delete tandoor.tf only once Tandoor is retired	hl-nknn closed; tandoor.tf has no live consumer
P5	NAS LDAP removal (hl-k8kp) + drop Prometheus LDAP scrape	Confirmed no LDAP consumer
P6	Drop enrollment/invitation flows; drop event-notification flows; file security-alert replacement bead (hl-qjh3)	§6.5 replacement bead filed
P7	Confirm humans in Keycloak; delete tf/authentik users/groups (lift prevent_destroy)	sean/dorothy/sophia present in realm (§6.6)
P8	Tear down deployment: Helm app, CNPG db-authentik + pooler, ingress, cert, whoami, DNS auth.fzymgc.house, Vault cluster/authentik, tf/authentik module, main-cluster-authentik TFC workspace	Nothing references Authentik — incl. P3+P4 done
P9	Doc / memory / CLAUDE.md sweep (also incrementally throughout)	rg -i authentik returns only historical spec/plan/ADR hits

6. Per-workstream teardown detail

6.1 OIDC leftovers

Delete the seven files containing only removed{} blocks once no other TF state references them. Zero runtime effect.

6.2 Forward-auth teardown

See §4. Teardown removes agentgateway.tf, temporal.tf, hubble-ui.tf, uptime-kuma.tf and the embedded-outpost middleware manifests (standalone deletes + inline/combined partial edits).

6.3 MCP OAuth leftover

Real cluster-internal Keycloak DCR is live (ADR hl-ll5p); the static tf/keycloak/mcp_public.tf client is the discovery anchor, and engram (agent-memory.yaml oidc.issuer) + agentgateway (mcp-engram.yaml, mcp-oauth-shared.yaml) already use https://id.fzymgc.house/realms/fzymgc. So mcp_public_oauth.tf is a removable leftover. Belt-and-suspenders: confirm (grep + ClickStack token-issuer telemetry) no live client still hits the Authentik issuer, then delete.

6.4 Tandoor OIDC — hard dependency on hl-nknn.

tandoor.tf is a live Authentik OIDC provider + app + Vault write. Tandoor is being retired, not migrated, so there is no Keycloak equivalent to cut over to. Deleting tandoor.tf (or the whole tf/authentik module at P8) before hl-nknn completes destroys Tandoor login. Gate: hl-nknn closed first.

6.5 NAS LDAP (hl-k8kp)

Confirm the NAS no longer binds the Authentik LDAP outpost, then remove authentik_provider_ldap, the LDAP outpost, and the monitoring-otel-scraper LDAP scrape job.

6.6 Identity teardown

Cross-file group invariant: users.tf references authentik_groups declared across many files (mealie.tf, agentgateway.tf, temporal.tf, hubble-ui.tf, uptime-kuma.tf, notifications.tf, argocd.tf, groups.tf, kubernetes-groups.tf, vault.tf). Terraform resolves refs at plan time, so every authentik_group deletion is deferred to this phase and done atomically with users.tf (rg -l 'resource "authentik_group"' tf/authentik/); earlier phases keep the groups. Exception: vault.tf is edited (its 2 groups removed) rather than deleted — it also holds data.vault_kv_secret_v2.authentik (the provider "authentik" token source), which must survive to the deployment teardown (§6.10). sean is grounded (data.keycloak_user.operator, sub ab096608-…). dorothy and sophia auto-provision via the GitHub social-login IDP (github_idp.tf, ADR hl-sqtn allow-auto-create, sync_mode = IMPORT) on first login — verify each exists before teardown. Their Authentik paperless_users / komodo_users memberships are moot: those apps are already retired (no Keycloak client, no argocd/app-configs/paperless*|komodo*). Then delete tf/authentik users.tf, groups.tf, kubernetes-groups.tf, and the retained Vault/enrollment groups (lift prevent_destroy in the same change).

6.7 Enrollment flows — drop.

All humans already exist / auto-provision in Keycloak; registration_allowed = false is intentional. Delete enrollment-flow.tf (flow, stages, invitation, bindings, and its invitation_created event rule/transport).

6.8 Security notifications — replace via observability (decision §9).

Authentik emails security-alerts on login_failed, suspicious_request, password_set, config_audit, and a per-user new-sign-in notice. File a follow-up (hl-qjh3 alignment) to rebuild equivalent alerting on Keycloak log events as ClickStack/HyperDX TILE alerts → the security-alerts Pushover contact before deleting notifications.tf + policies/; this also lets hl-iawe retire the shared SMTP secret. Keycloak brute-force detection continues independently.

6.9 Deployment + infra teardown (P8)

• ArgoCD: remove argocd/cluster-app/templates/authentik.yaml and argocd/app-configs/authentik/ (whoami, cert, ingress-route, jwks, secrets).
• CNPG: remove db-authentik.yaml + pooler-authentik.yaml, drop those 2 lines from the shared cnpg/kustomization.yaml, and remove the authentik entry from postgres-cluster.yaml’s spec.managed.roles (shared file — surgical edit, like vault.tf). Velero-verify before dropping the database.
• Cloudflare: remove the auth entry from var.authentik_services (tf/cloudflare/variables.tf → destroys the generated auth.fzymgc.house DNS record + tunnel ingress; a literal grep won’t find it) and tear down r2.tf’s Authentik-media integration (bucket fzymgc-authentik-media, media.auth.fzymgc.net, RW account token, cluster/authentik/media-storage Vault secret + policy grant) after a bucket-backup decision. Also delete the Traefik IngressRoute argocd/app-configs/authentik/ingress-route.yaml.
• Vault: remove vault_policy.terraform_authentik_admin (policy-terraform-workspaces.tf — cluster/authentik + .../media-storage paths) and vault_jwt_auth_backend_role.tfc_authentik (jwt-hcp-terraform.tf).
• Uptime Kuma: remove uptimekuma_monitor_http.authentik (tf/uptime-kuma/main.tf — monitors auth.fzymgc.house) + its output (main-cluster-uptime-kuma workspace).
• Terraform: delete tf/authentik; remove the main-cluster-authentik entry from tf/hcp-terraform local.all_workspaces (workspaces-as-code — operator applies the manager workspace to destroy it).

7. Documentation / memory / CLAUDE.md sweep

rg -il authentik currently hits 27 non-spec files (the live grep at teardown is authoritative). Concretely:

• CLAUDE.md (root, line 14): “Infrastructure as code (Vault, Authentik)” → “(Vault, Keycloak)”.
• tf/CLAUDE.md (lines 54, 61): remove the main-cluster-authentik workspace row + MCP note.
• docs/operations/authentik.md: delete (or archive with a “decommissioned , see ADR” stub).
• docs/operations/mcp-gateway-clients.md: stale — rewrite the “engram uses Authentik OAuth” / “Authentik mcp-public” lines to Keycloak.
• docs/operations/{hubble-ui,temporal}.md: rewrite auth sections for the shared oauth2-proxy + per-route group gate.
• docs/reference/{services,secrets,network,technologies,index}.md: drop Authentik rows / auth.fzymgc.house / cluster/authentik secret.
• docs/architecture/{overview,index}.md, docs/getting-started/*, docs/index.md, README.md, docs/operations/{index,hcp-terraform,vault,clickstack,octopus,cilium-migration,cloudflare,outbound-email}.md: prune references.
• ADR: add a decommission ADR (or addendum to hl-12i8) recording removal + the shared-oauth2-proxy forward-auth decision.
• Memory (engram): supersede the “Authentik decommissioning” in-progress records once done; store the shared oauth2-proxy + per-route allowed_groups forward-auth pattern and the oauth2-proxy split-horizon CA-bundle gotcha.

8. Risks & rollback

Risk	Mitigation
oauth2-proxy CA trust wrong → dashboards 500 on next restart (restart-latent)	Combined ICA1+Mozilla bundle at creation; smoke-test a pod restart before P2
Dashboard lockout during middleware swap	One-at-a-time cutover; Authentik stays up until all 5 verified
Shared oauth2-proxy = single blast radius	Accepted trade (§4); host-scoped route gating limits authZ scope; can split to per-app clients later without changing the model
?allowed_groups= query override (documented oauth2-proxy feature)	Low risk — confirm during the P1 smoke test; fall back to per-app instances if the pinned chart differs
Tandoor login destroyed if tandoor.tf/module deleted before hl-nknn	P4 + P8 gated explicitly on hl-nknn closed
MCP auth broken if a consumer still on Authentik issuer at P3	P3 verifies issuer via grep + ClickStack telemetry before delete
Deleting tf/authentik users destroys identities only in Authentik	P7 verifies Keycloak parity (GitHub-IDP auto-provision) first
Silent loss of security alerts	§6.5 replacement bead filed before P6
CNPG DB deletion irreversible	Velero backup + explicit operator confirmation at P8
TFC workspace-as-code footgun	Remove map entry, operator runs manager apply; never delete in UI

Rollback: each phase is a separate PR; until P8, Authentik is still deployed with state intact, so any phase reverts by restoring files + re-applying. After P8 (pod + DB gone) rollback means a fresh Authentik install — the point of no return; gate P8 on P3 + P4 done.

9. Decisions

Every choice is settled — no open questions carry into planning.

• Forward-auth = single shared oauth2-proxy with per-route ?allowed_groups=<app>-users overrides (per-app least-privilege authorization). Supersedes the rev-1 coarse gate and rev-2 per-instance design.
• Security alerts = rebuild the five Authentik security-event notifications (login_failed, suspicious_request, password_set, config_audit, new-sign-in) as ClickStack/HyperDX TILE alerts over Keycloak logs → the security-alerts Pushover contact (hl-qjh3 alignment; Grafana/Prometheus are sunset per hl-nlxl), filed as a P6-blocking follow-up bead so alerting lands before notifications.tf is deleted. This also frees hl-iawe to retire the shared SMTP secret. Not dropped; not re-emailed via SMTP.
• traefik-dashboard / longhorn-dashboard (no tf/authentik provider) both forwardAuth at the shared embedded-outpost endpoint (.../outpost.goauthentik.io/auth/traefik), so any Authentik proxy provider for them was created out-of-band. They cut over to oauth2-proxy identically at P2 (traefik-dashboard-users / longhorn-users); out-of-band Authentik objects are destroyed with the instance at P8 — no separate cleanup task.
• auth.fzymgc.house + R2-media teardown = at P8, remove the Traefik IngressRoute (argocd/app-configs/authentik/ingress-route.yaml), the Cloudflare tunnel/DNS (the auth entry in var.authentik_services, tf/cloudflare — the record is generated by "${svc}.fzymgc.house", so a literal grep won’t find it), and the whole tf/cloudflare/r2.tf Authentik-media integration (bucket fzymgc-authentik-media, custom domain media.auth.fzymgc.net, RW account token, cluster/authentik/media-storage Vault secret + policy grant) after a bucket-backup decision. Spans the main-cluster-cloudflare + main-cluster-vault workspaces, not just tf/authentik.

10. Follow-up beads to file (at plan-to-beads time)

• Keycloak: 5 per-app groups + the shared oauth2-proxy client + CA bundle (P1).
• Shared oauth2-proxy deployment (P1) + per-dashboard middleware cutover (P2) — one child per dashboard.
• MCP OAuth issuer verification + mcp_public_oauth.tf delete (P3).
• Security-alert replacement in ClickStack/Grafana (P6 dependency; hl-qjh3).
• auth.fzymgc.house DNS + Vault cluster/authentik teardown (P8).
• main-cluster-authentik TFC workspace removal (P8).
• Doc/memory/CLAUDE.md sweep (P9).
• Link hl-nknn (Tandoor, blocking), hl-k8kp (LDAP), hl-iawe (SMTP) as dependencies.