Authentik Decommission Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Remove Authentik from the cluster entirely — deployment, database, DNS, Vault paths, tf/authentik module, and the main-cluster-authentik TFC workspace — replacing its embedded-outpost forward-auth with a shared oauth2-proxy, without breaking any live auth path.

Architecture: Keycloak (fzymgc realm) is already the IdP for all OIDC apps, k3s, and MCP DCR. This plan (a) stands up one shared oauth2-proxy (keycloak-oidc) fronted by Traefik forwardAuth with per-route ?allowed_groups=<app>-users overrides to preserve per-app least-privilege on 5 internal dashboards, (b) tears down every Authentik capability in dependency order (proxies → MCP/Tandoor leftovers → LDAP → flows → identity → deployment), and (c) sweeps docs/memory/CLAUDE.md. Each phase is one PR; HCP TFC applies on merge, ArgoCD syncs manifests.

Tech Stack: Terraform (keycloak provider ~>5.8, vault, authentik provider — being removed), HCP Terraform (agent workspaces), ArgoCD + Kustomize, Traefik Middleware/IngressRoute CRDs, oauth2-proxy chart, External Secrets Operator, Vault KV-v2, CNPG.

Source of truth: design spec docs/engineering/specs/2026-07-03-authentik-decommission-design.md (design bead hl-0ahs).

Model intent (for plan-to-beads, Rule 5): mechanical file-delete / TF-forget tasks (T0, T4, T6, T7 Step 3, T8, T10) → model:sonnet. Live-auth cutover and irreversible teardown (T1–T3 oauth2-proxy stand-up + dashboard cutover; T9 deployment/DB/DNS teardown = point of no return) → model:opus, operator-driven. T5 (Tandoor) → model:sonnet, gated on hl-nknn. T7’s HyperDX-alerting build (Steps 1-2) is filed as its own observability child bead → model:sonnet.

Ground rules (read once)

GitOps only. Never kubectl apply or manually trigger a TFC run. Each task ends by committing to the phase branch and opening a PR; merge to main triggers the TFC plan/apply and ArgoCD sync. “Verify apply” = read the TFC run + argocd app get / smoke test, not a manual apply.
Terraform preview is read-only. terraform plan locally (or the Terraform MCP get_run_details) previews; it does not apply.
Vault dueling-writer rule (from hl-5g05 memory): tf/authentik and tf/keycloak both declared vault_kv_secret_v2.<app> at secret/fzymgc-house/cluster/<app>. Plain-deleting the tf/authentik resource destroys the shared secret. Always forget via removed { from = ...; lifecycle { destroy = false } }, never a bare delete. Confirm the plan shows “Plan to remove” (forget), never “destroy”, for any vault_kv_secret_v2.
Standing replacement cascade (memory hl-je6h): main-cluster-keycloak plans carry a baseline ~9-to-add / 9-to-destroy from the provider ~>5.8 cascade. A keycloak PR’s real footprint is the delta above that baseline. Don’t panic at 9/9.
Split-horizon CA (memory hl-9lcj): id.fzymgc.house serves the internal ICA1 chain in-cluster; oauth2-proxy (Go) needs a combined ICA1+Mozilla bundle. Failures are restart-latent — smoke-test a pod restart, not just first boot.
Each phase is gated: do not start phase N+1 until phase N’s PR is merged and verified.

Files touched (decomposition map)

Created

tf/keycloak/oauth2_proxy.tf — 5 flat <app>-users groups + operator bindings, one confidential oauth2-proxy client, groups scope binding, Vault secret write.
argocd/cluster-app/templates/oauth2-proxy.yaml — ArgoCD Application (oauth2-proxy chart).
argocd/app-configs/oauth2-proxy/kustomization.yaml
argocd/app-configs/oauth2-proxy/namespace.yaml
argocd/app-configs/oauth2-proxy/secrets.yaml — ExternalSecret (client + cookie secret).
argocd/app-configs/oauth2-proxy/ca-bundle.pem — vendored combined ICA1+Mozilla bundle, surfaced via a configMapGenerator in kustomization.yaml (matches miniflux/agent-memory).
argocd/app-configs/oauth2-proxy/middlewares.yaml — one oauth-auth-<app> Middleware per dashboard + shared oauth-errors.
docs/adr/hl-0ahs-decommission-authentik.md — decommission ADR (auto-filed by capture-adrs).

Modified (middleware swap — hubble & agentgateway keep Middleware and IngressRoute in separate files)

argocd/app-configs/hubble-ui/ingress-route.yaml (swap ref) + delete traefik-middleware.yaml + drop from hubble-ui/kustomization.yaml
argocd/app-configs/agentgateway/ui-ingress.yaml (swap ref) + delete modern-auth.yaml + drop from agentgateway/kustomization.yaml
argocd/app-configs/temporal-server/ingress.yaml (inline Middleware — partial edit)
argocd/app-configs/service-ingresses/traefik-dashboard.yaml (combined — partial edit)
argocd/app-configs/service-ingresses/longhorn-dashboard.yaml (combined — partial edit)

Deleted (teardown, dependency-ordered — all authentik_group deletions deferred to T8 because users.tf references them cross-file):

T0: 6 group-free OIDC removed{} files; providers trimmed from uptime-kuma.tf / mealie.tf (groups kept).
T3: proxies trimmed from agentgateway.tf / temporal.tf / hubble-ui.tf (groups kept); argocd/app-configs/authentik/traefik-middleware.yaml.
T7: enrollment-flow.tf, policies/; event rules trimmed from notifications.tf (security_alerts group kept).
T8: users.tf + every remaining group-holding file (groups.tf, kubernetes-groups.tf, argocd.tf, group-only remnants of mealie/uptime-kuma/agentgateway/temporal/hubble-ui/notifications); vault.tf is EDITED (groups removed) not deleted — its data.vault_kv_secret_v2.authentik (provider token) survives to T9.
T9: git rm -r tf/authentik (residual; mcp_public_oauth.tf gone T4, tandoor.tf T5, nas.tf T6); argocd/cluster-app/templates/authentik.yaml, argocd/app-configs/authentik/, argocd/app-configs/cnpg/{db-authentik,pooler-authentik}.yaml; main-cluster-authentik entry in tf/hcp-terraform local.all_workspaces; shared cnpg edits (kustomization.yaml −2 lines, postgres-cluster.yaml drop the authentik managed role); tf/cloudflare (auth entry in var.authentik_services + r2.tf authentik_media resources); tf/uptime-kuma (authentik monitor + output); tf/vault (vault_policy.terraform_authentik_admin + vault_jwt_auth_backend_role.tfc_authentik + cluster/authentik* grants) — separate main-cluster-cloudflare/vault/uptime-kuma PRs.

Task 0: Free wins — uptime-kuma proxy + OIDC `removed{}` leftovers

Group-retention invariant (applies to T0/T3/T7): tf/authentik/users.tf binds humans to authentik_groups declared across many files, and Terraform resolves those refs at plan time — so NO authentik_group may be deleted until users.tf goes in Task 8 (the repo documents this in vault.tf/mealie.tf comments). Earlier phases delete group-free files and edit group-holding files down to just their groups.

Files:

Delete (verified 0 groups, removed{}-only): tf/authentik/{clickstack,miniflux,karakeep,engram_ui,cloudflare-access,octopus}.tf
Edit (remove non-group resources, KEEP groups until T8): tf/authentik/uptime-kuma.tf (keep uptime_kuma_users), tf/authentik/mealie.tf (keep mealie_users/mealie_admins)
Step 1: Confirm the 6 delete-files are group-free removed{}-only

Run: rg -c 'resource "authentik_group"' tf/authentik/{clickstack,miniflux,karakeep,engram_ui,cloudflare-access,octopus}.tf; rg -L 'resource "authentik_(provider_oauth2|application|group)"' tf/authentik/{clickstack,miniflux,karakeep,engram_ui,cloudflare-access,octopus}.tf Expected: 0 groups in each; files hold only removed{} blocks + data sources. (mealie.tf retains 2 groups, uptime-kuma.tf 1 — those are edited, not deleted.) If any of the 6 declares a group or live provider, STOP.

Step 2: Edit uptime-kuma.tf and mealie.tf down to their groups

uptime-kuma is launcher-only (auth on Cloudflare Access — rg -n 'Cloudflare Access' tf/authentik/uptime-kuma.tf). Remove authentik_provider_proxy.uptime_kuma + authentik_application.uptime_kuma + authentik_policy_binding.uptime_kuma_access; keep authentik_group.uptime_kuma_users. In mealie.tf, remove the removed{} vault block + any data source; keep authentik_group.mealie_users/mealie_admins. Both group sets stay referenced by users.tf and are deleted in T8.

Step 3: Delete the 6 group-free files

Run: git rm tf/authentik/{clickstack,miniflux,karakeep,engram_ui,cloudflare-access,octopus}.tf (In jj: just rm them; the working copy auto-tracks the deletion.)

Step 4: Preview — removals/forgets + uptime-kuma destroy, NO group destroys

Run: cd tf/authentik && terraform validate && terraform plan Expected: validate succeeds (users.tf group refs still resolve). Plan: authentik_provider_oauth2.uptime_kuma + authentik_application.uptime_kuma + authentik_policy_binding.uptime_kuma_access destroy; removed{} blocks show vault_kv_secret_v2.<app> as “Plan to remove” (forget, NOT destroy). Zero authentik_group destroys; zero vault_kv_secret_v2 destroys. If a group or vault secret shows “destroy”, STOP.

Step 5: Commit + PR

Commit chore(authentik): remove uptime-kuma proxy + migrated OIDC leftover files [hl-0ahs], push the phase branch, open PR. Per references/vcs-preamble.md use jj: jj bookmark create authentik-decommission-p0 -r @ && jj git push -b authentik-decommission-p0.

Step 6: After merge — verify TFC applied cleanly

Check the main-cluster-authentik run applied; terraform plan on main shows no drift for these resources. The uptime-kuma app launcher entry disappears from the Authentik portal (cosmetic; auth still via Cloudflare Access at status.fzymgc.net).

Task 1: Keycloak side — per-app groups + oauth2-proxy client

Files:

Create: tf/keycloak/oauth2_proxy.tf
Reference: tf/keycloak/client_scopes.tf (existing shared groups scope + membership mapper), tf/keycloak/karakeep.tf (pattern for a per-app group + Vault write)
Step 1: Write tf/keycloak/oauth2_proxy.tf — the 5 flat groups

# Per-app groups + confidential client for the shared oauth2-proxy that replaces
# Authentik's embedded-outpost forward-auth (design hl-0ahs §4). Per-app
# least-privilege is enforced by the per-route ?allowed_groups=<app>-users
# override on each dashboard's Traefik forwardAuth address, NOT a global flag.

locals {
  oauth2_proxy_dashboards = ["agentgateway", "temporal", "hubble", "traefik-dashboard", "longhorn"]
}

resource "keycloak_group" "dashboard_users" {
  for_each = toset(local.oauth2_proxy_dashboards)
  realm_id = keycloak_realm.fzymgc.id
  name     = "${each.key}-users"
}

# FLAT groups, not nested. oauth2-proxy enforces one binary gate per route
# (?allowed_groups=<app>-users) — there is no admin/user tier to model (that was
# an Authentik policy-engine artifact). And with the shared mapper's
# full_path=false, the groups claim carries only DIRECT memberships, so a nested
# -admin child would NOT satisfy the -users gate anyway. Flat is simpler + correct.

# Bind the operator into every dashboard group so the gate has a member. WITHOUT
# this, the per-route gate denies EVERYONE (mirrors keycloak_user_groups.
# operator_karakeep). exhaustive=false so we don't clobber other memberships.
# Add family members here as needed.
resource "keycloak_user_groups" "operator_dashboards" {
  for_each   = toset(local.oauth2_proxy_dashboards)
  realm_id   = keycloak_realm.fzymgc.id
  user_id    = data.keycloak_user.operator.id
  exhaustive = false
  group_ids  = [keycloak_group.dashboard_users[each.key].id]
}

Step 2: Add the confidential client + groups-scope binding to the same file

resource "keycloak_openid_client" "oauth2_proxy" {
  realm_id  = keycloak_realm.fzymgc.id
  client_id = "oauth2-proxy"
  name      = "oauth2-proxy (dashboard forward-auth)"
  enabled   = true

  access_type           = "CONFIDENTIAL"
  standard_flow_enabled = true

  # One redirect URI per protected dashboard host; oauth2-proxy callback is /oauth2/callback.
  valid_redirect_uris = [
    "https://agentgateway.fzymgc.house/oauth2/callback",
    "https://temporal.fzymgc.house/oauth2/callback",
    "https://hubble.fzymgc.house/oauth2/callback",
    "https://traefik.k8s.fzymgc.house/oauth2/callback",
    "https://longhorn.fzymgc.house/oauth2/callback",
  ]
}

# Attach the shared groups client-scope so the session carries group membership
# for the per-route ?allowed_groups= checks (mirrors the OIDC apps).
resource "keycloak_openid_client_default_scopes" "oauth2_proxy" {
  realm_id  = keycloak_realm.fzymgc.id
  client_id = keycloak_openid_client.oauth2_proxy.id
  default_scopes = [
    "profile", "email", "roles", "web-origins", "acr", "basic",
    keycloak_openid_client_scope.groups.name, # existing shared scope (client_scopes.tf:6)
  ]
}

Verify the exact scope resource name: rg -n 'keycloak_openid_client_scope\|groups' tf/keycloak/client_scopes.tf and match the reference (keycloak_openid_client_scope.groups.name).

Step 3: Write the client secret + a generated cookie secret to Vault

resource "random_password" "oauth2_proxy_cookie" {
  length  = 32
  special = false
}

resource "vault_kv_secret_v2" "oauth2_proxy" {
  mount = "secret"
  name  = "fzymgc-house/cluster/oauth2-proxy/oidc"
  data_json = jsonencode({
    client_id     = keycloak_openid_client.oauth2_proxy.client_id
    client_secret = keycloak_openid_client.oauth2_proxy.client_secret
    cookie_secret = random_password.oauth2_proxy_cookie.result
  })
}

Step 4: Format + validate

Run: cd tf/keycloak && terraform fmt && terraform validate Expected: Success! The configuration is valid.

Step 5: Preview plan — confirm delta above the 9/9 baseline

Run: terraform plan Expected additions: 5 keycloak_group.dashboard_users, 5 keycloak_user_groups.operator_dashboards, 1 keycloak_openid_client.oauth2_proxy, 1 keycloak_openid_client_default_scopes.oauth2_proxy, 1 random_password, 1 vault_kv_secret_v2.oauth2_proxy — on top of the standing ~9-add/9-destroy provider cascade (memory hl-je6h). No unexpected destroys.

Step 6: Commit + PR + verify after merge

Commit feat(keycloak): add oauth2-proxy client + 5 dashboard groups [hl-0ahs]. After merge, confirm in Keycloak the oauth2-proxy client exists and the 5 <app>-users groups appear (with the operator as a member of each); the Vault secret is readable: vault kv get secret/fzymgc-house/cluster/oauth2-proxy/oidc.

Task 2: Deploy the shared oauth2-proxy (CA bundle + ExternalSecret + chart)

Files:

Create: argocd/app-configs/oauth2-proxy/{namespace,kustomization,secrets,ca-bundle}.yaml
Create: argocd/cluster-app/templates/oauth2-proxy.yaml
Reference: argocd/app-configs/miniflux/ (ICA1 CA-bundle + ESO pattern), argocd/cluster-app/templates/authentik.yaml (Application shape)
Step 1: Namespace + ExternalSecret

argocd/app-configs/oauth2-proxy/namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: oauth2-proxy

argocd/app-configs/oauth2-proxy/secrets.yaml:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: oauth2-proxy-secrets
  namespace: oauth2-proxy
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: oauth2-proxy-secrets
    creationPolicy: Owner
  data:
    - secretKey: client-id
      remoteRef: { key: fzymgc-house/cluster/oauth2-proxy/oidc, property: client_id }
    - secretKey: client-secret
      remoteRef: { key: fzymgc-house/cluster/oauth2-proxy/oidc, property: client_secret }
    - secretKey: cookie-secret
      remoteRef: { key: fzymgc-house/cluster/oauth2-proxy/oidc, property: cookie_secret }

Step 2: Combined ICA1 + Mozilla CA bundle (configMapGenerator — repo convention)

The repo surfaces the bundle via a configMapGenerator over a vendored .pem, NOT a static ConfigMap manifest (see argocd/app-configs/miniflux/kustomization.yaml:24-29 and argocd/app-configs/agent-memory/kustomization.yaml). Vendor it: cp argocd/app-configs/agent-memory/engram-ca-bundle.pem argocd/app-configs/oauth2-proxy/ca-bundle.pem — it already contains ICA1 fullchain + Mozilla roots (ICA1-only fails public discovery; Mozilla-only fails the split-horizon host; refresh both together from curl.se per the miniflux comment). Step 4’s kustomization.yaml surfaces it as ConfigMap oauth2-proxy-ca-bundle, key ca.crt.

Step 3: ArgoCD Application (oauth2-proxy chart, pinned latest GA)

Confirm the latest GA chart version first: helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests && helm search repo oauth2-proxy/oauth2-proxy --versions | head -3. Pin it in argocd/cluster-app/templates/oauth2-proxy.yaml targetRevision. Chart values (via helm.valuesObject):

config:
  # chart schema is config.existingSecret (NOT top-level); keys must be
  # client-id / client-secret / cookie-secret (matches the ExternalSecret above).
  existingSecret: oauth2-proxy-secrets
  configFile: |
    provider = "keycloak-oidc"
    oidc_issuer_url = "https://id.fzymgc.house/realms/fzymgc"
    email_domains = ["*"]
    cookie_domains = [".fzymgc.house"]
    whitelist_domains = [".fzymgc.house"]
    reverse_proxy = true
    set_xauthrequest = true
    provider_ca_files = ["/etc/ssl/oauth2-proxy/ca.crt"]
extraArgs:
  # authorization is per-route via ?allowed_groups=; do NOT set a global --allowed-group

Mount the oauth2-proxy-ca-bundle ConfigMap (key ca.crt) at /etc/ssl/oauth2-proxy/. Verify the provider_ca_files key name against the pinned chart’s values.yaml (helm show values oauth2-proxy/oauth2-proxy --version <v> | rg -i ca); if the chart exposes it differently, use SSL_CERT_FILE env → the combined bundle path.

Step 4: Kustomization

argocd/app-configs/oauth2-proxy/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - secrets.yaml
  - middlewares.yaml   # created in Task 3
configMapGenerator:
  - name: oauth2-proxy-ca-bundle
    files:
      - ca.crt=ca-bundle.pem
    options:
      disableNameSuffixHash: true

Step 5: Local render check

Run: kubectl --context fzymgc-house kustomize argocd/app-configs/oauth2-proxy Expected: renders Namespace + ExternalSecret + ConfigMap without error (middlewares.yaml added in Task 3 — either stub it now or add to kustomization in Task 3).

Step 6: Commit + PR + verify pod health AND a restart

Commit feat(oauth2-proxy): deploy shared Keycloak forward-auth proxy [hl-0ahs]. After merge + ArgoCD sync: kubectl -n oauth2-proxy get pods → Running; then delete the pod (kubectl -n oauth2-proxy delete pod -l app=oauth2-proxy) and confirm it comes back Ready — this catches the restart-latent CA-trust trap before any dashboard depends on it. Check logs for TLS errors against id.fzymgc.house.

Task 3: Cut dashboards over to oauth2-proxy (one at a time), then drop Authentik proxies

Files:

Create: argocd/app-configs/oauth2-proxy/middlewares.yaml
Modify (swap the middlewares: ref on the IngressRoute): argocd/app-configs/hubble-ui/ingress-route.yaml, argocd/app-configs/agentgateway/ui-ingress.yaml, argocd/app-configs/temporal-server/ingress.yaml (inline), argocd/app-configs/service-ingresses/{traefik-dashboard,longhorn-dashboard}.yaml
Delete + drop from kustomization: argocd/app-configs/hubble-ui/traefik-middleware.yaml (from hubble-ui/kustomization.yaml), argocd/app-configs/agentgateway/modern-auth.yaml (from agentgateway/kustomization.yaml)
Edit after cutover (remove proxy provider/app/bindings, KEEP the <app>_users/<app>_admin groups until T8): tf/authentik/agentgateway.tf, temporal.tf, hubble-ui.tf
Delete: argocd/app-configs/authentik/traefik-middleware.yaml
Step 1: Author the per-app + shared middlewares

argocd/app-configs/oauth2-proxy/middlewares.yaml — one auth middleware per dashboard (differing only in the ?allowed_groups= query) + a shared errors middleware:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata: { name: oauth-auth-hubble, namespace: oauth2-proxy }
spec:
  forwardAuth:
    address: "http://oauth2-proxy.oauth2-proxy.svc/oauth2/auth?allowed_groups=hubble-users"
    trustForwardHeader: true
    authResponseHeaders: [X-Auth-Request-User, X-Auth-Request-Groups, X-Auth-Request-Email]
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata: { name: oauth-errors, namespace: oauth2-proxy }
spec:
  errors:
    status: ["401-403"]
    service: { name: oauth2-proxy, port: 80 }   # adjust to the chart's Service name/port
    query: "/oauth2/sign_in?rd={url}"

Repeat the oauth-auth-<app> block for temporal (allowed_groups=temporal-users), agentgateway (agentgateway-users), traefik-dashboard (traefik-dashboard-users), longhorn (longhorn-users). Bare group names, NO leading slash — the shared groups mapper sets full_path = false (client_scopes.tf), so the groups claim carries hubble-users, not /hubble-users; with the keycloak-oidc provider a slash matches nothing and denies everyone (the exact mealie.tf/vault.tf footgun). Verify the oauth2-proxy Service name/port: kubectl -n oauth2-proxy get svc.

Step 2: Cut over dashboard #1 (hubble) — swap the middleware reference

hubble’s Middleware and IngressRoute are in separate files. In argocd/app-configs/hubble-ui/ingress-route.yaml (the IngressRoute), change the middlewares: list entry (currently - name: modern-auth) to the cross-namespace pair { name: oauth-auth-hubble, namespace: oauth2-proxy } + { name: oauth-errors, namespace: oauth2-proxy } — object form only (the @kubernetescrd string form is NOT valid in a native IngressRoute middlewares: list). Then delete the now-unused Authentik middleware file argocd/app-configs/hubble-ui/traefik-middleware.yaml and remove its line from argocd/app-configs/hubble-ui/kustomization.yaml (line 7), or the Kustomize build breaks.

Step 3: Verify hubble cutover (the “test”)

After merge + sync: browse https://hubble.fzymgc.house as a hubble-users member → Keycloak login → dashboard loads. As a non-member (or a Keycloak user not in hubble-users) → 403 from oauth2-proxy. curl -sI https://hubble.fzymgc.house unauthenticated → 302 to /oauth2/sign_in, not to auth.fzymgc.house. Expected: authenticated member OK; non-member denied; no redirect to Authentik.

Step 4: Roll the remaining 4 dashboards

Repeat Steps 2-3, minding each dashboard’s file layout: traefik-dashboard.yaml + longhorn-dashboard.yaml (both service-ingresses/, combined Middleware+IngressRoute — partial-edit in place); temporal-server/ingress.yaml (inline Middleware — partial-edit in place); agentgateway (separate files, like hubble) — swap the middlewares: ref in argocd/app-configs/agentgateway/ui-ingress.yaml, delete argocd/app-configs/agentgateway/modern-auth.yaml, and drop it from agentgateway/kustomization.yaml (line 23). One PR per dashboard is acceptable; verify each before the next.

Step 5: Remove the Authentik proxy providers (keep their groups until T8)

Edit tf/authentik/{agentgateway,temporal,hubble-ui}.tf to remove authentik_provider_proxy.* + authentik_application.* + authentik_policy_binding.*, but KEEP authentik_group.*_users/*_admin (still referenced by users.tf until T8). Delete argocd/app-configs/authentik/traefik-middleware.yaml. cd tf/authentik && terraform validate && terraform plan → expected: destroy 3 authentik_provider_proxy + 3 authentik_application + their policy bindings; zero authentik_group destroys; validate passes. No vault_kv_secret_v2 destroys.

Step 6: Commit + PR + final verify

Commit feat(oauth2-proxy): cut hubble/temporal/agentgateway/traefik/longhorn off Authentik forward-auth [hl-0ahs]. After merge, all 5 dashboards authenticate via oauth2-proxy; rg -rn 'outpost.goauthentik.io/auth/traefik' argocd/ returns nothing.

Task 4: MCP OAuth leftover — verify Keycloak issuer, delete `mcp_public_oauth.tf`

Files:

Delete: tf/authentik/mcp_public_oauth.tf
Step 1: Verify no live consumer references the Authentik MCP issuer

Run: rg -rn 'application/o/mcp' argocd/ ; rg -rn 'issuer' argocd/app-configs/agentgateway argocd/cluster-app/templates/agent-memory.yaml Expected: issuers are https://id.fzymgc.house/realms/fzymgc (Keycloak). No Authentik /application/o/mcp-public/ issuer. Cross-check via ClickStack: token-issuer telemetry shows only the Keycloak issuer over the last 24h.

Step 2: Delete the file + preview

git rm tf/authentik/mcp_public_oauth.tf; cd tf/authentik && terraform plan Expected: destroy authentik_provider_oauth2.mcp_public + authentik_application.mcp_public (+ any vault_kv_secret_v2 shown as forget, not destroy — check).

Step 3: Commit + PR + verify MCP still works

Commit chore(authentik): remove MCP OAuth leftover — consumers on Keycloak DCR [hl-0ahs]. After merge: exercise a Claude Code / agentgateway MCP connection to engram → still authenticates (via Keycloak mcp-public + DCR, ADR hl-ll5p). If it breaks, a consumer was still on Authentik — revert and re-audit.

Task 5: Tandoor OIDC — gated on `hl-nknn`

Files:

Delete: tf/authentik/tandoor.tf (only after hl-nknn closes)
Step 1: Gate check

Run: bd show hl-nknn Expected: status closed (Tandoor retired). If not closed, STOP — this task is blocked; skip to Task 6 and return here after hl-nknn. Deleting tandoor.tf while Tandoor is live destroys its login.

Step 2: Confirm Tandoor is actually gone from the cluster

Run: kubectl get ns tandoor 2>&1; rg -rn 'tandoor' argocd/cluster-app/templates/ Expected: namespace gone / Application removed. If Tandoor still runs, hl-nknn isn’t truly done — STOP.

Step 3: Delete + preview + commit

git rm tf/authentik/tandoor.tf; terraform plan → destroy authentik_provider_oauth2.tandoor + application + policy bindings; vault_kv_secret_v2.tandoor shown as forget (or destroy is acceptable here only if Tandoor’s Vault path is fully retired by hl-nknn — confirm). Commit chore(authentik): remove Tandoor OIDC provider (Tandoor retired, hl-nknn) [hl-0ahs].

Task 6: NAS LDAP removal (`hl-k8kp`)

Files:

Delete: tf/authentik/nas.tf (LDAP provider bits)
Modify: argocd/app-configs/monitoring-otel-scraper/scrape-collector-values.yaml (drop the ak-outpost-ldap scrape job, line ~182)
Step 1: Confirm the NAS no longer binds the Authentik LDAP outpost

Run: scripts/nas-playbook.sh check (per docs/operations/nas.md) that the NAS auth source is not the Authentik LDAP outpost (ak-outpost-ldap.authentik.svc:9300). Expected: NAS uses local/other auth, LDAP unused (hl-k8kp premise).

Step 2: Remove the LDAP provider + outpost + scrape job

Delete the authentik_provider_ldap (and its outpost) from tf/authentik/nas.tf; remove the ak-outpost-ldap job_name block (line ~182) from scrape-collector-values.yaml. cd tf/authentik && terraform plan → destroy authentik_provider_ldap + the LDAP outpost.

Step 3: Commit + PR + verify

Commit chore(authentik): decommission NAS LDAP integration [hl-k8kp]. After merge: kubectl -n authentik get pods shows no ak-ldap outpost pod; the Prometheus/OTel scraper no longer targets :9300.

Task 7: Drop enrollment + notification flows; file the security-alert replacement bead

Files:

Delete: tf/authentik/enrollment-flow.tf, tf/authentik/notifications.tf, tf/authentik/policies/
Step 1: File the security-alert replacement bead FIRST (blocks this task’s notification delete)

Run: bd create --type=task --title="Rebuild Authentik security-event alerting as ClickStack/HyperDX TILE alerts" --description="Replace the 5 Authentik security-alerts email notifications (login_failed, suspicious_request, password_set, config_audit, new-sign-in) with ClickStack/HyperDX TILE alerts over Keycloak otel_logs -> the security-alerts Pushover contact. This is the cluster's live alerting path (hl-qjh3/hl-toju DONE); Grafana+Prometheus are sunset (hl-nlxl). Unblocks hl-iawe SMTP retirement." --labels="authentik,keycloak,observability,hl-qjh3" --priority=2 --parent hl-0ahs Then bd dep add hl-0ahs <new-id> is not needed (parent link suffices); note the new ID here.

Step 2: Build the replacement alert rules (in the new bead’s PR, before deleting notifications.tf)

Create ClickStack/HyperDX TILE alerts on the otel_logs Keycloak stream for the 5 event types → the security-alerts Pushover contact, mirroring the existing TILE alerts from hl-qjh3.5 / hl-toju (NOT Grafana — it is sunset per hl-nlxl). Verify one fires (trigger a failed Keycloak login, confirm the Pushover alert). This is the new bead’s deliverable; Step 3 waits on it.

Step 3: Delete enrollment flow + notification rules (keep the security_alerts group until T8)

git rm tf/authentik/enrollment-flow.tf; git rm -r tf/authentik/policies. Edit tf/authentik/notifications.tf to remove the authentik_event_rule/event_transport/policy_event_matcher/policy_binding/policy_expression resources, but KEEP authentik_group.security_alerts (still referenced by users.tf until T8; its alerting function is already replaced by the Step-2 HyperDX TILE alerts). terraform validate && terraform plan → destroy the flow/stage/invitation/event resources; zero authentik_group destroys.

Step 4: Commit + PR + verify

Commit chore(authentik): drop enrollment + notification flows (alerting moved to ClickStack/HyperDX) [hl-0ahs]. After merge, confirm no Authentik email notifications remain and the HyperDX TILE alerts are the live alerting path.

Task 8: Identity teardown — delete `tf/authentik` users/groups

Files:

Delete: tf/authentik/users.tf + every remaining group-holding file — post T0/T3/T7: groups.tf, kubernetes-groups.tf, argocd.tf, and the group-only remnants of mealie.tf, uptime-kuma.tf, agentgateway.tf, temporal.tf, hubble-ui.tf, notifications.tf (rg -l 'resource "authentik_group"' tf/authentik/, minus vault.tf).
Edit (NOT delete): tf/authentik/vault.tf — remove the 2 authentik_groups, KEEP data.vault_kv_secret_v2.authentik (the provider "authentik" token source in terraform.tf; it dies with the module at T9).
Step 1: Verify all humans exist in Keycloak

In Keycloak admin: confirm sean (sub ab096608-…), dorothy, sophia exist in the fzymgc realm with expected group membership. dorothy/sophia auto-provision via the GitHub IDP (github_idp.tf, ADR hl-sqtn, sync_mode = IMPORT) — if either hasn’t logged in yet, have them log in once first. Expected: all three present.

Step 2: Delete users.tf + all group files atomically — but EDIT vault.tf (dual-role), don’t delete it

users.tf is the only thing still referencing the retained groups, so it and every group-holding file must change in the same atomic PR (delete groups before users.tf → dangling-ref error; delete users.tf first → groups unreferenced — one PR avoids both).

vault.tf carve-out: it is a MIXED file — it holds the two authentik_group resources AND data "vault_kv_secret_v2" "authentik", which terraform.tf’s provider "authentik" block reads for its API token. Deleting it whole breaks the provider (static “undeclared data source” error). So edit vault.tf to remove only authentik_group.vault_users/vault_admin, keeping the data source (it dies with the module at T9). Delete the rest:

Run: git rm tf/authentik/users.tf $(rg -l 'resource "authentik_group"' tf/authentik/ | grep -v '/vault\.tf$') Then edit tf/authentik/vault.tf down to just the data "vault_kv_secret_v2" "authentik" block.

The prevent_destroy guards on users.tf/groups.tf do NOT block this — that guard only protects a resource still present in config; removing the block entirely drops the guard, so Terraform destroys cleanly.

Step 3: Preview + confirm no Vault secret destroys

Run: cd tf/authentik && terraform plan Expected: destroy the 3 users + ~22 groups. data.vault_kv_secret_v2.authentik (the provider token source) is a data source — unaffected until Task 9. No vault_kv_secret_v2 resource destroys.

Step 4: Commit + PR + verify SSO unbroken

Commit chore(authentik): delete redundant identity (users/groups now Keycloak-owned) [hl-0ahs]. After merge, all humans still authenticate to every app via Keycloak (Authentik identities were already redundant).

Task 9: Deployment + infrastructure teardown (point of no return)

Files:

Delete: argocd/cluster-app/templates/authentik.yaml, argocd/app-configs/authentik/, argocd/app-configs/cnpg/{db-authentik,pooler-authentik}.yaml, remaining tf/authentik/* (module)
Edit (shared files / other workspaces — surgical): argocd/app-configs/cnpg/kustomization.yaml (−2 lines), argocd/app-configs/cnpg/postgres-cluster.yaml (drop authentik managed role), tf/hcp-terraform/workspaces.tf (drop main-cluster-authentik), tf/cloudflare/{variables,tunnel,r2}.tf, tf/uptime-kuma/{main,outputs}.tf, tf/vault/{policy-terraform-workspaces,jwt-hcp-terraform}.tf
Step 1: Gate — confirm Tasks 4–8 are all merged/closed

Run: ls tf/authentik/*.tf 2>&1; bd show hl-nknn; bd list --status open --parent hl-0ahs Expected: mcp_public_oauth.tf gone (T4); tandoor.tf gone + hl-nknn closed (T5); nas.tf LDAP gone (T6); the security-alert-replacement bead (filed in T7 Step 1) closed with HyperDX TILE alerts live (T7); enrollment-flow.tf gone + notifications.tf event rules removed (T7; the file + its security_alerts group go in T8); users.tf/groups.tf/kubernetes-groups.tf gone (T8). If ANY is still live/open, STOP — this phase destroys the pod they all depend on.

Step 2: Remove auth.fzymgc.house routing — Traefik IngressRoute + Cloudflare tunnel/DNS

The DNS record + tunnel ingress are generated from var.authentik_services — a plain rg 'auth\.fzymgc\.house' tf/ finds nothing (the literal is built by "${svc}.fzymgc.house" interpolation in tf/cloudflare/tunnel.tf:37). Two edits in two workspaces:

k8s (ArgoCD): the Traefik IngressRoute argocd/app-configs/authentik/ingress-route.yaml is removed by Step 3’s git rm -r argocd/app-configs/authentik.
Cloudflare (main-cluster-cloudflare workspace, separate PR): empty the auth entry from variable "authentik_services" default (tf/cloudflare/variables.tf:111-115) — or delete the variable and its two consumers in tf/cloudflare/tunnel.tf (ingress-rule for at line 37 + resource "cloudflare_dns_record" "authentik_services" at line 104). terraform plan in tf/cloudflare → destroys cloudflare_dns_record.authentik_services["auth"] + the tunnel ingress rule.
Step 2b: Tear down the Authentik R2 media integration (tf/cloudflare/r2.tf)

tf/cloudflare/r2.tf (hl-7qlb) backs Authentik media (app/source icons + admin backups) with an R2 bucket; once Authentik is gone it is orphaned. Backup decision (irreversible, like the CNPG DB): icons are disposable, but confirm no wanted admin backup remains in the bucket; if unsure, sync it out first. Then delete from tf/cloudflare/r2.tf: cloudflare_r2_bucket.authentik_media, cloudflare_r2_custom_domain.authentik_media (media.auth.fzymgc.net), cloudflare_account_token.authentik_media_r2 (revokes the RW token), vault_kv_secret_v2.authentik_media_storage (cluster/authentik/media-storage), and the two output blocks. In tf/vault/policy-terraform-workspaces.tf remove the dedicated cluster/authentik/media-storage grant (~lines 230-261). Lands with the main-cluster-cloudflare + main-cluster-vault PRs.

Step 3: Remove the ArgoCD Application + CNPG database, managed role, and shared-file refs

git rm argocd/cluster-app/templates/authentik.yaml; git rm -r argocd/app-configs/authentik; git rm argocd/app-configs/cnpg/db-authentik.yaml argocd/app-configs/cnpg/pooler-authentik.yaml. Then edit the two shared cnpg files (deleting the referenced files without these edits breaks the Kustomize build AND strands a live role):

argocd/app-configs/cnpg/kustomization.yaml: remove the - db-authentik.yaml + - pooler-authentik.yaml lines (11-12), or kubectl kustomize argocd/app-configs/cnpg fails (cluster-wide ArgoCD sync break — the app is shared by every DB tenant).
argocd/app-configs/cnpg/postgres-cluster.yaml: remove the authentik entry from spec.managed.roles (line ~64: - name: authentik + passwordSecret: main-authentik-credentials). db-authentik.yaml owned that secret; leaving the role strands a live login: true Postgres role with its password gone → unreconcilable managed-role state on the shared main cluster.

Before merge, confirm a Velero backup of the authentik namespace / CNPG DB exists (velero backup get | rg authentik). Render-check: kubectl kustomize argocd/app-configs/cnpg must still build.

Step 4: Delete the tf/authentik module + its Vault policy/role

git rm -r tf/authentik. In tf/vault (main-cluster-vault workspace) remove, by name: vault_policy.terraform_authentik_admin (policy-terraform-workspaces.tf:91-182, incl. the cluster/authentik data/metadata path blocks — the cluster/authentik/media-storage grant is a SEPARATE policy, already removed in Step 2b) and vault_jwt_auth_backend_role.tfc_authentik (jwt-hcp-terraform.tf:38-53, binds workspace:main-cluster-authentik → terraform-authentik-admin). Clean the now-dead cluster/postgres/users/main-authentik credential path if TF-managed. Confirm: rg -n 'authentik' tf/vault/ shows only comment mentions afterward.

Step 5: Remove the TFC workspace (workspaces-as-code)

In tf/hcp-terraform, delete the main-cluster-authentik entry from local.all_workspaces (workspaces.tf). Per tf/CLAUDE.md: an operator runs terraform apply in tf/hcp-terraform (Local exec, manual-apply) to destroy the workspace + its TFC_VAULT_* vars. Do NOT delete it in the HCP UI.

Step 5b: Remove the Uptime Kuma monitor of auth.fzymgc.house

tf/uptime-kuma (main-cluster-uptime-kuma workspace) has uptimekuma_monitor_http.authentik (main.tf:25, url https://auth.fzymgc.house/if/admin/) — once auth.fzymgc.house is gone it alerts forever. Remove that resource + its authentik = uptimekuma_monitor_http.authentik.id line in outputs.tf:6. Separate main-cluster-uptime-kuma PR.

Step 6: Commit + PR + verify Authentik is gone

Commit chore(authentik): tear down deployment, database, DNS, module, and TFC workspace [hl-0ahs]. After merge + apply: kubectl get ns authentik → NotFound; kubectl -n postgres get cluster → no authentik DB; curl -sI https://auth.fzymgc.house → no route; the main-cluster-authentik workspace is gone from HCP.

Task 10: Documentation / memory / CLAUDE.md sweep

Files: (~27 files; Step 3/4’s live rg -il authentik is authoritative — see spec §7)

Modify: CLAUDE.md, tf/CLAUDE.md, docs/operations/mcp-gateway-clients.md, docs/operations/{hubble-ui,temporal}.md, docs/reference/{services,secrets,network,technologies,index}.md, docs/architecture/{overview,index}.md, docs/getting-started/*, docs/index.md, README.md, docs/operations/{index,hcp-terraform,vault,clickstack,octopus,cilium-migration}.md
Delete: docs/operations/authentik.md
Create: docs/adr/hl-0ahs-decommission-authentik.md (if not already filed by capture-adrs)
Step 1: Root + tf CLAUDE.md

CLAUDE.md line 14: Infrastructure as code (Vault, Authentik) → (Vault, Keycloak). tf/CLAUDE.md: remove the main-cluster-authentik workspace-table row (line ~54) and the MCP note referencing it (line ~61).

Step 2: Rewrite the operational docs

Delete docs/operations/authentik.md (or replace with a 2-line “decommissioned 2026, see ADR hl-0ahs” stub). In docs/operations/mcp-gateway-clients.md rewrite the stale “engram uses Authentik OAuth” / “Authentik mcp-public” lines to Keycloak. In docs/operations/{hubble-ui,temporal}.md rewrite the auth section to describe the shared oauth2-proxy + per-route ?allowed_groups= gate.

Step 3: Prune reference/architecture/getting-started

rg -il authentik docs/ README.md → for each remaining file, remove Authentik rows / auth.fzymgc.house / cluster/authentik secret references. Also prune stale code comments/labels the teardown leaves behind: argocd/cluster-app/templates/hubble-ui.yaml header comment, argocd/app-configs/k8s-oidc-rbac/kustomization.yaml (app.kubernetes.io/part-of: authentik label), tf/cluster-bootstrap/README.md, and tf/keycloak/argocd.tf’s now-stale “ArgoCD keeps using Authentik until…” comment (the cutover is done — its own tf/authentik/argocd.tf says “verified green”). Keep historical docs/adr/*, docs/engineering/specs/*, docs/engineering/plans/* mentions (they are history).

Step 4: Verify the sweep is complete

Run: rg -il authentik docs/ README.md CLAUDE.md tf/CLAUDE.md tf/ argocd/ | rg -v 'engineering/(specs|plans)|/adr/' Expected: no output (every non-historical reference — including code comments/labels — gone).

Step 5: Update engram memory

Per engram:curating-memory: supersede the “Authentik decommissioning in-progress” records (they now describe a completed state) and confirm the durable pattern memory (e972552b: shared oauth2-proxy + per-route allowed_groups + CA-bundle) is accurate as-shipped. Store any as-built deltas (e.g., final chart version, exact provider_ca_files key).

Step 6: Commit + PR + close the epic

Commit docs(authentik): remove all references post-decommission [hl-0ahs]. After merge, rg -il authentik is history-only. Close hl-0ahs and its children.

Self-review coverage note

Every spec §2 area maps to a task: area 1→T0, 2→T1-T3, 3→T4, 4→T5, 5→T6, 6→T8, 7→T7, 8→T7, 9→T9; the §7 doc sweep→T10. Spec §9 decisions are all encoded (security alerts→T7 Step 1-2; traefik/longhorn→T3; auth.fzymgc.house grep→T9 Step 2). No open questions carried in.

Authentik Decommission Implementation Plan