Authentik upgrade 2026.2.2 → 2026.5.3

Design bead: hl-54n2 Status: design Author: Sean Brandt (with Claude) Date: 2026-06-14

Context

The cluster’s Authentik SSO is pinned at 2026.2.2; the current GA is 2026.5.3. Authentik gates every OIDC application in the cluster, so it is the single most blast-radius-sensitive component to upgrade. The goal is simply to get current, done ASAP with a brief, accepted login interruption — gated on a verified database backup because Authentik schema migrations are irreversible.

Authentik is deployed as a multi-source Argo CD Application (argocd/cluster-app/templates/authentik.yaml): the upstream authentik/authentik Helm chart pinned to the release version, plus the git source argocd/app-configs/authentik (HEAD). The server/worker image tag is pinned separately via global.image.tag. Postgres is the shared CloudNativePG main cluster (main-rw.postgres.svc.cluster.local); the chart’s bundled Postgres and Redis are both disabled (postgres.enabled: false, redis.enabled: false).

Grounding (Rule 7)

• deepwiki goauthentik/authentik — upgrades must be stepwise by version family; you cannot skip families. However no 2026.3 or 2026.4 family was released, so 2026.2 → 2026.5 is a single adjacent-family hop, which is supported (Authentik supports the two most-recent families). The cluster is already past the disruptive 2025-series breaks (RBAC overhaul 2025.12, Redis removal 2025.10, Postgres 15→17 in 2025.6, sessions-to-DB 2025.4) because it is on 2026.2. The one new 2026.5 change that touches this cluster is the User.ak_groups deprecation: it still functions but emits a configuration warning on each evaluation. We have a live mapping at tf/authentik/kubernetes-oidc.tf:12 (request.user.ak_groups.values_list('name', flat=True)); the replacement is request.user.groups. The warning is cosmetic and not a regression; the modernization is tracked separately in hl-lcy1 (see Risks / Out of scope). Database migrations are not reversible; a pg_dump backup is the recommended rollback path.
• helm search repo authentik/authentik — current 2026.2.2; latest GA 2026.5.3 (also a 2026.2.3 patch on the current family). No 2026.3.x / 2026.4.x exist.
• Render-diff (helm template our exact valuesObject, 2026.2.2 vs 2026.5.3) — the only structural delta is the image tag plus the removal of the explicit AUTHENTIK_LISTEN__HTTP/HTTPS/METRICS=0.0.0.0:… env vars from the rendered Deployments. This is the documented “default listen IP 0.0.0.0 → [::]” breaking change in concrete form. No new required values; same eight rendered resource kinds; redis.enabled: false already set.
• k8s inventory — authentik-server + authentik-worker (Helm, 2026.2.2) and ak-outpost-ldap (Authentik-managed, 2026.2.2). The LDAP outpost serves the NAS LDAP provider — which is unused: TrueNAS (nas.fzymgc.house) Directory Services is disabled (operator-confirmed via screenshot). The outpost-parity requirement is therefore moot for this upgrade.
• CNPG backup posture (main cluster) — continuous WAL archiving is live and healthy (barman-cloud plugin, archive_mode=on, ContinuousArchiving: Success), with 4-hourly Longhorn volumeSnapshot scheduled backups (main-backup-4h) and a PITR window back to 2025-06-10. Two caveats shape the rollback design: (a) the base backups are Longhorn-local volume snapshots, good for an intact-volume rollback but not off-site DR — so the off-pod pg_dump is the artifact that survives volume loss; (b) main is shared by 7 app DBs, so CNPG-native restore (PITR / snapshot recovery) is whole-instance and cannot surgically restore a single database.

Scope

One Argo CD PR editing argocd/cluster-app/templates/authentik.yaml:

1. Chart targetRevision: "2026.2.2" → "2026.5.3".

2. global.image.tag: "2026.2.2" → "2026.5.3".

3. Add to server.env and worker.env to preserve the IPv4 bind and neutralize the only breaking change:

   • AUTHENTIK_LISTEN__HTTP = 0.0.0.0:9000
   • AUTHENTIK_LISTEN__METRICS = 0.0.0.0:9300
   • AUTHENTIK_LISTEN__HTTPS = 0.0.0.0:9443 (server only)

   The worker’s __HTTP listener is its internal healthcheck endpoint on 9000, not a user-facing web server — which is why __HTTPS is server-only.

Validation: re-render the edited valuesObject against chart 2026.5.3 and confirm the diff vs the live 2026.2.2 render is exactly the image tag + the re-added listen env vars (no other drift).

Pre-flight gate (mandatory)

Two backups, taken/confirmed immediately before merge:

1. Native CNPG backup (primary safety net). Trigger an on-demand backup of the main cluster and confirm it completes — this rides the existing, proven infra (Longhorn volume snapshot + continuous WAL → PITR):

kubectl cnpg backup main -n postgres          # or apply a Backup CR (method: volumeSnapshot)
kubectl get backup -n postgres --sort-by=.metadata.creationTimestamp | tail -1   # phase: completed

2. Logical dump of ONLY authentik (surgical-rollback artifact). Because main is shared by 7 app DBs, this is the only artifact that can restore authentik alone, in place, without rewinding co-tenant DBs:

# resolve the primary: kubectl get pod -n postgres -l cnpg.io/cluster=main,role=primary -o name
kubectl exec -n postgres <main-primary-pod> -c postgres -- \
  pg_dump -Fc -d authentik > authentik-pre-2026.5.3.dump

Migrations are irreversible. Do NOT merge until both are confirmed: the CNPG backup reports completed, and the dump is restorable-looking (non-empty; schema and key tables present — pg_restore -l authentik-pre-2026.5.3.dump).

Rollout & blast radius

On merge, Argo CD syncs cluster-app → re-renders the authentik Application → the worker applies DB migrations and the server rolls (chart default strategy). Sessions are DB-backed (since 2025.4), so existing sessions survive; new logins blip for a few minutes during migration + roll. Everything behind SSO depends on this, which is why the backup gate is mandatory and the timing should be a low-traffic moment.

Rollback

1. Revert the PR (chart targetRevision + global.image.tag back to 2026.2.2, remove the added listen env vars). Removing the env vars is safe: the 2026.2.2 chart’s own default is 0.0.0.0, so the rolled-back state matches the original render.

2. Restore only the authentik database into the shared CNPG main cluster. Scale authentik-server + authentik-worker to zero first to drop live connections, then drop/recreate just that one database and restore the dump — this touches no other database on the instance:

   # terminate any stray connections, then drop/recreate ONLY the authentik DB
   kubectl exec -n postgres <main-primary-pod> -c postgres -- psql -d postgres -c \
     "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='authentik';"
   kubectl exec -n postgres <main-primary-pod> -c postgres -- \
     psql -c 'DROP DATABASE authentik;' \
          -c 'CREATE DATABASE authentik OWNER authentik;'
   kubectl exec -i -n postgres <main-primary-pod> -c postgres -- \
     pg_restore -d authentik < authentik-pre-2026.5.3.dump

   The in-place pg_dump restore is preferred precisely because it is surgical — it touches only authentik. The CNPG-native fallback (restore the pre-flight backup or PITR to just-before-upgrade via the Barman WAL archive) is whole-instance: it rewinds all 7 co-tenant DBs to that point, losing any interim writes. Use it only if the in-place logical restore is insufficient (e.g. instance-level corruption).

Reverting code alone is insufficient — the schema has migrated forward and 2026.5.3 migrations cannot be un-applied against a 2026.2.2 binary.

Verification

• authentik-server and authentik-worker pods report app.kubernetes.io/version=2026.5.3; both Running, no CrashLoop (confirms the listen-bind mitigation held).
• The Argo CD authentik Application returns to Synced/Healthy.
• Web login at auth.fzymgc.house works.
• At least one downstream OIDC app login works end-to-end (e.g. Grafana or ArgoCD).

Out of scope

• NAS LDAP decommission — the unused nas LDAP provider/app (tf/authentik/nas.tf) and the Authentik-managed ak-outpost-ldap are tracked separately in hl-k8kp. This upgrade does not touch LDAP; the outpost will auto-track the server version (or remain stale-but-harmless) until hl-k8kp removes it.
• ak_groups → groups property-mapping modernization — the 2026.5 deprecation warning is cosmetic and the mapping still functions; the fix is a tf/authentik (main-cluster-authentik workspace) change tracked in hl-lcy1.
• tf/authentik OIDC client definitions — version-independent, unchanged.
• The whoami test pod in the authentik namespace.

Risks

Risk	Likelihood	Mitigation
[::] listen default breaks probes/Service in an IPv4 path	Low	Re-add explicit 0.0.0.0 listen env vars (step 3); validated via render-diff
Irreversible migration + a bad release	Low	Mandatory verified pg_dump pre-flight; documented DB-restore rollback
Login interruption longer than expected	Low	DB-backed sessions persist; schedule low-traffic; rollback ready
User.ak_groups config warning after upgrade (live mapping in tf/authentik)	Certain (cosmetic)	Mapping still functions; warning is expected, not a regression. Modernize to request.user.groups in hl-lcy1
Hidden schema/value change beyond the render-diff	Very low	Render-diff showed only tag + listen env delta; re-validate after the edit