Skip to content
fzymgc-house Cluster Docs

Authentik Upgrade 2026.2.2 → 2026.5.3 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use dev-flow:subagent-driven-development (recommended) or dev-flow:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Bring the in-cluster Authentik SSO from 2026.2.2 to the current GA 2026.5.3 via a single Argo CD Helm change, gated on a verified Postgres backup.

Architecture: Authentik is a multi-source Argo CD Application (argocd/cluster-app/templates/authentik.yaml): upstream authentik/authentik chart + git source. The upgrade bumps the chart targetRevision and global.image.tag, and re-adds explicit AUTHENTIK_LISTEN__*=0.0.0.0:* env vars to neutralize the 2026.5 0.0.0.0 → [::] default-bind breaking change (validated by render-diff). Postgres is the shared CloudNativePG main cluster; migrations are irreversible, so verified backups (a native CNPG backup + an authentik-only pg_dump) precede the merge.

Tech Stack: Argo CD (GitOps, ServerSideApply), Helm (authentik chart), CloudNativePG, jj VCS, kubectl.

Spec: docs/engineering/specs/2026-06-14-authentik-upgrade-design.md Design bead: hl-54n2

Files touched

Section titled “Files touched”
  • Modify: argocd/cluster-app/templates/authentik.yaml — chart targetRevision (line ~13), global.image.tag (line ~19), add listen env vars under server.env and worker.env.

No new files. No Terraform changes (LDAP decommission = hl-k8kp; ak_groups modernization = hl-lcy1 — both out of scope).

Task 1: Pre-flight — capture & verify the Authentik DB backup (mandatory gate)

Section titled “Task 1: Pre-flight — capture & verify the Authentik DB backup (mandatory gate)”

Model: haiku (mechanical operational step)

This is a manual operational gate. Do NOT proceed to the merge (Task 3) until both backups are captured and verified. Migrations are irreversible; these are the rollback artifacts.

Files: none (operational step; the dump is stored off-cluster).

  • Step 1: Resolve the CNPG main primary pod

Run:

Terminal window
kubectl get pod -n postgres -l cnpg.io/cluster=main,role=primary -o name

Expected: one pod, e.g. pod/main-1. Note the name as <primary> below.

  • Step 2: Trigger and verify a native CNPG backup (primary safety net)

Confirm continuous archiving is healthy, then take an on-demand backup. This rides the existing proven infra (4-hourly Longhorn volumeSnapshot + continuous WAL → whole-instance PITR target).

Run:

Terminal window
kubectl get cluster -n postgres main \
  -o jsonpath='{range .status.conditions[?(@.type=="ContinuousArchiving")]}{.status}{"\n"}{end}'
kubectl cnpg backup main -n postgres   # needs the cnpg kubectl plugin; else apply a Backup CR (method: volumeSnapshot)
kubectl get backup -n postgres --sort-by=.metadata.creationTimestamp | tail -1

Expected: ContinuousArchiving True; the newly-created Backup reaches phase: completed within a few minutes.

  • Step 3: Capture a custom-format dump of ONLY the authentik database (surgical-rollback artifact)

Because main is shared by 7 app DBs, this logical dump is the only artifact that can restore authentik alone, in place, without rewinding co-tenant DBs. Run:

Terminal window
kubectl exec -n postgres <primary> -c postgres -- \
  pg_dump -Fc -d authentik > authentik-pre-2026.5.3.dump

Expected: command exits 0; authentik-pre-2026.5.3.dump is a non-trivial size (authentik DBs are typically multiple MB).

  • Step 4: Verify the dump is restorable-looking

Run:

Terminal window
pg_restore -l authentik-pre-2026.5.3.dump | head -20
ls -lh authentik-pre-2026.5.3.dump

Expected: pg_restore -l lists a table of contents including schema + tables (e.g. authentik_core_user); file size is non-zero. If the TOC is empty or the file is ~0 bytes, STOP — do not upgrade.

  • Step 5: Record both backup artifacts

In the bead/PR, note (a) the CNPG Backup name + its completed timestamp and (b) where authentik-pre-2026.5.3.dump is stored (off-pod), so the rollback step can find both. No commit.

Task 2: Apply and render-validate the manifest change

Section titled “Task 2: Apply and render-validate the manifest change”

Model: sonnet (manifest edit + render-diff judgment)

Files:

  • Modify: argocd/cluster-app/templates/authentik.yaml

  • Step 1: Capture the baseline render (current 2026.2.2) — the “before” for the diff

Run (extracts the live valuesObject, renders the current chart):

Terminal window
cd <repo-root>
awk '/^        valuesObject:/{f=1;next} /^    - repoURL:/{f=0} f' \
  argocd/cluster-app/templates/authentik.yaml | sed 's/^          //' > /tmp/ak-values-cur.yaml
helm repo add authentik https://charts.goauthentik.io >/dev/null 2>&1
helm repo update authentik >/dev/null
helm template authentik authentik/authentik --version 2026.2.2 -f /tmp/ak-values-cur.yaml \
  --set authentik.secret_key=dummy --set authentik.postgresql.password=dummy > /tmp/ak-before.yaml

Expected: /tmp/ak-before.yaml written, no error.

  • Step 2: Edit argocd/cluster-app/templates/authentik.yaml — bump the chart version

Change the chart source targetRevision:

    - chart: authentik
      repoURL: https://charts.goauthentik.io
      targetRevision: "2026.5.3"   # was "2026.2.2"
  • Step 3: Edit the same file — bump the image tag

Under valuesObject.global.image:

          global:
            image:
              tag: "2026.5.3"   # was "2026.2.2"
  • Step 4: Edit the same file — add listen env vars to server.env

The server.env list currently holds only AUTHENTIK_POSTGRESQL__SSLROOTCERT. Add the three listen vars so the block reads:

          server:
            # ...podAnnotations / envFrom unchanged...
            env:
              - name: AUTHENTIK_POSTGRESQL__SSLROOTCERT
                value: /certs/ca.crt
              - name: AUTHENTIK_LISTEN__HTTP
                value: "0.0.0.0:9000"
              - name: AUTHENTIK_LISTEN__HTTPS
                value: "0.0.0.0:9443"
              - name: AUTHENTIK_LISTEN__METRICS
                value: "0.0.0.0:9300"
  • Step 5: Edit the same file — add listen env vars to worker.env

The worker exposes only an HTTP healthcheck listener (no HTTPS). Add HTTP + METRICS so the block reads:

          worker:
            # ...podAnnotations / envFrom unchanged...
            env:
              - name: AUTHENTIK_POSTGRESQL__SSLROOTCERT
                value: /certs/ca.crt
              - name: AUTHENTIK_LISTEN__HTTP
                value: "0.0.0.0:9000"
              - name: AUTHENTIK_LISTEN__METRICS
                value: "0.0.0.0:9300"
  • Step 6: Render the target (2026.5.3) and diff against baseline — the validation gate

Run:

Terminal window
awk '/^        valuesObject:/{f=1;next} /^    - repoURL:/{f=0} f' \
  argocd/cluster-app/templates/authentik.yaml | sed 's/^          //' > /tmp/ak-values-new.yaml
helm template authentik authentik/authentik --version 2026.5.3 -f /tmp/ak-values-new.yaml \
  --set authentik.secret_key=dummy --set authentik.postgresql.password=dummy > /tmp/ak-after.yaml
diff /tmp/ak-before.yaml /tmp/ak-after.yaml | rg '^[<>]' | \
  rg -iv 'helm.sh/chart|app.kubernetes.io/version|image:.*authentik:|tag:|secret_key|password'

Expected: the ONLY remaining diff lines are the three AUTHENTIK_LISTEN__* env vars now present on the > (target) side that the 2026.2.2 chart used to emit itself — i.e. the listen env is restored, nothing else drifts. If any other structural line appears, STOP and investigate before committing.

  • Step 7: Commit

Create a feature branch and commit (jj):

Terminal window
jj new main@origin
# (edits already in the working copy)
jj describe -m "feat(authentik): upgrade 2026.2.2 -> 2026.5.3 (chart + image + IPv4 listen env) [hl-54n2]"
jj bookmark create feat/authentik-2026.5.3 -r @
jj git push -b feat/authentik-2026.5.3

Task 3: Open the PR, confirm backup gate, merge

Section titled “Task 3: Open the PR, confirm backup gate, merge”

Model: haiku (mechanical PR/merge step)

Files: none (review/merge step).

  • Step 1: Open the PR

Run:

Terminal window
gh pr create --head feat/authentik-2026.5.3 --base main \
  --title "feat(authentik): upgrade 2026.2.2 -> 2026.5.3" \
  --body "Chart + image bump to current GA; re-adds explicit AUTHENTIK_LISTEN__*=0.0.0.0 env to neutralize the 2026.5 [::] default-bind change (render-diff validated). Gated on a verified pg_dump (Task 1). LDAP decommission hl-k8kp + ak_groups modernization hl-lcy1 are out of scope. Spec: docs/engineering/specs/2026-06-14-authentik-upgrade-design.md"

Expected: PR URL printed.

  • Step 2: Confirm the Task 1 backup is captured and verified

Do NOT merge unless authentik-pre-2026.5.3.dump exists and Step 3 of Task 1 passed. This is the irreversible-migration gate.

  • Step 3: Merge the PR

Merge via GitHub once checks pass. Argo CD (cluster-app, ServerSideApply + selfHeal) will sync the new authentik Application revision automatically.

Task 4: Post-upgrade verification

Section titled “Task 4: Post-upgrade verification”

Model: haiku (mechanical verification step)

Files: none (verification step).

  • Step 1: Wait for the rollout and confirm pod versions

Run:

Terminal window
kubectl get pods -n authentik -l app.kubernetes.io/instance=authentik \
  -o custom-columns=NAME:.metadata.name,VER:.metadata.labels.'app\.kubernetes\.io/version',READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount

Expected: authentik-server and authentik-worker both show VER 2026.5.3, READY true, and RESTARTS not climbing (no CrashLoop — confirms the listen-bind mitigation held).

  • Step 2: Confirm the Argo CD Application is healthy

Run:

Terminal window
kubectl get application -n argocd authentik \
  -o custom-columns=SYNC:.status.sync.status,HEALTH:.status.health.status

Expected: Synced / Healthy.

  • Step 3: Verify the web login flow

Open https://auth.fzymgc.house in a browser and log in. Expected: login succeeds; admin UI loads on 2026.5.3 (version shown in the UI footer/admin overview).

  • Step 4: Verify a downstream OIDC app

Log in to one OIDC consumer (e.g. Grafana at grafana.fzymgc.house or Argo CD) via Authentik SSO. Expected: SSO redirect completes and the app authenticates. (A User.ak_groups config warning may appear in the Authentik admin event log — this is the expected, cosmetic 2026.5 deprecation tracked in hl-lcy1, NOT a regression.)

  • Step 5: Close out

Note the verified version + any observations on the PR / bead. The pre-upgrade dump can be retained until the upgrade is confirmed stable for a day, then discarded.

Rollback (if Task 4 fails)

Section titled “Rollback (if Task 4 fails)”
  1. Revert the PR (chart targetRevision + global.image.tag → 2026.2.2, remove the added listen env vars — safe because 2026.2.2 also defaults to 0.0.0.0). Let Argo CD sync.
  2. Restore ONLY the authentik DB into the shared main cluster:
Terminal window
kubectl scale -n authentik deploy/authentik-server deploy/authentik-worker --replicas=0
kubectl exec -n postgres <primary> -c postgres -- psql -d postgres -c \
  "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='authentik';"
kubectl exec -n postgres <primary> -c postgres -- \
  psql -c 'DROP DATABASE authentik;' -c 'CREATE DATABASE authentik OWNER authentik;'
kubectl exec -i -n postgres <primary> -c postgres -- \
  pg_restore -d authentik < authentik-pre-2026.5.3.dump
  1. Argo CD resyncs the reverted (2026.2.2) Application against the restored DB. Reverting code alone is insufficient — the schema migrated forward.