k3s Structured Authentication cutover to Keycloak (single-issuer)

Bead: hl-5g05.16 (child of epic hl-5g05, Keycloak Phase 2) · Status: design · Date: 2026-07-01

Context

Phase 2 of the Authentik→Keycloak IdP migration finished migrating every OIDC app client (epic hl-5g05). The Keycloak side of Kubernetes OIDC — a public PKCE client kubernetes plus flat groups k8s-admins / k8s-developers / k8s-viewers — was landed in hl-5g05.12 and verified in isolation. This spec covers the remaining piece: pointing the k3s apiserver at Keycloak.

This was originally sketched (design spec §8) as a dual-issuer, v1beta1 AuthenticationConfiguration. Grounding against the live cluster changed the shape:

• The cluster is k8s v1.36.0+k3s1. Structured AuthenticationConfiguration is stable at apiVersion: apiserver.config.k8s.io/v1 (since 1.34) — the v1beta1 the older sketch assumed is outdated.
• Legacy --oidc-* apiserver flags are still supported in 1.36 (deprecated, no announced removal) — but --oidc-* and --authentication-config are mutually exclusive: if both are present the apiserver reports an error and exits. So the switch must remove all --oidc-* args atomically.
• Existing RBAC already matches the Keycloak groups (see below), so the cutover needs zero RBAC changes.
• Only the operator uses k8s OIDC (3-user trust model), so a transitional dual-issuer trust adds risk with little value.

Decision (operator): structured config, single Keycloak issuer. Modernize the mechanism (--oidc-* → --authentication-config) and cut straight over; rely on reversible config + break-glass rather than dual-issuer trust.

Current state (grounding)

Aspect	Value
Cluster	k8s v1.36.0+k3s1, HA (tp_cluster_controlplane[0] + [1:] servers, tp_cluster_workers agents)
Current OIDC	legacy --oidc-* flags → Authentik https://auth.fzymgc.house/application/o/kubernetes/
Claims today	username-claim=email prefix oidc:; groups-claim=groups prefix oidc:; required-claim aud=kubernetes
Config path	/etc/rancher/k3s/config.yaml, templated from ansible/roles/k3s-common/templates/k3s-config.yaml.j2 (server only), backed up on change (config.yaml.* rotation), notify: Restart k3s
Rollout	ansible/k3s-playbook.yml runs serial: 1 (rolling, one node at a time)
CA	no k3s_oidc_ca_file set — issuer TLS verified via system roots (both IdPs serve Let’s Encrypt certs)
Break-glass	local admin kubeconfig /etc/rancher/k3s/k3s.yaml on any server (client-cert, non-OIDC)

RBAC (must be preserved, unchanged):

ClusterRoleBinding	Role	Subject (Group)
oidc-k8s-admins	cluster-admin	oidc:k8s-admins
oidc-k8s-developers	edit	oidc:k8s-developers
oidc-k8s-viewers	view	oidc:k8s-viewers

The Keycloak kubernetes client emits groups k8s-admins/developers/viewers; with groups prefix oidc: the apiserver yields oidc:k8s-admins etc. — an exact match. No binding changes.

Design

AuthenticationConfiguration

New file /etc/rancher/k3s/authentication-config.yaml on server nodes:

apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
jwt:
  - issuer:
      url: "https://id.fzymgc.house/realms/fzymgc"
      audiences: ["kubernetes"]              # replaces required-claim aud=kubernetes
    claimMappings:
      username:
        claim: "email"
        prefix: "oidc:"                       # preserve current identity shape
      groups:
        claim: "groups"
        prefix: "oidc:"                        # preserve RBAC match (oidc:k8s-*)

Notes:

• No certificateAuthority — id.fzymgc.house (Let’s Encrypt) verifies via the node system trust store; control-plane nodes resolve it internally (split-horizon → in-cluster Traefik serving the LE cert).
• audiences: ["kubernetes"] (default audienceMatchPolicy: MatchAny) subsumes the old aud=kubernetes required-claim. No extra claimValidationRules (YAGNI).
• Single jwt authenticator (single issuer) per the operator decision.

Ansible changes (k3s-common)

1. New template templates/authentication-config.yaml.j2 rendering the file above from k3s_oidc_* vars; a template task writing it to /etc/rancher/k3s/authentication-config.yaml (server only) with notify: Restart k3s. Extend the existing “Clean up old k3s config backups” task (tasks/main.yml:75-86, currently globs only config.yaml.*) to also rotate authentication-config.yaml.* — otherwise the fix-forward workflow (below), which edits this file via backup: true, accumulates backups unbounded.
2. Modify templates/k3s-config.yaml.j2: under kube-apiserver-arg, replace the entire --oidc-* block with a single - "authentication-config=/etc/rancher/k3s/authentication-config.yaml" (gated on k3s_oidc_enabled). Atomic: one render removes all --oidc-* and adds authentication-config, satisfying the mutual-exclusivity rule.
3. Modify defaults/main.yml: repoint k3s_oidc_issuer_url to https://id.fzymgc.house/realms/fzymgc; keep k3s_oidc_client_id=kubernetes, k3s_oidc_username_claim=email, k3s_oidc_groups_claim=groups, both prefixes oidc:; introduce k3s_oidc_audiences (default ["kubernetes"]) feeding the new template. k3s_oidc_required_claims becomes unused (removed).

Rollout & restart safety

serial: 1 restarts one control-plane node at a time; remaining apiservers keep etcd quorum + serve traffic, so no single node’s config can down the whole control plane. The two failure modes need different gating, because only one of them crashes:

• R1 (malformed auth-config → apiserver fails to init) is already caught by the platform — no custom gate needed. k3s runs apiserver + etcd + controllers in one process; an apiserver init failure triggers a full-process shutdown. The generated systemd unit is Type=notify/Restart=always, so systemctl restart k3s (the Restart k3s handler) returns non-zero on a real init crash rather than hanging. Ansible’s default abort-on-play-failure (no any_errors_fatal/max_fail_percentage is set anywhere in ansible/) plus serial: 1 then halts the run after node 1. So a bad file cannot roll past the first node on its own.
• R2 (issuer unreachable / OIDC authenticator non-functional) does NOT crash — this is what the gate is for. With a valid file but a bad/unreachable issuer, the apiserver starts, systemctl restart succeeds, and generic liveness looks fine while OIDC auth is silently dead. The gate must therefore assert the specific thing that can silently break, not aggregate health:
  • Probe the issuer’s discovery endpoint from the node (the direct R2 cause): GET https://id.fzymgc.house/realms/fzymgc/.well-known/openid-configuration via ansible.builtin.uri (with retries/delay) after the restart — this confirms the apiserver’s network position can reach Keycloak, which is the precondition for authenticator init.
  • Do not rely on the aggregate /readyz boolean for auth health. If a k8s liveness probe is used at all, target /readyz?verbose and assert the JWT/authenticator-specific check line, not the rollup (JWT-authenticator wiring into the aggregate boolean is nuanced/version-dependent). The aggregate /readyz is fine only as a “process is serving” check, not an “OIDC works” check.
  • TLS: the apiserver serves a self-signed cert on https://127.0.0.1:6443, so any local /readyz probe needs validate_certs: false (or the k3s server CA at /var/lib/rancher/k3s/server/tls/server-ca.crt).
  • The definitive R2 proof is the operator’s post-rollout kubelogin test (Verification below) — the in-playbook checks are the automated gate that prevents a silent bad rollout from reaching every node.
• Config file auto-backup (config.yaml.* and authentication-config.yaml.*) enables fast revert.

Break-glass & rollback

• Break-glass (no OIDC): the local admin kubeconfig /etc/rancher/k3s/k3s.yaml on a healthy server uses client-cert auth. SSH in and use it. Note the failed node’s own k3s (apiserver and embedded etcd) is down during an R1 crash, so break-glass must be run from a different, healthy control-plane node. Verify client-cert access works on all control-plane nodes before starting the rollout.
• Rollback: revert the Ansible change (or restore the backed-up config.yaml) and re-run → restart restores the prior working config.
• Fix-forward: once the initial switch is in place, authentication-config auto-reloads (validate-before-apply; on validation failure the previous config is retained and the error logged) — subsequent edits do not require a restart.

Verification

• Pre-rollout (do before touching any node):
  • Confirm client-cert break-glass works on all control-plane nodes.
  • Confirm the Keycloak kubernetes client’s token emits email_verified: true for the operator — the authenticator (shared code path with the legacy --oidc-* flags) rejects tokens with email_verified: false when username.claim is email. Verify parity with today’s Authentik behavior via the Keycloak admin API evaluate-scopes example ID token (same recipe used for the app cutovers) before cutting over. (Keycloak includes email_verified in the default email client scope.)
• Server-side (per node, gating the rollout): issuer discovery reachable from the node + apiserver serving (/readyz) — see Rollout & restart safety.
• End-to-end (operator, definitive): kubectl oidc-login (kubelogin) against the kubernetes Keycloak client → kubectl get nodes succeeds as oidc:k8s-admins → cluster-admin; confirm the local client-cert kubeconfig still works.

Risks

• R1 — invalid auth-config file fails apiserver startup (crashing). Confirmed the apiserver does not become ready unless the manifest loads; an invalid file fails init. This is already contained by the platform (not by a custom gate): k3s single-process shutdown on init failure + systemd Type=notify/Restart=always → the Restart k3s handler returns non-zero → Ansible aborts the run + serial:1 stops after node 1; HA quorum keeps the cluster serving; break-glass + backup recover. Additional mitigation: template the file (no hand-editing).
• R2 — issuer unreachable / OIDC authenticator non-functional (non-crashing). The apiserver starts and systemctl restart succeeds even though OIDC auth is dead, so R1’s crash-based containment does not apply — this is the failure the in-playbook gate specifically targets (issuer-discovery probe from the node; see Rollout & restart safety). Whether an unreachable issuer at boot blocks readiness vs. lazy-retries is not crisply documented for 1.36; historically discovery is retried and does not block startup, which is exactly why a generic liveness gate is insufficient. id.fzymgc.house resolves internally (split-horizon → Traefik) so it is reachable whenever Keycloak is up; break-glass covers the worst case.
• R3 — username/email claim. RBAC is entirely group-based, so the username (email) claim affects only audit identity, not authorization — low impact. One runtime caveat (verified upstream): with username.claim: email, tokens with email_verified: false are rejected (parity with the current --oidc-* path). Covered by the pre-rollout email_verified check (Verification).

Out of scope (YAGNI)

• Dual-issuer (transitional Authentik trust) — single issuer only.
• CEL claimValidationRules / userValidationRules beyond audiences.
• Any RBAC / ClusterRoleBinding change.
• Removing the Authentik kubernetes provider from tf/authentik — a separate teardown follow-up once this is green (parallels the other apps’ Step 6).