k3s Structured Authentication Keycloak cutover — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: use dev-flow:subagent-driven-development or dev-flow:executing-plans to implement task-by-task. Steps use checkbox (- [ ]) syntax.

Goal: Cut the k3s apiserver’s OIDC trust from Authentik (legacy --oidc-* flags) to a single-issuer file-based AuthenticationConfiguration trusting Keycloak, with a per-node health gate and break-glass so a bad rollout can’t lock the cluster out.

Architecture: All changes are in the ansible/roles/k3s-common role. A new authentication-config.yaml.j2 renders an apiserver.config.k8s.io/v1 config with one Keycloak JWT authenticator; k3s-config.yaml.j2 swaps the --oidc-* block for a single authentication-config= apiserver arg (atomic, since the two are mutually exclusive). Rollout rides the existing serial: 1 control-plane plays; a meta: flush_handlers + health-gate block restarts and verifies each node before the next.

Tech Stack: Ansible, k3s (k8s v1.36), Keycloak OIDC, Jinja2 templates.

Design: docs/engineering/specs/2026-07-01-k3s-structured-auth-keycloak-design.md. ADR: hl-idws (supersedes hl-9rhj). Bead: hl-5g05.16.

Model intent (for plan-to-beads): Tasks 1-4 are mechanical, self-contained Ansible/template edits → model:sonnet. Task 5 is the live-cluster rollout with real kubectl-lockout risk (break-glass, per-node gating, smoke test) → model:opus, and should be operator-driven, not autonomous.

---

File structure

File	Change	Responsibility
ansible/roles/k3s-common/templates/authentication-config.yaml.j2	Create	Render the AuthenticationConfiguration (single Keycloak JWT authenticator)
ansible/roles/k3s-common/defaults/main.yml	Modify (lines 57-65)	Repoint issuer to Keycloak, add k3s_oidc_audiences, drop k3s_oidc_required_claims
ansible/roles/k3s-common/templates/k3s-config.yaml.j2	Modify (the k3s_oidc_enabled block under kube-apiserver-arg)	Replace --oidc-* args with authentication-config= (atomic)
ansible/roles/k3s-common/tasks/main.yml	Modify (add template task; fix backup cleanup lines 75-86; append health gate)	Deploy the auth-config file; rotate its backups; per-node restart + OIDC health gate

No new handler — reuse the existing Restart k3s handler. No RBAC changes (existing oidc:k8s-* ClusterRoleBindings already match the Keycloak groups).

---

Task 1: Add the AuthenticationConfiguration template + defaults

Files:

• Create: ansible/roles/k3s-common/templates/authentication-config.yaml.j2
• Modify: ansible/roles/k3s-common/defaults/main.yml (lines 57-65)
• Step 1: Create the template

ansible/roles/k3s-common/templates/authentication-config.yaml.j2:

{# k3s apiserver structured authentication — Keycloak single-issuer OIDC (hl-5g05.16, ADR hl-idws) #}
{# Managed by Ansible. apiVersion v1 is GA since k8s 1.34; cluster is 1.36. #}
apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
jwt:
  - issuer:
      url: "{{ k3s_oidc_issuer_url }}"
      audiences:
{% for aud in k3s_oidc_audiences %}
        - "{{ aud }}"
{% endfor %}
    claimMappings:
      username:
        claim: "{{ k3s_oidc_username_claim }}"
        prefix: "{{ k3s_oidc_username_prefix }}"
      groups:
        claim: "{{ k3s_oidc_groups_claim }}"
        prefix: "{{ k3s_oidc_groups_prefix }}"

• Step 2: Edit defaults

In ansible/roles/k3s-common/defaults/main.yml, replace the current OIDC block including its comments (lines 55-65) — edit the whole range so no stale Authentik comments are left behind:

# OIDC Authentication Configuration
# Enables kubectl authentication via Authentik OIDC
k3s_oidc_enabled: true
k3s_oidc_issuer_url: "https://auth.fzymgc.house/application/o/kubernetes/"
k3s_oidc_client_id: "kubernetes"
k3s_oidc_username_claim: "email"
k3s_oidc_groups_claim: "groups"
k3s_oidc_username_prefix: "oidc:"
k3s_oidc_groups_prefix: "oidc:"
# Audience validation - prevents token reuse from other Authentik applications
k3s_oidc_required_claims: "aud=kubernetes"

with:

# OIDC Authentication Configuration (structured; kubectl auth via Keycloak).
k3s_oidc_enabled: true
# Keycloak realm issuer (structured AuthenticationConfiguration; ADR hl-idws).
k3s_oidc_issuer_url: "https://id.fzymgc.house/realms/fzymgc"
# audiences replaces the legacy required-claim aud=kubernetes; must equal the
# Keycloak client_id. Single-element list => default audienceMatchPolicy MatchAny.
k3s_oidc_audiences:
  - "kubernetes"
k3s_oidc_username_claim: "email"
k3s_oidc_groups_claim: "groups"
# Prefixes preserved so existing oidc:k8s-* ClusterRoleBindings keep matching.
k3s_oidc_username_prefix: "oidc:"
k3s_oidc_groups_prefix: "oidc:"

(k3s_oidc_client_id and k3s_oidc_required_claims are removed — the structured config uses audiences, not a client-id flag or required-claim string.)

• Step 3: Render the template locally to eyeball it

Run:

cd ansible
uvx --from ansible-core ansible -m template \
  -a "src=roles/k3s-common/templates/authentication-config.yaml.j2 dest=/tmp/authn-render.yaml" \
  -e k3s_oidc_issuer_url=https://id.fzymgc.house/realms/fzymgc \
  -e '{"k3s_oidc_audiences":["kubernetes"]}' \
  -e k3s_oidc_username_claim=email -e k3s_oidc_username_prefix=oidc: \
  -e k3s_oidc_groups_claim=groups -e k3s_oidc_groups_prefix=oidc: \
  localhost 2>/dev/null; cat /tmp/authn-render.yaml

Expected: valid YAML with apiVersion: apiserver.config.k8s.io/v1, one jwt[0].issuer.url = the Keycloak realm, audiences: ["kubernetes"], and claimMappings.username/groups with oidc: prefixes.

• Step 4: Commit

jj commit -m "feat(k3s): add Keycloak AuthenticationConfiguration template + defaults [hl-5g05.16]"

---

Task 2: Swap the apiserver arg block (atomic —oidc-* removal)

Files:

• Modify: ansible/roles/k3s-common/templates/k3s-config.yaml.j2 (the {% if k3s_oidc_enabled %} block under kube-apiserver-arg)

• Step 1: Replace the OIDC arg block

In k3s-config.yaml.j2, replace this block:

{% if k3s_oidc_enabled | default(false) %}
  # OIDC Authentication via Authentik
  - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
  - "oidc-client-id={{ k3s_oidc_client_id }}"
  - "oidc-username-claim={{ k3s_oidc_username_claim }}"
  - "oidc-groups-claim={{ k3s_oidc_groups_claim }}"
  - "oidc-username-prefix={{ k3s_oidc_username_prefix }}"
  - "oidc-groups-prefix={{ k3s_oidc_groups_prefix }}"
  - "oidc-required-claim={{ k3s_oidc_required_claims }}"
{% if k3s_oidc_ca_file is defined %}
  - "oidc-ca-file={{ k3s_oidc_ca_file }}"
{% endif %}
{% endif %}

with:

{% if k3s_oidc_enabled | default(false) %}
  # OIDC via structured AuthenticationConfiguration (Keycloak, hl-5g05.16 / ADR hl-idws).
  # Mutually exclusive with --oidc-* flags — do NOT re-add them here.
  - "authentication-config=/etc/rancher/k3s/authentication-config.yaml"
{% endif %}

• Step 2: Confirm no --oidc- args remain in the template

Run:

rg -n "oidc-issuer-url|oidc-client-id|oidc-username|oidc-groups|oidc-required|oidc-ca-file" \
  ansible/roles/k3s-common/templates/k3s-config.yaml.j2

Expected: no matches (the mutual-exclusivity rule requires zero --oidc-* args alongside authentication-config).

• Step 3: Commit

jj commit -m "feat(k3s): switch apiserver OIDC to authentication-config (atomic --oidc-* removal) [hl-5g05.16]"

---

Task 3: Deploy the auth-config file + fix backup rotation

Files:

• Modify: ansible/roles/k3s-common/tasks/main.yml (add a task after “Template k3s configuration” at line 60-73; fix “Clean up old k3s config backups” at lines 75-86)

• Step 1: Add the auth-config template task

Insert immediately after the “Template k3s configuration” task (after line 73), before “Clean up old k3s config backups”:

- name: Template k3s authentication configuration
  ansible.builtin.template:
    src: authentication-config.yaml.j2
    dest: /etc/rancher/k3s/authentication-config.yaml
    mode: "0640"
    owner: root
    group: root
    backup: true
  register: k3s_authconfig_result
  notify: Restart k3s
  when:
    - k3s_role | default('agent') == 'server'
    - k3s_oidc_enabled | default(false)
  tags:
    - k3s-config

• Step 2: Fix the backup-cleanup task (glob AND trigger)

Replace the “Clean up old k3s config backups” task (lines 75-86) with:

- name: Clean up old k3s config backups
  ansible.builtin.shell: |
    set -o pipefail
    for base in config.yaml authentication-config.yaml; do
      ls -t /etc/rancher/k3s/${base}.* 2>/dev/null \
        | tail -n +{{ k3s_config_backup_keep + 1 }} | xargs -r rm -f
    done
  args:
    executable: /bin/bash
  changed_when: false
  when:
    - k3s_role | default('agent') == 'server'
    - (k3s_config_result.changed | default(false)) or (k3s_authconfig_result.changed | default(false))
  tags:
    - k3s-config

This addresses the plan-reviewer carryover: extending the glob alone is insufficient — the when: also had to gain k3s_authconfig_result.changed, or a fix-forward edit touching only authentication-config.yaml.j2 would never trigger cleanup (k3s_config_result.changed stays false).

• Step 3: Syntax-check

cd ansible
uvx --from ansible-core ansible-playbook k3s-playbook.yml --syntax-check

Expected: playbook: k3s-playbook.yml with no errors.

• Step 4: Commit

jj commit -m "feat(k3s): deploy authentication-config file + rotate its backups [hl-5g05.16]"

---

Task 4: Add the per-node OIDC health gate (R2)

Files:

• Modify: ansible/roles/k3s-common/tasks/main.yml (append at the end of the file — currently 109 lines)

• Step 1: Append the flush-handlers + health-gate block

Add at the end of tasks/main.yml:

# --- OIDC cutover health gate (hl-5g05.16) -----------------------------------
# Force the Restart k3s handler NOW (mid-play) so the gate below runs against the
# restarted apiserver before serial:1 advances to the next control-plane node.
# R1 (malformed config) is already caught upstream: k3s single-process shutdown +
# systemd Type=notify/Restart=always makes the handler fail, and Ansible aborts.
# This gate targets R2 (issuer unreachable / authenticator non-functional), a
# NON-crashing failure where the restart succeeds but OIDC auth is silently dead.
- name: Flush handlers to apply k3s restart before the OIDC health gate
  ansible.builtin.meta: flush_handlers

- name: Wait for the local apiserver to report ready after restart
  ansible.builtin.uri:
    url: "https://127.0.0.1:6443/readyz"
    validate_certs: false # apiserver serves a self-signed cert on the loopback
    status_code: 200
  register: k3s_readyz
  until: k3s_readyz.status | default(0) == 200
  retries: 30
  delay: 5
  when:
    - k3s_role | default('agent') == 'server'
    - k3s_oidc_enabled | default(false)
  tags:
    - k3s-config
    - k3s-oidc-verify

- name: Verify the Keycloak issuer OIDC discovery is reachable from this node
  ansible.builtin.uri:
    url: "{{ k3s_oidc_issuer_url }}/.well-known/openid-configuration"
    status_code: 200
  register: k3s_oidc_discovery
  until: k3s_oidc_discovery.status | default(0) == 200
  retries: 12
  delay: 5
  when:
    - k3s_role | default('agent') == 'server'
    - k3s_oidc_enabled | default(false)
  tags:
    - k3s-config
    - k3s-oidc-verify

Rationale for the discovery probe: it directly tests R2’s precondition (the apiserver’s network position can reach Keycloak, so the JWT authenticator can complete OIDC discovery). The aggregate /readyz boolean is used only as a “process is serving” check, not an “OIDC works” check — per the design, do not rely on it for auth health.

• Step 2: Syntax-check again

cd ansible
uvx --from ansible-core ansible-playbook k3s-playbook.yml --syntax-check

Expected: no errors.

• Step 3: Lint

cd ansible
uvx ansible-lint roles/k3s-common/ 2>&1 | tail -20

Expected: no new errors introduced by the added tasks/template (pre-existing warnings unrelated to this role are acceptable; fix any that point at the new tasks).

• Step 4: Commit

jj commit -m "feat(k3s): per-node OIDC health gate (issuer discovery + readyz) [hl-5g05.16]"

---

Task 5: Pre-rollout verification, rollout, and smoke test

Files: none (operational). This task is executed against the live cluster with break-glass ready.

• Step 1: Confirm client-cert break-glass on ALL control-plane nodes

For each control-plane host, confirm the non-OIDC admin kubeconfig works (run from a healthy node; the failed node’s apiserver+etcd go down during an R1 crash):

for h in $(cd ansible && uvx --from ansible-core ansible -i inventory/hosts.yml \
  tp_cluster_controlplane --list-hosts 2>/dev/null | tail -n +2); do
  echo "== $h =="
  ssh "fzymgc@$h" 'sudo k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get --raw=/readyz'
done

Expected: each prints ok. This is the recovery path if the cutover breaks OIDC.

• Step 2: Verify Keycloak emits email_verified for the operator

The authenticator (shared code path with the legacy flags) rejects email_verified: false when username.claim: email. Confirm parity via the Keycloak admin API (same recipe used for the app cutovers):

ADMIN_PW=$(kubectl --context fzymgc-house -n keycloak get secret keycloak-bootstrap-admin -o jsonpath='{.data.password}' | base64 -d)
TOKEN=$(curl -s https://id.fzymgc.house/realms/master/protocol/openid-connect/token \
  -d client_id=admin-cli -d username=admin --data-urlencode password="$ADMIN_PW" -d grant_type=password \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["access_token"])')
UUID=$(curl -s -H "Authorization: Bearer $TOKEN" \
  "https://id.fzymgc.house/admin/realms/fzymgc/clients?clientId=kubernetes" | python3 -c 'import sys,json;print(json.load(sys.stdin)[0]["id"])')
OP=ab096608-3991-4465-9bb8-d1fe640650ac
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://id.fzymgc.house/admin/realms/fzymgc/clients/$UUID/evaluate-scopes/generate-example-id-token?userId=$OP&scope=openid" \
  | python3 -c 'import sys,json;d=json.load(sys.stdin);print("email_verified:",d.get("email_verified"));print("email:",d.get("email"));print("aud:",d.get("aud"));print("groups:",d.get("groups"))'

Expected: email_verified: True (or the field present/truthy), aud: kubernetes, groups includes k8s-admins. If email_verified is False/absent, STOP — fix the Keycloak user’s email verification before rolling out (an unverified email would be rejected by the apiserver).

• Step 3: Dry-run the config change (diff only, no restart)

cd ansible
uvx --from ansible-core ansible-playbook k3s-playbook.yml \
  --limit tp_cluster_controlplane --tags k3s-config --check --diff

Expected: the diff shows /etc/rancher/k3s/config.yaml losing the --oidc-* lines and gaining authentication-config=..., and a new /etc/rancher/k3s/authentication-config.yaml. No restart happens in --check.

• Step 4: Roll out (serial:1, per-node health-gated)

cd ansible
uvx --from ansible-core ansible-playbook k3s-playbook.yml \
  --limit tp_cluster_controlplane --tags k3s-config,k3s-oidc-verify

Expected: node-by-node — template change → restart → /readyz 200 → issuer discovery 200 → next node. If node 0 fails either the restart (R1) or the discovery probe (R2), the run aborts before touching node 1; recover via break-glass + revert (Step 6).

• Step 5: Operator end-to-end smoke test (definitive)

kubectl oidc-login get-token --oidc-issuer-url=https://id.fzymgc.house/realms/fzymgc \
  --oidc-client-id=kubernetes --oidc-extra-scope=openid,email,groups >/dev/null && \
kubectl --context fzymgc-house get nodes

Expected: browser auth via Keycloak → kubectl get nodes succeeds as oidc:k8s-admins → cluster-admin. Also confirm the client-cert kubeconfig still works.

• Step 6 (only if rollout fails): rollback

# Revert the ansible change (or restore the backup on the affected node) and re-run.
jj restore ansible/roles/k3s-common/   # if not yet merged; else revert the merge commit
# On the affected node, the previous config is at /etc/rancher/k3s/config.yaml.<ts>
# Restore it and: sudo systemctl restart k3s

• Step 7: File the Authentik kubernetes provider teardown follow-up

Once green, the Authentik-side kubernetes OAuth2 provider is dead. File a follow-up bead (Step-6 teardown, parallels the other apps) to remove it from tf/authentik — do NOT wire it here.

• Step 8: Commit any operational notes / close the bead

bd close hl-5g05.16 --reason="k3s cut to Keycloak single-issuer AuthenticationConfiguration; operator smoke test green"

---

Notes for the implementer

• Order is atomic at apply time, not commit time. Tasks 1-4 are separate commits, but nothing is live until the playbook runs (Task 5). The rendered config.yaml never contains both --oidc-* and authentication-config because Task 2 removes the former in the same template.
• Lockout safety: the whole point of the health gate + serial: 1 + break-glass is that a bad rollout stops after node 0 with the rest of the HA control plane intact. Never run this without confirming break-glass (Step 1) first.
• No RBAC changes: the existing oidc:k8s-{admins,developers,viewers} ClusterRoleBindings already match the Keycloak groups via the retained oidc: prefix.
• Out of scope: the Authentik kubernetes provider teardown (Task 5 Step 7 files it); any dual-issuer trust (superseded by ADR hl-idws).