Skip to content

Cluster Access

How to reach the Kubernetes API with kubectl. People log in through Keycloak with OIDC. The admin kubeconfig is the break-glass path.

Item Value Where it is set
API server https://k8s-cluster.fzymgc.house:6443, the routed VIP 192.168.250.254 and fddb:f665:73f7:250::254 that the three servers announce over BGP the Service kube-system/kubernetes-api in argocd/app-configs/cilium-lb/api-vip.yaml, the name in dns-control
OIDC issuer https://id.fzymgc.house/realms/fzymgc k3s_oidc_issuer_url in ansible/roles/k3s-common/defaults/main.yml
OIDC client kubernetes, public, PKCE, redirect to the kubelogin loopback ports 8000 and 18000 tf/keycloak/kubernetes.tf
Claims username from email, groups from groups, both prefixed oidc: the k3s_oidc_* values in the same defaults file
Access levels Keycloak group to ClusterRole bindings argocd/app-configs/k8s-oidc-rbac/
Token lifetimes Access token 1 h, SSO session idle 18 h, SSO session maximum 24 h tf/keycloak/main.tf
Context name fzymgc-house in every kubeconfig the documented commands use --context fzymgc-house
Keycloak group ClusterRole Permissions
k8s-admins cluster-admin Everything
k8s-developers edit Create and change workloads, no RBAC
k8s-viewers view Read only

Install kubectl, kubelogin and yq (brew install kubectl kubelogin yq).

  1. Read the cluster CA from a control plane node:

    Terminal window
    ssh fzymgc@tpi-alpha-1 'sudo cat /etc/rancher/k3s/k3s.yaml' | yq '.clusters[0].cluster.certificate-authority-data'
  2. Write ~/.kube/configs/fzymgc-house-oidc.yml:

    apiVersion: v1
    kind: Config
    clusters:
    - name: fzymgc-house
    cluster:
    server: https://k8s-cluster.fzymgc.house:6443
    certificate-authority-data: <the value from step 1>
    contexts:
    - name: fzymgc-house
    context:
    cluster: fzymgc-house
    user: oidc
    users:
    - name: oidc
    user:
    exec:
    apiVersion: client.authentication.k8s.io/v1beta1
    command: kubectl
    args:
    - oidc-login
    - get-token
    - --oidc-issuer-url=https://id.fzymgc.house/realms/fzymgc
    - --oidc-client-id=kubernetes
    - --oidc-extra-scope=profile
    - --oidc-extra-scope=email
    interactiveMode: IfAvailable
    current-context: fzymgc-house
  3. Point KUBECONFIG at it and run any command. The first command opens a browser for Keycloak:

    Terminal window
    export KUBECONFIG=~/.kube/configs/fzymgc-house-oidc.yml
    kubectl --context fzymgc-house get nodes # 8 nodes

kubelogin caches the token under ~/.kube/cache/oidc-login and refreshes it while the Keycloak session lives.

The admin kubeconfig carries the k3s client certificate and does not depend on Keycloak. Fetch it from the control plane with the playbook tag, which writes ~/.kube/configs/fzymgc-house-admin.yml with the same context name:

Terminal window
cd ansible
uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags k3s-kubeconfig
export KUBECONFIG=~/.kube/configs/fzymgc-house-admin.yml
kubectl --context fzymgc-house get nodes

The file grants cluster-admin. Use it when Keycloak is down or when a stream must outlive the SSO session. Re-fetch it when kubectl returns Unauthorized, which means that the client certificate expired.

The crew-homelab-ro ServiceAccount in namespace sysadm reads everything except Secrets. ansible/files/crew-homelab-ro-rbac.yaml declares it, and the k3s-readonly-access tag applies it:

Terminal window
cd ansible
uv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags k3s-readonly-access
kubectl --context fzymgc-house auth can-i --as=system:serviceaccount:sysadm:crew-homelab-ro list nodes # yes
kubectl --context fzymgc-house auth can-i --as=system:serviceaccount:sysadm:crew-homelab-ro list secrets -A # no
Symptom Cause Fix
The browser does not open kubelogin cannot reach a browser Run kubectl oidc-login get-token --oidc-issuer-url=https://id.fzymgc.house/realms/fzymgc --oidc-client-id=kubernetes and follow the printed URL
Unauthorized after a group change kubelogin holds a token issued before the change rm -rf ~/.kube/cache/oidc-login, then run any command to log in again
Unauthorized with the admin kubeconfig The client certificate expired Re-fetch it with the k3s-kubeconfig tag
A certificate error The CA in the kubeconfig does not match the cluster Read the CA again from a control plane node. Do not skip TLS verification
The login page does not load Keycloak is down Use the admin kubeconfig, then check the keycloak application in ArgoCD