Cluster Access
How to reach the Kubernetes API with kubectl. People log in through Keycloak with OIDC. The
admin kubeconfig is the break-glass path.
What exists
Section titled “What exists”| Item | Value | Where it is set |
|---|---|---|
| API server | https://k8s-cluster.fzymgc.house:6443, the routed VIP 192.168.250.254 and fddb:f665:73f7:250::254 that the three servers announce over BGP |
the Service kube-system/kubernetes-api in argocd/app-configs/cilium-lb/api-vip.yaml, the name in dns-control |
| OIDC issuer | https://id.fzymgc.house/realms/fzymgc |
k3s_oidc_issuer_url in ansible/roles/k3s-common/defaults/main.yml |
| OIDC client | kubernetes, public, PKCE, redirect to the kubelogin loopback ports 8000 and 18000 |
tf/keycloak/kubernetes.tf |
| Claims | username from email, groups from groups, both prefixed oidc: |
the k3s_oidc_* values in the same defaults file |
| Access levels | Keycloak group to ClusterRole bindings | argocd/app-configs/k8s-oidc-rbac/ |
| Token lifetimes | Access token 1 h, SSO session idle 18 h, SSO session maximum 24 h | tf/keycloak/main.tf |
| Context name | fzymgc-house in every kubeconfig |
the documented commands use --context fzymgc-house |
| Keycloak group | ClusterRole | Permissions |
|---|---|---|
k8s-admins |
cluster-admin |
Everything |
k8s-developers |
edit |
Create and change workloads, no RBAC |
k8s-viewers |
view |
Read only |
Create the OIDC kubeconfig
Section titled “Create the OIDC kubeconfig”Install kubectl, kubelogin and yq (brew install kubectl kubelogin yq).
-
Read the cluster CA from a control plane node:
Terminal window ssh fzymgc@tpi-alpha-1 'sudo cat /etc/rancher/k3s/k3s.yaml' | yq '.clusters[0].cluster.certificate-authority-data' -
Write
~/.kube/configs/fzymgc-house-oidc.yml:apiVersion: v1kind: Configclusters:- name: fzymgc-housecluster:server: https://k8s-cluster.fzymgc.house:6443certificate-authority-data: <the value from step 1>contexts:- name: fzymgc-housecontext:cluster: fzymgc-houseuser: oidcusers:- name: oidcuser:exec:apiVersion: client.authentication.k8s.io/v1beta1command: kubectlargs:- oidc-login- get-token- --oidc-issuer-url=https://id.fzymgc.house/realms/fzymgc- --oidc-client-id=kubernetes- --oidc-extra-scope=profile- --oidc-extra-scope=emailinteractiveMode: IfAvailablecurrent-context: fzymgc-house -
Point
KUBECONFIGat it and run any command. The first command opens a browser for Keycloak:Terminal window export KUBECONFIG=~/.kube/configs/fzymgc-house-oidc.ymlkubectl --context fzymgc-house get nodes # 8 nodes
kubelogin caches the token under ~/.kube/cache/oidc-login and refreshes it while the Keycloak
session lives.
Break-glass access
Section titled “Break-glass access”The admin kubeconfig carries the k3s client certificate and does not depend on Keycloak. Fetch
it from the control plane with the playbook tag, which writes
~/.kube/configs/fzymgc-house-admin.yml with the same context name:
cd ansibleuv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags k3s-kubeconfigexport KUBECONFIG=~/.kube/configs/fzymgc-house-admin.ymlkubectl --context fzymgc-house get nodesThe file grants cluster-admin. Use it when Keycloak is down or when a stream must outlive the
SSO session. Re-fetch it when kubectl returns Unauthorized, which means that the client
certificate expired.
Read-only service account for automation
Section titled “Read-only service account for automation”The crew-homelab-ro ServiceAccount in namespace sysadm reads everything except Secrets.
ansible/files/crew-homelab-ro-rbac.yaml declares it, and the k3s-readonly-access tag applies
it:
cd ansibleuv run ansible-playbook -i inventory/hosts.yml k3s-playbook.yml --tags k3s-readonly-accesskubectl --context fzymgc-house auth can-i --as=system:serviceaccount:sysadm:crew-homelab-ro list nodes # yeskubectl --context fzymgc-house auth can-i --as=system:serviceaccount:sysadm:crew-homelab-ro list secrets -A # noTroubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Fix |
|---|---|---|
| The browser does not open | kubelogin cannot reach a browser | Run kubectl oidc-login get-token --oidc-issuer-url=https://id.fzymgc.house/realms/fzymgc --oidc-client-id=kubernetes and follow the printed URL |
Unauthorized after a group change |
kubelogin holds a token issued before the change | rm -rf ~/.kube/cache/oidc-login, then run any command to log in again |
Unauthorized with the admin kubeconfig |
The client certificate expired | Re-fetch it with the k3s-kubeconfig tag |
| A certificate error | The CA in the kubeconfig does not match the cluster | Read the CA again from a control plane node. Do not skip TLS verification |
| The login page does not load | Keycloak is down | Use the admin kubeconfig, then check the keycloak application in ArgoCD |