Skip to content

k3s Versions and Upgrades

This page states where the k3s version of the cluster is pinned, which tool changes it, and how to upgrade k3s or add a node.

Part Location What it does
The version pin spec.version in argocd/app-configs/system-upgrade/plan-server.yaml and plan-agent.yaml The only pin for the k3s version of the cluster. The two values are always equal
system-upgrade-controller argocd/app-configs/system-upgrade/ Upgrades the nodes to spec.version: the server nodes first, one at a time, then the agent nodes. It cordons and drains each node
Renovate the k3s rule in .github/renovate.json Proposes a new spec.version for both Plans in one attended pull request, with a 7-day release-age floor
ansible/k3s-playbook.yml roles k3s-common, k3s-server, k3s-agent Installs k3s on a node that has no k3s. It never changes the version of a node that runs k3s
The API VIP 192.168.250.254 and fddb:f665:73f7:250::254, the Service kube-system/kubernetes-api in argocd/app-configs/cilium-lb/api-vip.yaml The three servers announce it over BGP through Cilium, and k8s-cluster.fzymgc.house resolves to it. The checks are in docs/operations/cilium-kpr.md

The Plans give rancher/k3s-upgrade with no tag. The controller adds the tag from spec.version, with + written as -. Do not add a tag to the image. A tagged image runs the version in its tag, and the controller then records an upgrade to spec.version that did not happen.

The controller marks each node with the label plan.upgrade.cattle.io/<plan>=<hash>. The hash covers spec.version, the service account, the annotations and the secrets of the Plan, but not the image. A node whose label equals the current hash of its Plan is done, and the controller does nothing more on it.

  1. Review the Renovate pull request update k3s. It changes spec.version in both Plans. A minor update is a Kubernetes minor version. Read the upstream release notes before you merge it.

  2. Before you merge, create a Velero backup: velero backup create pre-k3s-upgrade-$(date +%Y%m%d-%H%M%S) --wait.

  3. Merge the pull request. ArgoCD syncs the Plans, and the controller starts the upgrade.

  4. Watch the jobs and the node versions until every node shows the new version:

    Terminal window
    kubectl --context fzymgc-house -n system-upgrade get plans,jobs
    kubectl --context fzymgc-house get nodes -o wide

If a job fails, read its logs before you change the Plan again.

Run the k3s playbook for the new node only. The play installs the version that the first server runs, so the new node joins at the version of the cluster.

On a new cluster, no server runs k3s yet. The play then installs spec.version of the k3s-server Plan. To install another version, set k3s_version for that run. The variable has no effect on a node that already runs k3s.

Do not use the playbook to upgrade k3s. It does not change an installed version, and it does not drain nodes. Longhorn needs the drain to detach volumes cleanly (docs/operations/longhorn.md).

An agent joins through one address, k8s_cluster_endpoint_ip in ansible/inventory/group_vars/tp_cluster_nodes.yml, which is the API VIP 192.168.250.254. The install script writes it once as K3S_URL in /etc/systemd/system/k3s-agent.service.env. After the first join, the agent learns the addresses of the servers from the cluster and keeps them in /var/lib/rancher/k3s/agent/etc/k3s-agent-load-balancer.json, so it dials the servers directly and never the registration address again. An agent that joined before 2026-10 still carries the old address 10.255.254.6 in its unit environment, which no route reaches, and runs on the persisted server list. That file does not need a change.

If an agent must point at the current address, for example before every server address changes, run this on the agent:

Terminal window
sudo sed -i 's#^K3S_URL=.*#K3S_URL=https://192.168.250.254:6443#' /etc/systemd/system/k3s-agent.service.env
sudo systemctl restart k3s-agent