k3s Versions and Upgrades
This page states where the k3s version of the cluster is pinned, which tool changes it, and how to upgrade k3s or add a node.
What exists
Section titled “What exists”| Part | Location | What it does |
|---|---|---|
| The version pin | spec.version in argocd/app-configs/system-upgrade/plan-server.yaml and plan-agent.yaml |
The only pin for the k3s version of the cluster. The two values are always equal |
| system-upgrade-controller | argocd/app-configs/system-upgrade/ |
Upgrades the nodes to spec.version: the server nodes first, one at a time, then the agent nodes. It cordons and drains each node |
| Renovate | the k3s rule in .github/renovate.json |
Proposes a new spec.version for both Plans in one attended pull request, with a 7-day release-age floor |
ansible/k3s-playbook.yml |
roles k3s-common, k3s-server, k3s-agent |
Installs k3s on a node that has no k3s. It never changes the version of a node that runs k3s |
| The API VIP | 192.168.250.254 and fddb:f665:73f7:250::254, the Service kube-system/kubernetes-api in argocd/app-configs/cilium-lb/api-vip.yaml |
The three servers announce it over BGP through Cilium, and k8s-cluster.fzymgc.house resolves to it. The checks are in docs/operations/cilium-kpr.md |
The Plans give rancher/k3s-upgrade with no tag. The controller adds the tag from spec.version,
with + written as -. Do not add a tag to the image. A tagged image runs the version in its tag,
and the controller then records an upgrade to spec.version that did not happen.
The controller marks each node with the label plan.upgrade.cattle.io/<plan>=<hash>. The hash covers
spec.version, the service account, the annotations and the secrets of the Plan, but not the image.
A node whose label equals the current hash of its Plan is done, and the controller does nothing more
on it.
Upgrade k3s
Section titled “Upgrade k3s”-
Review the Renovate pull request
update k3s. It changesspec.versionin both Plans. A minor update is a Kubernetes minor version. Read the upstream release notes before you merge it. -
Before you merge, create a Velero backup:
velero backup create pre-k3s-upgrade-$(date +%Y%m%d-%H%M%S) --wait. -
Merge the pull request. ArgoCD syncs the Plans, and the controller starts the upgrade.
-
Watch the jobs and the node versions until every node shows the new version:
Terminal window kubectl --context fzymgc-house -n system-upgrade get plans,jobskubectl --context fzymgc-house get nodes -o wide
If a job fails, read its logs before you change the Plan again.
Add a node
Section titled “Add a node”Run the k3s playbook for the new node only. The play installs the version that the first server runs, so the new node joins at the version of the cluster.
On a new cluster, no server runs k3s yet. The play then installs spec.version of the k3s-server
Plan. To install another version, set k3s_version for that run. The variable has no effect on a
node that already runs k3s.
Do not use the playbook to upgrade k3s. It does not change an installed version, and it does not
drain nodes. Longhorn needs the drain to detach volumes cleanly (docs/operations/longhorn.md).
The registration address of an agent
Section titled “The registration address of an agent”An agent joins through one address, k8s_cluster_endpoint_ip in
ansible/inventory/group_vars/tp_cluster_nodes.yml, which is the API VIP 192.168.250.254. The
install script writes it once as K3S_URL in /etc/systemd/system/k3s-agent.service.env. After
the first join, the agent learns the addresses of the servers from the cluster and keeps them in
/var/lib/rancher/k3s/agent/etc/k3s-agent-load-balancer.json, so it dials the servers directly and
never the registration address again. An agent that joined before 2026-10 still carries the old
address 10.255.254.6 in its unit environment, which no route reaches, and runs on the persisted
server list. That file does not need a change.
If an agent must point at the current address, for example before every server address changes, run this on the agent:
sudo sed -i 's#^K3S_URL=.*#K3S_URL=https://192.168.250.254:6443#' /etc/systemd/system/k3s-agent.service.envsudo systemctl restart k3s-agent