Skip to content

etcd Snapshot Restore Runbook

This runbook restores the k3s control plane from an etcd snapshot. etcd is the key-value store that holds every Kubernetes object. It runs inside k3s on the three servers tpi-alpha-1, tpi-alpha-2 and tpi-alpha-3.

The procedure follows the k3s documentation for k3s etcd-snapshot. Rehearse it on a scratch cluster before the first real use.

etcd snapshot Velero
Mechanism k3s etcd-snapshot, built into k3s on each server Velero Schedule
What it captures Every API object in etcd, and the k3s bootstrap data Namespaced API objects and volume data, outside the excluded namespaces
Where /var/lib/rancher/k3s/server/db/snapshots/ on each server The b2 backup location: B2 bucket fzymgc-k8s-backups, prefix velero/backups
Schedule The k3s default. ansible/roles/k3s-common renders no snapshot key daily-backup at 0 2 * * * with a 720h ttl, weekly-full-backup at 0 3 * * 0 with a 2160h ttl
Size Read it with sudo ls -l on the snapshot directory Read it with velero backup describe <name>
Configuration /etc/rancher/k3s/config.yaml, rendered by ansible/roles/k3s-common argocd/app-configs/velero/backup-schedule.yaml

Each server takes its own scheduled snapshots. kubectl get etcdsnapshotfile lists the snapshots of all three servers. A listing of the snapshot directory on a node needs sudo.

Item Path
Snapshot directory /var/lib/rancher/k3s/server/db/snapshots/
etcd data directory /var/lib/rancher/k3s/server/db/
Server token /var/lib/rancher/k3s/server/token on the first server
k3s configuration /etc/rancher/k3s/config.yaml

The snapshot encrypts the bootstrap data with a key that k3s derives from the server token. A restore needs the token that was in use when k3s took the snapshot.

Do not print the token, and do not copy it off the node. The token is the root secret of the cluster.

Before a change to k3s on a server, take one etcd snapshot and one Velero backup:

Terminal window
# 1. Save an etcd snapshot on tpi-alpha-1. It writes on-demand-tpi-alpha-1-<unix time>.
ssh fzymgc@tpi-alpha-1 sudo k3s etcd-snapshot save
# 2. List the snapshots. Unknown flag warnings come before the list.
ssh fzymgc@tpi-alpha-1 sudo k3s etcd-snapshot ls
# 3. Take a Velero backup and wait for it to finish.
velero backup create pre-deploy-$(date -u +%Y%m%d-%H%M%S) --wait
# 4. Record the backup name and its status.
velero backup get

Restore on the first server only. The first server is the one whose configuration carries cluster-init: true. The other two carry server: https://<first server>:6443 and join it. If you restore on a joining server, the first server starts with an empty data directory under cluster-init and creates a new, empty cluster.

Find the first server before you start:

Terminal window
for n in tpi-alpha-1 tpi-alpha-2 tpi-alpha-3; do
echo "== $n"; ssh fzymgc@$n sudo grep -E '^(cluster-init|server):' /etc/rancher/k3s/config.yaml
done

The commands below assume that tpi-alpha-1 is the first server, which is what the Ansible inventory declares. If the read above says otherwise, swap the names.

A restore stops the Kubernetes API for the whole cluster. The five agents are not part of the restore. Do not stop k3s-agent on them.

Terminal window
# 1. Pick the snapshot on tpi-alpha-1. The directory listing works when k3s is down.
# `k3s etcd-snapshot ls` needs a running k3s server.
ssh fzymgc@tpi-alpha-1 sudo ls -l /var/lib/rancher/k3s/server/db/snapshots/
SNAP=<snapshot file name>
# 2. Stop k3s on the three servers.
for n in tpi-alpha-3 tpi-alpha-2 tpi-alpha-1; do ssh fzymgc@$n sudo systemctl stop k3s; done
# 3. Reset etcd and restore the snapshot on tpi-alpha-1.
ssh fzymgc@tpi-alpha-1 sudo k3s server --cluster-reset \
--cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/$SNAP
# Wait for: Managed etcd cluster membership has been reset, restart without --cluster-reset flag now
# 4. Start k3s on tpi-alpha-1.
ssh fzymgc@tpi-alpha-1 sudo systemctl start k3s
# 5. Wait until tpi-alpha-1 is Ready. If the wait fails, the loop does not start.
# The loop moves the old etcd data aside on the other two servers and starts k3s,
# one server at a time. The moved directory keeps the local snapshots of the server.
kubectl --context fzymgc-house wait --for=condition=Ready node/tpi-alpha-1 --timeout=10m &&
for n in tpi-alpha-2 tpi-alpha-3; do
ssh fzymgc@$n 'sudo mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.pre-restore-$(date -u +%Y%m%dT%H%M%SZ) && sudo systemctl start k3s'
done

k3s server reads /etc/rancher/k3s/config.yaml, so the reset in step 3 uses the same keys as the unit. --cluster-reset removes every etcd member except tpi-alpha-1.

Make sure that the cluster is back:

Terminal window
kubectl --context fzymgc-house get nodes # all eight nodes Ready
kubectl --context fzymgc-house get etcdsnapshotfile # snapshots from the three servers

After the restore, ArgoCD syncs the applications from Git again.

Step 5 leaves one db.pre-restore-<UTC time> directory on tpi-alpha-2 and one on tpi-alpha-3. Each directory holds the old etcd data and the local snapshots of that server. k3s does not read it. Keep it until all eight nodes are Ready and you no longer need the old snapshots. Then remove it:

Terminal window
for n in tpi-alpha-2 tpi-alpha-3; do
ssh fzymgc@$n 'sudo rm -rf /var/lib/rancher/k3s/server/db.pre-restore-*'
done
  • Any change to an API object after the snapshot time. This includes Secrets that External Secrets rotated, Longhorn volume and replica objects, cert-manager certificates and the state of ArgoCD applications. The Longhorn objects and the replicas on disk can disagree after a restore.
  • Data on disk: Longhorn replica data, container images, /var/lib/rancher/k3s/agent, /etc/rancher/k3s and the snapshot files.
  • A changed token. The restore needs the token that was in use when k3s took the snapshot.
  • Node objects for hosts that no longer exist. Delete each one with kubectl delete node <name>.

The Velero backup holds the application side. It excludes kube-system, the networking and storage namespaces, the operators and velero itself. The full list is in argocd/app-configs/velero/backup-schedule.yaml.