etcd Snapshot Restore Runbook
This runbook restores the k3s control plane from an etcd snapshot. etcd is the key-value store
that holds every Kubernetes object. It runs inside k3s on the three servers tpi-alpha-1,
tpi-alpha-2 and tpi-alpha-3.
The procedure follows the k3s documentation for k3s etcd-snapshot. Rehearse it on a scratch
cluster before the first real use.
Two backup paths
Section titled “Two backup paths”| etcd snapshot | Velero | |
|---|---|---|
| Mechanism | k3s etcd-snapshot, built into k3s on each server |
Velero Schedule |
| What it captures | Every API object in etcd, and the k3s bootstrap data | Namespaced API objects and volume data, outside the excluded namespaces |
| Where | /var/lib/rancher/k3s/server/db/snapshots/ on each server |
The b2 backup location: B2 bucket fzymgc-k8s-backups, prefix velero/backups |
| Schedule | The k3s default. ansible/roles/k3s-common renders no snapshot key |
daily-backup at 0 2 * * * with a 720h ttl, weekly-full-backup at 0 3 * * 0 with a 2160h ttl |
| Size | Read it with sudo ls -l on the snapshot directory |
Read it with velero backup describe <name> |
| Configuration | /etc/rancher/k3s/config.yaml, rendered by ansible/roles/k3s-common |
argocd/app-configs/velero/backup-schedule.yaml |
Each server takes its own scheduled snapshots. kubectl get etcdsnapshotfile lists the snapshots
of all three servers. A listing of the snapshot directory on a node needs sudo.
| Item | Path |
|---|---|
| Snapshot directory | /var/lib/rancher/k3s/server/db/snapshots/ |
| etcd data directory | /var/lib/rancher/k3s/server/db/ |
| Server token | /var/lib/rancher/k3s/server/token on the first server |
| k3s configuration | /etc/rancher/k3s/config.yaml |
The snapshot encrypts the bootstrap data with a key that k3s derives from the server token. A restore needs the token that was in use when k3s took the snapshot.
Do not print the token, and do not copy it off the node. The token is the root secret of the cluster.
Take an on-demand backup
Section titled “Take an on-demand backup”Before a change to k3s on a server, take one etcd snapshot and one Velero backup:
# 1. Save an etcd snapshot on tpi-alpha-1. It writes on-demand-tpi-alpha-1-<unix time>.ssh fzymgc@tpi-alpha-1 sudo k3s etcd-snapshot save
# 2. List the snapshots. Unknown flag warnings come before the list.ssh fzymgc@tpi-alpha-1 sudo k3s etcd-snapshot ls
# 3. Take a Velero backup and wait for it to finish.velero backup create pre-deploy-$(date -u +%Y%m%d-%H%M%S) --wait
# 4. Record the backup name and its status.velero backup getRestore the control plane
Section titled “Restore the control plane”Restore on the first server only. The first server is the one whose configuration carries
cluster-init: true. The other two carry server: https://<first server>:6443 and join it. If
you restore on a joining server, the first server starts with an empty data directory under
cluster-init and creates a new, empty cluster.
Find the first server before you start:
for n in tpi-alpha-1 tpi-alpha-2 tpi-alpha-3; do echo "== $n"; ssh fzymgc@$n sudo grep -E '^(cluster-init|server):' /etc/rancher/k3s/config.yamldoneThe commands below assume that tpi-alpha-1 is the first server, which is what the Ansible
inventory declares. If the read above says otherwise, swap the names.
A restore stops the Kubernetes API for the whole cluster. The five agents are not part of the
restore. Do not stop k3s-agent on them.
# 1. Pick the snapshot on tpi-alpha-1. The directory listing works when k3s is down.# `k3s etcd-snapshot ls` needs a running k3s server.ssh fzymgc@tpi-alpha-1 sudo ls -l /var/lib/rancher/k3s/server/db/snapshots/SNAP=<snapshot file name>
# 2. Stop k3s on the three servers.for n in tpi-alpha-3 tpi-alpha-2 tpi-alpha-1; do ssh fzymgc@$n sudo systemctl stop k3s; done
# 3. Reset etcd and restore the snapshot on tpi-alpha-1.ssh fzymgc@tpi-alpha-1 sudo k3s server --cluster-reset \ --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/$SNAP# Wait for: Managed etcd cluster membership has been reset, restart without --cluster-reset flag now
# 4. Start k3s on tpi-alpha-1.ssh fzymgc@tpi-alpha-1 sudo systemctl start k3s
# 5. Wait until tpi-alpha-1 is Ready. If the wait fails, the loop does not start.# The loop moves the old etcd data aside on the other two servers and starts k3s,# one server at a time. The moved directory keeps the local snapshots of the server.kubectl --context fzymgc-house wait --for=condition=Ready node/tpi-alpha-1 --timeout=10m &&for n in tpi-alpha-2 tpi-alpha-3; do ssh fzymgc@$n 'sudo mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.pre-restore-$(date -u +%Y%m%dT%H%M%SZ) && sudo systemctl start k3s'donek3s server reads /etc/rancher/k3s/config.yaml, so the reset in step 3 uses the same keys as
the unit. --cluster-reset removes every etcd member except tpi-alpha-1.
Make sure that the cluster is back:
kubectl --context fzymgc-house get nodes # all eight nodes Readykubectl --context fzymgc-house get etcdsnapshotfile # snapshots from the three serversAfter the restore, ArgoCD syncs the applications from Git again.
Step 5 leaves one db.pre-restore-<UTC time> directory on tpi-alpha-2 and one on
tpi-alpha-3. Each directory holds the old etcd data and the local snapshots of that server.
k3s does not read it. Keep it until all eight nodes are Ready and you no longer need the old
snapshots. Then remove it:
for n in tpi-alpha-2 tpi-alpha-3; do ssh fzymgc@$n 'sudo rm -rf /var/lib/rancher/k3s/server/db.pre-restore-*'doneWhat a restore does not bring back
Section titled “What a restore does not bring back”- Any change to an API object after the snapshot time. This includes Secrets that External Secrets rotated, Longhorn volume and replica objects, cert-manager certificates and the state of ArgoCD applications. The Longhorn objects and the replicas on disk can disagree after a restore.
- Data on disk: Longhorn replica data, container images,
/var/lib/rancher/k3s/agent,/etc/rancher/k3sand the snapshot files. - A changed token. The restore needs the token that was in use when k3s took the snapshot.
- Node objects for hosts that no longer exist. Delete each one with
kubectl delete node <name>.
The Velero backup holds the application side. It excludes kube-system, the networking and
storage namespaces, the operators and velero itself. The full list is in
argocd/app-configs/velero/backup-schedule.yaml.