Longhorn Operations
Longhorn is the cluster’s distributed block storage. This page states what exists, where it is defined, and the procedures that touch it.
What exists
Section titled “What exists”| Property | Value |
|---|---|
| Deployment | Terraform helm_release in tf/cluster-bootstrap/longhorn.tf, not ArgoCD |
| HCP Terraform workspace | main-cluster-bootstrap, local execution. A merge does not apply it |
| Version variable | var.longhorn_version in tf/cluster-bootstrap/variables.tf |
| Current version | 1.12.1 |
| Chart repository | https://charts.longhorn.io/ |
| Data engine | V1 (dataEngine = "v1" on every storage class) |
| Default replicas | 2 |
| Default data path | /data/longhorn |
| Backup target | default: s3://fzymgc-k8s-backups@us-west-000/longhorn-backups on Backblaze B2, key from Vault cluster/longhorn/b2 (Secret longhorn-backup-creds-b2) |
| Encryption key | Vault cluster/longhorn/crypto-key, delivered as Secret longhorn-crypto-config (LUKS, aes-xts-plain64, argon2i) |
snapshotMaxCount |
250 |
concurrentAutomaticEngineUpgradePerNodeLimit |
3 |
nodeDownPodDeletionPolicy |
delete-both-statefulset-and-deployment-pod |
Storage classes
Section titled “Storage classes”| Class | Defined in | Replicas | Encrypted | Notes |
|---|---|---|---|---|
longhorn |
the chart | 2 | no | the default class |
longhorn-encrypted |
longhorn.tf |
2 | LUKS | |
longhorn-1replica-encrypted |
longhorn.tf |
1 | LUKS | |
longhorn-retain |
argocd/app-configs/shared-resources/longhorn-storage-classes.yaml |
3 | no | |
longhorn-single |
the same file | 1 | no | Velero skips every volume on this class by resource policy (argocd/app-configs/velero/resource-policies-configmap.yaml). recurringJobSelector gives a new volume fstrim only, so it gets no Longhorn backup |
longhorn-highinode |
the same file | 2 | no |
Every class uses dataLocality: best-effort and WaitForFirstConsumer binding.
Node disks
Section titled “Node disks”The longhorn-disks Ansible role prepares extra disks on the nodes that declare
longhorn_additional_disks in their host_vars (tpi-beta-1 to tpi-beta-3). It loads the btrfs
module, checks that each device exists and is named once, formats it as btrfs, and mounts a
subvolume under /data with noatime and zstd compression. The role defaults are in
ansible/roles/longhorn-disks/defaults/main.yml.
Backups and recurring jobs
Section titled “Backups and recurring jobs”The recurring jobs are RecurringJob manifests in longhorn.tf. Longhorn renders them into
CronJobs with no spec.timeZone, so they run in the node zone, America/New_York from
ansible/inventory/group_vars/all.yml. Velero’s schedules run in UTC.
| Job | Schedule (local) | Task | Retain | Groups |
|---|---|---|---|---|
daily-backup |
0 3 * * * |
backup to the default target (B2), full backup every 7th run | 6 | default |
backup-snapshot-cleanup |
5 21 * * * |
snapshot delete | 7 | default |
fstrim |
0 20 * * * |
filesystem trim | 0 | default |
system-backup |
0 0 * * * |
system metadata backup, volume-backup-policy: disabled |
3 | none |
Longhorn adds the default group to any volume that has no recurring-job label. To keep a volume
out of the backups, give it a job label, not an empty label set. The ClickHouse volumes on
longhorn-single carry only recurring-job.longhorn.io/fstrim: enabled, because ClickHouse keeps
two replicas and backs up its tables with no TTL (docs/operations/clickstack.md, “Native ClickHouse
backups”). Volumes on
longhorn-highinode hold rebuildable caches, such as the Renovate cache, and carry the same single
label. Read a volume’s jobs:
kubectl -n longhorn-system get volumes.longhorn.io <pv-name> -o json \ | jq '.metadata.labels | with_entries(select(.key | test("recurring")))'fstrim at 20:00 local runs before every backup in the estate in both EDT and EST. Keep it
there. A later local hour loses that ordering against one or more backups, and the RecurringJob
spec has no timeZone field, so the cron string is the only lever.
system-backup saves Longhorn metadata without starting or waiting for volume data backups.
Recovery therefore depends on daily-backup. Creating a system backup does not make volume data
newer, and it does not give a new volume its first backup. Make sure that every volume that
matters is in the default group before a planned rebuild.
Each volume backup has one pruning path, and some backups have none:
| Backup name | Made by | Removed by |
|---|---|---|
backup-<hex>, status label RecurringJob: daily-backup |
daily-backup |
the job, which keeps the newest 6 per live volume |
backup-<hex>, snapshot snapshot-<uuid> |
Velero through VolumeSnapshotClass longhorn-bak |
expiry of the Velero backup, through CSI DeleteSnapshot |
system-backup-<hex> |
system-backup when its volume backup policy is not disabled |
nothing |
| any backup of a deleted volume | either job above | nothing |
Keep volume-backup-policy: disabled. Deletion of a SystemBackup leaves its volume backups in the
store, and daily-backup prunes only the backups that carry its own label. The retention of a
recurring job runs only for a live volume, so the backups of a deleted volume stay until you delete
them. Count the backups by origin:
kubectl -n longhorn-system get backups.longhorn.io -o json | jq -r '.items[] | if .metadata.name | startswith("system-backup-") then "system-backup" elif .status.labels.RecurringJob then .status.labels.RecurringJob elif .status.snapshotName | startswith("snapshot-") then "velero-csi" else "other" end' | sort | uniq -cA successful CronJob is not proof of a successful system backup. The recurring job accepts a
terminal Error state and deletes the failed SystemBackup during cleanup, so the older Ready
objects stay visible. Read the objects, not the Job:
kubectl -n longhorn-system get recurringjobs.longhorn.io system-backup \ -o jsonpath='{.spec.parameters.volume-backup-policy}{"\n"}' # disabled
kubectl -n longhorn-system get systembackups.longhorn.io \ -l recurring-job.longhorn.io/system-backup=system-backup \ -o custom-columns=NAME:.metadata.name,STATE:.status.state,CREATED:.status.createdAtFor a failure, query default.otel_logs for namespace longhorn-system and
SystemBackupController).LogErrorState. The log survives deletion of the failed object. Map an
incomplete volume backup through metadata.labels["backup-volume"], because status.volumeName
can be empty. A missing live volume is often a temporary Velero upload PVC. Correlate its ID with
CSI provisioning and Velero logs before any recovery or cleanup.
longhorn-manager co-owns .spec.cron on every RecurringJob, and server-side apply refuses a
change to a co-owned field when the value moves. fstrim and system-backup carry
field_manager { force_conflicts = true } for that reason. Add the same block to a job before
changing its cron. Do not kubectl edit a RecurringJob: Terraform owns it, and the next apply
reverts a live edit.
Trim reaches only snapshots already flagged markRemoved. Whether trim removes them is the
cluster setting remove-snapshots-during-filesystem-trim, which this repository does not set.
Read it and the node-drain policy live:
kubectl -n longhorn-system get settings.longhorn.io remove-snapshots-during-filesystem-trim node-drain-policySnapshot retention and snapshotMaxCount
Section titled “Snapshot retention and snapshotMaxCount”Retention comes from the recurring jobs. snapshotMaxCount is a backstop against runaway snapshot
chains, set to Longhorn’s upstream default of 250. Keep it above the retention floor:
backup-snapshot-cleanup retain 7 plus system-backup retain 3 is 10 before any in-flight backup
snapshot exists. A cap at or below the floor is self-reinforcing. Snapshot creation fails with
snapshot count usage N is equal or larger than snapshotMaxCount N, which surfaces as a generic CSI
waitForSnapshotToBeReady: timeout. The backup then fails, so its snapshot is never reaped, and the
next backup fails sooner.
The setting is a creation-time default, not a reconciled value. Existing volumes carry their own
spec.snapshotMaxCount and do not follow a Terraform change. After changing it, patch the volumes:
# Audit current per-volume valueskubectl get volumes.longhorn.io -n longhorn-system \ -o json | jq -r '[.items[].spec.snapshotMaxCount] | group_by(.) | map({snapshotMaxCount: .[0], volumes: length})'
# Patch every volume still on the old capkubectl get volumes.longhorn.io -n longhorn-system -o name | while read -r v; do kubectl patch -n longhorn-system "$v" --type=merge \ -p '{"spec":{"snapshotMaxCount":250}}'doneA volume already over its cap cannot be backed up until it drops under. Raising its per-volume value is what clears it, because a purge needs a healthy attached volume and cannot run while the volume is stranded.
Procedures
Section titled “Procedures”Upgrade Longhorn
Section titled “Upgrade Longhorn”Longhorn does not support skipping a minor version. Upgrade through each minor in order, for
example 1.10.x to 1.11.x to 1.12.x. Within one hop, go straight to the newest patch of the
target minor. An intermediate patch of the source minor buys nothing and costs a second
irreversible engine-image roll. Prefer a patch release over a .0 release. Make sure that the
target chart’s kubeVersion matches the running Kubernetes version.
-
Gather the pre-flight facts. Every volume is on the V1 data engine, so V2-only breaking changes do not apply:
Terminal window # Every volume on V1, attached and healthykubectl -n longhorn-system get volumes.longhorn.io \-o custom-columns=NAME:.metadata.name,ENGINE:.spec.dataEngine,STATE:.status.state,ROBUST:.status.robustness# Any backing imageskubectl -n longhorn-system get backingimages.longhorn.io# Any RWX volumeskubectl get pv -o jsonpath='{range .items[?(@.spec.accessModes[0]=="ReadWriteMany")]}{.metadata.name}{"\n"}{end}' -
In the Longhorn UI, make sure that no volume is degraded or faulted.
-
Update
var.longhorn_versionand open a PR. -
After the merge, apply from
tf/cluster-bootstrap:Terminal window cd tf/cluster-bootstrapterraform initterraform plan -out=tfplan # expect only the Helm release version changeterraform apply tfplan -
Watch
longhorn-system. The manager DaemonSet and the instance managers roll. Make sure that everylonghorn-managerpod reaches the new version and that every volume stays healthy and attached. -
Engine images upgrade by themselves, 3 per node, on healthy volumes only. Make sure that none remain on the old image:
Terminal window kubectl -n longhorn-system get volumes.longhorn.io \-o jsonpath='{range .items[*]}{.status.currentImage}{"\n"}{end}' | sort | uniq -cTo pause the automatic upgrade during an incident, set the limit to
0.
Longhorn does not support a downgrade once volume engine images are upgraded. Revert
var.longhorn_version only before the engine-image upgrade completes. For a failed in-progress
upgrade, roll forward to the next patch and restore from a backup if data integrity is affected.
Add a disk to a node
Section titled “Add a disk to a node”- Add the device to
longhorn_additional_disksin the node’s host_vars, with a name that is unique on the node and a device path that resolves. - Run the
longhorn-disksrole for that node only, with--limit. - In the Longhorn UI or with
kubectl -n longhorn-system get nodes.longhorn.io <node> -o yaml, make sure that the disk appears underspec.diskswith the mounted path.
Recover a volume stuck on a node reboot
Section titled “Recover a volume stuck on a node reboot”A node that reboots without a drain leaves Longhorn RWO VolumeAttachment objects on the old
node. The next pod placement then sits in ContainerCreating, and a StatefulSet rollout can
freeze. Two layers reduce this. The k3s system-upgrade Plans
(argocd/app-configs/system-upgrade/plan-{server,agent}.yaml) drain nodes, not only cordon them,
so pods terminate and volumes detach before the reboot. For an ungraceful node loss,
nodeDownPodDeletionPolicy lets Longhorn remove stranded pods so that their volumes re-attach
elsewhere.
If an orphaned attachment still occurs, delete the stale VolumeAttachment for the PVC, then
delete the stuck pod so that it re-attaches on its scheduled node:
kubectl get volumeattachment -o wide | rg <pvc-name> # find the stale onekubectl delete volumeattachment <name> # Longhorn detacheskubectl -n <ns> delete pod <stuck-pod> # forces a fresh attachDetach stuck on luksClose … is still in use
Section titled “Detach stuck on luksClose … is still in use”If the old node’s longhorn-csi-plugin logs NodeUnstageVolume failing with that error, another
pod still holds the volume’s globalmount in its own mount namespace, so the dm-crypt device
cannot close. Do not delete the VolumeAttachment. Find the holder and delete that pod:
ssh <old-node> 'for f in /proc/[0-9]*/mountinfo; do sudo grep -q <pvc-name> $f && p=${f#/proc/} && echo "${p%/mountinfo} $(cat /proc/${p%/mountinfo}/comm)"; done'Any pod that hostPath-mounts /var/lib, /var or / must set
mountPropagation: HostToContainer on that mount. The default None pins every Longhorn LUKS
globalmount on the node and blocks every detach. Vector
(argocd/cluster-app/templates/monitoring-vector.yaml) sets it. A new log or node agent with a
hostPath mount is the suspect.
Restore a volume from a backup
Section titled “Restore a volume from a backup”- In the Longhorn UI, open Backup, pick the volume and the backup, and restore it to a new volume name.
- Create a PV and PVC for the restored volume, or use the UI’s PV/PVC creation, in the namespace the workload runs in.
- Point the workload at the new PVC, or scale it down, swap the PVC name, and scale it up.
Velero excludes the longhorn-system namespace and skips longhorn-single volumes, so a Velero
restore never brings back a Longhorn object or a single-replica volume. Longhorn objects come from
Longhorn system backups. The ClickHouse tables with no TTL come from the native backups of
ClickHouse. The other ClickHouse tables come back only from the other replica.