Skip to content

Longhorn Operations

Longhorn is the cluster’s distributed block storage. This page states what exists, where it is defined, and the procedures that touch it.

Property Value
Deployment Terraform helm_release in tf/cluster-bootstrap/longhorn.tf, not ArgoCD
HCP Terraform workspace main-cluster-bootstrap, local execution. A merge does not apply it
Version variable var.longhorn_version in tf/cluster-bootstrap/variables.tf
Current version 1.12.1
Chart repository https://charts.longhorn.io/
Data engine V1 (dataEngine = "v1" on every storage class)
Default replicas 2
Default data path /data/longhorn
Backup target default: s3://fzymgc-k8s-backups@us-west-000/longhorn-backups on Backblaze B2, key from Vault cluster/longhorn/b2 (Secret longhorn-backup-creds-b2)
Encryption key Vault cluster/longhorn/crypto-key, delivered as Secret longhorn-crypto-config (LUKS, aes-xts-plain64, argon2i)
snapshotMaxCount 250
concurrentAutomaticEngineUpgradePerNodeLimit 3
nodeDownPodDeletionPolicy delete-both-statefulset-and-deployment-pod
Class Defined in Replicas Encrypted Notes
longhorn the chart 2 no the default class
longhorn-encrypted longhorn.tf 2 LUKS
longhorn-1replica-encrypted longhorn.tf 1 LUKS
longhorn-retain argocd/app-configs/shared-resources/longhorn-storage-classes.yaml 3 no
longhorn-single the same file 1 no Velero skips every volume on this class by resource policy (argocd/app-configs/velero/resource-policies-configmap.yaml). recurringJobSelector gives a new volume fstrim only, so it gets no Longhorn backup
longhorn-highinode the same file 2 no

Every class uses dataLocality: best-effort and WaitForFirstConsumer binding.

The longhorn-disks Ansible role prepares extra disks on the nodes that declare longhorn_additional_disks in their host_vars (tpi-beta-1 to tpi-beta-3). It loads the btrfs module, checks that each device exists and is named once, formats it as btrfs, and mounts a subvolume under /data with noatime and zstd compression. The role defaults are in ansible/roles/longhorn-disks/defaults/main.yml.

The recurring jobs are RecurringJob manifests in longhorn.tf. Longhorn renders them into CronJobs with no spec.timeZone, so they run in the node zone, America/New_York from ansible/inventory/group_vars/all.yml. Velero’s schedules run in UTC.

Job Schedule (local) Task Retain Groups
daily-backup 0 3 * * * backup to the default target (B2), full backup every 7th run 6 default
backup-snapshot-cleanup 5 21 * * * snapshot delete 7 default
fstrim 0 20 * * * filesystem trim 0 default
system-backup 0 0 * * * system metadata backup, volume-backup-policy: disabled 3 none

Longhorn adds the default group to any volume that has no recurring-job label. To keep a volume out of the backups, give it a job label, not an empty label set. The ClickHouse volumes on longhorn-single carry only recurring-job.longhorn.io/fstrim: enabled, because ClickHouse keeps two replicas and backs up its tables with no TTL (docs/operations/clickstack.md, “Native ClickHouse backups”). Volumes on longhorn-highinode hold rebuildable caches, such as the Renovate cache, and carry the same single label. Read a volume’s jobs:

Terminal window
kubectl -n longhorn-system get volumes.longhorn.io <pv-name> -o json \
| jq '.metadata.labels | with_entries(select(.key | test("recurring")))'

fstrim at 20:00 local runs before every backup in the estate in both EDT and EST. Keep it there. A later local hour loses that ordering against one or more backups, and the RecurringJob spec has no timeZone field, so the cron string is the only lever.

system-backup saves Longhorn metadata without starting or waiting for volume data backups. Recovery therefore depends on daily-backup. Creating a system backup does not make volume data newer, and it does not give a new volume its first backup. Make sure that every volume that matters is in the default group before a planned rebuild.

Each volume backup has one pruning path, and some backups have none:

Backup name Made by Removed by
backup-<hex>, status label RecurringJob: daily-backup daily-backup the job, which keeps the newest 6 per live volume
backup-<hex>, snapshot snapshot-<uuid> Velero through VolumeSnapshotClass longhorn-bak expiry of the Velero backup, through CSI DeleteSnapshot
system-backup-<hex> system-backup when its volume backup policy is not disabled nothing
any backup of a deleted volume either job above nothing

Keep volume-backup-policy: disabled. Deletion of a SystemBackup leaves its volume backups in the store, and daily-backup prunes only the backups that carry its own label. The retention of a recurring job runs only for a live volume, so the backups of a deleted volume stay until you delete them. Count the backups by origin:

Terminal window
kubectl -n longhorn-system get backups.longhorn.io -o json | jq -r '.items[]
| if .metadata.name | startswith("system-backup-") then "system-backup"
elif .status.labels.RecurringJob then .status.labels.RecurringJob
elif .status.snapshotName | startswith("snapshot-") then "velero-csi"
else "other" end' | sort | uniq -c

A successful CronJob is not proof of a successful system backup. The recurring job accepts a terminal Error state and deletes the failed SystemBackup during cleanup, so the older Ready objects stay visible. Read the objects, not the Job:

Terminal window
kubectl -n longhorn-system get recurringjobs.longhorn.io system-backup \
-o jsonpath='{.spec.parameters.volume-backup-policy}{"\n"}' # disabled
kubectl -n longhorn-system get systembackups.longhorn.io \
-l recurring-job.longhorn.io/system-backup=system-backup \
-o custom-columns=NAME:.metadata.name,STATE:.status.state,CREATED:.status.createdAt

For a failure, query default.otel_logs for namespace longhorn-system and SystemBackupController).LogErrorState. The log survives deletion of the failed object. Map an incomplete volume backup through metadata.labels["backup-volume"], because status.volumeName can be empty. A missing live volume is often a temporary Velero upload PVC. Correlate its ID with CSI provisioning and Velero logs before any recovery or cleanup.

longhorn-manager co-owns .spec.cron on every RecurringJob, and server-side apply refuses a change to a co-owned field when the value moves. fstrim and system-backup carry field_manager { force_conflicts = true } for that reason. Add the same block to a job before changing its cron. Do not kubectl edit a RecurringJob: Terraform owns it, and the next apply reverts a live edit.

Trim reaches only snapshots already flagged markRemoved. Whether trim removes them is the cluster setting remove-snapshots-during-filesystem-trim, which this repository does not set. Read it and the node-drain policy live:

Terminal window
kubectl -n longhorn-system get settings.longhorn.io remove-snapshots-during-filesystem-trim node-drain-policy

Retention comes from the recurring jobs. snapshotMaxCount is a backstop against runaway snapshot chains, set to Longhorn’s upstream default of 250. Keep it above the retention floor: backup-snapshot-cleanup retain 7 plus system-backup retain 3 is 10 before any in-flight backup snapshot exists. A cap at or below the floor is self-reinforcing. Snapshot creation fails with snapshot count usage N is equal or larger than snapshotMaxCount N, which surfaces as a generic CSI waitForSnapshotToBeReady: timeout. The backup then fails, so its snapshot is never reaped, and the next backup fails sooner.

The setting is a creation-time default, not a reconciled value. Existing volumes carry their own spec.snapshotMaxCount and do not follow a Terraform change. After changing it, patch the volumes:

Terminal window
# Audit current per-volume values
kubectl get volumes.longhorn.io -n longhorn-system \
-o json | jq -r '[.items[].spec.snapshotMaxCount] | group_by(.) |
map({snapshotMaxCount: .[0], volumes: length})'
# Patch every volume still on the old cap
kubectl get volumes.longhorn.io -n longhorn-system -o name | while read -r v; do
kubectl patch -n longhorn-system "$v" --type=merge \
-p '{"spec":{"snapshotMaxCount":250}}'
done

A volume already over its cap cannot be backed up until it drops under. Raising its per-volume value is what clears it, because a purge needs a healthy attached volume and cannot run while the volume is stranded.

Longhorn does not support skipping a minor version. Upgrade through each minor in order, for example 1.10.x to 1.11.x to 1.12.x. Within one hop, go straight to the newest patch of the target minor. An intermediate patch of the source minor buys nothing and costs a second irreversible engine-image roll. Prefer a patch release over a .0 release. Make sure that the target chart’s kubeVersion matches the running Kubernetes version.

  1. Gather the pre-flight facts. Every volume is on the V1 data engine, so V2-only breaking changes do not apply:

    Terminal window
    # Every volume on V1, attached and healthy
    kubectl -n longhorn-system get volumes.longhorn.io \
    -o custom-columns=NAME:.metadata.name,ENGINE:.spec.dataEngine,STATE:.status.state,ROBUST:.status.robustness
    # Any backing images
    kubectl -n longhorn-system get backingimages.longhorn.io
    # Any RWX volumes
    kubectl get pv -o jsonpath='{range .items[?(@.spec.accessModes[0]=="ReadWriteMany")]}{.metadata.name}{"\n"}{end}'
  2. In the Longhorn UI, make sure that no volume is degraded or faulted.

  3. Update var.longhorn_version and open a PR.

  4. After the merge, apply from tf/cluster-bootstrap:

    Terminal window
    cd tf/cluster-bootstrap
    terraform init
    terraform plan -out=tfplan # expect only the Helm release version change
    terraform apply tfplan
  5. Watch longhorn-system. The manager DaemonSet and the instance managers roll. Make sure that every longhorn-manager pod reaches the new version and that every volume stays healthy and attached.

  6. Engine images upgrade by themselves, 3 per node, on healthy volumes only. Make sure that none remain on the old image:

    Terminal window
    kubectl -n longhorn-system get volumes.longhorn.io \
    -o jsonpath='{range .items[*]}{.status.currentImage}{"\n"}{end}' | sort | uniq -c

    To pause the automatic upgrade during an incident, set the limit to 0.

Longhorn does not support a downgrade once volume engine images are upgraded. Revert var.longhorn_version only before the engine-image upgrade completes. For a failed in-progress upgrade, roll forward to the next patch and restore from a backup if data integrity is affected.

  1. Add the device to longhorn_additional_disks in the node’s host_vars, with a name that is unique on the node and a device path that resolves.
  2. Run the longhorn-disks role for that node only, with --limit.
  3. In the Longhorn UI or with kubectl -n longhorn-system get nodes.longhorn.io <node> -o yaml, make sure that the disk appears under spec.disks with the mounted path.

A node that reboots without a drain leaves Longhorn RWO VolumeAttachment objects on the old node. The next pod placement then sits in ContainerCreating, and a StatefulSet rollout can freeze. Two layers reduce this. The k3s system-upgrade Plans (argocd/app-configs/system-upgrade/plan-{server,agent}.yaml) drain nodes, not only cordon them, so pods terminate and volumes detach before the reboot. For an ungraceful node loss, nodeDownPodDeletionPolicy lets Longhorn remove stranded pods so that their volumes re-attach elsewhere.

If an orphaned attachment still occurs, delete the stale VolumeAttachment for the PVC, then delete the stuck pod so that it re-attaches on its scheduled node:

Terminal window
kubectl get volumeattachment -o wide | rg <pvc-name> # find the stale one
kubectl delete volumeattachment <name> # Longhorn detaches
kubectl -n <ns> delete pod <stuck-pod> # forces a fresh attach

Detach stuck on luksClose … is still in use

Section titled “Detach stuck on luksClose … is still in use”

If the old node’s longhorn-csi-plugin logs NodeUnstageVolume failing with that error, another pod still holds the volume’s globalmount in its own mount namespace, so the dm-crypt device cannot close. Do not delete the VolumeAttachment. Find the holder and delete that pod:

Terminal window
ssh <old-node> 'for f in /proc/[0-9]*/mountinfo; do sudo grep -q <pvc-name> $f && p=${f#/proc/} && echo "${p%/mountinfo} $(cat /proc/${p%/mountinfo}/comm)"; done'

Any pod that hostPath-mounts /var/lib, /var or / must set mountPropagation: HostToContainer on that mount. The default None pins every Longhorn LUKS globalmount on the node and blocks every detach. Vector (argocd/cluster-app/templates/monitoring-vector.yaml) sets it. A new log or node agent with a hostPath mount is the suspect.

  1. In the Longhorn UI, open Backup, pick the volume and the backup, and restore it to a new volume name.
  2. Create a PV and PVC for the restored volume, or use the UI’s PV/PVC creation, in the namespace the workload runs in.
  3. Point the workload at the new PVC, or scale it down, swap the PVC name, and scale it up.

Velero excludes the longhorn-system namespace and skips longhorn-single volumes, so a Velero restore never brings back a Longhorn object or a single-replica volume. Longhorn objects come from Longhorn system backups. The ClickHouse tables with no TTL come from the native backups of ClickHouse. The other ClickHouse tables come back only from the other replica.