Longhorn Operations

Operational guide for Longhorn distributed block storage.

Quick Reference

Property	Value
Deployment	Terraform helm_release (tf/cluster-bootstrap/longhorn.tf) — NOT ArgoCD
TFC Workspace	main-cluster-bootstrap (Local execution)
Version variable	var.longhorn_version (tf/cluster-bootstrap/variables.tf)
Current version	1.12.1
Chart repository	https://charts.longhorn.io/
Data engine	V1 (dataEngine = "v1" on all storage classes)
Default replicas	2
Backup target	Cloudflare R2 (s3://fzymgc-cluster-storage)

Storage Classes

Class	Encrypted	Replicas	Notes
longhorn	No	2 (default)	Default class
longhorn-encrypted	Yes (LUKS)	2	Crypto key from Vault
longhorn-1replica-encrypted	Yes (LUKS)	1	Single-replica encrypted

Upgrades

Longhorn is not managed by ArgoCD. Bump var.longhorn_version in tf/cluster-bootstrap/variables.tf and apply via Terraform. The main-cluster-bootstrap workspace runs Local execution — it is not auto-applied on PR merge.

Version policy

• Longhorn does not support skipping minor versions. Upgrade through each minor sequentially (e.g. 1.10.x -> 1.11.x -> 1.12.x). Within one minor hop, go straight to the newest patch of the target minor — an intermediate patch of the source minor buys nothing and costs a second irreversible engine-image roll (see Rollback).
• Prefer a mature patch release; avoid .0 releases on this storage layer until a patch follows.
• All volumes use the V1 data engine, so V2-data-engine breaking changes and known issues do not apply.
• Verify the target chart’s kubeVersion against the running Kubernetes version before upgrading.

Pre-flight evidence to gather

Record the answers rather than assuming them — several v1.12.0 breaking changes are V2-only or backing-image-only and are discharged by a one-line query:

# All volumes on the V1 data engine, attached and healthy?
kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,ENGINE:.spec.dataEngine,STATE:.status.state,ROBUST:.status.robustness

# Any backing images? (v1.12.0 REMOVED V2 backing images — migrate before upgrading)
kubectl -n longhorn-system get backingimages.longhorn.io

# Any RWX volumes? (the encrypted-migratable live-migration constraint needs one)
kubectl get pv -o jsonpath='{range .items[?(@.spec.accessModes[0]=="ReadWriteMany")]}{.metadata.name}{"\n"}{end}'

Upgrade procedure

1. Pre-flight: in the Longhorn UI, confirm there are no degraded or faulted volumes and all replicas are healthy.

2. Update var.longhorn_version and open a PR.

3. After merge, apply from tf/cluster-bootstrap:

   cd tf/cluster-bootstrap
   terraform init
   terraform plan -out=tfplan   # expect only the Helm release version change
   terraform apply tfplan

4. Watch the longhorn-system namespace: the manager DaemonSet and instance-managers roll. Confirm all longhorn-manager pods reach the new version and volumes stay healthy/attached.

5. Engine images upgrade automatically: concurrentAutomaticEngineUpgradePerNodeLimit is set to 3 in longhorn.tf defaultSettings, so Longhorn rolls healthy volumes’ engines to the new version in the background (3 per node). Confirm none remain on the old engine image:

   kubectl -n longhorn-system get volumes.longhorn.io \
     -o jsonpath='{range .items[*]}{.status.currentImage}{"\n"}{end}' | sort | uniq -c

   The setting stays enabled so future manager bumps also roll engines. Only healthy volumes are upgraded (degraded/faulted ones are skipped until they recover). To pause auto-upgrade during an incident, set the limit back to 0. Engine upgrades are not reversible — see Rollback below.

Rollback

• Longhorn does not support downgrading once volume engine images are upgraded. Revert var.longhorn_version only before the engine-image upgrade completes.
• For a failed in-progress upgrade, prefer rolling forward to the next patch and restore from R2 backups if data integrity is affected.

Backups

Recurring jobs are defined in tf/cluster-bootstrap/longhorn.tf:

Job	Schedule	Task
daily-backup	0 3 * * *	Backup to R2
backup-snapshot-cleanup	5 21 * * *	Snapshot delete
fstrim	0 20 * * *	Filesystem trim
system-backup	0 0 * * *	System metadata backup (volume backup policy disabled)

System metadata backups

system-backup uses volume-backup-policy: disabled: it saves Longhorn metadata without starting or waiting for volume data backups. daily-backup backs up volumes in the default group separately. Recovery therefore depends on those successful volume backups; creating a system backup does not make volume data newer or give a new volume its first backup. Check volume-backup coverage before a planned rebuild.

Issue #2151 exposed why these jobs must be independent. At 04:01 UTC on September 19, Velero created temporary upload PVC velero/agent-memory-pvc-backup-20260919040026-4nc7x (volume pvc-cc9b7b34-1b32-4208-bd77-15e1618a18b4). The system backup’s previous if-not-present policy attempted a data backup of that transient volume. During replica rebuilding, the engine controller deleted its snapshot CR at 04:03:18; backup system-backup-6d489bc0386d4291 then failed with a missing-snapshot error. The system-backup controller reported the resulting system backup as failed at 04:05:55. This was not the scheduled backup-snapshot-cleanup job. Velero’s own DataUpload for source PVC agent-memory/qdrant-data completed at 04:04:09; the failed Longhorn backup was of its temporary copy, not that source PVC.

A successful CronJob is not proof of a successful system backup. The recurring job accepted the terminal Error state and deleted the failed SystemBackup during cleanup, leaving the older Ready objects visible. The December 2025 objects and test object lack this recurring job’s label and are not evidence that its retention policy failed. Do not delete historical backups merely to make the list look fresh.

Deploy this change through the local-execution main-cluster-bootstrap Terraform workspace. Merging the PR alone does not apply it. The policy field is co-owned by longhorn-manager, so this resource uses field_manager.force_conflicts as well. After apply, check the policy and, after the next midnight-local run, require a new Ready system backup with a populated status.createdAt:

kubectl -n longhorn-system get recurringjobs.longhorn.io system-backup \
  -o jsonpath='{.spec.parameters.volume-backup-policy}{"\n"}'

kubectl -n longhorn-system get systembackups.longhorn.io \
  -l recurring-job.longhorn.io/system-backup=system-backup \
  -o custom-columns=NAME:.metadata.name,STATE:.status.state,CREATED:.status.createdAt

For failures, query default.otel_logs for namespace longhorn-system and SystemBackupController).LogErrorState; this survives deletion of failed objects. Map an incomplete volume backup through metadata.labels["backup-volume"], even when status.volumeName is empty. A missing live volume may have been a temporary Velero upload PVC; correlate its ID with CSI provisioning and Velero logs before attempting recovery or cleanup.

Freshness monitoring must measure the age of the newest successful system backup, failed backups, and missing telemetry independently. CronJob success cannot supply that signal. Follow-up #2153 tracks this under the metric-based alerting policy; no new backup alert is deployed by this configuration change.

Schedule ordering

Longhorn crons and Velero crons are not read in the same clock. Longhorn RecurringJobs are rendered into Kubernetes CronJobs with no spec.timeZone, so they evaluate in the node zone — America/New_York, set in ansible/inventory/group_vars/all.yml. Velero evaluates its own Schedule objects (argocd/app-configs/velero/backup-schedule.yaml, argocd/app-configs/velero/agent-memory-backup-schedule.yaml) in UTC. The Schedule column above is local time; Velero’s schedules elsewhere in the estate are not.

fstrim at 0 20 * * * runs ahead of every backup in the estate, year-round. 20:00 local is 00:00Z while EDT is in effect and 01:00Z once EST is, which puts the trim ahead of backup-snapshot-cleanup (01:05Z), Velero’s daily backup (02:00Z), both 04:00Z jobs, and the Longhorn daily-backup (07:00Z). A later local hour loses that ordering against one or more of them, and the RecurringJob CRD has no timeZone field — its spec is exactly concurrency, cron, groups, labels, name, parameters, retain, task — so the cron string is the only lever available.

Changing a RecurringJob cron needs force_conflicts. longhorn-manager co-owns .spec.cron on all four RecurringJobs — it claimed the field when it created each object on 2025-08-30 and has not rewritten it since, its ongoing writes being to .status only. Server-side apply refuses a change to a co-owned field, but only when the value actually moves, so a cron edit is the first thing that trips it; every earlier apply was a no-op and passed. Field-manager ownership does not lapse on its own — only the same manager re-applying without the field releases it, and longhorn-manager never does — so the resource carries a field_manager { force_conflicts = true } block. fstrim and system-backup have one today; the other two need it when a co-owned field changes. Do not kubectl edit the object instead: Terraform owns it, and a live edit is reverted on the next apply while leaving the codified and running values disagreeing meanwhile.

Trim removes already-marked snapshot chains, and that is intended. Volumes leave spec.unmapMarkSnapChainRemoved at ignored, which inherits the cluster-wide remove-snapshots-during-filesystem-trim setting, and that setting is true. Engines therefore report status.unmapMarkSnapChainRemovedEnabled: true, so snapshot-chain removal during trim is enabled. This is Longhorn’s intended behaviour: it reaches only snapshots already flagged markRemoved, and it is what lets the trim reclaim the blocks those snapshots pin — which is the point of running it before the backup window.

Snapshot retention and snapshotMaxCount

Retention is governed by the recurring jobs above, not by the cap. snapshotMaxCount (longhorn.tf defaultSettings) is a backstop against runaway snapshot chains and is set to Longhorn’s upstream default of 250.

It was previously 10, which sat below the retention policy’s own floor — backup-snapshot-cleanup retain=7 plus system-backup retain=3 reaches 10 before any in-flight backup snapshot exists. Volumes therefore hit the cap in normal operation, and hitting it is self-reinforcing:

1. Snapshot creation fails with snapshot count usage N is equal or larger than snapshotMaxCount N, surfacing as a generic CSI waitForSnapshotToBeReady: timeout.
2. The backup fails, so auto-cleanup-recurring-job-backup-snapshot never reaps that backup’s snapshot.
3. The count grows, and the next backup fails sooner.

The setting is a creation-time default, not a reconciled value. Existing volumes carry their own spec.snapshotMaxCount and are not updated when the Terraform value changes. After changing it, patch existing volumes:

# Audit current per-volume values
kubectl get volumes.longhorn.io -n longhorn-system \
  -o json | jq -r '[.items[].spec.snapshotMaxCount] | group_by(.) |
    map({snapshotMaxCount: .[0], volumes: length})'

# Patch every volume still on the old cap
kubectl get volumes.longhorn.io -n longhorn-system -o name | while read -r v; do
  kubectl patch -n longhorn-system "$v" --type=merge \
    -p '{"spec":{"snapshotMaxCount":250}}'
done

A volume already over its cap is unbackupable until it drops under — raising its per-volume value is what clears it, since purge itself requires a healthy attached volume and cannot run while the volume is stranded.

Node lifecycle and volume attachments

Node upgrades that reboot a node without draining it leave Longhorn RWO VolumeAttachment objects orphaned on the old node, which deadlocks the next pod migration (stuck ContainerCreating) and can freeze a StatefulSet rollout (root cause of the 2026-06-06 vault-2 incident, bead hl-1a4).

This is mitigated in two layers:

• Graceful upgrades: the k3s system-upgrade-controller Plans (argocd/app-configs/system-upgrade/plan-{server,agent}.yaml) drain nodes (not just cordon), so pods terminate gracefully and volumes detach cleanly before reboot. Longhorn’s node-drain-policy = block-for-eviction-if-contains-last-replica makes eviction wait until replicas are safe.
• Ungraceful node-down: Longhorn node-down-pod-deletion-policy = delete-both-statefulset-and-deployment-pod lets Longhorn remove stranded pods so their volumes can re-attach elsewhere.

Manual recovery (if an orphaned attachment still occurs): delete the stale VolumeAttachment for the affected PVC, then delete the stuck pod so it re-attaches on its scheduled node.

kubectl get volumeattachment -o wide | rg <pvc-name>   # find the stale one
kubectl delete volumeattachment <name>                 # Longhorn detaches
kubectl -n <ns> delete pod <stuck-pod>                 # forces fresh attach

Detach stuck on luksClose … is still in use: if the old node’s longhorn-csi-plugin logs NodeUnstageVolume failing with that error, another pod still has the volume’s globalmount in its own mount namespace, so the dm-crypt device cannot close. Do not delete the VolumeAttachment; find the holder and delete that pod. Vector used to be the holder on every node (hostPath /var/lib without mountPropagation, #2077). It now mounts with HostToContainer, so any other hostPath /var/lib agent is the suspect.

ssh <old-node> 'for f in /proc/[0-9]*/mountinfo; do sudo grep -q <pvc-name> $f && p=${f#/proc/} && echo "${p%/mountinfo} $(cat /proc/${p%/mountinfo}/comm)"; done'