Longhorn Operations

Operational guide for Longhorn distributed block storage.

Quick Reference

Property	Value
Deployment	Terraform `helm_release` (`tf/cluster-bootstrap/longhorn.tf`) — NOT ArgoCD
TFC Workspace	`main-cluster-bootstrap` (Local execution)
Version variable	`var.longhorn_version` (`tf/cluster-bootstrap/variables.tf`)
Current version	`1.11.2`
Chart repository	`https://charts.longhorn.io/`
Data engine	V1 (`dataEngine = "v1"` on all storage classes)
Default replicas	2
Backup target	Cloudflare R2 (`s3://fzymgc-cluster-storage`)

Storage Classes

Class	Encrypted	Replicas	Notes
`longhorn`	No	2 (default)	Default class
`longhorn-encrypted`	Yes (LUKS)	2	Crypto key from Vault
`longhorn-1replica-encrypted`	Yes (LUKS)	1	Single-replica encrypted

Upgrades

Longhorn is not managed by ArgoCD. Bump var.longhorn_version in tf/cluster-bootstrap/variables.tf and apply via Terraform. The main-cluster-bootstrap workspace runs Local execution — it is not auto-applied on PR merge.

Version policy

Longhorn does not support skipping minor versions. Upgrade through each minor sequentially (e.g. 1.10.x -> 1.11.x -> 1.12.x).
Prefer a mature patch release; avoid .0 releases on this storage layer until a patch follows.
All volumes use the V1 data engine, so V2-data-engine breaking changes and known issues do not apply.
Verify the target chart’s kubeVersion against the running Kubernetes version before upgrading.

Upgrade procedure

Pre-flight: in the Longhorn UI, confirm there are no degraded or faulted volumes and all replicas are healthy.
Update var.longhorn_version and open a PR.

After merge, apply from tf/cluster-bootstrap:

cd tf/cluster-bootstrap
terraform init
terraform plan -out=tfplan   # expect only the Helm release version change
terraform apply tfplan

Watch the longhorn-system namespace: the manager DaemonSet and instance-managers roll. Confirm all longhorn-manager pods reach the new version and volumes stay healthy/attached.
Engine images upgrade automatically: concurrentAutomaticEngineUpgradePerNodeLimit is set to 3 in longhorn.tf defaultSettings, so Longhorn rolls healthy volumes’ engines to the new version in the background (3 per node). Confirm none remain on the old engine image:
Terminal window
```
kubectl -n longhorn-system get volumes.longhorn.io \
  -o jsonpath='{range .items[*]}{.status.currentImage}{"\n"}{end}' | sort | uniq -c
```
The setting stays enabled so future manager bumps also roll engines. Only healthy volumes are upgraded (degraded/faulted ones are skipped until they recover). To pause auto-upgrade during an incident, set the limit back to 0. Engine upgrades are not reversible — see Rollback below.

Rollback

Longhorn does not support downgrading once volume engine images are upgraded. Revert var.longhorn_version only before the engine-image upgrade completes.
For a failed in-progress upgrade, prefer rolling forward to the next patch and restore from R2 backups if data integrity is affected.

Backups

Recurring jobs are defined in tf/cluster-bootstrap/longhorn.tf:

Job	Schedule	Task
`daily-backup`	`0 3 * * *`	Backup to R2
`backup-snapshot-cleanup`	`5 21 * * *`	Snapshot delete
`fstrim`	`0 4 * * *`	Filesystem trim
`system-backup`	`0 0 * * *`	System backup

Node lifecycle and volume attachments

Node upgrades that reboot a node without draining it leave Longhorn RWO VolumeAttachment objects orphaned on the old node, which deadlocks the next pod migration (stuck ContainerCreating) and can freeze a StatefulSet rollout (root cause of the 2026-06-06 vault-2 incident, bead hl-1a4).

This is mitigated in two layers:

Graceful upgrades: the k3s system-upgrade-controller Plans (argocd/app-configs/system-upgrade/plan-{server,agent}.yaml) drain nodes (not just cordon), so pods terminate gracefully and volumes detach cleanly before reboot. Longhorn’s node-drain-policy = block-for-eviction-if-contains-last-replica makes eviction wait until replicas are safe.
Ungraceful node-down: Longhorn node-down-pod-deletion-policy = delete-both-statefulset-and-deployment-pod lets Longhorn remove stranded pods so their volumes can re-attach elsewhere.

Manual recovery (if an orphaned attachment still occurs): delete the stale VolumeAttachment for the affected PVC, then delete the stuck pod so it re-attaches on its scheduled node.

kubectl get volumeattachment -o wide | rg <pvc-name>   # find the stale one
kubectl delete volumeattachment <name>                 # Longhorn detaches
kubectl -n <ns> delete pod <stuck-pod>                 # forces fresh attach