Longhorn vs OpenEBS: Persistent Storage for Kubernetes That Isn't a Nightmare

Kubernetes storage has a reputation, and the reputation is “here be dragons.” I earned my own scar early: a Postgres pod that rescheduled onto a different node during a routine reboot and came up staring at an empty disk, because its data had politely stayed behind on the node it left. That is the persistent-volume problem in one sentence. The moment you move past stateless web apps and want to run a database, a wiki, or anything that remembers things across a restart, you hit it. In the cloud you’d shrug and attach an EBS volume that follows the pod around. In a homelab or on-prem cluster, you have a pile of machines with local disks and nothing shared binding them together, so a pod that lands on a new node finds its data marooned on the old one.

The fix is replicated block storage: software that takes the disks scattered across your nodes and presents them as one logical pool, keeping synchronised copies of each volume on multiple machines so a pod can move and find its data already waiting for it. Two open-source projects own this space for self-hosters: Longhorn and OpenEBS. They overlap heavily, they solve the same fundamental problem, and they differ in exactly the ways that will decide which one you should run. This is the comparison I wish I’d had before I picked.

What replicated storage is doing for you

When a pod claims a PersistentVolumeClaim, the storage provider carves out a volume and synchronously replicates every write to copies on other nodes before acknowledging it. Lose a node, and a replica elsewhere takes over; the pod reschedules onto a surviving node and reattaches, none the wiser. That’s the whole pitch, and it’s genuinely magic the first time you watch a pod survive a node you’ve just yanked the power from. The cost is twofold and unavoidable: write latency, because every write must wait for the replicas to acknowledge over the network, and disk space, because three replicas means three times the raw storage for the same usable capacity. There is no free lunch here — you are trading capacity and speed for the ability to lose a machine without losing data. Sizing that trade against the workload is half the job, and it’s the same discipline as getting your resource requests and limits right: guess wrong and the cluster punishes you at the worst possible moment.

Both tools expose this through a standard StorageClass, so your application manifests don’t change. You just ask for storage and a healthy volume appears:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi

Swap longhorn for openebs-replicated and the manifest is otherwise identical. The interesting differences are underneath.

Longhorn: the friendly one

Longhorn, which came out of Rancher, is the easiest distributed storage you’ll meet. Installation is a Helm chart or a single manifest, and crucially it ships a genuinely good web UI. You can see every volume, every replica, where each copy lives, and whether the system is healthy — all without learning a new query language. Snapshots, scheduled backups to an S3 bucket or NFS share, and volume expansion are all point-and-click.

1
2
3

helm install longhorn longhorn/longhorn \
  --namespace longhorn-system --create-namespace \
  --set defaultSettings.defaultReplicaCount=3

Its classic (v1) engine runs entirely in userspace using a per-volume engine process, which keeps it approachable and easy to reason about. The trade-off is performance: that userspace path adds overhead, so v1 Longhorn is comfortable but not the fastest thing on the block. Recent releases add a v2 engine built on the SPDK framework that closes much of the gap and pushes Longhorn towards the high-performance camp, though it carries stricter hardware expectations of its own. For databases of modest size, wikis, and the general run of homelab workloads, classic Longhorn is more than quick enough, and the operational sanity it gives you is worth a great deal at 2am when you’re trying to work out why a volume is degraded.

The honest caveats: Longhorn really wants a clean filesystem path on each node (it defaults to /var/lib/longhorn), it expects the iSCSI client (open-iscsi) installed and running on every host — a missing open-iscsi is the single most common “why won’t my volume attach” support thread — and three replicas on a three-node cluster means losing one node leaves you with no spare node onto which the system can re-replicate, so you’re running exposed until it’s back. Plan for replicas + 1 nodes if you want the system to self-heal after a failure rather than merely survive it.

OpenEBS: the toolkit

OpenEBS is less a single product than a family, and that’s the first thing to get straight because it causes real confusion. Under the OpenEBS banner sit several distinct storage engines with wildly different characteristics. The older “local PV” engines just provision node-local volumes with no replication at all — genuinely fast and dead simple, but a node loss means data loss, so they’re for caches and scratch space, not your database. Older replicated engines (Jiva, cStor) offered mirrored pools with snapshots. The grown-up, current option is its Mayastor engine (now branded OpenEBS Replicated Storage), a cloud-native data plane written in Rust that uses NVMe-oF over TCP and the SPDK framework to push for much higher throughput and lower latency than Longhorn’s classic userspace approach. When people say “OpenEBS is fast,” they almost always mean Mayastor specifically — so be sure you’re comparing like with like and not accidentally benchmarking a local-PV engine against Longhorn’s replicated one.

That performance comes at a cost in fussiness. Mayastor wants hugepages configured on every node, strongly prefers a dedicated NVMe device, and pins a CPU core for its IO engine — meaning a core is simply gone from your general workload budget, permanently. There’s no comfortable web UI in the same league as Longhorn’s; you live in kubectl and custom resources, inspecting pools and volumes through CRDs rather than a dashboard. When it works it is genuinely quick — NVMe-over-TCP with a Rust data plane is a different class of throughput — but the setup is closer to commissioning a small SAN than clicking “install”, and it assumes you’re comfortable at that level. If your storage is really living on a separate box anyway, it’s worth first reading why I’d usually not bolt a QNAP NAS on as a Kubernetes backend — the same “keep the storage close and simple” instinct applies here.

Troubleshooting: what actually breaks

The failures cluster around a few predictable places, and knowing them saves a lot of 2am guessing:

• Longhorn volume won’t attach (“failed to attach”, stuck in attaching). Almost always missing or stopped open-iscsi on the target node, or the iSCSI kernel module not loaded. Install it, enable the service, reboot the node if in doubt.
• Degraded volumes that never rebuild. You don’t have a spare node to rebuild onto (three replicas, three nodes, one down). The system is working as designed; add a node or lower the replica count temporarily.
• Mayastor pool won’t come up. Usually hugepages aren’t configured, or the reserved count is too low. Check /proc/meminfo for HugePages_Total on each node before blaming Mayastor.
• Terrible write performance on Longhorn. Often the disks are slow and you’ve stacked three synchronous replicas on top. Measure the underlying disk first; replication multiplies whatever latency the raw device already has.
• Data intact but pod won’t start after a node loss. Check the volume actually finished re-attaching and the filesystem mounted cleanly; a ReadWriteOnce volume can be briefly stuck if the old node hasn’t fully released it. Fencing and the detach timeout settings govern how fast that clears.

How to actually choose

• Ease of operation: Longhorn wins decisively. The UI, the scheduled backups, the at-a-glance visibility of where every replica lives — it’s built for humans who have other jobs and don’t want to become full-time storage administrators.
• Raw performance: OpenEBS Mayastor wins on classic Longhorn, if you’re prepared to feed it hugepages, an NVMe device and a dedicated core. Longhorn’s newer v2 SPDK engine narrows this considerably, so the gap is smaller than it was a year ago.
• Hardware demands: classic Longhorn runs happily on ordinary SATA disks and modest nodes. Mayastor (and Longhorn v2) wants fast storage and tuned kernels to justify itself; put it on spinning rust and you’ve paid the complexity tax for nothing.
• Backups: Longhorn’s scheduled snapshot-to-S3-or-NFS is first-class straight out of the box, driven from the UI. OpenEBS leaves considerably more of that plumbing to you.
• Failure visibility: when a volume degrades at 2am, Longhorn shows you which replica on which node is unhealthy in a couple of clicks. With Mayastor you’re reading CRD status and cross-referencing pods. Both recover; one tells you what it’s doing.

The pattern is consistent: Longhorn optimises for the operator’s sanity, OpenEBS Mayastor optimises for throughput. Neither choice is wrong, but they’re answering different questions, and picking the one that answers yours matters more than any benchmark number.

The verdict

For ninety per cent of self-hosters and small clusters, install Longhorn and move on with your life. I mean that literally — it is the sensible default, and the fact that it’s the boring answer is exactly the point. It turns the genuinely scary problem of stateful Kubernetes into a solved one, and the web UI means that when something goes wrong you can actually see what, rather than divining it from logs. It is not the fastest option on paper, but “fast enough and I can understand it under pressure” beats “blistering and opaque” for anything you personally have to keep alive.

Reach for OpenEBS Mayastor when — and only when — you have measured a real performance need: a busy database that you’ve confirmed is IO-bound on Longhorn, not merely one you assume will be. Pair that with the NVMe hardware, the spare cores, and the patience to tune it, and it rewards you. It is the right tool for a specific, demanding job; it is not the sensible starting point, and choosing it prematurely means paying a steep operational tax for headroom you may never use.

Whichever you pick, do one thing before you trust either with anything you’d miss: confirm your backups actually restore. Replicated storage protects you from a dead node. It does absolutely nothing to protect you from a dropped table, a bad migration, or a rm in the wrong pod — those get faithfully replicated to all three copies in milliseconds. Take a snapshot, delete some data on purpose, restore it, and watch it come back. Do that once, on a quiet afternoon, and you’ll sleep better every night afterwards. Untested backups are just hope with extra steps, and hope is not a storage strategy.