ZFS Special VDEVs: Metadata on SSD, Bulk on Spinning Rust
The one tuning knob that made my pool feel like a different machine
Contents
The complaint that sent me down this road was mundane. ls -lR on a photo directory with about 90,000 files took eleven seconds on a pool of perfectly healthy 7200rpm drives. Nothing was broken. The pool scrubbed clean, the drives were fine, sequential throughput was a comfortable 400MB/s. But every operation that involved finding things rather than reading things felt like 2009 — rsync’s file-list phase, du -sh, a Nextcloud rescan, anything that walks a tree.
The reason is boring and structural. ZFS stores its metadata — the block pointers, the directory entries, the dnodes, the spacemaps — interleaved with your data on the same vdevs. Metadata reads are small and scattered by nature, and a scattered small read on a spinning disk is a seek, and a seek is roughly ten milliseconds no matter how fast the platter spins. Walk a directory tree and you are asking a mechanical arm to visit tens of thousands of locations. Eleven seconds is physics being charged for honestly.
A special vdev fixes this by moving that class of data off the disks entirely. It’s been in OpenZFS since 0.8, it’s one command to enable, and on my pool it turned that eleven seconds into about four hundred milliseconds. It also carries a caveat sharp enough that I want to state it before the walkthrough rather than after: get it wrong and you lose the entire pool. Every dataset on it.
What a special vdev actually holds
ZFS has an idea called allocation classes. Every block written gets classified, and the allocator picks a vdev based on that class. By default there’s one class and everything goes to the normal vdevs. Add a special vdev and three categories of block get routed to it instead.
All filesystem metadata. Block pointers, dnodes, directory entries, ZAP objects. This is the big win and it’s automatic — no tuning, no property to set. Anything that walks, stats or lists is now hitting flash.
Spacemaps and the pool’s own bookkeeping. These are read constantly during allocation, and putting them on flash makes writes to a fragmented pool noticeably less miserable.
Optionally, small data blocks. This is the special_small_blocks property, it’s per-dataset, and it defaults to zero — off. Set it to 32K and every block smaller than 32K lands on the special vdev along with the metadata.
That third one is where the judgement lives, and I’ll come back to it. The first two are the reason to do this at all, and they need no thought.
Worth being clear about what a special vdev is not, because ZFS has three flash-adjacent features and they get conflated constantly. A L2ARC is a read cache — losing it costs you nothing, it just goes cold. A SLOG absorbs synchronous write intent — losing it costs you the last few seconds of sync writes in a crash, and nothing in a clean shutdown. Both are caches, and both are optional in the strong sense that the pool works without them.
A special vdev is neither. It’s a member of the pool, holding the only copy of the blocks assigned to it. Pull it out and the pool has no metadata, which means the pool has nothing. There is no degraded mode, no reconstruct-from-elsewhere, no “it’ll be slow until you replace it”. The pool is gone. Everyone who has run ZFS for a while has internalised “cache devices are disposable”, and this is the one that punishes that instinct.
Adding one, properly
The rule that follows from the above: the special vdev’s redundancy must match or exceed your data vdevs’. If your pool is a three-way raidz2, a single SSD as special is a single point of failure sitting underneath twelve disks’ worth of protection, and you have carefully engineered a pool whose survival depends on your cheapest component. Mirror it. Two devices minimum, three if the data vdevs are raidz2 and you meant it.
Use different SSDs, ideally from different batches. Two identical drives that have been receiving identical writes since the same afternoon will wear out at approximately the same rate, and a mirror whose members fail together is a stripe with extra steps. This is the same argument as mirrors versus parity makes about correlated failure, and it’s more acute here because the write pattern is genuinely identical rather than merely similar.
| |
| |
Always use /dev/disk/by-id/ paths. /dev/sda is a name assigned by whatever order the kernel enumerated things in this boot, and a pool that reorders its own devices after a BIOS update is an evening you’ll remember.
The ashift check is the other trap. A special vdev added with a different ashift than the pool cannot be removed later, and on a pool containing raidz vdevs it cannot be removed at all regardless. Device removal in OpenZFS works for mirror and stripe vdevs; if there’s a raidz anywhere in the pool, zpool remove refuses. So on a raidz pool, adding a special vdev is a permanent, irreversible decision made with one command and no confirmation prompt. Read the command twice.
The part that needs judgement
Adding the vdev migrates nothing. ZFS routes new writes by allocation class; the metadata already on the disks stays there. So the day you add it, nothing gets faster. Metadata moves as it’s rewritten, which for a mostly-static photo library is approximately never.
The blunt fix is to rewrite the data. On a pool with room, zfs send | zfs recv into a new dataset and swap them — every block gets reallocated, and metadata lands where it should. On a pool without room, a rolling cp and delete per directory does the same job with more anxiety. This is the same mechanism as send/receive for time travel, pointed at a local target.
Then special_small_blocks, which is where you can genuinely make things worse:
| |
The property compares against the dataset’s recordsize. A block smaller than the threshold goes to special; anything at or above goes to the data vdevs. The subtlety people miss: a dataset with recordsize=128K writing a 4K file produces a 4K block, so small files land on flash even without touching the property — the threshold catches partial records, of which small files are the main population.
Set it to 32K on a dataset of source code, mail, or a Nextcloud store and you’ve moved the entire painful population onto flash. Set it equal to recordsize and you’ve told ZFS to put everything on the special vdev, which fills it, and when a special vdev fills, ZFS silently starts spilling to the normal vdevs. No error, no warning, just a pool that quietly stops behaving the way you configured it. Check the fill level like you’d check anything else that can silently degrade — the same instinct as watching SMART before a drive dies:
| |
One more property worth knowing about, because it interacts badly with all of this: dnodesize. ZFS defaults to 512-byte dnodes for compatibility reasons that stopped mattering years ago. Setting dnodesize=auto lets ZFS use larger dnodes, which lets it store extended attributes inline instead of spilling them into separate blocks. On datasets that carry xattrs — anything touched by SELinux, anything storing ACLs, a Nextcloud store — that spill is a second metadata read for every file. Combined with xattr=sa, which stores attributes in the dnode rather than as hidden directories, it removes a whole population of tiny objects your special vdev would otherwise be caching. Both are per-dataset, both only affect new writes, and both are approximately free.
Measuring whether it did anything
Do this before and after, or you’re guessing:
| |
zdb -Lbbbs takes a while and prints a block-level census of the pool by type and size. It’s the only honest answer to “did my metadata actually move”, and it’s how I found out my first attempt had migrated about 4% of it.
My numbers, on a four-disk raidz2 with a mirrored pair of small consumer NVMe drives as special, special_small_blocks=32K on the document datasets and off on media:
findacross 90,000 files: 11.2s → 0.41srsync --dry-runon the same tree: 38s → 3s- Scrub time: 9h20m → 7h05m
- Sequential read throughput: unchanged, as expected
The scrub number surprised me until I thought about it. A scrub reads everything including all the metadata, and the metadata portion was a large fraction of the seeks.
What actually goes wrong
“mismatched replication level” on zpool add. ZFS noticed you’re adding a single device to a redundant pool and is objecting. It’ll let you override with -f. Do not use -f here. This is the guard rail described above and it’s the one time ZFS’s nagging is load-bearing.
Added it as a normal vdev by mistake. Omit the word special and you’ve striped a fast SSD into a raidz pool as a data vdev, which is both catastrophic for redundancy and unremovable. Always zpool add -n first — dry run, prints what it would do.
The special vdev filled up. It spills to disk silently and performance drifts back toward baseline over months. There’s no alarm for this; you have to look. Sizing rule of thumb: metadata alone runs around 0.3% of pool capacity, so a 20TB pool wants roughly 60GB of metadata. With special_small_blocks on, add the actual size of your small-file population, and then double the whole figure, because being wrong upward costs you a few pounds and being wrong downward costs you the feature.
A special device dies and the pool faults. If it was mirrored, replace it exactly like any other member: zpool replace tank <old> <new>, and it resilvers. If it wasn’t mirrored, this is the paragraph I warned you about at the top, and your recovery is your offsite backup. Which you have, because snapshots and replication live in a different category entirely — the point of snapshots are not backups applies to every clever pool topology ever built.
Everything got slower for small writes. Check whether the special vdev is on a bus that’s actually fast. Two NVMe drives sharing a single PCIe lane through a cheap bifurcation adapter, or SATA SSDs behind an expander already saturated by the spinning disks, will underperform the disks they were meant to relieve. iostat -x 1 during a metadata-heavy operation shows the truth: if the special devices are at high utilisation while the disks idle, the bottleneck moved rather than disappeared.
Consumer SSDs wearing out fast. Metadata writes are small, constant and synchronous-ish. Cheap QLC drives with no power-loss protection are a poor match; the endurance maths on a metadata vdev is much less forgiving than on a general-purpose disk. Used enterprise SATA SSDs with PLP are cheap on the second-hand market and are the right tool.
The honest verdict
This is the highest-return-per-pound change I’ve made to a homelab pool, with a strict condition attached: mirror the thing, or don’t do it.
It’s worth it if your pool holds a lot of files and you interact with them — a document store, mail, source, a photo library, anything that gets scanned or indexed or rsynced. The improvement is not subtle. Directory operations stop being a thing you wait for, and a NAS that responds instantly to browsing feels categorically different from one that doesn’t, in a way the throughput numbers never capture. If your pool is on its own box already, as in the case for a dedicated ZFS machine, the drive slots are probably already there.
It’s not worth it if the pool holds forty large files and streams them sequentially. A media pool has almost no metadata pressure. You’d be spending money and adding a failure domain to fix a problem you don’t have — and for bulk media specifically, SnapRAID and mergerfs sidesteps the whole question.
And it’s worth being clear-eyed about the risk. You are adding a component whose failure destroys the pool, in exchange for latency. That’s a real trade, and a mirrored pair of decent SSDs with power-loss protection makes it a good one. A single cheap NVMe drive because you had one spare makes it a bad one, and the badness arrives eighteen months later on a Sunday. If you’d rather not think about metadata layout, a general-purpose grounding in how ZFS handles snapshots, scrubs and dead disks is a better use of the same afternoon.
Related Content
