Comment by Fr0styMatt88
12 hours ago
I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.
> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
> It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
Well, other filesystems can defragment.
ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).
Not sure why I'm being downvoted for being factually correct (and also, why I can't edit my above comment).
ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there.
I'm repeating this so that people who set up these drives know what is actually going to happen.
> Not sure why I'm being downvoted for being factually correct
Maybe because your statement about SSDs is incorrect. Consumer SSDs have spare blocks too, and wear leveling prevents the scenario you're describing.
2 replies →
For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.
The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.
In the Linux ISO community the math is different because none of the data is essential, however there is a lot of it. Backing it all up even one time to an off-site location would be a 2x expense for a hobby. So 3-2-1 is hard to justify.
Instead the math can be different e.g. it's ok to lose some data but not all. Therefore you might prefer the unraid approach over zfs where losing more than parity doesn't kill the whole pool.
I'm actually in this boat right now. I have 2x26TB drives in a 4 base nas (two empty, plus 3x M2 empty) and I'm trying to decide the best way to set them up. I have around 3TB of backups that I care about as backups (but they don't need to be online, just archive backups at this point), and around 5TB of media that can be replaced.
I don't want to lose 26TB just for a mirror, and I have a spare 8TB USB HDD. I'm torn between unraid and just Debian, and I'm torn between just two separate devices and one RAID 1 partition one RAID 0
For most people snapshots are enough backup. It is still useful to have an offsite backup, but realistically fire is rare enough that you can risk it, and that is about the only risk most people have (you can have your NAS in a location likely far from where a fire might break out to reduce the risk farther).
Fire, malware, accidental deletions, capricious RAID controller (pre-ZFS). And that’s only the stuff that happened to me. Add power surge, theft, correlation in SSD failures (eg power on counter overflow firmware bug), damaging the array while moving, etc.
2 replies →
The middle ground I go with is to structure the pool using datasets and zvols, and then periodically backup the critical ones using zfs send. For example, I can live without my movie collection, but I cannot afford to lose personal photos.
This isn’t black and white. You might have huge amounts of non-essential data that aren’t worth the cost of off-site backups, but worth the cost of an extra disk of redundancy to lessen the risk. Even when you do have backups, it will reduce the risk of extended downtime (and possibly egress costs) caused by having to restore large amounts of data from backups.
I don't think they are. I've already filled a ZFS pool almost to the brim with no adverse effects.
My experience has been escalating fragmentation past 90%, along with lower performance owing to the fragmentation.