Comment by crote
4 days ago
I think it's a huge caveat, because it makes upgrades a lot less efficient than you'd expect.
For example, home users generally don't want to buy all of their storage up front. They want to add additional disks as the array fills up. Being able to start with a 2-disk raidz1 and later upgrade that to a 3-disk and eventually 4-disk array is amazing. It's a lot less amazing if you end up with a 55% storage efficiency rather than 66% you'd ideally get from a 2-disk to 3-disk upgrade. That's 11% of your total disk capacity wasted, without any benefit whatsoever.
You have a couple options:
1. Delete the snapshots and rewrite the files in place like how people do when they want to rebalance a pool.
2. Use send/receive inside the pool.
Either one will make the data use the new layout. They both carry the caveat that reflinks will not survive the operation, such that if you used reflinks to deduplicate storage, you will find the deduplication effect is gone afterward.
Can you give sample commands on how to achieve both options that you gave?
Well, when you start a raidz with 2 devices you've already done goofed. Start with a mirror or at least 3 devices.
Also, if you don't wait to upgrade until the disks are at 100% utilization (which you should never do! you're creating massive fragmentation upwards of ~85%) efficiency in the real world will be better.
It still seems pretty minor. If you want extreme optimization, feel free to destroy the pool and create it new, or create it with the ideal layout from the beginning.
Old data still works fine, the same guarantees RAID-Z provides still hold. New data will be written with the new data layout.