Comment by Confiks
13 hours ago
Not a great time price-wise to be building a NAS, but I have been doing so for the last two weeks. Inside a Jonsbo N6 case, which is pretty nice with an 8x SATA backplane and drive bays (unlike the earlier Jonsbo variants).
I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-filled). The shucking was easier than I expected too, and the performance seems very comparable. The warranty is a definite downside (European, so no Magnuson-Moss), but I think I can even get them back in their enclosure should they fail during the 2-year warranty period.
I've put some second hand 256 GB M.2 SSDs in there as boot drives. It was a bit of a struggle to get it to work in a way that failure of one of the drives doesn't hold up booting, combined with LUKS, TPM keys and ZFS on root. Learned a lot about systemd-boot which I have never used before, but feels a lot saner to me than grub ever was. So now I have a large script which debootstraps a Debian based NAS into being.
I noticed that there are a lot of ZFS myths and cargo culting. For example TFA mentions ECC RAM, which in some circles is a must-have because ZFS would wreck your pool during a scrub otherwise, which is a myth. It's also very expensive, especially this year. You also don't need much RAM for ZFS, L2ARC doesn't use much RAM at all, to name a few others.
Still doubting about setting `dnodesize=auto` (which is the default), because there are some horror stories about that [1]. And it seems impossible to find a cloud storage provider with reasonable prices that supports `zfs send`. Rsync.net upped their minimum order to 10 TiB recently, which is far too much for my use case.
> Not a great time price-wise to be building a NAS
That under-states the matter. It is a terrible time, price-wise, to build a NAS.
I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
(If there were a magical choice between having AI and significantly more expensive storage, and having no AI and some program to dump that investment money into getting and somehow leveraging significantly more available storage, that is.)
You're telling me.
I had a drive go bad in my 4 year old home-built NAS a few weeks ago, and it cost 2x what it originally did to buy the same capacity drive, and that was going to a grey-market importer on eBay, it was more on "reputable" sites.
> I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
Almost? ALMOST!?!?!
If you handed me a button that would make it like LLMs had never existed I'd be slamming that button so hard Sam Altman's clothes would spin around.
Return memory and storage prices to normal, undo the sloppification of ~everything, remove all these annoying "features" that are so useful they have to force them upon people, and make scammers actually have to put in a slight bit of effort, all at the "cost" of real human developers, artists, writers, etc. getting paid for their work....
If this is a hard decision for you, you are the problem.
Same. Every time I think about this, I come back to the fact that I was excellent at my job before AI, and even enjoyed aspects of it, so I would be fine if tomorrow it all blew up.
Same, I'd smash it like it was offering me a million dollars. I cannot think of a single positive thing that has come out of the AI boom. It has been a net negative for all of society.
Oh come on! I'll take a few years of expensive RAM in order for humanity to get wide access to something near as makes no difference to the Star Trek Ship computer.
Which is 100% what this has felt like to work with, all spring and summer.
So we're in a slop phase, it'll pass. The first few years of youtube gave no hints that we'd get stuff like Veritasium.
Ignore the slop and use the tools to create something you never thought you could do. That's what it's for!
Same here, the usefulness of the slop machines is quite limited compared to the downsides.
Hey everyone, stop progress! Wolrah has decided that November 29, 2022 is as far as things should go. Because everyone knows before that day, there were no scams, every artist lived lavishly from commissions, and software was flawless and without any extraneous features.
4 replies →
I'm doing it anyway and I've found some secondhand deals. Maybe COVID homelabbers are offloading gear?
I built a 24 TB HDD NAS in a 36 bay chassis for around $1000 all-in: mb, ram, rails, chassis, rack, disks, hba, nic
Awful Watts/TB but the plan is to run it and the GPU rig on solar.
I obtained a used Dell PowerEdge and just stuffed a bunch of drives in there. It's old enough that the RAM is not overly expensive (DDR3), so it has 192GB of RAM to effectively cache the data, and then 32TB of raw storage. (Of course I regret not spending the extra money to take it to 384GB now.)
The nice thing about the array being big is that I can just RAID 1 it instead of worrying about tinkering with RAID 5, and leave a drive as a hot spare.
PCIe 2.5G NIC for the uplink, and then it can serve over SMB or iSCSI. The main use case for this thing, incidentally, is it just is a caching proxy that holds Docker images, models off of Huggingface, and so on.
36 bays and only 24T of storage?
Can't you get that in a single disk that costs less than a kilodollar?
36 bay? Why so many?
1 reply →
I would kill (figuratively, anyway) to have no AI whatsoever. No slop machine threatening to replace my job, or turn my job into babysitting its stupidity, and hardware would be reasonably priced? That would be awesome. AI has brought me nothing but downside.
AI has freed most people I know from the tedious job of writing actual emails and birthday greetings - leaving then more time to e.g. wash dishes or clean the floor.
Seriously, if arts and creativity is what sets humans apart from other animals, then AI has almost completely displaced our capacity to even consider doing these activities ourselves. People reach for AI when they should be composing a birthday greeting themselves.
14 replies →
But we now have trillionaires, aren't you happy about that? It means you might be the one some day too /s
1 reply →
[flagged]
I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.
> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
> It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
Well, other filesystems can defragment.
ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).
Not sure why I'm being downvoted for being factually correct (and also, why I can't edit my above comment).
ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there.
I'm repeating this so that people who set up these drives know what is actually going to happen.
3 replies →
For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.
The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.
In the Linux ISO community the math is different because none of the data is essential, however there is a lot of it. Backing it all up even one time to an off-site location would be a 2x expense for a hobby. So 3-2-1 is hard to justify.
Instead the math can be different e.g. it's ok to lose some data but not all. Therefore you might prefer the unraid approach over zfs where losing more than parity doesn't kill the whole pool.
1 reply →
For most people snapshots are enough backup. It is still useful to have an offsite backup, but realistically fire is rare enough that you can risk it, and that is about the only risk most people have (you can have your NAS in a location likely far from where a fire might break out to reduce the risk farther).
3 replies →
The middle ground I go with is to structure the pool using datasets and zvols, and then periodically backup the critical ones using zfs send. For example, I can live without my movie collection, but I cannot afford to lose personal photos.
This isn’t black and white. You might have huge amounts of non-essential data that aren’t worth the cost of off-site backups, but worth the cost of an extra disk of redundancy to lessen the risk. Even when you do have backups, it will reduce the risk of extended downtime (and possibly egress costs) caused by having to restore large amounts of data from backups.
I don't think they are. I've already filled a ZFS pool almost to the brim with no adverse effects.
My experience has been escalating fragmentation past 90%, along with lower performance owing to the fragmentation.
> combined with LUKS, TPM keys
Does it work? Server can reboot and use TPM to unlock rootfs? What about /boot - encrypted and tamper proof? Resists evil maid attack?
Rabbit hole, I know, but so fascinating if you solved it all :)
I run my cheap hosting box with a cleartext boot setup that uses ssh to automatically grab the key for the real root from my home server (or an alternate at my MILs house). Using FreeBSD, but similar concepts.
A previous hoster once gave me someone else's drives without wiping them. I don't want random customers snooping around on my data if a similar mistake happens with my disks.
For home use, I run without disk encryption. If I ever need to do data recovery, it's not going to be possible with encrypted disks and one point of a centralized NAS is to have stable long term storage.
If you are worried about someone breaking in to your house and replacing the bootloader while leaving your drives in place I probably wouldn't use the TPM auto unlock even if in theory secure boot should be able to handle this.
But in reality that will never happen and the only actual attack you need to be worried about is junkies breaking in and flogging the drives on facebook marketplace. For which, this level of security is fine.
> junkies breaking in and flogging the drives
For this you dont need TPM. Just a LUKS key in rootfs /etc. He said TPM :)
I've been thinking about building my own NAS as well. Mind sharing how much did you pay for those hard drives, and what motherboard did you choose?
https://diskprices.com/ is great for this
You have SATA or SAS to pick from. The CPU requirements for a storage server are not high. On a typical ATX board you have motherboard SATA and can put SAS controllers in the spare PCIe slots.
My first "NAS" was two 22TB hard drives in a ZFS pool on my motherboard SATA
There should be more sites like diskprices. So refreshing to see no styling, no cookie banners, just pure information.
ATA (SATA), SCSI (SAS), and NVM (NVMe).
1 reply →
It was €329 per hard drive (through a reputable store), and I chose the ASRock B650M-HDV/M.2 mATX motherboard combined with a Ryzen 5 8500G. Stock CPU cooler, replaced the Jonsbo case fans with Arctic P12 Pro PST LN. I only slightly regret the PSU (MSI MAG A650GL), which could be quieter. Not that it's very noisy, and it's a great PSU otherwise, but I should've chosen one that just shuts the fan down at low power usage.
> 4x the 14 TB WD Elements Desktop.
I recommend staying away from SATA drives (huge consumer rush) and look for SAS drives on eBay, particularly HC520 Helium drives from HGST/WD. Need a SAS3008 PCIe adapter ($20) and a SFF-8643 splitter ($30). No backplane is required. Huge lots of decommissioned drives frequently appear. I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
Yea may be some of the stuff is fear mongering and cargo culting. I was told ECC is necessary for ZFS. When the article was written, it was cheap af (2024) to buy ECC RAM so not much consideration was given to it.
-- (*) I have no idea what to do with it. Anyone has any good idea for using 120TB space? I have about 40TB unused bandwidth in the datacenter, may be host a Debian mirror? Donate storage/bandwidth to Internet Archive? Please contact me, appreciate it.
> I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
That's cheap indeed. Enough headroom for some failing disks too. How is the noise and power usage? I didn't look at SAS drives at all, because my impression was that they're very noisy. I can place my NAS in a closed off room, but it's not too far away and I was afraid SAS drives would be audible through the wall. At the same time, the shucked drive I'm using presents itself as an WD Ultrastar, which comes very close to a SAS drive, and isn't very noisy.
> How is the noise and power usage?
I think they're same as SATA drives, just different interface. AFAIK they have the same physical dimensions and same internals.
1 reply →
I'm also planning a new home server build now and the prices are definitely relatively ass. So far I've spent 820€ on two 22 TB WD Elements HDDs, 375€ on 2x16 GB DDR5 kit, and 520€ on two 2 TB M.2 SSDs (cache). About 1700€ and I still have no server to show for it. Doesn't help that I've been procrastinating on picking the CPU and motherboard.
You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.
Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?
Yes, indeed. ECC RAM is better than non-ECC RAM, also for ZFS.
The myth, popularized by a notorious thread on the TrueNAS forums [1], is specifically that ZFS requires ECC RAM, and will do worse than other filesystems without it, because scrubbing will multiply a single bitflip into a failed pool.
A ZFS core developer says that that isn't the case [2]. Here's some more reasoning [3], also about many other myths.
[1] https://kldload.com/zfs-wiki/myths
Usually if you’re using a NAS you don’t want to lose data, ZFS is not significantly more sensitive than everything else.
But everything is actually quite sensitive.
We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
It’s actually less of an issue these days because DDR5 has (by spec) some in-line ECC; won’t help with multi-bit errors but its an improvement on what came before.
3 replies →
The only hangup with the myth "debunking" is that the point is that the corruption doesn't happen to a per-disk buffer, but to the in-flight data before it's persisted to your stripe.
Which means all copies of the data will be corrupted. This can be anything from an irrecoverable file to complete filesystem corruption.
But generally, yeah, not any more dangerous than any other filesystem, and ECC used to be cheap so it was a no-brainer, you should have backups anyway TBH if we're being honest about storage resiliency.
What has been debunked is the "scrub of death" issue, on a scrub a bad bit flip would cause an error, which would be copied over with good data -- well it was technically good before. It would be statistically difficult to have a fault on a read, then a clean read, then a second bit flip destroying the data.
Updated link to the post from Matt Ahrens here[1], which is one of the ZFS creators[2] not "just" a core developer.
[1]: https://arstechnica.com/civis/threads/ars-walkthrough-using-...
[2]: https://en.wikipedia.org/wiki/ZFS#2004%E2%80%932010:_Develop...
Had it happen to my PBS backups running on ZFS without ECC just the other day. Turned out to be heat related, memory was fine but during long backups heat was causing bit flips.
> You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.
ZFS without ECC is no more risky than any other file system / software RAID without ECC.
As no one owes you an explanation, it would take you five seconds to Google this and discover:
1. It's been disproven, with one of the original ZFS developers chiming in.
2. The original source of the rumor was a forum post that somehow became canon.
> five seconds to Google this and discover
...(after much longer) that it's a rabbit hole with nuance. https://news.ycombinator.com/item?id=18481910
What an incredibly rude comment