Comment by LeifCarrotson
4 hours ago
It's crazy to me that rsync.net is buying mission-critical enterprise drives on Amazon.
I don't buy drives on Amazon for my 9 year old's laptop because of the rampant fraud and counterfeiting, I'm shocked that they're trusted for any business use-cases by anyone moderately savvy. I'm even more shocked that the takeaway is to blame the individual seller, rather than the marketplace that makes it possible.
First two lines of the article:
>At rsync.net we have trusted suppliers with verified supply chains and a long history of providing reliable service.
>However, from time to time, it is expedient to purchase parts from Amazon - something we do with care and suspicion.
That seems like a very reasonable and non-crazy approach to using Amazon.
As I pointed out in another post if you are using ZFS RAID Z2 you can use literal used garbage hard drives safely without risking data loss.
ZFS helped me discover that my motherboard SATA chip can't handle 6 drives; I had to purchase a cheap Chinese PCI Express SATA controller to communicate with my drives reliably and error-free.
Not for special VDEVs, which is the explicit purpose here.
3 replies →
I'd be more worried about a supply chain attack with malicious devices.
2 replies →
This is how we operate at my job. We go through our trusted and reliable vendor, who gets us good pricing but doesn't always control shipping times. If it's urgent, Amazon will be delivered within 48 hrs.
> I'm shocked that they're trusted for any business use-cases by anyone moderately savvy.
I buy drives on Amazon all the time. I check them all. Never had any problems.
The mistake they made was buying not from Amazon, but from "Maestro Technology" listing on Amazon. If you understand that Amazon is a marketplace and you take 10 seconds to read who you're buying from, it's not a problem.
Amazon returns are also extremely easy. I once gambled on a sketchy seller and received a bad product (not computer related). A couple clicks and it was on its way back for a refund.
The problems with inventory commingling are virtually a thing of the past. I went through the process of selling a product on Amazon and understanding their evolved inventory labeling and commingling procedures so I'm not worried. Many of the tech community are anchored to news articles from years ago, though.
If you have a highly trusted vendor who can deliver at great prices and have products in stock that show up at your door when you need them, then use that. For the rest of us, using Amazon to buy common parts isn't really the problem that it's made out to be in HN comments. I think a lot of people here only understand Amazon through the occasional article that makes it to the top of HN and they don't understand what it's really like because they've been too scared to use it for years.
The problem is that you might not be able to tell a used drive from a new drive if the scammer bothered to reset the SMART data.
> if the scammer bothered to reset the SMART data.
You can't universally reset SMART data on an SSD unless you happen to have a model where factory tools are available on the internet or something.
I have never gone through the process of selling on Amazon. For those of us anchored to news articles from decades in the past, is there any public documentation of these procedures?
I buy used enterprise hard drives that have been pushed hard. My current biggest NAS runs six used 14 terabyte enterprise hard drives and three have failed so far within a year. Each time I was able to get a warranty replacement and the replacement was in much better condition than the original ones I had. Zero data loss because of ZFS RAID Z2. I was able to measure the condition of the surface of the platters and other useful metadata using Victoria https://hdd.by/victoria/ included on Hiren's BootCD PE.
Which manfucaturer is the best and worst one?
It almost doesn't matter. There's three choices for spinning drives and all storage devices fail (even SSDs), and you need a plan for that. There's variance in failure rates and warranty periods, but most (new) drives last the warranty period, and used drives is more complex than just manufacturer --- how were they used, stored, and handled before resale probably makes more difference than who made them.
IMHO, it's more important to consider correlated failures rather than worry about getting the best or avoiding the worst drives. Try to avoid running an array that's built from same model, same firmware, same build time, same power on hours, same workload. Every so often, you get things like drive disappears when power on time overflows, or a manufacturing error that makes most drives fail after N weeks of use, having all of your drives in the same part of their lifecycle makes you more likely to experience a catastrophic failure that failure rate analysis wouldn't pick up.
Picking SSDs carefully bears more fruit, there are many more makers and wider variance in performance and reliability as well as characteristics during failure: everyone says SSDs go read only during failure, but my experience has been that lots of SSDs disappear from their interfaces during failure; you might reasonably have less redundancy if you have confidence the SSD will remain readable for recovery if it fails.
Do not buy Kioxia. If your drives dies after a few years, but can come back to life with an firmware update. They will not give it to you unless you have a support contract.
Solidigm have all their firmware available for everyone their website.
I just get whatever is cheapest with a warranty of one or more years. RMA ability is most important when buying these kinds of heavily used decommissioned drives. ZFS RAID Z2 is absolutely key to preventing data loss here.
https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-...
> For those of you tracking the stats closely, you’ll notice that the Seagate ST8000NM000A (8TB) is a frequent flier on this list. The last time it had a failure was in Q3 2024—and it was just a single failure for the whole quarter!
Check the Backblaze quarterly reports.
Marketplaces only work when the participants maintain a reputation. The buyer here is doing his part.
See also: https://news.ycombinator.com/item?id=45896707 (HDD shortage)
> blame the individual seller, rather than the marketplace
I'd have thought the fraud problems from "commingling" were well-enough known by now to avoid wanting to blame any specific Amazon Marketplace vendor, but perhaps not.
What are the reputable sellers these days? Wasn't there a recent German scandal where Seagate was selling used hard drives as new to its wholesalers?
"It's crazy to me that rsync.net is buying mission-critical enterprise drives on Amazon."
We don't.
"At rsync.net we have trusted suppliers with verified supply chains and a long history of providing reliable service."
...
"However, from time to time, it is expedient to purchase parts from Amazon - something we do with care and suspicion."
... and that care and suspicion takes the form of physical and logical inspections and extended part burn-in.
As you can see, this QC process caught these mis-labeled parts.
I mean, if you're a storage business, hopefully you've designed your architecture such that you assume drives will go bad, so you characterize the models of drive to make sure that not all the copies are on one manufacturer, and then you can take liberties finding the cheapest storage on the market. This only comes back to bite you when you didn't account for (because you didn't know) that there was decreased longevity, so your TCO calculation was off and you might not make as much money.
I agree with this.
However, an even more fundamental philosophy behind any work that we do is "defense in depth" which means that even after building the fault-tolerant, anti-fragile system, we also spend time and resources qualifying the inputs ...
... and then spend time and resources monitoring the outputs (error rates, failures, correlation, etc.).
Any one of those pieces is, theoretically, sufficient. Layering the pieces in a defense in depth strategy is what gives us the highly confident posture we enjoy.
from time to time your trusted supplier might be out of stock and you need drivers quickly
even backblaze bought drives in supermarket when there was HDD shortage
Jumping on two add I've now had 2 experiences of buying counterfeit drives on Amazon from different sellers. I've probably ordered only 4 hard drives off of Amazon ever, so that's a 50% counterfeit rate.
Both sellers issued refunds without trouble because I bought them under Prime. One seller seemed genuinely surprised they had a counterfeit in stock.
Both of these counterfeit drives look very, very convincingly authentic, except the serial numbers don't match real ones and don't validate as real ones with the OEM.
The first time, I actually argued with Seagate over it being real, until they pointed out that aside from the serial number not being in their databases, it's not even in the correct format for any of their drives.
If you care about your drives and you're buying on Amazon, only buy under Prime. And when they're delivered, check the serial numbers with the OEM first thing (usually via warranty validation). Don't buy anything not on Prime.