Comment by genman
3 years ago
There was a moment with the spinning rust drives where it would have made sense to have storage in a networked device and not locally but now it rarely makes sense unless an incredibly fast interconnect could be used.
Of course this example is still interesting and cool.
For a couple of years I had a Linux NAS box under my desk with like 8 Samsung 850 pros in a big array connected to my desktop over 40GbE. Then NVMe became a common thing and the complexity wasn't worthwhile.
Infiniband can match NVMe bandwidth and its latency is similar. Newer network cards can also present a NVMe-oF drive as a local NVMe drive
Yes, that was my point - 10Gbps is just way too slow, even full Thunderbolt bandwidth can be easily saturated in raid configuration - NVMe are just incredibly fast.
25 Gb Ethernet roughly matches a PCIe Gen 3 NMVe drive’s max throughput 50 Gb will match Gen 4. These are RDMA capable.
It seems 25Gb dual port Mellanox CX-4 cards can be found on eBay for about $50. The cables will be a bit pricey. If not doing back to back, the switch will probably be very pricey.
There are 100Gb/s Intel Onmi Path switches currently on ebay for cheap:
https://www.ebay.com/itm/273064154224
And yep, they do apparently work ok if you're running Linux. :)
https://www.youtube.com/watch?v=dOIXtsjJMYE
Haven't seen info about how much noise they generate though, so not sure if suitable homelab material. :/
2 replies →
It can match a single drive.
1 reply →