Comment by NavinF
3 years ago
Infiniband can match NVMe bandwidth and its latency is similar. Newer network cards can also present a NVMe-oF drive as a local NVMe drive
3 years ago
Infiniband can match NVMe bandwidth and its latency is similar. Newer network cards can also present a NVMe-oF drive as a local NVMe drive
Yes, that was my point - 10Gbps is just way too slow, even full Thunderbolt bandwidth can be easily saturated in raid configuration - NVMe are just incredibly fast.
25 Gb Ethernet roughly matches a PCIe Gen 3 NMVe drive’s max throughput 50 Gb will match Gen 4. These are RDMA capable.
It seems 25Gb dual port Mellanox CX-4 cards can be found on eBay for about $50. The cables will be a bit pricey. If not doing back to back, the switch will probably be very pricey.
There are 100Gb/s Intel Onmi Path switches currently on ebay for cheap:
https://www.ebay.com/itm/273064154224
And yep, they do apparently work ok if you're running Linux. :)
https://www.youtube.com/watch?v=dOIXtsjJMYE
Haven't seen info about how much noise they generate though, so not sure if suitable homelab material. :/
I saw that, but didn’t consider it particularly cheap. Also, the power draw of this things is also likely a concern if run continuously.
1 reply →
It can match a single drive.
Assuming jumbo packets are used with RoCE, every 4096 bytes of data will have 70 bytes of protocol overhead [1]. This means that a 25 Gb/s Ethernet link can deliver no more than 3.07 GB/s of throughput.
Each lane of PCIe Gen 3 can deliver 985 MB/s [2], meaning the typical drive that uses 4 lanes would max out at 3.9 GB/s. Surely there is some PCIe/NVMe overhead, but 3.5 GB/s is achievable if the drive is fast enough. There are many examples of Gen 4 drives that deliver over 7 GB/s.
Supposing NVMe-oF is used, the MVMe protocol overhead over Ethernet and PCIe will be similar.
1. https://enterprise-support.nvidia.com/s/article/roce-v2-cons...
2. https://en.wikipedia.org/wiki/PCI_Express