Comment by pzmarzly
5 hours ago
One of the YouTube comments pointed out the issue is most likely the lack of support for SMB Direct (RDMA) in macOS. Would be nice to repeat the test on a Windows/Linux laptop.
5 hours ago
One of the YouTube comments pointed out the issue is most likely the lack of support for SMB Direct (RDMA) in macOS. Would be nice to repeat the test on a Windows/Linux laptop.
https://developer.apple.com/documentation/technotes/tn3205-l...
MacOS has supported RDMA over thunderbolt since March 19th.
That is just RDMA itself and seems to be aimed directly at distributed inference (no surprise there!). Now they need to update the SMB driver to make use of it.
For the record this is officially called "SMB Direct":
* https://learn.microsoft.com/en-us/windows-server/storage/fil...
* https://docs.kernel.org/filesystems/smb/smbdirect.html
Yeah, raw RDMA is one thing, making sure other protocols can use it is another... hopefully Apple gets there, but right now SMB with RDMA is probably a tiny, tiny niche.
I do wonder what Apple's custom-built servers are running for networking. I wish they still made Mac Pro-level chassis specifically for better IO. Thunderbolt 5 is nice, but far inferior to a PCIe x16 slot (which can hit 400 Gbps networking) nowadays.
1 reply →
Is RDMA something worth looking into for connecting two random Linux computers? I've been extremely underwhelmed by the stability and performance of my NFS shares.
Yes, nfs with rdma will increase your performance significantly.
2 replies →
NFS 4.1 / 4.2?
This is a ConnextX4 on a thunderbolt bridge. This is well trodden territory. Unless some hero of open source has come to bat recently, MacOS doesn't support RDMA with this card or over Ethernet at all for that matter. It's only supported over Thunderbolt.
His iperf3 results show that it's able to go full beans on a single 25g port. A cross platform SMB file copy is kind of a meaningless benchmark in this context.
IMO a pretty cool product would be a network switch providing native thunderbolt interfaces to multiple upstream hosts. That wouldn't magically fix this RDMA issue, though. This is just a separate thought. Seems like the kind of thing Mikrotik should make.
Commented on another top comment but yeah, I've been running a similar setup for about $200 all in. Using the PCIe LX version takes care of much of the heat issue, and an eGPU dock (Razer Core in my case) takes care of the fan. Craigslist and eBay were the sources.
Speeds aren't any better, but at least it was cheap and fiber.
I have a 25GbE setup using Thunderbolt, kinda similar to the author's. Except that my fan is held together by a piece of string, not a 3D-printed enclosure.
I found that SMB on macOS is just bad. If I try to heavily load it (e.g. by doing mass conversions in my Audible library), it can cause stalls and disconnections.
I managed to find a set of parameters (in /etc/nsmbd.conf) that makes it work more-or-less reliably by trial-and-error: dir_cache_off=yes, file_ids_off=yes, notify_off=yes, mc_on=no.
It's still terrible, though. It's so bad that Linux in a Parallels VM works _better_.
It's always been like this, too! SMB or otherwise. Any company/person that has been using macs long enough has been burned by silently corrupted volumes or etc and has a thorough system of multiple on and offsite backups.
It's not just software companies or home enthusiasts either. There are a lot of ie medical practices or accounting/benefits agencies out there who have sworn off Apple for life after having to spend extreme amounts of money on data recovery to hopefully get their years of records back, usually in a do or die situation like an IRS audit which causes them to discover that their basement was flooding and all of the paper records are destroyed.