Comment by l1k
12 hours ago
This is using Thunderbolt networking as transport, which incurs a bit of overhead.
But starting with the upcoming Linux v7.2, there's a new feature called USB4STREAM to use raw Thunderbolt packets as transport with minimum overhead and a super simple user interface:
https://lore.kernel.org/r/20260511102744.1867485-1-mika.west...
Release of v7.2-rc1 is predicted for Jul 5, that's when this will first be available as a tarball. Until then you have to clone from thunderbolt.git/next:
https://git.kernel.org/pub/scm/linux/kernel/git/westeri/thun...
Or alternatively linux-next:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...
Press coverage:
author here! it's not on top of USB4NET, no (RXE can already do that, it's compared in the benchmarks). it's built with the same tb primitives as the networking stack obviously, just assembled differently to emulate a verbs device instead of a nic. happy to answer any other q!
A bit unclear, maybe I missed it in the article: how much of InfiniBand is there? Is it just the verbs interface? Or are there some (higher) layers of InfiniBand actually carried as bits on the cable? It can't be "bridged" into actual InfiniBand, right?
I actually didn't know there was more to InfiniBand than verbs (at least at this abstraction level, above PHY), so probably the answer is 'not much more'. The device imitates a RoCE V2 device and the higher level abstractions I used on top were GPU-ish libraries like NCCL and JACCL.
Good q about 'bridging into actual InfiniBand', I don't know the answer there either. My naive understanding would be that: since this is host-initiated RDMA (it's still the host cpu invoking into dma buffers, though they may be device-memory mapped), actually it should work fine, at least between two machines? I'm curious enough to try- I have a couple of machines with thunderbolt AND RoCE-capable NICs- the experiment is to see if we can use this across diverse transports simultaneously? I think this is what it does already (since the MacOS FA57 vs linux native are already 'different transports'), but say if you have a better scenario to demonstrate what 'bridging into actual infiniband' would look like!
2 replies →
> This is using Thunderbolt networking as transport,
Are you sure? It doesn't sound like it in some places in the text, e.g.:
>> a kernel driver that sits alongside thunderbolt-net, allocating DMA rings from the controller's NHI port in the same way
but I don't have the domain knowledge to tell…
Yes, the description from TFA does not match the traditional Thunderbolt networking protocol, whose performance may be as low as that of a 10 Gb/s Ethernet interface.
The description from TFA matches what the poster above you said about a new Linux device driver that allows access to the raw Thunderbolt protocol for transferring data between computers. This appears to be an independent implementation of the same principle as in the device driver that will be merged in the mainline Linux.
While the official Linux device driver makes the raw Thunderbolt appear like a file, which can be written and read to transfer data, this implementation emulates an Infiniband interface, which presumably was simpler to use for distributing work over multiple GPUs.
They actually mention that with traditional Thunderbolt networking on the same computers, they had obtained only 9 Gb/s, i.e. more than 5 times slower than what they obtained with raw Thunderbolt.
> traditional Thunderbolt networking protocol ... performance may be as low as that of a 10 Gb/s Ethernet interface.
Ouch. Why so much lower than the physical bandwidth (or what they've achieved here)?
3 replies →