Comment by toyg
3 years ago
> I don’t understand why it can only hit 11Gbps [...] other people building similar networks were able to hit 20Gbps. [...] So it could simply be that the machine only supports up to this speed [...] it might compete with [Intel's] network controllers, so they capped the speed
I'm bad at low-level networking, but could it be a routing issue? Effectively every machine is also a router, so there might be some wasting going on.
The machines are not configured as "routers" in that example. It is just a point-to-point layer 2 network, where each network link serves exactly one possible destination. As simple as it can be.
It's more likely to be offloads and processing time - none of these have any of the offloads a regular network card does. Even if the CPU can handle it, it increases latency, and with a single stream like this, that affects throughput badly.
At 10gigabit/second, with 1518 byte packets, it's 823451.9104 packets per second.
So in a single stream, you have to process each packet within 1.21microseconds to keep up
At 20gbps, you have 600nanoseconds per packet.
There are also almost certainly timing/synchronization issues between different stacks like this. It's horribly inefficient.
Network cards achieve >10gbps in part by offloading a lot of stuff.
Even if the CPU can handle the load, just going through different stacks like they are may add enough latency to throw single stream throughput off.
The posited reason of "not compete with network cards" is beyond stupid. It can't because you can only do a few meters this way, max.
That's not interesting at all. 25gbps network cards are cheap and for a few meters, a $10-15 25gbps DAC will suffice
For more than that, 25gbps transceivers are 25 bucks a pop for 25gbps-SR, which will do 100meters.
With none of the problems of trying to use longer thunderbolt cables, too.
Intel's 25gbps SKUs are not where they make their money anyway.
It's measuring single TCP connection performance which is already difficult to optimize. With jumbo frames and tuned buffer sizes I'd expect it to get higher, but it will likely be serialized to a single core's worth of CPU. Using multiple connections should give a better representation of available link bandwidth.
IIRC the specification of Thunderbolt from Intel (which was inherited by USB 4) limits the Ethernet emulation mode to 10 Gb/s.
Why did they set a limit so low is not known, but the supposition made by another poster that this is a market segmentation feature may be right, because such policies have always been typical for Intel.
This post is getting almost 12 Gbps and someone mentioned getting 16 Gbps on Mac so I don't think there's any intentional limit. Thunderbolt is really 22 Gbps anyway and networking over Thunderbolt is just inefficient.