Comment by gaudat
3 years ago
USB4/thunderbolt is a magical protocol. Turns out the fastest way to move data between 2 modern PCs is to connect their thunderbolt ports with a USB-C cable. The connection shows up as a ethernet port on Windows and I can easily saturate a SSD with its 1GB/s+ transfer rate. And it just works (tm). Reminds me of firewire on these 20 yo Macs.
And What happen if you wire both ports together on the same PC?.. Do you get a broadcast (thunder) storm?
Target disk mode over FireWire was magical back in the day. Nothing like turning someone’s laptop into an oversized external hard drive to rescue data or get a borked OS X installation booting again.
FireWire had IP networking stacks in Windows and OS X in 2007. You could daisy chain a bunch of devices together to share a network connection.
Sure but it was very expensive compared to USB and Ethernet so Firewire never caught on with mainstream conumers other than some niche cases like camcorders.
Thunderbolt was also expensive, which is why adoption was limited, but it's becoming more maisntream since Intel and Apple have been pushing it in the last years, adn piggibacking over USB-C makes it an easy sell comapred to requireing a separate connector like firewire.
Still, thunderbolt peripherals are way more expensive than USB ones, so like Firewire before, use is still more in the enthusianst/professional space.
10 replies →
Way before that there was SCSI.
The low cost is price. Asus for example with their ASUS ThunderboltEX 4 allows you to have TB4 via PCIe card.
The neat thing about USB4 was same as PATA and later SATA: widely and relatively cheap available in consumer hardware. SCSI and FireWire were technically superior but were neither cheap nor widely available.
Oh and I don't know about SCSI but FireWire was actually a security risk.
I know thunderbolt at least up through 3 was generally carte blanche DMA, so an obvious security nightmare (strictly speaking no worse than cold boot attacks and the like, but there's a practicality difference between dumping raw DIMMs and just plugging in a thumb drive -- or inter-machine links like TFA, for that matter). Does TB4 bother trying to solve this?
1 reply →
Just a fair warning about these cards, the support is flakey at best. You should research if it works with your motherboard and CPU before going down that route. I did a lot of research on this because I wanted to connect my Gaming-PC to an Apple Studio Display over optical Thunderbolt, but quickly decided against it.
Luckily there are good alternatives. I landed on a solution using a Belkin[0] DisplayPort and USB to Thunderbolt-cable. I just get USB2.0 speeds, but it's enough for my needs. I'm also able to extend it using an active DisplayPort 1.4 extender, for a total of 10 meters cable.
[0] https://www.belkin.com/support-article/?articleNum=316883
Target disk mode to my workstation and saving someone's whole system with Disk Warrior used to be my favourite and most rewarding task. APFS did away with that joy, if a Mac OS systems fails now you have almost no chance of saving the system from itself.
Yeah, we built a similar system a generation ago using FireWire on x86 and Linux.
FireWire at 800Mbps beat Gigabit Ethernet in terms of latency for a rather hard real-time system.
I remember the "good ol days" when I would always opt for a FireWire audio interface for music production and live performance over USB for exactly this reason. I'd get way better latency and stability.
4 replies →
I've been playing with Thunderbolt networking over the past week with mixed results. I can get 16Gbps between a couple of Macs. Between a Mac and a PC running Windows 10 I get similar speeds in one direction, but less than 1Gbps in the other direction.
In terms of scaling this to multiple hosts as the author does, I've read that it is possible to daisy chain, or even use a hub, but it doesn't strike me as the most reliable way to build a network. For an ad hoc connection, though (like null modem cables of yore), it's a great option.
I think reliability is a great metric to evaluate this on. I don't have a lot of experience with USB4/Thunderbolt networking, but as far as ring network principals go, when you have a network with only 3 nodes, a ring topology is also a fully connected topology. This means that connectivity between nodes should never fail due to the failure of a node. That screams reliable to me.
As far as points of failure, there's no additional hub/switch in between the devices, so you have a Thunderbolt controller on each device, two cables, and two ports. If a cable goes bad, so long as there isn't a silent/awkward failure mode, all three nodes can still talk to eachother, at degraded speed. If a switch goes bad, the whole network is down, unless you start talking about redundant switch topologies.
To your point though, there does seem to be plenty of shenanigans with performance, especially between devices with different Thunderbolt controllers, that may make this less ideal. But IMO, that's more of a question of do you want to with a more battle tested topology, or are you okay with a less battle tested, but still highly performant and "simple" (we won't go into how bonkers the USB/Thunderbolt spec is) topology?
You need more like 50 Gbps to saturate a modern Nvme SSD.
10 Gbps doesn't come close.
10 Gbps does not, but 10 GBps, as written above, is 80 Gbps, matches your estimate.
I've tried the same method and it's about 10 Gbps, not 10 GBps.
5 replies →
There was a moment with the spinning rust drives where it would have made sense to have storage in a networked device and not locally but now it rarely makes sense unless an incredibly fast interconnect could be used.
Of course this example is still interesting and cool.
For a couple of years I had a Linux NAS box under my desk with like 8 Samsung 850 pros in a big array connected to my desktop over 40GbE. Then NVMe became a common thing and the complexity wasn't worthwhile.
Infiniband can match NVMe bandwidth and its latency is similar. Newer network cards can also present a NVMe-oF drive as a local NVMe drive
7 replies →
> Turns out the fastest way to move data between 2 modern PCs ...
With the caveat being that's only if you're not up for adding PCIe cards.
If you are ok with adding some PCIe cards, then you can transfer things a lot faster than 1GB/s. :)
Care to elaborate?
Network cards that go a lot faster than 10GbE are common.
They're widely used (with many different types) in IT data centres, home labs, and probably other places too.
Heaps of them are on Ebay. As a random search just now for "Mellanox 25GbE" on US Ebay:
* https://www.ebay.com/itm/134435757546
* https://www.ebay.com/itm/355348422765
(there are hundreds of individual results)
Searching for "Mellanox 50GbE":
* https://www.ebay.com/itm/225021493021
* https://www.ebay.com/itm/233915360659
(less results)
There are older generation ones too, doing 40GbE:
* https://www.ebay.com/itm/305046322527
* https://www.ebay.com/itm/166350081025
(hundreds of results again)
With those older generation cards, some care is needed depending upon the OS being run. If you're using Linux you should be fine.
If you're running some other OS though (eg ESXi) then they might have dropped out of the "supported list" for the OS and not have their drivers included.
3 replies →
I assume they're referring to getting a couple of NICs
> The connection shows up as a ethernet port on Windows
Do you know if this is the case for all thunderbolt generations (speed differences aside)? Does it apply to thunderbolt using mini DisplayPort too or only over USB PHY?
It was possible on Thunderbolt 2 , on the Mac side OS X Mavericks enabled it, this Intel whitepaper from 2014 talks about the Windows side. https://www.thunderbolttechnology.net/sites/default/files/Th...
https://www.gigabyte.com/Press/News/1140 this doesn't mention it and the Intel whitepaper specifically requires TB2 so I would guess TB2 was it.
Pity Thunderbolt 2 is basically non-existent nowadays. I have a few Macbook Pro 13 (2015) and I'd love to be able to use the thunderbolt 2 ports, but peripherals were too expensive and the standard short-lived. Try finding a thunderbolt 2 dock anywhere. Filter out all the false-positives (USB-C docks) and the are maybe 10 total on ebay, and they're stupidly expensive for 6+ year old used devices, most without cables or power supplies. Such a pity because they can really extend the useful life of those laptops.
7 replies →
My TB3 mesh network shows interfaces as thunderbolt0 etc. this is on Linux using thunderbolt _net from the kernel. Latency is worse than regular twisted pair Ethernet.
Dang, that wouldn’t play nice for MPI then.
1 reply →
I think it works for older thunderbolt too. Been years since I tested it though.
> And What happen if you wire both ports together on the same PC?
A literal loopback interface? Two of them, most likely.
> The connection shows up as a ethernet port
OK but what then? I've had ethernet ports on my computers since I can remember, and that hasn't magically allowed me to transfer data back and forth just by plugging a patch cable into both machines. What software is at work here?
I’m guessing if they’re using Windows it’s boring old SMB?
Depending on what you want to do on what O/S, batteries may or may not be included. I use Moonlight on a Mac to control a Windows 10 PC running Sunshine--lower latency than Remote Desktop (which I don't have anyway in the Home edition), and nicer looking than Parsec.
When you connect a patch cable directly (no crossover cable needed in the 21st century), you'll likely find that each system has a self-assigned IP in the 169.254 network.
A friend has reminded me that with Linux, it can be as basic as piping tar between the two machines using nc. (There's probably a friendly frontend to this somewhere that will allow the machines to mount each other's filesystems with fuse, though...)
WTF!
I have two identical machines that I need to do this with... lemme test it and see.
...
ARGs usb-3... so nopes. (How FN lame is that)
Indeed... I just got my first computer with a USB-C port and have been puzzling over what to do with it. Most of the cool tricks seem to require Thunderbolt, which it is not.