Comment by eqvinox
7 months ago
Used, maybe, but [citation needed].
Built, no, definitely not voluntarily¹, Ethernet is the only non-legacy thing surviving for new installations for anything more than short range (few kilometer) runs. InfiniBand, CPRI and SDI are dying too and getting replaced with various over-Ethernet things, even for low-layer line aggregation there's FlexE these days.
¹ some installations are the exception confirming the rules; but as a telco sinking more money into keeping an old SONET installation alive is totally the choice of last resort. You'll have problems getting hardware too.
Disclaimer: I don't know what military installations do.
Infiniband is alive and well in HPC. 327 out of the top 500 machines use it according to this:
https://www.top500.org/statistics/sublist/
It is a nice interconnect. It is a shame that the industry does not revisit the idea of connecting all components of a computer over infiniband. Nvlink fusion is a spiritual successor in that regard.
Yes, IB hasn't died… yet. But the writing's probably on the wall with Ultra Ethernet; the corporate development of Mellanox (now nVidia) is not a great sign either.
(Also you don't use IB for 1800km WAN links.)
FWIW I actually run IB and think it's a nice interconnect too :)
This seems oddly appropriate:
https://xkcd.com/927/
There is one difference, however. As far as I know, they did not make Ultra Ethernet because the existing Infiniband standard did not cover everyone’s use cases. They made Ultra Ethernet because Intel killed QLogic’s Infiniband business in an attempt to replace an open standard (Infiniband) with a proprietary one (Omni-Path) that they made out of QLogic’s infiniband business’ corpse in an attempt to have a monopoly (which failed spectacularly in true Intel fashion), NVIDIA purchased Mellanox becoming the dominant Infiniband vendor, this move turned out to be advantageous for AI training and everyone else wanted an industry association in which NVIDIA would not be the dominant vendor. The main reason people outside of HPC care about Infiniband level performance is AI training and Nvidia’s dominance is not going anywhere. Now that NVIDIA has joined the UEC, it is unclear to me what the point was. NVIDIA will be dominant in Ultra Ethernet as soon as it ships Ultra Ethernet hardware. Are Nvidia’s competitors going to make a third industry association once they realize that Nvidia is effectively in control of the UEC because nobody can sell anything if it is not compatible with Nvidia’s hardware?
Had they just used Infiniband, which they had spent billions of dollars developing just a few decades prior, they would have been further along in developing competing solutions. Reinventing the wheel with Ultra Ethernet was a huge gift to Nvidia. If they somehow succeed in switching people to Ultra Ethernet, what guarantee do we have that they will not repeat this cycle in a few decades after they have left the technology to become a single vendor solution due to myopic decisions and they decide to reinvent the wheel again? We already have been through this with Infiniband and I do not see much reason that anyone should follow them down this road again.
4 replies →