Comment by chatmasta
19 hours ago
Inevitably the CSPs will make their own hardware, especially as we start to see specialized chips for specific models or generic inference. This is already happening with Google and TPUs.
It’s easier for the CSPs to move into hardware than it is for Nvidia to move into cloud hosting.
Although as a middle ground I’ve been quite happy with Nvidia Brev for on-demand GPU instances from a select marketplace of CSP offerings. It’s a well kept secret IMO — great product (from an acquisition iirc).
CSPs making own hardware still needs hardware companies, they reduce the Nvidia tax but still need the likes of TSMC, Broadcom, micron/sk hynix, Marvell, the truth semi-companies. CSPs will not have the patents, IPs and talent to replace any of them.
Also, not sure how well CSPs inference stack is compared with vllm + nvidia. A lot of open weight models uses MoE, making the inference stack more complex.
True, though they could always buy one. I’m surprised this hasn’t happened yet, maybe due to anticompetitive risk? Google bought Motorola long ago which seemed to work well for their mobile device offerings at least.