Comment by hedgehog
5 hours ago
How so? I don't know of any other mainstream platform with >1TB/s memory bandwidth. Personally I don't want to deal with macOS but between the memory bandwidth and out of box Thunderbolt networking it's hard to argue that Apple doesn't have a couple significant advantages over the current alternatives.
> I don't know of any other mainstream platform with >1TB/s memory bandwidth.
I mean, RTX 5080 has nearly 1TB/s, 5090 has nearly 2TB/s. Maybe you are talking about CPUs / unified memory platforms? I agree nobody else does it better. But for LLMs, GPUs can still be significantly faster than even the most advanced Apple silicon on the planet. TTFT in particular is super inferior with Apple, for now.
That's probably also the reason Apple had to reluctantly give into Nvidia servers for the initial rollout of Siri AI, though they claim to use trusted computing extensions to reach an acceptable level of privacy. (I do not trust that nearly as much as the Apple Silicon nodes)
It'll be amazing five years or whatever down the line to see Apple reaching those figures. They seem to be heading in that direction lately.