Comment by cbeach
17 hours ago
My 5090 has a bandwidth of 1,792 GB/s, and that is just about good enough for basic (i.e. 27B models) local LLM inference.
The fastest M6 MacBook Pro (24/32GB config) has a memory bandwidth of 170 GB/s. And the M5 Max is 614 GB/s.
Am I missing some nuance of the M6 MBP here, or am I right to assume it won't set the local LLM world on fire?
Arguably the thing you’re missing is the 5090 is $1999 MSRP and more like $5000+ retail currently, and then you still need to buy the rest of the computer. This on the other hand is a whole system for $2400 (for the 32gb config). Maybe won’t set the “local LLM” world on fire, but likely to do a lot more for it than the 5090s will for the moment.
You're correct, the base (non-Pro, non-Max) chips are still not great for local LLMs. M5 and M6 both improved prefill speed/time-to-first-token over prior gens, but bw is still a bottleneck.
There's rumored to be no M6 Pro or M6 Max coming, i.e. the higher bandwidth chips will skip a gen. M7 is also rumored to have better memory bandwidth across the range.