The memory bandwidth limitation is baked into the GB10, and every vendor is going to be very similar there.
I'm really curious to see how things shift when the M5 Ultra with "tensor" matmul functionality in the GPU cores rolls out. This should be a multiples speed up of that platform.
My guess is M5 Ultra will be like DGX Spark for token prefill and M3 Ultra for token generation, i.e. the best of both worlds, at FP4. Right now you can combine Spark with M3U, the former streaming the compute, lowering TTFT, the latter doing the token generation part; with M5U that should no longer be necessary. However given RAM prices situation I am wondering if M5U will ever get close to the price/performance of Spark + M3U we have right now.
The M3 Ultra was oddly delayed, though rumours are that the M5 Ultra should arrive much quicker. Most are estimating March-ish. We'll see. I think Apple has a much higher motivation to get the M5 higher end variants out given the enormous benefits the new matmul functionality offers.
The memory bandwidth limitation is baked into the GB10, and every vendor is going to be very similar there.
I'm really curious to see how things shift when the M5 Ultra with "tensor" matmul functionality in the GPU cores rolls out. This should be a multiples speed up of that platform.
My guess is M5 Ultra will be like DGX Spark for token prefill and M3 Ultra for token generation, i.e. the best of both worlds, at FP4. Right now you can combine Spark with M3U, the former streaming the compute, lowering TTFT, the latter doing the token generation part; with M5U that should no longer be necessary. However given RAM prices situation I am wondering if M5U will ever get close to the price/performance of Spark + M3U we have right now.
> you can combine Spark with M3U, the former streaming the compute, lowering TTFT, the latter doing the token generation part
Are you doing this with vLLM, or some other model-running library/setup?
1 reply →
The M3 ultra was released about 18 months after the original M3, so you could be waiting a while for the M5 Ultra.
The M3 Ultra was oddly delayed, though rumours are that the M5 Ultra should arrive much quicker. Most are estimating March-ish. We'll see. I think Apple has a much higher motivation to get the M5 higher end variants out given the enormous benefits the new matmul functionality offers.
At least for transformers, it can be kind of fixed with MOE + NVFP4 for small working set despite large resident size.