Comment by fotcorn
11 hours ago
The memory bandwith on M4 Max is 546 GB/s, M5 Max is 614GB/s, so not a huge jump.
The new tensor cores, sorry, "Neural Accelerator" only really help with prompt preprocessing aka prefill, and not with token generation. Token generation is memory bound.
Hopefully the Ultra version (if it exists) has a bigger jump in memory bandwidth and maximum RAM.
Do any frameworks manage to use the neural engine cores for that?
Most stuff ends up running Metal -> GPU I thought
It's referring to the neural cores(for matrix mul) in the GPU itself, not the NPU.
https://creativestrategies.com/research/m5-apple-silicon-its...
https://github.com/maderix/ANE