Comment by bitmasher9
4 months ago
512GB unified memory is targeting local inference of large models, or local training of non-frontier models.
4 months ago
512GB unified memory is targeting local inference of large models, or local training of non-frontier models.
I doubt you can run a model that requires hundreds of GB of RAM at an acceptable speed (tok/s) on a MacBook.
What would be the bottleneck?
Compute? Inference doesn't only need memory bandwidth. You need to actually do work with the memory you're loading which needs compute power. Which needs more electricity, which needs more cooling, which isn't practical for something as thin as a MBP.
4 replies →
The integrated GPU. Not enough compute onboard to handle prefill for 100gb+ models, and the decode is constrained by memory bandwidth that's lower than most dGPUs that price.
Apple would be in a much stronger spot right now if they didn't pretend like eGPUs were inconceivable black magic that Macs are incompatible with.
2 replies →