Comment by ranger_danger
6 days ago
A 10GB 3080 still beats even an M2 Ultra with 192GB... memory bandwidth is not the only factor.
https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
6 days ago
A 10GB 3080 still beats even an M2 Ultra with 192GB... memory bandwidth is not the only factor.
https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
If the model is small enough to fit in to 10GB of VRAM the GPU can win.
But the bigger models are more useful, so that’s what people fixate on.