Comment by jnwatson
13 hours ago
M5 Ultra has 4-5x the memory bandwidth of both. 1.2 TB/s memory bandwidth opens up good performance on relatively large models.
13 hours ago
M5 Ultra has 4-5x the memory bandwidth of both. 1.2 TB/s memory bandwidth opens up good performance on relatively large models.
you can get 4xGB10 for <20K so that gets you about the same tg and pp will be probably better. Power consumption though will be something like 200W idle so that's a bummer. And you get VLLM and SGLANG unlike them mac where one has to use oMLX (nice but not the same concurrency or cache performance) I have 128GB M5 Max laptop and sill prefer to run things on other boxes in basement because its no fun to have the primary device being overloaded.