Comment by correct_horse
5 hours ago
Mac minis are particularly suited to running AI models because they can have a pretty good quantity of RAM (64GB) assigned to the GPU at a reasonable price compared to Nvidia offerings. Mac minis have unified memory which means it can be split between CPU and GPU in a configurable way. I think apple didn’t price mac minis with AI stuff in mind, so they end up being good value.
Sure but the GPUs are fairly anemic, right? I get that they have more Gpu-addressable memory from the shared pool.
I have a 10900K with 65GB RAN and a 3090 24GB VRAM lying around gathering dust. 24GB isn't as much as a Mac but my cores run a whole lot faster. I may be able to run a 34B 4bit quantized model in that. Granted, the mofo will eat a lot of power.