Comment by D13Fd
16 hours ago
I’ve been running DeepSeek 4 Flash, Qwen 27B and Qwen 9B on local hardware. They work well for coding and document review tasks. I think Qwen 9B local on a 5090 might be legitimately helpful for small task agents in omp, since it is ridiculously fast. But my motivation is that I have data that I unfortunately can’t share with 3rd parties.
I have been eyeing a 512 GB Mac 5 Ultra to run full DS4 pro locally, which I expect would be pretty amazing as far as quality/recall. The only downside is that the speed is a lot slower than something like 27B on the 5090.
I have a RTX PRO 6000 96GB when the pricing was way better than now i also have a RTX 5090 too.
What I noticed is that (1) the great local models are optimized run inference (diffusion & LLMs) well on 32GB VRAM <= GPU's because that that's what the target has ...
(2) The quality of local models (esp. in diffusion) is increasing faster than the need for more VRAM - additional reason for the value of these FAST GPUs to increase!
(3) RTX PRO 6000 96GB is really great for fine tunes (ai-toolkit) :) but doesn't outperform my RTX 5090 with inference by anything significant on the good local models.
I have never run an AI job on a Mac, i also have doubts about performance and compatibilities - since the reviews almost never compare directly.