Comment by ActorNightly
16 hours ago
Man you Apple Fans really live under a rock.
Mini PC with Occulink, EGPU dock, 750w power supply and a refurbished 3090 is 2500. It can run Gemma4:31b at 1000+ tok/sec prefill, ~40 tok/sec generation (which is slow, but a fully connected model is way better for all tasks)
if you want more specialized partial activation models like Qwen, you can get 100+ tok/sec generation.
This beats anything that Apple has by a long shot. You don't need to run large models for usefulness. Large models can get things write the first time, but thats their only advantage. Running small models that are good enough like gemma4:31b fast allows you to do faster iterations.
And if you really want, you can also do multi GPU setups if you want slightly slower model loading with usb4/thunderbolt ports.
TLDR; local ai has already been useful for quite some time if you arent brainwashed into believing that Macs are the best for running local inference.
A $2500 Mac Studio with M5 Max running Qwen3.8-27b or Gemma4:31b in oMLX gets about the same speeds you quoted... and it comes with a warranty and has enough memory not to choke at longer context lengths.
> "Man you Apple Fans really live under a rock... ...refurbished 3090 is 2500"
Before you make assumptions, there are valid reasons for the choice I made:
I paid $1k for a refurbished M1 Max 64GB macbook that can run Qwen 3.8 27B at 24tok/sec (looks like they are about $1.5k on ebay). In my case, Apple was a lot cheaper (used) if you're looking at price-per-token. Yes, it's almost twice as slow as a 3090, but it's also half the price /performance ratio.
Qwen 3.8 27B Q4 with 64k:
3090: ~40tok/sec
M1 Max : ~24tok/sec (MLX)
In fact, the Max can even do 265k context which isn't possible with a single 3090 last I checked (without dropping to Q3-Q2)
>Qwen 3.8 27B Q4 with 64k, 3090: ~40tok/sec
Try more like 70-130 tok/sec. https://github.com/L-Forster/open-jet/tree/master/megakernel.
And also Qwen models famously chase benchmarks. They are good for writing code, but fall behind in other areas. Gemma4:31B is 6gb larger and does a much better job across all areas. https://www.reddit.com/r/LocalLLaMA/comments/1vyzopv/gemma4_...
Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec.
And this is also not touching prefill, which is the biggest thing that you need fast, and Apple sucks in this regard compared to nvidia cards. With gfx cards, you can throw big codebases at the local models and it will happily crunch through them in very little time.
Sure, spending 1.5k on a Macbook enables you to run a local model, and if you really have a good reason for running local (like running heretic/abliterated models) for tasks you don't mind waiting for, it works. The difference is, spend 1k more and let you never have to pay for tokens, and you can run inference all day long without your computer thermal throttling, like your Macbook.
And I don't need to make assumptions when you say idiotic stuff like "We're getting close to local AI being really useful", like you have any idea what you are talking about. On par with being an Apple fan though.
Most people don't run their own LLMs, let alone install GPUs themselves.
I'm one of those Mac users, but I do have a linux tower for remote dev work and LLMs.