Comment by Xeoncross
20 hours ago
We're getting close to local AI being really useful. Before the M1 is dead we might have already seen a huge shift. In my mind, there are two types of macs: those that have the bandwidth to run local models (Max & Ultra) and those that don't (Base & Pro chips).
The generation doesn't mater as much to me as the memory bandwidth. When that day comes, I'd rather have a M1 Max 64GB over a M5 Pro 64GB.
Local AI already is really useful. This has been true since Qwen Flash Next and Qwen 3.8 27B rolled out. The struggle with exactly 64GB of unified RAM is that you end up having to stream the engram table from SSD because you will need almost all of your 64GB to run QFN. A 27B dense model still runs a little slow, even on an M1 Max Mac. However if you have even an older gaming GPU with 12-16GB of RAM and enough system RAM you can run this model (QFN) at a very respectable token rate and intelligence level. It is a Sonnet 5 class model you can run at home. There are new quantization techniques that make these models very good at agentic tasks.
And now if you stretch you can run DeepSeek Flash 4.1.
And if you were willing to roll up your sleeves you could build your own AI lab at home and run this model and models like GLM Flash 5.3 and DeepSeek Flash V4.1 at lower quants. The budget GPUs (like V100) are kind of not budget now, but still <5K to acquire 128GB of fast VRAM that can compete with a 5090 and actually has a lot more interesting options available IMO.
Can share a good way of running Qwen 3.8 27B (or similar) on 16GB VRAM? I've been trying but I end up with either a heavily quantized model or a small context or a slow speed.
Dense models are not well suited to this. Depends on your system, but most inference engines let you put the KV cache on the host. They should let you use "Q8" KV cache format as well which takes half the storage of bf16, the default format. You essentially are stuck at having to fit the entire dense model into VRAM because it must compute across the entire model every token. Without it being in VRAM it is slow. MoE models such as Qwen 35B with 3B active parameters are much friendlier to limited VRAM systems. This is why you see systems with 16GB of VRAM running something like DeepSeek Flash if they have a bit of spare system RAM. Usually not at any good token rate, but tolerable.
Dense models also don't like quantization as much and are less tolerant of it. There are some very interesting smaller models like https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4... Dory that I would check out. You need system RAM or to stream a larger MoE model from disk and it won't be fast. You need to specialize the models some to get quality results if you have very limited system resources.
I've benchmarked quite a few models on my 9070XT, and I basically gave up hope of using models locally: https://claude.ai/artifact/77pHgucqQNsmic8ZEnqdBA
I asked Claude to come up with 7 problems to solve that were not part of any benchmarks. Besides speed, I asked it to put together a scoring that would reflect models that perform well on the 7 problems and in a reasonable time-frame (RWS, the definition is in the artifact). I valued models that produced good answers in a reasonable time better than models that had very high throughput but kept spitting useless stuff. Overall, IIRC most of them still had a >30% chance of producing wrong/non-working replies.
I then tested Strata ( https://github.com/niko1221/strata ) and the results are not even close. You can basically ignore everything in that table and use Strata instead, you can use bigger/less-quantized models and have way better results. Basically any otherwise-decent model degrades noticeably at Q < 5, and KV cache at Q4 also contributes a lot.
Still, a 20$/mo Claude subscription is way cheaper/more efficient if you code 2-3h in the evening than running stuff locally, unless you really have hardware than can hold models that get most things right at the first try. You can iterate way faster and cover a lot more ground in 2-3h using a Claude subscription, or OpenRouter if you prefer to change models. You get faster replies that are 99% correct at the first try, whereas using the local model you'd be spending 30% of your time fixing things that don't work and prompting it to do better.
Nowadays I use the local model to delegate tasks that would burn a lot of tokens, like sifting through big data archives for indexing and such.
3 replies →
The easiest thing to do (takes 5 minutes) is to use LMStudio and then search for the model inside LMStudios UI.
It will show you on the model selection IF a model fits and if you use a "Qwen 3.8-35B-A3B" type of model, it should just work out of the box. A3B stands for Active which means you need a lot less fast ram.
You will run into issues fast though: If you have big tasks for it to do, your pc/mac will get warm very fast and noisy and you will cosume relevant more energy.
And as you mention: yeah small context!
Its still not that practical in my opinion.
1 reply →
If you have enough system RAM (64GB), you can run Qwen 3.8 Flash Next with a GPU as small as 12GB, this repo is pretty amazing:
https://github.com/Niko1221/Strata/
My setup is a Ryzen 3600, 90GB DDR4 allocated to a VM (don’t even really need that much, just had it available, special thanks to the year 2024), and an RX 9060XT 16GB ($500 GPU).
I get around 40-60 tok/s depending on the quantization level I choose.
As you can see, the CPU is nothing special, don’t even need DDR5.
4 replies →
AI harnesses like Claude will set it all up for you, you can just say "I want to run a localllm on this machine" and it will find the best one for your hardware and even help you fine tune it etc
2 replies →
That's only part of the issue - prefill is horrible prior to M5 class chips.
So it depends on what you're doing. Using it for a fun oracle without feeding it too much context per turn? Sure, an older Max chip is great.
As a coding agent doing meaningful work on a sizable project? Compared to a cheap as chips cloud option, it will take an order of magnitude more time to complete.
SSD token caching (like oMLX does) goes a long way for prefill. Typical miss rates are under 5% in long coding sessions, which is mostly the output of a turn's tool use.
Prefill isnt a huge problem with an MoE, but ya, dense models are a non starter before the M5. You are still better off using DeepSeek than a local MoE, but locally running something gives you possibilities if you are on an airplane without internet, or if you want to jump guard rails, or if you want to run a home device server, etc…
Puh i don't know, there might be plenty of new/better chips coming out for AI next year or in 2 years.
Isn't it also rumored that the M7 is reworked for AI?
And Xiaomi announced their AI Cube with 1.2t of bandwidth. Just because Nvidia is protecting its business by giving consumers either compute or bandwidth and Apple being something in between, doesn't mean we can't get both a lot better balanced.
These M7 rumors and that they're skipping the M6 Pro/Max rumors are really making me try to stretch my M1 Pro until next year.
I'm gonna max out most of the specs on the next one I get, would feel insanely wasteful to do that now.
Yeah, the price is so high and things are moving so fast I don’t wanna risk committing to something that becomes obsolete in a year. I wanna see the rumoured M7 first. Hopefully hardware demand doesn’t make waiting a stupid decision…
Waiting is never a stupid decision because you can always save a good deal on apple refurbished. I got a mac on bestbuy refurbished a couple years ago shortly after it was released. It had like 4 cycles on the battery and I saved hundreds.
They've been saying the $LATEST_CHIP is better for AI since at least the M3.
It seems so wasteful to spend premium on the hardware for local ai to be shoved into macbook form though. You'd probably save a lot building a compute node you shove under your desk that you connect to with some more comfortable but cheap thin client, maybe a refurbished low spec macbook pro (inb4 I say air or neo and people say they cannot work on inferior screens).
Then again the people working on stuff involving this much local model compute demands probably make enough where they don't care about the couple grand they'd save building their own node. I guess apple will get the bag from being the nicest prebuilt out there in this case.
Running Gemma 4 26B A4B and Qwen 3.8 27B on a M4 Pro with 24 GB - Qwen 3.8 runs fine but is slow and taxes the machine a bit, Gemma 4 on the other hand, is a better experience.
Srouji is killing it on whatever he has on mind or is asked to do. Everyone is riling up on GPU and Cloud AI costs. 99.99% of what people use AI for today already processes fine on an M1 from 5 years ago.
Apple killed M6 Pro, M6 Ultra and M6 Max to focus on AI chip, so yeah that'd be a more sensible option.
I guess this means I’ll be hanging onto my M1 Pro for another year and seeing what M7 has to offer.
This is the first time I’ve ever kept a computer this long and never really felt like it needed to be upgraded. I suppose this is a bit of a gift and a curse for Apple.
>The generation doesn't mater as much to me as the memory bandwidth
I don't know man?
The generation kind of does matter. I'm not gonna lie, matrix multiplication units speed things up considerably. That hardware confers a material change in capability.
Maybe from M5 up, it doesn't matter? But you're definitely going to notice the difference between M1 and M5 for instance.
I'm running qwen 3.8 flash next with strata (q3 quant) on laptop with 5090 and 128G RAM and OMP harness. At 90t/s speed it is pretty much good enough already.
Will local AI eventually pull the proverbial rug out from under the big providers and the huge sums invested in building out data centers?
Probably not. The ROI for local AI is pretty terrible at best it's like 20 years to see a cost advantage. Most enterprises are going to be fine using hosted models. The two main reasons to use local AI are 1. education and 2. if you cannot use hosted AI for your use case (safety guardrails, payment restrictions, etc.)
Still on my M1 Max 64GB from release date. Love it. I don't do local AI but this laptop runs so cool and fast I have no reason to upgrade. They really knocked it out of the park on the design.
It feels just as fast as day 1 and I've never had any speed issues. Say what you want about Apple and their pricing and ecosystem but I've been using their hardware for over 20 years now and it's always been top notch.
If you buy their support/AppleCare, your support and warranty is excellent too. They don't skimp. I've seen them go so far past the extra mile it's unreal.
This is also my experience, but when I saw how few tokens/second with a large model it gets locally I was bummed. I’m wondering if I’ll be obligated to upgrade in the near future
Interesting. I have the same machine w/ same specs. I find it sluggish. I think it depends on what you're doing.
You might want to do a full restore or something, my M1 Max isn't sluggish at all.
1 reply →
What are you doing with yours? I recently started using Android/iOS emulators locally and combined with anything else my M1 Pro 32 GB feels unusable.
Definitely grabbing 128 GB + for the next machine.
1 reply →
> I'd rather have a M1 Max 64GB over a M5 Pro 64GB.
I am one of those: bought the M1 Max with 64gb thinking in running VMs, afterward bought a lighter Air with less memory and ended up using the M1 Max as a kind of luxury for LLMs and multicore computing.
Considering the insane memory price surge it's going to take years (or an eternity) before most people will consider upspecing their machines to run local over paying $20 per month to have the honor to loan a slice of Altman's hardware.
The only hope we have is the AI bubble bursting, but that will undoubtedly come with its own set of macroeconomic surprises.
The ~$1T/y AI capex isn't remotely sustainable. Memory fab capacity will go up, AI datacenter provisioning will go down, prices will become reasonable.
> Memory fab capacity will go up
The fabs are terrified to overscale in case the market collapses before the builds are complete.
> AI datacenter provisioning will go down
Depends on political opposition at both local and national levels. Hard to predict the future, which feeds into the previous point.
5 replies →
Man you Apple Fans really live under a rock.
Mini PC with Occulink, EGPU dock, 750w power supply and a refurbished 3090 is 2500. It can run Gemma4:31b at 1000+ tok/sec prefill, ~40 tok/sec generation (which is slow, but a fully connected model is way better for all tasks)
if you want more specialized partial activation models like Qwen, you can get 100+ tok/sec generation.
This beats anything that Apple has by a long shot. You don't need to run large models for usefulness. Large models can get things write the first time, but thats their only advantage. Running small models that are good enough like gemma4:31b fast allows you to do faster iterations.
And if you really want, you can also do multi GPU setups if you want slightly slower model loading with usb4/thunderbolt ports.
TLDR; local ai has already been useful for quite some time if you arent brainwashed into believing that Macs are the best for running local inference.
A $2500 Mac Studio with M5 Max running Qwen3.8-27b or Gemma4:31b in oMLX gets about the same speeds you quoted... and it comes with a warranty and has enough memory not to choke at longer context lengths.
> "Man you Apple Fans really live under a rock... ...refurbished 3090 is 2500"
Before you make assumptions, there are valid reasons for the choice I made:
I paid $1k for a refurbished M1 Max 64GB macbook that can run Qwen 3.8 27B at 24tok/sec (looks like they are about $1.5k on ebay). In my case, Apple was a lot cheaper (used) if you're looking at price-per-token. Yes, it's almost twice as slow as a 3090, but it's also half the price /performance ratio.
Qwen 3.8 27B Q4 with 64k:
3090: ~40tok/sec
M1 Max : ~24tok/sec (MLX)
In fact, the Max can even do 265k context which isn't possible with a single 3090 last I checked (without dropping to Q3-Q2)
>Qwen 3.8 27B Q4 with 64k, 3090: ~40tok/sec
Try more like 70-130 tok/sec. https://github.com/L-Forster/open-jet/tree/master/megakernel.
And also Qwen models famously chase benchmarks. They are good for writing code, but fall behind in other areas. Gemma4:31B is 6gb larger and does a much better job across all areas. https://www.reddit.com/r/LocalLLaMA/comments/1vyzopv/gemma4_...
Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec.
And this is also not touching prefill, which is the biggest thing that you need fast, and Apple sucks in this regard compared to nvidia cards. With gfx cards, you can throw big codebases at the local models and it will happily crunch through them in very little time.
Sure, spending 1.5k on a Macbook enables you to run a local model, and if you really have a good reason for running local (like running heretic/abliterated models) for tasks you don't mind waiting for, it works. The difference is, spend 1k more and let you never have to pay for tokens, and you can run inference all day long without your computer thermal throttling, like your Macbook.
And I don't need to make assumptions when you say idiotic stuff like "We're getting close to local AI being really useful", like you have any idea what you are talking about. On par with being an Apple fan though.
Most people don't run their own LLMs, let alone install GPUs themselves.
I'm one of those Mac users, but I do have a linux tower for remote dev work and LLMs.
..really useful on a Mac. FTFY