Comment by nozzlegear
18 hours ago
Qwen3.6-35B is my daily driver for AI, and what convinced me to cancel my Claude subscription back in April. The Qwen3.6 line is easily the best local model I've tried, and I've tried a lot. I've got it diligently grinding away on my laptop right now, reviewing and fixing some bugs in my F# code.
Qwen-3.6-35B-A3B was our "gateway drug" into switching our organisation to agent/harness-first coding.
Particularly, I had one team member who was extremely sceptical of AIs/LLMs/harnesses and refused to use them. One day he said "Well, I have an RTX 5090 doing nothing... should I try to get something up on it?" and a few minutes later he had 3.6-35B loaded up, running OpenCode.
It continues to be a workhorse to this day, running on both my local Mac for various types of jobs, an AMD R9700 at the office, and said teammember still uses it on his 5090, although in practical terms we do a lot more with DS-V4-Flash-0731 these days.
I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks. Let it run overnight and wake up to millions of output tokens worth of results without data having left my house, all for the price of electricity.
I haven’t found it very useful for code. It can do some code, but I’ve tried a dozen different quants and context lengths and the output is always bad enough that it has to be discarded for anything other than really easy tasks. It has been useful for exploring codebases for search and summary, though.
DS Flash is where local models begin to feel useful for coding, but the quants we run locally are sharply reduced in intelligence from the benchmarks for the full models.
For applications where data cannot leave the local network it’s good to have them. For actual coding work I can’t actually justify the power of electricity and cooling, let alone the expensive hardware, compared to hosted APIs.
But I admit I do enjoy playing with them anyway. I think it’s one of those hobbies where it’s most fun if you never do the math on how much you’re paying for the privilege. If someone has a requirement that data stay local then it’s different, of course.
> I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks.
Do you mind sharing your use cases?
5 replies →
That is exactly what got me past just enough of my cynicism to get started. I am still cynical but now I have meaningful knowledge.
> Qwen-3.6-35B-A3B
The A3B models are super fast but I found the A3B Q4 model ran in circles a lot and ended up taking longer to complete tasks that 27B Q6 because it kept having to redo/rethink/fix something.
I was writing extensive prompts to rein it in and it would still ignore basic directives like "never force push on the repo, ask me instead". I ended up switching back to 27B after about a week of frustration and lost productivity.
We’re using 6bit quants since we have 32GB cards.
Gemma QAT is an honourable mention.
What!? You are skeptical of AI but will go through the manual process of hosting a model that’s less than frontier intelligence (talking about Qwen 3.6)? Anti-AI folks are always odd to me
A local model needs 0 investment and 0 commitment, takes literal minutes to get started (especially if you have someone who is into that stuff showing you the ropes) and if you end up disliking the experience of using AI you can just `rm -fr` it and forget the whole thing existed.
38 replies →
This largely describes me. I'm skeptical of AI in that it's capabilities, while very impressive, are vastly oversold and overblown. Being skeptical of AI is not being "Anti-AI". That's largely the AI data centers are using up all the water and electricity types.
Maybe you're anti-AI because you're really anti-outsourcing your thinking to some remote corporation you don't control?
That's one of my main issues with AI anyways, the thought of having all my data go through some sketchy foreign (to me) entity with questionable motives and under a questionable regime.
Local AI solves for all of those.
2 replies →
35B MoE is certainly a good and fast local model. I find 27B dense to be quite a bit smarter, so I daily drive that. I wish there was a ~100B MoE with maybe 10B active. It would be super smart and fast!
There was a 3.5 122B 10A release -
https://huggingface.co/Qwen/Qwen3.5-122B-A10B
I tried it for a bit, and It was not really worth its size. It got swept up in all the other AI news recently, but laguna s 2.1 I think is the best ~100B moe model right now
25 replies →
A 35 A3B as smart as previous gen 27B would be a sweet point
> 35B MoE is certainly a good and fast local model. I find 27B dense to be quite a bit smarter
Isn't that just the definition of MoE vs dense ?
Full name is 35B-A3B. 3B Is the token generator thats selected out of the 35b available in the model, which is some layered jazz.
So it can be dumber but its quite capablr.
I use 27B in plan mode and 35B MoE in act mode. I noticed that is the best balance for me for consistent tool calls and intelligent planning. Takes some time to switch, but it's worth it for me.
Wonder if it's possible to share a common cache via cachyllama between 27B and 35B-A3B
I've heard 27B is smarter! I tried it some time ago but couldn't get it working with my oMLX. I need to try it again.
Honestly the 27b dense one punches way above its weight in a lot of domains, especially coding in my testing, so I think you will probably be disappointed.
In my case I would say they are comparable but moe models are looping and getting lost a lot more than dense models.
On the other hand having 90t/s with any local model is nice and Pi with loop police extension can prevent looping a lot.
1 reply →
It was for me too but the new deepseek pricing is too good to ignore for now.
I honestly think that with my electricity prices running qwen 36B myself is more expensive than hitting the cache rate at deepseek.
Can you elaborate on DeepSeek (deepseek-v4-flash, i assume?). What does your typical usage pattern look like and what is your weekly/monthly spend?
I gave it a try for a few days (pi + openrouter + deepseek-v4-flash via deepinfra) and ended up paying ~$18 for rather light usage. Yes it's still cheap, yes it's fast, but i feel i would still get a better deal with a Claude subscription plan.
DeepSeek without OpenRouter is wayyyy cheaper
7 replies →
Going directly though DeepSeek's API and hitting all day long with light/medium tasks I'm at $5/month. That's with pretty vanilla ohmypi.
[dead]
I would recommend looking into Ornith1.0 - it's using Qwen3.6 35B-A3B and excels in coding, at least for my coding needs, Python, web-dev, SQL scripting and some C#. Using Pi harness.
I have been using Qwen3.6-35B-A3B as my daily driver as well and its been phenomenal when it comes to coding
How do you use a 72GB model as your daily driver locally?
Apple Silicon. But: there's no need to use the FP16 version. At 8-bit precision the quality loss is almost imperceptible. That cuts the footprint to 36GB. Which is great for a 64GB Mac, because you have room for plenty of context. 6-bit also works nicely at 26GB + context.
You want to use the newer quantization formats like Unsloth's UD quants or oQe, where the weights are selectively quantized using a calibration dataset so that important weights are left at/closer to full precision.
use a quantized version. since it's MoE, what matters is that the 3b parameters that are used for every token fit in gpu vram, the rest can stay in system ram. really great if you don't have unified memory.
At 4 bit it easily fits in 32GB. That's what most people use.
What kind of machine do you have running that? My attempts at local have always resulted in a very hot lap
I host the models on my Mac Studio, an M1 Ultra with 64gb ram (I bought it when it came out, just happens to be good at LLMs). So when I work on my laptop, I have my oh-my-pi setup configured to use the models on my Mac over my local "bonjour" network or whatever Apple calls it. That way I have a nice cool lap, while using models that my M4 MacBook Air with its 16gb ram couldn't possibly run.
Cool yeah. I got a M3 with 32GB ram and it’s a little iffy. I’ve considered getting a MacMini to act as an in-house
Strix Halo for me. If I am running something on my laptop, it's a much smaller usually around 12b model, but those are a bit less functional. I mean I think there is a a ROG FLow Z that has the Strix Halo setup, but that thing was super expensive.
With laptop being ...?
It's just an M4 MacBook Air with 16gb ram – probably incapable of running models itself. I actually run the models on my Mac Studio which is an M1 Ultra with 64gb, and oh-my-pi on my laptop is configured to use the models over the local network.
Is your Qwen3.6 locally run on your laptop? What kind of tokens/s are you getting from your laptop GPU?
Qwen is running on my Mac Studio, an M1 Ultra 64gb. My harness (oh-my-pi) on my laptop is configured to use the models hosted on my local network, since it's just a MacBook Air 16gb and probably incapable of running anything useful itself.
I get about 45-55 tokens per second using Qwen with this setup. I could probably squeeze out more if I messed around with the settings, but I'm mostly using oMLX's defaults for the model.
I have Qwen3.6 35B-A3B on my laptop and it does 60 tokens/s
Could you share the specs of your laptop?
1 reply →
compared to claude - how 'fast' is it in terms of throughput on your laptop?
On my SpacemiT K3 SBC with 32GB RAM (where models run on the eight A100 RISC-V cores with 1024 bit vectors) doing the same task I got 5, 5.8, 6.5 tok/s using gemma-4-26B-A4B-it-QAT-Q4_0.gguf, Qwen3.6-35B-A3B-Q4_K_M.gguf, Qwen3.5-35B-A3B-Q4_K_M.gguf. The corresponding dense models are more in the 2.5-3 tok/s range.
Kind of slow, but using only 14W of electricity so the Wh per task is twice as good as using my i9-13900 laptop with 4060 GPU.
I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.
what hardware do you use or recommend for this? never heard of it until today.
5 replies →
On an 8GB GPU and 32GB laptop: ~5 words/s while running in Qubes via ollama with completely default settings (I don't have an install at the moment that'll tell me tokens/s). Not exactly a highly tuned setup, but it's a ballpark at least :)
Tolerable and usable for some things, though thinking makes it take about a minute to reply in many cases. But getting this kind of thing to run on 8GB of VRAM is the main benefit of the mix-of-experts setup: it can do partial GPU loading and get a ton better throughput than a similarly-sized dense model (like 5-10x, sometimes more).
It's pretty fast, faster than I could type anyway, but not as fast as Claude of course. My oMLX dashboard says I get about 45 tokens per second from the Qwen model I'm running (I host it on my M1 Mac Studio, not on my laptop).
[dead]
What do you use to pair it with web search?
Depends on how you are doing it. LM Studio and tool calling models can use the web, or you could go for something like Perplexica, or if you want to go real crazy, something like Hermes or OpenClaw.
I use searxng.
I'm a big ole noob when it comes to local AI. What are you using for a harness? Or platform to interact with it?
I recommend trying pi.dev as your agent harness for local models. In my experience it has been the sweet spot of functionality (which you can and should extend with plugins) vs performance (OpenCode just swamps local models on my hardware).
you can try ollama, omlx or llama.cpp for instance to download a model and get an inference server running locally. They expose „open ai compatible“ endpoints, so you can configure almost any harness to use them.
On what hardware do you run the model locally, if so?
Not the GP, but I run this model as daily driver too. It runs great on a Macbook Pro 64GB (M3 Max). Token generation speed can be about 100 tokens/sec with multi-token prediction, although it depends on the context. Worst case speed is around 50 tokens/sec.
The weaker point is prompt prefill, which starts at 1,400 tokens/sec but decreases significantly at high contexts. That said, for agentic scenarios, if you're using a harness that doesn't needlessly bust the cache, it doesn't feel slow.
I really hope they release a Qwen 3.8 35B, although the lack of a mention seems ominous.
What are the specs of your laptop and what tokens per second do you get?
It's just a Macbook Air with the base M4 and 16gb ram, but I'm hosting the models on a Mac Studio with M1 Ultra and 64gb ram that I had purchased when it came out. I get about 45-55 tokens per second with this setup. I think I could get more if I spent some time fiddling with the parameters, but I don't really know what I'm doing there so I've just left most of it on oMLX's defaults.
I'm on the verge over here, the new Anthropic models have been a disappointment. I've tried the A3B variant, but had mixed results. What do you use as the coding agent, and have you heavily customized your workflows?
What laptop?
It's just a MacBook Air with an M4, cheap and nothing special. I host Qwen on my Mac Studio, an M1 with 64gb ram. The model uses around 20-25gb ram depending on what it's doing.
I wonder if I could get this running on my 48gb M4 Pro. Haven't been able to load anything beyond 27B