Comment by toshinoriyagi

I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks. Let it run overnight and wake up to millions of output tokens worth of results without data having left my house, all for the price of electricity.

I haven’t found it very useful for code. It can do some code, but I’ve tried a dozen different quants and context lengths and the output is always bad enough that it has to be discarded for anything other than really easy tasks. It has been useful for exploring codebases for search and summary, though.

DS Flash is where local models begin to feel useful for coding, but the quants we run locally are sharply reduced in intelligence from the benchmarks for the full models.

For applications where data cannot leave the local network it’s good to have them. For actual coding work I can’t actually justify the power of electricity and cooling, let alone the expensive hardware, compared to hosted APIs.

But I admit I do enjoy playing with them anyway. I think it’s one of those hobbies where it’s most fun if you never do the math on how much you’re paying for the privilege. If someone has a requirement that data stay local then it’s different, of course.

What kind of hardware do you run that massive beast on (DS-V4-Flash-0731)?

That is exactly what got me past just enough of my cynicism to get started. I am still cynical but now I have meaningful knowledge.

> Qwen-3.6-35B-A3B

The A3B models are super fast but I found the A3B Q4 model ran in circles a lot and ended up taking longer to complete tasks that 27B Q6 because it kept having to redo/rethink/fix something.

I was writing extensive prompts to rein it in and it would still ignore basic directives like "never force push on the repo, ask me instead". I ended up switching back to 27B after about a week of frustration and lost productivity.

What!? You are skeptical of AI but will go through the manual process of hosting a model that’s less than frontier intelligence (talking about Qwen 3.6)? Anti-AI folks are always odd to me

There was a 3.5 122B 10A release -

https://huggingface.co/Qwen/Qwen3.5-122B-A10B

A 35 A3B as smart as previous gen 27B would be a sweet point

> 35B MoE is certainly a good and fast local model. I find 27B dense to be quite a bit smarter

Isn't that just the definition of MoE vs dense ?

I use 27B in plan mode and 35B MoE in act mode. I noticed that is the best balance for me for consistent tool calls and intelligent planning. Takes some time to switch, but it's worth it for me.

I've heard 27B is smarter! I tried it some time ago but couldn't get it working with my oMLX. I need to try it again.

Can you elaborate on DeepSeek (deepseek-v4-flash, i assume?). What does your typical usage pattern look like and what is your weekly/monthly spend?

I gave it a try for a few days (pi + openrouter + deepseek-v4-flash via deepinfra) and ended up paying ~$18 for rather light usage. Yes it's still cheap, yes it's fast, but i feel i would still get a better deal with a Claude subscription plan.

How do you use a 72GB model as your daily driver locally?

I host the models on my Mac Studio, an M1 Ultra with 64gb ram (I bought it when it came out, just happens to be good at LLMs). So when I work on my laptop, I have my oh-my-pi setup configured to use the models on my Mac over my local "bonjour" network or whatever Apple calls it. That way I have a nice cool lap, while using models that my M4 MacBook Air with its 16gb ram couldn't possibly run.

Strix Halo for me. If I am running something on my laptop, it's a much smaller usually around 12b model, but those are a bit less functional. I mean I think there is a a ROG FLow Z that has the Strix Halo setup, but that thing was super expensive.

It's just an M4 MacBook Air with 16gb ram – probably incapable of running models itself. I actually run the models on my Mac Studio which is an M1 Ultra with 64gb, and oh-my-pi on my laptop is configured to use the models over the local network.

Qwen is running on my Mac Studio, an M1 Ultra 64gb. My harness (oh-my-pi) on my laptop is configured to use the models hosted on my local network, since it's just a MacBook Air 16gb and probably incapable of running anything useful itself.

I get about 45-55 tokens per second using Qwen with this setup. I could probably squeeze out more if I messed around with the settings, but I'm mostly using oMLX's defaults for the model.

I have Qwen3.6 35B-A3B on my laptop and it does 60 tokens/s

On my SpacemiT K3 SBC with 32GB RAM (where models run on the eight A100 RISC-V cores with 1024 bit vectors) doing the same task I got 5, 5.8, 6.5 tok/s using gemma-4-26B-A4B-it-QAT-Q4_0.gguf, Qwen3.6-35B-A3B-Q4_K_M.gguf, Qwen3.5-35B-A3B-Q4_K_M.gguf. The corresponding dense models are more in the 2.5-3 tok/s range.

Kind of slow, but using only 14W of electricity so the Wh per task is twice as good as using my i9-13900 laptop with 4060 GPU.

I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.

On an 8GB GPU and 32GB laptop: ~5 words/s while running in Qubes via ollama with completely default settings (I don't have an install at the moment that'll tell me tokens/s). Not exactly a highly tuned setup, but it's a ballpark at least :)

Tolerable and usable for some things, though thinking makes it take about a minute to reply in many cases. But getting this kind of thing to run on 8GB of VRAM is the main benefit of the mix-of-experts setup: it can do partial GPU loading and get a ton better throughput than a similarly-sized dense model (like 5-10x, sometimes more).

It's pretty fast, faster than I could type anyway, but not as fast as Claude of course. My oMLX dashboard says I get about 45 tokens per second from the Qwen model I'm running (I host it on my M1 Mac Studio, not on my laptop).

[dead]

I recommend trying pi.dev as your agent harness for local models. In my experience it has been the sweet spot of functionality (which you can and should extend with plugins) vs performance (OpenCode just swamps local models on my hardware).

you can try ollama, omlx or llama.cpp for instance to download a model and get an inference server running locally. They expose „open ai compatible“ endpoints, so you can configure almost any harness to use them.

Depends on how you are doing it. LM Studio and tool calling models can use the web, or you could go for something like Perplexica, or if you want to go real crazy, something like Hermes or OpenClaw.

I use searxng.

It's just a Macbook Air with the base M4 and 16gb ram, but I'm hosting the models on a Mac Studio with M1 Ultra and 64gb ram that I had purchased when it came out. I get about 45-55 tokens per second with this setup. I think I could get more if I spent some time fiddling with the parameters, but I don't really know what I'm doing there so I've just left most of it on oMLX's defaults.

Not the GP, but I run this model as daily driver too. It runs great on a Macbook Pro 64GB (M3 Max). Token generation speed can be about 100 tokens/sec with multi-token prediction, although it depends on the context. Worst case speed is around 50 tokens/sec.

The weaker point is prompt prefill, which starts at 1,400 tokens/sec but decreases significantly at high contexts. That said, for agentic scenarios, if you're using a harness that doesn't needlessly bust the cache, it doesn't feel slow.

I really hope they release a Qwen 3.8 35B, although the lack of a mention seems ominous.

It's just a MacBook Air with an M4, cheap and nothing special. I host Qwen on my Mac Studio, an M1 with 64gb ram. The model uses around 20-25gb ram depending on what it's doing.

You can do whatever you want with the model within your own organization. If you use it commercially—either as a model-as-a-service business or in a very large-scale product—you should check the additional license terms, which go beyond MIT. My interpretation is that Moonshot cares about the exact inference behavior and accurate representation of their model or derivatives, and perhaps also about capturing some additional value despite their own GPU limitations, so the extra license terms focus on those large-scale commercial deployments.

Kimi’s policy is extreme reasonable.

  > In other recent news OpenAI also greatly cut their model prices, 20% for 5.6 Terra and 80% for 5.6 Luna, to stay competitive.

I’ve seen comments on HN saying how bad this is for the Chinese model developers since the cheaper option like Deepseek Flash are not longer as price competitive to justify the hassle/risk/lack of multimodal… but isn’t this a gigantic red flag for OpenAI/Anthropic at their current valuations?

Sure, it’s just the lowest end for now, and the enterprise money is at the top of the market. And there’s protectionism/enterprise lock-in/etc that complicate things somewhat.

But still, if the US AI labs ever tap the training brakes for a millisecond, the “inference is still a money maker” argument seems to evaporate when they’ll immediately have to fight a race to the bottom until margins are virtually nothing.

Or if the benchmaxing “line goes up” FOMO mindset starts to lose its luster and companies find their individual niches for productive use of AI and stop bothering with all the latest and greatest churn for top dollar.

Which might be even worse if it means the training arms race is still ongoing but neither Anthropic or OpenAI want to be the first to “lose”. While the marginal value of each new model training run keeps decreasing and enterprises signal they’re more concerned with cost reductions than solving ARC-AGI-7 puzzles.

That's exactly how i fried a laptop. The condensation killed it.

I use Modal for fine tuning and unsloth mostly.

Easy to do with Unsloth Studio.

You absolutely need to let models access the Internet if you want consistently good results. Pretty much any non-trivial task requires the model to do things like look up APIs, code examples, or existing discussions of a given topic.

It's the natural outcome of next-quarter short-termism. The board and C-suite will be fine (monetarily).