Comment by Casteil
2 days ago
One thing a lot of people don't seem to factor when hyping Qwen is how much models like this tend to 'overthink' with seemingly endless 'second guessing'. 3.8 seems no different from what I've tried thus far.
As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much higher tokens/second, and take a small fraction of the time. I suppose 'YMMV' depending on your use case.
Also, I haven't used it enough yet to see if it's prone to infinite looping, but its predecessors sure were.
IME this is a strong/reliable model smell, you typically see smarter and less benchmaxxed models' thinking traces spending more time exploring the solution space, and benchmaxxed models more time trying to refine/decide on the response contents. It has always been a big problem with qwen.
In general it's kind of a benchmaxxing/distillation (I don't mean that pejoratively here, read on) artifact where test time compute's purpose is to refine/zero-in on a particular input:output pair. Basically they're trying to "remember"/re-derive the answer rather than arrive at it deductively - it gets comical/absurd when you see it expending a huge amount of tokens trying to figure out the answer to some trivial question or response to some input, as if it were a trick question or the choice of wording was of the utmost importance. But it's not a bad thing when the output matters or the task is hard: basically the model has been trained to respond/act like a much smarter model and it's probably better for it to overgeneralize that behavior.
Also, IME Gemma and most other local models that support thinking tend to have the same problem, and it's partly only a "problem" because they give you access to the thinking tokens themselves (which you don't in many cases see when working with frontier american models) and you can actually see how they're getting spent. For the local or bulk (runnin on owned or rented hardware) use case where you are paying in time/watts rather than purely by the token, IMO it's a good idea to just look at the task success rate / time per task and not worry about the raw thinking token count.
Reduce or turn off thinking:
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Given that it apparently defaults to 'xhigh', this is probably the answer.
Granted, it's still much lower tokens/s than you'll get out of many MoE models.
Edit: Even set to medium or low there's still a lot of second guessing, less consistency, lower 'acceptable response' rate, and slower/more token churn vs gemma4:26b-a3b. I think gemma4 is just a better 'general purpose' model.
I haven't tried lowering thinking, however, I actually asked a solid question earlier regarding a real world scenario I encountered and all that excessive thinking made it give me an amazing answer. The thinking actually all made sense, and honestly I found it thought of similar stuff to what I thought when I drew my own conclusion.
I don't usually rely on AI for much (I'm actually kind of anti-AI, although I follow stuff like this enthusiastically because of the rapid advancements, "average joe" access, and openness), however for what I asked? It was spot on.
The subject was a bit personal, so I won't share. I was just curious what AI would say about the situation and it definitely surprised me, especially since Qwen, while usually great on development/coding stuff, has shown weaknesses in other areas.
Definitely a solid release, and this one runs on my 4090 with minimal loss of quality!
In my experience with Qwen3.6 35B-A3B, disabling thinking made the model generate inaccurate replies. Ask it for the recipe of egg salad and it gives you the recipe of an omelette.
Did I miss something, is it possible to have that model be reliable without thinking?
Turning off thinking mode you need a different tokenizer and a whole host of settings. It's more like switching your vehicle from gas to diesel, than turning your headlights from lo to hi.
The official docs recommend different sampler options for thinking off. Have you tried adjusting those?
2 replies →
Thank you, this is exactly what I needed.
“Thinking” is just a guiding methodology to help iterations (between the initial prompt, results, and a mixture of harness back and forth to the LLM) converge on something sane in a massive parameter space.
I like to think of it much like (as a common example most people can relate to) the Newton-Rhapson method for finding roots of a (mathematic) function. Your initial prompt runs, then the ‘harness’ kicks in using whatever methodologies are behind them to iterate on that prompt (back and forth with the model, occasionally with the user to get better guidance) and refine the outputs to hopefully converge back to some sensible output or actions the user was initially looking for.
So you’re hoping for an LLM that sort of ‘zero shots’ or needs minimal iterations from a prompt to give usable results. I find from my anecdata it varies across models and what I’m trying to get it to converge on. I tend to prefer models to not zero shot attempt because they tend to not do great, I want them to get feedback often to let me push them down the route of convergence in spaces I already understand well, meanwhile I like them to explore and give me new paths in spaces I’m not too familiar with.
That’s really what all that “second guessing” is, it’s making sure you’re following a sane path in a massive parameter space of an ambiguously defined problem. Imagine if in Newton’s method you checked the slope and it didn’t decrease from the last iteration and you just say “screw it let’s keep trying that direction.” LLMs and their harnesses tend not to have that base assumption like iteration on decreasing slopes to guide them closer to convergence, it’s a lot messier.
> As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much higher tokens/second, and take a small fraction of the time.
Not my experience at all. On my M1 Max MacBook Pro, Qwen 3.6 35B-A3B provides better responses in similar time (~60 tokens/second, vs. 50 for Gemma 4 26B-A3B). 3.8 27B is slower, being a dense model, but so are dense Gemmas.
This is why the "reduce thinking" approach of BottleCapAI is both quite interesting and very useful in practice:
https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B
Use 3.6 27b as a daily driver for months with charmbracelet crush. Gemma 26b-A3b is not even remotely comparable in terms of coding for me. YMMV depending on how you work, what harness you use, etc I suppose.
really? crush.... it's trash harness compared to pi.
I didn't realise there are people out there unironically using crush
Yeah. I use it to do extensive work on full source bootstrapping, deterministic operating systems, compiler debugging, kernel debugging... all with one tiny go binary without endless NPM deps like pi (which /I/ regard as trash)
Works better than opencode (pi based) or anything else I have tried for my needs, and by far the prettiest and easiest to reason about what is going on.
But I will bite. What does pi do today better than crush for your use cases?
What's wrong with it? I like Crush. It's hard to find good harnesses that don't pull in mounds of Javascript like Pi and OpenCode.
You can add a thinking budget thats not much effort in llamacpp. You can align the cut off message with your agent instructions.
What you describe is a engineering harness problem.
If you, and i mean the royal you, actually read tge thinking traces you can see and figure out where its stuck
This means an effective harness would observe when the model is overthinking and step in with reasonable redirection, like increasing logging.
Llamacpp can set reasoning budget and message per reauest, so it can be dynamic.
Your complaint is "skill issue" based and will be resolved by people who do something ither than vibe code react demos.
gemma4 can't hold a candle to 3.6