Comment by janalsncm
7 days ago
Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.
7 days ago
Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.
I think this is where leveraging classifier models will become important. The frontier LLM models do "everything", while we've known for a while that to truly scale this we will need to distill models into their individual functions. I don't see this as necessarily a bad thing and hope more is done in this space. Very promising.
Bitter lesson is knocking. Mixture of experts is essentially what you’re describing but free from unnecessary inductive biases.
Mixture of Experts is absolutely not what they're describing. MoE has to be one of the most misleadingly named things ever. It's completely confusing as to what it actually is.
3 replies →
1) Mixture of experts as an LLM architecture is not the same thing. In MoE each “expert” can activate for any given token.
2) Bitter lesson is misunderstood. Specialization and inductive biases still matter. ChatGPT isn’t the best chess player in the world just because it’s seen more math problems or read more Japanese poetry. Stockfish is, because it bakes in useful inductive biases like minimax.
There is value in splitting things. If all I ever do is local app automations, i don’t need model that knows how to code. If all I ever do is coding, i don’t need a model that translates english to slovakian.
4 replies →
I enjoy doing local image generation and this is one thing that the community around that has really optimized.
In some workflows you might have 20 different models doing their specialized tasks. Pose detection, hand/eye/face detailers, classifiers, refiners, up scalers, taggers, etc can all use their own models and that’s not even the including the model(s) used for the actual image generation part.
I’m interested to see the optimization when this concept gets applied to other general ai tasks.
FWIW my tests on my little puzzles suggest that it is not better than the Gemma 4 12B on SQL. It really does seem to get quite tangled up on stuff.
PHP/Wordpress code seems OK (better than the Gemma) but it gets stuck in reasoning loops.
Mind you, I am something of a cynic about the underlying 27B dense Qwen; I think the 35B MoE model is often better and it is just so, so much faster.
Qwen3.6-27B is the best model in that range that I’ve used for agentic coding by far. I think it’s kinda mid at everything else.
Surely not that good at vision. TBH none of these 14-27b models come close to even the cheapest Gemma 2.5 flash.
If these buddies are similarly bad on text, then they definitely don’t get anywhere close to big boys, no matter what the synthetic stats claim upon release.
From my own experiments with local, low VRAM model use vs. what I'm used to from using Claude at work is that being good at "coding" is of no use, if you're worse at "tool calling" as coding in an agentic way requires quite a bit of tool calling.
If you can "hide" different models of 8GB VRAM requirements each that have those specialties and mix and match them for me without having to manage it manually, I'll be impressed. Until then I will keep using my Claude, because "remarkably good _for their size_" models I've tried so far just sucked at trying to use them the way I code at work with Claude.
> slightly worse on knowledge and tool calling
Worse than Gemma at tool calling? Gemma's already bottom tier at that (at least when there's Qwen to compare to), that would just be unable to do tool calling at all.
I think it's extremely quantization and engine specific. I run Gemma4-31B at FP8 on vLLM and it's fantastic, no issues anymore[*].
* I will say that early on there were a LOT of issues with the chat template, across all engines. I dunno who decided using crappy Jinja templates was a good idea, but clearly it has its limitations. In the latest version of vLLM (0.25) they've ditched the Jinja templates for an in-engine parser and I've seen no issues.
I think that depends on how you run it. Llama.cpp has several fixes for the somewhat unusual tool call semantics in Gemma 4. I don't think I have noticed any issues.
To be fair, everything (roughly within an order of magnitude in size) is worse on vision. 12b is a beast for vision tasks, better than its bigger siblings, even.
More to the argument that we need a model of models - one general one that calls specialists in to do what they are good at and handles that like a foreman for you.
This is somewhat akin to the "one-expert-per-query" solution Apple are using in their small foundation models I think?
Is that different from mixture of experts?
Yes. A mixture of experts is a single model that activates different routes though the same weights, with the route possibly changing literally on every token. It's not experts as in a bunch of standalone models that are good at specific high-level tasks.
1 reply →
The things it loses are all the things that google models are historically excellent at, so that's a reasonable performance. I think the take home here is that the 1 bit models are probably better, but it's not a slam dunk given advanced quantization techniques.
The Gemma models are so good at vision. It seems particularly important for phones. Also, they write in a much more pleasant manner than Qwen imo.
I absolutely agree that Gemma 4 writes well out of the box. Free of a lot of the standard American model blog spam writing style but a little more fluid than Qwen.