Comment by ktosobcy
7 hours ago
Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather good (to the point that I use it mostly nowadays)
On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude.
Here are the tok/s I get:
- Gemma-4-26B-A4B (Q4_0) = 214 tok/s
- Gemma4-31B-QAT (Q4_0) = 58 tok/s
- Qwen3.6-35B-A3B (QB_0) = 30 tok/s
- Qwen3.6-27B (QB_0) = 9 tok/s
EDIT: Updated tok/s after updating llama.cpp
You can get easily over 100 tok/s on the Gemma 4 31B QAT if you enable MTP. Same goes for the 27B. I'm getting 880 tok/s of throughout on a single RTX 5090 for batch tasks.
These token numbers look off for a 5090. For comparison, an rtx 5000 Blackwell SFF with just 470gb/s bandwidth gets me 30 tok/s on gemma4-31B-QAT. Almost 60 tok/s with MTP enabled. A 5090 should get you much more than that!
Thanks for flagging, this is on my local rig and it's driving my display too. I'm curious now, will take a closer look. These are the tok/s as reported by LMStudio.
EDIT: Updating llama.ccp gets me 58 tok/s on Gemma 31b
4 replies →
Agree - I got 9.7 tok/s on an M1 Max with unsloth's gemma-4-31B-it-qat-UD-Q4_K_XL.
To the models you say for coding, can you give an example of what you code?
I find them insufficient for my projects (mid sized), but curious what people see working.
I use local models like qwen to do refactoring; the code and concept already exist, they need refining.
I use sota models for concept, implementation plans, and initial deployment.
I find that local models can perform at a very high level if given a very strict environment and the already bounded rules of an existing code base.
aye aye aye.
I see a lot of conflicting info about whether Qwen3.6-35B-A3B or Qwen3.6-27B is more capable. Is it one of those things where "it depends"?
I think sometimes the quantization is the source of the conflicting reports since those are often not included in discussions. Honestly, 27B is too slow for me to use very often, so I just do 35B-A3B and if it is underperforming I'll sometimes switch to 27B, and sometimes it does better, and other times it does not. That's when I reach for Kimi K2.6 or Claude (or just, y'know... use my noggin)
Keep in mind that lm studio is just a GUI with llama.cpp/llama-server under the hood, so if you want to compile the latest llama-server and load your own choice of model, then connect to it with pi, opencode, etc or your own other choices of harness, that's also a popular option.
Things are moving fast enough these days that llama-server needs to be built from source every 4 or 5 days to keep up with model support and various tweaks in published quantized GGUF files.
Additionally there are a few different tweaks/branches of llama.cpp/llama-server that you can grab and compile to take advantage of changes people have made specific to discrete models and/or types of GPUs.
That Gemma release is what opened my eyes. Even downloading the models on my iPhone was shocking. It reminded me when folks would put Wikipedia on a CD or something similar.
Obviously does not compare to a leading model but it’s impressive for something that was running on my phone. I could see thinking token output and it’s directionally interesting thought.
gemma 31b is my goto day-to-day model. It's adequately capable for coding and natural phenomena, and it follows instructions and skills reasonably well. My main negative trait is all the LLM-isms, but I can get past that for the value it brings