Comment by dofm
2 days ago
So far what I am seeing in my seemingly simple "Wordpress last-login plugin" test is that in xhigh reasoning mode (the default, seemingly) it overthinks so badly that it writes terrible bushy code with edge cases caused by going down rabbit holes.
In "medium" reasoning mode, you get the classic Qwen wait/actually thinking loops you see in 3.6 that I guess will need to be interrupted in the way others do already with an over-thinking guard proxy.
(In one of my test runs it is now on "OK TRULY FINAL APPROACH" after having got through "FINAL FINAL APPROACH". Can relate)
It gets stuck in a thinking loop regarding the WordPress API and (resolvable) ambiguity in my prompt, that I guess might be resolvable with a custom skill with hints on how to look it up (and maybe with the devdocs MCP).
In Low reasoning effort mode it flies through the task and writes pretty solid code. So maybe it is me overthinking what is needed here...
how do gemma4 or muse perform?
Gemma 4 26B does really well at this specific task (and a general MySQL-related puzzle I test on). I rather like it and now they have fixed tool calling, I would use it. I think maybe it has been trained well with "consumer" programming languages like PHP that are sort of commonplace things people want to do. I think for less commonplace programming languages, maybe it's worse.
Muse Glimmer thinks well and codes well in my tests; it does fine at this. I really like it so far, but my tests are fairly shallow.
One thing I have been struck by — my prompt includes this sentence:
"Please read the following and then ask me any further clarifying questions you need before proceeding with code generation."
Almost all models I've tested interpret this as an instruction to ask questions regardless. Qwen 3.8 27B is the only one that either expresses confidence that it doesn't need to ask clarifying questions, or in higher reasoning effort ultimately asks questions, but offers up defaults I can choose with a simple reply.