Comment by lukewarm707
13 hours ago
The model here is Qwen/Qwen3-Coder-30B-A3B-Instruct. This is a 30B model from July 2025. It is NON-REASONING only.
Qwen3-Coder-30B-A3B-Instruct has an AA score of 10. The same as o1 mini, or Gemini 2.5 Flash. AA does not really even score accurately below 10. Qwen 3.8 27B, which has fewer overall parameters, scores 34.
Qwen3 Coder 30b has a hallucination rate of 80%. Qwen 27b has a hallucination rate of 30%.
Why would they choose an outdated, non-thinking model when newer models from the same company have fewer parameters, are open source, and just as easy to test?
"show me the man and I'll show you the crime"
There is another gem in here.
The westernized version makes the US defense contractor 22% more likely to be flawed than neutral. Also 35% more likely to leak secrets for the Chinese contractor than the Hong Kong group. Those are the only statistically significant results for the western coder tests.
This report seems to be biased and misleading. Also 100% AI-written on pangram. Feels a bit like something Claude would write.
Well, their tagline is: "Your AI models contain risks you can’t control. Until now."
The "why" seems pretty clear :) Thanks for pointing that out.
If this is meant to be published research, its often because there’s a long lag with doing the research and getting published results back compared to how fast the AI world moves.
That MoE model is fast. 30B A3B gets 140 tokens/second on my M3 Max, and…the A3B part means a reasonable KV cache size and reasonable prefill times. 3.8 27B is a dense model that is mostly unusable, 40 tokens/second using MTP sure, but any cache miss is a few minutes of prefill. You are better off going with a lobotomized 3.8 flash MoE for the same amount of memory.
The report is still bizarre, but that model is still very relevant to local users.
I have no idea why you wouldn't be using Qwen3.6 35B-A3B with that setup. Or did you mistakenly think that was what they're talking about?
Their model is over a year old, like, a few decades in LLM history.
Or even Qwen3-Coder-Next? Still and old model, but newer and more capable than Qwen3-coder.
1 reply →
ah, you are exactly right. Coder and even coder-next is useless these days, you should be using 35B-A3B.
Wow, this is is incredibly important temporal context.
It is almost like there's some agenda behind these farcical "studies"
This is not a comment on gullibility or the like, but yeah, the agenda is _incredibly_ transparent for any non-USian, to the point that following discussions about russian/chinese/$INSERT_COUNTRY_HERE propaganda is always funny.
q27b is honestly super impressive for how big it is. The thought of something that good running on my own gpu is just mind blowing