Comment by seanmcdirmid
2 days ago
I'm struggling to figure out what to use this for. From the intelligence benchmarks in OMLX. If only they would release another MoE model.
Intelligence Benchmark Comparison
--- Detail ---
Model: scottlowry--Qwen3.8-27B-oQ4e-mtp
Benchmark Accuracy Correct Total Time(s) Think
--------------------------------------------------------------
GSM8K 93.3% 28 30 282 No
MATHQA 46.7% 14 30 26.3 No
HUMANEVAL 96.7% 29 30 156.5 No
MBPP 83.3% 25 30 71.5 No
LIVECODEBENCH 43.3% 13 30 1040.4 No
Model: stamsam--Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-MLX-oQ4-MTP
Benchmark Accuracy Correct Total Time(s) Think
--------------------------------------------------------------
GSM8K 96.7% 29 30 51.9 No
MATHQA 60.0% 18 30 9.1 No
HUMANEVAL 83.3% 25 30 82.9 No
MBPP 80.0% 24 30 29.6 No
LIVECODEBENCH 36.7% 11 30 283.7 No
Try manually asking both more discrete esoteric knowledge questions. Or use benchmarks which are less coding focused. The 3.6-35B-A3B with post-training may do well in coding type benchmarks and math but the density of its knowledge falls off in my experience (vs 3.6 27B dense Q8-K-XL unsloth GGUF) when you need to use it for less commonly used domains of knowledge.
My use cases try to avoid accessing world knowledge in the model (I give it access to web search for some adhoc RAG), and ya, I'm just focused on coding so that's the only place I'm looking at right now.
I think you may find that the dense 27B also does better if challenged with more rare coding tasks, less common or weird languages or things that aren't well represented in the active 3B parameters of the MoE model (eg: NOT css, javascript, python, c++, etc).
1 reply →