Comment by vikramkr
6 hours ago
So using this harness five qwen3.8-27b models roughly matched Fable 5 on a saturated coding benchmark from early 2025. No comparison to astra/6.1 sol/fable 5.1/opus 5.5. No comparison on a benchmark that reflects actual agentic work (and agentic behavior is what modern models optimize for anyway - probably makes qwen look worse than it is?). And also obviously using a very old benchmark like this:
> The updated release of the dataset with problems released between May 2023 and Apr 2025 containing 1055 problems
means essentially guarenteed contamination.
I have no idea if this harness is any good. Agressive orchestration could be interesting given how hard models are RLed on orchestration? But the methods used to evaluate this harness are not informative
I do not think we should trust this harnesses claims.
I don't think we should trust 'small model beats giant frontier model' either... Don't get me wrong I think small models are and could be great, but matching something like fable 5?
Also why not comparing harnessed Qwen to harnessed other LLMs.
I guess one point is you can run Qwen on a local machine.
It is not public knowledge how Fable 5 works - it is all hidden away behind a web service - for what it is worth it might a hybrid architecture with harness, loops, tools etc. combined with a traditional llm like what is done here.