Comment by walrus01
5 hours ago
The deepswe and terminalbench4.0 scores don't seem to jive with what many opencode users' experience has been of this model while it was being tested as "space bunny". I saw a lot of anecdotes of people trying it and going back to qwen 3.8-flash-next or deepseek v4 flash because "space bunny" made some rather egregious errors.
> [Space Bunny] was an early version of MiniMax M3.1 Flash.
They mention that space bunny was an earlier checkpoints, so it may have improved (or may it could be benchmaxxed)
I find it interesting that so many models seem to be at almost exactly 74% specifically which makes me suspicious of the benchmark. I want to say 5.6-sol, Astra, deepseek flash 4.1, opus 5 (and 5.5?), Gemini 3.8-flash and a few others all ended up there?
> so many models seem to be at almost exactly 74%
There was talk of deepswe tests being flawed. I have yet to find a better gauge of useful work for my case though, so I still keep referencing it and sampling it. On the lookout for improved benchmarks.
"We found issues in at least 23 out of 113 tasks (over 20.3%), leading us to designate the benchmark as Flawed." - https://epoch.ai/benchmarks/deepswe/review
I do agree that it’s generally matched my impressions of models for general SWE at least until we had Opus 5 and 5.6-Sol beating Fable 5. Which coincidentally is when they hit 74%.
Models like Deepseek 4.1 flash, Muse 1.3, and MiniMax 3.1 flash are certainly capable and very cost effective, but I’m not sure anyone would say they’re better than Fable 5/5.1 in capability