Comment by user43928

3 hours ago

I don't trust any of the benchmarks where Opus 5 surpasses Astra or Fable 5.1.

Maybe Terminal Bench 4.0 and ExploitGym are reasonable.

Terminal Bench 4.0

  GPT 6 Astra             59.6
  Claude Fable 5.1        55.1
  Claude Opus 5           49.0
  MiMo-V2.6-Pro           34.9
  MiMo-V2.6-Flash         28.8
  DeepSeek V4.1 Flash     26.8
  MiMo-V2.5-Pro            1.5

ExploitGym

  GPT 6 Astra             42.4
  Claude Fable 5.1        30.4
  Claude Opus 5           22.1
  MiMo-V2.6-Pro           17.8
  MiMo-V2.6-Flash          6.0
  MiMo-V2.5-Pro            0.1

DeepSWE v1.1

  DeepSeek V4.1 Flash     74.2
  Claude Opus 5           74.0
  GPT 6 Astra             74.0
  MiMo-V2.6-Pro           71.9
  Claude Fable 5          70.0
  MiMo-V2.6-Flash         67.9
  MiMo-V2.5-Pro           19.0

5 comments

user43928

mokre 3 hours ago

Maybe you should not trust any of the benchmarks!

dom96 1 hour ago

Why not? In my own benchmark Opus 5 does in fact come out on top[1]

1 - https://bench.killswitch-lang.org/

user43928 14 minutes ago

Good question, maybe I am underestimating it based on its absolutely horrible writing style.

varispeed 2 hours ago

They match my experience. Astra and Fable I rate below Sonnet. They are incredibly poor. They were excellent for a couple of days after release and then plummeted.

Maybe I am being routed to more quantised versions or less capable models with system prompt to fake Astra or Fable.