Comment by babelfish
15 hours ago
Taking the benchmarks in the article at face value, this is ~on par with Opus 5.5 for ~1/4 the price!
15 hours ago
Taking the benchmarks in the article at face value, this is ~on par with Opus 5.5 for ~1/4 the price!
Which is why I don't take the benchmarks at face value...
So you have decided the model is bad without having used it yet?
If they had a model on par with Opus 5.5 there would be no need to undercut it by that much, that's probably the logic.
1 reply →
No, I have decided I don't trust benchmarks without using the model first.
Taking the benchmarks (as reported by OpenAI) at face value, why wouldn't they show Opus 5.5 on the OS world benchmark, and show no Anthropic models at all on the "factual error rate" benchmark
...no it's not? At max effort, it scores 50% on terminal bench vs. Opus 5.5's 70%.
ALE (top benchmark in post) is 60% for Terra vs 63% for Opus - I guess we'll have to wait for AA Index!
48 for GPT-6.1 Terra Max vs 58 for Opus 5.5 Max on AA Index