Comment by theptip
6 days ago
Are we reading the same chart? They have Sonnet <= high as Pareto dominant on $/perf.
You have to test each task obviously but it is not a bad model on its face.
6 days ago
Are we reading the same chart? They have Sonnet <= high as Pareto dominant on $/perf.
You have to test each task obviously but it is not a bad model on its face.
They have updated it
Ha! So we were not looking at the same chart. That makes more sense.
> Anthropic did post an official explanation, stating the original chart used a "simpler methodology" that "underestimated Sonnet 5's performance." The new chart supposedly uses their "standard methodology."
Oops!
Did Anthropic have Opus 4.8 and Sonnet 5 switched in the Agentic Search chart at first?
No, and the original had everything more expensive. There's a comparison here:
https://www.reddit.com/r/ClaudeAI/comments/1ukgqwr/looks_lik...
The explanation Anthropic gave for the update doesn't address how the x-axis needed to range up to $50 previously and only $10 now. In any case the pass rates are also lower.
Probably the difference between whatever it is people notice when they say models become "nerfed".
1 reply →