Comment by x187463
6 hours ago
Carbon scoring below Opus 5.5 on coding highlights how magnificent Opus 5.5 is as a coding agent. With just a $20/month sub, it performs a ton of work at a very high quality.
6 hours ago
Carbon scoring below Opus 5.5 on coding highlights how magnificent Opus 5.5 is as a coding agent. With just a $20/month sub, it performs a ton of work at a very high quality.
The way benchmarks work is very very different from how you work with an agent.
The benchmarks often have ambiguous instructions, and there is no interaction with the agent. So a docker image is started, the agent is called in non-interactive mode with a prompt saying to read the instructions and finish the task. The instructions are often somewhat ambiguous and the agent has no way to ask for clarification. If it picks the wrong option out of many seemingly equally likely options that test will be mark as failed.
That's still the point though.
I get that you're saying the model might get a higher score because of luck, but at the same time you want the model to be "lucky".
I wish Argon were as lucky as Opus.