Comment by mickeyp

Model benchmarks are useful, to a point, but it is the long tail of things you do with the model that determines if it's good at a wide range of activities. Ant/OAI, to their credit, build their models -- even the small ones -- so they follow instructions and do tool calling well, without the system prompts confusing them. This is especially important for long-horizon tool calling.

So one open weight model might "meet" Opus or whatever on benchmarks, but then fail to follow a simple answer format and also tool call correctly. The models are whipped to within an inch of their lives to strictly adhere to their post training quality gates.