Comment by rbehrends
5 hours ago
I had a look at the changes and the biggest impact seems to be from adding Terminal-Bench-Science 0.1 and having it contribute 15% to the overall score.
Example problems are:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.
Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.
This is also arguably more of a test of a model's knowledge base than reasoning ability.
But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.
The coding agents subindex, in contrast, looks at least superficially plausible [1].
[1] https://artificialanalysis.ai/agents/coding-agents#coding-ag...
Ironically coding has no relevance to something like 95% of users (I forget the exact OAI statistic), but it is the mainstay LLM benchmark.
True, but coding is where something like 80%-90% of their revenue comes from; few other use cases are as token-hungry, and none of those are nearly as widespread. Many users will of course use the models just as chatbots, but they don't generate much in terms of revenue. And that is ultimately what determines where labs focus their resources.