Comment by CMay

The nature of LLM benchmarking is that they seem to saturate public benchmarks so quick, they are a uniquely efficient case of https://en.wikipedia.org/wiki/Goodhart%27s_law

I'm not asking anyone to take my word, they can believe or not and in practice people should be taking signals from a variety of places and doing their own testing to see how models behave in their own use cases. What I'm measuring and why I'm measuring it may not be the most important metric for your specific use case.

Most other models are simply failing at these tasks. I think the tasks are relevant to overall model capability, but they are not the only metric. You don't give a jellyfish a tool and expect it to produce wonders, so the other capabilities of the model matter.