Comment by Gareth321
13 hours ago
Benchmaxing is defined as targeted post-training attempts to optimise for higher benchmarks. This doesn’t change the aggregate intelligence or capability of the model, so it can make a model appear more capable than it is.
I’m pretty sure “benchmaxxing” as defined is a standard post-training procedure for ALL models.
One can argue though some models are post-trained heavily for specific benchmarks and less for other datasets. So they are weaker on out of distribution tasks.
So I assume it depends on what you train.
- Exact problems on benchmark.
- Similar problems to benchmark.
- Broad class of problems or techniques that might apply to benchmark
Each has very different ramifications for how the model performs. The current evidence is that Anthropic and OpenAI is generally operating lower in the list.