Comment by roenxi
13 hours ago
> ...or chinese models are benchmaxxed...
It is hard to tell what people are trying to say when they use undefined words with no particular meaning. This doesn't seem to change the relative position of the Chinese models much. The Chinese are a month or few behind and the state of intelligence is so volatile that the top 10 will be completely different in 6 months. I'd expect it to still be a mix of US and Chinese models with the US models leading.
Benchmaxing is defined as targeted post-training attempts to optimise for higher benchmarks. This doesn’t change the aggregate intelligence or capability of the model, so it can make a model appear more capable than it is.
I’m pretty sure “benchmaxxing” as defined is a standard post-training procedure for ALL models.
One can argue though some models are post-trained heavily for specific benchmarks and less for other datasets. So they are weaker on out of distribution tasks.
So I assume it depends on what you train.
- Exact problems on benchmark.
- Similar problems to benchmark.
- Broad class of problems or techniques that might apply to benchmark
Each has very different ramifications for how the model performs. The current evidence is that Anthropic and OpenAI is generally operating lower in the list.