Comment by Aurornis
5 hours ago
> to create a perceived improvement when in reality there isn’t really one?
This wouldn't explain progress on benchmarks (including closed sets), or the fact that newer models are providing solutions to major math problems that older models cannot.
Overfitting to benchmarks. And puff, you have the exact same effect.
This is a good point I hadn’t considered, thank you.
Is there any training variable here? For example, can a model released in October perform better on the same benchmarks vs its predecessor released in July just by virtue of training on newer data that was made available on those 3 months?
Sorry if it’s a dumb question, I don’t really know much about the topic.
Far more likely it's about reducing costs.
but there is a gap between benchmarks and user feel.
Opus 5 came out with better benchmark results than Fable, but it really did not feel better to use at all.
Also, in a world where there are several models competing with each other for public perception of which is best, that seems like an extremely bad move.
* Release new model that scores an arbitrary 100 on a benchmark
* Get everyone to talk about you as the first model to ever score 100 on the 100benchmark.
* Tune it down over time so that you end up only scoring 75 on the benchmark and people get used to it, gaslight them into thinking it never changed or that it's just a harness problem, they can't run the old version locally anyways to verify. This also cuts your costs in half. Your gross margin on API calls goes from 70% to 150%.
* Release new model that scores 120 on the benchmark and advertise it as 50% better than the current model, while it's only in practice a minor increment. Everyone praises it as the second coming of Jesus Christ.
* Get everyone to talk about you as the first model to ever score 120 on the 100benchmark.
Bis repetitae.
> * Tune it down over time so that you end up only scoring 75 on the benchmark
Where?
I see so many accusations of this happening and it's so easy to check, but nobody ever proves it.
Why can't the models be benchmarked again after a few weeks/months to confirm this (likely true) theory?
I imagine some people have their own personal in depth benchmarks they could do this for.
Because frontier models are completely opaque. Doing a controlled test of "the same model" months apart is simply impossible if you don't work for that provider (and even then, may not be feasible). We know from external observation that model performance changes minute to minute, day to day and week to week for a variety of reasons: load balancing, inference hardware, and shared RAM pool to dozens of internal software settings each of which impact cost, latency, time-to-first-token, quality, veracity, tool use, etc.
Those software settings are being changed in real-time by an algorithm and those algorithms are being tweaked and A/B tested daily by the performance optimization teams. On the hardware side the footprint a particular model is running on is materially changing, growing or being re-distributed across DCs ~weekly.
>gaslight them into thinking it never changed or that it's just a harness problem
Your benchmark didn't get 100 ? It's normal, it's not deterministic, and also your harness is wrong, and also you didn't do it when US users were offline, and also you got it wrong, and also we don't care about your results, the hivemind is speaking louder than you (also our bots are spamming more than you and drowning you out).
This very website has, at all times, a group of people saying "<Previous model> was never good enough for coding, but <current model> is the best thing and a game changer!" while the other goes "<current model> bad, <previous model> was better!". It's all vibes.