Comment by solarkraft
5 hours ago
Is anybody tracking these quality changes? All I've seen so far are accusations (quite a few at this point) but not really any actual data.
5 hours ago
Is anybody tracking these quality changes? All I've seen so far are accusations (quite a few at this point) but not really any actual data.
I don’t understand how there isn’t a website out there tracking this stuff already.
https://modelregression.com/
Haven't looked into how accurate the page is, but the list of regressions on the bottom looks terrifying, at first glance?
1 reply →
how the hell do we even track that? and before someone says...
—"Benchmarks!"
...I'll tell that they can be gamed so easily, and they are on a consistent basis.
Sure they are, but do you think they are continuing training to improve a model after release without bumping the version number, presumably only to game the benchmarks?
In a Codex subreddit there is a bunch of stats.