← Back to context Comment by pupppet 4 hours ago I don’t understand how there isn’t a website out there tracking this stuff already. 5 comments pupppet Reply copperx 2 hours ago https://modelregression.com/ hn8726 39 minutes ago Haven't looked into how accurate the page is, but the list of regressions on the bottom looks terrifying, at first glance? copperx 26 minutes ago Yes, it looks like regressions are frequent, but sometimes performance goes back to baseline quite fast. bpavuk 4 hours ago how the hell do we even track that? and before someone says...—"Benchmarks!"...I'll tell that they can be gamed so easily, and they are on a consistent basis. solarkraft 3 hours ago Sure they are, but do you think they are continuing training to improve a model after release without bumping the version number, presumably only to game the benchmarks?
copperx 2 hours ago https://modelregression.com/ hn8726 39 minutes ago Haven't looked into how accurate the page is, but the list of regressions on the bottom looks terrifying, at first glance? copperx 26 minutes ago Yes, it looks like regressions are frequent, but sometimes performance goes back to baseline quite fast.
hn8726 39 minutes ago Haven't looked into how accurate the page is, but the list of regressions on the bottom looks terrifying, at first glance? copperx 26 minutes ago Yes, it looks like regressions are frequent, but sometimes performance goes back to baseline quite fast.
copperx 26 minutes ago Yes, it looks like regressions are frequent, but sometimes performance goes back to baseline quite fast.
bpavuk 4 hours ago how the hell do we even track that? and before someone says...—"Benchmarks!"...I'll tell that they can be gamed so easily, and they are on a consistent basis. solarkraft 3 hours ago Sure they are, but do you think they are continuing training to improve a model after release without bumping the version number, presumably only to game the benchmarks?
solarkraft 3 hours ago Sure they are, but do you think they are continuing training to improve a model after release without bumping the version number, presumably only to game the benchmarks?
https://modelregression.com/
Haven't looked into how accurate the page is, but the list of regressions on the bottom looks terrifying, at first glance?
Yes, it looks like regressions are frequent, but sometimes performance goes back to baseline quite fast.
how the hell do we even track that? and before someone says...
—"Benchmarks!"
...I'll tell that they can be gamed so easily, and they are on a consistent basis.
Sure they are, but do you think they are continuing training to improve a model after release without bumping the version number, presumably only to game the benchmarks?