Comment by functionmouse

7 hours ago

Jokes on them, I don't trust benchmarks

Once something becomes a benchmark it is no longer a good benchmark.

4 comments

functionmouse

Reply
fluoridation  6 hours ago

No, it's when a benchmark becomes a target. You might have a private benchmark that you tell no one about. Would you not trust it?

  • charlieyu1  2 hours ago

    Good benchmarks are costly to build even for mid-large corporations. And once the benchmark is used on models you really can’t tell if the problems would be scrapped for training