Comment by janalsncm

> what if companies use this to fake benchmarks

How would this work? Are you talking about training on the test set as well? Some benchmarks have private test sets.

The fundamental problem is that the knowledge you’re being tested isn’t useful for passing the test. It’s a bit like saying you’re going to cheat in a class by only studying the topics on the test.

Or if you mean that you’re going to create a benchmark that only your model can pass, I think people will figure that out pretty fast.