Comment by charlieyu1

Compare:

"O3 performs spectacularly on a very hard dataset that was independently developed and that OpenAI does not have access to."

"O3 performs spectacularly on a very hard dataset that was developed for OpenAI and that only OpenAI has access to."

Or let's put it another way: If what they care about is benchmark integrity, what reason would they have for demanding access to the benchmark dataset and hiding the fact that they finance it? The obvious thing to do if integrity is your goal is to fund it, declare that you will not touch it, and be transparent about it.

If you’re a research lab then yes.

If you’re a for profit company trying to raise funding and fend off skepticism that your models really aren’t that much better than any one else’s, then…

It would be dishonest, but as long as no one found out until after you closed your funding round, there’s plenty of reason you might do this.

It comes down to caring about benchmarks and integrity or caring about piles of money.

Judge for yourself which one they chose.

Perhaps they didn’t train on it.

Who knows?

It’s fair to be skeptical though, under the circumstances.