Comment by fwlr
1 day ago
It is suspicious that OpenAI decided to generate 300 billion output tokens from a model still in training, right after learning there was a credible chance that a major math proof was in that model’s training data. Obviously there are reasonably plausible explanations for each step, but it does sort of feel like parallel construction.
I think people are focusing on the training data issue too much. If the data was contaminated, I can still blame that on negligence.
But, at least with the Navier-Stokes solution, it's clear [^1] that they learned that Alpöge and Buckmaster were getting close to a solution and learned of the general approach they were taking. Only after learning the secret to cracking the problem did they send the first prompt.
What makes this worse to me is the intention. They intentionally threw $15 million in compute at the problem in order to scoop the result. They intentionally left Buckmaster and Alpöge out of the citations.
Data contamination should be enough to disqualify them from the prize, but I can believe it to be accidental. On the other hand, someone made an intentional decision to scoop the result by throwing money at the problem. That's so much worse.
[^1]: That's the timeline claimed by Buckmaster, and no one from OAI has disputed it.
> and learned of the general approach they were taking. Only after learning the secret to cracking the problem did they send the first prompt.
Do you have any evidence of this? They don't dispute the timeline, but they never said they knew what Levant/Buckmaster were doing.
It's in OpenAI's first announcement that they had solved the problem.
10 replies →
I think you’re overlooking what I’m implying here. It’s not that they knew contamination was possible but they went ahead anyway. To spell it out just a little bit more: learning the answer might be in model X’s training data made them believe that model X specifically might be able to solve the question, and they were able to very quickly find enough certainty about the former to commit millions of dollars to the latter.
> the secret
So such thing existed. In fact, what they learnt was some progress existed, not what the specific progress was.
> They intentionally left Buckmaster and Alpöge out of the citations.
No, they asked if they could do a joint publish.
No, they asked one guy to do a joint publish conditioned on leaving the other collaborator out, with veiled threats. The joint publish part smells awfully like admission of guilt given there’s absolutely no reason to do it if you believe you independently arrived at the result using only public prior work. The leaving out collaborator part is outright academic malpractice. Disclosure: I was an academic once.
1 reply →
> but I can believe it to be accidental
What accident is it when the system is designed to function that way?
Their claim is that training on their solution is "unlikely but possible".
Consider this scenario.
Has a google crawler read my new novel, which I may or may not have posted on my blog, page by page, as I wrote it?
Can you, without knowledge of what I have actually done, claim that the google crawler has not seen the novel?
Without any evidence that I have posted the novel online, it might be tempting to say that the crawler has not seen the novel, but what if I were in an adversarial position against Google on this topic and were challenging them to make that claim. You would wonder if I were hoping Google to overreach by making a definitive claim without taking into account some action that they had no knowledge of. It becomes difficult to use the scientific expression "There is no evidence for this" when there is an accusation of malfeasance because it can be so easily be conflated as "You can't prove we did it". It seems like the best you could say would be 'Unlikely, but possible'
1 reply →
> They intentionally threw $15 million in compute at the problem
what? really?
Yes. Maybe much more:
> Such intensive use of AI doesn't come cheap. In a post on X, LisanBench, an LLM benchmark evaluator, estimated that the output tokens alone would cost about $6.5 million at OpenAI's average consumer price. Including the far larger volume of input tokens, the post estimated the total could reach $10 million to $40 million.
https://www.businessinsider.com/openai-math-problem-solved-t...
3 replies →
When I worked at Google, we spent $100M in power on protein folding and drug discovery (this was long before AlphaFold). Never underestimate the willingness of smart rich people to invest in speculative science.