Comment by qnleigh
5 days ago
> I think "novel" is ill defined here
That's exactly my point. When people say "LLMs will never do something novel," they seem to be leaning on some vague, ill-defined notion of novelty. The burden of proof is then to specify what degree of novelty is unattainable and why.
As for evidence that they can do novel things, there is plenty:
1. I really did ask Gemini to multiply 167,383 * 426,397 before posting this question. It answered correctly.
2. SVGs of pelicans riding bicycles
3. People use LLMs to write new apps/code every day
4. LLMs have achieved gold-medal performance on Math Olympiad problems that were not publicly available
5. LLMs have solved open problems in physics and mathematics [0,1]
That is as far as they have advanced so far. What's next? Where is the limit? All I want to say is that I don't know, and neither do you :).
Actually here's an even better list of progress on a number of open math problems, with plenty of caveats and exposition:
https://github.com/teorth/erdosproblems/wiki/AI-contribution...
This is great observational data but it's an early "step 1", I'd definitely need to see an actual analysis of these cases and likely want to have that analysis involve a review of relevant training data.
What you're asking for is exactly what's in the link you replied about. It collects analysis of each solution (or attempt), and info about whether the AI's solution could be found anywhere in the literature.
4 replies →