Comment by jeremyjh
9 hours ago
I'm curious how you think "word predictor" meaningfully describes an instruct model that has developed novel mathematical proofs that have eluded mathematicians for decades?
edit:
You cannot predict all the actions or words of someone smarter than you. If I could always predict Magnus Carlsen's next chess move, I'd be at least as good at chess as Magnus - and that would have to involve a deep understanding of chess, even if I can't explain my understanding.
I can't predict the next token in a novel mathematical proof unless I've already understood the solution.
I think that's more of a limitation in how people think about word predictors
If you can predict the words a bright person will say about X... Isn't that some truly astounding tool? That could be used in myriad useful ways if one is a little creative with it
Since it's also "alien" it can also detect and explore paths that we simply haven't noticed since their biases aren't quite the same as ours
Terence Tao himself answers that question (https://www.nature.com/articles/d41586-026-01246-9) :
"In almost any other application, the biggest Achilles heel of AI is that it makes unverifiable mistakes. But in mathematics, almost uniquely, you can automatically check the output — at least if the output is supposed to be the proof of a theorem, although that is not the only thing mathematicians do. So, AI companies have recognized that their most unambiguous successes — if they’re going to have any — are going to come from mathematics.
In my opinion, there are many use cases of AI that are risky and controversial. In mathematics, the downsides are much more limited"
AI successes in mathematics don't generalize to successes in other fields as the AI promoters want to suggest.
Magnus Carlsen understands chess, a machine designed to simply predict his next move would not necessarily understand chess. This is essentially the Chinese Room experiment.
So I think "word predictor" makes sense here. A word predictor can be really really cool.
What does "understand" even mean here? So many people arguing about this seem to assume they can just use words and everyone must accept that because the words have a certain connotation, their argument must be true.
I have no idea how Magnus Carlsen "understands" chess. Neither does anyone else. His brain is giant neural net, taking inputs, sending signals around, and coming out with an output. We think we understand the mechanics of this, but we do not understand exactly why or how sending these signals around produces such good outputs.
So to argue you know for certain that an LLM is not intelligent because it is "just" a next token predictor, without knowing if that is how the human brain operates, is thinking too highly of yourself.
I don't have to try and imagine how Magnus Carlsen understands chess, since I also understand chess, and I operate with the assumption that other people are not zombies and possess a similar form of consciousness. My comment works regardless of the skill of the player.
Imagine you have never played chess, you have no concept of the rules or how the game is played, yet you've learned the entirety of Stockfish's algorithms and can dutifully run them step by step on a piece of paper when you look at a chess position. You would be the strongest chess player ever, and yet you would have less understanding of the game than even a beginner. Just because you can take an input and produce an intelligent output does not mean there is any sort of underlying understanding. This is really just a modification of Searle's Chinese Room Argument, and one of the most famous refutations of functionalism.
https://plato.stanford.edu/entries/chinese-room/