Comment by onlyrealcuzzo
2 days ago
His main LLM predictions have almost nothing to do with Arc AGI...
What exactly was he dead wrong about that is proven by any of this?
GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying.
He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3.
He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value.
Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.
>He's merely said they don't think
He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.
> He said even 'GPT 5000' couldnt do things that they could do a month later, let alone by 5000.
What things specifically and when?
https://youtube.com/shorts/zQTt8TkcyfU?is=09r7XDqz2w6-Pygu
You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.
10 replies →
> What exactly was he dead wrong about that is proven by any of this?
He said as you need more and more tokens models will fall apart because each additional token is a chance for a mistake and they will just exponentially fall apart. But in practice models have learned to identify and self-correct mistakes and if you look at the graphs more inference reasoning tokens almost always give far better accuracy.
That's true, but he's still correct, it's just that the context is now so large that only people using agent loops see "context rot"
His other criticism of LLMs that I like better is that they try to predict tokens instead of learned embeddings. Tokens are arbitrary and in order to decode LLMs you need technical analysis (see mechanistic interpretability).
With JEPA models so far, it seems that PCA on latent vectors suffices.
tldr: embeddings have a lot more room for improvement
One of the big things that enabled extended context to work was training techniques for extending LLMs with reasoning, part of the thing he was saying wouldn't work.