Comment by thethimble
5 hours ago
I feel like LeCun has been plainly wrong about LLMs. He has been insisting that the stochastic nature of sampling tokens causes a non-zero hallucination property for any given next token such that as output length increases, this will inevitably converge towards garbage.
The reality is that while LLMs can make mistakes mid-output, those interim mistakes don't necessarily detract from the model's final output. We see a version of this all the time with agents as they make tactical mistakes but quickly backtrack and ultimately solve the root problem.
It really felt like LeCun was willing to die on this hill. He continued to argue about really pedantic things like the importance researchers, etc.
I'm glad he's gone and hopeful Meta can actually deliver real AI products for their users with better leadership.
I am a big fan of using LLMs although in my own limited way. I don't work at Meta and don't feel strongly about him leaving or staying there.
It's possible that he will turn out to be correct in the long run. From his viewpoint, the primary goal is research and any usefulness of intermediate advances is maybe (speculating) "beneath him". If this is the case, I completely understand why a corporation would want to eject him. LeCun probably sees the pretty amazing developments since ChatGPT first came out as incremental hacks. I am neutral about this aspect too. Maybe they are but the hacks have been useful to me.
Eventually this feels like the correction of a real misalignment between LeCun/FAIR and Meta. Hopefully now, they can both focus on what they are good at. I must admit that I have great sympathy for open-ended research but industry has always been fickle about it. That's where the government and universities are supposed to play a key role.