Comment by falcor84
14 days ago
I don't understand what LeCun is trying to say. Why does he give an interview saying that LLM's are almost obsolete just when they're about to release a model that increases the SotA context length by an order of magnitude? It's almost like a Dr. Jekyll and Mr. Hyde situation.
LeCun fundamentally doesn't think bigger and better LLMs will lead to anything resembling "AGI", although he thinks they may be some component of AGI. Also, he leads the research division, increasing context length from 2M to 10M is not interesting to him.
He thinks LLMs are a local maxima, not the ultimate one.
Doesn't mean that a local maxima can't be useful!
If that's what he said, I'd be happy, but I was more concerned about this:
> His belief is so strong that, at a conference last year, he advised young developers, "Don't work on LLMs. [These models are] in the hands of large companies, there's nothing you can bring to the table. You should work on next-gen AI systems that lift the limitations of LLMs."
It's ok to say that we'll need to scale other mountains, but I'm concerned that the "Don't" there would push people away from the engineering that would give them the relevant inspiration.
1 reply →
But ... that's not how science works. There are a myriad examples of engineering advances pushing basic science forward. I just can't understand why he'd have such a "fixed mindset" about a field where the engineering is advancing an order of magnitude every year
> But ... that's not how science works
Not sure where this is coming from.
Also, it's important to keep in mind the quote "The electric light did not come from the continuous improvement of candles"
2 replies →
Listening so Science Friday today on NPR, the two guests did not think AGI was a useful term and it would be better to focus on how useful actual technical advances are than some sort of generalized human-level AI, which they saw as more of a marketing tool that's ill-defined, except in the case of makes the company so many billions of dollars.
A company can do R&D into new approaches while optimizing and iterating upon an existing approach.