Comment by Etheryte
6 months ago
Many big names in the industry have long advocated for the idea that LLM-s are a fundamental dead end. Many have also gone on and started companies to look for a new way forward. However, if you're hip deep in stock options, along with your reputation, you'll hardly want to break the mirage. So here we are.
They're a dead end for whatever their definition of "AGI" is, but still incredibly useful in many areas and not a "dead end" economically.
Well, except for that needing a 40 year bond for a 3 year technology cycle thing.
I figure it's more like steam engines and flight. While steam engines were not suitable for aircraft, experience building them could carry over to internal combustion engines. I imagine something like that with LLMs and AGI.
"Fundamental dead end" strikes me as hyperbolic. Clearly they could be an import part of an "AGI" system, even if they're not sufficient for building an AGI in and of themselves?
"It is difficult to get a man to understand something when his salary depends upon his not understanding it" and "never argue with a man whose job depends on not being convinced" in full effect.
How can u look at progress in LLMs and think "this is mirage"
Because the depth of concept is so deep, and if you're just beginning then it's going to take a while for that statement to illuminate how obvious it is.
For those who've been sniffing this since early 2010, it's so blindly obvious they've already dropped llms on the floor and moved onto deeper alternative research.
For the rest of us, we're still catching coke bottles from the sky and building places of worship around them
Name some? Because YeCun's break is the first big name I've seen strike out in a major fashion away from the LLM trajectory.
> Many big names in the industry have long advocated for the idea that LLM-s are a fundamental dead end.
There should be papers on fundamental limitations of LLMs then. Any pointers? "A single forward LLM pass has TC0 circuit complexity" isn't exactly it. Modern LLMs use CoT. Anything that uses Gödel's incompleteness theorems proves too much (We don't know whether the brain is capable of hypercomputations. And, most likely, it isn't capable of that).
I have some idea of what the way forward is going to look like but I don't want to accelerate the development of such a dangerous technology so I haven't told anyone about it. The people working on AI are very smart and they will solve the associated challenges soon enough. The problem of how to slow down the development of these technologies- a political problem- is much more pressing right now.
> I have some idea of what the way forward is going to look like but I don't want to accelerate the development of such a dangerous technology so I haven't told anyone about it.
Ever since "AI" was named at Dartmouth, there have been very smart people thinking that their idea will be the thing which makes it work this time. Usually, those ideas work really well in-the-small (ELIZA, SHRDLU, Automated Mathematician, etc.), but don't scale to useful problem sizes.
So, unless you've built a full-scale implementation of your ideas, I wouldn't put too much faith in them if I were you.
Far more common are ideas that don't work on any scale at all.
If you have something that gives a sticky +5% at 250M scale, you might have an actual winner. Almost all new ML ideas fall well short of that.
2 replies →
999999 times out of a million you'd be right.
But, I shouldn't have said anything.
Let me guess, you have a truly marvelous demonstration of this proposition which this margin is too narrow to contain.
Well it kind of sucks honestly because im never going to get any sort of recognition, but that's partly by choice and partly because I don't have the right personal connections. I 100% understand why all of you think I'm a significant fool. Thats ok. Like i said, im burying the idea. But its obvious enough that someone else will discover it soon enough.
5 replies →
[dead]
cringe
Reject subtlety
Embrace cringe
[flagged]
If you're so confident that your discovery will be found by someone else soon why not just share it right here and get the credit you seem to want? Otherwise why even bring it up in the first place?
1 reply →
You're being downvoted for displaying the kind of overconfidence that people consider shameful.
Everyone in ML has seen dozens to thousands of instances of "I have a radical new idea that will result in a total AI breakthrough" already. Ever wondered why the real breakthroughs are so few and far in between?
3 replies →
[dead]