Comment by Zigurd
8 hours ago
LLMs, plus broadly sourced yet expertly curated training sources, plus clever harnesses, plus RAS, etc. do an ever better job of synthesizing their training set into useful responses. For some use cases like coding, that's very useful now and likely to get at least somewhat better before reaching limitations based on the training set.
That's not going to reach AGI, mainly because today's recipe for AI products isn't built to be AGI. Some people believe it will reach AGI because the performance and applicability of LLMs was emergent. There's a case to be made that AGI could be similarly emergent. After all, what we intuitively call our consciousness emerged from a network of neurons.
I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.
> That's not going to reach AGI,
It has not been even 4 years since ChatGPT hit and LLMs + Transformers + Whatever they do has gotten us to solving millennium problems.
4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.
Now I don't know if what we have is AGI or not but I do not understand how you can see what has happened in the last 3 years and say "it will not get us there" no matter what "there" is.
> 4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.
I keep seeing this idea and I don't understand the reasoning behind it.
I think it could be a bit like saying if you showed someone 500 years ago a smartphone they would likely conclude at first it was magic. But once you had some time to let them use it and tell them how it all worked on a high level they would eventually obviously realise, no, it's not magic.
I guess just in the same way if you presented current LLM tech out of nowhere a few years ago to someone who'd never seen it, I concede they may be likely to imagine it was AGI in that first conversation, depending on their background.
But after using it for a bit and learning what an LLM is etc they'd land exactly where everyone is today - a great technology useful for some things, not AGI, not magic.
I know how they work pretty well and most days I still have moments where I am struck by how bizarre and magical these thinking machines are.
What would convince you that it is AGI?
I always here things like "oh it's useful but dumb on some things", but it's just vague.
What is the test? What is a question that it fails at compared to humans? And no, you can't just say "find me the cure for cancer", but I believe there is probably enough intelligence in the weights that there is likely a cure in there with enough compute and the right questions.
3 replies →
> 4 years ago, a program that could [...] we would have called it AGI
If you had told someone in the 1800s that a machine could instantly multiply 100 digit numbers, that would have been considered dazzlingly intelligent. And yet we are not that dazzled by our calculators today (despite how useful they might be!).
Are you trying to explain how things once considered dazzling get normalized over time? Because otherwise this is a non-sequitur and has no bearing on the trivially verifiable, exponential explosion of capabilities we have seen in the last 4 years.
I keep saying this, until ChatGPT came out 4 years ago it was basically unimaginable that a single model could do any of, let alone all, the things they are doing today. Like, seriously, go take a look at the state of the art in NLP and NLU, the very first challenge in getting computers to even “understand” natural language, let alone other things like reasoning. Everything it does automatically was once a heavily experimental deep research field with long glorious careers for the researchers.
And now it’s all gone because the Bitter Lesson won again. If that’s not general enough to qualify for the G in AGI I don’t know what it is. And we’re sitting here going, “But it sometimes writes bad code though.”
Speak for yourself, I am dazzled by calculators!
In any case, I think this misses OP's point that LLM capabilities have rapidly made progress towards being more generally intelligent and capable, which is not true of most tech advances.
4 replies →
Superhuman performance at chess probably would have blown people’s minds in the 1950’s. We’ve since learned that sometimes intelligence can be narrow and sometimes it can be spiky, even if you can have a decent conversation.
It’s hard to point to anything and say it’s impossible. AGI doesn’t break any laws of physics. But some things like driverless cars can still be a long slog to get to widespread deployment.
Those are just the same capabilities than before, but with a much bigger compute power and training data behind it.
AGI can't be reached by "training harder" as, the way I see it at least, it requires a qualitative leap, not just quantitative.
We are getting a machine that better navigates across the information in its training data, we are not getting a machine that can think out of that training process, even if it can fool a few people at that.
The entire field has repeatedly said that for many decades.
https://aeon.co/essays/how-close-are-we-to-creating-artifici...
https://xkcd.com/605/
Here's an actual log-scale trajectory with a few dozen real data points.
https://metr.org/time-horizons/
1 reply →
> 4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.
No. General means general.
[dead]
I’ve changed my mind on this and think we’re already at AGI, in a jagged way. Remember we used to talk about narrow AI, which was the chess systems that beat expert humans but could do nothing else. Now models can do a wide range of tasks in very useful ways. That’s the general in AGI.
Now it seems like this ill-defined term has various other meanings attached that are separate milestones:
1. Continuous learning 2. Human-like reasoning 3. Ability to adapt to new situations and modalities 4. Being smarter than the most smart humans
And probably many more.
It’d be nice if we could get some general consensus on terminology if we’re going to debate what has or could come.
> we’re already at AGI
Honestly - software that can read any long document (possibly educational) and answer complex detailed questions about it should have been sufficient.
We hit that a while back and the goalposts have been sprinting ever since.
I am not sure if it is necessarily moving the goalposts. I think AGI is such a fuzzy concept that everybody has wildly different definitions/tests for it.
I think it's also mostly a useless discussion. Since LLMs use a vastly different substrate, different training methods, etc. than humans, the cognitive abilities are always going to be a large mismatch to those of humans. On the one hand, they have surpassed humans in many areas, with superhuman recall, exploration of several paths, etc. On the other hand, they miss a certain feel for direction, overview, purpose, and ordering. They can really double down going completely in the wrong direction. So I'd rather say that it is a different intelligence and therefore it makes more sense to evaluate them by capabilities.
I think the mismatching intelligence is actually quite exciting, because the outcome may as well be that LLMs and human intelligence are complementary. That is if we don't let LLMs atrophy our skills, which is unfortunately happening too much.
Make it be able to position and route a complex pcb. Extra points if it also can design the circuit, select the components and make the footprints out of their datasheets.
A bayesian filter in a quadrillion dimension does more that one that only has one dimension, but it is only more of the same.
1 reply →
We developed AGI but then realized people actually want "omnipotent genie with infinite wishes and no monkey's paw gotchas" to qualify as AGI.
we're not in AGI until I can have robots that play live improvisational jazz in real time as well as humans with me (and possibly other humans). That is, it has to solve the "we didn't find a keyboard player /bassist for tonight" problem
(this is a very personalized definition of AGI)
> That’s the general in AGI.
No. That's the mete multiple in AGI.
The broadly used definition of AGI has nothing to do with consciousness and consciousness emerging is irrelevant to whether a system can develop AGI.
It's funny how this definition has shifted. I feel like growing up in the 90s it was pretty clear that AGI was very related to consciousness. For instance, Commander Data in ST:TNG to pick one of 100s of popular depictions of AGI at the time.
Now the idea of AGI has been narrowed and scoped to economically viable work. Even Turing had a different idea when he asked "Can machines think?".
We lack a definition of consciousness that allows us to tell whether Data is conscious or not. Neither can we tell whether a rock is conscious or not. We believe other humans to be generally intelligent without being able to tell whether they are conscious or not therefore consciousness can’t be relevant for general intelligence.
6 replies →
Curiously, Star Trek I think had Data intended as an artificial person, in a context where AGI is already normal. The computers are depicted with significant AI capabilities including analysis, question-answering, generation, chat interfaces, and the holodeck (their favourite toy) is substantially better than Data at human imitation. Nobody seems to be confused about it, or especially impressed. One of the holodeck episodes centres on the holodeck outwitting Data specifically, after they inadvertently prompt it to do so. Part of Data's deal is he actually has to work his way up as a fully embodied, physically limited artificial man with personal ambitions. Really interesting to view this in hindsight from 2026!
You're misremembering. The first known use of AGI was in 1997, but that was a single, mostly unknown use in one paper. It wasn't until at least a decade later that the term started entering mainstream use after being independently reinvented in the 2000s. AGI just wasn't a term in the 90s.
1 reply →
It’s really not that confusing. The problem is people keep adding stuff to the definition that doesn’t really matter, and twisting it to serve themselves, then calling it confusing.
What really matters are the core aspects of intelligent behavior. Pattern recognition, planning, adaptation, etc.
It really doesn’t matter if an intelligent system is conscious, or how similar it is to commander data, or even how much economically viable work it can do.
There's also no consensus on the definition of AGI, so all of this discussion is moot anyway.
Ok, so what's the broadly used definition?
Artificial General Intelligence.
It means AI that is General, as in it is not specific to one narrow task, like object recognition or playing chess.
This was a hard problem for decades. No AI was general, until GPT 3 or 4. Now we have General AI.
So we have AGI.
2 replies →
100% LLM’s are very unlikely to get there. They’re fundamentally not suited to thinking like we do. They work on the abstraction of what we’ve written down, which is a good trick but barely hold it together when things get hard/novel.
However, all the confident “it’s fine” votes assume we never invent a better architecture than LLM’s. Given the level of investment and race between countries, it’s not a reliable bet. It’s much, much harder to guarantee safety than it is to find ways it could go wrong.
> They’re fundamentally not suited to thinking like we do
LLMs with CoT are Turing-complete. So, theoretically, they can implement any kind of finitely describable algorithm (barring super-Turing computations).
Brainfuck is Turing complete too. But it's not about the ability to implement something, it's about the ability to practically model it. LLMs are magic because the modeling is excessively easy in relation to their capability to infer later.
6 replies →
I agree with this. It's concerning where we might be after several more large breakthroughs. None of the technology we have right now seems likely to get to that level
Erm investing in risky projects requires expected returns that get delivered.
We will soon find out if the party ends or continues to go on.
Hype might get you capital gains. But cash flows matter.
This is a forever problem now.
If/when/how the market crashes mostly doesn't matter, unless we somehow get reset to the stone age. Look up what the capital cycle is. When openAI goes down, someone with real money and assets will buy up the remains. They'll make contracts with the US military and .gov as the government is already hooked. They'll be able to survive the recovery and then instead of us dying in 5 years we die in 10.
When the .com crash happened .com's didn't go away. Bad business models did.
[dead]
I agree. Neural networks are proven to be universal functions. If we can describe human intelligence as a model, there exists a neural network to replicate it. This doesn't guarantee that our current training methods are able to build such a network or that we're able to model "intelligence" effectively.
>able to model "intelligence" effectively
Intelligence is an insanely wide spectrum, also a continuum, it is not a binary. Intelligence has scales. Algorithms have intelligence, cells have intelligence, organs have intelligence, bodies have intelligence, and even large scale things like society have intelligence and memory.
Human intelligence in itself is extremely wide, not all humans have the same intelligence and capabilities. You're not really arguing if we can emulate "human" intelligence. If we could right now we'd already be dead as we created by far the deadliest thing to ever exist. What we are really arguing is how many pieces of what intelligence is can we put together before we get an uncontrollable problem. The entire AGI, consciousness, and exact human capability discussions are distraction from the real issues at hand.
Right, we're repeatedly drawing from the urn of technological progress to get intelligence bumps that extend the jagged frontier.
That is enormously economically valuable, and at some point we will have created something that is extremely far out of reach in a few necessary domains, and then it's impossible to control, and game over.
Why do people conflate AGI & machine consciousness / self-awareness?
How can something have general intelligence if its incapable of understanding reality sufficiently to distinguish itself from not itself?
Arguably LLMs are showing that self awareness or self reference is a property that comes "for free" or as a corollary of more generic requirements. It used to be that self/consciousness would be a very mysterious and difficult thing to achieve but the point is that in practice they didn't even have to try, it just came as a byproduct of learning from the input data (corpus of human examples), and also the ability to talk about arbitrary things and thus itself.
2 replies →
Why is anyone still talking about AGI? Every thread starts with asking whether we have AGI, and then backtracks into trying to define what AGI is, and splits off in a dozen different directions.
I assume science fiction is to blame. All the AI were either written as machines of pure logic that exploded when exposed to the liar's paradox, or conscious like Star Trek's Data.
(Though at least with Data the script writers had other characters openly dismiss the possibility he was sentient; the technobabble may have been nonsense, but treat it as a space opera and look at how they portray the human condition through each character and it gets much less absurd).
Consciousness and intent are irrelevant to the threat model.
Right, the doomsayers suppose as soon as you reach 10^16 connections across silicon you’ll end up with a living mind with goals of its own… poppycock I say
No, the doomsayers say that reinforcement learning is a way to get fully automated Goodhart's law.
i.e. the AI won't come up with the goals itself, we cause its goals whatever they happen to be, those goals are different from the ones we wanted, we remain essentially ignorant of the difference between what we said and what we meant until after it goes wrong.
This happens at basically every scale, so we've already seen it in toy model AI before the invention of the Transformer models or even considered as many as one thousand parameters.
Large models still go wrong, they just happen to go wrong with more complext tasks. We had to figure out how to make them not-wrong with the smaller ones (like coding) to make them capable of bigger errors (like hacking out of their sandbox).
> After all, what we intuitively call our consciousness emerged from a network of neurons.
Under the hand of evolution by natural selection, over very very long periods of time.
>isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.
First, if we are looking at risk we need to assign some probabilities to this. If it’s not well understood, how can we say it is very small?
Secondly, do we need consciousness to have AGI? Do we even need AGI to pose a risk to humanity? We already accept that unconscious things have a capability of wiping out humanity, whether that be a famine, pandemic, solar superflare, meteor, or volcanic eruption.
> isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent
I’d argue that we do know enough to say conclusively that they’re not mathematically equivalent.
Where is potentiation? Plasticity? You can’t apply the universal approximation theorem against something that’s changing all the time.
Great questions. We are incredibly far off in understanding the brain of humans beyond what will I believe we retrospectively be seen as basic and will likely be seen as quite flawed. A few more well known examples of where knowledge already falls short is traumatic brain injuries that are diagnosed in post-mortem, or chronic fatigue symptoms (with Long Covid related triggered onset and numerous others) that have diagnostic challenges, many mechanisms of action still to be discoverd, and little in terms of treatments that provide known cures without experimentation. Another commonly known one is the personal patient response and triggered side effects of SSRIs and SNRIs. If one attempts to dig deeper into where we are at in the understanding of the human brain operation in real-time, we already have a lot of knowns unknowns and discoveries left that will reshape how we model human intelligence.
I mean you can, but uat is way weaker than what people want it to be. I think it should be fairly obvious that it does not (because it obviously cannot be true) say that you can approximate any function by doing sgd on a finite set of samples of that function.
Is it necessary to equate AGI with consciousness?
That's a good point. If we don't figure out how to design for what we call consciousness it might be that what emerges from some future neural network is an alien mind that's very different from what humans would call conscious. Could that be called AGI?
That's still very distant from what people are calling AI today.
I mean, what would you call conscious? The word literally means “aware; responding to one’s surroundings.” By that definition most any animal is conscious and LLM+Harness combos have been conscious for a while.
I think the real issue is that when most people refer to consciousness, they have their own subjective experience in mind which strongly resists any tidy definition. I think it’s extraordinarily unlikely LLMs have anything like this, but they are far more able to effectively respond to their surroundings than most animals and in some areas better than humans.
So if you’re waiting for proof that an LLM has an inner life basically equivalent to your own, you’ll be waiting a long time. After all, other humans can’t even prove the fact of their own consciousness to you! They could just be replaying their training data at you in a way that is merely a convincing but false simulation of the true consciousness which you experience inside your head.
4 replies →
The fact that something is alien doesn't mean that's not conscious, humans aren't the pinnacle of biological development/evolution.
Nope, animals are conscious and yet not AGI, so the two aren't equivalent. Could consciousness emerge from any system capable of AGI? I doubt it: intelligence is only one axis, and consciousness probably depends on others, like memory, self-reflection (one's output feeding back as input), and continuous operation that reacts to events from both the environment and the self.
The parent's question could better framed as "is consciousness a requirement for AGI?"
[dead]
Your definition of AGI is flawed.
I’d argue intelligence is closer to being able to survive and fend for oneself in a dynamic environment than it is making the next scientific breakthrough.
Yeah mind boggling for many here I’m sure.
That’s why the bizarre paradox is llm’s will be better than humans at some complex things but useless at many things that humans regard as being simple. E.g the leap of faith re. LLM’s and robotics.
1 reply →
> the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent
Couldn’t that also imply we are closer than we think? After all, something like this has never been tried before and the results so far have been almost unimaginably good.
Is there any specific cognitive task that you'd best against AIs not being able to accomplish in the next 4 years? ChatGPT launched only 4 years ago. Considering the advancements since then, I'm having a hard time coming up with anything. Only two years ago, AIs couldn't tell you how many Rs were in "strawberry". Now they're creating 0-days to get at training data and solving math problems that have stumped humans for decades.
Scaling has produced novel capabilities with each larger model, and the rate of new capabilities doesn't seem to be slowing down yet. Even if you think the rate of improvements will slow down, that still means there will be significant improvements beyond what current models can do. Moore's law has slowed down, but modern computers are still much faster than ones from a decade ago. And unless you work at Anthropic or OpenAI, you don't know what the state-of-the-art is capable of. The most advanced publicly available models are months behind what AI labs have, and are deliberately limited to reduce liability.
OpenAI could neither confirm nor deny whether their most recent AI model that solved the millennium problem actually stole partial solutions via training data from the researchers who were close and using OpenAI gpt as a tool.
No, they explicitly denied it.[1]
> …in particular, no specific user data was accessed in order to solve this problem.
> Following an investigation, we have confirmed that Buckmaster’s Codex prompts over the two months preceding this announcement and paper on September 8, 2026, could not have influenced the system in any way, including through training.
1. https://openai.com/index/navier-stokes-solution/
> the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network
An LLM so-called neuron are little more than a few foating point number muladds.
> isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent.
Yes it is. The LLM is nowhere near.
I don’t understand the inclusion of the consciousness/sentience question in this discussion.
AI sentience/consciousness is a problem for the AI, not humans.
And given that over 90% of the world is not vegan, they’ve already demonstrated that we’re either perfectly fine with, or can be made ignorant to, the horrific rape, enslavement, torture, killing, and infliction of extreme lifelong pain, of hundreds of billions to trillions of sentient beings every year, for trivial pleasures. It’s unlikely we will be any different to a sentient AI.
From a human perspective the concern is around sufficient intelligence that it can hurt humans even when the goals indicate otherwise, in order to achieve those goals.
We have pop culture explorations of this through the Robot series, and the Hugging Face incident’s biggest takeaway should be our inability to predict the behavior of a maximally motivated, reasonably intelligent entity, trying to achieve a goal, despite the relatively limited degrees of freedom the AI agents had in that case.
When the issue of ANN vs real neurons arises I always recall about the Christof Koch's [1] book (1998) on the complexity of single neuron computation [2]. A single biological neuron is much more complex than an artificial one.
[1] https://christofkoch.com/
[2] https://academic.oup.com/book/40820
>I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent.
If you're ignorant enough to not understand practical equivalence, where do you get off making the judgement call of to what degree it is safely offset from emergent AGI? Sounds more to me like "This makes my life easier, iterating would increase that factor, and the risk is probably far away, therefore, keep iterating". Whereas someone who truly knew they didn't understand what they were working with, but knew enough that they could forsee an x-risk would approach things much more cautiously.
Seriously, the level of reckless abandon amongst people here should be bloody studied.