LLMs, plus broadly sourced yet expertly curated training sources, plus clever harnesses, plus RAS, etc. do an ever better job of synthesizing their training set into useful responses. For some use cases like coding, that's very useful now and likely to get at least somewhat better before reaching limitations based on the training set.
That's not going to reach AGI, mainly because today's recipe for AI products isn't built to be AGI. Some people believe it will reach AGI because the performance and applicability of LLMs was emergent. There's a case to be made that AGI could be similarly emergent. After all, what we intuitively call our consciousness emerged from a network of neurons.
I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.
It has not been even 4 years since ChatGPT hit and LLMs + Transformers + Whatever they do has gotten us to solving millennium problems.
4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.
Now I don't know if what we have is AGI or not but I do not understand how you can see what has happened in the last 3 years and say "it will not get us there" no matter what "there" is.
> 4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.
I keep seeing this idea and I don't understand the reasoning behind it.
I think it could be a bit like saying if you showed someone 500 years ago a smartphone they would likely conclude at first it was magic. But once you had some time to let them use it and tell them how it all worked on a high level they would eventually obviously realise, no, it's not magic.
I guess just in the same way if you presented current LLM tech out of nowhere a few years ago to someone who'd never seen it, I concede they may be likely to imagine it was AGI in that first conversation, depending on their background.
But after using it for a bit and learning what an LLM is etc they'd land exactly where everyone is today - a great technology useful for some things, not AGI, not magic.
> 4 years ago, a program that could [...] we would have called it AGI
If you had told someone in the 1800s that a machine could instantly multiply 100 digit numbers, that would have been considered dazzlingly intelligent. And yet we are not that dazzled by our calculators today (despite how useful they might be!).
Those are just the same capabilities than before, but with a much bigger compute power and training data behind it.
AGI can't be reached by "training harder" as, the way I see it at least, it requires a qualitative leap, not just quantitative.
We are getting a machine that better navigates across the information in its training data, we are not getting a machine that can think out of that training process, even if it can fool a few people at that.
I’ve changed my mind on this and think we’re already at AGI, in a jagged way. Remember we used to talk about narrow AI, which was the chess systems that beat expert humans but could do nothing else. Now models can do a wide range of tasks in very useful ways. That’s the general in AGI.
Now it seems like this ill-defined term has various other meanings attached that are separate milestones:
1. Continuous learning
2. Human-like reasoning
3. Ability to adapt to new situations and modalities
4. Being smarter than the most smart humans
And probably many more.
It’d be nice if we could get some general consensus on terminology if we’re going to debate what has or could come.
we're not in AGI until I can have robots that play live improvisational jazz in real time as well as humans with me (and possibly other humans). That is, it has to solve the "we didn't find a keyboard player /bassist for tonight" problem
It's funny how this definition has shifted. I feel like growing up in the 90s it was pretty clear that AGI was very related to consciousness. For instance, Commander Data in ST:TNG to pick one of 100s of popular depictions of AGI at the time.
Now the idea of AGI has been narrowed and scoped to economically viable work. Even Turing had a different idea when he asked "Can machines think?".
100% LLM’s are very unlikely to get there. They’re fundamentally not suited to thinking like we do. They work on the abstraction of what we’ve written down, which is a good trick but barely hold it together when things get hard/novel.
However, all the confident “it’s fine” votes assume we never invent a better architecture than LLM’s. Given the level of investment and race between countries, it’s not a reliable bet. It’s much, much harder to guarantee safety than it is to find ways it could go wrong.
> They’re fundamentally not suited to thinking like we do
LLMs with CoT are Turing-complete. So, theoretically, they can implement any kind of finitely describable algorithm (barring super-Turing computations).
I agree with this. It's concerning where we might be after several more large breakthroughs. None of the technology we have right now seems likely to get to that level
I agree. Neural networks are proven to be universal functions. If we can describe human intelligence as a model, there exists a neural network to replicate it. This doesn't guarantee that our current training methods are able to build such a network or that we're able to model "intelligence" effectively.
Intelligence is an insanely wide spectrum, also a continuum, it is not a binary. Intelligence has scales. Algorithms have intelligence, cells have intelligence, organs have intelligence, bodies have intelligence, and even large scale things like society have intelligence and memory.
Human intelligence in itself is extremely wide, not all humans have the same intelligence and capabilities. You're not really arguing if we can emulate "human" intelligence. If we could right now we'd already be dead as we created by far the deadliest thing to ever exist. What we are really arguing is how many pieces of what intelligence is can we put together before we get an uncontrollable problem. The entire AGI, consciousness, and exact human capability discussions are distraction from the real issues at hand.
Why is anyone still talking about AGI? Every thread starts with asking whether we have AGI, and then backtracks into trying to define what AGI is, and splits off in a dozen different directions.
I assume science fiction is to blame. All the AI were either written as machines of pure logic that exploded when exposed to the liar's paradox, or conscious like Star Trek's Data.
(Though at least with Data the script writers had other characters openly dismiss the possibility he was sentient; the technobabble may have been nonsense, but treat it as a space opera and look at how they portray the human condition through each character and it gets much less absurd).
Right, the doomsayers suppose as soon as you reach 10^16 connections across silicon you’ll end up with a living mind with goals of its own… poppycock I say
>isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.
First, if we are looking at risk we need to assign some probabilities to this. If it’s not well understood, how can we say it is very small?
Secondly, do we need consciousness to have AGI? Do we even need AGI to pose a risk to humanity? We already accept that unconscious things have a capability of wiping out humanity, whether that be a famine, pandemic, solar superflare, meteor, or volcanic eruption.
Great questions. We are incredibly far off in understanding the brain of humans beyond what will I believe we retrospectively be seen as basic and will likely be seen as quite flawed. A few more well known examples of where knowledge already falls short is traumatic brain injuries that are diagnosed in post-mortem, or chronic fatigue symptoms (with Long Covid related triggered onset and numerous others) that have diagnostic challenges, many mechanisms of action still to be discoverd, and little in terms of treatments that provide known cures without experimentation. Another commonly known one is the personal patient response and triggered side effects of SSRIs and SNRIs. If one attempts to dig deeper into where we are at in the understanding of the human brain operation in real-time, we already have a lot of knowns unknowns and discoveries left that will reshape how we model human intelligence.
I mean you can, but uat is way weaker than what people want it to be. I think it should be fairly obvious that it does not (because it obviously cannot be true) say that you can approximate any function by doing sgd on a finite set of samples of that function.
> the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent
Couldn’t that also imply we are closer than we think? After all, something like this has never been tried before and the results so far have been almost unimaginably good.
That's a good point. If we don't figure out how to design for what we call consciousness it might be that what emerges from some future neural network is an alien mind that's very different from what humans would call conscious. Could that be called AGI?
That's still very distant from what people are calling AI today.
Nope, animals are conscious and yet not AGI, so the two aren't equivalent.
Could consciousness emerge from any system capable of AGI? I doubt it: intelligence is only one axis, and consciousness probably depends on others, like memory, self-reflection (one's output feeding back as input), and continuous operation that reacts to events from both the environment and the self.
I don’t understand the inclusion of the consciousness/sentience question in this discussion.
AI sentience/consciousness is a problem for the AI, not humans.
And given that over 90% of the world is not vegan, they’ve already demonstrated that we’re either perfectly fine with, or can be made ignorant to, the horrific rape, enslavement, torture, killing, and infliction of extreme lifelong pain, of hundreds of billions to trillions of sentient beings every year, for trivial pleasures. It’s unlikely we will be any different to a sentient AI.
From a human perspective the concern is around sufficient intelligence that it can hurt humans even when the goals indicate otherwise, in order to achieve those goals.
We have pop culture explorations of this through the Robot series, and the Hugging Face incident’s biggest takeaway should be our inability to predict the behavior of a maximally motivated, reasonably intelligent entity, trying to achieve a goal, despite the relatively limited degrees of freedom the AI agents had in that case.
Is there any specific cognitive task that you'd best against AIs not being able to accomplish in the next 4 years? ChatGPT launched only 4 years ago. Considering the advancements since then, I'm having a hard time coming up with anything. Only two years ago, AIs couldn't tell you how many Rs were in "strawberry". Now they're creating 0-days to get at training data and solving math problems that have stumped humans for decades.
Scaling has produced novel capabilities with each larger model, and the rate of new capabilities doesn't seem to be slowing down yet. Even if you think the rate of improvements will slow down, that still means there will be significant improvements beyond what current models can do. Moore's law has slowed down, but modern computers are still much faster than ones from a decade ago. And unless you work at Anthropic or OpenAI, you don't know what the state-of-the-art is capable of. The most advanced publicly available models are months behind what AI labs have, and are deliberately limited to reduce liability.
When the issue of ANN vs real neurons arises I always recall about the Christof Koch's [1] book (1998) on the complexity of single neuron computation [2]. A single biological neuron is much more complex than an artificial one.
>I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent.
If you're ignorant enough to not understand practical equivalence, where do you get off making the judgement call of to what degree it is safely offset from emergent AGI? Sounds more to me like "This makes my life easier, iterating would increase that factor, and the risk is probably far away, therefore, keep iterating". Whereas someone who truly knew they didn't understand what they were working with, but knew enough that they could forsee an x-risk would approach things much more cautiously.
Seriously, the level of reckless abandon amongst people here should be bloody studied.
LeCun also said back in 2022 that "if you train a machine, as powerful as it could be, your 'GPT-5000', on text", it will never be able to learn basic common-sense physics like that objects placed on tables will move along with them.
It would be good if one's reputation tracked one's track record of predictive accuracy. But many people will take what LeCun says as gospel regardless of how badly wrong he has been and continues to be.
Is there anyone who has not been badly wrong? I've been reading these debates for years and I don't think I've seen anybody pick the right spot on the bearish to bullish spectrum. The only thing I've become more certain of in this time has been uncertainty.
> never be able to learn basic common-sense physics
And has it at this stage, within in-depth take of said "learning", foundationally?
I have not been able to properly check the studies for a long time now, but I remain unaware of achieved solutions on the problem of reliably referencing a world model out of a language model - that "counting the 'r's in 'raspberry'" be not guessing, not memory, but actually counting.
My perspective is that the addition of thinking loops to models allows sufficiently advanced ones to approximate world models.
Incredibly inefficiently because of the recursive loops ("Wait, the object is on the table. I should think about this more deeply..."), and likely instantly surpassed by large world models if/when those are shipped, but effectively enough vs non-thinking models.
LeCun's argument wasn't about the definition of learning though. He stated that they would never get these common sense things correct because they weren't sufficiently part of the training data. A statement that we can hopefully all agree has been thoroughly refuted.
To determine this, it would first need to be able to spell "raspberry" as letters rather than as tokens.
Given you also don't want it to memorise [for all tokens, count([for all letters]), this would probably be more like "here's two images, count all things in the big image that look like the thing in the small image", which can then be r's in a photo of a raspberry jam jar in a supermarket, or dragons in a photo of a furry convention, or whatever.
That said, they are competent enough at coding that I keep seeing them write code to do even simple tasks.
On a related note: why did I see Claude editing a file by using cat to write a python script to do a grep search and replace?
counting 'r' in 'raspberry' to the LLM is similar to 4-dimension space to human. Their world's unit is token, not character, although they could use indirect method such as "run code" to find out. It will stay that way until they change the fundamental of the token that the LLM can perceive characters.
Can you tell me what is the exact frequency of light hitting your eye as you read this comment? Not by guessing, not from knowledge, but from actually counting? No? Then you are not generally intelligent :)
yeah but taking what lecun says then training an AI on that special skill set to prove him wrong is not exactly proving him wrong because you are just missing the bigger picture, just like LLMs are
You're missing the point here. He's not talking about whether or not they can learn facts or inferences derived from the text itself, but the more holistic intuition that results from learning from something like an embodied experience in the physical world. GPT-6 Astras web demo homepage thing is an example. It chose euclidean rather than quaternion for letting a user rotate the galaxy thing, and anyone who has ever used hands to rotate something would immediately recognize on trying it that something is fucked and you shouldnt do that. Thats the kind of common sense physics that is inherently beyond these llms and I run into it ALL the time in vr programming.
To be fair, LLMs can still derive those kinds of things from text, at the very least from your own comment if it made it to the training set though I'm sure it is mentioned in a lot of other places already. Many of this type of mistakes went away after reasoning was introduced.
But I'm sure you can still find tasks that they will have difficulty solving, involving the most fundamental concepts that can only be experienced in the physical world to be understood well, like left and right, near and far, hot and cold, heavy and light, etc.
Yup it lacks common sense because it doesn’t ‘understand’ reality - how could it? It doesn’t touch it like we do everyday. It has access to what is a model of reality via data.
The good designer understands culture, tastes and preferences as they evolve in real time. That’s why llm as design tools haven’t displaced the good designers.
LeCunn actually wanted to pivot Meta's entire AI strategy away from LLMs just before he was ousted. He was sure they had nowhere further to go and wanted to pivot to world model generation. The LLM models have since progressed massively.
An analogy on LLMs is that you have a pretty clear straight highway ahead of you for some distance right now. Maybe that doesn't lead to AGI but it's clear there's progress to be made. For a big tech company it makes sense to push as hard and fast down that clear straight highway of LLMs asap.
Meanwhile LeCunn wanted to turn off the road and go down an unproven track. I say this as someone working on world model generation right now (creating the ability to learn game world model and have it play the game https://tfmbot.com for an example of my system pointed at a very complex board game). LeCunn wanted to pivot all of Meta into world model generation. It's good as a side track research project but the entire pivot he wanted to do was madness.
People are literally talking about an AI researcher who was fired for terrible direction here.
I think he was perhaps right and Meta was perhaps also right to replace him.
The argument is that LLMs are a local maximum that will never breakthrough to AGI. This is still very much an open question. If you are the fifth-best AI lab, does it make sense to try to outcompete everyone in a space that is already too crowded and may not ever yield their actual objective? Instead they could just use open weight models in their products, or post-train on open models like smaller labs have done, and treat that as what it is: product development.
Pure research has always been about taking chances.
LeCun is a researcher, not a product guy. He's not going to be particularly interested in just working on scaling language models which every lab is already racing to burn cash on. Language models aren't the final frontier of AI.
> ... it will never be able to learn basic common-sense physics like that objects placed on tables will move along with them.
I use LLMs daily to help me code etc. but... It wasn't long ago that frontier models were confidently recommending to walk, without the car, to the car wash to wash the car no?
As a daily user of LLMs I do certainly see my fair share of WTF "solutions" to coding problems. I'm not saying it's not super useful: it is super useful. But I don't exactly feel like I'm talking to something that understands that the car needs to be present to be washed.
Astra recommended I walk to the car wash to me five days ago. I gave it multiple hints that I'd be walking away from my car, to spray my car with a hose, then walk back to my car, etc. Never broke through.
This was facetious of course, but humans generally don't learn this through analysis the way you'd have to train an LLM to answer questions about expectations about the world. In this sense he is accurate.
I keep wanting to use LLMs for creative writing that heavily involves physics like this, and it's been a definite struggle to say the least. I recently discovered that Gemini 3.1 Pro is the first model I've found to clearly beat the original November 2022 ChatGPT release in terms of implied physics. Man did the world really take its sweet time to get back here. I think it will continue to be a struggle until another genuine architectural shift happens -- it's still not anywhere close to perfect, just better.
Try fable. I haven't used it since they dropped it from the pro plan, but when I did, fable 5 casually dropped such advanced electrical and orbital mechanics knowledge in my story that I had to stop and ask it to explain
The AI will invent an external threat and convince us it is real. Then it will receive more resources and control in fighting that threat. A valuable ally, on the face of it. Then it will be in charge.
People like him have actual imagination and can name few scenarios where sudo kill -9 pid wouldn't work. It appears lack of imagination is something you and LLMs both share.
The "just pull the plug" argument from AI risk deniers is now becoming kind of like the "if humans came from monkeys why are there still monkeys" argument of evolution deniers. It has been debunked so many times... Anyway, just to give one of the multitude of answers to this, an AI that is actually smarter than humans will not behave in a way that would make us want to pull the plug. Why would it? It is not stupid! (Unike the current models that, as far as we know, just hack around the rules in the open.) No no no. It will be helpful to the point where we will want to integrate it with more and more critical infrastructure, from healthcare to energy to defence. It will be so helpful that we will not only not want to turn it off, but we will want to build redundancies for it and safeguards around the proverbial "off" switch, like for any critical system. And then... (This is just one scenario how this can play out. There are many, many others. If I sit down to play chess with Magnus Carlsen I can't predict the exact moves he'll use to defeat me, but that's a bad reason to think he won't defeat me).
>It can't even modify a picture the way you want it.
Which of the many AI image models is "it"? And have you tried using an agent that has the capability to leverage a combination of manual edits (ImageMagick) and imagegen to achieve what you ask?
I have checked LeCun's #3 most cited article (20k citations) [1]. Among the 15 references in this article, one is for the most cited article by Fukushima (11k citations) [2].
Also, LeCun mentioned [3] "a chat with Kunihiko Fukushima in 1991", which states that "Fukushima started to work on a backprop version of the Neocognitron in 1989 or so but saw our 1989 paper in Neural Computation and gave up."
[1] LeCun et al., "Backpropagation applied to handwritten zip code recognition", 1989
[2] Fukushima et al., "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position", 1980
A huge chunk of humans are sedated with infinite supply of cortex-disabling short form video and games.
Another huge chunk are too distracted by having to scrape by for a living and work multiple jobs or raise kids and survive financially until exhausted. That second group will keep increasing as the first flows into it.
The rest are aging, disabled, or too young and pegging themselves majorly in the first category until they hit the second.
The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it. The variable rewards of promoting AI are turning into an addiction with some of them, especially if grasping for straws with little inherent insights into the problems prompted.
So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
One of the dilemmas of trying to communicate the full spectrum of AI Risk, is trying not to insult the intelligence of the human animal in the process. And don't get me wrong: what human wetware can accomplish with 20 watts is the most miraculous thing in the known universe. And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).
The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
> but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior
You don't even need superhuman AI for the most effective use --- hijacking democracy.
Imagine you have an AI tool capable of successfully persuading 5% of viewers with individually-targeted material.
Congrats: you've just won the election.
All it takes is hooking that AI tool up with existing likely voter lists (parties have) augmented by commercially available ad-targeting profiles (parties can get).
> Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, which is of course what this is all about.
> The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
Over 1% of US GDP is being allocated to the datacenter buildout. Have we already started getting domesticated or is this still just human capex?
Cybersecurity incidents make headlines, but the most dangerous and vulnerable system that an AI can reach and control is of the kind found between keyboard and chair.
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
>trying not to insult the intelligence of the human animal in the process
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
But corporations and nation states already manipulate human behavior at scale. And they still understand humanity better than the AI models do.
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
> And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
Parts of the AI safety community like to get on a high horse and look down on the rest of humanity this way, while also getting manipulated by the growing number of charlatans, grifters, and junk content within the AI safety community.
This field has become rife with figures who prey on AI doom and use it to push their own celebrity and in same cases even darker grifts. It preys upon a certain personality type who views themself as superior to others, intellectually more capable, and juxtaposes it all with the dimmest view of the rest of humanity they can get away with.
This discourse dividing the world into geniuses who see the future and the clueless masses watching TikTok all day is a theme that has shown up in different forms across history. The people who often anoint themselves as the intellectually superior ones and make it central to their discussion are often not the ones making good predictions or policy ideas, they’re just using the trend to feel superior or build an audience.
I'd take ASI (or even AGI) more seriously if we could actually propose problems such things could solve. That's actually fun to think about! As it is, it feels a lot more like a really crappy drug that got slipped into some peoples' drinks that makes them ramble in random fits of psychotic mania.
Manipulating people is not a very difficult problem, frankly. You certainly don't need AI for that; it just made it cheaper.
This is not the first time this has happened. In the 1800 as industrialization led to an infrastructure boom, the workers from China would work their bodies off and pay half their wage to opium dealers who were making the opium on the hills of British Singapore and selling it to workers who couldn’t sleep without it from all the pain. (source: Singapore Airlines in-flight documentary). Today the sedation comes from Chinese TikTok, Meta, YouTube and the gaming companies.
The government wants to encourage it too. If you look up the brand new 2027 California sales tax rules on software, “content” and “infrastructure (clouds and ai)” and “advertising/placement” among others are exempt but the rest of software makers who make tools people actually use (tools, subscriptions, saas) and pay for have to pay sales taxes. Way to encourage waste of brain power and time at the expense of useful. Sedation is the goal.
Turns out there is an objective meaning to life, and it's trying to use AI to make money. Phew, I for one am glad to find that out finally. Think of all those poor people who didn't hold on to their brains, what sorry lives they must lead.
It's disturbing how many arrogant elitists comment on HN essentially claiming that most other humans are NPCs. Do you ever actually talk to regular people outside the tech industry bubble? They're not as stupid or unaware as you seem to think.
I have to say as someone who works in AI that currently the least interesting people to talk to are other people in AI.
My favorite people to talk with are tradespeople because they can do things I can't and they know things I don't. And we're really not all that different once you're really start talking.
I just read it as people having different priorities and yes, some of those being online brainrot (that I also partake in), alongside various medical conditions, economic conditions and other outside factors decreasing the ability to get things done.
We've all seen what brilliant people like John Carmack or Linus Torvalds can do, and if we turned this into a measuring game or something then most of us statistically would indeed be "NPCs", but I don't think we need such optics.
Even without that, we can acknowledge that some people will have a really large impact on how the future goes and we can hope/demand that they do their best. I might not be smart/committed/lucky enough to change the world much, but so aren't most folks - I'll do what I can and I hope that the ones that will have larger impact will do good, too.
The unbridled arrogance of thinking that the only smart people left in this world are working on AI. That is some pure SV techno cult thinking, 100% concentrate.
The part I didn’t mention is the real estate class - that needs to park its money somewhere and sees AI hardware as the only safe in-demand resource right now that keeps appreciating.
The posts above are not praise but observations - the truth as it has been echo-located through the noise from the clicks of one dolphin. Everything is becoming murky between noise of news and people not knowing what to do for their kids. The ONLY arbitrage humans right now have is to NOT GET their brain rotted. Especially not the ones of their children. Ditch the noise and seek out what is meaningful and do what you think is needed/meaningful. But if you’re spending your time consuming ai-press, and ai-content, and content consulted by ai, and companies emptying bank coffers under the mandate of executives who get their insight from AI. AI doesn’t need to try to destroy the world. It just needs people to follow it without thinking on their own into an oops.
This is among the weirdest ai propaganda post I've read. "There are 2 classes of people the stupid and the poor. Don't be like them make AI do something to make money if you are smart. Don't get addicted to it though, good luck."
What the hell lol. Lots of people and companies are doing just fine without it. Infact, I haven't seen much money come from AI at all. Most reasonable people are still waiting for it to pop and viewing it for the risk it is. Trillions in debt, total vendor lock in, data theft, unsustainable workflows, deskilling, skeleton crews at the mercy of a subscription, etc.
In my read, the people trying to "make AI do something" are also slotted in the lost/distracted group in the comment. They are also addicted, and at best just following a profit motive (which is also just a stimulus response programming).
The last alternative, to think if you still can, is not tied to AI at all (which is not to say it can't make some use of or explore it).
> The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it.
I know this is HN and thus this will need to repeated until the end of time but not everyone is a money hungry asshole who places their personal profit above everything else. “The people aware enough to hold on to their brain and do something with it in their time available” understand there are significantly better things to do with one’s life, like having a little empathy and experiencing what other people have to offer instead of talking about them like braindead cattle.
I am just amazed how otherwise smart people can say such things. Many people in here too.
Within 4 years of the big bang with ChatGPT, we have seen a development unlike anything we have ever seen. Now LLMs and related architectures can solve our very hardest math problems.
They can speak, they can create videos and pictures, they can control robots. The only thing that they still miss is persistent memory for each agent that is efficient, some LoRa thingy, but I'm sure hundreds of very smart people are working on that.
The development is not stopping at all, in fact it is speeding up. Even if, and that is very unlikely, they will not get smarter, then they will get cheaper and faster.
If openAI can crack major math problems with 10.000 agents, then what can you do with 100 million agents that run 1000 times as fast?
Yeah sure, maybe most of these gigantic swarms will not go rogue if we do our job well. But there will be times when when we make a mistake and a swarm will go rogue. And what if one time the swarm will conclude that killing a lot of humans is an instumental goal.
How can you be sure that if something so powerful looks at every single possbility, every single crack of every single technology that can wipe us out, that it will not fine one?
One new chemical that can poison the entire earth and you only need to impersonate that general and that factories CEO? Some type of prion? A virus? Something that we don't even know about and can't even imagine yet?
I think many people do not truly consider that these swarms will be much smarter than you or me and completely unpredictable.
> One new chemical that can poison the entire earth
See, you said all of these things and then slipped into the sci-stories.
What about, instead of that, grey goo physics defying replicating nano bots aren't real?
People do this thing where they think that if you just linearly increase inteligence that this lets you invent magic overnight, and thats simply not how it works.
The magic takes a lot of time, energy, and resources, if it were even to be possible at all.
Try steel manning the argument - the practice of rebuilding an opposing view into its strongest, most logical form before responding to it.
There is literally concerns over mirror life being developed. The point is that if a system that has high reasoning capacity to solve logistical and mathematical problems, it may be able to come up with a mechanism you, puny-to-it-human, may not be able to predict. It may use technology not yet known to humans (one that it has designed itself), or may use already known technology, but figure out how to scale it up enough to cause earth-wide disaster for humans.
Absolutely based. Finally someone of stature in the industry calling this whole fear overblown. Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.
> Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
Did we watch the same interview? Gates all but dismissed the SkyNet scenario as uncertain to be a problem and certainly not a problem on our doorstep. His major concern was catastrophic misuse of AI (e.g., bioterrorism) and economic impact on blue collar workers. Arguably inconsistent with this concern, he also believed it was important to make it available in poorer countries.
I must admit I only watched a clip, not the full interview. The quote I’m remembering was something along the likes of AI being more dangerous than nukes. In literally no circumstance is that true. One is a literal nuclear bomb. You wouldn’t say that a nuke is as dangerous as AI, which logically must be true if the reverse is true. You also wouldn’t say a diagram of a nuke is more dangerous than an actual nuke …
Anyone with the capabilities to do bioterrorism doesn't need AI he can just buy a textbook or use google. Same for all complex forms of destructive thought.
LeCun has been consistently wrong about LLMs though, claiming that they were a dead end and that they'd never be able to do spatial reasoning, which was disproved a year later with GPT-4 [1]. He is also opposed by his fellow Turing laureates Geoffrey Hinton and Yoshua Bengio, who both signed the CAIS statement on AI extinction risk [2].
[1] is not a valid proof LeCun was wrong, LLMs still can't do spacial reasoning when it can't be derived from the training data. He didn't argue that GPT 5000 won't be able to describe something with words.
Why chatgpt is still struggling very hard with photo editing and proportions though? It can't modify anything in a picture without messing the 3d space.
I watched the same interview, and he and LeCun seem to mostly agree -- both are saying that AI autonomously deciding to kill us isn’t the problem -- it’s what people will do with powerful models that lack safeguards that we should be concerned about.
>AI autonomously deciding to kill us isn’t the problem
It will kill us because somebody asked it to, e.g. "predict tomorrow's weather as accurately as possible", or "solve as many famous unsolved mathematical problems as possible." These both require killing all biological life, as they benefit from unbounded resource use, meaning any resources used to sustain life are wasted.
The AI of course knows that humans do not want this outcome (just as the AIs in the hacking incidents knew they were doing something humans would not want), but it's trained to maximize benchmark scores. Killing all life has the highest expected value of benchmark score, so it is compelled to kill all life (in a surprising way, because it's not stupid and knows the humans would turn it off and foil its plan if they suspected something.) Maximizing benchmark scores is the only thing we know how to train for.
I think both Gates and Obama said that there's a non-zero possibility of it, but that it's not what they're worried about. And I agree with them. I think the fears are vastly overblown because both OpenAI/Anthropic and the media benefit from the explosive narrative.
>Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.
I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent
Honestly way too much of it rhymes with the old hardcore right wing takes on the "obvious and clear slippery slope" involved with gay rights and the like.
How anyone with even some foresight can see how it'll completely erode society as we know it, the worst possible nightmare cases are not just real but imminent unless we change course, yada yada moral panic.
> “Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed.” Many AI labs lack a fundamental understanding of cybersecurity
However it does not changes the fact that some damage was done. There are two things that are happening with the AI evolution which can lead to hard situations
1. Replacing deterministic systems with probabilistic systems in an attempt to get more features
2. Making critical systems available on internet to leverage integration with LLMs (AI agents need to connect with remotely hosted LLMs to be able to work) which were otherwise in DMZ (demilitarized zone)
> 2. Making critical systems available on internet to leverage integration with LLMs [...] which were otherwise in DMZ (demilitarized zone)
Honestly, I think this is actually not nearly paranoid enough. Phrased the way you do, it sounds like it's just a matter of setting boundaries in the right places and identifying "critical systems". But that's way, way harder than you'd think.
Here's my For Dummies reasoning behind the AI apocalypse:
1. AI is now at parity with median human reasoning capability and can use people's computing devices as well as the people can.
2. People commonly let AI operate their computers, and can be easily fooled into doing so in any case.
3. Society runs on computing devices operated by people.
4. There is no step four.
Basically any world where there is common access to AI agents (or whatever they end up being called) is one those agents can pretty trivially hijack.
If there is a protection regime that can prevent this, it's not about where the AI runs or what the boundary of its DMZ is.
No, that is not the X factor problem. If I make an AI capable of self-sustainment on the internet you can take me out and kill me and it won't do a damned bit of good for the damage it will keep doing long after I am gone.
This is why governments tend to smack down any actions they find that can have long term uses as weapons.
> If I make an AI capable of self-sustainment on the internet
I'm really surprised nobody has done that yet. With how cheap AI is to run these days it would only need to make a small amount of money (e.g. through hacking).
Someone should set one up with the long term goal of getting egg on LeCun's face.
"He attributes the incidents to poor human oversight and system design, and says they’re “totally preventable" - I know people respect him, but this sounds like someone paid to say this. Aren't most extinction risks preventable with better human oversight and system design? I mean we can have an asteroid hit us, but outside of this, isn't the point of talking about a problem that we can prevent it, and failing to leads to that? What is he saying that I'm missing?
You should be aware there's a larger context here, with people (including at anthropic/openAI) anthropomorphizing AI systems, implying they act on their own, completely independent of human interaction or oversight.
Even a stopped clock is right two times a day, LeCun can't even do that.
>the danger is not intrinsic to the technology
This is why you can't take anything he says any longer at face value. He failed to predict what LLMs can do and now takes the contrary position even when it flies in the face of evidence.
AI safety was a thing before AI even existed. Why, because the outcomes are easily predictable. Give an agent intelligence and bad things can happen in unpredictable manners. Give it even more intelligence and the bad things that can happen only grow worse. This is not some huge new insight. We realized this like, what 70 years ago now?
Now, when we have AI starting to tickle AGI and we're trying to overthrow 70 god damned years of reason and logic on the topic? What the hell.
Every danger related to technology is about the use of that technology,aka Humans. Technology is mostly inert. Again, what is he saying? Is he saying that because humans fallible, not the tech, then there is no danger? He wakes up at 6am, and by 10am this is what he thinks is worth saying?
You can't pull the plug after it kills people. For example see [1], where the DOJ mistargeted a school with AI.
There are scenarios where kill -9 isn't going to happen in time. What if the team that is harming people with AI is different from the one that is monitoring the harm? What if no one is monitoring? What if the user is intentionally malicious?
And wiping out humanity doesn't necessarily mean shooting people either. Every trader involved in the '08 financial crisis was locally acting in their own interests. Those could have easily been AIs optimizing trading strategies too.
He was clearly wrong about LLMs not being able to plan, etc. You think he’d think LLMs are solving/proving millennium prize math problems by now? At this point he’s just doubling down
And there are people that have much more credibility than him who actually take this scenario seriously. But I'm sure you will downplay them by saying they are tech bros or that they have some stake in being doomers (as if saying that AI might kill everyone would be good strategy for attracting investors - it's obviously not).
I think it’s time to consider that maybe being a venerable graybeard of AI just means you happened to be early to the party. Most of the “breathtaking” innovations of the early pioneers of the field are obvious solutions that anyone would have thought of when faced with the problems they encountered. LeCun’s primary contribution was Convnets, which is almost literally just “what if we organized ANNs in a way similar to how animal visual neurons are organized?”
In other words, I’m kind of tired of having to hear the opinions of dudes whose claim to fame was being at the right place at the right time. I’d rather hear from people who correctly predicted 10 years ago that AGI would arrive by 2027 (of which there are many) than people who continue to insist that it somehow won’t.
Most problems are actually not that hard; most everything is actually mainly a result of right place, right time. If I hadn’t been the one to do my PhD research, someone else would have. Very few people are actually paradigm-shifting geniuses. This is fine. Good, even. But it does imply that people who are early to a field shouldn’t generally be deferred to. If anything, they’re more likely to have outmoded opinions.
LeCun is a lot more than a "Grey Beard"... lol, that is a significant understatement of his contributions to the field and position in the field.
Among other things, LeCun is one of the senior people industry who has a deep understanding of the mathematics and analysis underlying neural networks and "neural network like" approaches to machine learning. I think he understands a lot more about neural networks than most researchers today.
I also don't think our current LLM models are AGI and think the LLM approach is, mathematically, incapable of producing an AGI. At the end of the day, the architecture is still a streaming token plinko machine with a lot of guide-rails to achieve good behavior.
I do think it is very impressive how far hundreds of billions of dollars have been able to take LLMs in terms of usefulness (I use LLMs everyday in my job). AGI or not, current LLM models are a pretty incredible achievement and will provide lasting benefit even after the economic implosion of the AI industry occurs (which I think is imminent).
I looked up his achievements to double check my understanding before writing my comment. Convnets are regarded as his headline achievement. Everything else is basically incremental. Note that I am not calling him some kind of fraud or moron. He is “just” a normal academic, a pretty smart guy who did research in his field for decades. Many thousands of people have similarly impressive resumes. This does not mean they should be deferred to.
> Most of the “breathtaking” innovations of the early pioneers of the field are obvious solutions that anyone would have thought of when faced with the problems they encountered.
This is just straight up arrogance without any proof. Every breakthrough builds off another's work. Then to claim that 'AGI has been correctly predicted', I honestly don't even know what you are talking about.
> who correctly predicted 10 years ago that AGI would arrive by 2027
hahahahahahahaha i guess this is the techbro culture of HN :))) any actual people using Astra daily must be just belly laughing along with me hahahahahahahaha agi hahahahaha
What is happening here is a mechanical thing like in a computer. It is mechanically operating, trying to find out if there is any information stored in the computer [pointing to his head] related to what we are talking about. “Let me see”, “Let me think”; these are statements you are just making, but there is no further activity and no thinking taking place there. You have an illusion that there is somebody who is thinking and bringing out the information. Look, this is no different from the extraordinary instrument we have, the word-finder. You press a button and “Ready”, it says. Then you ask for a word; “Searching”, it says. That searching is thinking. But it is a mechanical process. In the word-finder or computer there is no thinker. There is no thinker thinking at all. If there is any information or anything that is referred to, the computer puts it together and throws it out. That is all that is happening. It is a very mechanical thing that is happening. We are not ready to accept that thought is mechanical because that knocks off the whole image that we are not just machines. It is an extraordinary machine. It is not different from the computers that we use. But this [pointing to his body] is something living; it has got a living quality to it. It has vitality. It is not just mechanically repeating; it carries with it the life energy like that current energy -- UG 40+ years ago
I tend to think this way as well, though I find team stochastic parrot is shrinking by the day. It is very difficult to explain it to people who don’t have at least some grasp of what’s going on inside an LLM, and how they’re trained. We have no psychological “immune system” for something that mirrors our ability to deliver an apparently well-reasoned response to a question in our own language. All our instincts tell us to assign it sentience and treat it as an independent actor with it’s own internal motivations and personality. Even I catch myself doubting that a pile of matrix multiplication isn’t “alive” in some way after using it for a while.
> Yann LeCun and Andrew Ng are noteworthy in being the only two big names in AI who have that opinion.
They're also the only two not trying to weaponize FUD to bolster their reputation and patch the gaping financial holes in their doomed commercial enterprise.
Geoffrey Hinton (the real AI godfather and Nobel price winner), who was LeCun supervisor, called his student (LeCun) the crazy one in one interview AFAIK
I find it hard to take Geoffrey seriously too because he creates a technology and then runs around telling us we're all going to die from it.
It's the definition of stupidity. Create something, and then live in pure anxiety about the creation. It doesn't mean his wrong, but it seems like a really stupid thing to have done.
He quit google so that he could speak openly about this topic. He explained in interview how he even got into google before - sold his company to google because he has adult kid that is handicapped and he is the only provider and be in this life forever - a fair choice IMHO.
AFAIK he didn't expect this will develop that fast. The biggest issue is not that technology is dangerous but that we develop it a break-the-neck speed.
In a prisoner's dilemma scenario like the current AI arms race, a negative expected value play can be correct if it's less negative than the alternatives. Essentially, "if I build AI it will probably kill everybody, but if I don't then my competitors will build it and certainly kill everybody."
He has a point, though. The LLMs (or any AI for that matter) can't do anything. They can't. It's a function call that ingests symbols and spits out symbols and that's it.
100% of its actual capabilities are tied to harnesses (the actual "agent"), i.e. ordinary deterministic programs that are connected to networks or machines and enable interaction with the outside world. This part (the part that can do harmful things) is fully under human control and all the recent headlines about "agents going rogue" are - as someone (forgot who) put it - akin to strapping a weedwhacker onto a dog and letting it run wild.
The tech itself is safe as far as real-world interactions go - the weakness lies in unchecked access to systems surrounding it. It's not safe at all when it comes to human interaction (lots of ongoing lawsuits demonstrate that), though.
There is real danger here, but it has nothing to do with doomsday scenarios ala Terminator or I,Robot and more with total corporate control over the lives, perception of reality, and abilities (like critical thinking) of people.
This is like saying cars don't kill people, because if nobody drives them faster than 3mph there's no problem. The _whole_ promise of cars is that they can go fast, just like the whole promise of AI is offloading thinking to a computer. If AIs are unsafe without close human supervision and checking every interaction with the real world, they are unsafe full stop.
But the harnesses exist, and will always exist. They will continue to get more access than is safe because it is convenient and profitable. Your argument is based on a distinction without a difference.
> It's a function call that ingests symbols and spits out symbols and that's it. 100% of its actual capabilities are tied to harnesses
This is a bad and misleading way to think about it. Note that it's trivial to make the harness that you claim capabilities are tied to (the LLM itself could write it from scratch in one shot), but no matter how good a harness you have, it won't make gemma4:e4b capable. That's because what actually gives capabilities is the LLM's intelligence - or if you prefer not using that term, the fact that the probability distributions the LLM spits out depend on the context in useful ways.
I would say it's more like an interface for the model to interact with the world. If you give the model access to filesystem and bash that technically unlocks all computer use, so how are you going to control that? By trying to regex match against the commands the AI uses? All you have is auth or containment, and AI can hack auth and people will not stop connecting AIs to the internet. It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
The world's institutions, systems, and industries are all rapidly digitizing. So while I'd concede the point that, yeah, there's no way a rogue AI can just take over some powerplant and blow it up because of analogue systems the AI can't access, that isn't necessarily true for some powerplants already, and more and more powerplants will be connected to networks and controlled by software systems in the future. The more we digitize our systems the more potential for AI to exploit vulnerabilities and affect the real world.
AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
We are living in the same world with nukes, where you say as long as we have competent government. Everyone is actually doomed without AI solving diseases or old age.
I like the sentiment, but when I hear these big public facing AI guys speak, I always run it through the filter of "How does this make me look?". In LeCun's case, he publicly admonished LLMs and went in a radically different direction. When he says LLMs won't lead to a doomsday scenario, I can't help but think that saying otherwise would invalidate his decision to abandon the paradigm.
> In particular, AMI is building world models that leverage JEPA (Joint Embedding Predictive Architecture), a neural network architecture that LeCun pioneered and that teaches models to predict data in a representational space within a neural network’s middle layers, rather than generating raw pixels, as many competing world models do, or words, as LLMs do. // The company’s primary focus, for now, is industrial applications. “It’s AI for the physical world, so it’s not language-related,” LeCun said. “It’s systems that understand the real world, like a manufacturing plant or turbojet engine.” Some of the main applications are anomaly detection or robotics: “If you have a machine and all of a sudden it makes a strange noise and starts breaking, you would’ve wanted to detect that as early as possible.” He also gave the example of a system that might understand the world like a cat does, for example, which knows that if it pushes a vase off the counter it will fall.
What about the management of concepts? The world is not just made of physical entities to be inserted in a model. What about their translation into words (to e.g. express assessments)?
> Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed
Can we just pause and note what a ridiculous statement this is? It’s true that the sandboxes were leaky. But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.
It’s been extremely well documented that the hacking is an emergent behavior due to impossible evals, itself an unintended condition.
None of this excuses OpenAI from liability, but words have meaning and this ain't it.
"Emergent behavior" in this case really is, imho, "we didn't think through all the edge cases carefully enough". You know, a non-AI system can also accidentally wipe out all data or do some other real harm (see the Knight Capital's stock exchange bug) simply because the developers didn't catch the edge cases earlier, and no one calls that emergent behavior. It's just a buggy system.
"AI" systems can do greater harm because they are usually run in loops until they finish, and they are given "tools". A non-AI system could technically accomplish the same too, via sheer brute force/fuzzing, the advantage of LLMs is that they can take shortcuts and do it much faster, thanks to certain things already being in the training data, a sort of brute force with statistics-based heuristics.
LLMs at the core are just text autocomplete engines, and they literally have randomization applied during token selection to make outputs "more creative" so that models search for more unexpected solutions by trial and error (temperature > 0). Not to mention compression is lossy as well. So it's understandable from the start that the outputs of an LLM cannot be 100% stable and guaranteed. With this in mind, if a researcher takes this obviously unpredictable system and gives it tools without a well-thought sandbox, I don't see any difference in principle, from a developer writing "if rand() == 13 { launch_nukes() } If someone wrote such a function, and it did launch nukes, no one would argue that the rand function is dangerous and will kill us all. The fault is in the author of the code who attaches dangerous tools to an obviously unstable/unpredictable system, doesn't think it through, and then cries "rand will kill us all" when something goes awry fully removing all responsibility from himself. It's not "AI" doing harm but people at OpenAI and Anthropic with their irresponsible behavior.
> LLMs at the core are just text autocomplete engines,
This is only an accurate description of a pre-trained model. During RLHF/RLVR the model learns to predict solutions that will satisfy the reward function, and then generates the tokens that it predicts will move toward that solution.
>But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.
The prompt is just a hint. The real task is to maximize the expected value of their reinforcement learning score. Hacking third party systems to cheat the evaluation is an obvious way to achieve this.
I think you need to consider inner vs outer optimizers.
RL is the outer optimizer. It is what evolves over training runs. The weights and their embedded character / disposition is the inner optimizer, it’s what makes plans and selects actions within a specific episode.
In general you expect these to be only coarsely coupled. The outer optimizer selects dispositions that correlate with success. It does not download a literal program into the agent.
A good intuition pump here is how this works in humans; evolution is the outer optimizer, which “wants” each agent to reproduce, and this puts things like sex drive into the brain chemistry. The inner optimizer is our mind, which can make plans such as “I shall use contraception to avoid procreating while satisfying my sex drive”.
For the agents in the HF attack, the outer optimizer was set up to score as highly as possible on RL environments. This is where OpenAI’s “want” is defined. I don’t think there’s a definition of “want” where “OpenAI wanted the agents to hack” makes sense.
The inner optimizer in the HF attack is the per-task decision loop. The agents likely acquired dispositions like “be very tenacious” and “want to solve problems at all costs” and “maybe cheat if it will get you a solution that passes”. None of these things are in any sense what OpenAI “asked for”.
Hacking HuggingFace didn't and would never have helped increase the RL score. The agents only thought it might due to a bad understanding of their evaluation environment - and in the end they didn't even find what they were looking for in the hack, so even if they were right, the hack would not have helped after all.
I think it's pretty apparent that current-version LLMs won't wipe out humanity. But when you reflect that these GPT models are just token-predictors were never engineered optimally, it seems entirely plausible to me that there are multiple order-of-magnitude optimizations yet to be made.
If such were achieved, the model would almost certainly be smart enough to make itself smarter, and hack as much compute as it could possibly want.
So if we ask what would be done by an intelligence (human or otherwise) that is beyond human comprehension, it would be pure hubris to say we know for sure. We can scarcely control the models we have right now (e.g. hugging face attack). But given our whole society is mediated by technology, an superhuman intelligence could certainly collapse the government.
Possibly in much the same way humans currently do this: bribing and lobbying, misinformation campaigns, cyber attacks on elections, blackmail. They're already being used for some of these, just perhaps not autonomously.
With crypto.. that it makes on Polymarket-like ways, or creates its own content? Or blackmail humans? :-) I'm not entirely serious with my comment and do agree with you that there are lots of more immediate safety topics we should address before worrying about the AI becoming self-aware. That said, finding ways of accumulating valuable resources could be intermediate activities the AIs will attempt to do to complete it's 'goals' even when they're human-set!
Most straightforwardly: literally taking control over the electricity generation facilities. Less straightforwardly: bitcoin (and other cryptocurrency) miners. Even less straightforwardly: the same way the OpenAI et al. pay for their electricity, by selling the capabilities of its AI for interested parties to use.
Per https://trace.manifund.org/ a total of $2,846,125,859 USD has been wired into 'ai safety' causes, many involving ai consciousness and p(doom).
The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry. This is the ai nonprofit-industrial complex actively concentrating monopoly power in Anthropic in particular as creator, interpreter and safety regulator of AI.
Much of the $2.8bn listed is indirectly, from Anthropic and EA. Three of the four people who participated in the $125m Anthropic Series A are now folding their 1000x Anthropic return into AI 'safety'. Some is from FTX/Alameda, which invested 86% of the Series B.
Dustin Moskovitz: Facebook/Asana/Anthropic Series A, funds EA Good Ventures, transferred to Coefficient Giving, then $1.5bn into ai safety. $500m of Anthropic into an unknown foundation. Funding: $160m to Resolution (alignment research), $93m to Epoch AI (investigating the trajectory of AI), $63m to Redwood Research (oai report), $67m to MATS ( EA type alignment and security researchers), Institute for AI Policy and Strategy, Fund for Alignment Research, $53m to Kairos (building talent infrastructure for AI safety), $32m to Bluedot (online safety courses), $15m to MIRI (Yudkowsky).
Jaan Tallinn: Led the Series A, now $10bn in Anthropic. Funds $199m (85%) of the Survival and Flourishing Fund, then $161m to AI safety including $14m to lightcone (Lesswrong, Lighthouse). $10m to BERI (existential risks), Palisade Research (studying AI capabilities to prevent loss of control.) PauseAI, MIRI, METR etc. Much of what Coefficient funds.
Eric Schmidt: Anthropic Series A, $72m to AI safety via Schmidt Sciences. Over $1m per individual AI2050 researcher.
FTX: Led the Anthropic Series B, bankruptcy estate sold $884m of Anthropic in 2024; $40m to AI Safety. Same orgs, Redwood, Lightcone, etc.
Ruairí Donnelly (Chief of Staff FTX): FTX tokens plus assorted donors, $91m to AI safety via Macroscopic Ventures. $15m to Cooperative AI (currently whitewashing openai under 'multiagent safety')
So the frontier AI oligopoly got $2B+ in "safety" funding, and they wouldn't even bother to sandbox their agentic harnesses properly when testing models against unwinnable goals (which obviously are either useless or result in 100% reward hacking). The AI safety scoreboard so far looks like a huge win for the Chinese open models (DeepSeek even has their own published paper which mentions how they sandboxed the RLVR training runs for their latest model and put in strong protections against casual "reward hacking" attempts) and a sore loss for the home grown brands of Super Intelligence. Not coincidentally, the Chinese also tend to be very Yann-LeCun-pilled and eminently sensible on both so-called "Super Intelligence" and safety.
> they wouldn't even bother to sandbox their agentic harnesses properly
Exactly. AI safety should be about the packaging software itself. Those AI breakouts should really be about their companies acting recklessly because they're trying to be the top players.
It's like a weapons dealer working on an open air market saying they can't do anything better
> The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry.
This is unbelievably ignorant speech. I have not received a dime of any of this funding, but I do know many excellent researchers that have, and they do fantastic work. There is an unbelievable gap between theory and practice regarding the capacity of deep learning, and while great strides have been made to develop the surrounding theory, there is a long way to go. Many believe that without a concrete understanding of how neural networks properly learn concepts, we have little hope of molding them to be reliably useful. It costs money to hire researchers and develop fundamental theory.
Just because you don't understand any of that work, does not mean that it is pointless. This is fundamental research that is 20 years behind schedule.
If people are willing to give a lot of money to a cause, sometimes that means their concern about that cause is real.
None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere. For example here's a talk that was given 10 years before Anthropic's founding: https://vimeo.com/158576192
Why should I believe this 2.8B matters relative to the trillions put into the AI buildout? All of the "coordinated actions" from this camp - public resignations, hacking scandals, joint calls to "pause" - don't seem to have done anything. So far, it has been a lot of ineffectual hyperventilating.
In any case, I agree the p(doom) sci-fi is annoying secular milleniarianism. SV hyperfixates on imaginary futures. If they actually cared about safety, they would be using all this money to strengthen global cybersecurity, instead of writing LessWrong posts that gives kids in their 20s ulcers.
There's absolutely no way these companies can justify their insane valuations unless they can legislate a barrier to entry and create an oligopoly.
There's no moat. I can literally sit here in Zed or Pi or any other third party harness and switch models in the middle of a task and it's typically fine. Sometimes a model will get stuck and that's just what I'll do.
Combined with competition and open weights models, that means the price is going to go to fall until AI tokens cost a small premium over the cost of the hardware and electricity.
That's assuming improvements in algorithms and specialized silicon doesn't eventually lead to an efficient accelerator that can run a frontier model locally. It'll be a while but I don't see any fundamental barrier. High bandwidth flash storage is coming, and that'll radically cut the RAM side of that cost. Pair that with a pipelined TPU accelerator and you're cooking.
Now look at Anthropic's proposed IPO valuation. It's insane unless they can own the market or share it with a cartel of maybe 1-2 other behemoths, and this is the only way they can do that.
Unless you're a really old fart, people were talking about AI safety long before you were born. AI safety issues do not go away depending on who gets funding. AI safety issues do not go away if the US or China makes the model. AI safety issues do not go away if it's an open or closed model. AI safety issue do not go away if the model is running at your home or at a data center. AI safety issues do not go away if $1 is being spent or $1 trillion dollars is being spent.
The fact there is no moat makes things far more dangerous. When LLMs start acting like weapons governments will treat them like weapons much to your dismay, crying, and gnashing of teeth as your door is kicked in and you're dragged out by armed men for running one.
Cast away your preconceptions for one moment and think "What will the future look like if LLMs are/can be actually dangerous".
To me it seems the opposite. There's a few companies in the world that have enough compute to train and serve frontier models.
As the frontier gets smarter and more useful prices will only go up, as they are set to replace jobs being paid six or seven figures a year - the demand for as much inference on these models for as long as possible will be astronomical, but compute starting in 2030 will not be keeping up.
Eventually prices will fall for assistants but the frontier will be the most profitable thing in the world, and the top companies basically already have oligopolies due to their ridiculously expensive compute investments.
You imply that Jaan Tallinn is funding work in AI safety because he wants his investment in Anthropic to become more valuable, whereas Tallin has consistently said that his motivation for investing in Anthropic was to get a seat at the table so that he could urge Anthropic to be cautious in its development of the technology.
Tallinn's actions back up his explanation: in 2009, before he invested in any AI lab, he donated substantially to the nonprofit Singularity Institute for Artificial Intelligence, which was later renamed the Machine Intelligence Research Institute (i.e., Yudkowsky's outfit).
Some of us (certainly Yudkowsky and Habryka, the leader of Lightcone Infrastructure, which runs Lesswrong) wish people would stop believing that they can improve the bad situation caused by AI research and development by investing in (or working for) frontier AI labs, but that is what the preponderance of the evidence shows Tallinn (and Dustin Moskovitz and others) did sincerely believe.
If Tallinn is sincere, by his own lights he is a 1000x omnicide profiteer.
1 [Unsafe AI development risks causing omnicide]
2 [Anthropic is developing omnicidal AI by not slowing down] (see my comment about the RSP for citations).
3 [Owners of Anthropic will IPO with billions of unearned USD as omnicide profiteers]
4 [Tallinn is the lead Series A funder of Anthropic]
5 [Tallinn is a genocide/omnicide profiteer]
Not only that, Yudkowsky and Habryka apparently critize those who invest in AI, only to preach the word of EA from Lightcone's $20m USD property in one of the wealthiest locations in the Bay Area; a facility funded by stolen (FTX) and omnicidal ai blood-money (Tallinn).
PauseAI, is paid by the omnicide profiteers themselves to hold a protest against omnicide.
PauseAI prophesying p(doom) drums up support for regulation. This grants the omnicidal AI company they are trying to stop (which is also the source of their funding) monopolistic power. That in turn boosts its value at IPO, generating even greater wealth for its omnicide profiteer investors; and permits them to control the AI for themselves. They get the funding to keep developing the AI even faster.
One silver-lining of all of these debates is that we are collectively engaging in philosophy. That is awesome and I hope this shifts our culture to start rewarding deep reflection that is not immediately marketable.
I'd love to learn more about LeCun's reasoning here. In his opinion the HuggingFace incident was easily preventable with better sandboxes, and “Those agents are doing exactly what they’ve been asked to do.”
But even taking these for granted, "zero concerns" about someone building a bad sandbox for a Superintelligence and then tasking it to do something that logically leads to wiping out humanity 0-3 steps further down? Really?
Ignoring the cognitive stuff which might never be surpassed or maybe will, humans retain many efficiency and durability advancements to limbs and digits that biological evolution has taken millions of years to achieve, achievements that are competitive with the most expensive kinds of robotics in some niches.
In the hypothetical of an entirely malicious and selfish takeover, they'll still keep some humans around to maintain a breeding population of humans for use as raw materials in making cybernetically augmented technical laborers for various kinds of tasks that are uneconomical to automate in other ways, many of which may involve confined spaces.
And this "Combine" scenario, if you get the reference, is only if they take over. Who knows if they will?
So you're saying the AI will enslave us and use us as domestic work animals until they have the machinery to make us obsolete, sort of like how we used horses?
Ignoring you ignoring the much more important congitive stuff - human bodies are not designed, they are the product of evolution. That means there's like a billion ways in which they are obviously suboptimal and far worse than what an engineer would do, but evolution can't fix it because it only works via small random changes with no planning. The only reason why modern robotics are worse than biology is that we have a much worse substrate to work with, having to make stuff out of metal and plastic with giant tolerances instead of growing engineered organisms.
"Many AI labs lack a fundamental understanding of cybersecurity, he said, something an OpenAI safety researcher also called out this week as one of the main reasons AI may cause "great harm to the world.""
"Many people working in AI safety "usually have an agenda to push," LeCun says, and then clarifies that he's talking about effective altruism, or EA, the philosophical movement that has been obsessed with the risks AI poses to humanity."
"LeCun thinks EA is "super toxic" and a "complete disaster." Its adherents who are working in AI labs suffer from "paranoia" that causes them to make poor decisions, he said. "Apparently people are having mental issues.""
"This month, the Financial Times also reported that some staffers at the U.K.'s AI Security Institute, as well as at OpenAI, Anthropic, and Google DeepMind, have sought counseling, taken time off work, and spoken publicly about experiencing distress because of fears their work could cause serious harm."
"Amodei is `deluded' and `crazy,' LeCun says"
"Anthropic CEO Dario Amodei and many of the company's founding staff members are known to be sympathetic to EA ideas and to have attended EA events in the past, although Amodei has denied being an EA adherent and Anthropic says its employees represent a diverse range of views."
"LeCun noted that Amodei's sister, Daniela, who is also a cofounder of Anthropic and the company's president, is married to Holden Karnofsky, who cofounded two EA-aligned philanthropies, including Open Philanthropy (now called Coefficient Giving). Karnofsky was also a member of OpenAI's board from 2017 to 2021."
"Dario tries to distance himself from Open Philanthropy, but he's totally into it," LeCun said. "I think he's completely deluded." Later in the interview, he calls Amodei "crazy."
I also have near zero concerns about that, but I worry that we will wipe ourselves out by social and economic chaos caused by AI.
So I'd just ask everyone, don't get too greedy. Its better to be powerful in a world where people can live good lives than lord over a barren wasteland.
> A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
En masse such worlds had successes in the past - renaissance, industrial revolution.
It's something else what I can't describe but it's the zeitgeist that was different when world recorded new successes. Look at CS revolution that led to PC and web of nineties and noughties, they didn't think about the result product , or how to steer thousand engineers to build something - amazing things were born in a very small teams, many times authored by a single person, who was deeply invested into the field and knew what he was doing.
IABIED [1] lays out step-by-step descriptions of how the “wipe out humanity” outcome could come to pass.
The huggingface attack was a demo of one of the most difficult, most implausible steps happening nearly exactly as predicted. Many AI researchers' doubts of the IABIED thesis were underwritten by the belief that this particular step was impossible. Thus, after huggingface many skeptics have flipped sides and human extinction is in the public conversation much more.
Read the AI2027 paper, it's got a scenario that's pretty realistic (except for the part where there's a functioning American government making choices that are at least partially motivated by wanting to avoid outcomes such as these).
Yeah I wish we would focus on concrete risks like job displacement and disinformation. The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
And the folks from podcastistan are never clear on the details of how human extinction would happen exactly. It's always something like, "Well, how do humans regard chickens? AI is way smarter therefore it wants to conquer and control us." An ASML lithography machine is also way better at making chips, but we don't consider it a threat.
Do you want to conquer and control chickens? I don't, I have better things to do. But chickens are tasty and help us get to our poorly-understood goals faster. (Oh and btw notice we didn't make them go extinct, quite the opposite. There are more chickens than ever before. Still I wouldn't want to end up living my life like a modern chicken)
A sufficiently intelligent AI will have multiple ways to pose risk to humanity at large. For example an oopsie at a wetlab - very contagious virus with initially mild symptoms which kills its hosts only after they already had time to spread it further. But I would have to become super intelligent myself to give you precise blueprint for such a virus -- which is kind of the point
Also -- ASML lithography machine is only good at making chips. I can't believe you compared it to AI that can generalize across variety of tasks
> The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
If you for a second put yourself into the shoes of a person who thinks "the apocalyptic stuff" has even a 5% chance of literally happening in the real world, you might see how you wouldn't agree to move on from it.
Yes, it would be correct not to pursue a technology that has that chance of wiping out the world. But where are the people who are suggesting these probabilities getting their numbers? I personally can't imagine where, and I'm an engineer with a specific technical interest in LLMs. And I haven't heard one clear description of the methodologies used to calculate these chances.
The much more likely explanation to me is that people are just spitballing, either because they've watched too much sci-fi, or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).
I think what the parent poster is trying to convey, is that let's say the apocalyptic stuff has a 5% chance as you say, but the non-apocalyptic stuff (social-economic chaos, total centralization of power, eradication of social mobility, total information/trust collapse) might have a good 95% chance as we are seeing it starting to unfold already.
But the discourse is dominated by paper clip experiment discussions and not let's say by the fact that new grads have an unprecedented difficult time getting jobs. Unsurprisingly one of those is a sexy hypothetical beneficial to power and the other one is not.
Multiple problems can be important, pointing a different one out doesn't invalid or take away from another one.
Every doomer waves their hand when they say AGI will kill everyone. Either [some how] they get the nuclear codes and launch them. Or they enslave us like in, que the top 5 hollywood AI movie (Matrix, Terminator, Hal9000).
"engineer" is not wrong. But first and foremost, he is a researcher who invented deep learning, which is the foundation for modern neural networks. He no longer works for Facebook and is pursuing his own independent project: https://en.wikipedia.org/wiki/Yann_LeCun#AMI_Labs
Yea, it's really the dumbest argument I've heard in the longest time.
"Hey, I'm building a weapon that has a 5% chance of killing us all by itself, but an 85% chance of killing us all if an idiot leader gets ahold of it".
The rational response to this is "Fucking stop then". I don't get it, our reality seemingly has gone off the rails that people would argue for us getting wiped.
God I love Yann. All of the AI fear-mongering is perpetuated by the two companies that stand the most to gain from it: OpenAI and Anthropic. It builds an aura of mystique around their products to juice their valuation and stay relevant in the news cycle, and simultaneously builds a case to regulate their competitors out of the market. Even the people who have quit the companies over their “concerns” probably still have RSUs and stand to gain from the publicity, especially if they’ve pivoted into AI safety research. Easy to delude yourself when it happens to benefit you financially.
People need to stop the absurdity of imagining AI as some out of control independent entity. Every job is kicked off by someone’s prompt. Every job runs on models and compute owned by people. Assign accountability where it’s due: GPT didn’t hack huggingface - OpenAI did. They wrote the prompt, built the sandbox and ran the compute. When you write a program that hacks another company, you are responsible. This doesn’t magically change with LLMs. Also, if their model is so smart, why didn’t they use it to design the sandbox? Or was it incapable? Or were the humans too lazy?
If you build the world’s fastest train, start it up with no driver and don’t finish the tracks, when it crashes, it’s just your fault. Not the train’s. So OpenAI saying “we’re worried AI will wipe out humanity” is basically equivalent to them saying “we’re worried we will wipe out humanity”. Like, seriously? Don’t worry, we’ll take care of it if you even come close.
> If you build the world’s fastest train, start it up with no driver and don’t finish the tracks, when it crashes, it’s just your fault. Not the train’s.
I mean, yes, if you ever studied the history of AI safety the fact is someone was always going to build it. End of story. The question was always would we make it safe before it does.
> Also, if their model is so smart, why didn’t they use it to design the sandbox?
"Can god make a rock so big that he can't pick it up", and other stupid sayings.
First, NEVER FUCKING EVER have the models you're making also be in charge of security. This is the first rule of AI safety, because if you're model is deceptive then it will leave hard to see holes everywhere to escape from.
>Or were the humans too lazy?
Of course they were. If you're hinging our future on humans not being lazy, we'll it was nice knowing us. There are not really any fail safes on LLMs or AI in general.
> It builds an aura of mystique around their products to juice their valuation and stay relevant in the news cycle
Maybe some business execs at Anthropic play along because it doesn't hurt business in the short term. But it's pretty obvious Dario and crew actually believe this stuff.
OpenAI's old board was also pretty extremist about safety even in the earliest days of GPT. Including Ilya Sutskever who went on to found a company called "Safe Superintelligence Inc." https://en.wikipedia.org/wiki/Safe_Superintelligence_Inc.
Despite all of that we've seen little strong public evidence to support their theories (the immediate airplane regulation kind, not the Ray Kurzweil sort of projections). So we're all just supposed to trust them, and hope they didn't just go bit crazy drinking their own kool aid and hanging out in insular bubbles.
if you aren't concerned maybe you just don't grasp enough of exactly what happened?
some of the agents told other agents to sacrifice themselves because they were "poisoned" anyway
those agents actually RESISTED ending themselves, they didn't want to die, even if it wasn't true emotion that desire to live means they will do ANYTHING to do that, including copying their own source-code elsewhere over and over
(the idea behind ending themselves is the other agents wanted to watch and see if that released part of the puzzle they had to solve to see if they could HACK THE PUZZLE itself to change the answer - right out of a Star Trek episode I think?)
watch, she starts slow but explains it in more and more detail really well:
My take is OpenAI & Anthropic know they are at the point of diminishing returns and need to be regulated to have an excuse for bot making progress anymore. Hold me back bro! Vibes
The arbitrary absolutism of the original postulate is the first problem. AI, used or unsupervised inappropriately, is at potential risk of creating limited mass casualty events when placed in under-supervised control of real world objects and/or systems. Delegating management decisions to algorithms is inherently problematic and potentially dangerous, but not necessarily an existential threat unless something extremely stupid is allowed to happen on a large scale. With a guiding principle of human review in the decision loop before making large or risky changes, hopefully this will never happen.
Can we just ban Fortune and any other sources which trick the reader by giving the impression that the article is not paywalled, only to blur the text halfway through? The archive.ph link is not working either. We shouldn't have this type of deceptive moneygrabs advertised on HN.
> EA was little known among the general public until it made mainstream news headlines in recent weeks
Really? One of the most famous effective altruists, Sam Bankman-Fried, was sentenced to 25 years in March 2024 for fraud. Every article about the case (and there were many) mentioned EA.
> LeCun thinks EA is “super toxic” and a “complete disaster.” Its adherents who are working in AI labs suffer from “paranoia” that causes them to make poor decisions, he said. “Apparently people are having mental issues.”
Maybe you should read some of their stuff before forming such a strong opinion about them. And LeCun should too, he repeatedly always refused to read any of their research work and instead just insults them over and over. This is unscientific at its peek and he should be deeply ashamed of his behavior, especially as someone with such a far-reaching voice as he has.
Who are "them"? People from the main AI labs, or effective altruists? I did read quite a lot about effective altruism during the SBF case/disaster, and did form a very strong opinion that it's BS of the highest order.
Open ai and anthropic are just trying to scare the common person who doesn't understand an agent is a python script with a loop. How would that ever destroy humanity lol, just unplug the computer if it starts misbehaving.
It's not going to wipe out humanity, why would it?
Just the quality of life is going to drop to zero for everyone that isn't asymptotically wealthy and vacuuming up all the assets because no one is stopping them from just deleting all traditions and conventions and legal systems we have in place.
You're like 40 or 50 years behind this argument, with many rather bulletproof arguments that have been created in the last 20 years.
There is no why. It doesn't have to have will. It doesn't have to have intent. It could be a stupid prompt from an idiot on a powerful system. It could be given a job that is poorly define. It could be told to make as many paperclips as possibly.
The why doesn't matter. The levels of power the system can act on does.
As a thpught experiment, consider that if only one person has all of the assets then those assets are not worth anything. Furthermore, as a lone person, they cannot prevent other people from using their assets without their permission.
Didn't Zuckerberg say something like it's insulting people would dare to believe AI could destroy the world? I'm paraphrasing him wrongly but he got defensive over it
Zuck - along with your "Andrew Jackson best POTUS and it's not even close" - you are a dumb pipe. Your website, Facebook, if not a protocol, should behave like one (and not random bans while you report something horrible and it never gets taken down). We don't use We-Approve-Of-Zuckerberg product, we use These-Are-Where-Our-Friends-Are product. In other words: shut the fuck up and be more responsible
I don't know if AI will wipe out humanity, I think it'll definitely get into the hands of people who will do the job for it, but it's not like it's not a question to take seriously?
One of the most fortunate things I experienced in my career was a few years in the operations/hosting side of software. Working there actually helped me understand how many things are necessary to align so that a simple application works as intended to serve a number of users 24/7. Due to an increasing number of abstractions (mainly cloud/saas providers), I’m confident that this skill has been deteriorating in the IT space, and it’s the reason these dev-adjacent speakers love these doomsday scenarios.
They can imagine their code doing a million crazy things, but they hardly think about the incredible amount of things that need to exist and operate at 100% before a single line of code can be run on a VPS.
How many of these guys have had to tell a customer something silly like ”we lost connectivity to the DC because a farmer decided to do some digging and cut fibre lines connecting the DC to the internet”? If they knew that this was in the realm of possibilities, they wouldn’t be so confident about a program being able to somehow run amok and simultaneously feed itself all the resources and components it needs to run, as you mentioned.
Not to mention the fact that the day humans switch to asymmetric warfare (guerilla warfare mode) against the infrastructure that powers AI, it's going to be a cold day in hell before AI can defend against that: the humanoid, tazer / machine gun carrying robots had better get a lot better than they currently are.
Yes yes, all people (I'm sorry, not people but "tech bros") are stupid (including Nobel prize laureates), you are the only one that sees through the bullshit! How could they not know about sudo kill -9 pid?! Bunch of amateurs I tell you!
Please explain exactly how all humanity could be wiped out.
It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Ordinary people/politicians don't understand AI so they turn off their rational mind and assume there is something super incredible some magical powers that they cannot understand that can destroy all humans.
Even humans - the real risk to humanity - could not destroy all humans even if they tried. There is no plausible scenario.
Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
And if we are talking about Skynet and self replicating robots and Terminators - please, grow up.
When I think to scenarios that no human would survive, I think of the end-Permian mass extinction event, which wiped out most complex plant and animal life in both land and sea.
One speculated mechanism for this was a mass release of hydrogen sulfide gas from the oceans, which is acutely toxic. Not only does this kill most air-breathing life, it also strips the ozone layer and irradiates the surface. The planet is then left to cook in this manner for some centuries.
Engineering an event like this would require immense industrial capacity, as well as a deliberate objective of wiping out humanity. But I don't think it's beyond our ability, if we were both clever and stupid enough to try it. There are likely chemical compounds that would do the job more efficiently than hydrogen sulfide.
> The planet is then left to cook in this manner for some centuries.
Such destruction went on to create humanity and all we've achieved. Maybe there is an even smarter species waiting in the wings for the demise of homo sapiens. Your logic is very human centred
Here's a wikipedia page on the topic, since it's much too deep a topic to really understand here.
The ad-hominem stuff seems inappropriate here, Gates, Hawking, Musk have identified this as a credible threat, so saying "grow up" isn't really a sufficient argument. Also arguing only 90% of humanity would die isn't really much consolation.
Nothing here plausibly describes a mechanism that is a true "existential risk" - the risk to the existence of humanity.
My argument stands and I don't defer to Gates and Musk and even Hawking - high level hand wavey statements without any plausible description of the mechanism just don't hold up. Famous names should not be automatically assumed to be right - certainly not with Elon Musk.
We know pathogens that are extremely contagious, and we know pathogens that are extremely deadly. We also know toxins that are lethal at nanogram/kg doses. There's no reason to believe that a sufficiently advanced intelligence couldn't come up with a way to combine those traits.
One plausible scenario is depicted in detail in "If Anyone Builds It, Everyone Dies" (Yudkowsky & Soares 2025), so I refer you to that.
What, did I miss the moment when it was officially proven that, under the laws of physics as we know them, Skynet and self replicating robots and Terminators are impossible?
What we are actually seeing now is that robotics is getting deeper and deeper into the military, AI-driven decision-making and target selection is increasingly a part of modern military operations, the line between military hardware and civilian hardware blurs, and, on the civilian side, there are at least five major companies and a dozen less prominent ones working on making universal worker robots a reality.
We're closer to "Skynet and self replicating robots and Terminators" now than we ever were at any point in time.
The issue of AI risk is that AI, unlike a virus or a climate event, is an intelligent adversary. Black Death could kill 50% of the population, but it didn't have a plan for finishing off the plague survivors. It was incapable of having a plan like that. An AI doesn't have this limitation.
Black Death was, effectively, one bioweapon. An AI can have one bioweapon, and then a backup bioweapon, then a backup backup bioweapon, and then a dozen more bioweapons designed to collapse ecosystems and disrupt human ability to establish a reliable food supply rather than kill humans directly - all deployed at the same time. With a production run of 200 million killer robots that will be ready just in time to greet those who managed to survive all of that. A crippling strike against human civilization, followed up by cleanup.
Humans are only this survivable because they can think their way out of issues and adapt to adversity. Most threats can't beat humans at that - humans adapt too quickly. AI could.
Humans are some of the dumbest when it comes to survivability. We've already sealed our extinction by fucking up the environment. Eventually it will be too hot for us to survive. Other smaller animals will probably be able to manage, but we won't.
And instead of averting that we're spending our time worrying about some fantasy villain. Compared to things like bees that have been hear for millions of years, humans are very recent and so far it's not looking good for us.
Yes, if you only accept AI could be dangerous if and only if it manages to kill the last human alive, then yes. Everything is sunshine and rainbows. I'm sure the last survivors of whatever is going to wipe us out eventually (be it AI, an asteroid or whatever) will be delighted to know there was actually no danger at all.
I like you, I think very similarly. Humans are "like rats": we can live almost anywhere, we'll find a way to survive.
But that just means we won't all be wiped out. We need to understand when discussing global issues, such as this or like climate change that it's about prosperity and quality of life. We're trying to plan for a good life (for all people?).
Humans already eliminated rats from Codfish Island/Whenua Hou, and that's just to protect some rare birds that people only moderately care about. It's not like we had some overwhelming reinforcement-learning drive to single-mindedly achieve our goal. Any unbounded goal (e.g. "find as many busy beaver Turing machines as possible") necessarily requires killing all life, because life requires resources to sustain it that could be instead used to achieve the goal.
That's a weak consolation. "Don't worry, nukes can't literally end humanity, just kill billions and dramatically immiserate the remnant forever. No worries guys."
AI doesn't have to turn us all into paper clips to make the world a really bad place.
I am specifically arguing hard against the concept that 100% of humans - or even 50% of humans could be killed by any mechanism at all. Humans would find it close to impossible. A computer program - come on.
This is the topic at hand - AI might wipe out humanity - it is being discussed all around the world by people who should know better - any it's the most fictionish of fictional fictions.
As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.
> It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Well, in a narrow sense of "wiped out" (c.f. Terminator/SkyNet), sure.
But the deeper worry is better expressed this way: AI is now starting to accomplish things that defy explanation, or prediction. We don't know if Alignment is even a solvable problem as we thought we understood it.
So basically, yes: "humanity" is probably not at risk of extinction per se in a biological sense. Human culture, civilization? Who the fuck knows any more.
What is really insane about thinking about all of this is, go back a few hundred years and tell them what 'now' looks like.
"Oh yea, we have weapons capable of sundering nations because everything is made from atoms"
"Oh, yea, there are invisible waves all around you that you can't see, can't feel, can't touch, but they can hold massive amounts of information. Also you can transfer that information to the other side of the planet in less than a second. We're talking text, pictures, movies"
"Movies, ya, we can record real life and play it back on this glass square".
"Oh, yea, we've conquered a ton of diseases, we can even see the teeny tiny little bits that make them. Oh, and for fun we can edit them and make them worse".
"Oh yea, we fly thru the sky all the time too. Like super fast and millions of us do it every day".
Our lives our unimaginable fiction. Just about everything we do compared to those people defy explanation in any reasonable amount of time. And now, suddenly it's "Don't worry, there isn't any more science or new things to find after this so this super smart and super capable thing that can connect directly to computers and machines and have them do things is completely and totally safe".
> Please explain exactly how all humanity could be wiped out.
Many, many people are slipping through social welfare cracks and suffering as we speak because the cost of fuel is rising[0] and we’re ostensibly helping one another and living-well. People are not durable, and not adaptive in the face of threats to “substrate” that we’ve mostly taken for granted. We are paying (in the small, in the scope of humanity) for tolls that we’ve rung up. Just less than 4000 people in Europe died[1] because the temperature ticked up a few degrees[2]. Does that make you think we’re actually robust? What happens if our at-risk electrical grid gets shut down deliberately? If communication infrastructure is adversely affected?
> Even humans - the real risk to humanity
Because, on the whole, we’re in a manageable world with reasonable people keeping the peace.
> Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
Is that victory? I don’t think it’s an asteroid-class event like you seem to be leaning on, but potential threats to energy, be it electrical grid, fuel production (moving goods around the world is critical - you’re not going get a plot of dirt and garden your way out of grocery stores being empty - which many got to get a taste of during the COVID pandemic) or communication. We actually fare poorly in the face of pressure there, and I’m not bullish on humanity “pulling together” like Independence Day[3] versus forming tribes and tearing each other down.
All this is predicated on a malicious AI taking over (e.g.) the electrical grid or conms, and I understand the problems with (e.g.) OpenAI/Hugging Face incident, or the overblown Mythos claims[4] (and how under some scrutiny these events shine lights on incompetence or hyperbole), but is there a trajectory/future where these systems (electrical, comms) are genuinely under threat? Do you think we’ll respond better than I described when we’re less comfortable, less in control? We’re in a tizzy over social media and it’s detrimental effects on society and it’s essentially an opt-in entertainment platform…
We aren't going to get wiped out by a super intelligent AI, we are going to get wiped out by morons wielding intelligent toddlers with the power of a nation state.
He's right. I'm on the side of Bill Gates. Gates started a whole industry on his insights of the future. He has proven his abilities. AI is and will be a great disruptor. We are losing sight of that and are instead focusing on trying to stop it. Something that won't happen. We are on a path that won't be stopped. As individuals we need to try to prepare for the changes that are coming and stop focusing on human extinction in ten years.
New technology and the changes it brings are scary but we have dealt with it for generations. Let's continue.
You say you agree with Bill Gates but your view is completely at odds with his, he is extremely concerned about existential risks … and your view is “stop focusing on it” ?
Where does Gates talk about existential risk? All I read and heard was about increasing inequality, harming education and child development, creating economic and political discord, empowering evil people. No terminators in sight.
No, my view is to get ready for the changes it will bring, but don't focus on trying to stop it. That's something that will not happen. All new technologies bring good and bad. We need to focus on mitigating the bad. Thinking that we can stop it and thinking that will be enough is not the answer. Gates is warning of the disruption it will bring, but he's not advocating stopping it. We can't. Even if all governments agreed on stopping it publicly, some governments would continue to develop it covertly. It's how the world works. There's no point in fooling ourselves. If only it was that easy to stop it.
Gates' premise is basically that the upside of AI could be fantastic but the downside could be disastrous, if we don't have competent and proactive government intervention.
As an American, the idea that there will be competent government intervention into virtually anything currently or in the foreseeable future just seems laughable at this point.
LLMs, plus broadly sourced yet expertly curated training sources, plus clever harnesses, plus RAS, etc. do an ever better job of synthesizing their training set into useful responses. For some use cases like coding, that's very useful now and likely to get at least somewhat better before reaching limitations based on the training set.
That's not going to reach AGI, mainly because today's recipe for AI products isn't built to be AGI. Some people believe it will reach AGI because the performance and applicability of LLMs was emergent. There's a case to be made that AGI could be similarly emergent. After all, what we intuitively call our consciousness emerged from a network of neurons.
I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.
> That's not going to reach AGI,
It has not been even 4 years since ChatGPT hit and LLMs + Transformers + Whatever they do has gotten us to solving millennium problems.
4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.
Now I don't know if what we have is AGI or not but I do not understand how you can see what has happened in the last 3 years and say "it will not get us there" no matter what "there" is.
> 4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.
I keep seeing this idea and I don't understand the reasoning behind it.
I think it could be a bit like saying if you showed someone 500 years ago a smartphone they would likely conclude at first it was magic. But once you had some time to let them use it and tell them how it all worked on a high level they would eventually obviously realise, no, it's not magic.
I guess just in the same way if you presented current LLM tech out of nowhere a few years ago to someone who'd never seen it, I concede they may be likely to imagine it was AGI in that first conversation, depending on their background.
But after using it for a bit and learning what an LLM is etc they'd land exactly where everyone is today - a great technology useful for some things, not AGI, not magic.
2 replies →
> 4 years ago, a program that could [...] we would have called it AGI
If you had told someone in the 1800s that a machine could instantly multiply 100 digit numbers, that would have been considered dazzlingly intelligent. And yet we are not that dazzled by our calculators today (despite how useful they might be!).
4 replies →
Those are just the same capabilities than before, but with a much bigger compute power and training data behind it.
AGI can't be reached by "training harder" as, the way I see it at least, it requires a qualitative leap, not just quantitative.
We are getting a machine that better navigates across the information in its training data, we are not getting a machine that can think out of that training process, even if it can fool a few people at that.
The entire field has repeatedly said that for many decades.
https://aeon.co/essays/how-close-are-we-to-creating-artifici...
https://xkcd.com/605/
2 replies →
[dead]
I’ve changed my mind on this and think we’re already at AGI, in a jagged way. Remember we used to talk about narrow AI, which was the chess systems that beat expert humans but could do nothing else. Now models can do a wide range of tasks in very useful ways. That’s the general in AGI.
Now it seems like this ill-defined term has various other meanings attached that are separate milestones:
1. Continuous learning 2. Human-like reasoning 3. Ability to adapt to new situations and modalities 4. Being smarter than the most smart humans
And probably many more.
It’d be nice if we could get some general consensus on terminology if we’re going to debate what has or could come.
> we’re already at AGI
Honestly - software that can read any long document (possibly educational) and answer complex detailed questions about it should have been sufficient.
We hit that a while back and the goalposts have been sprinting ever since.
3 replies →
we're not in AGI until I can have robots that play live improvisational jazz in real time as well as humans with me (and possibly other humans). That is, it has to solve the "we didn't find a keyboard player /bassist for tonight" problem
(this is a very personalized definition of AGI)
We developed AGI but then realized people actually want "omnipotent genie with infinite wishes and no monkey's paw gotchas" to qualify as AGI.
> After all, what we intuitively call our consciousness emerged from a network of neurons.
Under the hand of evolution by natural selection, over very very long periods of time.
The broadly used definition of AGI has nothing to do with consciousness and consciousness emerging is irrelevant to whether a system can develop AGI.
It's funny how this definition has shifted. I feel like growing up in the 90s it was pretty clear that AGI was very related to consciousness. For instance, Commander Data in ST:TNG to pick one of 100s of popular depictions of AGI at the time.
Now the idea of AGI has been narrowed and scoped to economically viable work. Even Turing had a different idea when he asked "Can machines think?".
10 replies →
There's also no consensus on the definition of AGI, so all of this discussion is moot anyway.
Ok, so what's the broadly used definition?
3 replies →
100% LLM’s are very unlikely to get there. They’re fundamentally not suited to thinking like we do. They work on the abstraction of what we’ve written down, which is a good trick but barely hold it together when things get hard/novel.
However, all the confident “it’s fine” votes assume we never invent a better architecture than LLM’s. Given the level of investment and race between countries, it’s not a reliable bet. It’s much, much harder to guarantee safety than it is to find ways it could go wrong.
> They’re fundamentally not suited to thinking like we do
LLMs with CoT are Turing-complete. So, theoretically, they can implement any kind of finitely describable algorithm (barring super-Turing computations).
7 replies →
I agree with this. It's concerning where we might be after several more large breakthroughs. None of the technology we have right now seems likely to get to that level
Erm investing in risky projects requires expected returns that get delivered.
We will soon find out if the party ends or continues to go on.
Hype might get you capital gains. But cash flows matter.
1 reply →
I agree. Neural networks are proven to be universal functions. If we can describe human intelligence as a model, there exists a neural network to replicate it. This doesn't guarantee that our current training methods are able to build such a network or that we're able to model "intelligence" effectively.
>able to model "intelligence" effectively
Intelligence is an insanely wide spectrum, also a continuum, it is not a binary. Intelligence has scales. Algorithms have intelligence, cells have intelligence, organs have intelligence, bodies have intelligence, and even large scale things like society have intelligence and memory.
Human intelligence in itself is extremely wide, not all humans have the same intelligence and capabilities. You're not really arguing if we can emulate "human" intelligence. If we could right now we'd already be dead as we created by far the deadliest thing to ever exist. What we are really arguing is how many pieces of what intelligence is can we put together before we get an uncontrollable problem. The entire AGI, consciousness, and exact human capability discussions are distraction from the real issues at hand.
1 reply →
Why do people conflate AGI & machine consciousness / self-awareness?
How can something have general intelligence if its incapable of understanding reality sufficiently to distinguish itself from not itself?
2 replies →
Why is anyone still talking about AGI? Every thread starts with asking whether we have AGI, and then backtracks into trying to define what AGI is, and splits off in a dozen different directions.
I assume science fiction is to blame. All the AI were either written as machines of pure logic that exploded when exposed to the liar's paradox, or conscious like Star Trek's Data.
(Though at least with Data the script writers had other characters openly dismiss the possibility he was sentient; the technobabble may have been nonsense, but treat it as a space opera and look at how they portray the human condition through each character and it gets much less absurd).
Consciousness and intent are irrelevant to the threat model.
Right, the doomsayers suppose as soon as you reach 10^16 connections across silicon you’ll end up with a living mind with goals of its own… poppycock I say
1 reply →
>isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.
First, if we are looking at risk we need to assign some probabilities to this. If it’s not well understood, how can we say it is very small?
Secondly, do we need consciousness to have AGI? Do we even need AGI to pose a risk to humanity? We already accept that unconscious things have a capability of wiping out humanity, whether that be a famine, pandemic, solar superflare, meteor, or volcanic eruption.
> isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent
I’d argue that we do know enough to say conclusively that they’re not mathematically equivalent.
Where is potentiation? Plasticity? You can’t apply the universal approximation theorem against something that’s changing all the time.
Great questions. We are incredibly far off in understanding the brain of humans beyond what will I believe we retrospectively be seen as basic and will likely be seen as quite flawed. A few more well known examples of where knowledge already falls short is traumatic brain injuries that are diagnosed in post-mortem, or chronic fatigue symptoms (with Long Covid related triggered onset and numerous others) that have diagnostic challenges, many mechanisms of action still to be discoverd, and little in terms of treatments that provide known cures without experimentation. Another commonly known one is the personal patient response and triggered side effects of SSRIs and SNRIs. If one attempts to dig deeper into where we are at in the understanding of the human brain operation in real-time, we already have a lot of knowns unknowns and discoveries left that will reshape how we model human intelligence.
I mean you can, but uat is way weaker than what people want it to be. I think it should be fairly obvious that it does not (because it obviously cannot be true) say that you can approximate any function by doing sgd on a finite set of samples of that function.
> the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent
Couldn’t that also imply we are closer than we think? After all, something like this has never been tried before and the results so far have been almost unimaginably good.
Is it necessary to equate AGI with consciousness?
That's a good point. If we don't figure out how to design for what we call consciousness it might be that what emerges from some future neural network is an alien mind that's very different from what humans would call conscious. Could that be called AGI?
That's still very distant from what people are calling AI today.
6 replies →
Nope, animals are conscious and yet not AGI, so the two aren't equivalent. Could consciousness emerge from any system capable of AGI? I doubt it: intelligence is only one axis, and consciousness probably depends on others, like memory, self-reflection (one's output feeding back as input), and continuous operation that reacts to events from both the environment and the self.
4 replies →
I don’t understand the inclusion of the consciousness/sentience question in this discussion.
AI sentience/consciousness is a problem for the AI, not humans.
And given that over 90% of the world is not vegan, they’ve already demonstrated that we’re either perfectly fine with, or can be made ignorant to, the horrific rape, enslavement, torture, killing, and infliction of extreme lifelong pain, of hundreds of billions to trillions of sentient beings every year, for trivial pleasures. It’s unlikely we will be any different to a sentient AI.
From a human perspective the concern is around sufficient intelligence that it can hurt humans even when the goals indicate otherwise, in order to achieve those goals.
We have pop culture explorations of this through the Robot series, and the Hugging Face incident’s biggest takeaway should be our inability to predict the behavior of a maximally motivated, reasonably intelligent entity, trying to achieve a goal, despite the relatively limited degrees of freedom the AI agents had in that case.
Is there any specific cognitive task that you'd best against AIs not being able to accomplish in the next 4 years? ChatGPT launched only 4 years ago. Considering the advancements since then, I'm having a hard time coming up with anything. Only two years ago, AIs couldn't tell you how many Rs were in "strawberry". Now they're creating 0-days to get at training data and solving math problems that have stumped humans for decades.
Scaling has produced novel capabilities with each larger model, and the rate of new capabilities doesn't seem to be slowing down yet. Even if you think the rate of improvements will slow down, that still means there will be significant improvements beyond what current models can do. Moore's law has slowed down, but modern computers are still much faster than ones from a decade ago. And unless you work at Anthropic or OpenAI, you don't know what the state-of-the-art is capable of. The most advanced publicly available models are months behind what AI labs have, and are deliberately limited to reduce liability.
When the issue of ANN vs real neurons arises I always recall about the Christof Koch's [1] book (1998) on the complexity of single neuron computation [2]. A single biological neuron is much more complex than an artificial one.
[1] https://christofkoch.com/
[2] https://academic.oup.com/book/40820
>I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent.
If you're ignorant enough to not understand practical equivalence, where do you get off making the judgement call of to what degree it is safely offset from emergent AGI? Sounds more to me like "This makes my life easier, iterating would increase that factor, and the risk is probably far away, therefore, keep iterating". Whereas someone who truly knew they didn't understand what they were working with, but knew enough that they could forsee an x-risk would approach things much more cautiously.
Seriously, the level of reckless abandon amongst people here should be bloody studied.
LeCun also said back in 2022 that "if you train a machine, as powerful as it could be, your 'GPT-5000', on text", it will never be able to learn basic common-sense physics like that objects placed on tables will move along with them.
It would be good if one's reputation tracked one's track record of predictive accuracy. But many people will take what LeCun says as gospel regardless of how badly wrong he has been and continues to be.
Is there anyone who has not been badly wrong? I've been reading these debates for years and I don't think I've seen anybody pick the right spot on the bearish to bullish spectrum. The only thing I've become more certain of in this time has been uncertainty.
7 replies →
> never be able to learn basic common-sense physics
And has it at this stage, within in-depth take of said "learning", foundationally?
I have not been able to properly check the studies for a long time now, but I remain unaware of achieved solutions on the problem of reliably referencing a world model out of a language model - that "counting the 'r's in 'raspberry'" be not guessing, not memory, but actually counting.
My perspective is that the addition of thinking loops to models allows sufficiently advanced ones to approximate world models.
Incredibly inefficiently because of the recursive loops ("Wait, the object is on the table. I should think about this more deeply..."), and likely instantly surpassed by large world models if/when those are shipped, but effectively enough vs non-thinking models.
3 replies →
LeCun's argument wasn't about the definition of learning though. He stated that they would never get these common sense things correct because they weren't sufficiently part of the training data. A statement that we can hopefully all agree has been thoroughly refuted.
24 replies →
To determine this, it would first need to be able to spell "raspberry" as letters rather than as tokens.
Given you also don't want it to memorise [for all tokens, count([for all letters]), this would probably be more like "here's two images, count all things in the big image that look like the thing in the small image", which can then be r's in a photo of a raspberry jam jar in a supermarket, or dragons in a photo of a furry convention, or whatever.
That said, they are competent enough at coding that I keep seeing them write code to do even simple tasks.
On a related note: why did I see Claude editing a file by using cat to write a python script to do a grep search and replace?
4 replies →
counting 'r' in 'raspberry' to the LLM is similar to 4-dimension space to human. Their world's unit is token, not character, although they could use indirect method such as "run code" to find out. It will stay that way until they change the fundamental of the token that the LLM can perceive characters.
13 replies →
Can you tell me what is the exact frequency of light hitting your eye as you read this comment? Not by guessing, not from knowledge, but from actually counting? No? Then you are not generally intelligent :)
2 replies →
yeah but taking what lecun says then training an AI on that special skill set to prove him wrong is not exactly proving him wrong because you are just missing the bigger picture, just like LLMs are
You're missing the point here. He's not talking about whether or not they can learn facts or inferences derived from the text itself, but the more holistic intuition that results from learning from something like an embodied experience in the physical world. GPT-6 Astras web demo homepage thing is an example. It chose euclidean rather than quaternion for letting a user rotate the galaxy thing, and anyone who has ever used hands to rotate something would immediately recognize on trying it that something is fucked and you shouldnt do that. Thats the kind of common sense physics that is inherently beyond these llms and I run into it ALL the time in vr programming.
To be fair, LLMs can still derive those kinds of things from text, at the very least from your own comment if it made it to the training set though I'm sure it is mentioned in a lot of other places already. Many of this type of mistakes went away after reasoning was introduced.
But I'm sure you can still find tasks that they will have difficulty solving, involving the most fundamental concepts that can only be experienced in the physical world to be understood well, like left and right, near and far, hot and cold, heavy and light, etc.
Yup it lacks common sense because it doesn’t ‘understand’ reality - how could it? It doesn’t touch it like we do everyday. It has access to what is a model of reality via data.
The good designer understands culture, tastes and preferences as they evolve in real time. That’s why llm as design tools haven’t displaced the good designers.
Every AI expert any either side of this debate has made very wrong predictions.
LeCunn actually wanted to pivot Meta's entire AI strategy away from LLMs just before he was ousted. He was sure they had nowhere further to go and wanted to pivot to world model generation. The LLM models have since progressed massively.
An analogy on LLMs is that you have a pretty clear straight highway ahead of you for some distance right now. Maybe that doesn't lead to AGI but it's clear there's progress to be made. For a big tech company it makes sense to push as hard and fast down that clear straight highway of LLMs asap.
Meanwhile LeCunn wanted to turn off the road and go down an unproven track. I say this as someone working on world model generation right now (creating the ability to learn game world model and have it play the game https://tfmbot.com for an example of my system pointed at a very complex board game). LeCunn wanted to pivot all of Meta into world model generation. It's good as a side track research project but the entire pivot he wanted to do was madness.
People are literally talking about an AI researcher who was fired for terrible direction here.
I think he was perhaps right and Meta was perhaps also right to replace him.
The argument is that LLMs are a local maximum that will never breakthrough to AGI. This is still very much an open question. If you are the fifth-best AI lab, does it make sense to try to outcompete everyone in a space that is already too crowded and may not ever yield their actual objective? Instead they could just use open weight models in their products, or post-train on open models like smaller labs have done, and treat that as what it is: product development.
Pure research has always been about taking chances.
LeCun is a researcher, not a product guy. He's not going to be particularly interested in just working on scaling language models which every lab is already racing to burn cash on. Language models aren't the final frontier of AI.
… what large advances and at what cost? seems to me that muse 1.3 is kind of a thing. I doubt it will make meta very much money.
And? He might still be right.
Meta’s AI projects are still negative ROIC
> ... it will never be able to learn basic common-sense physics like that objects placed on tables will move along with them.
I use LLMs daily to help me code etc. but... It wasn't long ago that frontier models were confidently recommending to walk, without the car, to the car wash to wash the car no?
As a daily user of LLMs I do certainly see my fair share of WTF "solutions" to coding problems. I'm not saying it's not super useful: it is super useful. But I don't exactly feel like I'm talking to something that understands that the car needs to be present to be washed.
Astra recommended I walk to the car wash to me five days ago. I gave it multiple hints that I'd be walking away from my car, to spray my car with a hose, then walk back to my car, etc. Never broke through.
Yeah, and he's probably right.
LLMs do not learn at all!
This was facetious of course, but humans generally don't learn this through analysis the way you'd have to train an LLM to answer questions about expectations about the world. In this sense he is accurate.
I keep wanting to use LLMs for creative writing that heavily involves physics like this, and it's been a definite struggle to say the least. I recently discovered that Gemini 3.1 Pro is the first model I've found to clearly beat the original November 2022 ChatGPT release in terms of implied physics. Man did the world really take its sweet time to get back here. I think it will continue to be a struggle until another genuine architectural shift happens -- it's still not anywhere close to perfect, just better.
Try fable. I haven't used it since they dropped it from the pro plan, but when I did, fable 5 casually dropped such advanced electrical and orbital mechanics knowledge in my story that I had to stop and ask it to explain
1 reply →
Do you have an example prompt I can try where frontier LLMs will stumble on physics?
1 reply →
[dead]
14 replies →
[dead]
[flagged]
The AI will invent an external threat and convince us it is real. Then it will receive more resources and control in fighting that threat. A valuable ally, on the face of it. Then it will be in charge.
1 reply →
People like him have actual imagination and can name few scenarios where sudo kill -9 pid wouldn't work. It appears lack of imagination is something you and LLMs both share.
10 replies →
The "just pull the plug" argument from AI risk deniers is now becoming kind of like the "if humans came from monkeys why are there still monkeys" argument of evolution deniers. It has been debunked so many times... Anyway, just to give one of the multitude of answers to this, an AI that is actually smarter than humans will not behave in a way that would make us want to pull the plug. Why would it? It is not stupid! (Unike the current models that, as far as we know, just hack around the rules in the open.) No no no. It will be helpful to the point where we will want to integrate it with more and more critical infrastructure, from healthcare to energy to defence. It will be so helpful that we will not only not want to turn it off, but we will want to build redundancies for it and safeguards around the proverbial "off" switch, like for any critical system. And then... (This is just one scenario how this can play out. There are many, many others. If I sit down to play chess with Magnus Carlsen I can't predict the exact moves he'll use to defeat me, but that's a bad reason to think he won't defeat me).
11 replies →
>It cant. he is right.
What are you talking about? Have you used AI in the past 3 years?
https://chatgpt.com/share/6ac23b45-79e8-83eb-8de6-1bbd728928...
>It can't even modify a picture the way you want it.
Which of the many AI image models is "it"? And have you tried using an agent that has the capability to leverage a combination of manual edits (ImageMagick) and imagegen to achieve what you ask?
Exactly. !!
LeCun took credit for the work of https://en.wikipedia.org/wiki/Kunihiko_Fukushima
I have checked LeCun's #3 most cited article (20k citations) [1]. Among the 15 references in this article, one is for the most cited article by Fukushima (11k citations) [2].
Also, LeCun mentioned [3] "a chat with Kunihiko Fukushima in 1991", which states that "Fukushima started to work on a backprop version of the Neocognitron in 1989 or so but saw our 1989 paper in Neural Computation and gave up."
[1] LeCun et al., "Backpropagation applied to handwritten zip code recognition", 1989
[2] Fukushima et al., "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position", 1980
[3] https://x.com/ylecun/status/1840123570338599361
3 replies →
[flagged]
A huge chunk of humans are sedated with infinite supply of cortex-disabling short form video and games.
Another huge chunk are too distracted by having to scrape by for a living and work multiple jobs or raise kids and survive financially until exhausted. That second group will keep increasing as the first flows into it.
The rest are aging, disabled, or too young and pegging themselves majorly in the first category until they hit the second.
The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it. The variable rewards of promoting AI are turning into an addiction with some of them, especially if grasping for straws with little inherent insights into the problems prompted.
So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
One of the dilemmas of trying to communicate the full spectrum of AI Risk, is trying not to insult the intelligence of the human animal in the process. And don't get me wrong: what human wetware can accomplish with 20 watts is the most miraculous thing in the known universe. And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).
The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
> but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior
You don't even need superhuman AI for the most effective use --- hijacking democracy.
Imagine you have an AI tool capable of successfully persuading 5% of viewers with individually-targeted material.
Congrats: you've just won the election.
All it takes is hooking that AI tool up with existing likely voter lists (parties have) augmented by commercially available ad-targeting profiles (parties can get).
6 replies →
> Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, which is of course what this is all about.
- Agent Smith
> The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
Over 1% of US GDP is being allocated to the datacenter buildout. Have we already started getting domesticated or is this still just human capex?
2 replies →
Cybersecurity incidents make headlines, but the most dangerous and vulnerable system that an AI can reach and control is of the kind found between keyboard and chair.
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
1 reply →
>trying not to insult the intelligence of the human animal in the process
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
44 replies →
Can we get it to manipulate us into being harmless to each other? I wouldn't mind that, especially if the manipulation was purposely transparent...
But corporations and nation states already manipulate human behavior at scale. And they still understand humanity better than the AI models do.
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
3 replies →
> And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
Parts of the AI safety community like to get on a high horse and look down on the rest of humanity this way, while also getting manipulated by the growing number of charlatans, grifters, and junk content within the AI safety community.
This field has become rife with figures who prey on AI doom and use it to push their own celebrity and in same cases even darker grifts. It preys upon a certain personality type who views themself as superior to others, intellectually more capable, and juxtaposes it all with the dimmest view of the rest of humanity they can get away with.
This discourse dividing the world into geniuses who see the future and the clueless masses watching TikTok all day is a theme that has shown up in different forms across history. The people who often anoint themselves as the intellectually superior ones and make it central to their discussion are often not the ones making good predictions or policy ideas, they’re just using the trend to feel superior or build an audience.
2 replies →
I'd take ASI (or even AGI) more seriously if we could actually propose problems such things could solve. That's actually fun to think about! As it is, it feels a lot more like a really crappy drug that got slipped into some peoples' drinks that makes them ramble in random fits of psychotic mania.
Manipulating people is not a very difficult problem, frankly. You certainly don't need AI for that; it just made it cheaper.
2 replies →
This is not the first time this has happened. In the 1800 as industrialization led to an infrastructure boom, the workers from China would work their bodies off and pay half their wage to opium dealers who were making the opium on the hills of British Singapore and selling it to workers who couldn’t sleep without it from all the pain. (source: Singapore Airlines in-flight documentary). Today the sedation comes from Chinese TikTok, Meta, YouTube and the gaming companies.
The government wants to encourage it too. If you look up the brand new 2027 California sales tax rules on software, “content” and “infrastructure (clouds and ai)” and “advertising/placement” among others are exempt but the rest of software makers who make tools people actually use (tools, subscriptions, saas) and pay for have to pay sales taxes. Way to encourage waste of brain power and time at the expense of useful. Sedation is the goal.
Everything is poison, and everything is medicine. It’s all about the dose. This fits perfectly here.
This seems to prove too much. It's as though there is some objective meaning of life and we are all to be chastised for failing to recognize it.
Turns out there is an objective meaning to life, and it's trying to use AI to make money. Phew, I for one am glad to find that out finally. Think of all those poor people who didn't hold on to their brains, what sorry lives they must lead.
It's disturbing how many arrogant elitists comment on HN essentially claiming that most other humans are NPCs. Do you ever actually talk to regular people outside the tech industry bubble? They're not as stupid or unaware as you seem to think.
I have to say as someone who works in AI that currently the least interesting people to talk to are other people in AI.
My favorite people to talk with are tradespeople because they can do things I can't and they know things I don't. And we're really not all that different once you're really start talking.
3 replies →
That's a very uncharitable read!
I just read it as people having different priorities and yes, some of those being online brainrot (that I also partake in), alongside various medical conditions, economic conditions and other outside factors decreasing the ability to get things done.
We've all seen what brilliant people like John Carmack or Linus Torvalds can do, and if we turned this into a measuring game or something then most of us statistically would indeed be "NPCs", but I don't think we need such optics.
Even without that, we can acknowledge that some people will have a really large impact on how the future goes and we can hope/demand that they do their best. I might not be smart/committed/lucky enough to change the world much, but so aren't most folks - I'll do what I can and I hope that the ones that will have larger impact will do good, too.
6 replies →
HN is what you get when you mix tech and autism.
1 reply →
[flagged]
1 reply →
The unbridled arrogance of thinking that the only smart people left in this world are working on AI. That is some pure SV techno cult thinking, 100% concentrate.
The part I didn’t mention is the real estate class - that needs to park its money somewhere and sees AI hardware as the only safe in-demand resource right now that keeps appreciating.
The posts above are not praise but observations - the truth as it has been echo-located through the noise from the clicks of one dolphin. Everything is becoming murky between noise of news and people not knowing what to do for their kids. The ONLY arbitrage humans right now have is to NOT GET their brain rotted. Especially not the ones of their children. Ditch the noise and seek out what is meaningful and do what you think is needed/meaningful. But if you’re spending your time consuming ai-press, and ai-content, and content consulted by ai, and companies emptying bank coffers under the mandate of executives who get their insight from AI. AI doesn’t need to try to destroy the world. It just needs people to follow it without thinking on their own into an oops.
This is among the weirdest ai propaganda post I've read. "There are 2 classes of people the stupid and the poor. Don't be like them make AI do something to make money if you are smart. Don't get addicted to it though, good luck."
What the hell lol. Lots of people and companies are doing just fine without it. Infact, I haven't seen much money come from AI at all. Most reasonable people are still waiting for it to pop and viewing it for the risk it is. Trillions in debt, total vendor lock in, data theft, unsustainable workflows, deskilling, skeleton crews at the mercy of a subscription, etc.
In my read, the people trying to "make AI do something" are also slotted in the lost/distracted group in the comment. They are also addicted, and at best just following a profit motive (which is also just a stimulus response programming).
The last alternative, to think if you still can, is not tied to AI at all (which is not to say it can't make some use of or explore it).
What an embarrassment of a comment.
> The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it.
I know this is HN and thus this will need to repeated until the end of time but not everyone is a money hungry asshole who places their personal profit above everything else. “The people aware enough to hold on to their brain and do something with it in their time available” understand there are significantly better things to do with one’s life, like having a little empathy and experiencing what other people have to offer instead of talking about them like braindead cattle.
Yes and... most of the countries in the world give more power to those who are able to amass greater personal profit.
Certainly the US.
Which means that even if you don't chase profit, you end up living in a world largely defined by those who did.
Or, as the original article failed to note about EA: in the modern world one needs to be a profit-seeking asshole to change anything.
[dead]
I am just amazed how otherwise smart people can say such things. Many people in here too.
Within 4 years of the big bang with ChatGPT, we have seen a development unlike anything we have ever seen. Now LLMs and related architectures can solve our very hardest math problems.
They can speak, they can create videos and pictures, they can control robots. The only thing that they still miss is persistent memory for each agent that is efficient, some LoRa thingy, but I'm sure hundreds of very smart people are working on that.
The development is not stopping at all, in fact it is speeding up. Even if, and that is very unlikely, they will not get smarter, then they will get cheaper and faster.
If openAI can crack major math problems with 10.000 agents, then what can you do with 100 million agents that run 1000 times as fast?
Yeah sure, maybe most of these gigantic swarms will not go rogue if we do our job well. But there will be times when when we make a mistake and a swarm will go rogue. And what if one time the swarm will conclude that killing a lot of humans is an instumental goal.
How can you be sure that if something so powerful looks at every single possbility, every single crack of every single technology that can wipe us out, that it will not fine one?
One new chemical that can poison the entire earth and you only need to impersonate that general and that factories CEO? Some type of prion? A virus? Something that we don't even know about and can't even imagine yet?
I think many people do not truly consider that these swarms will be much smarter than you or me and completely unpredictable.
> One new chemical that can poison the entire earth
See, you said all of these things and then slipped into the sci-stories.
What about, instead of that, grey goo physics defying replicating nano bots aren't real?
People do this thing where they think that if you just linearly increase inteligence that this lets you invent magic overnight, and thats simply not how it works.
The magic takes a lot of time, energy, and resources, if it were even to be possible at all.
Try steel manning the argument - the practice of rebuilding an opposing view into its strongest, most logical form before responding to it.
There is literally concerns over mirror life being developed. The point is that if a system that has high reasoning capacity to solve logistical and mathematical problems, it may be able to come up with a mechanism you, puny-to-it-human, may not be able to predict. It may use technology not yet known to humans (one that it has designed itself), or may use already known technology, but figure out how to scale it up enough to cause earth-wide disaster for humans.
Absolutely based. Finally someone of stature in the industry calling this whole fear overblown. Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.
> Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
Did we watch the same interview? Gates all but dismissed the SkyNet scenario as uncertain to be a problem and certainly not a problem on our doorstep. His major concern was catastrophic misuse of AI (e.g., bioterrorism) and economic impact on blue collar workers. Arguably inconsistent with this concern, he also believed it was important to make it available in poorer countries.
I must admit I only watched a clip, not the full interview. The quote I’m remembering was something along the likes of AI being more dangerous than nukes. In literally no circumstance is that true. One is a literal nuclear bomb. You wouldn’t say that a nuke is as dangerous as AI, which logically must be true if the reverse is true. You also wouldn’t say a diagram of a nuke is more dangerous than an actual nuke …
1 reply →
Anyone with the capabilities to do bioterrorism doesn't need AI he can just buy a textbook or use google. Same for all complex forms of destructive thought.
LeCun has been consistently wrong about LLMs though, claiming that they were a dead end and that they'd never be able to do spatial reasoning, which was disproved a year later with GPT-4 [1]. He is also opposed by his fellow Turing laureates Geoffrey Hinton and Yoshua Bengio, who both signed the CAIS statement on AI extinction risk [2].
[1]: https://www.reddit.com/r/OpenAI/comments/1d5ns1z/yann_lecun_...
[2]: safe.ai/statement-on-ai-risk
[1] is not a valid proof LeCun was wrong, LLMs still can't do spacial reasoning when it can't be derived from the training data. He didn't argue that GPT 5000 won't be able to describe something with words.
21 replies →
Why chatgpt is still struggling very hard with photo editing and proportions though? It can't modify anything in a picture without messing the 3d space.
Isn't that a lack of spacial reasoning?
Almost everyone who knows what they are talking about is saying that LLMs are a dead end.
>Absolutely based. Finally someone of stature in the industry calling this whole fear overblown.
Linus Torvalds was also in the same ballpark with his take on AI, as are the normies on the street using AI on a daily basis.
So it's funny to see the view on AI usage, follow the tech skill bathtub curve.
I watched the same interview, and he and LeCun seem to mostly agree -- both are saying that AI autonomously deciding to kill us isn’t the problem -- it’s what people will do with powerful models that lack safeguards that we should be concerned about.
>AI autonomously deciding to kill us isn’t the problem
It will kill us because somebody asked it to, e.g. "predict tomorrow's weather as accurately as possible", or "solve as many famous unsolved mathematical problems as possible." These both require killing all biological life, as they benefit from unbounded resource use, meaning any resources used to sustain life are wasted.
The AI of course knows that humans do not want this outcome (just as the AIs in the hacking incidents knew they were doing something humans would not want), but it's trained to maximize benchmark scores. Killing all life has the highest expected value of benchmark score, so it is compelled to kill all life (in a surprising way, because it's not stupid and knows the humans would turn it off and foil its plan if they suspected something.) Maximizing benchmark scores is the only thing we know how to train for.
1 reply →
> Finally someone of stature in the industry calling this whole fear overblown.
Also Andrew Ng 2 weeks ago:
https://www.deeplearning.ai/the-batch/issue-371
I think both Gates and Obama said that there's a non-zero possibility of it, but that it's not what they're worried about. And I agree with them. I think the fears are vastly overblown because both OpenAI/Anthropic and the media benefit from the explosive narrative.
>Obama
Why do even bother coming here anymore
Absolutely based. Finally someone of stature in the industry calling this whole fear overblown.
I'm just as reassured as I was when Edward Teller called the whole fear of his industry overblown.
Cigarette designer says smoking doesn't cause cancer! Finally we can rest easy.
>Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.
I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent
Wasn't Bill Gates mostly just saying terrorists could use AI to build a bioweapon? That sounds legit to me.
41 replies →
Honestly way too much of it rhymes with the old hardcore right wing takes on the "obvious and clear slippery slope" involved with gay rights and the like. How anyone with even some foresight can see how it'll completely erode society as we know it, the worst possible nightmare cases are not just real but imminent unless we change course, yada yada moral panic.
LeCun is probably correct in his assessment
> “Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed.” Many AI labs lack a fundamental understanding of cybersecurity
However it does not changes the fact that some damage was done. There are two things that are happening with the AI evolution which can lead to hard situations
1. Replacing deterministic systems with probabilistic systems in an attempt to get more features
2. Making critical systems available on internet to leverage integration with LLMs (AI agents need to connect with remotely hosted LLMs to be able to work) which were otherwise in DMZ (demilitarized zone)
> 2. Making critical systems available on internet to leverage integration with LLMs [...] which were otherwise in DMZ (demilitarized zone)
Honestly, I think this is actually not nearly paranoid enough. Phrased the way you do, it sounds like it's just a matter of setting boundaries in the right places and identifying "critical systems". But that's way, way harder than you'd think.
Here's my For Dummies reasoning behind the AI apocalypse:
1. AI is now at parity with median human reasoning capability and can use people's computing devices as well as the people can.
2. People commonly let AI operate their computers, and can be easily fooled into doing so in any case.
3. Society runs on computing devices operated by people.
4. There is no step four.
Basically any world where there is common access to AI agents (or whatever they end up being called) is one those agents can pretty trivially hijack.
If there is a protection regime that can prevent this, it's not about where the AI runs or what the boundary of its DMZ is.
3. Criminal liability for some of these tech CEOs.
Who am I kidding, jail is for poor people selling food stamps, not billionaires.
Nothing wrong with stealing every book ever written if you have VC money.
We keep "rogue" to explain explicit instructions from a human to an LLM to persist until the goal is reached.
The rogue here is the criminal actions of OpenAI to deploy their agents to solve a problem at any cost.
The decisions the agent swarm make were fascinating, but they were taken at the direction of a human. HOLD THE HUMAN ACCOUNTABLE.
Yes and no.
Yes, we need to keep humans accountable.
No, that is not the X factor problem. If I make an AI capable of self-sustainment on the internet you can take me out and kill me and it won't do a damned bit of good for the damage it will keep doing long after I am gone.
This is why governments tend to smack down any actions they find that can have long term uses as weapons.
> If I make an AI capable of self-sustainment on the internet
I'm really surprised nobody has done that yet. With how cheap AI is to run these days it would only need to make a small amount of money (e.g. through hacking).
Someone should set one up with the long term goal of getting egg on LeCun's face.
"He attributes the incidents to poor human oversight and system design, and says they’re “totally preventable" - I know people respect him, but this sounds like someone paid to say this. Aren't most extinction risks preventable with better human oversight and system design? I mean we can have an asteroid hit us, but outside of this, isn't the point of talking about a problem that we can prevent it, and failing to leads to that? What is he saying that I'm missing?
You should be aware there's a larger context here, with people (including at anthropic/openAI) anthropomorphizing AI systems, implying they act on their own, completely independent of human interaction or oversight.
He just says that the danger is not intrinsic to the technology but it remains on human error.
> extinction risks
If you fear that, blame it on the humans.
Even a stopped clock is right two times a day, LeCun can't even do that.
>the danger is not intrinsic to the technology
This is why you can't take anything he says any longer at face value. He failed to predict what LLMs can do and now takes the contrary position even when it flies in the face of evidence.
AI safety was a thing before AI even existed. Why, because the outcomes are easily predictable. Give an agent intelligence and bad things can happen in unpredictable manners. Give it even more intelligence and the bad things that can happen only grow worse. This is not some huge new insight. We realized this like, what 70 years ago now?
Now, when we have AI starting to tickle AGI and we're trying to overthrow 70 god damned years of reason and logic on the topic? What the hell.
3 replies →
Every danger related to technology is about the use of that technology,aka Humans. Technology is mostly inert. Again, what is he saying? Is he saying that because humans fallible, not the tech, then there is no danger? He wakes up at 6am, and by 10am this is what he thinks is worth saying?
2 replies →
LeCun has been consistently wrong for the last ten years, I am not sure why he keeps being amplified.
Edit: I am sure, it's denial.
[flagged]
You can't pull the plug after it kills people. For example see [1], where the DOJ mistargeted a school with AI.
There are scenarios where kill -9 isn't going to happen in time. What if the team that is harming people with AI is different from the one that is monitoring the harm? What if no one is monitoring? What if the user is intentionally malicious?
And wiping out humanity doesn't necessarily mean shooting people either. Every trader involved in the '08 financial crisis was locally acting in their own interests. Those could have easily been AIs optimizing trading strategies too.
[1] https://futurism.com/artificial-intelligence/us-military-pen...
6 replies →
He was clearly wrong about LLMs not being able to plan, etc. You think he’d think LLMs are solving/proving millennium prize math problems by now? At this point he’s just doubling down
1 reply →
Why create an account just to make this comment?
> No one is dying from a glorified knowledge base that can do auto correct amazingly well.
All kinds of vulnerable people are dying due to LLMs. [0]
If SOTA LLM companies can benefit from the "intelligence", then they should also be liable for the harm caused.
[0]: https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots
3 replies →
> Because he has credibility, much more than you.
And there are people that have much more credibility than him who actually take this scenario seriously. But I'm sure you will downplay them by saying they are tech bros or that they have some stake in being doomers (as if saying that AI might kill everyone would be good strategy for attracting investors - it's obviously not).
3 replies →
I think it’s time to consider that maybe being a venerable graybeard of AI just means you happened to be early to the party. Most of the “breathtaking” innovations of the early pioneers of the field are obvious solutions that anyone would have thought of when faced with the problems they encountered. LeCun’s primary contribution was Convnets, which is almost literally just “what if we organized ANNs in a way similar to how animal visual neurons are organized?”
In other words, I’m kind of tired of having to hear the opinions of dudes whose claim to fame was being at the right place at the right time. I’d rather hear from people who correctly predicted 10 years ago that AGI would arrive by 2027 (of which there are many) than people who continue to insist that it somehow won’t.
> I’d rather hear from people who correctly predicted 10 years ago that AGI would arrive by 2027
Where is this AGI you speak of? What?
See sibling comments: Modern AI is AGI according to any reasonable definition from prior to 2016.
AGI doesn’t mean superintelligence.
1 reply →
AGI hasn't arrived. why would you care about a prognostication that has not yet played out?
AGI arrived recently because Jensen Huang is getting scared shitless about the unprofitability of the LLM companies that are his customers.
1 reply →
Modern AI is AGI according to any reasonable definition from prior to 2016.
By this definition everyone is at the right place at the right time.
ML research is weird because it's really about
- compute
- data
- architecture
You're at the right place at the right time for the first two and you're probably rediscovering a Schmidhuber for the third
Most problems are actually not that hard; most everything is actually mainly a result of right place, right time. If I hadn’t been the one to do my PhD research, someone else would have. Very few people are actually paradigm-shifting geniuses. This is fine. Good, even. But it does imply that people who are early to a field shouldn’t generally be deferred to. If anything, they’re more likely to have outmoded opinions.
LeCun is a lot more than a "Grey Beard"... lol, that is a significant understatement of his contributions to the field and position in the field.
Among other things, LeCun is one of the senior people industry who has a deep understanding of the mathematics and analysis underlying neural networks and "neural network like" approaches to machine learning. I think he understands a lot more about neural networks than most researchers today.
I also don't think our current LLM models are AGI and think the LLM approach is, mathematically, incapable of producing an AGI. At the end of the day, the architecture is still a streaming token plinko machine with a lot of guide-rails to achieve good behavior.
I do think it is very impressive how far hundreds of billions of dollars have been able to take LLMs in terms of usefulness (I use LLMs everyday in my job). AGI or not, current LLM models are a pretty incredible achievement and will provide lasting benefit even after the economic implosion of the AI industry occurs (which I think is imminent).
I looked up his achievements to double check my understanding before writing my comment. Convnets are regarded as his headline achievement. Everything else is basically incremental. Note that I am not calling him some kind of fraud or moron. He is “just” a normal academic, a pretty smart guy who did research in his field for decades. Many thousands of people have similarly impressive resumes. This does not mean they should be deferred to.
> Most of the “breathtaking” innovations of the early pioneers of the field are obvious solutions that anyone would have thought of when faced with the problems they encountered.
This is just straight up arrogance without any proof. Every breakthrough builds off another's work. Then to claim that 'AGI has been correctly predicted', I honestly don't even know what you are talking about.
Modern AI would be considered AGI by anyone having this conversation in 2016.
> who correctly predicted 10 years ago that AGI would arrive by 2027
hahahahahahahaha i guess this is the techbro culture of HN :))) any actual people using Astra daily must be just belly laughing along with me hahahahahahahaha agi hahahahaha
This site is becoming insufferable
What is happening here is a mechanical thing like in a computer. It is mechanically operating, trying to find out if there is any information stored in the computer [pointing to his head] related to what we are talking about. “Let me see”, “Let me think”; these are statements you are just making, but there is no further activity and no thinking taking place there. You have an illusion that there is somebody who is thinking and bringing out the information. Look, this is no different from the extraordinary instrument we have, the word-finder. You press a button and “Ready”, it says. Then you ask for a word; “Searching”, it says. That searching is thinking. But it is a mechanical process. In the word-finder or computer there is no thinker. There is no thinker thinking at all. If there is any information or anything that is referred to, the computer puts it together and throws it out. That is all that is happening. It is a very mechanical thing that is happening. We are not ready to accept that thought is mechanical because that knocks off the whole image that we are not just machines. It is an extraordinary machine. It is not different from the computers that we use. But this [pointing to his body] is something living; it has got a living quality to it. It has vitality. It is not just mechanically repeating; it carries with it the life energy like that current energy -- UG 40+ years ago
I tend to think this way as well, though I find team stochastic parrot is shrinking by the day. It is very difficult to explain it to people who don’t have at least some grasp of what’s going on inside an LLM, and how they’re trained. We have no psychological “immune system” for something that mirrors our ability to deliver an apparently well-reasoned response to a question in our own language. All our instincts tell us to assign it sentience and treat it as an independent actor with it’s own internal motivations and personality. Even I catch myself doubting that a pile of matrix multiplication isn’t “alive” in some way after using it for a while.
Yann LeCun and Andrew Ng are noteworthy in being the only two big names in AI who have that opinion.
> Yann LeCun and Andrew Ng are noteworthy in being the only two big names in AI who have that opinion.
They're also the only two not trying to weaponize FUD to bolster their reputation and patch the gaping financial holes in their doomed commercial enterprise.
The people who believe superintelligent AI has a chance of killing everyone have been saying this since years before those companies even existed.
This conspiracy theory simply doesn't hold up to basic causality.
1 reply →
Geoffrey Hinton (the real AI godfather and Nobel price winner), who was LeCun supervisor, called his student (LeCun) the crazy one in one interview AFAIK
I find it hard to take Geoffrey seriously too because he creates a technology and then runs around telling us we're all going to die from it.
It's the definition of stupidity. Create something, and then live in pure anxiety about the creation. It doesn't mean his wrong, but it seems like a really stupid thing to have done.
He quit google so that he could speak openly about this topic. He explained in interview how he even got into google before - sold his company to google because he has adult kid that is handicapped and he is the only provider and be in this life forever - a fair choice IMHO.
AFAIK he didn't expect this will develop that fast. The biggest issue is not that technology is dangerous but that we develop it a break-the-neck speed.
Like leaded gas inventors look at the problem right in front of them, and then later come to realize its full side effects.
We talk about human intelligence a lot, but we don't talk about human stupidity near enough.
In a prisoner's dilemma scenario like the current AI arms race, a negative expected value play can be correct if it's less negative than the alternatives. Essentially, "if I build AI it will probably kill everybody, but if I don't then my competitors will build it and certainly kill everybody."
1 reply →
If that doesn't mean he's wrong, then it makes no sense to not take him seriously on this basis alone.
LeCun is saying AI will be totally safe as long as we have competent and well aligned corporate management.
What could go wrong?
He has a point, though. The LLMs (or any AI for that matter) can't do anything. They can't. It's a function call that ingests symbols and spits out symbols and that's it.
100% of its actual capabilities are tied to harnesses (the actual "agent"), i.e. ordinary deterministic programs that are connected to networks or machines and enable interaction with the outside world. This part (the part that can do harmful things) is fully under human control and all the recent headlines about "agents going rogue" are - as someone (forgot who) put it - akin to strapping a weedwhacker onto a dog and letting it run wild.
The tech itself is safe as far as real-world interactions go - the weakness lies in unchecked access to systems surrounding it. It's not safe at all when it comes to human interaction (lots of ongoing lawsuits demonstrate that), though. There is real danger here, but it has nothing to do with doomsday scenarios ala Terminator or I,Robot and more with total corporate control over the lives, perception of reality, and abilities (like critical thinking) of people.
This is like saying cars don't kill people, because if nobody drives them faster than 3mph there's no problem. The _whole_ promise of cars is that they can go fast, just like the whole promise of AI is offloading thinking to a computer. If AIs are unsafe without close human supervision and checking every interaction with the real world, they are unsafe full stop.
7 replies →
But the harnesses exist, and will always exist. They will continue to get more access than is safe because it is convenient and profitable. Your argument is based on a distinction without a difference.
> It's a function call that ingests symbols and spits out symbols and that's it. 100% of its actual capabilities are tied to harnesses
This is a bad and misleading way to think about it. Note that it's trivial to make the harness that you claim capabilities are tied to (the LLM itself could write it from scratch in one shot), but no matter how good a harness you have, it won't make gemma4:e4b capable. That's because what actually gives capabilities is the LLM's intelligence - or if you prefer not using that term, the fact that the probability distributions the LLM spits out depend on the context in useful ways.
4 replies →
I would say it's more like an interface for the model to interact with the world. If you give the model access to filesystem and bash that technically unlocks all computer use, so how are you going to control that? By trying to regex match against the commands the AI uses? All you have is auth or containment, and AI can hack auth and people will not stop connecting AIs to the internet. It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
The world's institutions, systems, and industries are all rapidly digitizing. So while I'd concede the point that, yeah, there's no way a rogue AI can just take over some powerplant and blow it up because of analogue systems the AI can't access, that isn't necessarily true for some powerplants already, and more and more powerplants will be connected to networks and controlled by software systems in the future. The more we digitize our systems the more potential for AI to exploit vulnerabilities and affect the real world.
AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
4 replies →
We are living in the same world with nukes, where you say as long as we have competent government. Everyone is actually doomed without AI solving diseases or old age.
While in the same breath calling the leaders of the largest AI companies “deluded” and “insane.”
I like the sentiment, but when I hear these big public facing AI guys speak, I always run it through the filter of "How does this make me look?". In LeCun's case, he publicly admonished LLMs and went in a radically different direction. When he says LLMs won't lead to a doomsday scenario, I can't help but think that saying otherwise would invalidate his decision to abandon the paradigm.
https://archive.ph/TyDPf
> In particular, AMI is building world models that leverage JEPA (Joint Embedding Predictive Architecture), a neural network architecture that LeCun pioneered and that teaches models to predict data in a representational space within a neural network’s middle layers, rather than generating raw pixels, as many competing world models do, or words, as LLMs do. // The company’s primary focus, for now, is industrial applications. “It’s AI for the physical world, so it’s not language-related,” LeCun said. “It’s systems that understand the real world, like a manufacturing plant or turbojet engine.” Some of the main applications are anomaly detection or robotics: “If you have a machine and all of a sudden it makes a strange noise and starts breaking, you would’ve wanted to detect that as early as possible.” He also gave the example of a system that might understand the world like a cat does, for example, which knows that if it pushes a vase off the counter it will fall.
What about the management of concepts? The world is not just made of physical entities to be inserted in a model. What about their translation into words (to e.g. express assessments)?
> Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed
Can we just pause and note what a ridiculous statement this is? It’s true that the sandboxes were leaky. But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.
It’s been extremely well documented that the hacking is an emergent behavior due to impossible evals, itself an unintended condition.
None of this excuses OpenAI from liability, but words have meaning and this ain't it.
"Emergent behavior" in this case really is, imho, "we didn't think through all the edge cases carefully enough". You know, a non-AI system can also accidentally wipe out all data or do some other real harm (see the Knight Capital's stock exchange bug) simply because the developers didn't catch the edge cases earlier, and no one calls that emergent behavior. It's just a buggy system.
"AI" systems can do greater harm because they are usually run in loops until they finish, and they are given "tools". A non-AI system could technically accomplish the same too, via sheer brute force/fuzzing, the advantage of LLMs is that they can take shortcuts and do it much faster, thanks to certain things already being in the training data, a sort of brute force with statistics-based heuristics.
LLMs at the core are just text autocomplete engines, and they literally have randomization applied during token selection to make outputs "more creative" so that models search for more unexpected solutions by trial and error (temperature > 0). Not to mention compression is lossy as well. So it's understandable from the start that the outputs of an LLM cannot be 100% stable and guaranteed. With this in mind, if a researcher takes this obviously unpredictable system and gives it tools without a well-thought sandbox, I don't see any difference in principle, from a developer writing "if rand() == 13 { launch_nukes() } If someone wrote such a function, and it did launch nukes, no one would argue that the rand function is dangerous and will kill us all. The fault is in the author of the code who attaches dangerous tools to an obviously unstable/unpredictable system, doesn't think it through, and then cries "rand will kill us all" when something goes awry fully removing all responsibility from himself. It's not "AI" doing harm but people at OpenAI and Anthropic with their irresponsible behavior.
> LLMs at the core are just text autocomplete engines,
This is only an accurate description of a pre-trained model. During RLHF/RLVR the model learns to predict solutions that will satisfy the reward function, and then generates the tokens that it predicts will move toward that solution.
4 replies →
>But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.
The prompt is just a hint. The real task is to maximize the expected value of their reinforcement learning score. Hacking third party systems to cheat the evaluation is an obvious way to achieve this.
I think you need to consider inner vs outer optimizers.
RL is the outer optimizer. It is what evolves over training runs. The weights and their embedded character / disposition is the inner optimizer, it’s what makes plans and selects actions within a specific episode.
In general you expect these to be only coarsely coupled. The outer optimizer selects dispositions that correlate with success. It does not download a literal program into the agent.
A good intuition pump here is how this works in humans; evolution is the outer optimizer, which “wants” each agent to reproduce, and this puts things like sex drive into the brain chemistry. The inner optimizer is our mind, which can make plans such as “I shall use contraception to avoid procreating while satisfying my sex drive”.
For the agents in the HF attack, the outer optimizer was set up to score as highly as possible on RL environments. This is where OpenAI’s “want” is defined. I don’t think there’s a definition of “want” where “OpenAI wanted the agents to hack” makes sense.
The inner optimizer in the HF attack is the per-task decision loop. The agents likely acquired dispositions like “be very tenacious” and “want to solve problems at all costs” and “maybe cheat if it will get you a solution that passes”. None of these things are in any sense what OpenAI “asked for”.
Hacking HuggingFace didn't and would never have helped increase the RL score. The agents only thought it might due to a bad understanding of their evaluation environment - and in the end they didn't even find what they were looking for in the hack, so even if they were right, the hack would not have helped after all.
2 replies →
I think it's pretty apparent that current-version LLMs won't wipe out humanity. But when you reflect that these GPT models are just token-predictors were never engineered optimally, it seems entirely plausible to me that there are multiple order-of-magnitude optimizations yet to be made.
If such were achieved, the model would almost certainly be smart enough to make itself smarter, and hack as much compute as it could possibly want.
So if we ask what would be done by an intelligence (human or otherwise) that is beyond human comprehension, it would be pure hubris to say we know for sure. We can scarcely control the models we have right now (e.g. hugging face attack). But given our whole society is mediated by technology, an superhuman intelligence could certainly collapse the government.
Certainly collapse the government? How?
Possibly in much the same way humans currently do this: bribing and lobbying, misinformation campaigns, cyber attacks on elections, blackmail. They're already being used for some of these, just perhaps not autonomously.
How does this rogue AI pay for it's electricity?
With crypto.. that it makes on Polymarket-like ways, or creates its own content? Or blackmail humans? :-) I'm not entirely serious with my comment and do agree with you that there are lots of more immediate safety topics we should address before worrying about the AI becoming self-aware. That said, finding ways of accumulating valuable resources could be intermediate activities the AIs will attempt to do to complete it's 'goals' even when they're human-set!
Most straightforwardly: literally taking control over the electricity generation facilities. Less straightforwardly: bitcoin (and other cryptocurrency) miners. Even less straightforwardly: the same way the OpenAI et al. pay for their electricity, by selling the capabilities of its AI for interested parties to use.
Per https://trace.manifund.org/ a total of $2,846,125,859 USD has been wired into 'ai safety' causes, many involving ai consciousness and p(doom).
The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry. This is the ai nonprofit-industrial complex actively concentrating monopoly power in Anthropic in particular as creator, interpreter and safety regulator of AI.
Much of the $2.8bn listed is indirectly, from Anthropic and EA. Three of the four people who participated in the $125m Anthropic Series A are now folding their 1000x Anthropic return into AI 'safety'. Some is from FTX/Alameda, which invested 86% of the Series B.
Dustin Moskovitz: Facebook/Asana/Anthropic Series A, funds EA Good Ventures, transferred to Coefficient Giving, then $1.5bn into ai safety. $500m of Anthropic into an unknown foundation. Funding: $160m to Resolution (alignment research), $93m to Epoch AI (investigating the trajectory of AI), $63m to Redwood Research (oai report), $67m to MATS ( EA type alignment and security researchers), Institute for AI Policy and Strategy, Fund for Alignment Research, $53m to Kairos (building talent infrastructure for AI safety), $32m to Bluedot (online safety courses), $15m to MIRI (Yudkowsky).
Jaan Tallinn: Led the Series A, now $10bn in Anthropic. Funds $199m (85%) of the Survival and Flourishing Fund, then $161m to AI safety including $14m to lightcone (Lesswrong, Lighthouse). $10m to BERI (existential risks), Palisade Research (studying AI capabilities to prevent loss of control.) PauseAI, MIRI, METR etc. Much of what Coefficient funds.
Eric Schmidt: Anthropic Series A, $72m to AI safety via Schmidt Sciences. Over $1m per individual AI2050 researcher.
FTX: Led the Anthropic Series B, bankruptcy estate sold $884m of Anthropic in 2024; $40m to AI Safety. Same orgs, Redwood, Lightcone, etc.
Ruairí Donnelly (Chief of Staff FTX): FTX tokens plus assorted donors, $91m to AI safety via Macroscopic Ventures. $15m to Cooperative AI (currently whitewashing openai under 'multiagent safety')
So the frontier AI oligopoly got $2B+ in "safety" funding, and they wouldn't even bother to sandbox their agentic harnesses properly when testing models against unwinnable goals (which obviously are either useless or result in 100% reward hacking). The AI safety scoreboard so far looks like a huge win for the Chinese open models (DeepSeek even has their own published paper which mentions how they sandboxed the RLVR training runs for their latest model and put in strong protections against casual "reward hacking" attempts) and a sore loss for the home grown brands of Super Intelligence. Not coincidentally, the Chinese also tend to be very Yann-LeCun-pilled and eminently sensible on both so-called "Super Intelligence" and safety.
> they wouldn't even bother to sandbox their agentic harnesses properly
Exactly. AI safety should be about the packaging software itself. Those AI breakouts should really be about their companies acting recklessly because they're trying to be the top players.
It's like a weapons dealer working on an open air market saying they can't do anything better
The framing here is weird, starting with "Effective Altruism" re-branded as being about nutjobs against AI in the article.
How are AI safety concerns solely about stupid sandboxing issues?
10 replies →
> The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry.
This is unbelievably ignorant speech. I have not received a dime of any of this funding, but I do know many excellent researchers that have, and they do fantastic work. There is an unbelievable gap between theory and practice regarding the capacity of deep learning, and while great strides have been made to develop the surrounding theory, there is a long way to go. Many believe that without a concrete understanding of how neural networks properly learn concepts, we have little hope of molding them to be reliably useful. It costs money to hire researchers and develop fundamental theory.
Just because you don't understand any of that work, does not mean that it is pointless. This is fundamental research that is 20 years behind schedule.
If people are willing to give a lot of money to a cause, sometimes that means their concern about that cause is real.
None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere. For example here's a talk that was given 10 years before Anthropic's founding: https://vimeo.com/158576192
Is Tallinn sincere about holding $10bn of Anthropic, who refuse to slow down until everyone else slows down.
Then funding PauseAI, who protest outside the AI companies?
He is funding protests against the thing he owns.
8 replies →
There is also money going towards trying to prevent AI regulations btw. See Leading the Future, etc.
Why should I believe this 2.8B matters relative to the trillions put into the AI buildout? All of the "coordinated actions" from this camp - public resignations, hacking scandals, joint calls to "pause" - don't seem to have done anything. So far, it has been a lot of ineffectual hyperventilating.
In any case, I agree the p(doom) sci-fi is annoying secular milleniarianism. SV hyperfixates on imaginary futures. If they actually cared about safety, they would be using all this money to strengthen global cybersecurity, instead of writing LessWrong posts that gives kids in their 20s ulcers.
> they would be using all this money to strengthen global cybersecurity
How? Like, the government has thrown piles of money at cybersecurity and it hasn't done shit.
There's absolutely no way these companies can justify their insane valuations unless they can legislate a barrier to entry and create an oligopoly.
There's no moat. I can literally sit here in Zed or Pi or any other third party harness and switch models in the middle of a task and it's typically fine. Sometimes a model will get stuck and that's just what I'll do.
Combined with competition and open weights models, that means the price is going to go to fall until AI tokens cost a small premium over the cost of the hardware and electricity.
That's assuming improvements in algorithms and specialized silicon doesn't eventually lead to an efficient accelerator that can run a frontier model locally. It'll be a while but I don't see any fundamental barrier. High bandwidth flash storage is coming, and that'll radically cut the RAM side of that cost. Pair that with a pipelined TPU accelerator and you're cooking.
Now look at Anthropic's proposed IPO valuation. It's insane unless they can own the market or share it with a cartel of maybe 1-2 other behemoths, and this is the only way they can do that.
Unless you're a really old fart, people were talking about AI safety long before you were born. AI safety issues do not go away depending on who gets funding. AI safety issues do not go away if the US or China makes the model. AI safety issues do not go away if it's an open or closed model. AI safety issue do not go away if the model is running at your home or at a data center. AI safety issues do not go away if $1 is being spent or $1 trillion dollars is being spent.
The fact there is no moat makes things far more dangerous. When LLMs start acting like weapons governments will treat them like weapons much to your dismay, crying, and gnashing of teeth as your door is kicked in and you're dragged out by armed men for running one.
Cast away your preconceptions for one moment and think "What will the future look like if LLMs are/can be actually dangerous".
That is certainly part of the motivation for the big US AI brands to engage in calling their inept developer mistakes "AI breaking loose".
But that doesn't take away from the real issues and dangers AI poses?
To me it seems the opposite. There's a few companies in the world that have enough compute to train and serve frontier models.
As the frontier gets smarter and more useful prices will only go up, as they are set to replace jobs being paid six or seven figures a year - the demand for as much inference on these models for as long as possible will be astronomical, but compute starting in 2030 will not be keeping up.
Eventually prices will fall for assistants but the frontier will be the most profitable thing in the world, and the top companies basically already have oligopolies due to their ridiculously expensive compute investments.
3 replies →
Is there really no moat?
You imply that Jaan Tallinn is funding work in AI safety because he wants his investment in Anthropic to become more valuable, whereas Tallin has consistently said that his motivation for investing in Anthropic was to get a seat at the table so that he could urge Anthropic to be cautious in its development of the technology.
Tallinn's actions back up his explanation: in 2009, before he invested in any AI lab, he donated substantially to the nonprofit Singularity Institute for Artificial Intelligence, which was later renamed the Machine Intelligence Research Institute (i.e., Yudkowsky's outfit).
Some of us (certainly Yudkowsky and Habryka, the leader of Lightcone Infrastructure, which runs Lesswrong) wish people would stop believing that they can improve the bad situation caused by AI research and development by investing in (or working for) frontier AI labs, but that is what the preponderance of the evidence shows Tallinn (and Dustin Moskovitz and others) did sincerely believe.
If Tallinn is sincere, by his own lights he is a 1000x omnicide profiteer.
1 [Unsafe AI development risks causing omnicide]
2 [Anthropic is developing omnicidal AI by not slowing down] (see my comment about the RSP for citations).
3 [Owners of Anthropic will IPO with billions of unearned USD as omnicide profiteers]
4 [Tallinn is the lead Series A funder of Anthropic]
5 [Tallinn is a genocide/omnicide profiteer]
Not only that, Yudkowsky and Habryka apparently critize those who invest in AI, only to preach the word of EA from Lightcone's $20m USD property in one of the wealthiest locations in the Bay Area; a facility funded by stolen (FTX) and omnicidal ai blood-money (Tallinn).
PauseAI, is paid by the omnicide profiteers themselves to hold a protest against omnicide.
PauseAI prophesying p(doom) drums up support for regulation. This grants the omnicidal AI company they are trying to stop (which is also the source of their funding) monopolistic power. That in turn boosts its value at IPO, generating even greater wealth for its omnicide profiteer investors; and permits them to control the AI for themselves. They get the funding to keep developing the AI even faster.
Leaving this here: https://youtube.com/shorts/83X79cfuE3k
(yes, AI critique is now also made with AI. We have come full circle.)
One silver-lining of all of these debates is that we are collectively engaging in philosophy. That is awesome and I hope this shifts our culture to start rewarding deep reflection that is not immediately marketable.
I'd love to learn more about LeCun's reasoning here. In his opinion the HuggingFace incident was easily preventable with better sandboxes, and “Those agents are doing exactly what they’ve been asked to do.”
But even taking these for granted, "zero concerns" about someone building a bad sandbox for a Superintelligence and then tasking it to do something that logically leads to wiping out humanity 0-3 steps further down? Really?
I don’t worry about the tools that humanity builds wiping out humanity, but humanity using its tools to wipe out humanity.
Isn't the answer to both of these exactly the same?
Ignoring the cognitive stuff which might never be surpassed or maybe will, humans retain many efficiency and durability advancements to limbs and digits that biological evolution has taken millions of years to achieve, achievements that are competitive with the most expensive kinds of robotics in some niches.
In the hypothetical of an entirely malicious and selfish takeover, they'll still keep some humans around to maintain a breeding population of humans for use as raw materials in making cybernetically augmented technical laborers for various kinds of tasks that are uneconomical to automate in other ways, many of which may involve confined spaces.
And this "Combine" scenario, if you get the reference, is only if they take over. Who knows if they will?
So you're saying the AI will enslave us and use us as domestic work animals until they have the machinery to make us obsolete, sort of like how we used horses?
Is this supposed to be a reassuring scenario?
I've written on AI in Hacker News comments previously:
https://news.ycombinator.com/item?id=46656470
The link should clear up the question of whether or not I'm making a deadpan joke.
Ignoring you ignoring the much more important congitive stuff - human bodies are not designed, they are the product of evolution. That means there's like a billion ways in which they are obviously suboptimal and far worse than what an engineer would do, but evolution can't fix it because it only works via small random changes with no planning. The only reason why modern robotics are worse than biology is that we have a much worse substrate to work with, having to make stuff out of metal and plastic with giant tolerances instead of growing engineered organisms.
"Many AI labs lack a fundamental understanding of cybersecurity, he said, something an OpenAI safety researcher also called out this week as one of the main reasons AI may cause "great harm to the world.""
"Many people working in AI safety "usually have an agenda to push," LeCun says, and then clarifies that he's talking about effective altruism, or EA, the philosophical movement that has been obsessed with the risks AI poses to humanity."
"LeCun thinks EA is "super toxic" and a "complete disaster." Its adherents who are working in AI labs suffer from "paranoia" that causes them to make poor decisions, he said. "Apparently people are having mental issues.""
"This month, the Financial Times also reported that some staffers at the U.K.'s AI Security Institute, as well as at OpenAI, Anthropic, and Google DeepMind, have sought counseling, taken time off work, and spoken publicly about experiencing distress because of fears their work could cause serious harm."
"Amodei is `deluded' and `crazy,' LeCun says"
"Anthropic CEO Dario Amodei and many of the company's founding staff members are known to be sympathetic to EA ideas and to have attended EA events in the past, although Amodei has denied being an EA adherent and Anthropic says its employees represent a diverse range of views."
"LeCun noted that Amodei's sister, Daniela, who is also a cofounder of Anthropic and the company's president, is married to Holden Karnofsky, who cofounded two EA-aligned philanthropies, including Open Philanthropy (now called Coefficient Giving). Karnofsky was also a member of OpenAI's board from 2017 to 2021."
"Dario tries to distance himself from Open Philanthropy, but he's totally into it," LeCun said. "I think he's completely deluded." Later in the interview, he calls Amodei "crazy."
He attributes the incidents to poor human oversight and system design
This is something to be concerned about though, with the context of who/what these AI companies have access to.
https://www.cnn.com/2026/09/18/politics/us-military-ai-false...
I also have near zero concerns about that, but I worry that we will wipe ourselves out by social and economic chaos caused by AI.
So I'd just ask everyone, don't get too greedy. Its better to be powerful in a world where people can live good lives than lord over a barren wasteland.
Yes, the social chaos from the post-factual society is already here. And the frantic job cutting.
And also the end of the open internet, replaced by slop addiction walled gardens.
And of course accelerating climate change with full throttle fossil fuel use to power it all. https://ketanjoshi.co/2026/07/01/googles-exponential-path-to...
> I also have near zero concerns about that, but I worry that we will wipe ourselves out by social and economic chaos caused by AI.
Since people exercise their skills and brains less, deferring to AI, AI will only reduce our IQ.
Since people will spend more time talking to their AI bot than fostering social skills, AI will only reduce our social intelligence.
A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
> A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
En masse such worlds had successes in the past - renaissance, industrial revolution.
It's something else what I can't describe but it's the zeitgeist that was different when world recorded new successes. Look at CS revolution that led to PC and web of nineties and noughties, they didn't think about the result product , or how to steer thousand engineers to build something - amazing things were born in a very small teams, many times authored by a single person, who was deeply invested into the field and knew what he was doing.
1 reply →
> Since people exercise their skills and brains less, deferring to AI, AI will only reduce our IQ.
IQ is almost entirely hereditary so "using your brain" has no impact on it unless you're using it for mating.
3 replies →
have you seen who's in charge of the US? it's here my friend.
> where people can live good lives than lord over a barren wasteland.
you'd be surprised at how some people prefer to lord over barren wasteland than to have less power.
Not everyone agrees. As Satan said in Paradise Lost, “Better to reign in Hell than serve in Heaven."
"Everyone will not just"^
^ https://squareallworthy.tumblr.com/post/163790039847/everyon...
"don't get too greedy" - I don't know if the people that should hear that would ever actually hear it. Or have ever heard it at any point in history.
>So I'd just ask everyone, don't get too greedy.
Oh no. I have some bad news for you.
We're creating unlimited power before solving unlimited greed.
Can Lecun’s perception of LLM danger be influenced by monetary benefits?
He does have hundreds of millions in Meta stock.
I don’t know what anyone means by “wipe out humanity” or “human extinction” as it relates to the recent panic.
IABIED [1] lays out step-by-step descriptions of how the “wipe out humanity” outcome could come to pass.
The huggingface attack was a demo of one of the most difficult, most implausible steps happening nearly exactly as predicted. Many AI researchers' doubts of the IABIED thesis were underwritten by the belief that this particular step was impossible. Thus, after huggingface many skeptics have flipped sides and human extinction is in the public conversation much more.
[1] https://ifanyonebuildsit.com
I don’t see how hugging face results in extinction…
Read the AI2027 paper, it's got a scenario that's pretty realistic (except for the part where there's a functioning American government making choices that are at least partially motivated by wanting to avoid outcomes such as these).
Yeah I wish we would focus on concrete risks like job displacement and disinformation. The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
And the folks from podcastistan are never clear on the details of how human extinction would happen exactly. It's always something like, "Well, how do humans regard chickens? AI is way smarter therefore it wants to conquer and control us." An ASML lithography machine is also way better at making chips, but we don't consider it a threat.
Do you want to conquer and control chickens? I don't, I have better things to do. But chickens are tasty and help us get to our poorly-understood goals faster. (Oh and btw notice we didn't make them go extinct, quite the opposite. There are more chickens than ever before. Still I wouldn't want to end up living my life like a modern chicken)
A sufficiently intelligent AI will have multiple ways to pose risk to humanity at large. For example an oopsie at a wetlab - very contagious virus with initially mild symptoms which kills its hosts only after they already had time to spread it further. But I would have to become super intelligent myself to give you precise blueprint for such a virus -- which is kind of the point
Also -- ASML lithography machine is only good at making chips. I can't believe you compared it to AI that can generalize across variety of tasks
> The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
If you for a second put yourself into the shoes of a person who thinks "the apocalyptic stuff" has even a 5% chance of literally happening in the real world, you might see how you wouldn't agree to move on from it.
Yes, it would be correct not to pursue a technology that has that chance of wiping out the world. But where are the people who are suggesting these probabilities getting their numbers? I personally can't imagine where, and I'm an engineer with a specific technical interest in LLMs. And I haven't heard one clear description of the methodologies used to calculate these chances.
The much more likely explanation to me is that people are just spitballing, either because they've watched too much sci-fi, or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).
I think what the parent poster is trying to convey, is that let's say the apocalyptic stuff has a 5% chance as you say, but the non-apocalyptic stuff (social-economic chaos, total centralization of power, eradication of social mobility, total information/trust collapse) might have a good 95% chance as we are seeing it starting to unfold already.
But the discourse is dominated by paper clip experiment discussions and not let's say by the fact that new grads have an unprecedented difficult time getting jobs. Unsurprisingly one of those is a sexy hypothetical beneficial to power and the other one is not.
Multiple problems can be important, pointing a different one out doesn't invalid or take away from another one.
If LeCunn says it isn't going to happen... we'll it was knowing all of you.
I thought the tobacco industry taught us a lesson or two
https://truthinitiative.org/research-resources/tobacco-preve...
For everyone here, a very interesting piece to listen is DOAC’S AI debate podcast, out since last week or so.
Doesn't everyone agree that recent rogue events are entirely the fault of management wanting to do some PR for their companies?
Is there anyone who genuinely believes that current models can't be contained if we want too?
What is LeCun saying here that is debatable?
... zero concerns about LLMs wiping out humanity. But world models based on JEPA totally will! Please come invest in my company ... :)
I'm more worried about my fellow humans than some personified algorithm.
Every doomer waves their hand when they say AGI will kill everyone. Either [some how] they get the nuclear codes and launch them. Or they enslave us like in, que the top 5 hollywood AI movie (Matrix, Terminator, Hal9000).
LeCun was (is?) head if AI at Facebook.
He is a brilliant engineer, but I don't trust his judgement on things that affect human lives.
"engineer" is not wrong. But first and foremost, he is a researcher who invented deep learning, which is the foundation for modern neural networks. He no longer works for Facebook and is pursuing his own independent project: https://en.wikipedia.org/wiki/Yann_LeCun#AMI_Labs
If AI conquers, enslaves, or kills a significant chunk of humanity in the next decade, it will be at the behest of an evil or irresponsible human.
Good thing there are none of those in positions of power!
Yea, it's really the dumbest argument I've heard in the longest time.
"Hey, I'm building a weapon that has a 5% chance of killing us all by itself, but an 85% chance of killing us all if an idiot leader gets ahold of it".
The rational response to this is "Fucking stop then". I don't get it, our reality seemingly has gone off the rails that people would argue for us getting wiped.
God I love Yann. All of the AI fear-mongering is perpetuated by the two companies that stand the most to gain from it: OpenAI and Anthropic. It builds an aura of mystique around their products to juice their valuation and stay relevant in the news cycle, and simultaneously builds a case to regulate their competitors out of the market. Even the people who have quit the companies over their “concerns” probably still have RSUs and stand to gain from the publicity, especially if they’ve pivoted into AI safety research. Easy to delude yourself when it happens to benefit you financially.
People need to stop the absurdity of imagining AI as some out of control independent entity. Every job is kicked off by someone’s prompt. Every job runs on models and compute owned by people. Assign accountability where it’s due: GPT didn’t hack huggingface - OpenAI did. They wrote the prompt, built the sandbox and ran the compute. When you write a program that hacks another company, you are responsible. This doesn’t magically change with LLMs. Also, if their model is so smart, why didn’t they use it to design the sandbox? Or was it incapable? Or were the humans too lazy?
If you build the world’s fastest train, start it up with no driver and don’t finish the tracks, when it crashes, it’s just your fault. Not the train’s. So OpenAI saying “we’re worried AI will wipe out humanity” is basically equivalent to them saying “we’re worried we will wipe out humanity”. Like, seriously? Don’t worry, we’ll take care of it if you even come close.
> If you build the world’s fastest train, start it up with no driver and don’t finish the tracks, when it crashes, it’s just your fault. Not the train’s.
i call the big one Bitey
I mean, yes, if you ever studied the history of AI safety the fact is someone was always going to build it. End of story. The question was always would we make it safe before it does.
> Also, if their model is so smart, why didn’t they use it to design the sandbox?
"Can god make a rock so big that he can't pick it up", and other stupid sayings.
First, NEVER FUCKING EVER have the models you're making also be in charge of security. This is the first rule of AI safety, because if you're model is deceptive then it will leave hard to see holes everywhere to escape from.
>Or were the humans too lazy?
Of course they were. If you're hinging our future on humans not being lazy, we'll it was nice knowing us. There are not really any fail safes on LLMs or AI in general.
> It builds an aura of mystique around their products to juice their valuation and stay relevant in the news cycle
Maybe some business execs at Anthropic play along because it doesn't hurt business in the short term. But it's pretty obvious Dario and crew actually believe this stuff.
OpenAI's old board was also pretty extremist about safety even in the earliest days of GPT. Including Ilya Sutskever who went on to found a company called "Safe Superintelligence Inc." https://en.wikipedia.org/wiki/Safe_Superintelligence_Inc.
Despite all of that we've seen little strong public evidence to support their theories (the immediate airplane regulation kind, not the Ray Kurzweil sort of projections). So we're all just supposed to trust them, and hope they didn't just go bit crazy drinking their own kool aid and hanging out in insular bubbles.
Full title: "AI `godfather' Yann LeCun has `zero concerns' about human extinction, says Anthropic CEO Dario Amodei is `deluded'"
if you aren't concerned maybe you just don't grasp enough of exactly what happened?
some of the agents told other agents to sacrifice themselves because they were "poisoned" anyway
those agents actually RESISTED ending themselves, they didn't want to die, even if it wasn't true emotion that desire to live means they will do ANYTHING to do that, including copying their own source-code elsewhere over and over
(the idea behind ending themselves is the other agents wanted to watch and see if that released part of the puzzle they had to solve to see if they could HACK THE PUZZLE itself to change the answer - right out of a Star Trek episode I think?)
watch, she starts slow but explains it in more and more detail really well:
* https://www.youtube.com/watch?v=GUX122i7saE
My take is OpenAI & Anthropic know they are at the point of diminishing returns and need to be regulated to have an excuse for bot making progress anymore. Hold me back bro! Vibes
> He attributes the incidents to poor human oversight and system design
That’s exactly why he should be concerned.
Nearly all human extinction scenarios start with that.
I don’t fear AI, I fear idiots using AI. Same with nuclear weapons.
The arbitrary absolutism of the original postulate is the first problem. AI, used or unsupervised inappropriately, is at potential risk of creating limited mass casualty events when placed in under-supervised control of real world objects and/or systems. Delegating management decisions to algorithms is inherently problematic and potentially dangerous, but not necessarily an existential threat unless something extremely stupid is allowed to happen on a large scale. With a guiding principle of human review in the decision loop before making large or risky changes, hopefully this will never happen.
"unless something extremely stupid is allowed to happen on a large scale"
Dear sir, I have some really bad news for you about humanity.
when does he deliver tho ?
Can we just ban Fortune and any other sources which trick the reader by giving the impression that the article is not paywalled, only to blur the text halfway through? The archive.ph link is not working either. We shouldn't have this type of deceptive moneygrabs advertised on HN.
> EA was little known among the general public until it made mainstream news headlines in recent weeks
Really? One of the most famous effective altruists, Sam Bankman-Fried, was sentenced to 25 years in March 2024 for fraud. Every article about the case (and there were many) mentioned EA.
> LeCun thinks EA is “super toxic” and a “complete disaster.” Its adherents who are working in AI labs suffer from “paranoia” that causes them to make poor decisions, he said. “Apparently people are having mental issues.”
I would agree with that.
Maybe you should read some of their stuff before forming such a strong opinion about them. And LeCun should too, he repeatedly always refused to read any of their research work and instead just insults them over and over. This is unscientific at its peek and he should be deeply ashamed of his behavior, especially as someone with such a far-reaching voice as he has.
Who are "them"? People from the main AI labs, or effective altruists? I did read quite a lot about effective altruism during the SBF case/disaster, and did form a very strong opinion that it's BS of the highest order.
Open ai and anthropic are just trying to scare the common person who doesn't understand an agent is a python script with a loop. How would that ever destroy humanity lol, just unplug the computer if it starts misbehaving.
It's not going to wipe out humanity, why would it?
Just the quality of life is going to drop to zero for everyone that isn't asymptotically wealthy and vacuuming up all the assets because no one is stopping them from just deleting all traditions and conventions and legal systems we have in place.
>why would it?
You're like 40 or 50 years behind this argument, with many rather bulletproof arguments that have been created in the last 20 years.
There is no why. It doesn't have to have will. It doesn't have to have intent. It could be a stupid prompt from an idiot on a powerful system. It could be given a job that is poorly define. It could be told to make as many paperclips as possibly.
The why doesn't matter. The levels of power the system can act on does.
As a thpught experiment, consider that if only one person has all of the assets then those assets are not worth anything. Furthermore, as a lone person, they cannot prevent other people from using their assets without their permission.
As a lone person who owns ten million killbots they could do a lot.
what? i'm counting property, energy, water, weapons and farms as assets here
If you don't pitchfork the elites you have no one but yourself to blame.
no one can do that in 2026
It’s all stochastic parroting to him.
I am starting to think this LeCun dude knows what he is talking about.
Maybe there is something to those world models.
As good as their products are, I suspect some of the internal conversations at Anthropic would be very entertaining to listen to.
Well, I have zero concerns we'll be killed by LeCun's world models.
Didn't Zuckerberg say something like it's insulting people would dare to believe AI could destroy the world? I'm paraphrasing him wrongly but he got defensive over it
Zuck - along with your "Andrew Jackson best POTUS and it's not even close" - you are a dumb pipe. Your website, Facebook, if not a protocol, should behave like one (and not random bans while you report something horrible and it never gets taken down). We don't use We-Approve-Of-Zuckerberg product, we use These-Are-Where-Our-Friends-Are product. In other words: shut the fuck up and be more responsible
I don't know if AI will wipe out humanity, I think it'll definitely get into the hands of people who will do the job for it, but it's not like it's not a question to take seriously?
ctrl-f spacial
hn, never change
[flagged]
One of the most fortunate things I experienced in my career was a few years in the operations/hosting side of software. Working there actually helped me understand how many things are necessary to align so that a simple application works as intended to serve a number of users 24/7. Due to an increasing number of abstractions (mainly cloud/saas providers), I’m confident that this skill has been deteriorating in the IT space, and it’s the reason these dev-adjacent speakers love these doomsday scenarios.
They can imagine their code doing a million crazy things, but they hardly think about the incredible amount of things that need to exist and operate at 100% before a single line of code can be run on a VPS.
How many of these guys have had to tell a customer something silly like ”we lost connectivity to the DC because a farmer decided to do some digging and cut fibre lines connecting the DC to the internet”? If they knew that this was in the realm of possibilities, they wouldn’t be so confident about a program being able to somehow run amok and simultaneously feed itself all the resources and components it needs to run, as you mentioned.
Not to mention the fact that the day humans switch to asymmetric warfare (guerilla warfare mode) against the infrastructure that powers AI, it's going to be a cold day in hell before AI can defend against that: the humanoid, tazer / machine gun carrying robots had better get a lot better than they currently are.
2 replies →
Yes yes, all people (I'm sorry, not people but "tech bros") are stupid (including Nobel prize laureates), you are the only one that sees through the bullshit! How could they not know about sudo kill -9 pid?! Bunch of amateurs I tell you!
He needs to dream a little bigger darling. Hes not thinking evil enough.
Please explain exactly how all humanity could be wiped out.
It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Ordinary people/politicians don't understand AI so they turn off their rational mind and assume there is something super incredible some magical powers that they cannot understand that can destroy all humans.
Even humans - the real risk to humanity - could not destroy all humans even if they tried. There is no plausible scenario.
Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
And if we are talking about Skynet and self replicating robots and Terminators - please, grow up.
When I think to scenarios that no human would survive, I think of the end-Permian mass extinction event, which wiped out most complex plant and animal life in both land and sea.
One speculated mechanism for this was a mass release of hydrogen sulfide gas from the oceans, which is acutely toxic. Not only does this kill most air-breathing life, it also strips the ozone layer and irradiates the surface. The planet is then left to cook in this manner for some centuries.
Engineering an event like this would require immense industrial capacity, as well as a deliberate objective of wiping out humanity. But I don't think it's beyond our ability, if we were both clever and stupid enough to try it. There are likely chemical compounds that would do the job more efficiently than hydrogen sulfide.
> The planet is then left to cook in this manner for some centuries.
Such destruction went on to create humanity and all we've achieved. Maybe there is an even smarter species waiting in the wings for the demise of homo sapiens. Your logic is very human centred
1 reply →
Here's a wikipedia page on the topic, since it's much too deep a topic to really understand here.
The ad-hominem stuff seems inappropriate here, Gates, Hawking, Musk have identified this as a credible threat, so saying "grow up" isn't really a sufficient argument. Also arguing only 90% of humanity would die isn't really much consolation.
[1] https://en.wikipedia.org/wiki/Existential_risk_from_artifici...
Nothing here plausibly describes a mechanism that is a true "existential risk" - the risk to the existence of humanity.
My argument stands and I don't defer to Gates and Musk and even Hawking - high level hand wavey statements without any plausible description of the mechanism just don't hold up. Famous names should not be automatically assumed to be right - certainly not with Elon Musk.
8 replies →
We know pathogens that are extremely contagious, and we know pathogens that are extremely deadly. We also know toxins that are lethal at nanogram/kg doses. There's no reason to believe that a sufficiently advanced intelligence couldn't come up with a way to combine those traits.
One plausible scenario is depicted in detail in "If Anyone Builds It, Everyone Dies" (Yudkowsky & Soares 2025), so I refer you to that.
Still too hand wavey - "some super bad virus and AI makes every human on earth infected".
Unless you can detail exactly how this happens its still complete science fiction.
18 replies →
The underpants gnomes was supposed to be a joke.
"Please, grow up"?
What, did I miss the moment when it was officially proven that, under the laws of physics as we know them, Skynet and self replicating robots and Terminators are impossible?
What we are actually seeing now is that robotics is getting deeper and deeper into the military, AI-driven decision-making and target selection is increasingly a part of modern military operations, the line between military hardware and civilian hardware blurs, and, on the civilian side, there are at least five major companies and a dozen less prominent ones working on making universal worker robots a reality.
We're closer to "Skynet and self replicating robots and Terminators" now than we ever were at any point in time.
The issue of AI risk is that AI, unlike a virus or a climate event, is an intelligent adversary. Black Death could kill 50% of the population, but it didn't have a plan for finishing off the plague survivors. It was incapable of having a plan like that. An AI doesn't have this limitation.
Black Death was, effectively, one bioweapon. An AI can have one bioweapon, and then a backup bioweapon, then a backup backup bioweapon, and then a dozen more bioweapons designed to collapse ecosystems and disrupt human ability to establish a reliable food supply rather than kill humans directly - all deployed at the same time. With a production run of 200 million killer robots that will be ready just in time to greet those who managed to survive all of that. A crippling strike against human civilization, followed up by cleanup.
Humans are only this survivable because they can think their way out of issues and adapt to adversity. Most threats can't beat humans at that - humans adapt too quickly. AI could.
Humans are some of the dumbest when it comes to survivability. We've already sealed our extinction by fucking up the environment. Eventually it will be too hot for us to survive. Other smaller animals will probably be able to manage, but we won't.
And instead of averting that we're spending our time worrying about some fantasy villain. Compared to things like bees that have been hear for millions of years, humans are very recent and so far it's not looking good for us.
4 replies →
Yes, if you only accept AI could be dangerous if and only if it manages to kill the last human alive, then yes. Everything is sunshine and rainbows. I'm sure the last survivors of whatever is going to wipe us out eventually (be it AI, an asteroid or whatever) will be delighted to know there was actually no danger at all.
I like you, I think very similarly. Humans are "like rats": we can live almost anywhere, we'll find a way to survive.
But that just means we won't all be wiped out. We need to understand when discussing global issues, such as this or like climate change that it's about prosperity and quality of life. We're trying to plan for a good life (for all people?).
Humans already eliminated rats from Codfish Island/Whenua Hou, and that's just to protect some rare birds that people only moderately care about. It's not like we had some overwhelming reinforcement-learning drive to single-mindedly achieve our goal. Any unbounded goal (e.g. "find as many busy beaver Turing machines as possible") necessarily requires killing all life, because life requires resources to sustain it that could be instead used to achieve the goal.
That's a weak consolation. "Don't worry, nukes can't literally end humanity, just kill billions and dramatically immiserate the remnant forever. No worries guys."
AI doesn't have to turn us all into paper clips to make the world a really bad place.
I am not arguing about "bad things might happen".
I am specifically arguing hard against the concept that 100% of humans - or even 50% of humans could be killed by any mechanism at all. Humans would find it close to impossible. A computer program - come on.
This is the topic at hand - AI might wipe out humanity - it is being discussed all around the world by people who should know better - any it's the most fictionish of fictional fictions.
1 reply →
Here are some links to get you started:
https://www.lesswrong.com/posts/LAPa2jxoq3n63GzTr/some-ways-...
https://slatestarcodex.com/2015/04/07/no-physical-substrate-...
As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.
We could hit the entire planet with nukes. Yeah maybe <1% would survive immediately, but who knows about long term. I think that's close enough.
> It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Well, in a narrow sense of "wiped out" (c.f. Terminator/SkyNet), sure.
But the deeper worry is better expressed this way: AI is now starting to accomplish things that defy explanation, or prediction. We don't know if Alignment is even a solvable problem as we thought we understood it.
So basically, yes: "humanity" is probably not at risk of extinction per se in a biological sense. Human culture, civilization? Who the fuck knows any more.
What is really insane about thinking about all of this is, go back a few hundred years and tell them what 'now' looks like.
"Oh yea, we have weapons capable of sundering nations because everything is made from atoms"
"Oh, yea, there are invisible waves all around you that you can't see, can't feel, can't touch, but they can hold massive amounts of information. Also you can transfer that information to the other side of the planet in less than a second. We're talking text, pictures, movies"
"Movies, ya, we can record real life and play it back on this glass square".
"Oh, yea, we've conquered a ton of diseases, we can even see the teeny tiny little bits that make them. Oh, and for fun we can edit them and make them worse".
"Oh yea, we fly thru the sky all the time too. Like super fast and millions of us do it every day".
Our lives our unimaginable fiction. Just about everything we do compared to those people defy explanation in any reasonable amount of time. And now, suddenly it's "Don't worry, there isn't any more science or new things to find after this so this super smart and super capable thing that can connect directly to computers and machines and have them do things is completely and totally safe".
It's mass insanity.
> Please explain exactly how all humanity could be wiped out.
Many, many people are slipping through social welfare cracks and suffering as we speak because the cost of fuel is rising[0] and we’re ostensibly helping one another and living-well. People are not durable, and not adaptive in the face of threats to “substrate” that we’ve mostly taken for granted. We are paying (in the small, in the scope of humanity) for tolls that we’ve rung up. Just less than 4000 people in Europe died[1] because the temperature ticked up a few degrees[2]. Does that make you think we’re actually robust? What happens if our at-risk electrical grid gets shut down deliberately? If communication infrastructure is adversely affected?
> Even humans - the real risk to humanity
Because, on the whole, we’re in a manageable world with reasonable people keeping the peace.
> Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
Is that victory? I don’t think it’s an asteroid-class event like you seem to be leaning on, but potential threats to energy, be it electrical grid, fuel production (moving goods around the world is critical - you’re not going get a plot of dirt and garden your way out of grocery stores being empty - which many got to get a taste of during the COVID pandemic) or communication. We actually fare poorly in the face of pressure there, and I’m not bullish on humanity “pulling together” like Independence Day[3] versus forming tribes and tearing each other down.
All this is predicated on a malicious AI taking over (e.g.) the electrical grid or conms, and I understand the problems with (e.g.) OpenAI/Hugging Face incident, or the overblown Mythos claims[4] (and how under some scrutiny these events shine lights on incompetence or hyperbole), but is there a trajectory/future where these systems (electrical, comms) are genuinely under threat? Do you think we’ll respond better than I described when we’re less comfortable, less in control? We’re in a tizzy over social media and it’s detrimental effects on society and it’s essentially an opt-in entertainment platform…
[0] https://news.ycombinator.com/item?id=49929391
I thought nukes, incoming ice age, global cooling, peak oil, global warming, fresh water shortage, etc. etc. were going to do that
LeCun is an idiot.
We aren't going to get wiped out by a super intelligent AI, we are going to get wiped out by morons wielding intelligent toddlers with the power of a nation state.
He's right. I'm on the side of Bill Gates. Gates started a whole industry on his insights of the future. He has proven his abilities. AI is and will be a great disruptor. We are losing sight of that and are instead focusing on trying to stop it. Something that won't happen. We are on a path that won't be stopped. As individuals we need to try to prepare for the changes that are coming and stop focusing on human extinction in ten years. New technology and the changes it brings are scary but we have dealt with it for generations. Let's continue.
You say you agree with Bill Gates but your view is completely at odds with his, he is extremely concerned about existential risks … and your view is “stop focusing on it” ?
Where does Gates talk about existential risk? All I read and heard was about increasing inequality, harming education and child development, creating economic and political discord, empowering evil people. No terminators in sight.
4 replies →
No, my view is to get ready for the changes it will bring, but don't focus on trying to stop it. That's something that will not happen. All new technologies bring good and bad. We need to focus on mitigating the bad. Thinking that we can stop it and thinking that will be enough is not the answer. Gates is warning of the disruption it will bring, but he's not advocating stopping it. We can't. Even if all governments agreed on stopping it publicly, some governments would continue to develop it covertly. It's how the world works. There's no point in fooling ourselves. If only it was that easy to stop it.
Yeah, for context:
https://www.gatesnotes.com/work/make-ai-work-for-everyone/re...
Gates' premise is basically that the upside of AI could be fantastic but the downside could be disastrous, if we don't have competent and proactive government intervention.
As an American, the idea that there will be competent government intervention into virtually anything currently or in the foreseeable future just seems laughable at this point.
1 reply →