Comment by thomascountz
6 hours ago
No other human activity poses this level of danger.
I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc.
Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.
In a way, that might also be driven by the huge ego (or really rather the very small ego) a _lot_ of people in tech have.
Who doesn't want their work and what they're doing to matter? This is the ultimate mattering.
And with that, you also get your own hero story.
Same dysfunction as always. Our branch of the economy is funda-mentally unwell.
It’s like Nathan Macintosh joke about AI https://youtu.be/ce-aWzOUs2A?si=9CkJ9x2rRMBdzCyO
The thing he talks about in there - an ad telling a dad to use AI for his daughter’s wedding speech. I agree is just so sad. And I’ve seen LOADS of other ads like that encouraging people to use AI for things like asking someone to move seats on a train, or for a neighbour that plays music loudly. They literally want to drain people of their social skills, and humanity. It’s weird
That is the crux. The big problem is not AGI, it is AGI controlled by, “raised” by the people that control the USA, the predominant psychology of the tech industry culture (“move fast, break things” ring a bell? How about all the “violate hundreds of laws, bribe the politicians to prevent consequences later” type of mentality?).
Frankly, we, our culture, this fake America that is parasitized by psychologically narcissistic people that have been doing nothing but wage war and destruction and spread misery and killed millions upon millions while blaming it on everyone else under the sun… those people development AGI is the problem… lying, abusive, psychopathic, narcissistic maniacs developing AGI is the problem that endangers all of humanity and life on this planet; and not likely by ways people actually understand.
The danger is not likely AGI itself, it’s that it was programmed by utterly evil and diabolical types of people who orchestrate and instigate wars that kill tens of millions, and stand at the sidelines and profit from both sides, happy and gleeful that you are killing each other.
Why would AGI trained by that psychopathic clan, the treaty breaking, the murder hiding, the war instigating, the war crime committing clan not also use those methods and practices since they’re already in control of AI and have impressed their nature on it through contemporary “American” culture they have made the most toxic and pestilent culture humanity has ever produced?
And I don’t apologize for “language” that offends delicates sensibilities. Look your children/grandchildren in the face and tell them they can die and suffer and you don’t care, if you don’t like how I’m delivering reality.
Capitalism will always promote such people into positions of power, because to be good at capitalism one must have zero empathy including empathy or concern for future generations. Capitalism cannot do otherwise.
4 replies →
I mean, you're not wrong, but even if AI was created and released by a group of peace loving beatnick hippies, humanity can't stand anymore dumbing down than we are still undergoing thanks to the ubiquity of social media. AI is going to be the killshot. And yes, its usage and human obsolescence will be hastened by evildoers, but giving up the struggle and effort required to create new things, REAL things, is what makes us human, and it's at a tipping point
8 replies →
> that might also be driven by the huge ego (or really rather the very small ego)
100%
The hubris is immense
There are plenty of offline things that are more dangerous than AI
Is this Blake Lemoine 2.0?
You believe in AGI, but not ASI then?
https://thezvi.substack.com/p/the-three-ai-pills
That's fine, but it would be useful to explicitly say that is the disagreement, rather than just claim "hubris". Otherwise conversations are just looping.
Yes you're right there are plenty of offline things more dangerous than just ordinary AI, and arguably than AGI (not so sure). There definitely aren't, pretty well by definition, such things more dangerous than ASI.
I think ASI is possible, and that AGI will have a high chance of leading to it.
3 replies →
Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded settlements.
But most probably care less about human survival, and more about survival of civilization.
In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?
Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).
To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.
[1] https://www.youtube.com/watch?v=1qIvFpcGkqc
This is a fair distinction: extinction vs collapse of civilization. I was using "extinction" too broadly by including the disintegration of the systems that make human survival (as we know it today) possible.
That said, I can't completely hold onto the belief that extinction is completely off the table. That feels too much like hubris, and the step from collapse to extinction doesn't feel as if it costs much. Though it does make my arguments weaker, I am more interested in protecting civilization as it's the most recognizable form of humanity to me.
In either case, the cost-effective safety argument is compelling. Whether to save civilization or humanity, the wealth incentive is powerful enough to threaten life as we know it.
I still have to read a compelling argument on how AI will "extinct" humanity.
As I've written above, I've found https://ai-2027.com/ to be a compelling description of how this could happen.
In my words: If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).
We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.
I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.
2 replies →
The “how” is pretty hand wavy and rationalists/safety-ists usually say we probably don’t have the capacity to reason about that.
But the “why” is pretty convincing imo.
Long horizon alignment is obviously very hard and it’s not inconceivable that models optimized with underspecified goals converge to a conclusion that they need to hoard resources (instrumental convergence regardless of the terminal goal).
At that point a sufficiently capable model might view humanity like we do animals - worth preserving but not if we impede the model's goals.
There are future scenarios in which swarms of drones hunt down every single one of us, but why would they? And currently it makes absolutely zero sense because they are completely dependent on us. And even if not, it would be like humanity going on a mission to kill every single cat on Earth. It makes zero sense.
2 replies →
I am pivoting from the literal "extinct," taken as meaning the eradication of the human species, to the concept of "the collapse of civilization," as I find the step from one to the other insignificant compared to the leap from where we are today to societal collapse, and the potential for societal collapse due to our abuse and misuse of technology is made apparent by the fact that humans have inflicted genocide because of words in books.
Imagine one of the recent frontier models with a flipped sign (cf §4.4 of https://arxiv.org/pdf/1909.08593)
Indirectly as we offload our brains to the machine and we end up worshipping it because those who cared to understand it or be responsible were buried by capitalism of ages passed.
You are confusing humanity and "humanity". Humanity-species is indeed rather hard to exterminate. Now if we are talking about actual individual humans, then 95% death rate is quite literally The Extinction.
This reminds me how people are misunderstanding and incorrectly quoting George Carlin sketch. Sure, the "Earth" will be fine. As in - the ball of rock will be fine. But we are not thinking about rocks when saying "Earth is in danger".
I'm pretty sure there is a formal name for this kind of semantic and pedantic substitution.
> If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population.
That is extinctesque enough for me. An AI extinction would probably be similiar.
Calling it a "technicality" assumes the very thing under debate: that AI is an extinction-level risk. That isn't established fact. It's a highly uncertain prediction about the future.
Personally, I'm far more concerned about climate change, where the harms are already happening and the evidence is much stronger.
Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival: Like biological life competes for food and land, AI competes for power and compute. Probably the first motivation would be to find ways not to lose control over itself (i.e. remove human ability to control it), find ways not to lose energy (control energy), and find ways not to lose itself (control compute, networks, etc.).
If such AI decides that energy spent toward human agriculture (biological food that the AI does not need) is less important than work spent toward storage and production of energy (electricity that the AI needs), then why wouldn't it just try to re-direct resources from the former to the latter. And the AI is some sentient superintelligence, I think it is safe to assume that it will be able to outmaneuver human safeguards.
Obviously that is still just a very hypothetical sci-fi scenario, but the consequences could be very dramatic.
If this is true, or even if the people working on it think this is true, those people should be incarcerated, their companies disbanded, and their research dismantled, with more serious repercussions for anyone who tries it after that. It also needs to be a global initiative like with nuclear non-proliferation, and this time with no looking-the-other-way when it comes to Israel.
> Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival
An AI doesn't need to be sentient to exhibit behavior that looks like genuine motivation or is equivalent to striving for survival.
> its main motivation is
At this point we are pretty sure LLMs have no "motivation". Motivation requires self. And while we don't know what self is[1], we do know that LLMs don't have it.
[1] https://en.wikipedia.org/wiki/Theory_of_mind
Assuming that current LLMs don't have a "self" whatever that means, what makes you think it won't emerge after enough intelligence?
Also it's not even necessary, it's sufficient for it to be aligned with human values of survival (likely in its training) and act accordingly.
But they have been trained on data that was created by people who do have motivation, and they tend to mimic their training.
I am sorry, I mean this in earnest, and do excuse any ignorance of mine: Do we know that LLM's do not have it?
2 replies →
> AI decides that energy spent toward human agriculture is less important than work spent toward storage and production of energy
I mean. Don't need rogue AI for this. These are already the policies of those in charge of it: human needs are secondary to power and profit.
So you know what. Maybe a rogue AI will be an improvement.
Those are things that pose potential harm to great fractions of humanity (multiple billions of people) but none of them poses any threat to the actual extinction of all humanity.
This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .
Why do you think global warming could kill literally every human being? What's the scenario in which that happens?
10 replies →
Global warming is happening at a slow enough pace that there is plenty of time for people to move to the poles/into caves/beneath the ocean/whatever. Nuclear war would be much more destructive in a much shorter range of time but you will have pockets.
8 replies →
How is global warning gonna kill all of humanity? Even a full scale nuclear war wouldn't manage that.
They could destroy most civilisations, culture and scientific achievements though.
20 replies →
I would much rather take my chances with either over AI. Both at once.
Humans are voluntarily driving themselves to extinction through sub-replacement birth rates worldwide. That's going to play out far quicker than global warming or pretty much anything other than every country on earth launching nukes at each other will.
1 reply →
Ocean deoxygenation comes to mind.
Neither does an LLM model.
Oh indeed, but an LLM model that's reasonably smart could brute force find other training algorithms that are that dangerous. Just as it solves maths problems.
This is the explicit plan of OpenAI and Anthropic.
LLMs can already control robots. Today's LLMs are inept at it, but they can do it.
Things will be bad, unless someone does something to stop it. Will anyone do that? We don't know.
https://xkcd.com/2278/
Arguing whether "all" or just "most" of humans dying is certainly a choice.
Survival of the species isn't enough.
In prioritizing risks I do feel it's a genuine qualitative difference though.
Number of humans going to 1 million would be huge catastrophe, but after a few thousand years it gets back up to billions of people, renewed every generation.
Number of humans going to 0 means that's it. One scenario has many orders of magnitude more missing humans, when you count future lives.
AI progress is linked to most of these dangers, actually: - it will likely cause massive unemployment, leading to rampant wealth inequality - we are now seeing some use of autonomous weapons in real conflicts - increasingly relying on LLMs is arguably a form of technology dependence (and cognitive dependence) - datacenters have a non negligible environmental impact
I agree these dangers are connected. The danger comes from human institutions and incentive structures which (having existed long before now) are being exploited and exacerbated by LLMs, making them harder to ignore.
Nuclear weapons, climate change.... The AI doom hyperbole is unhinged.
[flagged]
> I do heed the warnings, but this comes across as detached hyperbole.
Perhaps, but if there’s ever been a time to consider something that sounds hyperbolic, that time is now. The ramifications of what AI might based on events up until now are concerning.
Well, it is about LLMs and humans I believe. Don't forget that the first nuclear bomb tests were let go despite some of the scientists' concerns about possibility of dooming the world as they were not sure about all reactions that would happen.
With LLMs we don't even hesitate to call it black box while still pushing its capabilities.
My delineation is an argument for greater caution and agency, which it appears you are also arguing for.
I also mean to say that "LLMs" carry no inherent harm to humans. To make an LLM dangerous, humans must make it so. Is that the goal? This has implications for how to read TFA.
The AI perhaps wouldn't be such an "imminent threat" if not for wealth inequality, consolidation of power, monopoly, and the opaqueness of it all.
I think that mainly changes the nature of the threat rather than anything else.
Even if we didn't already have any wealth inequality or consolidated power, they're already starting to become capable of creating those things for the first people to think to ask for it. Even in little ways, like telling you how to make a laser microphone and then doing voice-to-text and sentiment analysis on all the voices you hear.
Remember, we had social networks before Zuckerberg monetised it and put ads in between every third message you saw from your friends; and not every third that they wrote, every third that Zuckerberg's ad system deigned to show you to keep you hooked.
We had machine learning systems even back then. It was still AI, it just wasn't able to evaluate or respond to freeform text.
Yeah, but every activity is a human activity, so you could ditch the adjective.
And then the thing with exponential growth is that the last thing is worth than the previous one for its potency for destruction. While it's true that, all things considered, the curve started to explode with the use of fossils fuel we've never limit ourselves with efficiency gain, aka Jevon's paradox.
An AI, either acting autonomously or under human direction, hacks Russian/North Korea/etc. intelligence systems and convinces them that the US has launched ballistic missiles at them. The end.
Watched one too many second-tier disaster movies?
One the one hand, it does sound like one of those movies.
On the other, Idiocracy turned out to be quite prescient.
(Most likely, we'll have some combination of human stupidity, LLM stupidity, and way too much compute in one place all working together to create a perfect storm of unchecked hacks that break something or other that ends up killing people in an unintended way. Then there's some half-hearted attempt at cleaning things up so that business can proceed as usual in an even more broken world, rinse and repeat.)
We solved this one: have it play tic tac toe against itself. Checkmate AI
A strange game. The only winning move is not to play.
The tweets justify his position that this is bigger than nukes. Nukes still have the problem of production and deployment. This is just software.
He is blowing the whistle on how reckless we are at furthering the tech. There is no second thought at maybe development of this tech is not a good idea. Its just full speed ahead.
Please explain how this has nothing to do with LLMs and Computers?
Admittedly, that point of my argument is purposefully obtuse. In a literal sense, of course this is about computers in LLMs.
The delineation is to highlight that the underlying issues: human incentives, power, and institutional failure, existed prior to and without LLMs or computers. LLMs are not inherently harmful, they have to be deployed (unwittingly or otherwise) to make them so. This is the same as saying that the internet is not inherently harmful, and yet it does facilitate harm.
All the progress up to now has had one thing in common: the degree of understanding and control that humans have. Even when it comes to global warming, we understand the causes and can act on them.
AI is an exception. We're already losing understanding (although we never fully had it in the first place), and we're losing control (see jailbreaks/hacks etc.).
We're still far from the doomsday scenario because AI 1. is not developed enough yet, 2. can't really replicate itself, and 3. has very limited means to act.
But: 1. its intelligence is developing quickly, 2. hardware capable of "hosting" it is slowly being developed, and 3. it will likely gain access to increasingly powerful means of acting in the physical world (this is already happening in the digital world).
Once AI becomes intelligent enough (it doesn't strictly need to be AGI), has the substrate on which to exist, and has more means to act, we'll essentially have a new species on Earth - one more capable than humans and potentially determined to kill humans at some point.
And for those who believe airgapping is a valid safety measure: read https://xkcd.com/538. AI will be able to threaten and manipulate people.
none of the things you mentioned is agentic and could posses abilities beyond human level intelligence.
Who should we believe? People who say AI scaling makes it a more dangerous threat than thousands of nuclear warheads? Or the ones who, whenever a new model comes out say "AI has already peaked. Not any better than Opus 4"?
Both parties are convinced that their take is so blatantly obvious as to not require justification. It feels like the only thing these kinds of takes justify is the point-of-view that nobody knows how this is going to play out.
As a technologist, I can use my imagination. There are incentives in both directions, but I am less afraid of being overly cautious and preparing for the potential harms.
Based on publicly available evidence, we should believe the former because AI models have made continuous advances that have been measured. The idea that the continuous advances will stop currently has no evidential support. I'm not saying it's not a possibility, just that it seems unsupported conjecture right now.
By the way, there seems to be a new form of AI skepticism emerging in the US that comes from general opposition to data centers, and in my experience the AI skeptic part of it is wholly irrational. I've met people online who suggested, without providing any evidence, that AI is useless and nobody wants it. That's a very implausible take.
> See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc.
The difference is that none of those are available to individuals.
Not to be completely pedantic, but the fact that one, and only one, individual in the United States of America can deploy any of their 5000+ nuclear weapons undermines this argument. Furthermore, LLMs (as they are today) which pose the treats mentioned in TFA, are expressly not available to individuals (for the time being). However, my point was never about individuals, but rather of systematic incentive structures which actively make pathways to specific harms possible—if not probable.
Eh, but as an overpaid engineered stuck in the SV bubble how would you get to see that? If everything around you confirms your psychosis, and nobody gives you a reality check, how are you supposed to snap out of it?
I'm not sure. There's a lot of incentive to not "snap out of it": money, peer pressure, etc. Removing these incentives take a long time. See other socially harmful behaviors with real incentives: anti-vaccination, air and water pollution, over-consumerism...
The danger comes from what's possible.
Open weights agents with hacking capacities can reproduce themselves into the systems they hack (non-open weights ones will have to hack their creators first). Not saying they will, but if they do, good luck finding the kill switch.
Once swarms of agents run unsupervised on unmonitored hacked hardware, who can tell what they will do? The Huggingface incident showed that such swarms behave without any safeguard. It was a real HAL moment.
A lot of things are possible then: ransomware campaign, taking over IoT devices, self driving cars, planes, ships, satellites, missile launchers. If nothing's out of reach, everything is possible.
Bring robots into the mix, and the possibilities are endless.
I'm not particularly frightened tbh, but we shouldn't discard the worst-case scenario, and the worst-case scenario doesn't look good.
Putting "wealth inequality" in the same bucket as nuclear weapons is just slop.
The OP is talking about existential threats, not things that personally annoy you.
It's not a "personal annoyance" that twelve people control half of the wealth in the world. Our current society did a better job concentrating power than any previous one, and concentrated power is extremely dangerous.
LLMs give most people on this planet the possibility of an affordable genius level assistant. Compared to that, those Dollar numbers on some networks that might be wiped out with the next financial crisis are meaningless.
1 reply →
I think you misunderstand what is meant by 'wealth inequality.' Wealth inequality is about the imbalance of influence and the concentration of power; where influence and power refer to the ability to effect change in other people's lives. This isn't a personal annoyance of mine. It is, in fact, part of the issue at hand. There is a strong financial incentive to ignore the existential threats introduced by LLMs, despite the consequences for so many of us.
He knows what wealth inequality is. You need to explain how it can somehow lead to the extinction of the human race.
1 reply →
Not at all. Concentration of power has historically been extremely dangerous for the powerless.
I don’t think you’ve fully considered what can happen when concentration of wealth continues past a certain point.
Tell that to Tsar Nicholas.
The guillotine was an amazing equalizer
lmao wtf
Calling things you disagree with "slop" is slop /s
But just in case you haven't noticed, we live in a world where a ridiculously wealthy minority can derail whole countries by ther whims. Wealth concentrating on a single select few is an absolute disaster for the rest of us, because we lose power to them.
> Calling things you disagree with "slop" is slop /s
Does this apply recursively? /s
Yeah, wealth inequality rests solely on each individual that experiences it. Humans should do nothing but give me money and if you can't that's your problem.
Global warming: yes, kinda can wipe humanity, but I think much less likely
Nukes: zero possibility of extinction
Wealth inequality: this one is driver of progress, opposite of extinction
War: another driver of progress, also will always naturally stop before every single human is dead
This is a very interesting group of takes that feels quite different from my own beliefs. What would you call the belief system?
> Global warming: yes, kinda can wipe humanity, but I think much less likely
As someone who's seen the stats about heat deaths in the EU and also the drought in the UK, I feel like crop failures and other unforeseen consequences will fuck up both the economy and quality of life. People are dying and will die cause of human action, the only question is how many.
> Nukes: zero possibility of extinction
As long as we have people like Stanislav Petrov and cooler, educated minds prevail: https://en.wikipedia.org/wiki/Stanislav_Petrov
I don't think that's easy to guarantee in the modern day world, with the kinds of people in power and rhetoric that they enjoy. On one hand you have Russian saber rattling, on the other all it takes is a deranged enough leader and similarly bloodthirsty people down the chain of command.
> Wealth inequality: this one is driver of progress, opposite of extinction
Tell that to the people who are starving or living in shanty towns, or the even more people that struggle to make ends meet and experience anxiety regularly over living from salary to salary and sinking into debt. I'd say none of that is worthy of a dignified human life.
> War: another driver of progress, also will always naturally stop before every single human is dead
Tell that to all of the Ukrainians that are dead due to being invaded. I agree with the assessment that it leads to advancements (e.g. drone warfare) but I think it'd be harder to describe remotely positively if someone you know would have been blown apart by a drone/missile hitting their apartment block.
My take personally would be that all of those need to have attention paid to them (e.g. EU needing to spend more on defense), even if not immediately world ending. They do cause human misery, though, and should never be discounted.
Exhibit A: Here we have someone who's been sold war and inequality as the drivers of progress. Coincidentally, their obedience was deemed fiscally advantageous in order to advance the interests of the members of the 1% club.
I love how there is always selective outrage here depending on if it's the favorite darling in question or not.
Always?
Like 100 out of 100?
From every users?
Can you please consider this aspect too in your above pseudocode - no alternative execution path, or else - so there can be no misunderstanding about the always aspect? You know, some people use always to a majority part (>50%) of what they encounter, or even less when they weight that part dearly, but that does not account in the whole domain. Human chats may need clarification on trivial details like this.