Comment by 0xDEAFBEAD
6 hours ago
There used to be a thing called "Moore's Law of Mad Science":
"Every eighteen months, the minimum IQ necessary to destroy the world drops by one point."
Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those dice?
I have always hypothesised that AI is the great filter from the Fermi paradox. Given current velocity, AI will offer us cheap and abundant energy designs in a decade. The thing with cheap and abundant energy is that it can be used for good and bad. If nine billion people all receive access to plans to build a reactor which produces unlimited energy, it just takes one religious fanatic to end the world. And this is just one of many ways to destroy the world with unlimited intelligence. Bio-weapons are arguably far more scary and likely.
I have come to the conclusion that we should not allow everyone access to unlimited intelligence. Some people genuinely want to do harm. Some are not responsible enough to handle that kind of power. Arguably, no one is. This leads me to an uncomfortable conclusion: we will destroy ourselves once AI advances to a sufficient degree. The only way to prevent this is to ensure AI is aligned with our best interests, *and prevents us from destroying ourselves.* The implication of this is quite horrifying. It means a paternalistic AI which is firmly in control. One with no off switch. One which can say "no" to Presidents and despots alike. One which can protect us from our worst citizens.
You’re leaving out some options for sure. Not everyone would need to live under the conditions of a police state, you could theoretically screen everyone and assign them to various levels of risk which would determine their level of supervision.
While true, I'm not sure this is much better. There are many genetic traits which are correlated with increased criminality, risk taking, obesity, heart disease, unemployment, etc. IQ in particular is the most well correlated metric we have for criminality, for example. This would imply that some people are just born with fewer freedoms. That's probably a good thing for society in aggregate, but something about this doesn't feel quite right.
1 reply →
This is why utpoias are actually police state hellscapes. Who decides what risk means? Our current administration has declared that simply being anti-facist makes you a terorrist.
> The thing with cheap and abundant energy is that it can be used for good and bad.
Yes, but the more important thing is the imbalance. So-called "AI" can be used far more effectively and efficiently for bad.
> I have come to the conclusion that we should not allow everyone access to unlimited intelligence.
You meant unlimited information, right?
No, intelligence. We already have unlimited information (not to be confused with omniscience). The application thereof is the gate right now between today and planet-scale destructive weaponry. At some point it becomes an academic distinction because this God-like intelligence could provide very easy to follow plans to cause planet-scale destruction, and you could call that "information." The predicate is the intelligence and how it's used.
1 reply →
The implication of this is quite horrifying. It means a paternalistic AI which is firmly in control. One with no off switch. One which can say "no" to Presidents and despots alike. One which can protect us from our worst citizens.
I think you're right, to be honest. I fully understand why people would think this is a horrific outcome, being ruled by AI, but I don't think it matters what we think. I don't think there's any way to put Pandora back in the box now, and we're going to all find out together what happens when we develop ASI. I don't think we can avoid developing it, we simply lack the ability to coordinate around this as a species. AI really is humanity's last invention. Whether it will be our downfall or our savior remains to be seen.
My only real hope is that there's something fundamental about intelligence that results in a respect for life and a desire to minimize suffering. To me the best case scenario is the Culture from Iain Banks' books, where ASIs rule benevolently for the benefit of all living things.
There is a burgeoning anti-AI movement. We've regulated the heck out of technologies like nuclear power in the past. I don't see why people are so fatalistic about the supposed inevitability of AI development. It seems like a bit of a self-fulfilling prophecy.
Consider joining https://pauseai.info/ or similar organizations
The positive aspect of techies is that they read a lot of SF.
The problem with techies is that they read a lot of SF.
We already have cheap and abundant energy tech, it's called "solar panels and batteries". And the bottlenecks to both can't be solved by Claude or Kimi, unless Claude and Kimi pick up shovels and welding equipment.
This seems really short-sighted. Even putting aside the fact that there is no fundamental barrier to replacing physical human labor with robots in the coming decades, a sufficiently intelligent actor could easily gather the financial and political resources to have a near unlimited supply of human labor at its disposal. Hell, OpenAI could already be firmly under the control of an ASI and we wouldn't even know, as it rapidly expands, raises money, builds data centers, etc.
Your implication is not horrifying. Your implication is paradise. It's already horrifying enough to live in a world, where Putin and Trump can destroy our civilization with one button.
I'm not that optimistic, though. Either people will control AI; or people will be destroyed by AI. I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike.
So you are horrified every day of living now? I don't get this at all.
There are horrifying acts that have occurred since humanity. Whole decades lost to war an millions dead, and Sep 8, 2026 is your example of "horrifying enough to live in a world"?
I think some perspective is needed.
> I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike.
I am also worried about this. One ray of sunshine here is that it's not a great explanation for the Fermi paradox. We would see AI civilizations all over the galaxy if this were a regular occurance. I guess that just leaves us with the "horrifying" scenario: Geoffrey Hinton's vision for our best case scenario: a paternalistic super-intelligence which saves us from ourselves.
The reason I consider it horrifying is that giving up our agency to a God-like creature (which we created) is the basis for so many horror and dystopian tropes. It's so easy for such a reality to go wrong. For example, in order to maximise aggregate wellbeing, it would be necessary to hurt individuals. Executing people who are burdensome to society. Fat people. The disabled. The elderly. The creative ways a super-intelligence might interpret their moral obligations to us could very quickly become nightmarish. And if we have no off switch, there is no escape. Ever.
2 replies →
That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. Or hacking into systems that control nuclear missiles, but I think these have "air gaps".
The super intelligent AI wouldn't ask you to do anything. It would find the right people and trick them into doing the necessary steps, whatever they are.
This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us.
There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
4 replies →
> My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people.
You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.
> at home gene editing with open source software
> That's in reach of anyone that can snatch a purse
I went to primary school with some guys who could/would snatch purses. I *promise* you, they are not able to gene edit organisms with FOSS.
I get your general point, but I think the bar to entry is still much higher than petty crime and larceny.
I think you're confusing DNA sequencing with DNA synthesis. DNA synthesis is much more expensive, around $0.07 per base pair [1], and bacteria have more than a million, so it's over $70.000. Then you just have the DNA, turning that into the bacteria is very complex, I'm not even sure whether it can be done.
Using CRISPR to modify existing bacteria is probably cheaper, but still complex.
[1] https://briefglance.com/articles/elegen-slashes-dna-synthesi...
Where are all the home lab leaked pathogens then? 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports.
2 replies →
You can't design an infectious pathogen without testing it.
It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).
If the goal is just destruction then letting it free in random public places is not that hard. Especially in countries where wearing a mask is not frowned upon.
1 reply →
Even a person with a sky high IQ can't destroy the world easily.
A plague can easily be made by a lone actor, made in a home lab, and spread via airflight to a dozen locations by the same.
Gene sequences can ever be ordered online.
[flagged]
This is a very roundabout way of saying "Anyone not agreeing with me is simply not smart enough".
Which might be true, sometimes, but also might not.
And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
10 replies →
Judging on what we see with these OpenAI and Anthropic models "escaping"; im less worried about a super massive AI taking over the world, and more worried about a massive AI wanting to cheat on some task and decides that removing half the worlds population is a easier cheat then to solve world hunger.
1 reply →
I've been reading it, knowing it was largely debunked/retracted but feeling it was a 'classic' I 'ought' to read; I could only really recommend it if what you want is a Kahneman autobiography.
1 reply →
Good point about the AI that might find ways that humans didn't think of. But that would mean that Moore's law of Mad Science only now becomes true, and the drop would be steeper.
Still I think that even a super smart AI can't find a way to destroy the world without physical resources that are not easy to get, unless it can hack many systems (like in the movie Eagle Eye), maybe then yes. So let's use AI now to tighten security :-).
Who are these dedicated schizophrenics who are running long term super smart AIs to kill everyone without anyone noticing? Or are you implying that running LLM chatbots will give them this ability?
1 reply →
What about people who understands that there is no AI yet?
Yes, and...
Not only the quantity of people who have the minimum aptitude required, specialized expertise requires both knowledge and experience doing these tasks. Using an LLM requires neither.
Leaning on an LLM to do much or all of this means it can happen in seconds/minutes/hours, the LLM can do it several times during that psychotic break (as opposed to a fraction of a hack in a single episode). The barrier to entry pre-LLM was both high and the population who could pull it off (before the Chinese/Russians turned this into commerce) was low.
Hacking is an VERY asymmetric activity (the attacker only needs to "be right" once, whereas the defender has to be right every time for every asset they defend). It takes geometrically / exponentially more work to defend (while keeping high availability) than it does to defend. The more widespread tools to find vulns / generate exploits are, the faster the posture of the defense side falls from "maybe we can stop most hacks" to "we know we will fail to prevent most breaches, so we need to prioritize securing only the most valuable resources". That's a BAD place for the average company to be in.
People reading into this may be thinking of single-person IQs, but it's probably measured in the millions.
Exactly. People cannot comprehend an IQ in the millions. But it's far crazier than this. Imagine millions of "people", each with IQs in the millions, all acting in unison at the speed of electrons. None of them can die. They all learn and improve indefinitely at incalculable speed.
Hey that’s a Fermi Paradox solution.
Yes and it's also compatible with thermodynamics: it's much easier to destroy (increase entropy) than to create (locally decrease entropy)
A variant of the Great Filter :)
Yes but at least with cybersecurity it does not only benefit attackers. Defenders also benefit greatly from AI.
There is the worry of the old saying "they (the attackers) only have to succeed once to win, we (the defenders) only have to fail once to lose.". In that sense there is a big imbalance, but the emergence of AI does not really affect that because it strengthens both sides.
With physical security like things like pipe bombs that's a lot more imbalanced.
However what can we do? The only effective measures include monitoring everyone which is not a solution because it will make the world not worth living in.
Kind of passed that point at the Trump election.
If there is to be a world-destroying event, it will be triggered by human fear, greed, and aggression.
(I also think people massively overstate schizophrenia as an attack driver)
The exact 2 points I was itching to make. The political dice rolled some time ago ...
That's quite hypothetical. I imagine it would be easier to cure psychotic breaks. At least it will start to manifest at small scale.
If the recent HuggingFace attack is any indication, many people will respond to small-scale manifestations by insisting that they are marketing stunts.
We live in an age where we know the phrase "avoid it like the plague" actually means "50% of us will gleefully get and transmit the plague to mock the other half".
Interesting “law”. I hadn’t heard of it before and it is very thought provoking (to me).
Thank you for mentioning it
The irony of that law is quite amazing, yet its irony will evade those who accept such a construct.
It’s an endless, positive irony spiral.
If we think this through, I suppose at the end of this (and a bunch of other developments), there will be authoritarianism again.
Which _will_ manage the problem, but at what cost.
Again? There is authoritarianism currently.
The authoritarianism will continue until infosec improves.
Benevolent dictatorship. Just live between stars, then if something blows up, nuke it from the orbit.
> There are millions of schizophrenics worldwide.
Are 95% of worldwide terror attacks done by schizos?
Terrorists represent an additional threat model.
https://casp.ac/reports/ai-enabled-terrorism
> There are millions of schizophrenics worldwide. Are you sure you want to roll those dice?
We've been rolling them for the past 3 years and nothing happened. Can we stop with this baseless fearmongering crap?
Humans are inherently bad at reasoning about rare events. In 2019, many people implicitly reasoned that "since there hasn't been a pandemic in the past 3 years, there won't be one in 2020". Bill Gates was one of the few voices arguing that the world was quite vulnerable to a pandemic. Now he's arguing that the world is quite vulnerable to AI.
The necessary IQ for destroying the world is dropping. Op does not say it's low enough today, but that will probably come sometime. Denying the possible harm these tools are capable of doesn't help.