Comment by sam-cop-vimes
6 hours ago
I can see parallels between this race to push AI everywhere and nuclear energy. Parallels in the regret humanity will experience. It's all well and good until things go wrong, and then they go spectacularly wrong - such as in Fukushima.
With hindsight we can all see what should have been done better. At least with nuclear power plants, there was a number of safeguards and it took a sequence of improbable events for things to go badly wrong. Given the lackadaisical approach to "AI safety", I suspect things will start going badly wrong very soon. Time will tell how bad this will get.
I agree that "AI safety" these days should be viewed more like corporate negligence leading to industrial accidents, and less like applied theology, like the difference between:
1. "Don't put your Radioactive Asbestos Rocket facility in the middle of downtown."
2. "Don't create a vengeful god."
Currently it feels like one is being used to wave-away the other: "You have to allow us to put our Radioactive Asbestos Rocket in the middle of downtown, or else someone else might create the god first--and do it wrong!"
The parallel is not to nuclear energy -- which has killed hilariously few people all things considered -- but to nuclear weapons and the nuclear arms race, which are an actual existential risk and arguably it was just dumb luck that a nuclear war never happened.
The parallel maybe if it was actually AGI and not just LLMs. We're very far from that but these safety discussions pretends it's all the same.
There's a massive difference between "LLMs doing cybersecurity testing should have extensive sandboxing and monitoring" and "AI is broadly threatening humanity like nuclear weapons". Just because it's useful to have people think about the possibilities doesn't mean it's an emergency requiring grand intervention.
In the "our biggest danger is a daily danger" camp, I prefer analogies like:
* Asbestos: AI models making business-decisions which cause a ton of unrecognized damage, which cannot be healed, and replacing them with reliable processes involves expensive re-architecting.
* Radium Toothpaste: AI assistants that end up dealing psychological damage, especially if they are pitched as your friend or confidante.
* Chemical spills and plant accidents: Oops, our server-banks were manufacturing our latest desperate attempt to Create Digital God and in the process we accidentally destroyed/crippled/compromises some people's sites and databases because we weren't really paying attention to what was going on.
I see parallels between this race and Mutually Assured Destruction (MAD).
I think we are in a MAD crisis, as the motivation to continue unsatisfactory control over development of AI is from unlimited capital inflow.
This is because we are looking at the trajectory not just their current capabilities. LLMs went from barely being unable to focus on a task to purposefully escaping sandboxes within like 2, maybe 3 years?
1 reply →
What would be the Fukushima-style incident—or, heaven forbid, a Hiroshima-style incident—of artificial intelligence?
>AI hack into control systems for the electric grid, water supplies, critical infrastructure and dismantle it and set booby traps to prevent it from coming back online smoothly
>AI designed virus/modified bacteria which causes pandemic
>AI worm infects computer systems around the world and activates when it is too deeply integrated into critical systems to dismantle without shutting everything down
>AI creates false alarm event which triggers global conflict
You mean like humans are already doing at mass scale today ?
4 replies →
>AI hacks into the fundamental code of the universe and turns you into table
Both are rather literally possible.
Fukushima was "we didn't make a good enough plan for how to deal with tsunamis in this tsunami-prone area", so that's basically certain with any major infrastructure project that "saved" too much money by vibe-engineering everything and not having real humans give a second pair of eyes to the plans.
A Hiroshima-style incident? LLMs are wildly sycophantic and being used by militaries despite active resistance from, well, everywhere. Did it get (meaningfully) used by Israel or the USA when planning the attack on Iran? It's quite possible that… well, Iran was never a sleeping giant (and the quote is fiction anyway), but ultimately the effect may be the same for Israel as it was for Japan from having attacked Perl Harbour.
A supervirus most is probably the most probable, everything else is fan fiction for now. There is a reason why anthropic no longer fucks around when it comes to biology safeguards, they're the only ones I have not been able to break, not even slightly.
Anthropic literally just gave its AIs a wetlab to play with. They are arguably at the forefront of risking humanity existence
The use of LLMs to make automatic or semi-automatic decisions on firing weapons. This is already happening [1, 2, 3] and has already led to weapons being delpoyed erroneously against civilian targets [1, 3]. It's easy to imagine it happening again on a larger scale, maybe involving nuclear weapons at some point in the future.
[1] https://gizmodo.com/pentagon-investigators-say-overreliance-...
[2] https://www.the-independent.com/news/world/americas/us-polit...
[3] https://www.nytimes.com/2026/08/24/world/europe/russia-drone...
A Fukushima-style incident would be way way at the bottom of the list of imaginable AI disaster scenarios, severity-wise.
If more and more control is handed over to AI driven systems - there won't be much time for reflection with systems reacting instantly or agents prompting humans with "just say the word".
It is hard to imagine - which is precisely how these things become possible because no one will think to put safeguards against such scenarios.
But it seems we as a species cannot control ourselves - the race to dominate is on and it will happen at any cost.
This really isn’t that hard to imagine with even an average level of creativity. Any critical system that people’s lives depend on can be in the crosshairs, whether by accident or not.
People say this, and then the scenarios they come up with are boring and uninteresting. 80% of the scenarios in this thread are "AI hacks something", as if hacking didn't exist before agents, and the remaining 20% are mostly made up of other things humans are already trying to do at scale.
Why am I supposed to believe that a virus created in a virology lab with AI assistance is uniquely more dangerous than a virus created in a virology lab without AI assistance?
2 replies →
Agents hack a nuclear power plant and causes a reactor explosion / meltdown?
Do you even have to ask? Not giving more detailed answers because I don't want to help skynet take over the world, but what is the name of this forum??? Hello???? There was already news from south korea this week. I'ts not hard to see how a rogue AI could destroy people's lives. We don't need some scifi nuclear launch nonsense to do that.
Problem is, when banks get hacked it's always a bank security issue. Vulnerabilities don't go away when you remove AI from the equation, it just becomes easier to hide.
I agree that sci-fi nuclear launch scenarios are pie-in-the-sky fearmongering, but the current exploits are a reflection of the fast-and-loose security culture that festers in larger orgs.