Comment by vickychijwani
9 hours ago
I agree it’s not likely, but I really don’t see how one can dismiss the possibility of immense danger outright. I can think of some scenarios that are not far off from current capability and I wouldn’t be too surprised if the first one occurred within ~1 year from now if there are more “ambitious” unmonitored training runs like OpenAI’s:
Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.
Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown.
Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.
Example 6 is a good one. Iran attacked water infra in the US recently and maybe they would have done a “better” job (from their point of view) had they used Fable.
The “worst case” with 6 is potentially very bad but I think we are currently using advanced AI models to harden systems and patch vulnerabilities more aggressively than anyone is trying to bring down the whole power grid (for example).
I think it’s a potentially harmful case but my take is defensive capabilities are scaling as fast as offensive capabilities but defense is being implemented faster than anyone is going on offense?
Example 7 is Russia and Ukraine right now according to public information. It sounds like entirely autonomous weapons are deployed to the battlefield already. I put this in the “not likely to be a widespread problem” category for now.
How is bringing down the whole power grid in any particular country an extinction level event? I'm pretty sure that even in the worst case scenario it would be like a month of chaos in one particular part of the world at most, hardly something that would have a long-lasting impact on the humankind's ability to survive at large.
If the answer is "they'd at least try to nuke the country that did it in response", then once again, LLMs are not the main threat.
> inherent non-determinism in current AI architectures
There's nothing inherent about non-determinism in transformer architectures. All of it is removable.
Again comes to use of deterministic. Maybe calling AI varyingly chaotic is more helpful but would also be misunderstood. And I use that in meaning of slight changes in input generating large and somewhat unpredictable changes in output...
I see. Can you say more about this? What’s the trade-off of removing it?
You get a probability curve for the next token prediction. You can just pick the highest probability. That said the non-determinism serves a real purpose- it allows different outputs and paths to be explored. So that's kind of the tradeoff.
This explains it pretty well: https://academy.claude.com/courses/building-with-the-claude-...
You made up some cool sci-fi.
Example 5: how does a giant LLM that needs million-dollar server racks just to run, replicate itself over the internet?
We’re not far off from the point where a 30B parameter model could do that and run on not-too-expensive hardware. See recent Qwen releases for example and extrapolate the current rate of progress from there.