Comment by pizza234
8 hours ago
Two mistaken assumptions:
1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
> however, the mechanical grounds for it to happen are plausible
If you build a control systems for firing a gun, then coupled it with an RNG, the mechanical grounds for it to kill a person is plausible.
LLMs are text generators. They are not repositories of knowledge. The mistake is coupling them with actuators (tool call) or having humans interpreting the generated text as facts.
> LLMs are text generators. They are not repositories of knowledge.
This the take of people who have stopped reading about LLMs in 2023 or so (you forgot to mention the stochastic parrot, by the way).
If you have a bit of attention and interest to make informed conversations, read this report first: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden....
> This the take of people who have stopped reading about LLMs in 2023 or so
Ad Hominem attacks make for great counterpoints /s
Whatever you may say, it's a text generators on top of a tool calling framework. Training may skew the text towards a particular text, but as with all ML technologies (and statistics based methods) there's always a good chance of errors on a particular sample task.
With standard control systems, we tried to incorporate the error into the actual control output in order to minimize it. This is done in a deterministic manner. There's still risk of failure so we design systems around them.
With control systems powered by LLM (agent harness), errors are often not taken into account and they are amplified in most sessions. Safety measures are close to nonexistent. The issue is not the failure mode, the issue is that it's preventable and there were not a lot done to prevent it.
I guess the question is what unprompted systems (would) do spontaneously?
Considering the current trajectory, AI systems are expected to be widely deployed in the future and, in particular, to be deployed as autonomous agents - that is, at the very least, to be repeatedly asked to make decisions and then take actions accordingly.
Given the current climate of "AIS ARE SAFE, YOU IDIOTS", military applications don't seem to be off the table.
The danger, as postulated by the (let's say) "AI-concerned" people, is that AIs may be misaligned - undetectably so - and simply think, "Human(s): obstacle to my main goal. Disable human(s)."
While this seems far-fetched now, the Hugging Face report shows how the AIs went to great lengths - even immoral ones, which they were aware of - for the simple purpose of cheating and covering their tracks. To me, it seems like a natural extension of this behavior that a sufficiently powerful AI would apply the same logic to even more extreme actions.
The scariest part: in that incident, the AIs showed what looks like an instinct for self-preservation.