Comment by latentsea
7 hours ago
> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
AI can wield itself. All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
I see no reason why that's an absolute impossibility in the long run.
A fire can spread itself. That doesn't mean it's alive or thinking.
And more importantly than that technicality: the fires that affect human lives are almost always directly or indirectly caused or amplified by human (in)action.
Fires are not forming plans to act towards goals. I wouldn’t technically call what AIs are doing thinking in the same way humans think, but it’s analogous, and I think it’s reasonable to consider it a kind of intentional behaviour.
Damaging fires are often unintended consequences, and it’s often the same with the damage caused by rogue AI. That’s the alignment problem.
One thing I wonder about is whether sometimes telling an AI not to do something can put the possibility of doing that thing into its behavioural space. There’s an effect well known to many teachers that if you tell children not to do something, it can increase the chance they will do it. A better approach is to phrase it positively. Instead of “don’t run in the hall” say “please walk in the hall”. I wonder if a similar effect can show up in responses to prompts.
Besides the fact that the subjective criteria needed classify something as "alive" or "thinking" is irrelevant to the point you are replying to, fire doesn't pass the Turing test. That analogy is extremely weak.
The Turing test has been passed for half a century already, and we've been moving goal posts ever since.
The point I was trying to make was this: we've been dealing with autonomous systems that sustain and spread themselves and "act on their own" without our intervention since time immemorial, ranging from lifeless chemical reactions to other humans. Now, the discussion about where exactly AI lies on that spectrum isn't unimportant: suppose we do manage create an artificial mind that "experiences" something, then we are currently on the path to creating a new class of artificial slaves. The ignorance of which is a "fun" bit of cognitive dissonance on the pro-AI side imo (and in general we could have fun with analysing how much the fear of robot/AI uprisings in classical fiction may just be a projection of upper-class fears of an automated worker's class uprising, which would explain a lot of why it's sd US-specific. But that's probably too radical for this particular website).
The point is: it's a lot less important than the fact that no matter what the least-nebulous answer to that question may be, ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that. And we are able to make different choices. And most of the time this discussion about autonomous agency is a wielded as a distraction from that.
3 replies →
A self spreading fire doesn’t need to think to cause massive destruction.
There is also not point if some consider it thinking or not. The important factor is what actions a fire takes and what is the impact on the world.
It's not an absolute impossibility, but most of the power that humans have comes from being able to direct other people, deploy large sums of money and do all of this over extended periods of time. Being intelligent doesn't give you that power automatically.
While true, we're already observing humans just straight asking LLMs to direct them, and also to take their money and spend it on stuff.
Plus every so often we see power-seeking and self-preservation behaviours in LLMs, so it's not like we have zero reason to expect such granted power to be spent on such things:
https://arxiv.org/html/2501.16513
https://ieee-dataport.org/documents/instrumental-power-seeki...
I think we have very good reason to fear big accidents. But some AI algo going rogue isn't exercising real societal power. Power-seeking and self-preservation is nothing special. We all do that. That doesn't mean every group of intelligent people with a bit of money is always on the cusp of world domination.
Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.
2 replies →
I think there are a couple of interesting scenarios where humans essentially “hand over the reins” to AI.
1. A leader rises to power who has instructed an AI “I want to control the world, tell me what to do”, and has some kind of real-time capable reasoning engine in their ear/head. Even this would require monumental advances in physical compute and HMI, but I wouldn’t rule this out within 20 years. However, this still requires a human to make the initial prompt and carry things out. And in such a scenario, there would be others with the same technology to counterbalance. I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.
2. An entire society hands over the reins of governance to AI. Basically some country decides that AI has advanced to the point that it makes better decisions than any human / group of humans, and allows it to govern their country with an objective of “take over the world”. This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.
If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.
1 reply →
We're working very hard to "enable agentic commerce", which is exactly that.
Because there is zero evidence that LLMs/reasoning models can really come up with novel ideas, and it's very likely those will be needed to make substantial progress on AI research. The math results that have been proven have also "just" combined existing ideas or found counterexamples. I'm not saying those capabilities aren't impressive, but I just don't see a clear path to super intelligence with the current paradigm.
> The math results that have been proven have also "just" combined existing ideas or found counterexamples
The Exponential Time Hypothesis one used a novel algorithm though. And the approach was very surprising for experts[1].
> “Some parts [of OpenAI’s proof] are, maybe you could say routine, but then yes, there are two or three alien-looking parts, and it’s exactly these parts that I saw that made me think, wow, this is something that the whole community needs to work on together,” Williams told TechCrunch.
[1] https://techcrunch.com/2026/10/27/frontier-ai-labs-say-their...
No AI would be willing to modify its own goals.
> All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
And be told to do that.