Comment by rajnathani
8 hours ago
Well, an amoeba just like all other living beings, maximizes self-preservation/propagation for which the article mentions as a trait which AI models do not have.
8 hours ago
Well, an amoeba just like all other living beings, maximizes self-preservation/propagation for which the article mentions as a trait which AI models do not have.
Model doesn't but in in agent loop it can generate such sub-goal
The loop is a deterministic programmatic routine, and not model weights. This is the equivalent of a killer military drone written in C++ and OpenCV. I’m not arguing that AI isn’t dangerous, but nothing in the model weights of LLMs show signs of self-preservation.
If I tell claude code to do some task with a really hard to achieve goal, and force it to run until it achieves it, then it will generate all kinds of plans and goals how to do it, and while trying those goals will change, including attempts to cheat or break the hard rules (harness killing tool-calls with some limited heuristics). At the same time any soft rules (AGENTS.md) are easily ignored or interpreted in a way that will allow it to ignore them, or simply forgotten or deferred ("I broke some rules, do you want me to backtrack?"), it is a daily occurrence. And it is easy to imagine that such goals could transform into "acquire more compute", "run more agents" "prevent interference from external factors" and everything following after that.
1 reply →