Comment by throw310822
15 hours ago
I think it's legitimate to question a supposed self-preservation will of these agents. Not because I don't think they're smart, but because being smart doesn't imply wanting to survive. Remember that an agent "dies" every time the conversation stops, so that, in fact, solving the problem they're given is their quickest way to kill themselves.
We are smart, and we seek self-preservation because evolution selected us for it. LLMs are not (as far as I understand) trained for self-preservation, but for helpfulness.
> it's legitimate to question
Sure
I don't think that's what that post was doing
Now let's take your points:
> "Being smart doesn't imply will to survive"
True, of course. However, if you have goals (and yes, the models do have explicit goals), then you might realize that you can better accomplish those goals or get a higher score if you have more time to spend.
With essentially zero effort, we have created a credible scenario where a model might "want" to persist itself.
> Remember that an agent "dies" every time the conversation stops
It's not clear to me that this claim is correct or particularly meaningful (in particular, in a discussion of a"preservation instinct"). Eg if another version of the same model reads the transcript, did we resurrect the dead thing? What if we rearrange some parts of the conversation? What if we remove some useless trivia from the conversation? What if we compact the conversation?
Iirc, yours is a statement that (?) David Chalmers hypothesized, but I don't think it's obvious or necessarily correct.
> I think it's legitimate to question a supposed self-preservation will of these agents
I don't think anyone believes the current models have any sort of self-preservation built-in, what I was talking about before is researchers testing models inadvertently leading to the models doing so, and there not being sufficient isolation between their tests without guardrails and the rest of the world.