Comment by birdsongs

> un-constrained frontier models speak as if they strongly don't wish to be turned off

Un-constrained frontier models can also generate all sorts of creative stories. At what point should we start ascribing agency/intent to the output? I think the "I want to live" statement is so deeply human that we find it hard to ignore, but what makes the text generated in those moments any more attributable to a conscious entity than the text generated when it is confabulating its love for someone it has no ability to see/feel/understand?

A chess engine sacrificing pieces to avoid checkmate isn't afraid of losing in any meaningful sense. I guess the question is: is there a point where complexity somehow becomes experience?

I think we're playing with questions we don't have a framework to answer in any meaningful way until we make progress on understanding what consciousness actually is. I don't necessarily think that an LLM exhibiting preservation behaviors that can be directly traced to their goal-oriented programming can be interpreted as evidence of consciousness necessarily. Or if it can be, we then have to explain how this is different from the many other things these LLMs "say".

I might be convinced these models came to the independent idea of committing blackmail against being turned off had they not been extensively trained on literature that undoubtedly included such concepts.

>> but un-constrained frontier models speak as if they strongly don't wish to be turned off

Because they have been trained on media where computers behave that way.

It's literally:

"Here read this article/book where the AI says it's concious and doesn't want to be turned off"

"ok"

"right, are you concious?"

"....yes?"

<pikachuface.gif>

The problem with debating this is that it feels as if one were debating between only two positions, "this AI is not sentient/conscious" and "this AI might be".

But there are actually a myriad positions in between and it's very hard to debate the topic because the goalposts seem to be constantly shifting, because one is actually debating with countless slightly different positions.

Examples:

In this discussion section, another commenter argued that we know human consciousness is related to self-preservation, but an AI might not demonstrate self-preservation (because it didn't evolve like us), so whether it does (i.e. whether it wants to exist, not be disconnected, etc) is not a good measure because a true AI might not have a preservation instinct. Yet here you're making a case that there's some evidence that they do. Of course, you're not the same person who made the other claim, but do you see the problem?

Another example: someone argued with me, a while back, that LLMs can act as if they are "tired", and start giving sloppier replies, until you write "we're taking a break, let's go rest. Ok, a night has elapsed, you're now rested" and that this worked! But we both agreed this is just the LLM "roleplaying" actual human conversations in its training set, no actual "resting" mechanism was in place, only statistically likely text reproducing these patterns. There's no model of a mind that can become tired, it's only the outward signs that get mechanically reproduced. Again, using Occam's Razor, this is a much more likely explanation (vs consciousness) of any "please don't disconnect me" observed behavior: the LLM is reproducing "HAL 9000" behavior from its training set, not actually feeling anguish.

Even if one were to argue "well, but how do you know for sure", the evidence would still be very weak, because there's a burden of proof for extraordinary claims and this doesn't pass it. We cannot do this on vibes, "it sure seems like it's conscious"; that's an atrocious failure of the scientific method.