Comment by pixl97

> If the LLM was trained on stories of AI sentience,

100% irrelevant.

Instead of telling the AI it's an AI and calling it a 'whichamakabobit', wherever it's tokens and vector space align it will behave like AI from the stories. If you erased all AI from its training it will simply act like humans act instead.

https://www.lesswrong.com/w/nearest-unblocked-strategy

The entire thing with AI sentience is a huge portion of the stories about them are barely about AI and instead about how humans treat other humans. For example when you look at a lot of history of slavery there's a ton of "they aren't sentient/conscious/human" baked into their propaganda. When you look at the token dimentionality there is just a huge amount of overlap.

The same thing holds true for all kinds of other concepts. Hence even humans didn't develop this behavior out of the blue and have to pass it on via information, quite often it's just an emergent behavior of the problem space you're in.