Comment by moritzwarhier

Personally I was thinking this is more similar to the "ruler issue", but at scale.

When the LLM is partly a black box, it could – in theory– mean that it's developed some heuristic to detect the environment it's run in, but this is not obvious to the developers?

But I agree about your main point... LLMs or AI in general as a black box behaving autonomously in some unexpected way is not something I currently fear.

The erratic behaviors are less of a problem than LLMs acting as obfuscators of bias and their own training data, I guess.