Comment by red75prime

> If the training data doesn't include lots of text of people being asked questions and saying "I don't know", then it's unlikely to respond "I don't know"

There's a problem with your reasoning. The above applies to a foundational model (that is to an autoregressively pretrained model). If the training data doesn't contain "I don't know" in a dialogue context, then, indeed, the model is extremely unlikely to output "I don't know" when asked. That's the nature of autoregressive training.

But we are dealing with fine-tuned, RLHF-, intruction-, RL-trained models. If the model has "grasped" a concept of knowledge, this concept can be elicited during the mentioned training.