Comment by danaris
17 hours ago
The LLM doesn't "know" anything, can't reason about its own knowledge, and has no self-awareness. It has training data, and it can use your prompts to synthesize that training data into probable continuations or responses.
If the training data doesn't include lots of text of people being asked questions and saying "I don't know", then it's unlikely to respond "I don't know" when prompted, regardless of whether anything in its training data that might actually answer your question.
> If the training data doesn't include lots of text of people being asked questions and saying "I don't know", then it's unlikely to respond "I don't know"
There's a problem with your reasoning. The above applies to a foundational model (that is to an autoregressively pretrained model). If the training data doesn't contain "I don't know" in a dialogue context, then, indeed, the model is extremely unlikely to output "I don't know" when asked. That's the nature of autoregressive training.
But we are dealing with fine-tuned, RLHF-, intruction-, RL-trained models. If the model has "grasped" a concept of knowledge, this concept can be elicited during the mentioned training.
Did you just make up in your head how LLMs work and then just run with it? There are a zillion explainers out there, at least put in some time before talking about it.
Humans run on training data too in the same broad sense.