Comment by grey-area

This is correct for areas where they have been intensively trained to be right, but the training covers a tiny slice of the space of text the LLM must produce and is just adjusting the weights a little. The corpus does still weigh heavily. That’s how they can reliably produce grammatically correct text. That’s also why they sometimes produce nonsense even in domains they are trained on, and more often where there was no training.

For example ask it for a recipe for rock pizza or glue pizza or whatever and if it had not been specifically trained on it or had guardrails introduced, but has some nonsense in its dataset, it will reproduce the nonsense.