Comment by Workaccount2

I think people give training data too much credit. Obviously it's important, but it also isn't a database of knowledge like it's made out to be.

You can see this in riddles that are obviously in the training set, but older or lighter models still get them wrong. Or situations where the model gets them right, but uses a different method than the ones used in the training set.