Comment by pbmonster
4 hours ago
But this is different, right?
The equivalent would be taking a (fully offline) LLM and asking it about the ending of one specific Goosebumps book, and it revealing the twist. And although that specific book was (probably) only once in the training data, a high parameter LLM can usually "remember" the twist.
The only way it would be able to tell you the ending is if it was somehow given more importance in pretraining, loaded into context, or represented in multiple sets of training samples. I have a blog that I make very LLM friendly and usually load posts up into context when I’m working on something relevant. I’ve also opted to improve models for everyone. Despite this, the model can’t recognize my site or any of my posts when I ask it to recall without internet usage (I also turn memory off btw).