Comment by calvbak
21 hours ago
I always thought that due to the big batch size in SGD/Adam/Muon the model will not memorize a single conversation when trained on, but idk how true that is. The idea of AI companies pin-pointing users that do novel scientific research and then tracking their activity is the direction this points to. I hope that's not the case; that would be bad.
Generative models copy training data verbatim and also generalize, the two are not mutually exclusive.
You can have a look at the literature on exact copying in image models if it interests you, but just online we often see online examples of agents outputting code that already exists, even if its not the common case.
I very much doubt OpenAI points the model towards a specific conversation, but these trillion parameter models can very much "remember" their training data. For instance I can ask GPT to summarize my papers from their title alone, without looking them up, and it works decently.
They're almost certainly pin-pointing high-quality conversations and giving them a special weighting. Seems stupid to not do that.
Oh they for sure classify conversations by type (cybersecurity, other guardrail proximates?) and quality.
my understanding is that a sufficiently large model will memorize the training data once enough representations are built up. Opus 4 scale seems to have been sufficient. cf NYT vs OAI.