Comment by piperswe
15 days ago
Nowadays the scraped training data is only one part of the training process. The RL phase of training can instill a knowledge of tactics like sub-agent delegation without documentation about them being present in the training corpus. Context isn't the only way for a model with an early knowledge cutoff to know about it.
Still a crutch around incomplete pre-training data, a ton of useful knowledge wont be properly represented on the weights.