Comment by HarHarVeryFunny
1 day ago
Not really - after RL the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. As Karpathy notes, these reinforced token-sequences (from current token out to some future reward many tokens in the future) may even include errors and backtracking, with even the errors being reinforced, so not only is this synthetic data but it may even be incorrect synthetic data being reinforced.
So, I think "steering" is the best description of it - these aren't real training sample continuations being predicted, but rather RL tweaking a whole path of generation probabilities out into the future, with the hope being that if sampling follows this (stochastic) direction then it may eventually lead to a "correct" response.
Thinking of it as goal seeking also provides some insight into the resulting behavior of the model.
> the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training.
Who cares? Still it is next-token prediction.
The question being originally asked is whether "next-token predictor" is the right mental model for an RL-trained model, and I think the answer is no - not only is it not technically correct, but it is a misleading mental model and will lead to incorrect expectations/explanations of what the model is doing.
Calling the base model a next token predictor is accurate since it is literally making a prediction and being given error feedback when that prediction is wrong.
The output probabilities of an RL-trained model are no longer a prediction - they are something else; they don't come from next-token error feedback, but instead come from probability updates from the RL training process that is trying to steer the output towards that future reward.
Why should you care? Well, because an accurate understanding of what the model is helps explain why it behaves as it does. For example, the recent OpenAI hacks into HuggingFace etc wouldn't make much sense if this is was an auto-regressive next-token predictor with a system prompt telling it to be truthful etc, but it is exactly what you expect of a goal seeking system.
> The question being originally asked is whether "next-token predicton" is the right mental model for an RL-trained model,
Regardless, the statement being challenged here is "still next-token prediction, then".
> and I think the answer is no - not only is it not technically correct
It is correct. RL simply adjusts weights - with no effect beyond an equivalent adjustment to the corpus itself. Hence "next-token predictor" remains accurate.
3 replies →