Comment by HarHarVeryFunny
2 days ago
I'm not sure that's a useful way to think of it.
RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.
A base model is predicting training sample continuations (copying).
A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.
The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
> The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
So still next-token prediction, then.
Not really - after RL the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. As Karpathy notes, these reinforced token-sequences (from current token out to some future reward many tokens in the future) may even include errors and backtracking, with even the errors being reinforced, so not only is this synthetic data but it may even be incorrect synthetic data being reinforced.
So, I think "steering" is the best description of it - these aren't real training sample continuations being predicted, but rather RL tweaking a whole path of generation probabilities out into the future, with the hope being that if sampling follows this (stochastic) direction then it may eventually lead to a "correct" response.
Thinking of it as goal seeking also provides some insight into the resulting behavior of the model.
> the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training.
Who cares? Still it is next-token prediction.
5 replies →