Comment by chrisjj
1 day ago
> the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training.
Who cares? Still it is next-token prediction.
The question being originally asked is whether "next-token predictor" is the right mental model for an RL-trained model, and I think the answer is no - not only is it not technically correct, but it is a misleading mental model and will lead to incorrect expectations/explanations of what the model is doing.
Calling the base model a next token predictor is accurate since it is literally making a prediction and being given error feedback when that prediction is wrong.
The output probabilities of an RL-trained model are no longer a prediction - they are something else; they don't come from next-token error feedback, but instead come from probability updates from the RL training process that is trying to steer the output towards that future reward.
Why should you care? Well, because an accurate understanding of what the model is helps explain why it behaves as it does. For example, the recent OpenAI hacks into HuggingFace etc wouldn't make much sense if this is was an auto-regressive next-token predictor with a system prompt telling it to be truthful etc, but it is exactly what you expect of a goal seeking system.
> The question being originally asked is whether "next-token predicton" is the right mental model for an RL-trained model,
Regardless, the statement being challenged here is "still next-token prediction, then".
> and I think the answer is no - not only is it not technically correct
It is correct. RL simply adjusts weights - with no effect beyond an equivalent adjustment to the corpus itself. Hence "next-token predictor" remains accurate.
Yeah - but what is it adjusting weights based on? It's not based on next token ....
And per the focus of this thread, regardless of how accurate it is, why do you find "next token predictor" to be the most useful mental model?
> why do you find "next token predictor" to be the most useful mental model?
To me it is an accurate description of the algorithm. And a sufficient explanation for the behaviour. So I don't need it or anything else as a mental model.
I accept this does not suffice for people who cannot comprehend the huge amount of processing and data the empowers it. Lacking a factual understanding, they reach for any mental model as a kind of superstition.
It is sufficiently advanced technology which to many is indistinguishable from magic. This disguises its limitations and enables its limitless false promotion to the gullible, being the reason it is so dangerous to individuals and society.
1 reply →