Comment by garrinm
2 days ago
The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.
What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.
So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.
And I think effectively the same distinction applies here.
I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing.
There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.
It's not a prediction of the next move though, and that is the point. It's a prediction of what will happen if you make that move.
So, it's not a next move predictor. It's a game result predictor.
Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?
3 replies →