Comment by jameshart
2 days ago
I think it’s a disagreement about what ‘predict’ means.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
That seems overly pedantic to me. If I asked you "What's your prediction for the Super Bowl?", I'm pretty confident you would infer that I mean predicting the outcome, not the event itself.
But if you are an NFL coach and I ask you to decide your next action in order to maximize your odds of winning the superbowl, while yes that does involve you having some predictive ability to think about what impact your actions would have on your odds of winning the Super Bowl… I don’t think you would call the process that you use to decide that next action ‘prediction’.
I don't find the fact that I don't call any humans "action predictors" to be a particularly meaningful insight because my rationale is that it's a weird thing to call a human; football coaches can do plenty of other things besides just coaching football.
3 replies →
This is correct for areas where they have been intensively trained to be right, but the training covers a tiny slice of the space of text the LLM must produce and is just adjusting the weights a little. The corpus does still weigh heavily. That’s how they can reliably produce grammatically correct text. That’s also why they sometimes produce nonsense even in domains they are trained on, and more often where there was no training.
For example ask it for a recipe for rock pizza or glue pizza or whatever and if it had not been specifically trained on it or had guardrails introduced, but has some nonsense in its dataset, it will reproduce the nonsense.
> that isn’t really ‘prediction’, so much as ‘generation’
It's both. Generation in a predictive way. That's what G and P in GPT stand for.
Or about what the word “is” means. If you’re Bill Clinton. :-D
yes, it's exactly this. And it's not a trivial distinction.
It’s deterministically trivial.
Post train a model, you'll be able to determine it is not.
18 replies →
Well said!