Comment by Barrin92
3 days ago
> LLM systems are strongly predictive
No, they're not predictive at all. They're generative. A prediction is a forecast based on a theoretical model about a state of affairs in the world. Making a guess what the weather tomorrow will be is a prediction, and a prediction can be true or false. An LLM generates text based on certain parameters, and output it generates aren't true or false, they're simply whatever the system is tuned to produce.
LLMs are terrible at making forecasts or world modelling precisely because they don't actually operate on much of any model or theory, they have no notion of truth or falsehood, they simply generate whatever you want them to generate.
If you'd like to use that definition of "predictive" in this discussion that's 100% fine with me. But, you should do so knowing that in ML and AI "prediction" is a term of art that means "a statistical model's output given some input." LLMs are one example, in that they take as input a sequence of tokens and predict the next token.
In this specific context, the point is that LLMs and MT "predict" lucid-sounding sentences that incorporate linguistic structure (syntactic and semantic) that are well beyond the predictive abilities of the Chomskyan theory of syntax (which is itself a generative model). In other words, the Chomskyan model of syntax is descriptive of much of the rich variation in real-world human language. But it is not predictive in the sense that it does not provide a constructive algorithm to emit lucid-sounding sentences (and does not attempt to, either).
They're generating by prediction. That's how they work - the predict the most plausible next token.