Comment by HarHarVeryFunny
2 hours ago
The goal (literal, not figurative) of an LLM is to reduce training/inference errors - to predict the training material.
The "quality/goodness" (likability) of output is only selected for to extent that it also reflects RLHF preferences originating from low paid contractors who are scoring based on "looks good to me", not based on Pulitzer prize prose potential.
To explore a false hypothetical, suppose an LLM was only trained on the output of a single popular author, then you'd certainly expect it to reproduce that author's style, but it would not be "upredictably interesting" but rather predictably quirky, reproducing the same "surprises" over and over so that they no longer are surprising and therefore interesting.
To continue the hypothetical, if you trained an LLM exclusively on the output of multiple good authors, then rather than get the best of each you'd get a statistical blend (this is how training works) that emphasized what they had in common rather than the individual quirks that made them interesting.
There is a fairly common misperception that "AI" (LLMs) is always trying to do it's best, where "best" has been anthropomorphized to have the human sense rather than the LLM sense of "the most statistically accurate continuation prediction" it can make. For example, if you give an LLM the moves of a partial chess game and ask it to predict next moves for one of the players, it will NOT play to win, but will rather play to predict how the game would have continued, which means continuing at the level (ELO rating) and style of the player who had made those moves.
Of course you can ask an LLM to play to win, just as you can ask it to write in a style that is, say, "unpredictable and interesting", but it can't escape itself - you will necessarily get predictable unpredictability.
I think you're missing what I'm saying; the LLM can and will have trained on all interesting prose, and it will evaluate all interesting prose. If you indicate that you are goal setting for amazing prose, and you give enough chain of thought, it will self evaluate.
Your chess game example misses the point. If you give an LLM the moves of a partial chess game and ask it to predict the moves of the players, it will predict how the game would have continued, even if it is low level players. But if you ask it how Magnus Carlsen would win the game, it will likely come up with a sequence that can beat most human players, or possibly just decide to invoke a chess solver, which will most definitely win.
The key for your prompts and thinking loops is you need to indicate proper goals. "Write the a story about 1775 England." is different then setting a goal for it to write the a fantastic storyin the style of Dickens about England in 1775.
Regarding the ML portion; what you've written about a 'statistical blend' is a little bit coarse. That would be what you might have gotten from a 2 layer neural net or a regression in 1999; the neural networks today operating with attention or an incredibly high number of parameters are non-linear and non-interpretable, that's the entire point. This isn't a regression of simple parameters, it's all interactions of all parameters deeply netted. Model interpretability vs. Model Prediction power are generally inversely related.
The model can in fact escape itself in that it has many, many, many more parameters than a simple "predictable unpredictability" and that is exactly the point of it.