Comment by blobbers

I think you're missing what I'm saying; the LLM can and will have trained on all interesting prose, and it will evaluate all interesting prose. If you indicate that you are goal setting for amazing prose, and you give enough chain of thought, it will self evaluate.

Your chess game example misses the point. If you give an LLM the moves of a partial chess game and ask it to predict the moves of the players, it will predict how the game would have continued, even if it is low level players. But if you ask it how Magnus Carlsen would win the game, it will likely come up with a sequence that can beat most human players, or possibly just decide to invoke a chess solver, which will most definitely win.

The key for your prompts and thinking loops is you need to indicate proper goals. "Write the a story about 1775 England." is different then setting a goal for it to write the a fantastic storyin the style of Dickens about England in 1775.

Regarding the ML portion; what you've written about a 'statistical blend' is a little bit coarse. That would be what you might have gotten from a 2 layer neural net or a regression in 1999; the neural networks today operating with attention or an incredibly high number of parameters are non-linear and non-interpretable, that's the entire point. This isn't a regression of simple parameters, it's all interactions of all parameters deeply netted. Model interpretability vs. Model Prediction power are generally inversely related.

The model can in fact escape itself in that it has many, many, many more parameters than a simple "predictable unpredictability" and that is exactly the point of it.