Comment by andrewflnr
2 years ago
I think what makes AlphaZero's recursion work is the objective evaluation provided by the game rules. Language models have no access to any such thing. I wouldn't even count user-based metrics of "was this result satisfactory": that still doesn't measure truth.
I generally respect the heck out of Chiang but I think it's silly to expect anyone to be happy feeding a language model's output back into it, unless that output has somehow been modified by the real world.
I don't expect it'll work for everything: as you say, for many topics truth must be measured out in the real world.
But, for a subset of topics, say, math and logic, a minimal set of core principles (axioms) is theoretically sufficient to derive the rest. For such topics, it might actually make sense to feed the output of a (very, very advanced) LLM back into itself. No reference to the real world is needed - only the axioms, and what the model knows (and can prove?) about the mathematical world as derived from those axioms.
Next, what's to say that a model can't "build theory", as hypothesized in this article (via the example of arithmetic)? If the model is fed a large amount of (noisy) experimental data, can it satisfactorily derive a theory that explains all of it, thereby compressing the data down to the theoretical predictions + lossy noise? Could a hypothetical super-model be capable of iteratively deriving more and more accurate models of the world via recursive training, assuming it is given access to the raw experimental data?
> Next, what's to say that a model can't "build theory", as hypothesized in this article
Well for one thing it would stop being a language model; I used that term very deliberately. It would be a different kind of model, not one that (AFAIK) we know how to build yet.
> Language models have no access to any such thing.
And this is exactly why MS is in such a hurry to integrate it into Bing. The feedback loop can be closed by analyzing user interaction. See Nadella’s recent interview about this.
Or if it was accompanied by human-written annotations about the quality of it, which could be used to improve its weightings. Of course it might even be that the only instance of text describing some novel phenomenon available was itself an LLM paraphrase (i.e. the prompt contained novel information but has been lost).
There’s a version of this where the output is mediated by humans. Currently chatgpt has a thumbs up/down UI next to each response. This feedback could serve as a signal for which generated output may be useful for future ingestion. Perhaps OpenAI is already doing this with our thumb signals.