Comment by HanClinto

Keep in mind that the Prover-Verifier game is not that it's training on AI-generated data (as if to imitate it) -- rather, it's training against a discriminator that verifies for correctness (a calculator) and understandability (a smaller, less-capable language model). You can think of this as a distillation method, but it's not like it's generating large amounts of source data and then retraining on it. This method only works on specific problems where there is an absolute right answer that can be verified with an independent heuristic (in this case, a math calculation).

However, there is a lot of potential in the world of self-play and adversarial-training to improve the quality of our LLMs with true reinforcement learning.

For one recent paper on this topic, also check out SPAG -- I found this one to be fascinating:

https://github.com/Linear95/SPAG

I've been keeping notes on this topic in a WIP paper, and if you'd like to read my (rambling) ravings about it, you can find more info here:

https://github.com/HanClinto/MENTAT

I think that self-play and reinforcement learning are going to absolutely be important for the next level of LLM development. If you use AI-generated data, then you must have an objective metric to verify "goodness". Nothing is free, and simply asking an LLM to rate the quality of its own data is not going to cut it. I think that's the point of the article.