Comment by abeppu
2 years ago
No, bad/wrong/nonsense is not the only risk here. You're missing the main point that the authors are making: the shape of the distribution gets changed by this process. A model trained on human data will produce fewer high-perplexity examples than it was trained on (you can see this in Fig 1b, even between generation 0 and 1). In a literal information theory sense, these perplexity values indicate how much information is in each example. Over successive generations models have less actual information to learn from even if they have the same volume of text.
LLMs are milking us of knowledge and skills, repackage them and give it back to us. Models interact with the internet, humans and code execution. They are exploring. Lots of exploring now happens in the chat room, a place where ideas are first tried out. With billions of users, the volume of information LLMs collect from us is huge. We bring references, guidance and feedback right into its mouth, the LLM doesn't even need to do anything like crawling.
Imagine how many things we know, things we accumulated in our life experience, that were never written down anywhere. That information was lost to others. But now we use LLM assistants, so they get to be in the loop and collect tidbits of human life experience that is not written on the internet. And soon they will also work on audio/video and travel with us everywhere, seeing what we show them.
I think that maybe we are too harsh in expecting LLMs to be perfect. If they are based off of human input that is incorrect then we might propagate such errors. But they will still be quicker and much more reliable than most people. Isn’t this good enough? After all, we are willing to accept flaws in people, even including the president. I suspect that the way forward will be to progressively clean the LLM input data as each error gets identified.