It’s a lossy transformation, so you’re losing information each time. It’s never going to add information.
However, some information is junk that obscures the good stuff. It’s likely that how they train today is very inefficient compared to what’s possible, and there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much.
> there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much
Like, take for example search. Instead of training on a bunch of scraped texts, you take one prompt, select 10 references, and use it to synthesize an answer. Referencing multiple texts gives you more than training on them directly. The LLM could catch contradictions, observe the distribution of human opinions, note if the topic is controversial. And then output a wikipedia-like article. Do this billions of times, and you got a refined dataset. You can iterate on top, using the articles as source and writing meta articles. Or just silly studies like writing a paper about "Characters named Charlie in literature". You can slice and dice the data in any way, and analyze the cross section.
It’s a lossy transformation, so you’re losing information each time. It’s never going to add information.
However, some information is junk that obscures the good stuff. It’s likely that how they train today is very inefficient compared to what’s possible, and there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much.
Papers like this one show what not to do.
> there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much
Like, take for example search. Instead of training on a bunch of scraped texts, you take one prompt, select 10 references, and use it to synthesize an answer. Referencing multiple texts gives you more than training on them directly. The LLM could catch contradictions, observe the distribution of human opinions, note if the topic is controversial. And then output a wikipedia-like article. Do this billions of times, and you got a refined dataset. You can iterate on top, using the articles as source and writing meta articles. Or just silly studies like writing a paper about "Characters named Charlie in literature". You can slice and dice the data in any way, and analyze the cross section.
It'll never add information, but one may think it would be useful to refine information as you feed 'good' model outputs into itself for training.
Floating point precision is not involved (most LLM models still function after floating-point quantization).
I am puzzled that some find this result at all surprising. You simply cannot generate information from nothing.
I'm not surprised you can't use it to make it better, but one might imagine gradients would go to zero as you fed the model its own output.
No, not even close. Gradients don't come to zero in the first place. Training is never perfect.
3 replies →