Comment by echelon
2 years ago
I'm long on synthetic data.
If you think about evolution and hill climbing, of course it works.
You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima.
RLHF, synthetic data, and enrichment are all we need.
This misunderstands fitness. Its not a sure bet what is most optimal is what you see. “Good enough” given environmental context is what you see. Just like with certain crystal structures in chemistry, you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness).
In other words you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.
> Its not a sure bet what is most optimal is what you see.
I wouldn't ever make "most optimal" a criteria. We're looking for measurable improvements, not a jump to god emperor or apex predator.
> you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness).
Optimization is like that. But unlike genetics, where we can't re-route the recurrent laryngeal nerve or change fundamental biochemistry, these are engineered systems where we can set up wildly different experiments at any time. Just to cite one of many different research threads, there's now research now going into developing models from small scale training data.
> you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.
We can know if the synthetic data is better. We have objective measures, a scientific process, and we'll always be striving for improvement.
Only if you have a valid fitness metric. If you have humans looking at hands, then that's a good metric, as long as you really do have a human in the loop. Any automated metric (eg something that can evaluate hands) is great for measuring that specific dimension of fitness (after all, it was developed by a human, so it's really just an indirect way of feeding the human's evaluation into the machine). But it's useless for any other dimension. It'll happily rate the perfect hand coming out of a dogchickenpeach above the deformed hand petting the perfectly formed dog.
It's the same as any other kind of signal processing. You can increase the noise, but you can't get more signal than you started with.
Here, if the LLM decides that "monkey" is most often followed by "butt" and occasionally by "trainer", then it'll generate synthetic data with those frequencies and training on that data will not change its probability estimates at all. It will, however, drown out the signal that "you are a monkey butt" is more likely than "phlegm cigar monkey butt", if you'll forgive me the liberty of using those phrases to represent statistical correlations just beyond the frontier of what the LLM has learned. The synthetic data will teach it that everything it doesn't already know is equally probable, which will overwhelm human source data in which it isn't.
Synthetic data has to work if we hope to have ML models that can improve themselves in a similar fashion as humans when it comes to advancing knowledge.
They mathematically cannot unless they have access to a way of measuring fitness. One that goes beyond an evaluation based on what they have already learned.
Data created automatically is not the same as human curated data, though both are synthetic. Auto-created data often suffers from a host of demerits (duplication, bias, error, unnatural distribution, irrelevance to learn the intended domain, etc, etc). Human curated data usually avoids these pitfalls, and thus is far more valuable when training -- otherwise all human teachers would be equally good. So auto- vs curated- data are incomparable when training naive neophytes like ML models, or children.
> If you think about evolution and hill climbing, of course it works.
You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced.
Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe from scratch to learn some basic maths.
the equivalent here would be a child learning from a textbook he has written himself.
not sure how effective that would be, if it was his only source of learning.
Well that’s what the TFA is about. If you indiscriminately ingest synthetic data into training - the child learning from their own textbook - the model collapses.
The SOTA is to use a discriminator (often another LLM or ML algo) to select the best output before feeding it into the training data. That’s what OpenAI, Anthropic, et al have been doing. One of them just published a paper about it a few weeks ago.
That isn't synthetic data in any reasonable or meaningful sense of the term.
You could describe a textbook as a synthesis, sure, in a sense which absolutely does not track with the 'synthetic' in 'synthetic data'.
Unless the textbook is AI-generated, and I expect that in 2024, the number of AI-generated textbooks is not zero.
It’s an analogy. The learning materials teachers create for students is very much like synthetic data, it’s just not assembled from algorithmic output.
2 replies →
By this reasoning wouldn’t all information that you didn’t discover yourself be synthetic data?
Yeah and that’s why we call it “standing on the shoulders of giants.” Humans went through tons of trial and error in every facet of life to get where we are today. We kept the stuff that worked and taught it.
But before humans can understand enough language to ingest that synthetic data, they do a lot of their own discovery based training where they learn about the world physically and absorb the language people around them use, kind of like throwing random internet data at an LLM.
https://en.wikipedia.org/wiki/Infinite_monkey_theorem
It's important to climb the right hill.
And to have a very well-tuned sense up vs down when the hill is almost flat...
The paper is not talking about verifiable synthetic data generated by some means other than LLMs.
Are you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.
I think the direct action of a person taking their idea and thoughts and going through it many times (making changes / updates / fixes) fits better than eating something. however, I do think you still some form of validation data to ensure these are good changes.
However, I do get the spirit of the article, that as more information generated online is done by LLms, the validity and use of the output decreases
What exactly is doing the validation?
1 reply →
Not sure why you’re downvoted, I think a comparison with prions seems apt and interesting, and bad protein copies that can replicate is essentially an information process. GAN research in recent years showing how you can sabotage a working dog/cat classifier with a one pixel change feels similar to how the tiniest parts of large systems can sometimes undermine the whole completely, albeit with low probability. And finally, since models will bootstrap models that bootstrap models, inevitably there are already subtle issues out there in the wild that may have an incubation period of many years before the downstream effects are completely clear.
The problem is systemic. People believe that the pursuit of monetary and financial profits by corporations will lead to the creation of benevolent artificial intelligence. I personally think this is essentially a religion because it is obvious that the pursuit of profits can not actually create anything benevolent, let alone intelligence.