Comment by mcswell
2 years ago
I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM.
In this case of course there are multiple LLMs that are creating text which finds its way to the web, but to the extent that the output of the different LLMs have commonalities, this still seems problematic.
And afaik, there are no metrics or algorithms that reliably distinguish between human-generated and LLM-generated text, at least not for the current generations of LLMs.
What am I missing?
> Training on the output of your system as if it were validated input seems like an obvious no-no.
Imagine a scientist inventing theories without testing anything, and then continuing to build on top. Crazy. Not even humans can create absent some kind of feedback or validation from outside. That's why we invented the scientific method.
Isn’t that how math works in some respects? In that, there’s only a hierarchy of consistency (no absolute consistency) for most of math. And we just keep building and building. We tried the absolute consistency route and found it too limiting.
Maybe that this doesn’t work for LLMs is a sign they aren’t on the path to AGI…
Personally I found LLMs horrendous at this kind of stuff. I’m basically a RLHF peon by trade and if I’m ever needing a quick way to fool a model, I go to simple logical problems, where it can’t lean on external structures, only itself. I don’t mean logical syntax but logical reasoning. I can’t share recent stuff but a just a few months ago the models I work with failed to reason removing 12 cards from a regular deck couldn’t remove an entire suit. That kind of stuff. Why would I want to make my prompt longer and more detailed to provide it extra structure (which is logically superfluous) to ensure it gets the right answer. Im sure a wordy prompt could get it to the right answer. I’m interested in its ability to “reason”, not prompt engineering.
Given that math is devoid of external structure, I wonder if there something to this (it’s at least interesting to speculate)
I think you're right. When I was experimenting with llama 1, I was able to easily observe that with a short prompt and a long response, the response _rapidly_ degraded the longer it went, because it was seeing and amplifying the patterns in its context window so far.
It is intuitively obvious that these problems would get even worse if the garbage output found its way into the training set, and not just into the context window.
It's _relatively_ easy, I think to filter out sites with a large proportion of low quality ai-generated glurge.
Then you're left with a lot of AI generated or assisted content that has quite often been filtered and modified by humans, so that might mitigate some of the problems that cause model collapse because the filtered content _should_ better reflect reality or desirable output?
I mean a fair bit of content on Reddit and Twitter is machine generated now, right? And content on Reddit and Twitter is being used to train new models, right?
You would think so, but people like Sam Altman have suggested that they can use AI-generated data to train their own models. See here:
https://www.nytimes.com/2024/04/06/technology/tech-giants-ha...
At no point should you trust anything Sam Altman says.
Training on ai-generated data isn't a problem, and has been routinely done by everyone for 18 mo +.
The issue is training on 'indiscriminate' ai-generated data. This just leads to more and more degenerate results. No one is doing this however, there is always some kind of filtering to select which generated data to use for training. So the finding of that paper are entirely not surprising, and frankly, intuitive and already well known.