Comment by mrtesthah
14 hours ago
>>When the AI inputs start coming from the AI outputs the wobble is introduced which results inevitably into delirium.
>This is asserted without evidence. There is absolutely no proof that AI requires humans in the loop to function or improve itself.
Nope, it's actually fairly well established.
That paper has had a pretty turbulent reception and looks pretty conclusively wrong at this point.
It used an incorrect theoretical framing that assumed that data was being replaced rather than accumulated as a result of more training (see https://arxiv.org/abs/2404.01413 which explores this). This is incorrect because this simply isn't how real-world datasets are created via synthetic data generation (which generally accumulate more data over time rather than replace their data). As a result most of the theoretical results were invalid.
Empirical evidence has also cast a considerable amount of doubt on the paper. For example, Microsoft Phi-4 was an empirical test in specifically what happens if the majority of your training data is synthetic rather than human and it turns out that Phi-4 did significantly better than previous models which relied primarily on human data.
There's some nuance to all of this in how exactly you do this, but the original claims of the paper are looking really shaky at this point.
We've already seen it start and they had to actively work against it: https://openai.com/index/where-the-goblins-came-from/
The key word in that paper is “indiscriminate”, as in:
> We find that _indiscriminate_ use of model-generated content in training causes irreversible defects in the resulting models
If you view AI training as lossy compression of their training data, then lossily compressing the same data repeatedly will result in data degredation; this is well known from other domains (try repeatedly compressing a JPEG image, for example).
That means it's extremely important that there is some content curation in the loop. But there is no reason to believe this content curation must be done by humans, or that it must exclude all AI-generated content by default.
For example, the recent LLM-generated disproof of the Jacobian conjecture would probably be beneficial to include in the training data, despite being the result of an LLM.