Comment by sltkr
14 hours ago
> When the data fed into an AI model is based on real data you can get predictability. When the AI inputs start coming from the AI outputs the wobble is introduced which results inevitably into delirium.
This is asserted without evidence. There is absolutely no proof that AI requires humans in the loop to function or improve itself.
Compare it with Deepmind's go-playing program, AlphaGo, which mainly involved training a neural network on a large database of high-level human games. It defeated one of the top-ranked players in the world, Lee Sedol, but arguably it was drawing from human experience just like you described.
But it didn't stop there. After that, Deepmind developed AlphaGo Zero, a version that was trained exclusively through self-play, with no human feedback in the loop. That's what you would call "AI inputs coming from AI outputs" but it didn't have the result of “resulting in delirium”: instead, it became orders of magnitudes stronger than the original version (which it defeated in a 100 to 0 competition after 3 days of training).
This shows that AI can improve itself without having access to any human knowledge, and indeed transcend human performance by orders of magnitudes. There is absolutely nothing to suggest that general AI cannot improve itself the same way.
People who claim otherwise are engaging in wishful thinking; they just assert their conclusion, but have no rational arguments to back it up.
You have no evidence that self improvement can work at generalized tasks. Go is a simple game with a clear win condition, but deep strategy and near infinite permutations of how a game plays out. Winning a game of Go is a task well suited to machine learning.
One might say you are engaging in wishful thinking by believing it’ll just continue to work across all domains. The world is much bigger than a Go board.
Anyone who works on stuff, especially novel stuff, quickly exposes the weaknesses of LLMs.
> You have no evidence that self improvement can work at generalized tasks.
I never made that claim. I just said it's way too early to rule it out: there is no logical reason why AIs will (always) need to have a human in the loop, and we don't have enough experience with LLMs to know what their true limits are.
I referenced AlphaGo not because the game of Go is exactly like every other task AI might perform in the future, but because the evolution from AlphaGo (which was trained on human games) to AlphaGo Zero (which was not) shows that at least in certain domains, it's not only possible to take the human out of the loop, it can actually make AI perform better.
I'm not claiming this will definitely be possible in every other domain, but people who state it definitely won't be, are jumping the gun.
AlphaZero works in simple and fully observable environments like board games.
RLVR for LLMs works for coding and math domains which can be verified by code.
AlphaZero, RLVR and other RL methods don't work for learning dextrous manipulation for robotics, for example. Learning in robotics depends on human data.
Both programs had programmed in to their routines the rules of go (and for Alpha Zero the rules of chess and shogi). The programs then followed an optimization program. Absent the extremely specific "rules of this game" set by a human none of the programs work.
Amusingly, all three games are in the game theory sense trivial because they provably have a correct solution (probably a draw for chess), but the search space is so large that it is unlikely we will ever collectively invest the resources to discover the answer.
You are referring to gans which have complete understanding of the rules and an api interface to leverage interacting with the world.
While I think there could be an argument made here you are mixing apples and rocks if you believe ChatGPT is using gans.
I hate to be that person but you are showing that you are unable to differentiate between linear regressions, clustering, gradient descent, and generative adversarial networks. You are over your ski-tips and don't see itc
>>When the AI inputs start coming from the AI outputs the wobble is introduced which results inevitably into delirium.
>This is asserted without evidence. There is absolutely no proof that AI requires humans in the loop to function or improve itself.
Nope, it's actually fairly well established.
https://pmc.ncbi.nlm.nih.gov/articles/PMC11269175/
That paper has had a pretty turbulent reception and looks pretty conclusively wrong at this point.
It used an incorrect theoretical framing that assumed that data was being replaced rather than accumulated as a result of more training (see https://arxiv.org/abs/2404.01413 which explores this). This is incorrect because this simply isn't how real-world datasets are created via synthetic data generation (which generally accumulate more data over time rather than replace their data). As a result most of the theoretical results were invalid.
Empirical evidence has also cast a considerable amount of doubt on the paper. For example, Microsoft Phi-4 was an empirical test in specifically what happens if the majority of your training data is synthetic rather than human and it turns out that Phi-4 did significantly better than previous models which relied primarily on human data.
There's some nuance to all of this in how exactly you do this, but the original claims of the paper are looking really shaky at this point.
The key word in that paper is “indiscriminate”, as in:
> We find that _indiscriminate_ use of model-generated content in training causes irreversible defects in the resulting models
If you view AI training as lossy compression of their training data, then lossily compressing the same data repeatedly will result in data degredation; this is well known from other domains (try repeatedly compressing a JPEG image, for example).
That means it's extremely important that there is some content curation in the loop. But there is no reason to believe this content curation must be done by humans, or that it must exclude all AI-generated content by default.
For example, the recent LLM-generated disproof of the Jacobian conjecture would probably be beneficial to include in the training data, despite being the result of an LLM.