> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models
The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately.
I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that well under control.
The question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way to use web scraping, because you can't know for sure what's human-generated and what's LLM-generated.
I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be.
Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement.
OpenAI are licensing data from sources like the Associated Press.
> Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.
This is a similar problem to what was observed in Diffusion models going "MAD" when trained on synthetic data. https://arxiv.org/abs/2307.01850 . Therefore, going forward AI companies will find it increasingly difficult to get their data by scraping the. web, because web will be full of synthetically generated data.
How do you "discriminate" data gathering at web-scale, though? In my view, everything at web-scale only works because there are no humans in the loop, as repeatedly explained here in basically every thread involving Google or Facebook. Yes, since it's a scientific paper they should have defined their usage of the word, but I see nothing wrong with the basic premise that automation at large-scale implies indiscrimate use of content.
" The people training models seem to have that well under control." The people training the models are not the C-Suite, and that is an element of entropy there is currently zero accounting for.
The paper is interesting, but it seems to focus on iteratively training models on synthetic copies of the same data. Obviously, this is going to cause problems.
They did not address what happens if the model is trained on synthetic data that is distinct from the source corpus.
They make it clear in the paper that their primary "real-world" concern is that it's difficult to distinguish synthetic data from real human interaction when scraping data from the web. This will only get worse over time with our current way of doing things.
How are they supposed to deliberately train on synthetic data when they don't know whether it is (synthetic) or not?
Also, do you not feel that it is presumptuous to dismiss a body of work in a few sentences with a "seems fine to me"?
In this case I wasn't reacting to this specific paper so much as to the widespread idea (at least that I've observed among AI skeptics) that "model collapse" is a huge problem.
I find nothing wrong with your statement. I am curious about the paper's use of "indiscriminate." I read this as "just feed the AI more AI output without care" which one can indeed do deliberately.
Seems to me that deliberate discriminate use should yield better against expectations.
> I don't think the "model collapse" problem is particularly important these days.
I think you might misunderstand what model collapse is. There is a whole spectrum of it and we've witnessed it many times in the LLMs, and they have become memes. A fairly recent example is the Golden Gate Claude[0]. This is mode{,l} collapse. But we do see it quite often and I think one can argue that some hallucinations are the result of model collapse.
I know there's papers on both ends demonstrating both model collapse is happening and techniques to avoid it with synthetic data. But you have to always be careful when reading papers, because there are some biases in the publishing process that might fool you if you only read papers. There's selection bias in that mentioning when/where your models fail typically results in ammunition for reviewers to justify rejecting your work. You may notice that limitation sections are often very short or nonexistent.[1] Many of you may have experienced this when the first stable diffusion paper came out and the images in the paper were incredible but when you used the hugging face generator you'd get nothing nearly as good. Hell, try even now[2]. Can you do better than I did? Sure! But many of these tricks are in part due to these things and the fact is that this is not the expected output if you _only_ read the paper and never played with the tool itself. That there's a big difference between these.
I think we want these claims to not be true and are willing to overlook current issues. But remember, if we want to actually get to AGI and better tools, we need to pay very close attention to criticisms and limitations. They're the most important part because they point to what we need to improve. Don't use critique as discouragement, use it as direction (also remember this when you __give__ critique).
This has happened with much simpler models than LLMs, eg. Google Suggest became noticeably worse when everybody started using Google Suggest to input their queries, because it was trained on real query logs and those query logs started to simply reproduce the output of the Suggest model. SEO and Webspam have similar problems within Google Search.
More broadly, this is a reflection of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." The issue is that any model's purpose is to capture novel, useful data about real human behavior. Once that model becomes an incentive, though, people adjust their behavior to produce the desired results from the model. Authentic behavior disappears, which means there's no useful information content for the model to capture, and future generations of the model instead just reproduce behaviors of the previous generation they were trained on, including quirks. Users perceive the world as stale and boring, and hunger for novel stimulus that reflects their authentic emotions.
You could look at this as a full-employment theorem for entrepreneurs and artists.
Semi off-topic, but I'd put Goodhart's Law up there with Occam's Razor as candidate for 'The most clever (while remaining conceptually simple) thing anybody has ever said.'
It amazes me how often it gets to the heart of a problem.
> Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works.
They got a secret ace in their pocket - chat logs created with human in the loop. Of course those might still have errors, but much fewer. They can infer from a human response if it was accepted or not.
I think OpenAI generates at least 1B sessions per month and 2 Trillion interactive tokens. Those can go into the LLM again for analysis and synthetic content generation, or for RLHF with the whole conversation as guidance. Having access to the following interactions can shed light on previous answers.
Even more, they can correlate chats across days, presumably humans try out LLM ideas in reality and return for iteration. That way LLMs indirectly get real world grounding.
They can't directly train on chat transcripts, because they contain private information and other things you don't want appearing in answers. I doubt they even look at them unless you press the thumbs down, in which case they probably use it in some indirect way.
They might try to look for trends or what questions are popular of course.
Yes -- said another way, if you're an ML researcher and you have human-provided (scraped) data, and an ability to generate synthetic data, then until recently, you had a controllable parameter: how much of your training data for your new model should be synthetic? You can vary this, run multiple experiments, and choose how much synthetic data to use -- and you can vary the specific configs about how that synthetic data is generated.
If synthetic data is mixed into your upstream data sources in a way you cannot control, then your ML team loses a valuable controllable parameter.
Keep in mind that the Prover-Verifier game is not that it's training on AI-generated data (as if to imitate it) -- rather, it's training against a discriminator that verifies for correctness (a calculator) and understandability (a smaller, less-capable language model). You can think of this as a distillation method, but it's not like it's generating large amounts of source data and then retraining on it. This method only works on specific problems where there is an absolute right answer that can be verified with an independent heuristic (in this case, a math calculation).
However, there is a lot of potential in the world of self-play and adversarial-training to improve the quality of our LLMs with true reinforcement learning.
For one recent paper on this topic, also check out SPAG -- I found this one to be fascinating:
I think that self-play and reinforcement learning are going to absolutely be important for the next level of LLM development. If you use AI-generated data, then you must have an objective metric to verify "goodness". Nothing is free, and simply asking an LLM to rate the quality of its own data is not going to cut it. I think that's the point of the article.
If you think about evolution and hill climbing, of course it works.
You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima.
RLHF, synthetic data, and enrichment are all we need.
This misunderstands fitness. Its not a sure bet what is most optimal is what you see. “Good enough” given environmental context is what you see. Just like with certain crystal structures in chemistry, you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness).
In other words you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.
Only if you have a valid fitness metric. If you have humans looking at hands, then that's a good metric, as long as you really do have a human in the loop. Any automated metric (eg something that can evaluate hands) is great for measuring that specific dimension of fitness (after all, it was developed by a human, so it's really just an indirect way of feeding the human's evaluation into the machine). But it's useless for any other dimension. It'll happily rate the perfect hand coming out of a dogchickenpeach above the deformed hand petting the perfectly formed dog.
It's the same as any other kind of signal processing. You can increase the noise, but you can't get more signal than you started with.
Here, if the LLM decides that "monkey" is most often followed by "butt" and occasionally by "trainer", then it'll generate synthetic data with those frequencies and training on that data will not change its probability estimates at all. It will, however, drown out the signal that "you are a monkey butt" is more likely than "phlegm cigar monkey butt", if you'll forgive me the liberty of using those phrases to represent statistical correlations just beyond the frontier of what the LLM has learned. The synthetic data will teach it that everything it doesn't already know is equally probable, which will overwhelm human source data in which it isn't.
Synthetic data has to work if we hope to have ML models that can improve themselves in a similar fashion as humans when it comes to advancing knowledge.
Data created automatically is not the same as human curated data, though both are synthetic. Auto-created data often suffers from a host of demerits (duplication, bias, error, unnatural distribution, irrelevance to learn the intended domain, etc, etc). Human curated data usually avoids these pitfalls, and thus is far more valuable when training -- otherwise all human teachers would be equally good. So auto- vs curated- data are incomparable when training naive neophytes like ML models, or children.
> If you think about evolution and hill climbing, of course it works.
You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced.
Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe from scratch to learn some basic maths.
I think this paper is more focused on figuring out what would happen in the theoretical scenario that most data on the web in the future might be AI generated without being marked as such. As they say,
> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear.
The companies you listed are surely not training the models indiscriminately. In particular they have piles of data for which they can have high confidence that they are written by humans.
Here's Gretel's response to the Nature paper on model collapse. Covers the methodology, and how it's flawed, in detail and highlights a lot of other great synthetic data research.
I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM.
In this case of course there are multiple LLMs that are creating text which finds its way to the web, but to the extent that the output of the different LLMs have commonalities, this still seems problematic.
And afaik, there are no metrics or algorithms that reliably distinguish between human-generated and LLM-generated text, at least not for the current generations of LLMs.
> Training on the output of your system as if it were validated input seems like an obvious no-no.
Imagine a scientist inventing theories without testing anything, and then continuing to build on top. Crazy. Not even humans can create absent some kind of feedback or validation from outside. That's why we invented the scientific method.
Isn’t that how math works in some respects? In that, there’s only a hierarchy of consistency (no absolute consistency) for most of math. And we just keep building and building. We tried the absolute consistency route and found it too limiting.
Maybe that this doesn’t work for LLMs is a sign they aren’t on the path to AGI…
Personally I found LLMs horrendous at this kind of stuff. I’m basically a RLHF peon by trade and if I’m ever needing a quick way to fool a model, I go to simple logical problems, where it can’t lean on external structures, only itself. I don’t mean logical syntax but logical reasoning. I can’t share recent stuff but a just a few months ago the models I work with failed to reason removing 12 cards from a regular deck couldn’t remove an entire suit. That kind of stuff. Why would I want to make my prompt longer and more detailed to provide it extra structure (which is logically superfluous) to ensure it gets the right answer. Im sure a wordy prompt could get it to the right answer. I’m interested in its ability to “reason”, not prompt engineering.
Given that math is devoid of external structure, I wonder if there something to this (it’s at least interesting to speculate)
I think you're right. When I was experimenting with llama 1, I was able to easily observe that with a short prompt and a long response, the response _rapidly_ degraded the longer it went, because it was seeing and amplifying the patterns in its context window so far.
It is intuitively obvious that these problems would get even worse if the garbage output found its way into the training set, and not just into the context window.
It's _relatively_ easy, I think to filter out sites with a large proportion of low quality ai-generated glurge.
Then you're left with a lot of AI generated or assisted content that has quite often been filtered and modified by humans, so that might mitigate some of the problems that cause model collapse because the filtered content _should_ better reflect reality or desirable output?
I mean a fair bit of content on Reddit and Twitter is machine generated now, right? And content on Reddit and Twitter is being used to train new models, right?
Training on ai-generated data isn't a problem, and has been routinely done by everyone for 18 mo +.
The issue is training on 'indiscriminate' ai-generated data. This just leads to more and more degenerate results. No one is doing this however, there is always some kind of filtering to select which generated data to use for training.
So the finding of that paper are entirely not surprising, and frankly, intuitive and already well known.
Good venues include main track NeurIPS, ICML, ACL, e.g.
Nature is notorious for publishing PR pieces that don't reproduce, and their ML theory publishing has been quite poor. They do pretty well on things like AlphaGo, materials science, or weather modeling because it's more in their wheelhouse and the results don't require a deep understanding of info theory or ML practice.
I call this "LLM inbreeding." It's a vicious loop where new models are trained on AI-generated content, resulting in the quality degenerating with each generation.
The article contains no proof of theorem 3.1 and finding counterexamples seems trivial. Adult male weight can be modeled by N(85, 20). You can recursively "train" the model on data it generates without having it collapse. It will stay stationary as long as the samples are large enough.
I believe that counterexample only works in the limit where the sample size goes to infinity. Every finite sample will have μ≠0 almost surely.(Of course μ will still tend to be very close to 0 for large samples, but still slightly off)
So this means the sequence of μₙ will perform a kind of random walk that can stray arbitrarily far from 0 and is almost sure to eventually do so.
Maybe. "Overall, this only shows us how far on average we go from the original distribution, but the process can only ’terminate’ if the estimated variance at a certain generation becomes small enough, i.e. we effectively turn into a delta function." Iiuc, variance is modeled as a random walk that will sooner or later reach on zero. I'm not sure I buy that because the variance "walks" orders of magnitudes slower than the mean and is much more robust for large sample sizes.
3. you still won't get beyond the imititation game boundary without exploration & feedback, i.e. the recursive improvement doomers are, as of now, still wrong
> 1. this is nothing that should surprise anyone who has an intuition on control theory and the evolution of unconstrained markov chains
You don't even need to know what a markov chain is. It is intuitively obvious to anyone with two brain cells to rub together that AI can't improve by eating its own vomit.
"Given that training a single moderately large model produces twice the American lifetime’s worth of CO2 (ref. 15), we opted to not run such an experiment and instead focus on a more realistic setting for a proof of concept."
There are other ways AI can help train other AI that aren't generating data. AI could remove low quality data from a training set. It could assist humans in structuring video, 3D and physics simulation datasets for the best learning results.
So they fine tuned an existing model using its own completions to produce the training set for the next run which uses the fine tuned model as the base. They mention catastrophic forgetting so they are aware of it. I suppose they wanted to get results as quickly as possible but this isn’t an accurate model of reality (pun not intended). They’ve only succeeded in demonstrating something that is well known. If they had made the effort to simulate mitigation of bad data and a growing corpus that included proportionally more synthetic data over time it would have been interesting.
I thought this was fairly obvious. Imperfections would only compound over time. Does anyone remember recursively inter-translating between two languages?
If I'm correct, we generally percieve AI generated data to be indistinguishable from a human sourced data and we don't have a tool to reliably assess whether a text is AI generated.
However, could it be that texts generated by AI models posses some kind of statistical property which causes training to collapse?
Then, would it allow us to use it to detect AI texts?
Maybe this is true test of intelligence instead of "emulating intelligence"?
I can learn from Pythagorus' work, extend it, combine it, apply it, and produce works that are more valuable than the original. Perhaps that gets recognized as important, and others then take that, learn, and repeat the process adding their own experience, increasing the general intelligence.
This is about language models. They include plenty of real-world concepts that are essential to language. But they are not models of intelligence or knowledge or reasoning.
Using generated training data is a good way to ensure that the training includes things that are too obvious to appear in normal writing. (Such as "there are zero giraffes in this photo.") This paper describes the limits of using transformer-generated data to train other transformers.
Conceptually, a LLM is a lossy compression of all of the data it saw during training. If you feed it lossy data, at each iteration you will get poorer and poorer signal and more noise.
Prior generations learned this by copying VHS tapes over and over and making photocopies of photocopies. You can see it today by opening and saving a JPG over and over again.
If they're considering Reddit content to be free of generated material, I've got bad news for them. It's not quite the Chernobyl-grade hole that Pinterest has become, but it's hardly "low background".
I still believe reddit is an amazing source. Any article you read on reddit, chances are the comments are better than the original text. They will debunk the article, present a diversity of reactions, and most importantly, they will be grounded in public opinion unlike the press which caters to money interests.
You just copy-paste a conversation into the LLM and ask for an article. For taste, here is one generated from this very conversation. https://pastebin.com/raw/JFH6PGqg
Given a time snapshot and enough computing power, isn't recursion inevitable? It's like running out of known universe given time x. So then we're back creating data without a prior dataset, which is still a human domain.
It’s a lossy transformation, so you’re losing information each time. It’s never going to add information.
However, some information is junk that obscures the good stuff. It’s likely that how they train today is very inefficient compared to what’s possible, and there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much.
> there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much
Like, take for example search. Instead of training on a bunch of scraped texts, you take one prompt, select 10 references, and use it to synthesize an answer. Referencing multiple texts gives you more than training on them directly. The LLM could catch contradictions, observe the distribution of human opinions, note if the topic is controversial. And then output a wikipedia-like article. Do this billions of times, and you got a refined dataset. You can iterate on top, using the articles as source and writing meta articles. Or just silly studies like writing a paper about "Characters named Charlie in literature". You can slice and dice the data in any way, and analyze the cross section.
If the model collapse means that the text produced by it is not statistically identical to the garbage that fills the Internet - then I guess a collapse is the goal.
Or navel-gazing. In fact, that's one of the classically known flaws. (So well known that it has many names: ivory tower, navel gazing, getting stuck in your own head...)
If you don't compare your thoughts to the outside world, it's easy for them to diverge more and more from reality.
It's important to note that outside world means the actual world, not the thoughts of other humans. You need a way to establish ground truth, which comes from observing the actual outcome of actions and experiments.
As far as I understand Douglas Hofstadter's Godel, Escher, Bach - self-referential recursive structures (strange loops) are the foundation of consciousness (among other interesting things). I've been watching to see if LLM's becoming self-referential actually improves them as opposed to degrades them.
Back when I was getting my econ degree, we were taught about the Ultimatum game, which goes like this: You get two participants who don't know each other and will (ostensibly) never see each other again. You give one of them $100, and they make an offer of some portion of it to the other. If the other accepts, both parties keep their portion - so, if A offers B $20, and B accepts, A keeps $80 and B keeps $20, if B rejects, both parties get nothing. Standard economic theory suggests A can offer $1 and B will accept, because otherwise B gets nothing. Spoiler for those of you who haven't seen how standard economic theory plays out in real life, that's not how the game went - typically, offers below ~$30 or so got rejected, because B was a real feeling person who felt like they were getting screwed and opted to punish A for doing so. The exception to this - the people who would take the $1 offer - were people who had been taught economic theory. It turns out you _could_ screw them over and they'd pat themselves on the backs for being very wise.
The "tragedy of the commons" is another one of those parts of standard economic theory that never actually played out in reality - we've got examples from all over the world of communities implementing practices and often entire belief systems that led them to be responsible stewards of shared resources without requiring unilateral ownership of that resource and singular acquisition of the benefits of that stewardship, and yet first on the lips of every modern capitalist when describing why they're at a disadvantage if they're not the ones polluting the water supply is the tragedy of the commons.
Rebecca Solnit wrote a book, "A Paradise Built in Hell", on how people behave during disasters, and found broadly the same thing - contra the prepper myths, most people most of the time faced with disaster come together to work cooperatively to help each other.
We're a fundamentally social species - we've got smaller brains than Neanderthals did, we're not a particularly tough species, but we're very, very good at cooperating with each other.
Each player can limit the other's income to $0 - the offerer can offer $0 and the receiver can reject any deal.
So then what's optimal? $50 seems obviously fair, but does that mean we ought to reject offers of $49 100% of the time? Not quite, to limit the opponent's expected income for an offer of $49 to $50 instead of the $51 they left for themselves, we can use a mixed strategy that only accepts the offer with probability 50/51. Extending that gives the opponent a benefit curve that is linear as they leave themselves more money up to $50 and then flat at $50 afterwards.
That's good, but we can make it better - if we accept offers for $X<$50 with probability 50/(100-X) - epsilon*(50-X), then their expected benefit curve is smooth and has a peak at $50, which is the most we can expect to make except against a generous opponent.
After all that, playing this game as stated against an unknown opponent there's a lot of uncertainty. Maybe all your opponents are entirely irrational and move at random. Maybe all your opponents have colluded and decided that $66 for the offerer and $34 for the receiver is fair and that's the only deal they'll make. But if you think that random actors in the universe are reasonably intelligent and can discover the equilibrium above with the thought worth putting into this Ultimatum game, the receiver strategy above properly aligns incentives.
> It turns out you _could_ screw them over and they'd
End up with a dollar in their pocket which they otherwise wouldn't have.
The Ultimatum game is a useful insight into human psychology: for one thing, it tells us who thinks that the defector in this equilibrium is better off than a counterfactual cooperator.
Ah, but they have their pride! Ok. My pride is not affected by someone else having 99 bucks they didn't earn, and myself $1 likewise. Maybe that other fellow really needed the money.
I don't know what the hell you're talking about. Your argument is incoherent. If you wanted to allocate the money according to the individual's utility of money, then a rule of thumb of $1 is going to be wrong. You should, given no information, assume that both have the same utility of money and that the utility of money is diminishing, favouring an even split.
You may be interested in some of the foundational papers exploring game theory models similar to the Ultimatum game[1][2]. These are known as Iterated Prisoner's Dilemmas.
It's crazy how most political or economic systems would very obviously collapse in the real world almost instantly without some kind of voluntary moral contract (explicit or implied), yet we've got huge clumps of people demonizing one system or another based on the context of what happens when you implement it in a morally dead societal context.
Like there are a ton of people who smirk at your last paragraph and go "nuh uh, hashtag late stage capitalism"
A hundred percent. I've said this elsewhere, but a primary problem for at least American society at this point is we don't have a commonly-agreed upon moral system other than the market - things like Martin Shkreli buying drugs people need to live and jacking the price up are Bad, but we don't have a common language for describing why it's immoral, whereas our only real common shared language, the market, is basically fine with it as long as it's legal. A lot of the market logic works fine for society within constraints - optimize your costs, but not at the expense of your workers; increase your prices if you can, but don't be a ghoul about it; lobby for your position, but don't just buy a supreme court judge.
If you iterate the game, it’s obvious. I, as the responder, control the proposer’s income. Extend to infinity with knowledge of iteration and you reach symmetry between proposer and responder.
We're shockingly bad at doing this in modern society. Our temporal planning horizon is somewhere between 6 months and 5 years, whereas our lifespans are around 75-80.
...in the real world, A tells B that he "sourced" the deal and therefore deserves a bigger cut and in the real world, B agrees up to a point (the $30 mark). Over time and rounds of playing the game, the A's of the world learn where the line is and optimize to stay on the correct side of it, only testing the other side 1-2% of the time to see if rules/behavior has changed.
This seems extremely interesting, but I don't have the time right now to read this in depth (given I would also need to teach myself a bunch of technical concepts too).
Anyone willing to weigh in with a theoretical intuition? The one in the paper is just a little inaccessible to me right now.
> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models
The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately.
I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that well under control.
The question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way to use web scraping, because you can't know for sure what's human-generated and what's LLM-generated.
I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be.
Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement.
OpenAI are licensing data from sources like the Associated Press.
Andrej Karpathy said this: https://twitter.com/karpathy/status/1797313173449764933
> Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.
10 replies →
This is a similar problem to what was observed in Diffusion models going "MAD" when trained on synthetic data. https://arxiv.org/abs/2307.01850 . Therefore, going forward AI companies will find it increasingly difficult to get their data by scraping the. web, because web will be full of synthetically generated data.
They needed to deal with degenerate data on the Web anyway. It's always been full of trash and spam.
1 reply →
At least some of the LLM generated content will be vetted/selected for by a human being though.
2 replies →
Or to consider the inverse of indiscriminate, selection.
Mutation = bad.
Mutation + selection = good.
(given enough iterations)
wow this is such a good point! Evolution is just that!
How do you "discriminate" data gathering at web-scale, though? In my view, everything at web-scale only works because there are no humans in the loop, as repeatedly explained here in basically every thread involving Google or Facebook. Yes, since it's a scientific paper they should have defined their usage of the word, but I see nothing wrong with the basic premise that automation at large-scale implies indiscrimate use of content.
You can use LLMs to vet the relevancy of the content, so you only select the most useful data. I believe most labs are doing this today.
> I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that well under control.
And you base this on what? Vibes?
Basically yes. Vibes based on reading between the lines of various papers, blog announcements and tweets from people better informed than I am.
" The people training models seem to have that well under control." The people training the models are not the C-Suite, and that is an element of entropy there is currently zero accounting for.
The paper is interesting, but it seems to focus on iteratively training models on synthetic copies of the same data. Obviously, this is going to cause problems.
They did not address what happens if the model is trained on synthetic data that is distinct from the source corpus.
They make it clear in the paper that their primary "real-world" concern is that it's difficult to distinguish synthetic data from real human interaction when scraping data from the web. This will only get worse over time with our current way of doing things.
How are they supposed to deliberately train on synthetic data when they don't know whether it is (synthetic) or not?
Also, do you not feel that it is presumptuous to dismiss a body of work in a few sentences with a "seems fine to me"?
In this case I wasn't reacting to this specific paper so much as to the widespread idea (at least that I've observed among AI skeptics) that "model collapse" is a huge problem.
> All of the big AI labs have been training on synthetic data for at least a year at this point
Curious how you know this and the actual extent of such training.
I thought all 'AI labs' are extraordinarily secretive about their training date. Do you have any inside connections to ' All of the big AI labs' ?
I find nothing wrong with your statement. I am curious about the paper's use of "indiscriminate." I read this as "just feed the AI more AI output without care" which one can indeed do deliberately.
Seems to me that deliberate discriminate use should yield better against expectations.
Came here to say the same. "indiscriminate" doesn't really make sense. It's very deliberate.
However, there is one scenario: Scraping of web data. In that case, AI labs might know what is model generated.
> I don't think the "model collapse" problem is particularly important these days.
I think you might misunderstand what model collapse is. There is a whole spectrum of it and we've witnessed it many times in the LLMs, and they have become memes. A fairly recent example is the Golden Gate Claude[0]. This is mode{,l} collapse. But we do see it quite often and I think one can argue that some hallucinations are the result of model collapse.
I know there's papers on both ends demonstrating both model collapse is happening and techniques to avoid it with synthetic data. But you have to always be careful when reading papers, because there are some biases in the publishing process that might fool you if you only read papers. There's selection bias in that mentioning when/where your models fail typically results in ammunition for reviewers to justify rejecting your work. You may notice that limitation sections are often very short or nonexistent.[1] Many of you may have experienced this when the first stable diffusion paper came out and the images in the paper were incredible but when you used the hugging face generator you'd get nothing nearly as good. Hell, try even now[2]. Can you do better than I did? Sure! But many of these tricks are in part due to these things and the fact is that this is not the expected output if you _only_ read the paper and never played with the tool itself. That there's a big difference between these.
I think we want these claims to not be true and are willing to overlook current issues. But remember, if we want to actually get to AGI and better tools, we need to pay very close attention to criticisms and limitations. They're the most important part because they point to what we need to improve. Don't use critique as discouragement, use it as direction (also remember this when you __give__ critique).
[0] https://arxiv.org/abs/2112.10752
I don't think Golden Gate Claude was related to model collapse. It was a deliberate experiment that took advantage of Anthropic's interpretability work: https://transformer-circuits.pub/2024/scaling-monosemanticit...
3 replies →
How do you envision thwsw companies aee discriminaying ans how many manhours goes into disxriminating an hour?
I fine your optimism here delusional at best.
This has happened with much simpler models than LLMs, eg. Google Suggest became noticeably worse when everybody started using Google Suggest to input their queries, because it was trained on real query logs and those query logs started to simply reproduce the output of the Suggest model. SEO and Webspam have similar problems within Google Search.
More broadly, this is a reflection of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." The issue is that any model's purpose is to capture novel, useful data about real human behavior. Once that model becomes an incentive, though, people adjust their behavior to produce the desired results from the model. Authentic behavior disappears, which means there's no useful information content for the model to capture, and future generations of the model instead just reproduce behaviors of the previous generation they were trained on, including quirks. Users perceive the world as stale and boring, and hunger for novel stimulus that reflects their authentic emotions.
You could look at this as a full-employment theorem for entrepreneurs and artists.
Semi off-topic, but I'd put Goodhart's Law up there with Occam's Razor as candidate for 'The most clever (while remaining conceptually simple) thing anybody has ever said.'
It amazes me how often it gets to the heart of a problem.
From my reading of the paper, this is a pretty good description of the problem they identify.
Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works.
https://openai.com/index/prover-verifier-games-improve-legib...
https://www.anthropic.com/research/claude-character
> Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works.
They got a secret ace in their pocket - chat logs created with human in the loop. Of course those might still have errors, but much fewer. They can infer from a human response if it was accepted or not.
I think OpenAI generates at least 1B sessions per month and 2 Trillion interactive tokens. Those can go into the LLM again for analysis and synthetic content generation, or for RLHF with the whole conversation as guidance. Having access to the following interactions can shed light on previous answers.
Even more, they can correlate chats across days, presumably humans try out LLM ideas in reality and return for iteration. That way LLMs indirectly get real world grounding.
They can't directly train on chat transcripts, because they contain private information and other things you don't want appearing in answers. I doubt they even look at them unless you press the thumbs down, in which case they probably use it in some indirect way.
They might try to look for trends or what questions are popular of course.
5 replies →
This is likely one of the main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss.
3 replies →
Cheese and Chalk.
It is very different to generate synthetic datasets to assist in targeted training , vs ingesting LLM output from web scraping.
Yes -- said another way, if you're an ML researcher and you have human-provided (scraped) data, and an ability to generate synthetic data, then until recently, you had a controllable parameter: how much of your training data for your new model should be synthetic? You can vary this, run multiple experiments, and choose how much synthetic data to use -- and you can vary the specific configs about how that synthetic data is generated.
If synthetic data is mixed into your upstream data sources in a way you cannot control, then your ML team loses a valuable controllable parameter.
4 replies →
Yeah, I raised the same issue before reading your post; ninja'd I am.
I like your "cheese and chalk".
3 replies →
I think this is it.
Generated data is ok if you're curating it to make sure nothing bad, wrong or insensible comes in.
Basically still needs a human in the loop.
10 replies →
Keep in mind that the Prover-Verifier game is not that it's training on AI-generated data (as if to imitate it) -- rather, it's training against a discriminator that verifies for correctness (a calculator) and understandability (a smaller, less-capable language model). You can think of this as a distillation method, but it's not like it's generating large amounts of source data and then retraining on it. This method only works on specific problems where there is an absolute right answer that can be verified with an independent heuristic (in this case, a math calculation).
However, there is a lot of potential in the world of self-play and adversarial-training to improve the quality of our LLMs with true reinforcement learning.
For one recent paper on this topic, also check out SPAG -- I found this one to be fascinating:
https://github.com/Linear95/SPAG
I've been keeping notes on this topic in a WIP paper, and if you'd like to read my (rambling) ravings about it, you can find more info here:
https://github.com/HanClinto/MENTAT
I think that self-play and reinforcement learning are going to absolutely be important for the next level of LLM development. If you use AI-generated data, then you must have an objective metric to verify "goodness". Nothing is free, and simply asking an LLM to rate the quality of its own data is not going to cut it. I think that's the point of the article.
I'm long on synthetic data.
If you think about evolution and hill climbing, of course it works.
You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima.
RLHF, synthetic data, and enrichment are all we need.
This misunderstands fitness. Its not a sure bet what is most optimal is what you see. “Good enough” given environmental context is what you see. Just like with certain crystal structures in chemistry, you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness).
In other words you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.
1 reply →
Only if you have a valid fitness metric. If you have humans looking at hands, then that's a good metric, as long as you really do have a human in the loop. Any automated metric (eg something that can evaluate hands) is great for measuring that specific dimension of fitness (after all, it was developed by a human, so it's really just an indirect way of feeding the human's evaluation into the machine). But it's useless for any other dimension. It'll happily rate the perfect hand coming out of a dogchickenpeach above the deformed hand petting the perfectly formed dog.
It's the same as any other kind of signal processing. You can increase the noise, but you can't get more signal than you started with.
Here, if the LLM decides that "monkey" is most often followed by "butt" and occasionally by "trainer", then it'll generate synthetic data with those frequencies and training on that data will not change its probability estimates at all. It will, however, drown out the signal that "you are a monkey butt" is more likely than "phlegm cigar monkey butt", if you'll forgive me the liberty of using those phrases to represent statistical correlations just beyond the frontier of what the LLM has learned. The synthetic data will teach it that everything it doesn't already know is equally probable, which will overwhelm human source data in which it isn't.
Synthetic data has to work if we hope to have ML models that can improve themselves in a similar fashion as humans when it comes to advancing knowledge.
1 reply →
Data created automatically is not the same as human curated data, though both are synthetic. Auto-created data often suffers from a host of demerits (duplication, bias, error, unnatural distribution, irrelevance to learn the intended domain, etc, etc). Human curated data usually avoids these pitfalls, and thus is far more valuable when training -- otherwise all human teachers would be equally good. So auto- vs curated- data are incomparable when training naive neophytes like ML models, or children.
> If you think about evolution and hill climbing, of course it works.
You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced.
Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe from scratch to learn some basic maths.
8 replies →
https://en.wikipedia.org/wiki/Infinite_monkey_theorem
It's important to climb the right hill.
1 reply →
The paper is not talking about verifiable synthetic data generated by some means other than LLMs.
Are you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.
5 replies →
I think this paper is more focused on figuring out what would happen in the theoretical scenario that most data on the web in the future might be AI generated without being marked as such. As they say,
> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear.
The companies you listed are surely not training the models indiscriminately. In particular they have piles of data for which they can have high confidence that they are written by humans.
Here's Gretel's response to the Nature paper on model collapse. Covers the methodology, and how it's flawed, in detail and highlights a lot of other great synthetic data research.
https://towardsdatascience.com/addressing-concerns-of-model-...
I must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM.
In this case of course there are multiple LLMs that are creating text which finds its way to the web, but to the extent that the output of the different LLMs have commonalities, this still seems problematic.
And afaik, there are no metrics or algorithms that reliably distinguish between human-generated and LLM-generated text, at least not for the current generations of LLMs.
What am I missing?
> Training on the output of your system as if it were validated input seems like an obvious no-no.
Imagine a scientist inventing theories without testing anything, and then continuing to build on top. Crazy. Not even humans can create absent some kind of feedback or validation from outside. That's why we invented the scientific method.
Isn’t that how math works in some respects? In that, there’s only a hierarchy of consistency (no absolute consistency) for most of math. And we just keep building and building. We tried the absolute consistency route and found it too limiting.
Maybe that this doesn’t work for LLMs is a sign they aren’t on the path to AGI…
Personally I found LLMs horrendous at this kind of stuff. I’m basically a RLHF peon by trade and if I’m ever needing a quick way to fool a model, I go to simple logical problems, where it can’t lean on external structures, only itself. I don’t mean logical syntax but logical reasoning. I can’t share recent stuff but a just a few months ago the models I work with failed to reason removing 12 cards from a regular deck couldn’t remove an entire suit. That kind of stuff. Why would I want to make my prompt longer and more detailed to provide it extra structure (which is logically superfluous) to ensure it gets the right answer. Im sure a wordy prompt could get it to the right answer. I’m interested in its ability to “reason”, not prompt engineering.
Given that math is devoid of external structure, I wonder if there something to this (it’s at least interesting to speculate)
I think you're right. When I was experimenting with llama 1, I was able to easily observe that with a short prompt and a long response, the response _rapidly_ degraded the longer it went, because it was seeing and amplifying the patterns in its context window so far.
It is intuitively obvious that these problems would get even worse if the garbage output found its way into the training set, and not just into the context window.
It's _relatively_ easy, I think to filter out sites with a large proportion of low quality ai-generated glurge.
Then you're left with a lot of AI generated or assisted content that has quite often been filtered and modified by humans, so that might mitigate some of the problems that cause model collapse because the filtered content _should_ better reflect reality or desirable output?
I mean a fair bit of content on Reddit and Twitter is machine generated now, right? And content on Reddit and Twitter is being used to train new models, right?
You would think so, but people like Sam Altman have suggested that they can use AI-generated data to train their own models. See here:
https://www.nytimes.com/2024/04/06/technology/tech-giants-ha...
At no point should you trust anything Sam Altman says.
Training on ai-generated data isn't a problem, and has been routinely done by everyone for 18 mo +.
The issue is training on 'indiscriminate' ai-generated data. This just leads to more and more degenerate results. No one is doing this however, there is always some kind of filtering to select which generated data to use for training. So the finding of that paper are entirely not surprising, and frankly, intuitive and already well known.
The source code that accompanies the paper is available in a zip file here: https://zenodo.org/records/10866595
I copied that into a Gist to make it easier to browse here: https://gist.github.com/simonw/b3ab1588a681dda821da9fb57290d...
A lot of these papers are wrong. They do something wrong in their setup and then claim their conclusion shows show general truth.
Publishing in nature in ML can actually be a red flag, because they're really not well equipped to evaluate a lot of claims.
The latest llama model got a lot of its data using labels from llama2, and every frontier lab is talking about self training as the future.
Who are "they"? And do you actually believe the practice of publishing unvetted preprints is a good thing in ML research?
Non sequitur? I never said that.
Good venues include main track NeurIPS, ICML, ACL, e.g.
Nature is notorious for publishing PR pieces that don't reproduce, and their ML theory publishing has been quite poor. They do pretty well on things like AlphaGo, materials science, or weather modeling because it's more in their wheelhouse and the results don't require a deep understanding of info theory or ML practice.
1 reply →
I call this "LLM inbreeding." It's a vicious loop where new models are trained on AI-generated content, resulting in the quality degenerating with each generation.
I like this analogy. With the Cambrian explosion of LLMs, we are getting into safe territory, aren't we? Aren't we?
The article contains no proof of theorem 3.1 and finding counterexamples seems trivial. Adult male weight can be modeled by N(85, 20). You can recursively "train" the model on data it generates without having it collapse. It will stay stationary as long as the samples are large enough.
I believe that counterexample only works in the limit where the sample size goes to infinity. Every finite sample will have μ≠0 almost surely.(Of course μ will still tend to be very close to 0 for large samples, but still slightly off)
So this means the sequence of μₙ will perform a kind of random walk that can stray arbitrarily far from 0 and is almost sure to eventually do so.
Fair point about the mean, but I don't see how the random walk causes the standard deviation to shrink towards zero.
2 replies →
Does the Supplementary Information (starting on p. 4, for example) help?
https://static-content.springer.com/esm/art%3A10.1038%2Fs415...
In your counterexample, can you quantify "as long as the samples are large enough"? How many samples do you need to keep the s.d. from shrinking?
Maybe. "Overall, this only shows us how far on average we go from the original distribution, but the process can only ’terminate’ if the estimated variance at a certain generation becomes small enough, i.e. we effectively turn into a delta function." Iiuc, variance is modeled as a random walk that will sooner or later reach on zero. I'm not sure I buy that because the variance "walks" orders of magnitudes slower than the mean and is much more robust for large sample sizes.
It should be noted that
1. this is nothing that should surprise anyone who has an intuition on control theory and the evolution of unconstrained markov chains
2. there appear to be relatively easy mitigations https://news.ycombinator.com/item?id=41061085 (made a separate post because it might be of independent interest to discuss)
3. you still won't get beyond the imititation game boundary without exploration & feedback, i.e. the recursive improvement doomers are, as of now, still wrong
> 1. this is nothing that should surprise anyone who has an intuition on control theory and the evolution of unconstrained markov chains
You don't even need to know what a markov chain is. It is intuitively obvious to anyone with two brain cells to rub together that AI can't improve by eating its own vomit.
I've been telling people this for the past few years. They would like to find out the hard way what control theorists already know.
Nature published a computer science paper???!
"Given that training a single moderately large model produces twice the American lifetime’s worth of CO2 (ref. 15), we opted to not run such an experiment and instead focus on a more realistic setting for a proof of concept."
Dunno why people posted this around so much when it’s hard to take it seriously when you read this.
There are other ways AI can help train other AI that aren't generating data. AI could remove low quality data from a training set. It could assist humans in structuring video, 3D and physics simulation datasets for the best learning results.
So they fine tuned an existing model using its own completions to produce the training set for the next run which uses the fine tuned model as the base. They mention catastrophic forgetting so they are aware of it. I suppose they wanted to get results as quickly as possible but this isn’t an accurate model of reality (pun not intended). They’ve only succeeded in demonstrating something that is well known. If they had made the effort to simulate mitigation of bad data and a growing corpus that included proportionally more synthetic data over time it would have been interesting.
I thought this was fairly obvious. Imperfections would only compound over time. Does anyone remember recursively inter-translating between two languages?
If I'm correct, we generally percieve AI generated data to be indistinguishable from a human sourced data and we don't have a tool to reliably assess whether a text is AI generated.
However, could it be that texts generated by AI models posses some kind of statistical property which causes training to collapse? Then, would it allow us to use it to detect AI texts?
Maybe this is true test of intelligence instead of "emulating intelligence"?
I can learn from Pythagorus' work, extend it, combine it, apply it, and produce works that are more valuable than the original. Perhaps that gets recognized as important, and others then take that, learn, and repeat the process adding their own experience, increasing the general intelligence.
This is about language models. They include plenty of real-world concepts that are essential to language. But they are not models of intelligence or knowledge or reasoning.
Using generated training data is a good way to ensure that the training includes things that are too obvious to appear in normal writing. (Such as "there are zero giraffes in this photo.") This paper describes the limits of using transformer-generated data to train other transformers.
Conceptually, a LLM is a lossy compression of all of the data it saw during training. If you feed it lossy data, at each iteration you will get poorer and poorer signal and more noise.
Prior generations learned this by copying VHS tapes over and over and making photocopies of photocopies. You can see it today by opening and saving a JPG over and over again.
Related ongoing thread:
The problem of 'model collapse': how a lack of human data limits AI progress - https://news.ycombinator.com/item?id=41058867 - July 2024 (6 comments)
Which is good background to this story about Reddit locking down robots.txt and trying to get money from the AI teams scraping their content.
https://news.ycombinator.com/item?id=41057033
If they're considering Reddit content to be free of generated material, I've got bad news for them. It's not quite the Chernobyl-grade hole that Pinterest has become, but it's hardly "low background".
I still believe reddit is an amazing source. Any article you read on reddit, chances are the comments are better than the original text. They will debunk the article, present a diversity of reactions, and most importantly, they will be grounded in public opinion unlike the press which caters to money interests.
You just copy-paste a conversation into the LLM and ask for an article. For taste, here is one generated from this very conversation. https://pastebin.com/raw/JFH6PGqg
2 replies →
Sure. I think Reddit is aware though, that time is running out to get paid for whatever human generated content is there that isn't already scraped.
1 reply →
Given a time snapshot and enough computing power, isn't recursion inevitable? It's like running out of known universe given time x. So then we're back creating data without a prior dataset, which is still a human domain.
Is this an artifact of floating point precision or a fundamental mathematical truth.
It’s a lossy transformation, so you’re losing information each time. It’s never going to add information.
However, some information is junk that obscures the good stuff. It’s likely that how they train today is very inefficient compared to what’s possible, and there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much.
Papers like this one show what not to do.
> there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much
Like, take for example search. Instead of training on a bunch of scraped texts, you take one prompt, select 10 references, and use it to synthesize an answer. Referencing multiple texts gives you more than training on them directly. The LLM could catch contradictions, observe the distribution of human opinions, note if the topic is controversial. And then output a wikipedia-like article. Do this billions of times, and you got a refined dataset. You can iterate on top, using the articles as source and writing meta articles. Or just silly studies like writing a paper about "Characters named Charlie in literature". You can slice and dice the data in any way, and analyze the cross section.
It'll never add information, but one may think it would be useful to refine information as you feed 'good' model outputs into itself for training.
Floating point precision is not involved (most LLM models still function after floating-point quantization).
I am puzzled that some find this result at all surprising. You simply cannot generate information from nothing.
I'm not surprised you can't use it to make it better, but one might imagine gradients would go to zero as you fed the model its own output.
4 replies →
I don't see how this hurts training unless you hurl all hallucinations back at the model.
Alpha zero used a similar approach where it trained against itself and that only made it better. I don't think collapse is real.
If the model collapse means that the text produced by it is not statistically identical to the garbage that fills the Internet - then I guess a collapse is the goal.
Of course it will collapse if you don’t verify it, I remember OpenAI talking about its research into having a different model verify that data somehow
Seems analogous to the effect of echo chambers on humans
Or navel-gazing. In fact, that's one of the classically known flaws. (So well known that it has many names: ivory tower, navel gazing, getting stuck in your own head...)
If you don't compare your thoughts to the outside world, it's easy for them to diverge more and more from reality.
It's important to note that outside world means the actual world, not the thoughts of other humans. You need a way to establish ground truth, which comes from observing the actual outcome of actions and experiments.
you are right, navel-gazing describes it perfectly
I Am Sitting In A Room https://en.wikipedia.org/wiki/I_Am_Sitting_in_a_Room
Very interesting. But wouldn't human preferences still find their way into the datasets of the future?
It’s like the ai generated version of index funds.
There's a complexity missing there. It's like the effects of incest upon dna. Or an echo chamber upon conversation.
As far as I understand Douglas Hofstadter's Godel, Escher, Bach - self-referential recursive structures (strange loops) are the foundation of consciousness (among other interesting things). I've been watching to see if LLM's becoming self-referential actually improves them as opposed to degrades them.
The interesting thing about loops is that they can generate fields (think motion if current generating a magnetic field).
Consciousness is more like a field than like a particle (which are also fields), but we haven’t determined how conscious fields fit in physics models.
Data in --> slop out.
Slop in --> yikes
"Breathing in your own exhaust can be fatal"
[dead]
No say it ain't so /s
I wrote this over a year ago about this. Don't build a city on rock and roll. Don't build a business on a fractal.
https://theluddite.org/#!post/the-snake-eats-itself
[flagged]
Back when I was getting my econ degree, we were taught about the Ultimatum game, which goes like this: You get two participants who don't know each other and will (ostensibly) never see each other again. You give one of them $100, and they make an offer of some portion of it to the other. If the other accepts, both parties keep their portion - so, if A offers B $20, and B accepts, A keeps $80 and B keeps $20, if B rejects, both parties get nothing. Standard economic theory suggests A can offer $1 and B will accept, because otherwise B gets nothing. Spoiler for those of you who haven't seen how standard economic theory plays out in real life, that's not how the game went - typically, offers below ~$30 or so got rejected, because B was a real feeling person who felt like they were getting screwed and opted to punish A for doing so. The exception to this - the people who would take the $1 offer - were people who had been taught economic theory. It turns out you _could_ screw them over and they'd pat themselves on the backs for being very wise.
The "tragedy of the commons" is another one of those parts of standard economic theory that never actually played out in reality - we've got examples from all over the world of communities implementing practices and often entire belief systems that led them to be responsible stewards of shared resources without requiring unilateral ownership of that resource and singular acquisition of the benefits of that stewardship, and yet first on the lips of every modern capitalist when describing why they're at a disadvantage if they're not the ones polluting the water supply is the tragedy of the commons.
This reminds me of Lord of the Flies. The real version of the events turned out very differently.
https://www.newsweek.com/real-lord-flies-true-story-boys-isl...
Rebecca Solnit wrote a book, "A Paradise Built in Hell", on how people behave during disasters, and found broadly the same thing - contra the prepper myths, most people most of the time faced with disaster come together to work cooperatively to help each other.
We're a fundamentally social species - we've got smaller brains than Neanderthals did, we're not a particularly tough species, but we're very, very good at cooperating with each other.
Each player can limit the other's income to $0 - the offerer can offer $0 and the receiver can reject any deal.
So then what's optimal? $50 seems obviously fair, but does that mean we ought to reject offers of $49 100% of the time? Not quite, to limit the opponent's expected income for an offer of $49 to $50 instead of the $51 they left for themselves, we can use a mixed strategy that only accepts the offer with probability 50/51. Extending that gives the opponent a benefit curve that is linear as they leave themselves more money up to $50 and then flat at $50 afterwards.
That's good, but we can make it better - if we accept offers for $X<$50 with probability 50/(100-X) - epsilon*(50-X), then their expected benefit curve is smooth and has a peak at $50, which is the most we can expect to make except against a generous opponent.
After all that, playing this game as stated against an unknown opponent there's a lot of uncertainty. Maybe all your opponents are entirely irrational and move at random. Maybe all your opponents have colluded and decided that $66 for the offerer and $34 for the receiver is fair and that's the only deal they'll make. But if you think that random actors in the universe are reasonably intelligent and can discover the equilibrium above with the thought worth putting into this Ultimatum game, the receiver strategy above properly aligns incentives.
> It turns out you _could_ screw them over and they'd
End up with a dollar in their pocket which they otherwise wouldn't have.
The Ultimatum game is a useful insight into human psychology: for one thing, it tells us who thinks that the defector in this equilibrium is better off than a counterfactual cooperator.
Ah, but they have their pride! Ok. My pride is not affected by someone else having 99 bucks they didn't earn, and myself $1 likewise. Maybe that other fellow really needed the money.
I don't know what the hell you're talking about. Your argument is incoherent. If you wanted to allocate the money according to the individual's utility of money, then a rule of thumb of $1 is going to be wrong. You should, given no information, assume that both have the same utility of money and that the utility of money is diminishing, favouring an even split.
Indeed. You’re very wise!
You may be interested in some of the foundational papers exploring game theory models similar to the Ultimatum game[1][2]. These are known as Iterated Prisoner's Dilemmas.
---
[1] The Evolution of Cooperation (https://ee.stanford.edu/~hellman/Breakthrough/book/pdfs/axel...)
[2] Evolutionary Dynamics of Spatial Games (https://www.sciencedirect.com/science/article/abs/pii/016727...)
It's crazy how most political or economic systems would very obviously collapse in the real world almost instantly without some kind of voluntary moral contract (explicit or implied), yet we've got huge clumps of people demonizing one system or another based on the context of what happens when you implement it in a morally dead societal context.
Like there are a ton of people who smirk at your last paragraph and go "nuh uh, hashtag late stage capitalism"
A hundred percent. I've said this elsewhere, but a primary problem for at least American society at this point is we don't have a commonly-agreed upon moral system other than the market - things like Martin Shkreli buying drugs people need to live and jacking the price up are Bad, but we don't have a common language for describing why it's immoral, whereas our only real common shared language, the market, is basically fine with it as long as it's legal. A lot of the market logic works fine for society within constraints - optimize your costs, but not at the expense of your workers; increase your prices if you can, but don't be a ghoul about it; lobby for your position, but don't just buy a supreme court judge.
If you iterate the game, it’s obvious. I, as the responder, control the proposer’s income. Extend to infinity with knowledge of iteration and you reach symmetry between proposer and responder.
> If you iterate the game, it’s obvious.
We're shockingly bad at doing this in modern society. Our temporal planning horizon is somewhere between 6 months and 5 years, whereas our lifespans are around 75-80.
...in the real world, A tells B that he "sourced" the deal and therefore deserves a bigger cut and in the real world, B agrees up to a point (the $30 mark). Over time and rounds of playing the game, the A's of the world learn where the line is and optimize to stay on the correct side of it, only testing the other side 1-2% of the time to see if rules/behavior has changed.
Game theory only applies to sociopaths and economists, but I repeat myself.
This seems extremely interesting, but I don't have the time right now to read this in depth (given I would also need to teach myself a bunch of technical concepts too).
Anyone willing to weigh in with a theoretical intuition? The one in the paper is just a little inaccessible to me right now.