Comment by CuriouslyC
2 months ago
#1 isn't going to happen because we're actually data limited, not compute limited. You can throw all the compute in the world at bad data and it won't make a difference, but an undertrained model with perfect training data will absolutely slay.
#2 isn't going to happen, because these labs have shown they have limited app/design sense, and they also lack the industry connections and domain wisdom to execute.
The way things are actually going to go is that these labs will set up partnerships with huge biotech/engineering/etc firms, and do custom training/inference on specific tasks that promise to be wildly profitable with them, then take royalties on the creation in perpetuity. Why sell inference when you can partner with Pfizer to make a version of Ozempic that also makes people freaky jacked, or partner with Bectel to make a radically safer, more efficient Nuclear power plant?
I don't think "data limited" is true anymore outside of very specialized cases (for instance: https://arxiv.org/abs/2510.01631). As weird as it sounds, training improves a lot with synthetic data.
You do need business development to create those relationships. Saying they "have limited ___" mostly means they "haven't yet hired people who are good at ___". That's been changing already; the Claude app is steadily improving and handling more use cases simply through understanding which tools to use, Anthropic is building more relationships to create more tools, and all the frontier model companies are building relationships with companies that have specialized data and want specialized solutions.
I think we're also seeing the frontier model companies offer partners their own ability to run RL on their own data, and then retrain new models on the same data. That's going to make those relationships VERY sticky in ways that won't be obvious from the outside.
Can you point me to the parts in that paper that meet those claims, I am reading something different and want to know what I am missing.
This study seems to show that there are places where synthetic data, especially related to common crawl.
> Pure synthetic data remains non-advantageous over CC; notably, models trained on pure rephrased synthetic data will underperform those trained on CC at larger models.
But the tradeoffs seem to be different at large scale.
> Overall, these model scaling results suggest synthetic data appears comparably less favorable for pre-training larger LMs relative to its utility in data scaling scenarios. Despite outperforming training on CC, larger models are not as tolerant to a higher ratio synthetic data as larger data budgets. This observation aligns with practices where synthetic data is effective for smaller LMs or specific pre-training phases, but less predominantly used for the largest models.
How I am reading it is there are places where it is useful:
> Notably, any mixture involving synthetic data, or pure synthetic data (except pure QA), is projected to achieve a lower irreducible loss than training only on CommonCrawl.
But it also seems that on textbook scale synthetic data, they did show model collapse vs rephrased data.
> These results contribute mixed evidence on “model collapse" during large-scale single-round (n=1) model training on synthetic data–training on rephrased synthetic data shows no degradation in performance in foreseeable scales whereas training on mixtures of textbook-style pure-generated synthetic data shows patterns predicted by “model collapse".
IMHO there are some very specific areas where we aren't "data limited", like math, but as your reference states "Our work demystifies synthetic data in pre-training, validates its conditional benefits, and offers practical guidance."
Note the cost of 30% of the total dataset being synthetic, where the model starts amplifying the generator's biases, leading to a permanent degradation in downstream zero-shot capability on unseen out-of-domain natural tasks.
My takeaway is there is nuance where synthetic data is an amplifier and where it is a problem, and in my mind that paper demonstrates it will not solve the data problem in general.
> we're actually data limited
Correction: public text data limited.
There's a ridiculous amount of proprietary text and non-text data out there that much of society is run on.
what is 'bad data' and 'perfect data' according to you?
Worst possible bad data is where the data is orthogonal to the task, so increasing the data never provides information on the task. Perfect data is where the data exactly encapsulates the task being trained.