Comment by overfeed

> I believe that’s post-training, not pre-training

Aren't AI companies already butting against the limits of usable training data?

Outside of undiscovered techniques, additional costly pre-training will likely yield diminishing returns if you've already scraped the whole internet and destructively scanned as many 20th century books as you could lay your hands on.