Comment by WarmWash

Pre-trains take a huge chunk of your compute offline, incurring both an raw expense (24/7 max power for all training clusters) and an opportunity cost (could have sold excess compute during that time). They also don't come with any great guarantees, as lots of techniques look good on small scale and crumble or plateau once scaled.