Comment by rhdunn
5 hours ago
It only makes sense to train a frontier model if you are trying a different architecture to one that is available from an existing frontier model. This is because the different model architecture will learn the weights differently.
It may make sense to train a frontier model on an existing architecture if the base model is not available and the instruction trained version doesn't fit with what you want. There are techniques like ablation, but those could have other effects on the model, and there can still be lingering effects of the instruction training in the model that surface less frequently (e.g. on an input not covered by the ablation training).
Otherwise, fine tuning is definitely the way to go. However, you need to be careful not to over-tune the model such that it is only tuned to the data you are training it on.
On a higher level it might make sense to build the expertise that comes with base training. I dont know enough about the process to estimate these gains, but china has been doing it in manufacturing for decades. All the money in the world is useless when no one knows how to do the thing