Comment by zmmmmm
1 day ago
> Train your own base model - but tune it off Claude output to make it perform more in line with Claude
Is that actually genuine distillation though? Distillation suggests the core model is being pre-trained using output from another model. For the above to work, you have to already have all the core intelligence trained into your base model.
If distillation just comes down to post-training then it's tantamount to admitting that the Chinese base models are just as good as frontier US lab models. Because you can't post-train frontier intelligence into a model. It has to be there in the base. Then you can change how that intelligence is expressed through post-training.
What's in the base model is "bits and pieces of intelligence".
You have to bring those bits and pieces together, put them into the right shapes and fill in the gaps to get a model that actually performs. This is what post-training is all about. It's not at all a trivial thing.
Reasoning, tool use, agentic behavior - all of those are post-training performance gains. Getting a good well trained base model is putting your foot in the door of frontier performance - post-training is how you actually get inside.
See: GPT-4.5 vs o1. One went for "build a bigger better more capable base model", the other went for "take the old base and post-train it for advanced capabilities". The results: a wider base with basic post-training loses to a narrower base with advanced post-training. Or, hell: GPT-3 vs GPT-3.5. One was largely a research lab curio, and the other kicked off the AI revolution as we know it.
The gains compound. Getting a better base model with the same type of post-training helps, see: the jump from Opus to Mythos/Fable. But post-training techniques account for a lot of the performance juice.
And yes, reasoning trace post-training distillation is "genuine distillation". As is logit distillation in pre-training. "Distillation" isn't a single training recipe that you have to follow to a tee - it's a large group of training methods. I've seen plenty of wacky things like inverse distillation bootstrap and post-training self-distillation that use distillation in strange ways at different stages of the training run to get results.