Comment by IncreasePosts
2 days ago
It's much easier to distill a model than create one from scratch. Part of the reason the open source model factories have been able to keep par with the frontier model factories is that they distill the frontier models, not recreate something as good from scratch.
> It's much easier to distill a model than create one from scratch. Part of the reason the open source model factories have been able to keep par with the frontier model factories is that they distill the frontier models, not recreate something as good from scratch.
I think the "why" was "why would the US companies have models that can't be distilled?", not "why does distillation work"?
I suppose it would be due to intense research and development. Obviously anyone could come upon these models if they exist, so it isn't necessarily a given that frontier labs would do it first, but probably the odds should be given to the groups with the biggest pull and the largest budgets