I feel like there's an implication here that distillation is a problem but I don't understand what you mean. I thought distillation was generating text from a model and then training another model on it. Is the something unethical in that? You're paying the API costs to generate the tokens, right?
Or I guess more to the point: is this something frontier labs have said is (or tried to paint at any rate) problematic? This feels like an "out of the loop" situation because I've only ever heard "distillation" with a positive connotation before.
it doesn't matter the reason. This is a race and nobody will care or remember how the winners got there.
Mistral looks like it's fading away to irrelevance unless they can play alongside the similar sized models, or have some unique advantage other than being in Europe, for Europe. I was really excited for them back when they were startup that had the biggest European venture round ever. This space will have a few winners, and many losers. Google, plus either Anthropic or OpenAI most likely. Big models will see breakthroughs in inference performance/cost fall precipitously and small models will only exist on devices (Pixels and iPhones, cars, watches, bluetooth speakers, etc)
It’s not that I don’t agree with you, I am just pointing out why it’s hard to catch up to scaling laws given the European economic (capital) and political (US would be upset if they found out Europeans distill) constraints. China is only bound by economic constraints.
> This is a race and nobody will care or remember how the winners got there.
For consumer AI, yes. For coding assistants, probably.
For specific application "business" AI like the things Airbus announced the other day? Not at all. What matters for an Airbus using Mistral to build compliance documentation based on AI generated physics simulations is the enterprise relationship, reliability, compliance, forward deployed engineers helping with the fine tuning, quality, predictability, support. A Chinese lab having a better at benchmarks model that is cheaper is just irrelevant for that.
And IMO, the real money in AI is this type of "business AI" deployment. Developer tooling tends to converge on becoming commoditised. Once you're a core supplier for a big bank and embedded in their processes, you're there untill you screw up with the pricing (like Broadcom), and even then.
Because they distill
I feel like there's an implication here that distillation is a problem but I don't understand what you mean. I thought distillation was generating text from a model and then training another model on it. Is the something unethical in that? You're paying the API costs to generate the tokens, right?
Or I guess more to the point: is this something frontier labs have said is (or tried to paint at any rate) problematic? This feels like an "out of the loop" situation because I've only ever heard "distillation" with a positive connotation before.
Whether it's a 'problem' or not is viewpoint-dependent but it's against the OpenAI ToU:
> You may not use our Services for any illegal, harmful, or abusive activity. For example, you may not:
> [...]
> * Use Output to develop models that compete with OpenAI.
Source: https://openai.com/policies/row-terms-of-use/
(I'm also curious whether they consider developing a competing model to be illegal, or harmful, or abusive...?)
4 replies →
it doesn't matter the reason. This is a race and nobody will care or remember how the winners got there.
Mistral looks like it's fading away to irrelevance unless they can play alongside the similar sized models, or have some unique advantage other than being in Europe, for Europe. I was really excited for them back when they were startup that had the biggest European venture round ever. This space will have a few winners, and many losers. Google, plus either Anthropic or OpenAI most likely. Big models will see breakthroughs in inference performance/cost fall precipitously and small models will only exist on devices (Pixels and iPhones, cars, watches, bluetooth speakers, etc)
It’s not that I don’t agree with you, I am just pointing out why it’s hard to catch up to scaling laws given the European economic (capital) and political (US would be upset if they found out Europeans distill) constraints. China is only bound by economic constraints.
1 reply →
> This is a race and nobody will care or remember how the winners got there.
For consumer AI, yes. For coding assistants, probably.
For specific application "business" AI like the things Airbus announced the other day? Not at all. What matters for an Airbus using Mistral to build compliance documentation based on AI generated physics simulations is the enterprise relationship, reliability, compliance, forward deployed engineers helping with the fine tuning, quality, predictability, support. A Chinese lab having a better at benchmarks model that is cheaper is just irrelevant for that.
And IMO, the real money in AI is this type of "business AI" deployment. Developer tooling tends to converge on becoming commoditised. Once you're a core supplier for a big bank and embedded in their processes, you're there untill you screw up with the pricing (like Broadcom), and even then.
Why doesn't Mistral distill?
Good question, given that American companies basically threw copyright law into the trash, I think they should.
4 replies →
I suppose losing with dignity is a consolation.