Comment by HarHarVeryFunny
1 month ago
No - you can't distill if what you are given doesn't have the thing in it that you want to distill out of it.
I don't know why it's so important to you to use the word "distillation", but it's the wrong word to use.
BTW OpenAI on twitter also said that Kimi 3 "cannot be explained away by distillation or anything like that". The timeline of how long it takes to train a model and when Fable was released don't even line up. This is just Anthropic as usual trying to manipulate the US government into helping them shut down competition.
Distillation is absolutely - and uncontroversially - a valid term for what is happening here.
This isn't really a debate, I'm not making a fine point - just check with all of the various defintions of the term.
Moreover - the 'reasoning traces' are not required for distillation at all.
Finally - it's entirely possible for them to have used Fable for later stage fine tuning.
It's fair to be skeptical of Anthropic (and everyone else) - but this is 'distilling'.
Words have meaning - you cant just redefine them because you want to.
Are reasoning traces required for distillation? Well they are if what you are trying to distill is reasoning, such as coding expertise.
Do you need reasoning traces for "LLM as judge"? No, but it would be highly perverse to call that distillation when there is a more accurate name for it - LLM as judge.
If you want to call use of Anthropic's redacted model outputs in any fashion that violates their terms of service (using them them to help develop anything that competes with Anthropic) as "distillation" then I can't stop you, but it reduces their claims to a joke.
Finally, as noted, OpenAI (who are just as anti-Chinese as Anthropic) said that Kimi 3 can't be explained via distillation (even true distillation!!), or even "anything like it". But random internet guy, you, disagrees. OK.
> Words have meaning - you cant just redefine them because you want to.
You are redefining words. The consensus among people who work in this space is that "distilling" is what's actually going on here.
> who are just as anti-Chinese as Anthropic
Conflating criticism of IP theft with being "anti-Chinese" is a standard PRC influence playbook technique.
And furthermore, OpenAI's market strategy is to win through regulatory capture. They are financially incentivized for Anthropic to be distilled by PRC labs and to be undercut by open models. Their claim about Kimi not being explainable due to distillation is not a factual claim - it's marketing from a company owned by Sam Altman.
Although, it does conclusively disprove your claim about the meaning of distillation, because you cannot say that "Kimi can't be explained by distilling" unless the consensus definition of "distillation" is such that it could be done on the summarized reasoning traces that Anthropic models expose.
4 replies →
The GP (HarHarVeryFunny) is not operating in good faith. They've repeatedly lied about my own words to me, and are making up definitions that people who actually work at a frontier lab would disagree with.