Comment by colingauvin
6 hours ago
Where is the actual evidence of distillation? I keep seeing this repeated ad nauseam but I must have somehow missed the evidence.
6 hours ago
Where is the actual evidence of distillation? I keep seeing this repeated ad nauseam but I must have somehow missed the evidence.
Been using a lot of Kimi K3 lately and the answers have been… „load-bearing“ to the point of hilariousness. It‘s obvious from where they distilled, even if sceptics rightly point out it can‘t have been the only source of their secret sauce, as it‘s been better than the current Opus 4.x at the time of release.
Qwen 3.8 27b also enjoys their genuinely load-bearing seams.
Distillation a pretty well documented technique that actually pre-dates LLMs https://arxiv.org/pdf/1503.02531
Here is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit
That distillation exists isn't the question.
It's about evidence this is an active force in competition in LLMs.
I think the biggest actual piece of evidence is how hard the major players are trying to stop it
I took GP as asking for evidence that the reduced-cost Sol is actually a distillation of the previous-cost Sol. AFAIK, providers distilling or quantising models and offering them as the same model have not been proven.
I doubt he was claiming that. He's probably saying that the ability of Chinese companies to be able to distill frontier US models has put downwards pressure on the price of all models.
2 replies →
To be precise, the distillation mentioned in this paper is not the distillation used by other model companies. In the one mentioned in the paper, your teacher and student model typically have similar architectures - and you typically need access to the full logits. What happens here instead is motivated by the fact that these companies don't have access to the training data and compute that anthropic/openai have. The distillation they do basically amounts to using traces from ant/oai models trained on much more data with a lot more compute (in many cases including the hidden intermediate tokens! turns out there were many ways to coax it out) and then either directly training on it or using it in many ways in post training pipelines. It falls under imitation learning, IMO.
It turns out you can train a 1b model at almost 1000 tokens/s on a m5 max laptop. As a personal experiment, I've been asking Sol for synthetic training data and synthetic agentic training data (model distillation in it's purest form), plus modified opencode, codex transcripts etc for training data, and nobody's even paying me to do it. If I'm doing it has a hobby, you can bet industrial users are doing it.
Why does Kimi insist its name is Claude?
Please elaborate the mechanisms by which a LLM would know what model it is.
By being trained on text containing "I am X" in the model response section.
ask it what type of model it is or what it's name is... it's weird that Kimi will say it's claude...
3 replies →
The evidence is Anthropic's own reporting [1]. You may doubt that they're telling the truth, but that's what they're reporting.
[1] https://www.anthropic.com/news/detecting-and-preventing-dist...
Musk confirmed in federal court that xAI does it: https://techcrunch.com/2026/04/30/elon-musk-testifies-that-x...
It's also how providers build their smaller models out of their larger ones; they publicly talk about the process.
Here's an example: https://github.com/microsoft/Build25-LAB329
[flagged]
there is no evidence. it shortcuts post training by a huge margin this is true. but that is all.
https://techcrunch.com/2026/04/30/elon-musk-testifies-that-x...
Make sure to stay updated!
Nothing against what I said in there. Distillation is far from enough to get to the frontier. Its at best a ramp (the most efficient one used by everyone ) that shortcut and saves millions of rl runs before a model moves.
1 reply →