Comment by tristanj
9 hours ago
There's little doubt that Kimi K3 was distilled off Claude.
Anthropic stated in February that Moonshot AI (the creator of Kimi) distilled ~3.4 million exchanges from Claude models, as explained in their press release https://www.anthropic.com/news/detecting-and-preventing-dist...
While it sounds like a lot, do you suppose 3.4 million sessions come even close to being sufficient to train a frontier model?
Assuming each session was 10,000 words each, that's 34 billion words; lets call it 50 billion tokens (0.05 trillion) unfairly pilfered from Claude. That left Moonshot needing to scrounge for the other 14.950 trillion training tokens required for a baseline frontier model.
3.4 million is the number of sessions Anthropic detected. The actual number of Claude sessions trained on is likely >100 million. There are tens of thousands of accounts funneling Claude sessions into Chinese labs https://www.chinatalk.media/p/how-to-buy-cheap-claude-tokens...
They are used for post-training, i.e. calibrating the model to understand and use tools/command line more effectively.
> 3.4 million is the number of sessions Anthropic detected. The actual number of Claude sessions trained on is likely >100 million.
That's an increase of only a single order of magnitude, increasing my estimate of exfiltrated tokens from 0.05 to 0.15 trillion - a far cry from the 15 trillion required.
> They are used for post-training
Possibly - it may be too much data for post-training, unless further curation was done. However, this is not distillation; you know it, I know it, Dario knows it, but "Distillation Attack" is a short, memorable, sciencey-sounding, political sound-bite with enough malevolence to be deployed on the floors of congress, or by the usual fear-mongering newstainment talking heads.
2 replies →
It’s so funny to me that Anthropic can make claims like this one with zero evidence provided.
DeepSeek and others like Minimax are publishing deep research on Multi-Head Latent Attention and Mixture of Experts, Multi-Token Prediction, novel Sparse Attention approaches, I mean they trained long context models on a fraction of the resources and gave everyone the recipe.
Chinese labs might not have the funding of labs like Anthropic, but at least they provide the receipts.
There's reproducible evidence of Kimi K3 spontaneously identifying itself as Claude https://x.com/denisewu/status/2077984660211269870
This behavior is exactly what you'd expect from a model distilled from Claude.
Someone even took the time to analyze Kimi's ambiguous identity, in great detail: https://github.com/rgreenblatt/which_claude_is_k3/blob/main/...
And there's an entire Reddit thread discussing this https://www.reddit.com/r/LocalLLaMA/comments/1m2w5ge/did_kim...
That doesn’t prove Anthropic’s specific 3.4m-session allegation, but calling it “zero evidence” is no longer credible.
Kimi K2.5 was worse in a hilarious way, it identified itself as Claude and referenced Anthropic's Constitutional AI as some of its guiding principles https://huggingface.co/moonshotai/Kimi-K2.5/discussions/38
> This behavior is exactly what you'd expect from a model distilled from Claude.
This is not at all what I would expect because it's trivial to change the training data to replace Claude with Kimi. In fact I'd argue it's almost certainly not saying that due to distillation.
2 replies →
By evidence I mean logs, I mean IP addresses, I mean timestamps. They claim millions of requests, let’s see literally any of them?
I don’t consider a tweet by Denise Wu, who works at Anthropic, to be reproducible evidence.
I don’t consider “Caveat: fully AI-generated research.” To be someone taking time to analyze anything in great detail.
Because two AI models produce vaguely similar front-end styles when generating similar prompts I also do not consider to be of much value?
I think this is what I mean when I say the U.S. has its head in the sand. The Chinese labs are releasing ~60 page research reports with citations and analyses and evidence and Anthropic is throwing up defensive blog posts with zilch. I’ve seen more detail in a tech blog from Uber than anything I’ve seen from Anthropic.
2 replies →