← Back to context

Comment by kamranjon

19 hours ago

The fact that API based distillation is even a conversation right now makes me feel like the U.S. has their heads so far in the sand that it’s not really excusable.

These Chinese labs are producing novel models, publishing their techniques and sharing their open weights and the first topic of conversation is how they stole from U.S. AI labs.

Setting aside the fact that it doesn’t make any feasible sense to do API distillation, these models are outperforming frontier models on a number of benchmarks, and often times run more efficiently by several orders of magnitude.

We have to stop crying distillation, it’s getting embarrassing and at this point feels even a bit delusional.

23 comments

kamranjon

Reply
overfeed  17 hours ago

> We have to stop crying distillation, it’s getting embarrassing and at this point feels even a bit delusional.

It's a PR campaign - when they say its an "attack" they don't mean on Anthropic - but on America itself. What kind of American can let such a brazen attack go unanswered? At the very least, they ought to demand the dangerous, pinko, stolen models be banned in all 50 states, and pay whatever price demanded by the patriotic, freedom-loving, all-American AI labs that can never be accused of stealing.

tristanj  18 hours ago

There's little doubt that Kimi K3 was distilled off Claude.

Anthropic stated in February that Moonshot AI (the creator of Kimi) distilled ~3.4 million exchanges from Claude models, as explained in their press release https://www.anthropic.com/news/detecting-and-preventing-dist...

  • kamranjon  14 hours ago

    It’s so funny to me that Anthropic can make claims like this one with zero evidence provided.

    DeepSeek and others like Minimax are publishing deep research on Multi-Head Latent Attention and Mixture of Experts, Multi-Token Prediction, novel Sparse Attention approaches, I mean they trained long context models on a fraction of the resources and gave everyone the recipe.

    Chinese labs might not have the funding of labs like Anthropic, but at least they provide the receipts.

  • overfeed  17 hours ago

    While it sounds like a lot, do you suppose 3.4 million sessions come even close to being sufficient to train a frontier model?

    Assuming each session was 10,000 words each, that's 34 billion words; lets call it 50 billion tokens (0.05 trillion) unfairly pilfered from Claude. That left Moonshot needing to scrounge for the other 14.950 trillion training tokens required for a baseline frontier model.

    • ACCount37  9 hours ago

      What do you think those tokens are used for?

      Distillation attacks aren't about replacing the entire pretraining dataset with questionably sourced synthetics. It's all about post-training.

      Train your own base model - but tune it off Claude output to make it perform more in line with Claude. Yoink the products of Anthropic's expensive SFT, RLHF and RLVR work for yourself by training on the outcomes.

      The post-training datasets are small, but they are what controls the final model behavior.

      4 replies →