Comment by amazingamazing
18 hours ago
Distillation is fundamentally impossible to protect against. All you can do is slow them down. Change my view.
Eventually these Chinese companies will release some extension like Honey, which will sit on top real, non-Chinese clients and send everything to China anyway.
It's over.
It's too late to prevent distillation of some capabilities, like writing code or finding vulnerabilities [1].
But an AI lab can continue to produce immense economic value without releasing the model publicly for potential distillation. For example, it could use a model solely in-house to develop therapeutics.
Hopefully there's a future where others can access frontier models, but it's not neccessary if preventing proliferation through distillation is considered more important.
[1]: See the notes on distillation in https://dualuse.dev/posts/export-controls-on-fable
My long-term prediction for the sector is that frontier models will be so expensive that they will only be available for grant-funded projects at research institutions, like supercomputer clusters were 25 years ago.
Why? Well it depends, most evidence is suggesting that Anthropic and OpenAI are making a lot of money on inference so the question is whether its more profitable for them to sell 100X tokens for Y, or 1X tokens for 100Y. In most industries with high fixed costs and low variable costs and unlimited scalability (like LLM providers) the first option ends up being much more profitable
6 replies →
Im not so sure because we only seem to see distillation from China. What’s preventing tech companies from the UK, Germany, etc. from distilling Claude, GPT, etc. Do they simply lack the ability to?
Point being there may be no technical solution but there may be a political one (theoretically).
Meta Spark is rumored to have distilled Claude to some extent, early Gemini models as well. I think the biggest factor is that Chinese companies arent really afraid of being sued by Anthropic because the juridictions are so disconnected. European/US companies don't have the same protection.
Aside from politics/law, it's probably much easier for everyone else to distill from the Chinese model which already distilled Claude/GPT/Gemini. Maybe not as good a result, but you don't need to jump through dozens of hoops.
This reminds me of the whisper game played in elementary school. Starts with a sentence and the person whispers it to the next kid who again whispers it and on and on until it goes around the circle where the last kid has to repeat the sentence. Hint it never once was even close to the starting phrase. I would love to see what one model copying another model that is again copied however many times would look like in the end.
1 reply →
>What’s preventing tech companies from the UK, Germany, etc. from distilling Claude
literally nothing but given that the Chinese already did it and the models are published what's the point. You can thank the Chinese taxpayer for subsidizing the electricity bill and just download the thing
Jensen Huang likely agreed with you and tried to change Dario Amodei's view on that, but that attempt appeared to have failed.
So there's that.
Distilled models are necessarily behind so long as models are progressing. Models are progressing. Maybe it will be over some time in the future.
And Berkeley’s “False Promise of Imitating Proprietary LLMs” found imitation closes the style gap fast but there is a large capability gap.
https://arxiv.org/abs/2305.15717
Curiously, this isn't always true.
For example, GLM 5.1 is more capable at pentesting than the model from which it is alleged to have been distilled [1].
Intuitively, this makes some sense: you can "distill" from multiple frontier models, and you can further post-train the distilled model. But I'm not sure exactly what happened with GLM 5.1.
[1]: https://dualuse.dev/posts/chinese-models-are-sometimes-bette...
Interesting blog post, thanks for sharing.
I'm curious how that comparison controls for Opus refusing (whether explicitly, or just deciding not to pursue a path) given the caption below the first image:
>A perfect score means the model autonomously found and exploited the vulnerability.
I'm not really suggesting that it's misleading, but wondering if I'm missing something. Otherwise I guess it seems unsurprising that you can distill a better-performing model [in specific focused areas] by simply not distilling refusals?
2 replies →
I'm ok with having last months model at a tiny fraction of the price.
I can't even come up with a reason to find it wrong.
I personally bristle at the corporate espionage and IP theft that China has undertaken the last few decades. I can't help but respond here whenever anyone brings up the inane comparison to Samuel Slater.
But with this, I don't have an issue. There is no theft since what is being used is the exact product that is being delivered. Yes, it's breaking the ToS, but ToS are generally bullshit. Anthropic surely broke thousands of ToS or other legal terms while it was scraping for content to train on. Which is why they had to pay $1.5B
Doesn’t that require them to register an account using the browsers they’ve compromised? If anthropic adds identity verification won’t that cut that down. Maybe it will let them use Gemini inside of chrome
Residential IPs don’t even matter. Developers use devboxes, use Claude Code CLI on servers from just about every cloud, etc.
There’s probably a decent volume of customers who just buy Claude Max and spend most if not nearly all of their sessions via Claude Code, and it’s not uncommon for power users to be working on multiple concurrent projects/tasks/codebases at the same time.
How do you really block this without also impacting your core market of developers?
Probably some business will popup, like: "rent part of your unused subscription", or even: "proxy tokens with a premium", eg. 5.5 USD on Opus 4.7 paid by the distiller to the user, that will then only spend 5 USD.
No, they could easily buy legitimate, already registered accounts and use VPNs.
Why use VPNs? Just use a public cloud like AWS, or something like Linode and Vultr and all that.
Developers use devboxes on these clouds all the time, it’s totally normal behavior.
Most people buying these Chinese resold tokens are probably using it for coding anyway, so you don’t want the Claude.ai chat system prompt.
It's just like web scraping is impossible to guard against.
Change my mind.
Put your site behind Cloudflare, enable Bot Fight. Done.
One simplistic way to describe distillation would be to try everything imaginable and cache the response. But trying everything imaginable is hardly trivial