Comment by tristanj
7 days ago
1) Model distillation is the process of transferring knowledge from a large model to a smaller one. It doesn't require logits. https://en.wikipedia.org/wiki/Knowledge_distillation
2) The word "attack" is standard security vocabulary. Per RFC 4949:
attack
1. (I) An intentional act by which an entity attempts to evade
security services and violate the security policy of a system.
That is, an actual assault on system security that derives from an
intelligent threat. (See: penetration, violation, vulnerability.)
2. (I) A method or technique used in an assault (e.g.,
masquerade). (See: blind attack, distributed attack.)
There are hundreds of named "attacks".
3) The "attack" part of "distillation attack" refers to distillers creating tens of thousands of fraudulent accounts, using proxies to bypass georestrictions, deepfaked IDs, and paying real people to pass biometric KYC checks. Who then blended this in with real user traffic to conceal their behavior.
It doesn't refer to the AI training technique in any way.
If they acquired this data without the fraud, you'd have a point.
Sure that can be called an attack, but then we must also concede these labs essentially massively attacked everyone else in existence to get the data, and continue attacking as we speak.
In a way you could see this as a case of Robin Hood. The US companies exfiltrated all the data on the planet just to hoard it for themselves now and accuse anyone who tries to get a piece of that back from them, and the Chinese labs are distilling it to offer it for cheap.
Obviously a bit more complicated than that but it still holds pretty well.
> Model distillation is the process of transferring knowledge from a large model to a smaller one
Sure, and large-to-small is a key part of the definition, and why it's called distillation (cf concentrating something). When Anthopic use synthetic data generated by Opus to train Sonnet or Haiku, then this can correctly be considered a type of distillation.
When Anthropic accuse Chinese companies of "distillation", it seems they are using this word to refer to two potential uses of their model outputs:
1) Using Anthropic model outputs (aka synthetic data) as training data, especially for reasoning, for Chinese models. This really isn't distillation though, since (unlike when they distill their own models) Anthropic don't actually provide the reasoning in their model output, only a "summary" designed to hide the actual reasoning. You can't distill what you are not given!
2) Another way Chinese companies may be using US LLMs is for "LLM as judge" where you are just asking the model to use it's expertise to judge/rate something that you provided yourself (to provide RL training rewards), although for coding you really want hard rewards which are easy to obtain, not fuzzy "looks good to me" ones.
Of course Anthropic are trying to pull the drawbridge up after themselves and their TOS says you can't use their models to develop anything that competes with them, and this seems to be what they are generically referring to as "distillation" - any use of their models that they suspect is being used by the Chinese to improve their own models, not just what what might more technically be called distillation, unless you want to define that word so broadly that it does mean this!
Not to mention the "distilled" models almost certainly have other training inputs as well, again watering down the meaning of the word.
And if just partial output is all that it takes to declare a model distilled, then every model trained on Internet content since 2023 is now technically a "distilled" ChatGPT and Claude model.
I'm not even convinced that this fits your definition. A distillation "attack" doesn't evade the security system in the sense of hacking past a login. The only part of the "security system" that it bypasses is the Terms of Service. And that's only after said data was acquired legally and correctly and normally.
It's a post-facto attack, which doesn't sit right linguistically to me.
That's understating things. The efforts to violate the ToS involve what amounts to large scale organized fraud. But I'm not convinced that should be considered an "attack" rather than merely "piracy" and I certainly don't feel like there's any ethical issue. It's nothing more than an attempt to politicize competition.
1) Your own Wikipedia link goes on to describe using logits. Yes, language evolves to mean multiple things, and that is my point: Anthropic is pushing for a watered down definition. Furthermore, Anthropic hides thinking, so you do not even really get model outputs, you get some downstream partials. Further-furthermore, Anthropic does not describe their own models as distilled when they produce training data. Why? Because generating training data != distillation.
2) This is a stretch: it allows Anthropic to arbitrarily define "attack" via TOS, and ignores the fact that the generated training data is literally paid for by the "attackers".
So what kind of attack are AI companies doing by scraping up copyrighted info to build these LLMs?
"You're trying to kidnap what I've rightfully stolen."
> 3) The "attack" part of "distillation attack" refers to distillers creating tens of thousands of fraudulent accounts, using proxies to bypass georestrictions, deepfaked IDs, and paying real people to pass biometric KYC checks. Who then blended this in with real user traffic to conceal their behavior.
Lol. Isn't this literally many of the same tactics OpenAI and Anthropic used to scrape the internet? So now it's an "attack", but previously it was just "training".
Sun Tzu said: if you scrape your enemy, call it training; if your enemy scrapes you, call it an attack.