Comment by dlandis
6 hours ago
Hmm, don't people think that if the frontier labs really put enough engineering effort into preventing distillation that they would be able to do that, or at least diminish it significantly? I'm sure there are variety of additional techniques they could use on top of what they already do, but I suspect it just hasn't been at the top of their priorities yet. Maybe that will change soon. Worst case they could add additional hurdles to account creation ("know your customer" type of thing).
> Hmm, don't people think that if the frontier labs really put enough engineering effort into preventing distillation that they would be able to do that, or at least diminish it significantly?
Distillation was big news a year or even 6 months ago, but as far as we can tell it's not really a moat anymore. Now that multiple players have trillion+ parameter models and the capacity to post-train them, there's no putting the genie back in the lamp.
The frontier labs have competing goals in mind. They want high growth (which means little friction for account creation), API access (because enterprise money is the best money to have), and distillation protection.
Besides, identity verification that actually works at scale is a much harder problem than identity verification which is good enough to satisfy your compliance people and regulators. Especially if the fraudsters have a major world government standing behind them, and if their aim is to be identified as a real customer, not one customer in particular.
In particular, harvesting identities for online fraud is an industrial market for various criminal organizations.
The nature of LLMs is that you give them input, they give you output. That allows for distillation. They do try to make it harder by hiding the chain of thought stuff, but fundamentally, if your model is publicly available, its distillable.
At the end of the day, while you can do your best to obfuscate your reasoning tokens, it's a losing battle to hide actual user-visible output tokens. The very nature of API offerings is that you can't do KYC on where that API's output is going - there's a rich secondary market that's not going away.
And with the sheer volume of data created from that, coupled with benign-seeming prompts like "plan out your reasoning in a document before implementing" that could never be patched without breaking existing customer workflows... there's more than enough for someone to distill on. Even if that only gets them to not-quite-frontier, if you're pushing the frontier every few months, they're only ever a few months behind you.
Even if it were possible it wouldn't change the outcome. China is capable of training frontier models even without distillation. Distillation is only an accelerant.
The primary resource you need to train LLMs is money and China has plenty of that.
Only the Chinese authorities can stop Chinese labs from distilling from western labs. And they won’t do that, for obvious reasons.
i trained another AI on all my codex logs... it's pretty good actually