Matmuls need access to decrypted weights to do their work.
Which means that getting the full weights out isn't even an "if" - it's "how much effort". The encryption wouldn't do much more than a gentleman's agreement would.
The only real move for Anthropic there is to outline contract penalties for letting weights get leaked, and never give less trusted external inference providers access to cutting edge system weights.
Exposure is limited either way. Opus 4.7 weights are a deprecating asset - it's bleeding edge today, very valuable now, but it'll lose a lot of its value the moment Opus 5.0 drops.
Dude, Chinese labs distil attack via the APIs, if Musk wanted to do something like that, technically he could. Legally it would be a giant slam dunk liability though
Well, knowledge distillation requires a teacher model and a student model and the student model attempts to learn and extract and (preferrably) compress the information of the teacher model, so it is possible for model collapse due to high SNR in between [1].
What I suggested is to steal the (possibly intermediate) weight in between by sniffing the network communication bus, which means MITM for getting the exact values. Or unless it turns out OpenAI or Anthropic leveraged homomorphic encryption, or I'm not certain how is Anthropic would safely allow Mythos to run on AWS without their control.
Distilling is different from "siphoning the model weights". I would think that Anthropic has a system for this. After all, they deploy to different clouds already. Their weights are worth billions, I presume that they take security very seriously and have done a lot of homework to trust no one.
Can't run inference on encrypted weights and get any kind of performance out of it.
The overhead shrinks with larger models. It doesn't seem that bad.
https://arxiv.org/pdf/2409.03992v2
The whole system has encryption all the way through.
Otherwise, OpenAI/Anthropic would never use external clouds since the weights are some of the most valuable assets in the world.
Matmuls need access to decrypted weights to do their work.
Which means that getting the full weights out isn't even an "if" - it's "how much effort". The encryption wouldn't do much more than a gentleman's agreement would.
The only real move for Anthropic there is to outline contract penalties for letting weights get leaked, and never give less trusted external inference providers access to cutting edge system weights.
Exposure is limited either way. Opus 4.7 weights are a deprecating asset - it's bleeding edge today, very valuable now, but it'll lose a lot of its value the moment Opus 5.0 drops.
2 replies →
Dude, Chinese labs distil attack via the APIs, if Musk wanted to do something like that, technically he could. Legally it would be a giant slam dunk liability though
Well, knowledge distillation requires a teacher model and a student model and the student model attempts to learn and extract and (preferrably) compress the information of the teacher model, so it is possible for model collapse due to high SNR in between [1].
What I suggested is to steal the (possibly intermediate) weight in between by sniffing the network communication bus, which means MITM for getting the exact values. Or unless it turns out OpenAI or Anthropic leveraged homomorphic encryption, or I'm not certain how is Anthropic would safely allow Mythos to run on AWS without their control.
[1]: https://en.wikipedia.org/wiki/Knowledge_distillation
Distilling is different from "siphoning the model weights". I would think that Anthropic has a system for this. After all, they deploy to different clouds already. Their weights are worth billions, I presume that they take security very seriously and have done a lot of homework to trust no one.