Comment by ben_w
13 hours ago
Good.
If you think closed source software/binaries only is bad, wait until you see how awful the state of the art is with a clear-as-mud bucket of matrix weights.
We know it's possible to train an LLM to secretly respond to certain trigger phrases, and last I checked these could only be detected with the assistance of whoever chose those phrases.
The trigger condition for such backdoors is not something anyone can do a systematic brute-force check for, for the same reason we had to invent LLMs in order to do natural language processing: combinatorial explosion.
Passing around open weight models from known sources is already asking you to trust those sources; because of how difficult this is to do correctly even without deliberately inserting such things, we still don't know if China has already put such trigger conditions into their models despite headlines such as these: https://venturebeat.com/security/deepseek-injects-50-more-se...
Regardless of if it was deliberate or not, we don't know if we caught all of these misbehaviours. We don't know how to.
And note, I'm not saying "and therefore you should trust the Big Name Models". If open weight models score 2/100 in this context, closed ones score 1/100.
You can actually discover those in open weight artifacts, reproduce them, study them and issue a security bulletin.
With proprietary hosted weights you can be specifically targeted and you would not be able to reproduce nor prove anything.
Poisoning open models would be of short-term benefit to China only if they could target US (and maybe EU + Commonwealth) specifically. Damaging anyone else would be a net loss and would erode the partnerships and alliances they are trying to build elsewhere. So it's a fire-once weapon with a huge risk of collateral damage.
Much more plausible is simply making the models ideologically biased, but as history teaches us, preferring ideology or religion over science is a well-known path to ruin. It would be weird to simultaneously warn public not to use their own open models, so.
I think the most plausible explanation for open models is simply that Huawei wants more customers and is willing to compete on the hardware front.
> You can actually discover those in open weight artifacts, reproduce them, study them and issue a security bulletin.
Finding unknown backdoors in models is NP hard.
> You can actually discover those in open weight artifacts, reproduce them, study them and issue a security bulletin.
No, you actually cannot. Not in general and without already knowing what the whole trigger pattern is. It's absolutely possible to put in a trigger that only fires while working on backend code on a specific date in a specific company by a specific github username, and no way to find this except by trying that combination, thanks to the terrible state of current mechanistic interpretability tools.
Remember: an AI model is not code. Solving this problem is as hard as the entire alignment problem.
The companies at the bleeding edge of research into this topic do not know how to reliably perform the kind of thing you suggest here.
The only reason we can point at DeepSeek-R1 and say the following, is because we can guess the magic keywords:
- https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
> Poisoning open models would be of short-term benefit to China only if they could target US (and maybe EU + Commonwealth) specifically. Damaging anyone else would be a net loss and would erode the partnerships and alliances they are trying to build elsewhere. So it's a fire-once weapon with a huge risk of collateral damage.
This "fire-once weapon" has already been fired, and appears to be a massive foot-gun for every model on a near-continuous basis.
Nobody would use LLMs if the trust deficit alone was a sufficient argument.
> Much more plausible is simply making the models ideologically biased, but as history teaches us, preferring ideology or religion over science is a well-known path to ruin. It would be weird to simultaneously warn public not to use their own open models, so.
"Ideologically biased" is the alternative explanation for the already-observed output of DeepSeek-R1. We can't tell which explanation, malicious or accidental bias, is the actual cause.