Comment by jke_kang
7 days ago
People seem to conflate "made in China" with "can't be trusted." id argue the bigger distinction is open vs. closed. An open model can be audited, fine-tuned, and technically run entirely on your own hardware. A closed model is basically "trust us."
Open weight models are much more auditable than closed models, but could still hide backdoors that could be near impossible to detect.
Correct. We need open weights, open code and open data. If nobody else can reproduce what someone did there will always be security questions. Even if we can reproduce it there could still be security concerns but it's more realistic to investigate yourself.
I'm all for open models, but people seem to misunderstand what they are. They aren't the same thing as open source code!
> open weights, open code and open data
Even if you have all these things you still can't replicate a model because of randomness.
You can backdoor a model with less than 1000 examples and it is impossible to detect.
7 replies →
Exactly what are the possible 'security issues' of self hosting an open weights model?
6 replies →
> We need open weights, open code and open data.
Even with this, the cost of verification would be enormous. You would need a massive cluster to repeat the training E2E.
> could still hide backdoors that could be near impossible to detect.
But it won't change after you download it, so you can isolate those problematic cases and use another model for different use cases
In my opinion, the big issue with that argument is that advances in interpretability research and steering conceivably could, and probably will, render moot that (as of now, purely hypothetical) risk of subtle sabotage for open-weight models... but not for closed models.
It’s not hypothetical. Magic strings are a known and implemented feature for standard model interaction. Nearly impossible to detect unless you know where to look with current technology.
13 replies →
which is moot point, if open model is hard to fully audit, then closed model is complete enigma and you should be more scared about closed models
Oh really. How'd that work out for security in open source.
I think that perception of China has been shifting and will look quite different over the next few years