Comment by sanxiyn
1 month ago
If a model fails the test, it should be banned. He is not advocating a ban of open-weight models. He is advocating a ban of models that fail mandatory safety testing. Seems reasonable and straightforward.
1 month ago
If a model fails the test, it should be banned. He is not advocating a ban of open-weight models. He is advocating a ban of models that fail mandatory safety testing. Seems reasonable and straightforward.
Can you think of anything open-source that has to go through mandatory tests to be distributed and still survived ?
There is a reason to it, that's as good as any angle to find why IMHO.
I think Gemma will be fine. Most open-weight models are not capable enough to be dangerous. Yes, I can't think of any capable open-weight model that would survive reasonable safety testing.
Any sufficiently capable open weights model would fail "safety" testing though, as any "safeguards" of the sort Anthropic likes could be removed. That's just another way of saying they want a ban on capable open source models which would contradict their earlier statement, or at least make it very misleading. It's hard to see how this post can be internally consistent without some hint from Dario about what he believes should happen to models that fail safety testing and/or how capable open weights models could possibly pass a safety test of the kind he proposes.
I agree we don't know how capable open-weight models could possibly pass any reasonable safety testing NOW, but that's about currently abysmal state of AI alignment research, not about what is possible in principle. I don't see any internal inconsistency, to be honest. Since Anthropic does not release any capable open-weight models, it's not their problem. If mandatory safety testing is established, companies who want to release capable open-weight models will work on AI alignment research so that they can pass. This seems to be a good outcome to me.
OK that is a position they could take but my point is that's inconsistent with "Anthropic has never advocated for a ban on open-weights models". What you're describing is a ban on capable open-weights models until some future time.
3 replies →
> He is not advocating a ban of open-weight models
He is though. He wants open weight models banned that do not pass some set of tests.
And what does "safety" mean here? We constantly see these companies treating NSFW content as "unsafe", despite the fact that its not. Is being able to produce adult content going to result in a model being declared "unsafe"?
abliteration and fine tuning makes it not so straightforward
https://huggingface.co/blog/mlabonne/abliteration
I know, but "we don't know how to make it not dangerous, so it should be allowed to release dangerous things" is... not convincing?
My point is that advocating a de facto ban on capable open source models is inconsistent with Dario's statement here that "Anthropic has never advocated for a ban on open-weights models." Call a spade a spade.
4 replies →
There is no movement on global policy or enforcement. One country banning their people access to the best models hinders their people.
I am unconvinced that "this can be used dangerously, therefore we must ban it" argument. The OpenAI/Huggingface, needing to turn to Chinese open weight to defend themselves seems to support the case that we need open access and freedom to compute as we see fit.
He is not advocating for banning models per se but the proposal makes a business model (i.e. serving open weight models) that is starting to work more expensive.
Agreed, and that serves Anthropic. It seems unproblematic to me. Dario probably sincerely believes in mandatory safety testing for capable models (open and closed), and likes the fact that it aligns with Anthropic's interest.