Comment by cubefox
3 days ago
> Open Source: We will release the weights in two weeks after launch, once safety evaluation and hardening are complete.
What safety evaluation? What safety hardening? They already evaluated it and found it to be highly capable at exploiting security vulnerabilities. So we know it is not "safe", and they don't seem to plan to do anything against it. What could be more dangerous than hacking? Biological weapons research? I don't think Chinese labs are doing anything against this either.
Are you against open-source models?
Data and content related to "biological weapons" already exist on the internet, in books, etc. The real issue is access to facilities and tools. There are models that help researchers, but they are not LLMs, rather they are models trained specifically on biological data (like AlphaFold).
Cybersecurity is basically used like a dog whistle pioneered by Anthropic to achieve regulatory capture. Otherwise, the widespread availability of good tooling for security analysis would eliminate more of these cyber threats, rather than gatekeeping them for a few private companies.
> Data and content related to "biological weapons" already exist on the internet, in books, etc. The real issue is access to facilities and tools.
No, I think tools are easy to come by (unlike in nuclear research), the real issue is the know-how to create biological weapons, which you can't easily get out of books, but much more easily out of an amoral LLM.
This is just wrong. Standard textbooks like "Bioprocess Engineering Principles" [1] already contain step by step industrial engineering protocols and formulas. LLMs only summarize what print literature has documented for decades. The real bottleneck has always been hands on physical execution, not access to text. (This is not the case with actual models trained on biological data, which are not LLMs)
> amoral LLM
Are you a Lesswrong member? What makes Anthropic moral when their priority is automating people out of jobs instead of developing better medicine or advancing healthcare?
[1] https://www.sciencedirect.com/book/monograph/9780122208515/b...
1 reply →
I wouldn't be surprised if more resources were put into abliteration resistance the more capable open weight models become. It's something you don't need at all to start hosting the model on your own, but something you need to take care of before you release the weights (if you do care about it at all).
I'm curious what they mean by that too... They might be trying to weaken the cyber capabilities... Or I guess they might mean safety evaluation and hardening of the open source (and perhaps closed source Chinese) software ecosystem...
They need to make money. Let them do it. They deserve it. Also, this is what inference engines like vLLM want to have "zero day" supporr