Comment by duplessitous
1 month ago
You can put lipstick on a pig, it'll still be a pig
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.
I have no idea what to do about the government interference. There's probably not a lot anyone can do.
However, your last point is quite a strong one. Corpos aren't just going to stand there with their collective pants down, and there's not a lot anyone can do to stop them from protecting themselves. There are ways they can get what they want without getting caught.
Remember when the US tried to ban strong cryptography in the 1990s, and how well that went? They may have more leverage with AI because it's a bit harder to hide large scale computing usage, but I don't think it's impossible at all.
What do they even really mean by "safety"? I mean, I can have an Anthropic model do something incredibly unsafe if, for example, I put it in charge of a hydroelectric dam and don't explain properly how the controls work. On some level, everything is simultaneously "safe" and "unsafe". I've never found Amodei's reasoning here to be particularly well thought-through. I think he, like a lot of folks in the area, are starting to realize that they may never be able to build a moat around their businesses.
It's political doublespeak. They want to have their cake and eat it too.
> an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China.
HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly.
> these statements are counter-factual.
The OpenAI incident is a single example. You're massively overgeneralizing. You can't refute an entire class of possible outcomes based on a single event where it went the other way.
I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise.
Edit: just to clarify my position, I don't love Anthropic so much. I think they're marginally better, but I'd still like to see regulation strangle everyone so we get another 20 years to figure this shit out.
"HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly."
HF released a statement and made it clear a closed source model specialized in cyber security refused them. They stated they had to use open source. What model is specialized in cyber security, closed, and frequently denies users access other than Mythos/Fable and 5.5Cyber? If not these two, what was HF referring to? It sounds like you have a source, I would like to read it.
fwipsy is right, cnbc has a story on this. they only had fable. I still think this is horrible for closed source, get on a list or else, but i was wrong
"You can't refute an entire class of possible outcomes based on a single event where it went the other way."
But Dario can dream up and entire class of outcomes based on the zero events that have never gone his way? Convenient.
The OpenAI incident is singular and HF was clear, it went exactly how I wrote it: a closed source American AI decided to perform corporate espionage and the only tool available was open source AI from China
"I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise."
We literally just saw an advanced model from openAI commit a cyber crime. I can't take hypotheticals that ignore reality seriously and it shouldn't be lauded as some higher form of thought
I assume you mean https://huggingface.co/blog/security-incident-july-2026. It says that they used frontier models, not frontier cybersecurity models. My reading is that they asked Fable and it refused; if they'd had access to Mythos, it would have helped. You're mixing the two but they're NOT the same model.
Source is here: https://thezvi.substack.com/p/more-on-an-internal-openai-mod... ctrl+f "Skill issue." No source is cited, but I'm fairly confident it's correct. If Mythos/5.5Cyber specifically had refused to help, then HF would have made a much bigger deal out of it. The whole point of these models is that they have relaxed guardrails and specialty cybersecurity training relative to the publicly-available ones.
> zero events
What about all of the vulnerabilities already patched under Project Glasswing?
In the quote you provided Amodei is expressing uncertainty, saying we don't know which way things will go. You're the one making strong assertions; the burden of proof is on you.
2 replies →
Amodei isn't ignoring reality; he's just proposing a different solution to the problem. If it were one of Anthropic's models, then that would be a much stronger case.
Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable.
4 replies →
I don't really understand how they can argue the security angle with a straight face. It's not like GLM 5.2 is a slouch. I've seen it do things like exploit an IDOR issue when I was experimenting with a quick-and-dirty web automation task. I simply fixed it, as one does. Open models make the world better to a far greater degree than they set it aflame.
Their position is analogous to trying to, say, ensure digital privacy for everyone not by making encryption freely available (because that would let the bad guys use it!), but by making it so you can't use general purpose communications devices that can listen to transmissions not intended for you. Do they hear how moronic that sounds?
Each passing frontier-level open model release makes Anthropic's patronizing rhetoric a little more insufferable, because it becomes clearer how unmoored from reality they've become in pursuit of profit.