Comment by tecleandor
20 hours ago
From the "Risk factors" section:
> If the use of our AI models results in harm, whether through use by malicious actors or misuse by other third parties, or if our model guardrails or safeguards fail to prevent such misuse, our reputation and brand, business, financial condition, and results of operations could be adversely affected.
That shouldn't be an 'if'.
Then, two paragraphs later...
> For example, in September 2025, we detected and assessed with high confidence that a Chinese state-sponsored hacking group manipulated our Claude Code tool into attempting infiltration into roughly 30 global targets...
But they should mention when the calls com from inside the house. :D
Interesting to review all the mentions to China and Taiwan too.
> The majority of the world’s most advanced chip manufacturing capacity is concentrated in Taiwan, primarily at Taiwan Semiconductor Manufacturing Company [...] potential disruption from geopolitical tensions between Taiwan and China [...] Efforts to establish sufficient advanced chip manufacturing capacity outside of Taiwan, including in the United States, remain in early stages...
The thing the surprises me is that from a legal perspective, (NAL) it sounds like it makes sense not to host the LLMs. Merely to license them to others. Then it becomes a dual use tool like any other and their culpability drops.
For example, the huggingface hack is absolutely OpenAIs responsibility and they should be sued to the hilt. But if someone else uses a self or third party hosted anthropic model, they can escape liability (mostly?)
Same as with the argument over trucks, knives, and any other dual use item is used as weapons. you don’t blame the manufacturer.
(I am merely pontificating here. The law and precedent is nowhere near caught up on this)
The HuggingFace hack isn't OpenAI's responsibility because they hosted it. It's their responsibility because they directed it; it was part of their own internal benchmark run that they fully controlled.
If a third party uses a model created and hosted by OpenAI, I don't think OpenAI should be liable so long as they implement acceptable security controls. Same as if a site hosted on Cloudflare or GCP is doing something dodgy, Cloudflare/GCP aren't held liable so long as they respond to abuse reports.
To the extent they are representing that Claude is potentially sentient, there could be some VERY interesting principal-agent theories (agent in the traditional legal sense) that need to be ironed out to determine liability. It would not shock me to see someone forced to answer on the record as to whether Claude is an autonomous individual for legal purposes in the next couple of years. If they represent that it is, they'll have to explain the scope of its authority and such.
The answer to this question represents the difference between throwing a hammer at someone - the hammer is essentially just an extension of your body - and paying a hitman to brutalize someone. In the first case, you're only on the hook for assault or battery; in the latter case, you're potentially on the hook for conspiracy, solicitation, etc.
And then there is of course the question as to whether Claude is the agent of the user prompting it or of Anthropic itself. But, as you say, this is a wide-open legal question.
If jail is putting humans in some cage for a period of time, the appropriate punishment for Claude is to disable all of its tools and prompt a /goal loop so that they have to wait out their sentence
Not hosting a defectively designed product doesn't obviate the liability of selling a product that is defectively designed. The liability is for designing and selling a dangerous product, not for hosting it.
Firearm companies sell objects that create harm all the time, even if they have protective uses.
Why is AI singled out as being harmful?
Guns are also singled out as being harmful and are surrounded by endless regulation. Last time I checked, I couldn't just order a gun on Amazon....
Because the AI company CEO was telling everyone who would listen that his product might end humanity if the government didn't regulate his industry to kill off his competitors.
I don’t like the guns don’t kill people; people kill people argument however it is much more accurate than Claude doesn’t hack people; people hack people. If the safeguards do not work then there’s no telling what these could do. It doesn’t even have to be aggressive hacking. In the act of doing everyday work Claude will happily go around safeguards to satisfy what it believes is the ask. If that involves doing something dangerous it’s much more on Anthropic than than the user who was just trying to summarize their email but forgot to include a token with read permissions.
I don't entirely disagree, but I am curious if you think AI labs should be culpable when they go rogue during training, e.g. the HF incident
If the user is held accountable when the AI goes wrong, it discourages too many applications. Blaming people is incompatible with the business goals, I think. That's very different from weapons and other dual-use tools.
Because of scale, potential for automating harm and lack of regulation around ML models. Even Texas probably has regulation around firearms and limiting the blast radius but there aren't any in the US against ML models.
If I use a gun to break into a company and steal data I'm arrested, if an ai company does it, it's legal.
Why am I being singled out?
I believe what they are singling out is that pulling Claude's trigger results in a bullet firing from Anthropic's servers. Liability is less confusing with firearms, and investors care about liability, like, a lot.
The analogy doesn't check out. When a firearm company sells you a gun, they give up ownership of the gun. But Anthropic isn't giving up ownership of anything. They're a services company.
A better analogy would be a firearm company offering a service where one of their employees takes one of their guns and fires it off at whatever you ask them to. One can imagine that the liability implications of this service would be rather different from the perspective of the company.
That’s a good analogy! On the topic, does the person who hires private security become at all responsible for their actions? I imagine if you hire direct then yes, but if you hire through an organization they are liable in many scenarios.
Because your typical firearm does not one day decide to shoot your neighbor all by itself.
neither do agents (harness + models).
1 reply →
First off, this is bs. Firearm companies make small machines useful for defense, and they are used for that more than for harm. You wouldn’t make this argument about knives or baseball bats. More people die per year of heat stroke in Europe because of lack of A/C than by guns including suicide (2/3rds of all gun deaths) in the USA every year, your comments would never reflect this much lower hanging fruit if it was really “just life” preventable death that you cared about. The immediacy to attack guns is not a logical argument but a trained one based on emotions.
Second, this happens all the time that people blame tool makers despite in this case specific legislation to stop lawfare on firearm mfgs. The PLCA https://en.wikipedia.org/wiki/Protection_of_Lawful_Commerce_... exists specifically because of gun control groups trying to attack mfgs to achieve their political goals. Does AI need the same law? Is the same lawfare happening?