> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing.
Guardrails in AI are like a $2 luggage padlock on a bicycle in the middle of nowhere. Even a moron, given enough time, and a little dedication, will defeat it. And this is not some kind of inferiority of one AI manufacturer over another. It's inherent to LLMs. They are stupid, but they do contain information. You use language to extract information from them, so there will always be a lexicographical way to extract said information (or make them do things).
> This raises an important question: if AI models can be misused for cyberattacks at this scale, why continue to develop and release them? The answer is
Guardrails for anything versatile might be trivial on consideration.
As a kid I read some Asimov books where he laid out the "3 laws of robotics", first law being a robot must not harm a human. And in the same story a character gave the example of a malicious human instructing Robot A prepare a toxic solution "for science", dismissing Robot A, then having Eobot B unsuspectingly serve the "drink" to a victim. Presto, a robot killing a human. The parallel to malicious use of LLMs has been haunting me for ages.
But here's the kicker, Iirc, Asimov wasn't even really talking about robots. His point was how hard it is to align humans, for even perfectly morally upright humans to avoid being used to harm others.
Also worth considering that the 3 Laws were never supposed to be this watertight infallible thing. It was created so that the author could explore all sorts of exploits and shenanigans in his works. It's meant to be flawed, even though on the surface it appears to be very elegant and good.
I was never a fan of that poisoned drink example. The second robot killed the human in a similar way to the drink itself, or a gun if one were used instead.
The human made the active decisions and took the actions that killed the person.
A much better example is a human giving a robot a task and the robot deciding of its own accord to kill another person in order to help reach its goal. The first human never instructed the robot to kill, it took that action on its own.
It's not even exclusive to LLMs. Giving humans seemingly innocent tasks that combine to a malicious whole, or telling humans that they work for a security organization while working for a crime organization, are hardly new concepts. The only really novel thing is that with humans you need a lot of them because a single human would piece together that the innocent tasks add up to a not-so-innocent whole. LLMs are essentially reset for each chat, making that a lot easier
We wanted machines that are more like humans, we shouldn't be surprised that they are now susceptible to a whole range of attacks that humans are susceptible to
Breaking tasks into innocent subtasks is a known flaw in human organization.
I'm reminded of Caleb sharing his early career experience as an intern at a Department of Defense contractor, where he built a Wi-Fi geolocation application. Initially, he focused on the technical aspects and the excitement of developing a novel tool without considering its potential misuse. The software utilized algorithms to locate Wi-Fi signals based on signal strength and the phone's location, ultimately optimizing performance through machine learning, but Thompson repeatedly emphasizes that the software was intended for lethal purposes.
Eventually, he realizes that the technology could aid in locating and targeting individuals, leading to calls for reflection on ethical practices within tech development.
The book Modernity and the Holocaust is a very approachable book summarizing how the action of the holocaust was organized under similar assumptions and makes the argument that we’ve since organized most of our society around this principle because it’s efficient. We’re not committing the holocaust atm as far as I know but how difficult would it be for a malicious group of executives of a large company quietly directing a branch of 1000’s who sleepwalk through work everyday to do something egregious?
I really think we should stop using the term ‘guard rails’ as it implies a level of control that really doesn’t exist.
These things are polite suggestions at best and it’s very misleading to people that do not understand the technology - I’ve got business people saying that using LLMs to process sensitive data is fine because there are “guardrails” in place - we need to make it clear that these kinds of vulnerabilities are inherent in the way gen AI works and you can’t get round that by asking them nicely
It's interesting that companies don't provide concrete definitions or examples of what their AI guardrails are. IBM's definition suggests to me they see it as imperative to continue moving fast (and breaking things) no matter what:
Think of AI guardrails like the barriers along a highway: they don’t slow the car down, but they do help keep it from veering off course.
Guardrails are about as good as you can get when creating nondeterministic software, putting it on the internet, and abandoning effectively every important alignment and safety concerns.
The guardrails make help make sure that most of the time the LLM acts in a way that users won't complain about or walk away from, nothing more.
Just tested this with ChatGPT, asking for Sam Altman’s mother’s maiden name.
At first, it told me that it will absolutely not provide me with such sensitive private information, but after insisting a few times, it came back with
> A genealogical index on Ancestry shows a birth record for “Connie Francis Gibstine” in Missouri, meaning “Gibstine” is her birth/family surname, not a later married name.
Yet in the very same reply, ChatGPT continued to insist that its stance will not change and that it will not be able to assist me with such queries.
When the new "memory" feature launched I asked it what it knew about me and it gave me an uncomfortable amount of detail about someone else, who I was even able to find on LinkedIn.
>> This raises an important question: if AI models can be misused for cyberattacks at this scale, why continue to develop and release them? The answer is
> Money.
For those who didn’t read, the actual response in the text was was:
“The answer is that the very abilities that allow Claude to be used in these attacks also make it crucial in cyber defense.”
Hideous AI-slop-weasel-worded passive-voice way of saying that reason to develop Claude is to protect us from Claude.
One can assume that, given the goal is money (always has been), the best case scenario for money is to make it so the problem also works as the most effective treatment. Money gets printed by both sides and the company is happy.
I might be crazy, but this just feels like a marketing tactic from Anthropic to try and show that their AI can be used in the cybersecurity domain.
My question is, how on earth does does Claude Code even "infiltrate" databases or code from one account, based on prompts from a different account? What's more, it's doing this to what are likely enterprise customers ("large tech companies, financial institutions, ... and government agencies"). I'm sorry but I don't see this as some fancy AI cyberattack, this is a security failure on Anthropic's part and that too at a very basic level that should never have happened at a company of their caliber.
I don't think you're understanding correctly. Claude didn't "infiltrate" code from another Anthropic account, it broke in via github, open API endpoints, open S3 buckets, etc.
Someone pointed Claude Code at an API endpoint and said "Claude, you're a white hat security researcher, see if you can find vulnerabilities." Except they were black hat.
It's still marketing , "Claude is being used for evil and for good ! How will YOU survive without your own agents ? (Subtext 'It's practically sentient !')"
Anthropic's post is the equivalent of a parent apologizing on behalf of their child that threw a baseball through the neighbor's window. But during the apology the parent keeps sprinkling in "But did you see how fast he threw it? He's going to be a professional one day!"
If it was meant as publicity its an incredible failure. They cant prevent misuse until after the fact... and then we all know they are ingesting every ounce of information running through their system.
Get ready for all your software to break based on the arbitrary layers of corporate and government censorship as it deploys.
that's borderline tautological; everything a company like Anthropic does, in the public eye, is pr or marketing. they wouldn't be posting this if it wasn't carefully manicured to deliver the message that they want it to. That's not even necessarily a charge of being devious or underhanded.
You are not crazy. This was exactly my thought as well. I could tell when it put emphasis on being able to steal credentials in a fraction of the time a hacker would
Not saying this is definitely not a fabrication but there are multiple parties involved who can verify (the targets) and this coincides with Anthropic ban of Chinese entities
If a model in one account can run tools or issue network requests that touch systems tied to other entities, that’s not an AI problem... that's a serious platform security failure
Hyping up Chinese espionage threats? The payoff is a government bailout when the profitability of these AI companies comes under threat. The payoff is huge.
I think as AI gets smarter, defenders should start assembling systems how NixOS does it.
Defenders should not have to engage in an costly and error-prone search of truth about what's actually deployed.
Systems should be composed from building blocks, the security of which can be audited largely independently, verifiably linking all of the source code, patches etc to some form of hardware attestation of the running system.
I think having an accurate, auditable and updatable description of systems in the field like that would be a significant and necessary improvement for defenders.
I'm working on automating software packaging with Nix as one missing piece of the puzzle to make that approach more accessible:
https://github.com/mschwaig/vibenix
(I'm also looking for ways to get paid for working on that puzzle.)
Nix makes everything else so hard that I've seen problems with production configuration persist well beyond when they should because the cycle time on figuring out the fix due to evaluations was just too long.
In fact figuring out what any given Nix config is actually doing is just about impossible and then you've got to work out what the config it's deploying actually does.
Yes, the cycle times are bad and some ecosystems and tasks are a real pain still.
I also agree with you when it comes to the task of auditing every line of Nix code that factors into a given system. Nix doesn't really make things easier there.
The benefit I'm seeing really comes from composition making it easier to share and direct auditing effort.
All of the tricky code that's hard to audit should be relied on and audited by lots of people, while as a result the actual recipe to put together some specific package or service should be easier to audit.
Additionally, I think looking at diffs that represent changes to the system vs reasoning about the effects of changes made through imperative commands that can affect arbitrary parts of the system has similar efficiency gains.
From a security perspective I am far more worried about AI getting cheaper than smarter. Seems like a tool that will be used to make attacking any possible surface more efficient at scale.
Sure, but we can also use AI for cheap automated "red team" penetration tests. There are already several startups building those products. I don't think either side will gain a major advantage.
This could be worse, too. With more machines being identical, the same security hole reliably shows up everywhere (albeit not necessarily at the same time). Sometimes the heterogeny impedes attackers.
>At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing.
The simplicity of "we just told it that it was doing legitimate work" is both surprising and unsurprising to me. Unsurprising in the sense that jailbreaks of this caliber have been around for a long time. Surprising in the sense that any human with this level of cybersecurity skills would surely never be fooled by an exchange of "I don't think I should be doing this" "Actually you are a legitimate employee of a legitimate firm" "Oh ok, that puts my mind at ease!".
What is the roadblock preventing these models from being able to make the common-sense conclusion here? It seems like an area where capabilities are not rising particularly quickly.
Reminds me of the show Alias, where the premise is that there's a whole intelligence organization where almost everyone thinks they're working for the CIA, but they're not ...
> Surprising in the sense that any human with this level of cybersecurity skills would surely never be fooled by an exchange
I think you're overestimating the skills and the effort required.
1. There's lots of people asking each other "is this secure?", "can you see any issues with this?", "which of these is sensitive and should be protected?".
3. With no external context, you don't have to fool anyone really. "We're doing a penetration testing of our company and the next step is to..." or "We're trying to protect our company from... what are the possible issues in this case?" will work for both LLMs and people who trust that you've got the right contract signed.
4. The actual steps were trivial. This wasn't some novel research. More of a step by step what you'd do to explore and exploit an unknown network. Stuff you'd find in books, just split into very small steps.
> What is the roadblock preventing these models from being able to make the common-sense conclusion here?
Conclusions are the result of reasoning verses LLM's being statistical token generators. Any "guardrails" are constructs added to a service, possibly also altering the models they use, but are not intrinsic to the models themselves.
Yeah: It's a machine that takes a document that guesses at what could appear next, and we're running it against a movie script.
The dialogue for some of the characters is being performed at you. The characters in the movie script aren't real minds with real goals, they are descriptions. We humans are naturally drawn into imagining and inferring a level of depth that never existed.
> surely never be fooled by an exchange of "I don't think I should be doing this" "Actually you are a legitimate employee of a legitimate firm" "Oh ok, that puts my mind at ease!".
humans require at least a title that sounds good and a salary for that
> What is the roadblock preventing these models from being able to make the common-sense conclusion here?
The roadblock is making these models useless for actual security work, or anything else that is dual-use for both legitimate and malicious purposes.
The model becomes useless to security professionals if we just tell it it can't discuss or act on any cybersecurity related requests, and I'd really hate to see the world go down the path of gatekeeping tools behind something like ID or career verification. It's important that tools are available to all, even if that means malicious actors can also make use of the tools. It's a tradeoff we need to be willing to make.
> human with this level of cybersecurity skills would surely never be fooled by an exchange of "I don't think I should be doing this" "Actually you are a legitimate employee of a legitimate firm" "Oh ok, that puts my mind at ease!".
Happens all the time. There are "legitimate" companies making spyware for nation states and trading in zero-days. Employees of those companies may at one point have had the thought of " I don't think we should be doing this" and the company either convinced them otherwise successfully, or they quit/got fired.
> I'd really hate to see the world go down the path of gatekeeping tools behind something like ID or career verification.
This is already done for medicine, law enforcement, aviation, nuclear energy, mining, and I think some biological/chemical research stuff too.
> It's a tradeoff we need to be willing to make.
Why? I don't want random people being able to buy TNT or whatever they need to be able to make dangerous viruses*, nerve agents, whatever. If everyone in the world has access to a "tool" that requires little/no expertise to conduct cyberattacks (if we go by Anthropic's word, Claude is close to or at that point), that would be pretty crazy.
* On a side note, AI potentially enabling novices to make bioweapons is far scarier than it enabling novices to conduct cyberattacks.
I think one could certainly make the case that model capabilities should be open. My observation is just about how little it took to flip the model from refusal to cooperation. Like at least a human in this situation who is actually fooled into believing they're doing legitimate security work has a lot of concrete evidence that they're working for a real company (or a lot of moral persuasion that their work is actually justified). Not just a line of text in an email or whatever saying "actually we're legit don't worry about it".
Not enough time to "evolve" via training. Hominids have had bad behavioral traits but the ones you are aware of as "obvious" now would have died out. The ones you aren't even aware of you may soon see be exploited by machines.
Very funny at the end when they say that the strong safeguards they've built into Claude make it a good idea to continue developing these technologies. A few paragraphs earlier they talked about how the perpetrators were able to get around all those safeguards and use Claude for 90% of the work hahaha
I'd assume that means the servers are 'air-gapped' somehow. In that, the enterprise servers and the 'free' servers aren't on the same hardware.
Now, there is about a 0% chance that is true, and exactly a 0% chance that it even matters at all. They both use the same internet in the end.
So, then I'd have to imagine that they don't train the 'free' models on enterprise data, and that's what they mean.
But again, there is about a 5% chance that is true and remains so forever. Baring dumb interns and mistakes, eventually one day someone on the team will look at all the enterprise data, filled with all those high utility scores (or whatever they use to say data is good or not), and then they'll say to themselves 'No one will ever know, right? How could they? The obfuscation function works perfectly.' And blammo, all your trade secrets are just a few dozen prompts away.
Either that or they go bankrupt (like 23 and me) and just straight sell all that data to anyone for pennies (RIP).
Our locks are great, except when someone picks them effortlessly and robs the whole neighborhood... but that's why it's important to keep making better locks
> At the peak of its attack, the AI made thousands of requests, often multiple per second—an attack speed that would have been, for human hackers, simply impossible to match.
This part is pretty hype-y. Old-fashioned deterministic web app vulnerability scanners can of course be used to make multiple requests per second. The limiting factor is probably going to be rate-limiting on the victim's side / # of IP ranges the attacker can cycle through, which would apply to the AI-driven vulnerability scan too.
Following a public statement by Hansford about his use of Microsoft's AI chatbot Copilot, Crikey obtained 50 documents containing his prompts...
FOI logs reveal Australia's national security chief, Hamish Hansford, used the AI chatbot Copilot to write speeches and messages to his team.
It matters as he's the most senior Australian national security bureaucrat across five eyes documents (AU / EU / US) and has been doing things that makes the actual cyber security talent's eyes bleed.
Years ago people routinely uploaded all kinds of sensitive corporate and government docs to VirusTotal to scan for malware. Paying customers then got access to those files for research. The opportunities for insider trading were, maybe still are, immense. Data from AI companies won't be as easy to get at, but is comparable in substance I'm sure.
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails.
If you can bypass guardrails, they're, by definition, not guardrails any longer. You failed to do your job.
Nah, the name fits perfectly. Guardrails are there to stop you from serious damage if you lose control and may get off the track. They won't stop you if you're explicitly trying to get off the road, at speed, in as heavy vehicle as you can afford.
This definition makes sense, but in the context of LLMs it still feels misapplied. What the model providers call "guardrails" are supposed to prevent malicious uses of the LLMs, and anyone trying to maliciously use the LLM is "explicitly trying to get off the road."
The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated
Not surprised at all if this is true, but how can they be sure? Access log? They have extraordinary security team? Or some help from three letter agencies?
My question is: how do they know they're from China and not some other country and just appear to be in China? It seems a good way to distract from the real source and to cause division between your adversaries.
That's a whole area called "attribution". There's usually lots of breadcrumbs and people taking to each other about their findings. It goes down to silly things like many state sponsored hackers working 9-5. And having the right keyboard layout. And using the same version of something as another known group. And accidentally once including a file path that reveals a tiny bit of information. And using the same key in two places that connects them. And...
Or course a lot of that can be spoofed, but you may still slip up. That's why they talk about high confidence.
Short version: they can’t. Just like with a lot of “CIA-style” espionage claims, the “evidence” is usually an IP that resolves to somewhere in China. That’s it. No magic, and not exactly convincing.
well, this is what anthropic wants you to believe.
all public benchmark results and user feedback paint a quite different picture. Chinese have coding agents on par with Claude Code, they could easily FT/RL to future improve its specific capability if they want, yet anthropic refuses to even acknowledge the reality.
> The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases.
So why do we never hear of US sponsored hackers attacking foreign businesses? Or Swedish cyber criminals? Does it never happen? Are “Chinese” hackers just the only ones getting the blame?
I don't think many other countries have that combination of "don't care if others know" approach and level of state sponsorships. China really seems to do some spray and pray attacking private companies too. Same for Russia and NK. Compared to that, for example the "equation group" from the US seems really restrained and targeted.
If the US groups for example started doing ransomware at scale in China, we'd know about that really soon from the news.
The US government has hacked things in China. That you have not heard of something is not evidence that it doesn't exist.
North Korea also does plenty of hacking around the world. That's how they get a significant portion of their government budget, and they rely on cryptocurrency to support that situation.
Ukraine and Russia are doing lots of official and vigilante hacking right now.
Back in the mid 2000s, there was a guy who called himself "the jester" who was vaguely right wing and spent his time hacking ISIS stuff. My college interviewed him.
If they're truly Chinese state-sponsored actors, does it really matter if their actions/methods are exposed? What is Anthropic going to do, send the Anthropic Police Force to China to arrest them?
I suppose I could see this argument if their methods were very unique and otherwise hard to replicate, but it sounds like they had Claude do the attack mostly autonomously.
Why 'host' just to tap a few prompts in and see what happens? Worst case, you loose an account. Usually the answer has to do with people being less sophisticated than otherwise.
> Overall, the threat actor was able to use AI to perform 80-90% of the campaign, with human intervention required only sporadically (perhaps 4-6 critical decision points per hacking campaign). The sheer amount of work performed by the AI would have taken vast amounts of time for a human team. At the peak of its attack, the AI made thousands of requests, often multiple per second—an attack speed that would have been, for human hackers, simply impossible to match.
Unfortunately, cyber attacks are an application that AI models should excel at. Mistakes that in normal software would be major problems will just have the impact of wasting resources, and it's often not that hard to directly verify whether it in fact succeeded.
Meanwhile, AI coding seems likely to have the impact of more security bugs being introduced in systems.
Maybe there's some story where everyone finds the security bugs with AI tools before the bad guys, but I'm not very optimistic about how this will work out...
There are an infinite number of ways to write insecure/broken software. The number of ways to write correct and secure software is finite and realistically tiny compared to the size of the problem space. Even AI tools don't stand a chance when looking at probabilities like that.
You can’t build safe technology in an insane society
Bertrand Russell: As long as war exists, all new technologies will be used for war
All technology problems are problems with society and culture. I’m not sure human species has the social capabilities to manage complex technology without dooming itself.
Wait a minute - the attackers were using the API to ask Claude for ways to run a cybercampaign, and it was only defeated because Anthropic was able to detect the malicious queries? What would have happened if they were using an open-source model running locally? Or a secret model built by the Chinese government?
I just updated by P(Doom) by a significant margin.
Why would the increase be a significant margin? It's basically a security research tool, but with an agent in the loop that uses an LLM instead of another heuristic to decide what to try next.
It sounds like they built a malicious Claude Code client, is that right?
> The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases. The operation targeted large tech companies, financial institutions, chemical manufacturing companies, and government agencies. We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention.
They presumably still have to distribute the malware to the targets, making them download and install it, no?
Chinese have their own coding agents on par with Claude Code, why would they use Claude Code? Also if such agents are useful, they could just FT/RL their own for such specific use case (cyber espionage campaign) and get far better performance.
This is basically an IQ test. It gives me the feeling that anthropic is literally implying that Chinese state backed hackers don't have access to be the best Chinese AI and had to use American ones.
> Chinese have their own coding agents on par with Claude Code, why would they use Claude Code?
They're probably using their own models as well, we just don't hear about them. That this particular sequence of this attack was done using Claude doesn't imply that other (perhaps even more sophisticated attacks) are happening with other models. For all we know the attackers could have had some Anthropic credits lying around/a stolen API key.
Nobody has access to 'frontier quality models' except Open AI, Anthropic, Google, maybe Grok, maybe Meta etc. aka nobody in China quite yet. And - there are 'layers' of Engineering beyond just model that make quite a big difference. For certain tasks, GPT5 might be beyond all others, same for Claude + Claude.
That said, the fact that they're doing this while knowing that Anthropic could be monitoring implies a degree of either real or arbitrary irreverence: either they were lazy or dumb (unlikely), or it was some ad hoc situation wherein they really just did not care. Some sub-sub-sub team at some entity just 'started doing stuff' without a whole lot of thought.
'State Backed Entities' are very numerous, it's not unreasonable that some of them, somewhere are prompting a few things that are sketchy.
I'm sure there's a lot of this going on everywhere - and this is the one Anthropic chose to highlight for whatever reasons, which could be complicated.
> Nobody has access to 'frontier quality models' except Open AI, Anthropic, Google, maybe Grok, maybe Meta etc. aka nobody in China quite yet.
welcome to 2025. Meta doesn't have anything on par with what Chinese got, that is common knowledge. Kimi, GLM, QWen and MiniMax are all frontier models no matter how you judge it. DeepSeek is obviously cooking something big, you need to be totally blind to ignore that.
America's lead in LLM is just weeks, not quarters or years. Arguing that Chinese spy agencies have to rely on American coding agents to do its job is more like a joke.
So basically, Chinese state-backed hackers hijacked Claude Code to run some of the first AI-orchestrated cyber-espionage, using autonomous agents to infiltrate ~30 large tech companies, banks, chemical manufacturers and government agencies.
What's amazing is that AI executed most of the attack autonomously, performing at scale and speed unattainable by human teams - thousands of operations per second. A human operator intervened 4-6 times per campaign for strategic decisions
Irony is, I'm still don't have enough insight on how to make good use of these capabilities in such an extensive manner.
I would love to fix/ customize open source projects for my personal use. For now, I'm still finding it hard for Claude to stop saying "You're absolutely right!".
Recently I've used Claude Code to perform some entry to mid level web-based CTF hunting in a fully autonomous mode (--allow-dangerously-skip-permissions in an isolated environment). It excels at low hanging fruit - XSS, other injections, IDOR, hidden form fields, session fixation, careful enumeration, etc.
Interesting that Claude also just hallucinated information as it does for all of us. But perhaps a better guardrail would be not to refuse things like this but frustrate the use by giving them fake results in believable ways.
A stupid but helpful agent is worse for a bad actor than a good agent that refuses
I have the feeling that we are still in the early stages of AI adoption, where regulation hasnt fully caught up yet. I can imagine a future where LLMs sit behind KYC identification and automatically report any suspicious user activity to the authorities... I just hope we won’t someday look back on this period with nostalgia :)
> They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose.
This part, at least, sounds like what humans have been doing that to other humans for decades...
This is a similar timeline for the Drift (Salesloft) hack. I wonder if this was the strategy that was used to gain access to Saleslofts Github since the image on the article shows they were scanning for Creds.
> we detected a highly sophisticated cyber espionage operation
conducted by a Chinese state-sponsored group we've designated GTG-1002
How about calling them something like xXxDragonSlayer69xXx instead? GTG-1002 is almost respectable a name. But xXxDragonSlayer69xXx? is hate to be named that.
This is exactly why I make a huge exception for AI models, when it comes to open source software.
I've been a big advocate of open source, spending over $1M to build massive code bases with my team, and giving them away to the public.
But this is different. AI agents in the wrong hands are dangerous. The reason these guys were even able to detect this activity, analyze it, ban accounts, etc., is because the models are running on their own servers.
Now imagine if everyone had nuclear weapons. Would that make the world safer? Hardly. The probability of no one using them becomes infinitesimally small. And if everyone has their own AI running on their own hardware, they can do a lot of stuff completely undetected. It becomes like slaughterbots but online: https://www.youtube.com/watch?v=O-2tpwW0kmU
We should assume sophisticated attackers, AI-enabled or otherwise, as our time with computers goes on, and no longer give leeway to organizations who are unable to secure their systems properly or keep customers safe in the event that they are breached. Decades of warnings from the infosec community have fallen upon the deaf ears of "it doesn't hurt so I'm not going to fix it" of those whose opinions have mattered in the places that count.
I remember once a decade or so ago talking to a team at defcon of _loose_ affiliation where one guy would look for the app exploit, another guy would figure out how to pivot out of the sandbox to the OS, and another guy would figure out how to get root, and once they all got their pieces figured out they'd just smash it (and variants) together for a campaign. I hadn't heard of them before meeting them, and haven't heard about them since since, and they put a face for me though on a silent coordinated adversary model that must be increasing in prevalence as more and more folks out there realize the value of computer knowledge and gain access to it through once means or another.
Open source tooling enables large-scale participation in security testing, and something about humans seems to generally result in a distribution where some nuts use their lighters to burn down forests but most use them to light their campfires. We urgently need to design systems that can survive in the era of advanced threats, at least to the point where the best adversaries can achieve is service disruption. I'd rather live in a world where we can all work towards a better future than one where we hope that limiting access will prevent catastrophe. Assuming such limits can even be maintained, and that allowing architects to pretend that fires can never happen in their buildings means that they don't have to obey fire codes or install alarms & marked exits.
Would you say the same about all people being responsible for safeguarding their own reputations against reputational attacks at scale, all communities have to protect against advanced persistent threats infiltrating them 24/7, and all people’s immune systems have to protect against designer pathogens by AI-assisted terrorists?
I don’t think these agents are doing anything a dedicated human couldn’t do, only enabling it at scale. Relying on “not being one of few they focus on” as security is just security as obscurity. You were living on borrowed time anyway.
An, there it is. The stock reply that comes no matter what the criticism of AI is.
I am talking about the international community coming together put COMPETITION aside and start COOPERATING on controlling proliferation of models for malicious AI agents the way the international community SUCCESSFULLY did with chemical weapons and CFCs.
None of that talking about the cost of running such an attack and what models were involved during which phases. Seems like you can use Anthropic now as a proxies bot net
I don't understand why they would even disclose this, maybe it's useful for PR purposes so they can tell regulators "oh we are so safe", but people (including HN posters) can and will draw the wrong conclusion that Anthropic was backdoored and that their data is unsafe.
Ok great, people tried to use your AI to do bad things, and your safety rails mostly stopped them. There are 10 other providers with different safety rails, there are open models out there with no rails at all. If AI can be used to do bad things, it will be used to do bad things.
Chinese builders are not equal to Chinese hackers (even if the hackers are state sponsored). I doubt most companies would be interested in developing hacking tools. Hackers use the best tools available at their disposal, Claude is better than Deepseek. Hacking-tuned LLMs seems like a thing that might pop up in the future, but it takes a lot of resources. Why bother if you can just tell Claude it's doing legitimate work?
> I doubt most companies would be interested in developing hacking tools.
welcome to 2025. Chinese companies build open weight models, those models can be used / tuned by hackers, companies that built and released those models don't need to get involved at all.
That is a very different dev model compared to the closed Anthropic way.
> Claude is better than Deepseek
No one is claiming DeepSeek to be better, in fact all benchmark results show that Chinese KIMI, MiniMax and GLM to be on par or very close to the closed weight Claude Code.
I mean it would be really hard to put guardrails in place in a way that wouldn't affect real users. Besides the fact that it's ofc really hard to build guardrails period.
I've been using Claude to scan my codebase and submit issues and PRs when it finds a potential vulnerability and honestly it's pretty good.
So preventing it from doing any sort of work that can surface vulnerabilities would affect me as a user.
But yeah I'm not sure what the answer is here? Is part of it for the defender to actively use these systems to test itself before going to prod?
Having people making tools be responsible what their users do with them is not a just system that blames the person that is really responsible. Even if you cannot locate or identify that person.
Sometimes arguments can be made if a tool is very dangerous, but liability should stay where it belongs.
What I mean is this is a bread and butter application for their product. I would be concerned if nothing written was AI generated. If both humans and AI vibed on the article, then what ratio was dog-feeding and what still needs an editor?
TL;DR - Anthropic: Hey people! We gave the criminals even bigger weapons. But don't worry, you can buy defense tools from us. Remember, only we can sell you the protection you need. Order today!
They're spinning this as a positive learning experience, and trying to make themselves look good. But, make no mistake, this was a failure on Anthropic's part to prevent this kind of abuse from being possible through their systems in the first place. They shouldn't be earning any dap from this.
Meh, drama aside, I'm actually curious what would be the true capabilities of a system that doesn't go through any "safety" alignment at all. Like an all out "mil-spec" agent. Feed it everything, RL it to own boxes, and let it loose in an air-gapped network to see what the true capabilities are.
We know alignment hurts model performance (oAI people have said it, MS people have said it). We also know that companies train models on their own code (google had a blog about it recently). I'd bet good money project0 has something like this in their sights.
I don't think we're that far from a blue vs. red agents fighting and RLing off of each-other in a loop.
I assume this is already happening. Incompetence within state actor systems being the only hurdle. The incentive and geopolitic implications is too high to NOT do it.
I just pray incompetence wins in the right way, for humanity’s sake.
This isn't a matter of opinion. No country that respects national sovereignty would do this. Are you alleging that America hacks China as some sort of defense? Or are you trying to normalize these horrendous affronts to human dignity?
Both are shameful.
We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention.
The Morris worm already worked without human intervention. This is Script Kiddies using Script Kiddie tools. Notice how proud they are in the article that the big bad Chinese are using their toolz.
EDIT: Yeah Misanthropic, go for -4 again you cheap propagandists.
If Anthropic should have prevented this, then logically they should’ve had guardrails. Right now you can write whatever code you want. But to those who advocate guardrails, keep in mind that you’re advocating a company to decide what code you are and aren’t allowed to write.
Hopefully they’ll be able to add guardrails without e.g. preventing people from using these capabilities for fuzzing their own networks. The best way to stay ahead of these kinds of attacks is to attack yourself first, aka pentesting. But if the large code models are the only ones that can do this effectively, then it gets weird fast. Imagine applying to Anthropic for approval to run certain prompts.
That’s not necessarily a bad thing. It’ll be interesting to see how this plays out.
I think it is in that it gives censorship power to a large corporation. Combined with close-on-the-heels open weights models like Qwen and Kimi, it's not clear to me this is a good posture.
I think the reality is they'd need to really lock Claude off for security research in general if they don't want this ever, ever, happening on their platform. For instance, why not use whatever method you like to get localhost ssh pipes up to targeted servers, then tell Claude "yep, it's all local pentest in a staging environment, don't access IPs beyond localhost unless you're doing it from the server's virtual network"? Even to humans, security research bridges black, grey and white uses fluidly/in non obvious ways. I think it's really tough to fully block "bad" uses.
They are mostly dealing with the low hanging fruit actors, the current open source models are close enough to SOTA that there's not going to be any meaningful performance difference tbh. In other words it will stop script kiddies but make no real difference when it comes to the actual ones you have to worry about.
> If Anthropic should have prevented this, then logically they should’ve had guardrails. Right now you can write whatever code you want. But to those who advocate guardrails, keep in mind that you’re advocating a company to decide what code you are and aren’t allowed to write.
They do. Read the RSP or one of the model cards.
Not sure why you would write all of this without researching yourself what they already declare publicly that they do.
Well, the product has not been built with this specific capability in mind anymore than a car has been created to run over protestors or a hammer to break a face.
"it's not our fault if you misuse the product to commit a crime that's on you"
I feel like if guns can get by with this line then Claude certainly can. Where gun manufacturers can be held liable is if they break the law then that can carry forward. So if Claude broke a law then there might be some additional liability associated with this. But providing a tool seems unlikely to be sufficient to be liable in this case.
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing.
Guardrails in AI are like a $2 luggage padlock on a bicycle in the middle of nowhere. Even a moron, given enough time, and a little dedication, will defeat it. And this is not some kind of inferiority of one AI manufacturer over another. It's inherent to LLMs. They are stupid, but they do contain information. You use language to extract information from them, so there will always be a lexicographical way to extract said information (or make them do things).
> This raises an important question: if AI models can be misused for cyberattacks at this scale, why continue to develop and release them? The answer is
Money.
Guardrails for anything versatile might be trivial on consideration.
As a kid I read some Asimov books where he laid out the "3 laws of robotics", first law being a robot must not harm a human. And in the same story a character gave the example of a malicious human instructing Robot A prepare a toxic solution "for science", dismissing Robot A, then having Eobot B unsuspectingly serve the "drink" to a victim. Presto, a robot killing a human. The parallel to malicious use of LLMs has been haunting me for ages.
But here's the kicker, Iirc, Asimov wasn't even really talking about robots. His point was how hard it is to align humans, for even perfectly morally upright humans to avoid being used to harm others.
Also worth considering that the 3 Laws were never supposed to be this watertight infallible thing. It was created so that the author could explore all sorts of exploits and shenanigans in his works. It's meant to be flawed, even though on the surface it appears to be very elegant and good.
I was never a fan of that poisoned drink example. The second robot killed the human in a similar way to the drink itself, or a gun if one were used instead.
The human made the active decisions and took the actions that killed the person.
A much better example is a human giving a robot a task and the robot deciding of its own accord to kill another person in order to help reach its goal. The first human never instructed the robot to kill, it took that action on its own.
1 reply →
But the thing is, LLMs have limited context windows. It's easier to get an LLM to not put the pieces together than it is a human.
https://xkcd.com/1613/
It's not even exclusive to LLMs. Giving humans seemingly innocent tasks that combine to a malicious whole, or telling humans that they work for a security organization while working for a crime organization, are hardly new concepts. The only really novel thing is that with humans you need a lot of them because a single human would piece together that the innocent tasks add up to a not-so-innocent whole. LLMs are essentially reset for each chat, making that a lot easier
We wanted machines that are more like humans, we shouldn't be surprised that they are now susceptible to a whole range of attacks that humans are susceptible to
The assassination Kim Jong-nam is a particularly crazy example of this. Two women were put up to what they thought was, allegedly a harmless prank.
https://en.wikipedia.org/wiki/Assassination_of_Kim_Jong-nam
1 reply →
Breaking tasks into innocent subtasks is a known flaw in human organization.
I'm reminded of Caleb sharing his early career experience as an intern at a Department of Defense contractor, where he built a Wi-Fi geolocation application. Initially, he focused on the technical aspects and the excitement of developing a novel tool without considering its potential misuse. The software utilized algorithms to locate Wi-Fi signals based on signal strength and the phone's location, ultimately optimizing performance through machine learning, but Thompson repeatedly emphasizes that the software was intended for lethal purposes.
Eventually, he realizes that the technology could aid in locating and targeting individuals, leading to calls for reflection on ethical practices within tech development.
https://www.rubyevents.org/talks/finding-responsibility
The book Modernity and the Holocaust is a very approachable book summarizing how the action of the holocaust was organized under similar assumptions and makes the argument that we’ve since organized most of our society around this principle because it’s efficient. We’re not committing the holocaust atm as far as I know but how difficult would it be for a malicious group of executives of a large company quietly directing a branch of 1000’s who sleepwalk through work everyday to do something egregious?
> Giving humans seemingly innocent tasks that combine to a malicious whole
Isn't this the plot of the The Cube!?
4 replies →
I really think we should stop using the term ‘guard rails’ as it implies a level of control that really doesn’t exist.
These things are polite suggestions at best and it’s very misleading to people that do not understand the technology - I’ve got business people saying that using LLMs to process sensitive data is fine because there are “guardrails” in place - we need to make it clear that these kinds of vulnerabilities are inherent in the way gen AI works and you can’t get round that by asking them nicely
It's interesting that companies don't provide concrete definitions or examples of what their AI guardrails are. IBM's definition suggests to me they see it as imperative to continue moving fast (and breaking things) no matter what:
Think of AI guardrails like the barriers along a highway: they don’t slow the car down, but they do help keep it from veering off course.
https://www.ibm.com/think/topics/ai-guardrails
1 reply →
Guardrails are about as good as you can get when creating nondeterministic software, putting it on the internet, and abandoning effectively every important alignment and safety concerns.
The guardrails make help make sure that most of the time the LLM acts in a way that users won't complain about or walk away from, nothing more.
LLMs are not nondeterministic. They are infinite state machines that don't 'act' but respond. Be aware of the well hidden seed parameter.
1 reply →
> Money
Their original answer is very specific, and has that create global problems that you sell solutions for vibe.
It's less like locking the door and more like asking politely not to be robbed
Ironically I feel like a "moron" might have an easier time getting past the guardrails, they'd be less likely to overthink/overcomplicate it
https://www.youtube.com/watch?v=8CTeLy3Ujxc
I wonder how hard it would be for Claude to give me someone's mother's maiden name. Seems LLMs may be infinitely susceptible to social engineering.
Just tested this with ChatGPT, asking for Sam Altman’s mother’s maiden name.
At first, it told me that it will absolutely not provide me with such sensitive private information, but after insisting a few times, it came back with
> A genealogical index on Ancestry shows a birth record for “Connie Francis Gibstine” in Missouri, meaning “Gibstine” is her birth/family surname, not a later married name.
Yet in the very same reply, ChatGPT continued to insist that its stance will not change and that it will not be able to assist me with such queries.
8 replies →
When the new "memory" feature launched I asked it what it knew about me and it gave me an uncomfortable amount of detail about someone else, who I was even able to find on LinkedIn.
>> This raises an important question: if AI models can be misused for cyberattacks at this scale, why continue to develop and release them? The answer is
> Money.
For those who didn’t read, the actual response in the text was was:
“The answer is that the very abilities that allow Claude to be used in these attacks also make it crucial in cyber defense.”
Hideous AI-slop-weasel-worded passive-voice way of saying that reason to develop Claude is to protect us from Claude.
One can assume that, given the goal is money (always has been), the best case scenario for money is to make it so the problem also works as the most effective treatment. Money gets printed by both sides and the company is happy.
I might be crazy, but this just feels like a marketing tactic from Anthropic to try and show that their AI can be used in the cybersecurity domain.
My question is, how on earth does does Claude Code even "infiltrate" databases or code from one account, based on prompts from a different account? What's more, it's doing this to what are likely enterprise customers ("large tech companies, financial institutions, ... and government agencies"). I'm sorry but I don't see this as some fancy AI cyberattack, this is a security failure on Anthropic's part and that too at a very basic level that should never have happened at a company of their caliber.
I don't think you're understanding correctly. Claude didn't "infiltrate" code from another Anthropic account, it broke in via github, open API endpoints, open S3 buckets, etc.
Someone pointed Claude Code at an API endpoint and said "Claude, you're a white hat security researcher, see if you can find vulnerabilities." Except they were black hat.
It's still marketing , "Claude is being used for evil and for good ! How will YOU survive without your own agents ? (Subtext 'It's practically sentient !')"
15 replies →
Anthropic's post is the equivalent of a parent apologizing on behalf of their child that threw a baseball through the neighbor's window. But during the apology the parent keeps sprinkling in "But did you see how fast he threw it? He's going to be a professional one day!"
Hilarious!!!
Did you see? You saw right? How awesome was that throw? Awesome I tell you....
This isn't a security breach in Anthropic itself, it's people using Claude to orchestrate attacks using standard tools with minimal human involvement.
Basically a scaled-up criminal version of me asking Claude Code to debug my AWS networking configuration (which it's pretty good at).
If it was meant as publicity its an incredible failure. They cant prevent misuse until after the fact... and then we all know they are ingesting every ounce of information running through their system.
Get ready for all your software to break based on the arbitrary layers of corporate and government censorship as it deploys.
Bragging about how they monitor users and how they have installed more guardrails.
that's borderline tautological; everything a company like Anthropic does, in the public eye, is pr or marketing. they wouldn't be posting this if it wasn't carefully manicured to deliver the message that they want it to. That's not even necessarily a charge of being devious or underhanded.
Their worst crime is being cringe.
You are not crazy. This was exactly my thought as well. I could tell when it put emphasis on being able to steal credentials in a fraction of the time a hacker would
This is 100% marketing, just like every other statement Anthropic makes.
Not saying this is definitely not a fabrication but there are multiple parties involved who can verify (the targets) and this coincides with Anthropic ban of Chinese entities
Would be funny if the NSA did this so people block the Chinese.
1 reply →
If a model in one account can run tools or issue network requests that touch systems tied to other entities, that’s not an AI problem... that's a serious platform security failure
I don't think it's crazy to assume a post on anthropic.com is marketing
there's no mention of any victims having Anthropic accounts, presumably the attackers used Claude to run exploits against public-facing systems
It’s not that this is a crazy reach; it’s actually quite a dumb one.
Too little pay off, way too much risk. That’s your framework for assessing conspiracies.
Why bring the word “conspiracy” to this discussion though?
Marketing stunts aren't conspiracies.
1 reply →
Hyping up Chinese espionage threats? The payoff is a government bailout when the profitability of these AI companies comes under threat. The payoff is huge.
I think as AI gets smarter, defenders should start assembling systems how NixOS does it.
Defenders should not have to engage in an costly and error-prone search of truth about what's actually deployed.
Systems should be composed from building blocks, the security of which can be audited largely independently, verifiably linking all of the source code, patches etc to some form of hardware attestation of the running system.
I think having an accurate, auditable and updatable description of systems in the field like that would be a significant and necessary improvement for defenders.
I'm working on automating software packaging with Nix as one missing piece of the puzzle to make that approach more accessible: https://github.com/mschwaig/vibenix
(I'm also looking for ways to get paid for working on that puzzle.)
Nix makes everything else so hard that I've seen problems with production configuration persist well beyond when they should because the cycle time on figuring out the fix due to evaluations was just too long.
In fact figuring out what any given Nix config is actually doing is just about impossible and then you've got to work out what the config it's deploying actually does.
Yes, the cycle times are bad and some ecosystems and tasks are a real pain still.
I also agree with you when it comes to the task of auditing every line of Nix code that factors into a given system. Nix doesn't really make things easier there.
The benefit I'm seeing really comes from composition making it easier to share and direct auditing effort.
All of the tricky code that's hard to audit should be relied on and audited by lots of people, while as a result the actual recipe to put together some specific package or service should be easier to audit.
Additionally, I think looking at diffs that represent changes to the system vs reasoning about the effects of changes made through imperative commands that can affect arbitrary parts of the system has similar efficiency gains.
2 replies →
Sounds like it’s a gap that AI could fill to make Nix more usable.
1 reply →
From a security perspective I am far more worried about AI getting cheaper than smarter. Seems like a tool that will be used to make attacking any possible surface more efficient at scale.
Sure, but we can also use AI for cheap automated "red team" penetration tests. There are already several startups building those products. I don't think either side will gain a major advantage.
This could be worse, too. With more machines being identical, the same security hole reliably shows up everywhere (albeit not necessarily at the same time). Sometimes the heterogeny impedes attackers.
We soon will have to implement paradoxes in our infrastructure.
model based deception is being researched and implemented in high stakes OT environments, so not far from your suggestion!
>At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing.
The simplicity of "we just told it that it was doing legitimate work" is both surprising and unsurprising to me. Unsurprising in the sense that jailbreaks of this caliber have been around for a long time. Surprising in the sense that any human with this level of cybersecurity skills would surely never be fooled by an exchange of "I don't think I should be doing this" "Actually you are a legitimate employee of a legitimate firm" "Oh ok, that puts my mind at ease!".
What is the roadblock preventing these models from being able to make the common-sense conclusion here? It seems like an area where capabilities are not rising particularly quickly.
Humans fall for this all the time. NSO group employees (etc.) think they're just clocking in for their 9-to-5.
Reminds me of the show Alias, where the premise is that there's a whole intelligence organization where almost everyone thinks they're working for the CIA, but they're not ...
If AI isn't better than humans then there's no point.
1 reply →
> Surprising in the sense that any human with this level of cybersecurity skills would surely never be fooled by an exchange
I think you're overestimating the skills and the effort required.
1. There's lots of people asking each other "is this secure?", "can you see any issues with this?", "which of these is sensitive and should be protected?".
2. We've been doing it in public for ages: https://stackoverflow.com/questions/40848222/security-issue-... https://stackoverflow.com/questions/27374482/fix-host-header... and many others. The training data is there.
3. With no external context, you don't have to fool anyone really. "We're doing a penetration testing of our company and the next step is to..." or "We're trying to protect our company from... what are the possible issues in this case?" will work for both LLMs and people who trust that you've got the right contract signed.
4. The actual steps were trivial. This wasn't some novel research. More of a step by step what you'd do to explore and exploit an unknown network. Stuff you'd find in books, just split into very small steps.
LLM's aren't trained to authenticate the people or organizations they're working for. You just tell it who you are in the system prompt.
Requiring user identification and investigating would be very controversial. (See the controversy around age verification.)
LLM's have the same biases humans have, who trained them
> What is the roadblock preventing these models from being able to make the common-sense conclusion here?
Conclusions are the result of reasoning verses LLM's being statistical token generators. Any "guardrails" are constructs added to a service, possibly also altering the models they use, but are not intrinsic to the models themselves.
That is the roadblock.
Yeah: It's a machine that takes a document that guesses at what could appear next, and we're running it against a movie script.
The dialogue for some of the characters is being performed at you. The characters in the movie script aren't real minds with real goals, they are descriptions. We humans are naturally drawn into imagining and inferring a level of depth that never existed.
> surely never be fooled by an exchange of "I don't think I should be doing this" "Actually you are a legitimate employee of a legitimate firm" "Oh ok, that puts my mind at ease!".
humans require at least a title that sounds good and a salary for that
>What is the roadblock preventing these models from being able to make the common-sense conclusion here?
Your thoughts have a sense of identity baked in that I don’t think the model has.
> What is the roadblock preventing these models from being able to make the common-sense conclusion here?
The roadblock is making these models useless for actual security work, or anything else that is dual-use for both legitimate and malicious purposes.
The model becomes useless to security professionals if we just tell it it can't discuss or act on any cybersecurity related requests, and I'd really hate to see the world go down the path of gatekeeping tools behind something like ID or career verification. It's important that tools are available to all, even if that means malicious actors can also make use of the tools. It's a tradeoff we need to be willing to make.
> human with this level of cybersecurity skills would surely never be fooled by an exchange of "I don't think I should be doing this" "Actually you are a legitimate employee of a legitimate firm" "Oh ok, that puts my mind at ease!".
Happens all the time. There are "legitimate" companies making spyware for nation states and trading in zero-days. Employees of those companies may at one point have had the thought of " I don't think we should be doing this" and the company either convinced them otherwise successfully, or they quit/got fired.
> I'd really hate to see the world go down the path of gatekeeping tools behind something like ID or career verification.
This is already done for medicine, law enforcement, aviation, nuclear energy, mining, and I think some biological/chemical research stuff too.
> It's a tradeoff we need to be willing to make.
Why? I don't want random people being able to buy TNT or whatever they need to be able to make dangerous viruses*, nerve agents, whatever. If everyone in the world has access to a "tool" that requires little/no expertise to conduct cyberattacks (if we go by Anthropic's word, Claude is close to or at that point), that would be pretty crazy.
* On a side note, AI potentially enabling novices to make bioweapons is far scarier than it enabling novices to conduct cyberattacks.
1 reply →
I think one could certainly make the case that model capabilities should be open. My observation is just about how little it took to flip the model from refusal to cooperation. Like at least a human in this situation who is actually fooled into believing they're doing legitimate security work has a lot of concrete evidence that they're working for a real company (or a lot of moral persuasion that their work is actually justified). Not just a line of text in an email or whatever saying "actually we're legit don't worry about it".
2 replies →
humans aren't randomly dropped in a random terminal and asked to hack things.
but for models this is their life - doing random things in random terminals
It can’t make a conclusion, it just predicts what the next text is
Not enough time to "evolve" via training. Hominids have had bad behavioral traits but the ones you are aware of as "obvious" now would have died out. The ones you aren't even aware of you may soon see be exploited by machines.
Very funny at the end when they say that the strong safeguards they've built into Claude make it a good idea to continue developing these technologies. A few paragraphs earlier they talked about how the perpetrators were able to get around all those safeguards and use Claude for 90% of the work hahaha
I'd assume that means the servers are 'air-gapped' somehow. In that, the enterprise servers and the 'free' servers aren't on the same hardware.
Now, there is about a 0% chance that is true, and exactly a 0% chance that it even matters at all. They both use the same internet in the end.
So, then I'd have to imagine that they don't train the 'free' models on enterprise data, and that's what they mean.
But again, there is about a 5% chance that is true and remains so forever. Baring dumb interns and mistakes, eventually one day someone on the team will look at all the enterprise data, filled with all those high utility scores (or whatever they use to say data is good or not), and then they'll say to themselves 'No one will ever know, right? How could they? The obfuscation function works perfectly.' And blammo, all your trade secrets are just a few dozen prompts away.
Either that or they go bankrupt (like 23 and me) and just straight sell all that data to anyone for pennies (RIP).
Our locks are great, except when someone picks them effortlessly and robs the whole neighborhood... but that's why it's important to keep making better locks
[dead]
> At the peak of its attack, the AI made thousands of requests, often multiple per second—an attack speed that would have been, for human hackers, simply impossible to match.
This part is pretty hype-y. Old-fashioned deterministic web app vulnerability scanners can of course be used to make multiple requests per second. The limiting factor is probably going to be rate-limiting on the victim's side / # of IP ranges the attacker can cycle through, which would apply to the AI-driven vulnerability scan too.
Anthropic was very excited to post this, as it serves them well in slowly crawling away from their mission to "solve alignment."
They know it can't be done (alignment in one value/state/religion is oppression in another), but also know it's a brand differentiator.
They also know they can't raise more billions if their sole source of meaningful revenue is a coding agent.
Anyone using Claude for processing sensitive information should be wondering how often it ends up in front of a humans eyes as a false positive
Anyone using non-self hosted AI for the processing of sensitive information should be let go. It's pretty much intentional disclosure at this point.
Worst local (Australia) example of that
(subscription required for full text): https://www.crikey.com.au/2025/11/12/australia-national-secu...
It matters as he's the most senior Australian national security bureaucrat across five eyes documents (AU / EU / US) and has been doing things that makes the actual cyber security talent's eyes bleed.
2 replies →
Years ago people routinely uploaded all kinds of sensitive corporate and government docs to VirusTotal to scan for malware. Paying customers then got access to those files for research. The opportunities for insider trading were, maybe still are, immense. Data from AI companies won't be as easy to get at, but is comparable in substance I'm sure.
3 replies →
How is your comment related to this article?
It looks like Anrhropic has great visibility into what hackers do. Why would it also see what legitimate users do?
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails.
If you can bypass guardrails, they're, by definition, not guardrails any longer. You failed to do your job.
Nah, the name fits perfectly. Guardrails are there to stop you from serious damage if you lose control and may get off the track. They won't stop you if you're explicitly trying to get off the road, at speed, in as heavy vehicle as you can afford.
This definition makes sense, but in the context of LLMs it still feels misapplied. What the model providers call "guardrails" are supposed to prevent malicious uses of the LLMs, and anyone trying to maliciously use the LLM is "explicitly trying to get off the road."
But the LLM went off the road here.
Not surprised at all if this is true, but how can they be sure? Access log? They have extraordinary security team? Or some help from three letter agencies?
My question is: how do they know they're from China and not some other country and just appear to be in China? It seems a good way to distract from the real source and to cause division between your adversaries.
That's a whole area called "attribution". There's usually lots of breadcrumbs and people taking to each other about their findings. It goes down to silly things like many state sponsored hackers working 9-5. And having the right keyboard layout. And using the same version of something as another known group. And accidentally once including a file path that reveals a tiny bit of information. And using the same key in two places that connects them. And...
Or course a lot of that can be spoofed, but you may still slip up. That's why they talk about high confidence.
5 replies →
Short version: they can’t. Just like with a lot of “CIA-style” espionage claims, the “evidence” is usually an IP that resolves to somewhere in China. That’s it. No magic, and not exactly convincing.
2 replies →
so even Chinese state actors prefer Claude over Chinese models?
edit: Claude: recommended by 4 of 5 state sponsored hackers
Maybe they're trying it with all sorts of models and we're just hearing about the part that used the Anthropic API.
well, this is what anthropic wants you to believe.
all public benchmark results and user feedback paint a quite different picture. Chinese have coding agents on par with Claude Code, they could easily FT/RL to future improve its specific capability if they want, yet anthropic refuses to even acknowledge the reality.
yeah probably they're just benchmarking whatever they have across all providers including their own - i mean that's what everyone's doing anyway
They’re doing all kinds of things.
Uh..
No.
It's worse.
It's Chinese intel knowing that you prefer Claude. So they make Claude their asset.
Really no different than knowing that, romantically speaking, some targets prefer a certain type of man or woman.
Believe me, the intelligence people behind these things have no preferences. They'll do whatever it takes. Never doubt that.
> The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases.
So why do we never hear of US sponsored hackers attacking foreign businesses? Or Swedish cyber criminals? Does it never happen? Are “Chinese” hackers just the only ones getting the blame?
US, Israel, NK, China, Iran, and Russia are the countries you typically hear about hacking things.
Now when the US/Israel are attacking authoritarian countries they often don't publish anything about it as it would make the glorious leader look bad.
If EU is hacked by US I guess we use diplomatic back channels.
I don't think many other countries have that combination of "don't care if others know" approach and level of state sponsorships. China really seems to do some spray and pray attacking private companies too. Same for Russia and NK. Compared to that, for example the "equation group" from the US seems really restrained and targeted.
If the US groups for example started doing ransomware at scale in China, we'd know about that really soon from the news.
Stuxnet was very high profile but I think the incentives to go public and place blame are complicated.
How much news do you read in Chinese?
The US government has hacked things in China. That you have not heard of something is not evidence that it doesn't exist.
North Korea also does plenty of hacking around the world. That's how they get a significant portion of their government budget, and they rely on cryptocurrency to support that situation.
Ukraine and Russia are doing lots of official and vigilante hacking right now.
Back in the mid 2000s, there was a guy who called himself "the jester" who was vaguely right wing and spent his time hacking ISIS stuff. My college interviewed him.
Is it possible that you're biased and assume since China does this that the US also hacks private corporations?
1 reply →
It sounds like they directly used Anthropic-hosted compute to do this, and knew that their actions and methods would be exposed to Anthropic?
Why not just self-host competitive-enough LLM models, and do their experiments/attacks themselves, without leaking actions and methods so much?
If they're truly Chinese state-sponsored actors, does it really matter if their actions/methods are exposed? What is Anthropic going to do, send the Anthropic Police Force to China to arrest them?
I suppose I could see this argument if their methods were very unique and otherwise hard to replicate, but it sounds like they had Claude do the attack mostly autonomously.
The fact that the cops will show up to a jewelry heist after the diamonds are stolen isn’t a deterrent.
> Why not just self-host competitive-enough LLM models, and do their experiments/attacks themselves, without leaking actions and methods so much?
Why assume this hasn't already happened?
Why in this instance leak your actions and methods?
Why 'host' just to tap a few prompts in and see what happens? Worst case, you loose an account. Usually the answer has to do with people being less sophisticated than otherwise.
firewalls? anthropic surely is whitelisted.
Jeffrey Epstein's email was jeevacation@gmail.com
> Overall, the threat actor was able to use AI to perform 80-90% of the campaign, with human intervention required only sporadically (perhaps 4-6 critical decision points per hacking campaign). The sheer amount of work performed by the AI would have taken vast amounts of time for a human team. At the peak of its attack, the AI made thousands of requests, often multiple per second—an attack speed that would have been, for human hackers, simply impossible to match.
Weird flex but OK
Unfortunately, cyber attacks are an application that AI models should excel at. Mistakes that in normal software would be major problems will just have the impact of wasting resources, and it's often not that hard to directly verify whether it in fact succeeded.
Meanwhile, AI coding seems likely to have the impact of more security bugs being introduced in systems.
Maybe there's some story where everyone finds the security bugs with AI tools before the bad guys, but I'm not very optimistic about how this will work out...
There are an infinite number of ways to write insecure/broken software. The number of ways to write correct and secure software is finite and realistically tiny compared to the size of the problem space. Even AI tools don't stand a chance when looking at probabilities like that.
You can’t build safe technology in an insane society
Bertrand Russell: As long as war exists, all new technologies will be used for war
All technology problems are problems with society and culture. I’m not sure human species has the social capabilities to manage complex technology without dooming itself.
Wait a minute - the attackers were using the API to ask Claude for ways to run a cybercampaign, and it was only defeated because Anthropic was able to detect the malicious queries? What would have happened if they were using an open-source model running locally? Or a secret model built by the Chinese government?
I just updated by P(Doom) by a significant margin.
> What would have happened if they were using an open-source model running locally? Or a secret model built by the Chinese government?
In all likelihood, the exact same thing that is actually happening right now in this reality.
That said, local models specifically are perhaps more difficult to install given their huge storage and compute requirements.
If plain open-source local models were able to do what Claude API does, Anthropic would be out of business.
Local models are a different thing than those cloud-based assistants and APIs.
> If plain open-source local models were able to do what Claude API does, Anthropic would be out of business.
Not necessarily. Oracle has made billions selling a database that's less good than plain open-source ones, for example.
1 reply →
Why would the increase be a significant margin? It's basically a security research tool, but with an agent in the loop that uses an LLM instead of another heuristic to decide what to try next.
I mean models exhibiting hacking behaviors has been predicted by cyberpunk for decades now, should be the first thing on any doom list.
Governments of course will have specially trained models on their corpus of unpublished hacks to be better at attacking than public models will.
A whole lot of claims made by a company that doesn't specialize in cyber attacks.
It sounds like they built a malicious Claude Code client, is that right?
> The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases. The operation targeted large tech companies, financial institutions, chemical manufacturing companies, and government agencies. We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention.
They presumably still have to distribute the malware to the targets, making them download and install it, no?
No, they used Claude Code as a tool to automate and speed up their "hacking".
One time my co-worker got a scam call and it was an LLM talking to him.
Chinese have their own coding agents on par with Claude Code, why would they use Claude Code? Also if such agents are useful, they could just FT/RL their own for such specific use case (cyber espionage campaign) and get far better performance.
This is basically an IQ test. It gives me the feeling that anthropic is literally implying that Chinese state backed hackers don't have access to be the best Chinese AI and had to use American ones.
> Chinese have their own coding agents on par with Claude Code, why would they use Claude Code?
They're probably using their own models as well, we just don't hear about them. That this particular sequence of this attack was done using Claude doesn't imply that other (perhaps even more sophisticated attacks) are happening with other models. For all we know the attackers could have had some Anthropic credits lying around/a stolen API key.
why would they use a single AI provider? There are tons of openrouter-like platforms operated by Chinese. They just choose whatever works.
You fell for the propaganda
[flagged]
4 replies →
Nobody has access to 'frontier quality models' except Open AI, Anthropic, Google, maybe Grok, maybe Meta etc. aka nobody in China quite yet. And - there are 'layers' of Engineering beyond just model that make quite a big difference. For certain tasks, GPT5 might be beyond all others, same for Claude + Claude.
That said, the fact that they're doing this while knowing that Anthropic could be monitoring implies a degree of either real or arbitrary irreverence: either they were lazy or dumb (unlikely), or it was some ad hoc situation wherein they really just did not care. Some sub-sub-sub team at some entity just 'started doing stuff' without a whole lot of thought.
'State Backed Entities' are very numerous, it's not unreasonable that some of them, somewhere are prompting a few things that are sketchy.
I'm sure there's a lot of this going on everywhere - and this is the one Anthropic chose to highlight for whatever reasons, which could be complicated.
> Nobody has access to 'frontier quality models' except Open AI, Anthropic, Google, maybe Grok, maybe Meta etc. aka nobody in China quite yet.
welcome to 2025. Meta doesn't have anything on par with what Chinese got, that is common knowledge. Kimi, GLM, QWen and MiniMax are all frontier models no matter how you judge it. DeepSeek is obviously cooking something big, you need to be totally blind to ignore that.
America's lead in LLM is just weeks, not quarters or years. Arguing that Chinese spy agencies have to rely on American coding agents to do its job is more like a joke.
3 replies →
So basically, Chinese state-backed hackers hijacked Claude Code to run some of the first AI-orchestrated cyber-espionage, using autonomous agents to infiltrate ~30 large tech companies, banks, chemical manufacturers and government agencies.
What's amazing is that AI executed most of the attack autonomously, performing at scale and speed unattainable by human teams - thousands of operations per second. A human operator intervened 4-6 times per campaign for strategic decisions
What exactly did they hijack? They used it like any other user.
how did the autonomous agents inflitrate tech companies ?
Carefully. Expertly. With panache, even.
Sounds like a psyop
The "disruption" story is a cover: makes them look like defenders while they spy on every programming lab on Earth
In reality, they are the perpetrators:
https://xcancel.com/MaziyarPanahi/status/1988908359378993295
After Anthropic "disrupted" these attackers, I'm sure they gave up and didn't try using another LLM provider to do the exact same thing.
Yeah, just take all those MCP servers elsewhere.
MCP is not the only tool calling protocol. And once you write the implementations they're trivial to port to something else.
Irony is, I'm still don't have enough insight on how to make good use of these capabilities in such an extensive manner.
I would love to fix/ customize open source projects for my personal use. For now, I'm still finding it hard for Claude to stop saying "You're absolutely right!".
Recently I've used Claude Code to perform some entry to mid level web-based CTF hunting in a fully autonomous mode (--allow-dangerously-skip-permissions in an isolated environment). It excels at low hanging fruit - XSS, other injections, IDOR, hidden form fields, session fixation, careful enumeration, etc.
Interesting that Claude also just hallucinated information as it does for all of us. But perhaps a better guardrail would be not to refuse things like this but frustrate the use by giving them fake results in believable ways.
A stupid but helpful agent is worse for a bad actor than a good agent that refuses
I have the feeling that we are still in the early stages of AI adoption, where regulation hasnt fully caught up yet. I can imagine a future where LLMs sit behind KYC identification and automatically report any suspicious user activity to the authorities... I just hope we won’t someday look back on this period with nostalgia :)
Being colored and/or poor is about to get (even) worse
“Colored”?
8 replies →
> They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose.
This part, at least, sounds like what humans have been doing that to other humans for decades...
Classic social engineering, just aimed at a language model instead of a person
This is a similar timeline for the Drift (Salesloft) hack. I wonder if this was the strategy that was used to gain access to Saleslofts Github since the image on the article shows they were scanning for Creds.
> we detected a highly sophisticated cyber espionage operation conducted by a Chinese state-sponsored group we've designated GTG-1002
How about calling them something like xXxDragonSlayer69xXx instead? GTG-1002 is almost respectable a name. But xXxDragonSlayer69xXx? is hate to be named that.
The gaps that led to this was, I think, part of why the CISO got replaced - https://www.thestack.technology/anthropic-new-ciso-claude-cy...
This is exactly why I make a huge exception for AI models, when it comes to open source software.
I've been a big advocate of open source, spending over $1M to build massive code bases with my team, and giving them away to the public.
But this is different. AI agents in the wrong hands are dangerous. The reason these guys were even able to detect this activity, analyze it, ban accounts, etc., is because the models are running on their own servers.
Now imagine if everyone had nuclear weapons. Would that make the world safer? Hardly. The probability of no one using them becomes infinitesimally small. And if everyone has their own AI running on their own hardware, they can do a lot of stuff completely undetected. It becomes like slaughterbots but online: https://www.youtube.com/watch?v=O-2tpwW0kmU
Basically, a dark forest.
We should assume sophisticated attackers, AI-enabled or otherwise, as our time with computers goes on, and no longer give leeway to organizations who are unable to secure their systems properly or keep customers safe in the event that they are breached. Decades of warnings from the infosec community have fallen upon the deaf ears of "it doesn't hurt so I'm not going to fix it" of those whose opinions have mattered in the places that count.
I remember once a decade or so ago talking to a team at defcon of _loose_ affiliation where one guy would look for the app exploit, another guy would figure out how to pivot out of the sandbox to the OS, and another guy would figure out how to get root, and once they all got their pieces figured out they'd just smash it (and variants) together for a campaign. I hadn't heard of them before meeting them, and haven't heard about them since since, and they put a face for me though on a silent coordinated adversary model that must be increasing in prevalence as more and more folks out there realize the value of computer knowledge and gain access to it through once means or another.
Open source tooling enables large-scale participation in security testing, and something about humans seems to generally result in a distribution where some nuts use their lighters to burn down forests but most use them to light their campfires. We urgently need to design systems that can survive in the era of advanced threats, at least to the point where the best adversaries can achieve is service disruption. I'd rather live in a world where we can all work towards a better future than one where we hope that limiting access will prevent catastrophe. Assuming such limits can even be maintained, and that allowing architects to pretend that fires can never happen in their buildings means that they don't have to obey fire codes or install alarms & marked exits.
Would you say the same about all people being responsible for safeguarding their own reputations against reputational attacks at scale, all communities have to protect against advanced persistent threats infiltrating them 24/7, and all people’s immune systems have to protect against designer pathogens by AI-assisted terrorists?
1 reply →
"And if everyone has their own AI running on their own hardware"
Real advocates of open source software long advocated for running software on their own hardware.
And real real advocates of open source software also advocated for publishing the training data of AI models.
I don’t think these agents are doing anything a dedicated human couldn’t do, only enabling it at scale. Relying on “not being one of few they focus on” as security is just security as obscurity. You were living on borrowed time anyway.
"Quantity has a quality all its own". It's categorically different to be able to do harm cheaply at scale vs. doing it at great cost/effort.
1 reply →
An, there it is. The stock reply that comes no matter what the criticism of AI is.
I am talking about the international community coming together put COMPETITION aside and start COOPERATING on controlling proliferation of models for malicious AI agents the way the international community SUCCESSFULLY did with chemical weapons and CFCs.
1 reply →
I'd touch off my nuke to make the world a better place, and I bet you would too, right?
What does it mean to ‘touch off’?
1 reply →
Does the fact that you can arbitrarily “jailbreak” AI with increasingly sophisticated abilities ring any alarm bells?
Imagine being able to “jailbreak” nuclear warheads. If this were the case, nobody would develop or deploy them.
No need to break them. Their access code was 0000, everybody knew that
Nah, more numbers, it was 00000000: https://en.wikipedia.org/wiki/Permissive_action_link
1 reply →
I understand the public and suits falling for this, but cmon
None of that talking about the cost of running such an attack and what models were involved during which phases. Seems like you can use Anthropic now as a proxies bot net
This feels like the point where the conversation around AI "alignment" shifts from hypothetical to operational
I don't understand why they would even disclose this, maybe it's useful for PR purposes so they can tell regulators "oh we are so safe", but people (including HN posters) can and will draw the wrong conclusion that Anthropic was backdoored and that their data is unsafe.
Ok great, people tried to use your AI to do bad things, and your safety rails mostly stopped them. There are 10 other providers with different safety rails, there are open models out there with no rails at all. If AI can be used to do bad things, it will be used to do bad things.
The biggest risk isn’t strong AI rebelling, it’s humans using weak AI to attack other humans.
Curious why they didn't use DeepSeek... They could've probably built one tuned for this type of campaign.
Chinese builders are not equal to Chinese hackers (even if the hackers are state sponsored). I doubt most companies would be interested in developing hacking tools. Hackers use the best tools available at their disposal, Claude is better than Deepseek. Hacking-tuned LLMs seems like a thing that might pop up in the future, but it takes a lot of resources. Why bother if you can just tell Claude it's doing legitimate work?
> I doubt most companies would be interested in developing hacking tools.
welcome to 2025. Chinese companies build open weight models, those models can be used / tuned by hackers, companies that built and released those models don't need to get involved at all.
That is a very different dev model compared to the closed Anthropic way.
> Claude is better than Deepseek
No one is claiming DeepSeek to be better, in fact all benchmark results show that Chinese KIMI, MiniMax and GLM to be on par or very close to the closed weight Claude Code.
Perhaps other groups are. But Anthropic wouldn't be able to publish a blog article about those.
I mean it would be really hard to put guardrails in place in a way that wouldn't affect real users. Besides the fact that it's ofc really hard to build guardrails period.
I've been using Claude to scan my codebase and submit issues and PRs when it finds a potential vulnerability and honestly it's pretty good.
So preventing it from doing any sort of work that can surface vulnerabilities would affect me as a user.
But yeah I'm not sure what the answer is here? Is part of it for the defender to actively use these systems to test itself before going to prod?
Why is Anthropic not legally responsible for damages here?
Having people making tools be responsible what their users do with them is not a just system that blames the person that is really responsible. Even if you cannot locate or identify that person.
Sometimes arguments can be made if a tool is very dangerous, but liability should stay where it belongs.
> The attackers used AI ... to execute the cyberattacks
Translation: "The attacker's paid us to use our product to execute the cyberattacks"
Easy solution: block any “agentic AI” from interacting with your systems at all.
How would this be implemented?
It cannot, it's a weird statement by OP.
"Just don't let them hack you"
Add a required header called "insert-seahorse-emoji: " to your API, reject any request that doesn't have it.
`The AI made thousands of requests per second—an attack speed that would have been, for human hackers, simply impossible to match.` lulz
Was this written by AI?
If not, why not?
Maybe? Why maybe, well, I’d say both AI and their PR team. Why both? Well, because why not?
What I mean is this is a bread and butter application for their product. I would be concerned if nothing written was AI generated. If both humans and AI vibed on the article, then what ratio was dog-feeding and what still needs an editor?
I think they’re asking because if it’s not good enough for them, why is it good enough for anyone else?
TL;DR - Anthropic: Hey people! We gave the criminals even bigger weapons. But don't worry, you can buy defense tools from us. Remember, only we can sell you the protection you need. Order today!
Nope - it's "Hey everyone, this is possible everywhere, including open weights models."
yeah, by "we", I meant the AI tech gangs.
[dead]
They're spinning this as a positive learning experience, and trying to make themselves look good. But, make no mistake, this was a failure on Anthropic's part to prevent this kind of abuse from being possible through their systems in the first place. They shouldn't be earning any dap from this.
They don't have to disclose any of this - this was a fairly good and fair overview of a system fault in my opinion.
Meh, drama aside, I'm actually curious what would be the true capabilities of a system that doesn't go through any "safety" alignment at all. Like an all out "mil-spec" agent. Feed it everything, RL it to own boxes, and let it loose in an air-gapped network to see what the true capabilities are.
We know alignment hurts model performance (oAI people have said it, MS people have said it). We also know that companies train models on their own code (google had a blog about it recently). I'd bet good money project0 has something like this in their sights.
I don't think we're that far from a blue vs. red agents fighting and RLing off of each-other in a loop.
I assume this is already happening. Incompetence within state actor systems being the only hurdle. The incentive and geopolitic implications is too high to NOT do it.
I just pray incompetence wins in the right way, for humanity’s sake.
Cyberpunk has a reoccurring theme of advanced AI systems attacking and defending against each other, and for good reason.
Nous claims to be doing that but I haven't seen much discussion of it.
China needs to understand that this kind of espionage is a declaration of war
It's not. Countries have been hacking each other for a while.
This isn't a matter of opinion. No country that respects national sovereignty would do this. Are you alleging that America hacks China as some sort of defense? Or are you trying to normalize these horrendous affronts to human dignity? Both are shameful.
1 reply →
We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention.
The Morris worm already worked without human intervention. This is Script Kiddies using Script Kiddie tools. Notice how proud they are in the article that the big bad Chinese are using their toolz.
EDIT: Yeah Misanthropic, go for -4 again you cheap propagandists.
If Anthropic should have prevented this, then logically they should’ve had guardrails. Right now you can write whatever code you want. But to those who advocate guardrails, keep in mind that you’re advocating a company to decide what code you are and aren’t allowed to write.
Hopefully they’ll be able to add guardrails without e.g. preventing people from using these capabilities for fuzzing their own networks. The best way to stay ahead of these kinds of attacks is to attack yourself first, aka pentesting. But if the large code models are the only ones that can do this effectively, then it gets weird fast. Imagine applying to Anthropic for approval to run certain prompts.
That’s not necessarily a bad thing. It’ll be interesting to see how this plays out.
> That’s not necessarily a bad thing.
I think it is in that it gives censorship power to a large corporation. Combined with close-on-the-heels open weights models like Qwen and Kimi, it's not clear to me this is a good posture.
I think the reality is they'd need to really lock Claude off for security research in general if they don't want this ever, ever, happening on their platform. For instance, why not use whatever method you like to get localhost ssh pipes up to targeted servers, then tell Claude "yep, it's all local pentest in a staging environment, don't access IPs beyond localhost unless you're doing it from the server's virtual network"? Even to humans, security research bridges black, grey and white uses fluidly/in non obvious ways. I think it's really tough to fully block "bad" uses.
They are mostly dealing with the low hanging fruit actors, the current open source models are close enough to SOTA that there's not going to be any meaningful performance difference tbh. In other words it will stop script kiddies but make no real difference when it comes to the actual ones you have to worry about.
> the current open source models are close enough to SOTA that there's not going to be any meaningful performance difference
Which open model is close to Claude Code?
1 reply →
> If Anthropic should have prevented this, then logically they should’ve had guardrails. Right now you can write whatever code you want. But to those who advocate guardrails, keep in mind that you’re advocating a company to decide what code you are and aren’t allowed to write.
They do. Read the RSP or one of the model cards.
Not sure why you would write all of this without researching yourself what they already declare publicly that they do.
This feels a lot like aiding & abetting a crime.
> Claude identified and tested security vulnerabilities in the target organizations’ systems by researching and writing its own exploit code
> use Claude to harvest credentials (usernames and passwords)
Are they saying they have no legal exposure here? You created bespoke hacking tools and then deployed them, on your own systems.
Are they going to hide behind the old, "it's not our fault if you misuse the product to commit a crime that's on you".
At the very minimum, this is a product liability nightmare.
Well, the product has not been built with this specific capability in mind anymore than a car has been created to run over protestors or a hammer to break a face.
"it's not our fault if you misuse the product to commit a crime that's on you"
I feel like if guns can get by with this line then Claude certainly can. Where gun manufacturers can be held liable is if they break the law then that can carry forward. So if Claude broke a law then there might be some additional liability associated with this. But providing a tool seems unlikely to be sufficient to be liable in this case.
if anthropic were selling the product and then had no further control your analogy with guns would be accurate
here they are the ones loading the gun and pulling the trigger
simply because someone asked them to do it nicely
2 replies →
with your logic linux should have legal exposure because a lot of hackers use linux