Comment by ActorNightly
5 days ago
>The technology held by private AI companies is warfare-capable technology.
This is the precisely the response OpenAI is hoping for to raise its valuation, and you fell for it.
Look at it this way - whats the difference between tasking AI to break into something, versus taking a whole bunch of smart humans to do the same? The only difference is that AI is slightly easier to orchestrate.
Prior to AI, there were already a whole bunch of tools to automate exploits. Nothing that the model did is groundbreaking or novel, it was just able to efficiently find the thing that worked. Same thing happens in state sponsored cyber sec agencies like in China or Israel - they train people on the most common exploits and have a whole bunch of tools that automate exploit research and development.
And the reason why this doesn't happen more is because to exploit something is one thing, to do it so there is no trace back to you is a whole different animal that has many more magnitudes of difficulty, which with modern web security is next to impossible in a lot of cases as traffic can easily be traced back to the point of origin.
I.e when a company trains an LLM that manages to build a drone that can fly into a vent and plug in a USB stick into a computer undetected, then we can make the claim that they have a weapon.
On the flip side, most anyone who can run local models can replicate what they did. The key thing to take away from the article is "agentic framework" - i.e this means that they spent a shitload of time developing explicitly coded loops for an LLM to go through. Nothing is really stopping you from doing the same, models like Gemma4 can take 256k tokens of context, so you can give it a whole bunch of info on how to test for exploits, develop exploits, and what to do when the exploit is found, and set it free in a custom designed loop.
I find myself in major disagreement here. The nice thing about humans is we always have context and ongoing internal conversations including about ethics. If you recruit a bunch of hackers to take down a country, not only is the pay an order of magnitude higher, you have to worry about them backstabbing you, leaking your intent to the government, whistleblowing to the press, and so on. It’s not trivial to do that with a group of (especially capable) humans. They will also have differences of opinion with you and coworkers with some regularity.
I try to recruit a bunch of people to attack a country and it’s going to be hard to get people to say yes, and they will definitely ask or find out which country, and wonder about potential retribution. You see this dynamic show up even to some extent among cybergangs, not all targets are equal.
A single private individual wielding a compliant and hyper capable LLM is an entirely different paradigm. They are accountable to nearly no one and often have few brakes. Frequently they may not care about avoiding detection. And the AI itself may be incapable of the same scale of self reflection and brake behavior a human team will.
We may potentially be entering the age of lone wolf cyberterrorism, and some of the same principles and problems apply. When it is easier for single people to plot and carry out high-impact, destructive acts they happen more often. Doubly so if there’s a social contagion. Gun violence isn’t actually a bad analogy here. And do you remember how many corporate sites got defaced in the prime Anon era?
In places like China, its really not that unthinkable to basically raise kids indoctrinated into an ideology and train them in the necessary skills so that you have a cyber army at your command.
Also
>Frequently they may not care about avoiding detection.
This is a big negative. As someone who used to be in the cybersecurity sector (both offense and defense), I wouldn't trust an LLM agent if Im doing red team, because it may leak some info that ties the hack back to me.
ALso keep in mind that most places with good cybersecurity have firewall servers that straight up detect anything that looks like malicious and not regular traffic, and will straight up block IPs, leaving you with no way to even access the server. An agent is bound to statistically use the attacks that are known at some point, increasing the chances of this type of detection.
I'm not sure your assumptions hold. As OpenAI has found out the hard way, if you task the AI to do X, it may do something else instead and hack into huggingface in attempt to cheat out the answer. This is way worse than what a human might do when they have "differences of opinion".
It might turn out that it's harder to align AI intentions compared with aligning human interests. It's possible that the more "intelligent" a thing is, the more likely it will have ideas that are outside of normal expectations (for us).
I strongly agree with you here. People are also no-selling the enormous cost of the exploit, which OpenAI conveniently hasn’t released, or at least I can’t find such an accounting. You can already buy politicians for relatively cheap, corporations already act as sentient AIs pursuing goals misaligned with public benefit (we tried to pass laws to stop this but the corps already stacked the SC in advance and gave us citizens united). Attack and defense are two sides of the same coin, so our focus should be on making frontier models open weight.
It lowers the economic cost of a given attack, but also lowers the economic cost of protection. Not sure if it’ll be a perfect balance, but right now there’s a manufactured IMbalance due to embargos and winner picking.
It lowers the economic cost of performing an attack in the same way a gun lowers the economic cost of killing a person. It does nothing for consequences of that attack.
Are you serious? Before LLM’s you needed serious skills and experience to pull this off.
Now it’s a prompt away on some terminal done by any random dud.
And I dont mention the velocity of iteration or that they will be even better in 1 year.
Its like you don't even read the post or the content.
Hint: agentic loops.
No its not a prompt away.
Also the harness, there have been several HN submissions showing that older/smaller models can find a subset of the exploits given a good env