Comment by minraws
4 days ago
As much as I am with the author in I don't like the marketting around it, let's be real it must have really happened because it's very risky to try to frame/lie about it because if it leaks in one of their court cases OpenAI is beyond screwed and honestly modern LLMs are really that good.
I am not saying LLMs are super hackers but I don't think people understand serious hacking, most of the time is about silently hiding tracks and slowly trying ideas and waiting for opportunities to go from step 1 to step 2 in random chains of sub issues/bugs/vulnerabilities.
It's the perfect hill climbing problem, and one we can validate since it's about access.
Another big part of the story is believing most software is terribly written and very insecure which is the reality and you really should believe it.
Now the second part about silently doing it, the reason for that is if the data is important enough any serious attack should result in me in unplugging my servers period.
Huggingface not doing that is either stupid or something I am not sure. Maybe it's cause downtime is worse than being pwned??
Either way there are other options but most saas software don't build these options to help with defense maybe they will now.
Lastly if there is 1 attacker trying 1/2 different small scale ideas it's very easy to stop, most hacking related steps are hard to automate but LLMs are very good at massively parallel agent swarms trying completely orthogonal but related strategies and with enough resources it can definitely pwn most SaaS services today I wouldn't be surprised.
Though the result for a normal person doing it would be jail hence we don't see a group of small time hackers trying these sort of attacks...
I don't even think openai's agent tried to hide it's traces so I am surprised huggingface didn't realize it was OpenAI. But since we don't have the details I won't speculate further on my misgivings about HFs handling of this attack.
But it's certain the security on OpenAI's end was shoddy, it's also certain HF bungled their reaction, but the LLM did something that wasn't a risk before.
Post Kimi K3 a few rich folks now have as much hacking capabilities as they used to have before if they hired a few hundred russian hackers.
But it's surprising it's slowly feeling like it might just trickle down from centi-millionare to multi-millionare levels of affordability range.
But it should definitely give nightmares to people shipping slop security SaaS apps which now might be beyond trivial to pwn for users with ability to pay for privately hosting open models.
Ya, we're seeing posts talking about the absolutely massive increase in the number of patches in the past few months. It seems some people cannot connect that to the increases in model capabilities.
In groups that have been given a large amount of capacity by the providers, they tend to find huge numbers of new vulns in most of their existing software, and the models can chain together exploits very well.
A number of large companies are absolutely panicked about this now after using these models on their internal systems and the ease at which they broke in. Of course they are not going to discuss this widely as it makes them look bad.
I mean if you gave me direct access to all your source code and fuzzers I could find vulnerabilities as well, these models also are allowed to use fuzzing and static analysis tools to help guide them.
The problem is the sheer scale of it, I could find 1 in a day or two.
They find dozens well depending on how much you are willing to spend ofc. I don't think it's massive in terms of how intelligent these are but how much they can understand intent and execute with relatively fuzzing or incorrectly built tools.
In big companies even most employees don't have access to all the code to be able to figure out the attacks quickly enough.
Somehow they are now willing the red tape since these systems could theoretically with much greater effort(read spend) reverse engineer APIs and break through even without access to the source code.