Comment by nickpsecurity
5 days ago
If anything, it shows they lost control of an attack tool that exploited preventable, security flaws in another company. Then, they both wrote a lot of press about how amazing that is. Now, people want to buy it.
Why do you think my head is in the sand if I think that is either a marketing stunt or (more likely) reflects total negligence which was exploited for marketing?
I think your head is in the sand if you prefer to believe that this was a hoax for marketing purposes as opposed to accepting how effective these models have become at exploit research.
You're welcome to think they are exploring what happened for marketing if you like. I didn't get that tone from their post about it myself but I don't hold a particularly strong opinion on that.
I didn't say it's a hoax. AI's exploiting common, unpatched, or preventable vulnerabilities just isn't far above what script kiddies and network scanners have done for decades. I literally pitched it as a business model to some people before Windows XP was invented. Ultimately, I decided against what I felt was unethical but would've scored easy money with scary, automated reports.
How these events are described in most articles uses wording that makes people feel the whole situation has changed and you might want to buy these products due to their scary reports. If they said what I said, or what tptacek said, many people wouldn't care about it much more than non-AI, security products or pentesting services they've been buying (or ignoring) for 10-20 years.
Also, you shouldn't interpret posts in isolation: we must consider patterns of behavior (character). They've consistently overhyped what AI's do and what value it provides to businesses and how we're all going to be unemployed/dead. They've done this to increase sales or market value pre-IPO. Then, some of them publish another set of articles promoting a specific, AI tool in a similar way.
So, the proper interpretation is to see this as the kind of talk they're always doing for marketing. The AI sellers are creatures of habit. We should highly-skeptically and scientifically evaluate every model. We should also compare them to existing, security practices. For instance, would the attack have happened with memory-safe systems, proper hardening, and network/web apps with built-in security?
Do we need an AI? Or should we use techniques like Burroughs B5000's memory safety (1961) or secure, distributed libraries? Will OpenAI and HuggingFace tell you to spend more money on the latter to their AI's can't hack your systems without inventing RAM-based attacks or something? Probably not because these are marketing pieces, not security advice.
Exploiting, not exploring.
We don't know the details of the exploit still, so saying it was preventable, and that they were negligent is the head in the sand part. It's entirely possible that everyone at OpenAI is out to lunch and negligent and don't know what they're doing, but maybe, just maybe, they're not all total idiots over there, so the exploit was actually surprising.
Two, business partners put much press into announcing one of their products does something game-changing but won't share details. Take their word for it. Keep writing checks to both. Get ready to write bigger checks, too.
Shouldn't we be skeptical of this? Or should I also believe the sugar industry when it says their internal studies show their products don't cause obesity or diabetes?