Comment by aftbit
17 hours ago
The coordinated disclosure thing will have to stop soon. Ideally, if you found a vuln from the defensive side, you'd tell all the "good guys" about it and none of the bad guys. The world doesn't work that way, so instead, the compromise was to tell a small number of important big actors who were unlikely to leak it, so the most good could be done before everyone found out.
Now the simple knowledge that there might be an exploitable memory issue in some given piece of software is enough for bad actors to point LLMs at it and find the issue in hours. That will only keep getting worse.
At some point, it's going to become a speed game. If you find a vuln, you might as well disclose to everyone immediately. Anyone who runs internet-exposed software will need an agent watching for disclosures around the clock and triaging them automatically (either by upgrading, improving filtering with some protective shell, or taking the whole service offline until humans can help) to keep up with the offensive agents watching around the clock and attacking automatically.
I'm wondering how long till we see the first exploit that goes like this:
* A bad actor finds a real, theoretically severe but not easy-to-actually-exploit vulnerability in some popular software. E.g. it's an exploit that can get you pwned 100% with RCE, but only if you are using it in configuration X on platform Y with kernel version Z on a Tuesday.
* They design a patch that closes that vulnerability but subtly opens a more dangerous one known only to them. They privately test their patch against LLM reviewers, tuning the approach until the introduced vulnerability typically escapes detection by automated review.
* They "responsibly disclose" the original exploit and helpfully share the patch. In the hurry to fix the original exploit, the patch itself gets only cursory (claudy, codexy) review. By now this has happened so many times that just applying whatever code changes ClodSed were GTG is the status quo.
* When the new vulnerability is eventually discovered, they can plausibly say "oh my agent didn't catch that", which is true. In the meantime, they use the new exploit that only they know about.
One can defend by quarantining any submitted fix, developing your own without reading it. Then only after that compare them.
In any case you don't even need malice --- sincere reporter can vibe up a fix that accidentally introduced a vulnerability, owing to their own lack of domain expertise and model errors... and then this bad fix makes it in because in the thrall of confirmation bias you fail to notice its shortcomings. Reporter also presumably tested it against their own LLM, so that will pass too.
This happened recently in a cryptocurrency project and contributed to a significant exploit.
This was already a thing before AI, the term is "variant analysis". But it's been way juiced up by LLMs and is 100% a major cause for concern.
> the patch itself gets only cursory (claudy, codexy) review
FWIW Claude is actually quite good at discovering variants of existing bugs.
> Now the simple knowledge that there might be an exploitable memory issue in some given piece of software is enough for bad actors to point LLMs at it and find the issue in hours.
It is all but certain there are exploitable memory issues in Chrome, Firefox, Node.js, Linux kernel, Nginx, etc.
Can we expect CVEs in a few hours? I'm sorry I ruined everyone's weekend.
You may joke, but I know someone who told Kimi to "go wild and fuzz filesystems in the Linux kernel", woke up the next morning to "26 bugs found and fixes prepared".
I understand several have been accepted since, and he stopped the fuzzers because his backlog of submissions to make was getting out of hand.
> If you find a vuln, you might as well disclose to everyone immediately.
This still doesn't work, since attackers are much more motivated and able to pay attention to such things.
If your goal is finding vulnerabilities / stealing money / whatever, you focus all your energy looking for new exploits.
The rest of the world has other things to do (listen to users, run services, whatever) and by definition cannot spend as much of their time watching every social media in the world for the chance of an announcement.
That's why anyone who has a lot to lose from a data breach had better either harden the shell of their system such that it's not vulnerable (very hard to do correctly) or automate the process of listening for and reacting to vulns.
> or automate the process of listening for and reacting to vulns
1. This is also very hard to do correctly.
2. What automated steps would you take without any patches available to apply?
3. Potentially makes you more vulnerable to supply chain attacks