Comment by thaumaturgy
20 hours ago
There are two aspects: first that there is currently a much higher pace of critical vulnerabilities and exploits being discovered, and second that there is a much shorter window between discovery and large-scale exploitation.
People are saying the pace will settle down once LLMs find and fix all the bugs. Okay, sure, if you want to believe that sort of thing, but that still leaves the much shorter time window, and that's not going to go away anytime soon.
If you're on the side of the industry where your job or hobby involves breaking things, this is all very exciting. But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks.
I lost a full day yesterday because some big players discovered a thing, told other big players about it, "oopsed" enough public information to let everyone else know about it, and then left everyone else to figure out what to do with almost no information to act upon.
This cycle has happened several times this year now. If you're on the short list of big hosting providers, you get advance notice of what's coming and early fixes; if you're not, sucks to be you, hope you didn't have any plans today.
For people like me, this is just yet one more way LLMs are making my life materially worse.
> But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks
Is it? In using it at an opportunity to improve our devops pipelines, software supply chains, etc. Perhaps I'm weird and enjoy strange things? Not to mention that all these security incidents will require plenty of people to fix things = more paid work for us humans going forward.
The coordinated disclosure thing will have to stop soon. Ideally, if you found a vuln from the defensive side, you'd tell all the "good guys" about it and none of the bad guys. The world doesn't work that way, so instead, the compromise was to tell a small number of important big actors who were unlikely to leak it, so the most good could be done before everyone found out.
Now the simple knowledge that there might be an exploitable memory issue in some given piece of software is enough for bad actors to point LLMs at it and find the issue in hours. That will only keep getting worse.
At some point, it's going to become a speed game. If you find a vuln, you might as well disclose to everyone immediately. Anyone who runs internet-exposed software will need an agent watching for disclosures around the clock and triaging them automatically (either by upgrading, improving filtering with some protective shell, or taking the whole service offline until humans can help) to keep up with the offensive agents watching around the clock and attacking automatically.
I'm wondering how long till we see the first exploit that goes like this:
* A bad actor finds a real, theoretically severe but not easy-to-actually-exploit vulnerability in some popular software. E.g. it's an exploit that can get you pwned 100% with RCE, but only if you are using it in configuration X on platform Y with kernel version Z on a Tuesday.
* They design a patch that closes that vulnerability but subtly opens a more dangerous one known only to them. They privately test their patch against LLM reviewers, tuning the approach until the introduced vulnerability typically escapes detection by automated review.
* They "responsibly disclose" the original exploit and helpfully share the patch. In the hurry to fix the original exploit, the patch itself gets only cursory (claudy, codexy) review. By now this has happened so many times that just applying whatever code changes ClodSed were GTG is the status quo.
* When the new vulnerability is eventually discovered, they can plausibly say "oh my agent didn't catch that", which is true. In the meantime, they use the new exploit that only they know about.
One can defend by quarantining any submitted fix, developing your own without reading it. Then only after that compare them.
In any case you don't even need malice --- sincere reporter can vibe up a fix that accidentally introduced a vulnerability, owing to their own lack of domain expertise and model errors... and then this bad fix makes it in because in the thrall of confirmation bias you fail to notice its shortcomings. Reporter also presumably tested it against their own LLM, so that will pass too.
This happened recently in a cryptocurrency project and contributed to a significant exploit.
This was already a thing before AI, the term is "variant analysis". But it's been way juiced up by LLMs and is 100% a major cause for concern.
> the patch itself gets only cursory (claudy, codexy) review
FWIW Claude is actually quite good at discovering variants of existing bugs.
> Now the simple knowledge that there might be an exploitable memory issue in some given piece of software is enough for bad actors to point LLMs at it and find the issue in hours.
It is all but certain there are exploitable memory issues in Chrome, Firefox, Node.js, Linux kernel, Nginx, etc.
Can we expect CVEs in a few hours? I'm sorry I ruined everyone's weekend.
You may joke, but I know someone who told Kimi to "go wild and fuzz filesystems in the Linux kernel", woke up the next morning to "26 bugs found and fixes prepared".
I understand several have been accepted since, and he stopped the fuzzers because his backlog of submissions to make was getting out of hand.
> If you find a vuln, you might as well disclose to everyone immediately.
This still doesn't work, since attackers are much more motivated and able to pay attention to such things.
If your goal is finding vulnerabilities / stealing money / whatever, you focus all your energy looking for new exploits.
The rest of the world has other things to do (listen to users, run services, whatever) and by definition cannot spend as much of their time watching every social media in the world for the chance of an announcement.
That's why anyone who has a lot to lose from a data breach had better either harden the shell of their system such that it's not vulnerable (very hard to do correctly) or automate the process of listening for and reacting to vulns.
1 reply →
> People are saying the pace will settle down once LLMs find and fix all the bugs
Yeah I don't buy it either. Look at github right now, it is a total slopfest with people vibing projects they don't know the first thing about. Plenty of LLM use will be delivering code full of new exploits. It will be interesting to see the frequency of these exploits being found correlated to LLM friendly/unfriendly contribution policies once the dust has settled to tell for certain though.
I have the opposite view: these vibe-coded projects are MUCH higher security than even top-talent human.
Ask a top agent today to create a to-do app and watch it deploy enterprise grade security: anti CSRF, XSS, CSP, secure cookies, HMACs, rate-limiting of logins, and all kinds of things you never heard of.
I've asked it to implement a simple OAuth and saw it classify inbound IPs for private/public subnets, and put in workarounds for very theoretical DNS re-bindings attacks.
We all know LLMs love to over-engineer stuff, they are having a field day turning your static 3 page blog into crypto-exchange grade security.
That may be true, but we're talking about core library vulnerabilities that major projects use, not one-off's that nobody else cares about.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
I wonder if we start to hit a point where the pace that exploits are found and deployed increases faster than they get repaired, because there is a natural higher risk appetite for the bad actors, whereas the good actors still require careful thought and prudence. As a result of this, would the internet/digital economy largely seize up and crash? That would be bad, especially for a lot of people on this site. Maybe this leads to a regression in convenience, but requires more in-person interaction in order to function in society. You can already see this pattern today emerging in higher education. Maybe, in a deeper way, this improves humanity?
> People are saying the pace will settle down once LLMs find and fix all the bugs.
Only for projects that are in a maintenance state. New features will produce new bugs, unless everyone pays for the tokens to scan every release with the top-of-the-line agents or otherwise spends more time than many do ATM to ensure that changes and new code don't introduce significant exploit paths. Things will slow down once the low-hanging fruit from the past is mostly detected and pulled, but the level of vigilance will always have to be at least a little higher than it has been until recently.
Time between updates releasing and people installing them has been an increasing problem for some time, we are just going to have to be a lot more disciplined about that. This will be a problem for places that have strict availability requirements as they might have to compromise on the testing time they spend on new patches before updating their installations, and we might have to move to having more flexible maintenance windows (the old “an hour or two once a month after Patch Tuesday” hasn't cut it for some time and certainly won't going forward) to allow faster updates.
The bigger problem, compared to the patch-availability-to-general-use time which can be reduced somewhat via automation (though that might accelerate bad-patch based attacks…), is greater constraints for the time between bug discovery and patch release: as soon as there is a rumour of an issue many will start scanning for it (as that is easier now) so the time to wide exploitation is going to be much shorter. This might lead to buggy patches that add new issues of their own.
Perhaps we'll see a lot more projects having explicit LTS releases that only get security updates or signifiant bug fixes, which will reduce the attack surface area (and fault surface area) most users, letting those with less stability concerns deal with the bleeding edge.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
I'm with you there…
> New features will produce new bugs, unless everyone pays for the tokens to scan every release with the top-of-the-line agents
Insufficient. Models available to you are already cyber-gimped today and seems likely that they will be moreso in the future (see the recent K3.1 announcement for example). There will be attackers that lack such handicap.
As an industry, we also know two more aspects
1. many have underinvested in security for a long time
2. many are checking out of the development process
I generally think we can improve software with Ai tools, but I'm skeptical the current incentives will lead to this.
> But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks.
It's called job security :)
That’s job security in the same way that a building maintenance guy gets job security from somebody filling the toilets with garbage everyday and flooding the bathrooms
Like sure it’s true that you’re more necessary now, but when you took the job you weren’t expecting to need to be elbow deep in shit every day, and you’d probably prefer to go back to the days of changing lightbulbs and vacuuming hallways
I've been on software projects where more time was spent shoveling shit like this than doing real work. It's only going to get worse.
The web as we know it was never securable. LLMs made it obvious. We made a wrong turn somewhere back around JavaScript. Eventually we'll get fed up with the quick fixes and design something better. I don't blame the LLMs for this. I blame whoever made that wrong turn back in the 90's (and us, for not turning around sooner).
You're blaming JavaScript? The language that succeeded at delivering fully sandboxed rich client-side applications better than any other tech? With the worst incidents generally being 'just' various forms of fingerprinting?
Or, if you're thinking server-side JavaScript: would the web have been more secure running PHP and Python?
The language that made it normal to blindly run code that you found lying around on some page? Elsewhere we'd call that a remote code execution exploit. JavaScript just calls it another day at the office.
And it's not fully sandboxed. It betrays its user all the time by abusing unrestricted access to the network, or activating browser functionality that most of us would disable if we could.
I'm not saying it's THE bad decision, but it is the product of a generation of thinkers who were not thinking about threats the way we do now. If we were to go back in time and have the opportunity to do it differently, I think we would take that opportunity.
But we don't actually need a time machine. We can just accept some sunk costs and move forward anyhow.
9 replies →
Perhaps the web was more secure when it wasn't running anything—just serving up static web pages?
8 replies →
I have to wonder, what is the real substantive number difference in exploits. Its in the hype train's interest to pump the CVEs in both quantity and severity so I do wonder how much of that is happening.
The time to exploit is also somewhat hypothetical. We're assuming infinite attacker resources which hasn't really been the assumption.
We see a lot more news and fear but are we seeing more actual "large scale exploitation"?
I'd love to see some numbers in that regard.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
AI skynet is killing humans. We know that already though.
You were the one hosting an exploitable application.
You say this like it's their fault. The only way not to be sure you won't end up hosting an application vulnerable to a 0-day is not to host anything at all.
> first that there is currently a much higher pace of critical vulnerabilities and exploits being discovered
The slowest that will ever be.
> second that there is a much shorter window between discovery and large-scale exploitation.
The longest that will ever be.
Aren’t exponentials beautiful?
That would also be true if change was linear, or logarithmic.