Comment by flerchin

10 hours ago

The vast majority of CVEs are not exploitable, basically noise. I suspect that the overwhelming majority of the CVEs being generated by LLMs are either noise of the sort in the linked article or noise of the sort that is not exploitable.

This is what I truly don't like about the CVE system.

One such example is CVE-2023-45853 [1]. Zlib included in it's source an extra set of utilities and add-ons. One such utility, MiniZip, had a buffer overflow vulnerability. BAM, 8.8 CVE (was a 9, looks like they pulled it back a bit). But not one that the 99% of applications using zlib would ever be vulnerable to because almost nobody used the MiniZip utility. It was so unused that the solution for zlib was to simply remove it.

I know about this one particularly because our security policy required us to do a BUNCH of pointless updates for it since zlib is in just about everything.

[1] https://app.opencve.io/cve/CVE-2023-45853

  • In the Spack package manager [1] we're trying to work around this with directives:

        deprecated("@:1.3 +minizip", reason="cve", severity="high")
    

    So, you can define conditional deprecation, and the dependency resolver can still pick up `zlib ~minizip` (i.e. with a variant/component disabled).

    Deprecation on versions only isn't the right granularity.

    [1]: https://github.com/spack/spack/pull/52372

  • Also a buffer overflow in user space for a tool not serving internet traffic is not a big deal.

    • I would be careful with this line of thought: opening a malformed archive that gets you into user space is often the first step in a chain-of-attacks that ends up winning Pwn2Own, so I think that a 8-ish score makes sense for it. It won't be enough to do much on its own, but for the past decade or so it's been all about chaining enough small things together to get root, rather than single points of failure.

      I keep feeling like the entire CVE system is creaking and about to fail- the signal-to-noise on that has gone so far down, the gate-keeper function that MITRE et al are supposed to be providing clearly has fallen apart, but we do need some way of tracking problems and alerting on bugs in commonly reused libraries. So I can see the problems with the current system, I just can't envision the new, better system to build to replace it.

      1 reply →

  • Reminds me of the time our “security” team fought with me for a year to fix an issue in a third-party dependency that was failing their scanners. The issue was only a problem for and intentionally left in to not break support for Internet Explorer 6 on Windows XP, neither of which anyone at the company had used in 10 years, and everyone using this software was on a Mac anyway.

    My condolences to anyone who’s got to deal with all these slop-y CVEs on one side and brain dead security teams on the other.

  • IMHO that’s why binaries and libraries should be separate packages. Drives me buts with npm packages.

    • Thats not enough. This problem pops up everywhere people do CVE analysis. If a tertiary part of some library is hit by some arbitrary CVE, why should that affect the core functionality people actually use? If I use ChaCha20, and that's bundled in a library with md5.do I care that their md5 library has a vulnerability? No, not really.

      If you keep driving the "seperate the packages" angle you end up with is-odd, and that not a nice play to be either.

      5 replies →

    • They should be in separate packages so you won't force yourself to patch CVEs in the parts you don't use?

  • But could an attacker with access to zip tools exploit it to get a root shell? Probably not, but maybe.

    • Well that's the thing, to build the utilities required extra flags which were pretty rarely enabled. You can, in most distros, ultimately install minizip as a separate package but few do. It's not that useful of a utility. Most people will likely just grab the full blown "zip" application https://infozip.sourceforge.net/Zip.html . Though on linux, even more people are simply using tar with a zlib extension. That's the `tar.gz` files.

      2 replies →

    • You do not run it as root, all you get is the possibility to crash or execute code. Nodes that handle such things are all heavily locked down.

      2 replies →

  • Isn't the real difficulty in how vulnerabilities can be mixed? A given vulnerability might be extremely hard to execute, but very damaging. Another vulnerability might be easier to execute, but it can't do much. But if the second one can be used to trigger the first one, you then have attack surface area of the second vulnerability with the damage of the first one.

    Even if some individual case can be shown to be safe from being combined, can we identify such cases with enough confidence to justify using it reduce severity warnings?

    • Yea, this is where the problems start coming in.

      In testing with LLMs with good exploit finding capabilities and a lack of guardrails on writing exploits, quite often LLMs will chain together a surprising stack of exploits to get what they want. They'll get access via a weak, but limited user. They'll search around laterally until they find something else. For example User A (weak exploit) -> User B -> User B bad configuration -> root (or application takeover).

      They don't get bored like humans and they can be more robust than a set of scripts by far in catching errors when their scripts don't work or interesting things show up in their findings. The huggingface break down is a good example of just how much lateral movement an LLM can try with enough processing power behind it.

      This is where the security onion gets messy. It can be hard to predict when compromising one layer actually bypasses many other layers of your security.

Yes, the CVE system is broken. LLM-generated CVEs is making it even worse.

I suppose all these fake issues and the many more that have absurdly elevated severities could be considered an attack on the system itself, stripping it of credibility.

  • That's the worst case scenario right there: so many fake reports that the real vulnerabilities can no longer be reported properly and fixed quickly. Assuming some LLMs really are capable of finding serious vulnerabilities (which in my experience is absolutely the case : just let a good model run on your security code base and see for yourself) that means that we can more easily find vulnerabilities, but it's much harder to report them and be taken seriously - so the incentive to exploit such vulnerabilities instead has become much, much higher.

    • Having your LLM look at the security really isn't a great option. Most users don't have unlimited budgets and can't dump the necessary time and funds needed to find these flaws.

      On top of that, ability to prompt about flaws makes all the difference in the world. If you know what you're doing you can get better results. Problem, very few people know what they are doing.

      Going back to the money/time problem, LLMs are not deterministic. Much like the day you forgot to drink coffee, running LLM systems against code can get different results where things are found or not found. That or two different things are found on two different runs (with the same code) because attention was stolen by the first issue it found.

      So yea, in the LLM age proof is in the exploit pudding. Who cares if you're taken seriously, if you dump out an exploit that works the seriousness is now on the defenders time and people will scramble to fix it.

      "But what about exploit embargos"... Well, you just found the exploit with an LLM, how many other nefarious groups do you think are already exploiting it? The age of sitting on problems is dead.

CVE-2026-4I5109 - Your kitchen drawer contains forks, which can maim and/or kill. An attacker with kitchen access could access the drawer. See...

CVE-1972-404: The code is written in C. ^_^

The majority these days seems to be real issues, but probably not exploitable. The latest LLMs are very good at reading code and finding issues where the code is wrong. They are somewhat less good at finding exploits - both because finding exploits is harder than finding code that is clearly wrong (even if not exploitable), and also because the better ones are intentionally made to not create exploits (since they don't know if you are a good or bad - they assume bad.

However if I was writing this response just one year ago I would instead be saying: the majority off LLM CVS are noise where the code is correct, and often they are writing up for code that doesn't even exist.

Which is to say I suspect the repo in question was generated with a year-old LLM, since they act like that. The new ones [mostly?] are much better.

Still, if a modern LLM points out something you should fix it. Even if we can't figure out how to exploit it today that doesn't mean we won't figure it out in the future.

  • > I suspect the repo in question was generated with a year-old LLM

    I suspect there are a lot of people running inexpensive models that are searching for vulnerabilities across a lot of projects, probably in an automated way (ex with openclaw or similar) in the hopes of winning a bounty and/or noteriety.

    • Most bounty programs are aware of this. Many projects have closed their bounty program in response to that type of attack.

> The vast majority of CVEs are not exploitable, basically noise.

This is not true if you consider security-in-depth. Many of them are exploitable on their own but maybe not in combination with other issues that are as yet unknown or known but not patched everywhere.

As a simple example a local privilege escalation issue that is not exploitable on a device that only I ever have access to, essentially becomes a remote root access flaw if you have untrusted or unreliable users (clients with accounts for instance) on the system. This works on a finer grain too, seemingly minor issues spread through the kernel and user space can add up to a serious exploit.

  • This is the most common "not exploitable" CVE I have to deal with at work all the time. Things like buffer overruns when passing headers, but my server shouldn't be accessible to the public Internet anyway, and if it is, then we have much bigger problems. So yes it's technically exploitable IF other safeguards fail, but the reason it's irrelevant is that if those other safeguards fail, we are in deep shit anyway.

    • It's been proven over and over and over and over and over and over and over and over and over and over and over ......

      that requiring access to a privileged network is NOT a sufficient security boundary! Someone you don't like always somehow ends up getting access to the network.

    • But if you have proper risk-based vulnerability management, you should already have classified that CVE as a lesser priority. This is something that modern LLMs should in theory make a lot easier, although I’m unsure if workflows exist to continuously build a proper architectural understanding of your systems, at least at a feasible cost.

    • This sounds like a piss poor implementation of zero trust on your part.

      Security is an onion, you have lots of layers because layers peel off and fail all the time. With your current setup a single foothold gives an attacker all the lateral movement they need for full exploitation of every single thing you own blowing past the other layers.

    • Or you can just fix the issue that way if IT screws up you are not vulnerable. Or maybe next year you will decide it is useful to connect to the internet.

I once had a customer threaten to reject a delivery over a CVE. I dug into it. The CVE was that an Ocaml PostgreSQL client didn’t implement an authentication feature.

I convinced the customer to accept the delivery by pointing out that (1) our app had zero lines of ocaml and (2) the feature had been implemented in the ocaml driver since the CVE was issued.

>The vast majority of CVEs are not exploitable

Perhaps in isolation. The issue is when you can chain exploits to bypass multiple layers.