One can hope that will put pressure on the industry to design a better system than CVEs. The signal noise ratio was already terrible before LLMs, I cannot imagine that will still be a meaningful system in 10y.
But I’m too cynical to not consider all the middlemen who benefit from the status quo
It's going to be very very difficult to build this system as the exploitability of any particular CVE can massively vary depending on your system configuration.
There are a lot of things that are bugs, but cannot be exploited in a standard configuration, most people would wonder why this is even a CVE.
But then you have those users that would have the application, then something like a report module that was imported, and another 3rd party module that imports images, and maybe another one that arranges the images in a chart. And suddenly that isolated CVE is now exploitable by user written data because of the original bug, it just required a complex and unexpected chain to get there.
> There are a lot of things that are bugs, but cannot be exploited in a standard configuration
And this describes the vast majority of CVEs you see when you scan your dependencies. Typical case: Your regex library has a denial of service vulnerability for crafted regular expressions but your program never allows users to have any influence over the regex.
I'm very curious what organisations would have such a policy. I can't imagine it being viable for any size of org without significant self-deception (or banning the use of all open source at which point CVEs are moot anyway).
> I'm very curious what organisations would have such a policy.
I would humbly suggest any org of any size that has insurance cover that covers anything tech related (e.g. data loss/recovery, cyber etc.) has a very good look at the small print.
Over the last few years insurers have aggressively been adding "no vulnerability patch, no claim" exclusion clauses.
Yeah, policies like this are often not coming from engineering directly but often through other parts of the company like legal, or even sales from contract negotiations. Not that it's entirely comparable, but I was at AWS when the big log4j vulnerability happened, and the handling for it was not left up to individual engineering teams, which I don't think would surprise anyone.
At a large enough company, processes for handling things like security vulnerabilities will have a lot of stakeholders with incentives that are not necessarily perfectly aligned.
Many orgs (esp w ISO27000) have a vulnerability management policy that involves patching at least critical CVEs within a short timeline. Tools like trivvy make it possible to do the scans…
I've been in such an org, & I've led initiatives to set up automated detection at very large scale. We started by issuing tickets to teams to resolve CVEs within varying timelines - ranging from a 24hr fix to 6 months - connected to the CVSS score. It wasn't viable.
- Firstly, you quickly realise how irrelevant CVSS scores are - initiatives like First's EPSS are designed to fix this but they aren't there yet
- Secondly, you need to begin implementing localised heuristics to determine exploitable code paths. This has generally been incredibly difficult to do reliably - LLMs have started to make it easier, but it's expensive.
- Lastly, you need to factor in consideration of actionable remediation pathways. A dependency upgrade for critical infrastructure might contain breaking changes that take months to fix, or two competing CVEs might be present in interdependent versions of transitive dependencies in your sbom tree.
Most orgs aren't applying any of the above three filters to reduce their CVE remediation burden, & even if they are, it's still too high to make zero a viable target.
In reality, most orgs aren't doing comprehensive detection to begin with - if you haven't discovered all of your CVEs, your remediation burden is going to be a lot more manageable.
I have been given a list by security. "We had an automated tool scan that machine. It reported these. Fix anything medium severity and above.
Never mind that some of them involved vulnerabilities in some part of the bluetooth stack (servers in our datacenter don't even have bluetooth). But they just didn't care
This does make some sense if it's considered a valid fix to document that you have verified that Bluetooth is disabled on the servers and therefore not vulnerable. But that assumes that the scanning tool can be told about this kind of fix, so that it stops warning about it, which I guess it might not.
So run apt full-upgrade and get the new bluetooth driver. Why bother with a fight over something that isn't even used? Just do the quickest thing to get it off your plate.
SOC2 CC7.1 [1] requires a vulnerability scanner, findings tracked with tickets, assigned severities according to a documented risk-based system, severity-based SLAs for remediation, and that the SLAs mostly be complied with or have tracked exceptions.
However it doesn't mandate any particular SLA, or the details of how risks are to be evaluated.
Organisations get to write their own policy, and they don't need to commit to patching every CVE within 24 hours or anything like that.
Any org large enough to have separated the people responsible for the security exposure of the organisation from the developers with familiarity of what's deployed is likely to have done exactly this.
The thing you have to remember is that CVEs can be a) scanned for without exerting mental effort, and b) counted.
For one thing, bigcorps in regulated areas like it a lot. They push hard to get it required by the regulations (in practice if not directly). Although it's quite inefficient, it becomes a regulatory moat. A cost they can bear that potential upstart competitors cannot.
Many large organizations like banks have requirements like this and they solve it through a mix of automatic scanners, e.g. Trivvy, and self-deception as not all systems are actually scanned in any sufficiently large org.
> I'm very curious what organisations would have such a policy
Do you provide SOC2, HIPAA, GDPR, or similar certifications to your b2b customers? Then your tech stack undergoes an annual audit, and in your audit you will need to provide a paper trail for every single vulnerability in your stack.
In practice, this means that your audit compliance software (something like Vanta.com) is going to be setup to mandate every CVE in the whole stack is patched within SLA.
ITAR has no such hard requirements. Might be some orgs that tell themselves they're attempting this under ITAR but they're not doing it in any comprehensive way.
The only thing within ITAR that I'm aware of concerning itself with software supply chain is SP 800-218 requirements & that's just a load of open-to-interpretation weasel words about having CVE detection & automations in place & some defined plans for reducing the number of vulns. Pretty sure that component of it is even eligible for self-assessment.
If by "terrible" you mean "amazing". Imagine being able to tear down an entire Jenga tower of bullshit corpospeak, and replace it with actual security.
It’s honestly not great. The security guys are completely exasperated at my job, we’re wasting time having with these. You take the scam really, investigate for a bit, write up a DNF with justification, they go and up date records, and we all just kinda hope that someone updates the scans so it stops showing up.
Something is going to give, and I suspect that the optimistic open filling is going to get canceled.
One can hope that will put pressure on the industry to design a better system than CVEs. The signal noise ratio was already terrible before LLMs, I cannot imagine that will still be a meaningful system in 10y.
But I’m too cynical to not consider all the middlemen who benefit from the status quo
It's going to be very very difficult to build this system as the exploitability of any particular CVE can massively vary depending on your system configuration.
There are a lot of things that are bugs, but cannot be exploited in a standard configuration, most people would wonder why this is even a CVE.
But then you have those users that would have the application, then something like a report module that was imported, and another 3rd party module that imports images, and maybe another one that arranges the images in a chart. And suddenly that isolated CVE is now exploitable by user written data because of the original bug, it just required a complex and unexpected chain to get there.
> There are a lot of things that are bugs, but cannot be exploited in a standard configuration
And this describes the vast majority of CVEs you see when you scan your dependencies. Typical case: Your regex library has a denial of service vulnerability for crafted regular expressions but your program never allows users to have any influence over the regex.
I'm very curious what organisations would have such a policy. I can't imagine it being viable for any size of org without significant self-deception (or banning the use of all open source at which point CVEs are moot anyway).
> I'm very curious what organisations would have such a policy.
I would humbly suggest any org of any size that has insurance cover that covers anything tech related (e.g. data loss/recovery, cyber etc.) has a very good look at the small print.
Over the last few years insurers have aggressively been adding "no vulnerability patch, no claim" exclusion clauses.
Yeah, policies like this are often not coming from engineering directly but often through other parts of the company like legal, or even sales from contract negotiations. Not that it's entirely comparable, but I was at AWS when the big log4j vulnerability happened, and the handling for it was not left up to individual engineering teams, which I don't think would surprise anyone.
At a large enough company, processes for handling things like security vulnerabilities will have a lot of stakeholders with incentives that are not necessarily perfectly aligned.
Many orgs (esp w ISO27000) have a vulnerability management policy that involves patching at least critical CVEs within a short timeline. Tools like trivvy make it possible to do the scans…
I've been in such an org, & I've led initiatives to set up automated detection at very large scale. We started by issuing tickets to teams to resolve CVEs within varying timelines - ranging from a 24hr fix to 6 months - connected to the CVSS score. It wasn't viable.
- Firstly, you quickly realise how irrelevant CVSS scores are - initiatives like First's EPSS are designed to fix this but they aren't there yet
- Secondly, you need to begin implementing localised heuristics to determine exploitable code paths. This has generally been incredibly difficult to do reliably - LLMs have started to make it easier, but it's expensive.
- Lastly, you need to factor in consideration of actionable remediation pathways. A dependency upgrade for critical infrastructure might contain breaking changes that take months to fix, or two competing CVEs might be present in interdependent versions of transitive dependencies in your sbom tree.
Most orgs aren't applying any of the above three filters to reduce their CVE remediation burden, & even if they are, it's still too high to make zero a viable target.
In reality, most orgs aren't doing comprehensive detection to begin with - if you haven't discovered all of your CVEs, your remediation burden is going to be a lot more manageable.
5 replies →
> Tools like trivvy make it possible to do the scans...
Only if you didn't rip trivvy out of your organisation when it had two supply chain compromises within a month of each other earlier this year
3 replies →
I have been given a list by security. "We had an automated tool scan that machine. It reported these. Fix anything medium severity and above.
Never mind that some of them involved vulnerabilities in some part of the bluetooth stack (servers in our datacenter don't even have bluetooth). But they just didn't care
This does make some sense if it's considered a valid fix to document that you have verified that Bluetooth is disabled on the servers and therefore not vulnerable. But that assumes that the scanning tool can be told about this kind of fix, so that it stops warning about it, which I guess it might not.
So run apt full-upgrade and get the new bluetooth driver. Why bother with a fight over something that isn't even used? Just do the quickest thing to get it off your plate.
2 replies →
SOC2 CC7.1 [1] requires a vulnerability scanner, findings tracked with tickets, assigned severities according to a documented risk-based system, severity-based SLAs for remediation, and that the SLAs mostly be complied with or have tracked exceptions.
However it doesn't mandate any particular SLA, or the details of how risks are to be evaluated.
Organisations get to write their own policy, and they don't need to commit to patching every CVE within 24 hours or anything like that.
[1] https://www.compliancebase.org/controls/soc-2/cc7-1
If I remember correctly, we had to patch or provide justification for CVEs flagged by tools like AWS Inspector for SOC2 as well.
So you didn't have to patch all of them.
1 reply →
Any org large enough to have separated the people responsible for the security exposure of the organisation from the developers with familiarity of what's deployed is likely to have done exactly this.
The thing you have to remember is that CVEs can be a) scanned for without exerting mental effort, and b) counted.
It's quite common in enterprisey environments.
For one thing, bigcorps in regulated areas like it a lot. They push hard to get it required by the regulations (in practice if not directly). Although it's quite inefficient, it becomes a regulatory moat. A cost they can bear that potential upstart competitors cannot.
It's more common than many think https://old.reddit.com/r/kubernetes/comments/1vb3x2c/where_a...
Many large organizations like banks have requirements like this and they solve it through a mix of automatic scanners, e.g. Trivvy, and self-deception as not all systems are actually scanned in any sufficiently large org.
> I'm very curious what organisations would have such a policy
Do you provide SOC2, HIPAA, GDPR, or similar certifications to your b2b customers? Then your tech stack undergoes an annual audit, and in your audit you will need to provide a paper trail for every single vulnerability in your stack.
In practice, this means that your audit compliance software (something like Vanta.com) is going to be setup to mandate every CVE in the whole stack is patched within SLA.
ITAR
ITAR has no such hard requirements. Might be some orgs that tell themselves they're attempting this under ITAR but they're not doing it in any comprehensive way.
The only thing within ITAR that I'm aware of concerning itself with software supply chain is SP 800-218 requirements & that's just a load of open-to-interpretation weasel words about having CVE detection & automations in place & some defined plans for reducing the number of vulns. Pretty sure that component of it is even eligible for self-assessment.
2 replies →
[flagged]
This was my first thought, this could be terrible if used offensively.
The best defense I can imagine is to have an agent reproduce the issues before a human sees it, but even that will cost money.
If by "terrible" you mean "amazing". Imagine being able to tear down an entire Jenga tower of bullshit corpospeak, and replace it with actual security.
It’s honestly not great. The security guys are completely exasperated at my job, we’re wasting time having with these. You take the scam really, investigate for a bit, write up a DNF with justification, they go and up date records, and we all just kinda hope that someone updates the scans so it stops showing up.
Something is going to give, and I suspect that the optimistic open filling is going to get canceled.
Those organizations will have to adapt to new reality, ie, that some CVEs are not real.
Nothing new about that
All organisations also have exceptions to policies. This one would be one
Yes, but getting an exception approved can be a huge pain in some places.
If you work with the government, you don’t really have this luxury
You'd be surprised...
Not really. LLMs can hallucinate the patches too :-)
Steps to patch a hallucinated CVE:
Create the referenced but non-existent file and then fix it /s