Comment by bluGill
4 days ago
OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed.
If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then severity may come into place - all 6 are rated low impact)
I think you might be misunderstanding this? This is, from my understanding, what went down:
1. curl was scanned by many different things, including AISLE, and many bugs were fixed <- all this was in the past 2. curl a week ago was scanned again my Mythos and Codex Security, and both of them said: 0 issues found 3. the same curl was scanned by AISLE a day later, resulting in ~29 reports (based on the blog post and mastodon posts from Daniel Stenberg) 4. of these 29, 6 cleared the bar and got CVEs in curl 5. these 6 CVEs were just announced as fixed in curl 8.22.0 today, together with 4 more CVEs that were detected by other people prior to point 2. of this list
so imho it was head-to-head, the very same codebase => it's a legit comparison
Curl has been scanned by mythos and several other AI tools several times over the past year already. Mythos found nothing this round, but when it was first released it found issues which have been fixed - and several other AIs had already scanned curl for issues which had been fixed by then.
We can say that this is a useful tool, but is it better or worse than the others - there is no way to make that conclusion.
Edit, wait, are you claiming that asile was already used and those issues fixed, and it still found more. That is valid, but it only says that asile is better by enough that is worth an upgrade, while the others probably are not. It is not valid in comparing to other tools. (assuming this is true, I've never heard of asile before this)
I don't understand your objection. If tool A says "job is done" and tool B says "found N additional tasks that need doing" how does the history matter? B is turning up things that A isn't thus B is performing better. They both had access to the same inputs here.
I suppose it's interesting to wonder if B would have turned up issue X which A previously found. But that seems largely academic to me. There is a code base right now with N known issues (thanks to B) and A is saying everything is good. It seems like that's all that should matter here.
1 reply →
"How many total vulnerabilities can your tool alone identify?" and "How many unique vulnerabilities can your tool identify?" are both valid comparisons to make IMO.
But that isn't what happened! Mythos has found issues in the past - which are now fixed (or so we should assume, I didn't verify but curl is very good about fixing issues). At most we can say the current version of mythos isn't better than the last version (a new version of mythos was just released, I'm not sure if that was even the one used in this scan)
I guess this would also require models trained on pre-2023 data - or not trained on later curl code, changelogs, blog posts discussing curl security fixes, etc.
i don't think this is doable fairly. as they say in the blog post, the only fair way to is to look for new, previously undiscovered zero-days, otherwise you always risk the model has in some way been trained on the vulnerabilities. looking for legit new stuff is the only way to prevent leakage (even accidental one)
That’s my point too.