Comment by melvinroest
4 days ago
Wow, this announcement is good content marketing.
Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.
It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.
> We then ran AISLE's autonomous AI system against curl.
They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.
The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.
Thanks for figuring that out. Sort of sounds like AI programming programs to find vulnerabilities, of which fuzzing is one of the proven techniques to do it.
Back when I first heard of it, the HuggingFace hack reminded me about how you don't have to be particularly smart to find vulns, just aggressive in looking for them. AFL on its own "learned" how to construct well-formed JPEGs and PDFs by fuzzing decoders for these formats, back in the pre-LLM dark ages.
Of course the HuggingFace hack proved to be much more than that...
It's an effective approach. Google project zero started doing this in 2024
https://security.googleblog.com/2024/11/leveling-up-fuzzing-...
Also sounds incredibly compute intensive.
Setting a swarm of agents loose for hours to look for software vulnerabilities is far more compute-expensive than fuzzing. The industry has never thrown this kind of compute resources at pure fuzzing, in part because you can't get much VC money for that.
2 replies →
Fuzzing or having the LLM sort through where might be most useful to fuzz & sorting the results? Neither seem particularly compute intensive to me, fuzzing is a pretty standard step and having the LLM read through to find the most interesting areas to fuzz sounds a lot more efficient than leaving the whole task to the LLM.
[dead]
Their system can run with various models, they go into more details in this article.
https://aisle.com/blog/system-over-model-zero-day-discovery-...
It wouldn’t surprise me if AISLE uses many different providers’ models, and what’s holding back OpenAI and Anthropic is only using first-party models. Just because OpenAI and Anthropic have arguably the strongest models overall doesn’t mean their models are the strongest at finding any given class of vulnerability or lead to follow.
> Wow, this announcement is good content marketing.
Why do you think companies hire PMMs?
>All six are rated Low severity
This says it all. Nothing important was missing. This is marketing hype.
> what models the use under the hood
Presumably their own, wouldn’t they?
You mean their own trained models, or do you think it's an open source model that they fine-tuned? If they use their own, I'd guess it's the latter.
Maybe the model doesn’t matter, maybe you just need something minimally intelligent to seed the fuzzer, generate a test case, and rinse and repeat when the fuzzer gets stuck.
Default to gpt 5.4 nano
https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
It defaults to gpt5.4 nano
https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
A repo named "nano-analyzer" unsurprisingly uses gpt5.4 nano. I doubt their "pay them money" version uses nano.
I am pretty sure that the nano-analyzer is just a limited open source demo of their "System over Model" thesis from https://aisle.com/blog/system-over-model-zero-day-discovery-..., not the main product.