Comment by mgaldys4
4 days ago
Data-only attacks are somewhat low-hanging fruit. Classical static analysis could already find them before AI got this strong, and LLMs make identification even easier. But the real threat is risk buried in business logic, especially abuse of normal business logic. Take e-commerce refund abuse. Bug hunters would not even call it a risk, yet fraud rings have arbitraged millions off this kind of logic. And because the logic is legitimate business logic, it is very hard to detect.
Going on a bit of a tangent:
'Classic' non-AI fuzzers like AFL are still insanely useful and powerful, as are static analysis tools.
LLMs make all of these much, much easier to use. The other night, before I went to bed I told Kimi to go and fuzz filesystem code in the latest Linux kernel. I woke up to 26 crashes with reproducers and fixes. I'm still busy reviewing and upstreaming them. (Some have already landed.)
Similar use case here! Combining AI with fuzzers is so powerful, especially for creating a special fuzzing harness, or generating seeds for hard to reach code. That was taking hours/days and was frustratingly boring work before. Unfortunately the Codex models refuse a lot for me, I’m mostly using the cheapest models because they refuse the least, have you found Kimi to be a good alternative? Any other you tested that you can recommend? Thinking of switching.
So Codex with Daybreak Blue refuses less. You need to join OpenAI's cybersecurity program. But it seems to be pretty simple: I just told them that I do some Linux kernel work for fun, but that codex refuses anything that touches C and the kernel.
I've also tried DeepSeek and now Mimo. DeepSeek was really quite useful before the price increase, because I didn't care too much about burning a lot of tokens. But it was less useful since, especially since my timezone here in Singapore is the same as in China, so my waking and working hours have a lot of overlap with their peak pricing. However, it's gotten a bit cheaper effectively, because their new flash model is supposedly as good as the old pro model.
Which cheap models are you using (especially those that refuse less)?
So far I've only really used Mimo to drive Sashiko, which is a kernel review tool. I haven't used it for actually writing code.
Kimi is pretty decent overall. But I haven't given it really hard work. Running a fuzzer is pretty simple, and so's the other stuff I asked it so far.
At work, I am arguing with tech leads that they should sanitize user inputs for a public facing chatbot.
I want to cry.
Oh, I'm doing the fuzzing for fun.
But yeah, you can't trust user inputs!
> (Some have already landed.)
Do you have links to some of these?
Accepted by maintainers so far:
In mainline:
ext4, xattr cache: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
ocfs2, cluster accounting: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
Taken into Jan Kara's filesystem tree this week, all isofs:
An out-of-bounds read in the multi-extent directory walk: https://lore.kernel.org/all/20260923153220.777907-1-matthias...
A hang in a fix Jan had just written, which he is folding into it: https://lore.kernel.org/all/20260923153134.771632-1-matthias...
A bug Jan fixed himself after my report: https://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-f...
Sashiko, the LLM patch reviewer that now comments on kernel mailing lists, merged a fix of mine: https://github.com/sashiko-dev/sashiko/pull/556 (a patch that contained one of its prompt placeholders as text had it replaced).
Not accepted yet, but I think interesting:
libata, reviewed by the maintainer but not applied yet: a faulty ATAPI device could make the kernel write past its sense buffer. https://lore.kernel.org/all/20260922182655.2423663-1-matthia...
e2fsck, the repair tool, could deadlock on some corrupt images and hang forever. Reviewed by Darrick Wong. https://lore.kernel.org/all/20260922110435.1528332-1-matthia...
ntfs: a crafted image makes mount hang forever, because the mount waits on a lock its own read already holds. One of the maintainers asked for a wider fix, which I am testing. https://lore.kernel.org/all/20260922153931.1976405-1-matthia...
Sashiko was reviewing ext4 patches against a tree from 2020: https://github.com/sashiko-dev/sashiko/issues/559, fix in https://github.com/sashiko-dev/sashiko/pull/560, plus a one-line MAINTAINERS patch naming the ext4 branch, which Jan acked: https://lore.kernel.org/all/20260923101749.3505886-1-matthia...
Not posted yet, still being reviewed on my side:
ntfs: writing to a compressed file on a volume whose free space is fragmented silently overwrote clusters belonging to other files. write() reported success; the damage only showed after a remount.
ntfs: reads of a damaged file could return zeros for data that is on disk, and writes to it could be dropped.
Sashiko: on its Claude, Vertex and Gemini backends, a connection that drops mid-response is treated as a permanent error rather than retried.
The rest, mostly more ntfs, are still in review or on my desk. Not all of these came from that first night; the ext4 and ocfs2 ones are older.
I also have a lot of bcachefs contributions, but that's because I'm actively using that filesystem on my desktop; instead of a pure fuzzing run.
It is more than LLMs are faster using those tools, than they are finding more errors by themselves analysing the code.
These tools find different errors, I'd say?
Fuzzers execute, which has different failure modes that having an LLM do informal reasoning on the code. (I haven't played enough with static analysis to say anything insightful about how they compare with LLM.)
The Linux kernel is a really big piece of software. Using a fuzzer to point you at an interesting (and potentially dodgy) piece of code is a good start. Often once you found the first problem, you or your LLM can read around and find more problems.
I still remember me and my friends on club live finding that the games you could just submit the scores for and get free xbox stuff, and then doing some research online years later we found the entire thing was setup by employees to abuse themselves with plausible deniability.
Club live lost msft millions of dollars by itself.
Lol, heh, how long before we find AI is setting up hidden doors like this to extract money from software they make via external methods.
> And because the logic is legitimate business logic, it is very hard to detect.
Hard to detect at n=1, yes. But larger scale - are you assuming that no Accounting or Sales managers are watching the returns ratios, nobody in Shipping is minding carrier delivery failure metrics, and nobody in Returns is raising alarms about the bricks they're receiving?