Data-only attacks are somewhat low-hanging fruit. Classical static analysis could already find them before AI got this strong, and LLMs make identification even easier. But the real threat is risk buried in business logic, especially abuse of normal business logic. Take e-commerce refund abuse. Bug hunters would not even call it a risk, yet fraud rings have arbitraged millions off this kind of logic. And because the logic is legitimate business logic, it is very hard to detect.
'Classic' non-AI fuzzers like AFL are still insanely useful and powerful, as are static analysis tools.
LLMs make all of these much, much easier to use. The other night, before I went to bed I told Kimi to go and fuzz filesystem code in the latest Linux kernel. I woke up to 26 crashes with reproducers and fixes. I'm still busy reviewing and upstreaming them. (Some have already landed.)
Similar use case here! Combining AI with fuzzers is so powerful, especially for creating a special fuzzing harness, or generating seeds for hard to reach code. That was taking hours/days and was frustratingly boring work before. Unfortunately the Codex models refuse a lot for me, I’m mostly using the cheapest models because they refuse the least, have you found Kimi to be a good alternative? Any other you tested that you can recommend? Thinking of switching.
I still remember me and my friends on club live finding that the games you could just submit the scores for and get free xbox stuff, and then doing some research online years later we found the entire thing was setup by employees to abuse themselves with plausible deniability.
Club live lost msft millions of dollars by itself.
> And because the logic is legitimate business logic, it is very hard to detect.
Hard to detect at n=1, yes. But larger scale - are you assuming that no Accounting or Sales managers are watching the returns ratios, nobody in Shipping is minding carrier delivery failure metrics, and nobody in Returns is raising alarms about the bricks they're receiving?
> Data-only attacks, those that do not affect a program’s control flow, have long been considered too sophisticated and niche to pose a practical threat.
Leveraging user data to get malicious behavior is the basis of interpreter eval injection (php, js, perl, shell calls, SQL ...). These attacks are like 50 years old. What do I miss?
I think of “data-only” as weird shorthand for “app-specific exploit primitives.”
An example I’ve seen somewhere is if a buffer overflow lets you change the value of another variable, but not directly control the instruction pointer. The exploit developer then has to figure out a way to turn their very constrained primitive into something useful, versus having access to a more powerful and generic primitive (stack return pointer, write-what-where, etc).
The example in the post is, basically, command injection, but it requires you to manipulate the app-specific state into a vulnerable state.
Computing is a huge subject, and there are many branches. For context, one branch, which I call "memory corruption studies", ever since the 1990s, is almost an entire research discipline with its own standing in infosec. As a result, entire OS concepts were invented to mitigate them (there even exists entire operating systems nearly dedicated to this this, such as OpenBSD and HardenedBSD), CPU hardware was literally modified to help mitigating them, new complier code generators were invented to mitigate them. Thousands of PhD degrees were awarded based studies on them. The research arm of every major tech firm has teams dedicated to them. There are infosec companies dedicated to this single subject.
In this context, practical data-only attacks as reported by this paper are the holy grail in this field.
As an outsider critic, you may say it's a side-effect of C/C++. If you use dynamic programming languages, this issue doesn't exist in your universe because there's no fundamental difference between code and data. But in another parallel universe of systems programming, it's a huge subject. Both the Morris worm and the publication of the article Smashing the Stack for Fun and Profit in Phrack were regarded by many as the milestones of hacker culture and canonical models of hacking, all the exploitation and mitigation studies that followed it were partially motivated by new hackers who wanted to "advance the field of hacking", so any advancement would be considered significant by a hacker. This is Hacker News, and I thought most people would understand this background. But apparently many developers are application-focused nowadays and hack in different universes, and it's not the case.
> The attack effectively modifies only the arguments of the execve syscall
I feel this checklist of shell-tools [0] is relevant, although the focus is more on how setuid is dangerous because you might not know the fancier arguments someone could supply.
> GTFOBins is a curated list of Unix-like executables that can be used to bypass local security restrictions in misconfigured systems.
Corrupting program memory via malicious input data is known as a code-execution attack, not a data-only attack. The fuzzed program usually crashes because its executable code or the control flow got overwritten directly by the input, or indirectly by the program code itself when it tries to process bad data. An exploit involves injecting external code, or overwriting memory addresses (like a virtual table or a stack return address) to override the original logic flow to do something else.
A data-only attack would be an attack that reuses the original logic by only corrupting data inputs (such as a flag or a file path), without overwriting code or overriding the logic. W^X, stack canary, or CFI won't work in these cases since no code is tampered by the attacker. In almost ever talk about compiler mitigations, you always hear a passing-by mention of data-only attacks - before the speaker immediately dismisses them as an academic curiosity when the software industry is still facing a flood of stack smashing and ROP attacks.
> The fuzzed program usually crashes because its executable code or the control flow got overwritten directly by the input, or indirectly by the program code itself when it tries to process bad data.
Add assertions to your code. Voila, your run-of-the-mill fuzzer can now hunt for arbitrary problems with your program by turning them into crashes.
When fuzzing C programs, I usually also add undefined-behaviour sanitizers and friends, in the mode where they crash when you run into the kinds of UB they can detect.
If "the server has a memory safety bug that allows a malicious client to overflow some buffer and overwrite, for instance, the contents of the cgi_bin_path variable", then why is this a data-only attack? Instill need to overflow a buffer the "traditional" way.
The boundaries are a _little bit_ vague of course. Traditional attacks transition from "data" (the buffer) to a reality where all control is handed to the attacker (ROP, traditional executable stack pivots, etc.)
The attacker controls what's executed, and stitches side-effects together to achieve the desired attack. This also starts with "data" but ends with "instructions of the attacker's choosing doing what the attacker wants".
In a data-flow attack (which _still_ has some characteristics of the former), the attacker does _not_ have the ability to pick-and-choose what gets executed, even in the final stages of their attack. They _still_ need to overflow the buffer, and put some other data in some other registers, but their attack _never_ gets to the point where they pick the next instruction.
Likely, at the end of the exploit, arbitrary code execution or privilege escalation has been unlocked. But the exploit chain itself doesn't go through a stage of arbitrary execution (either through ROP, or executable stack). That's how I like to distinguish anyway.
(2024) in title/article date already spoils that, this article comes from BC/AC era boundary, so the tool itself must've been written in BC era. LLMs wouldn't be useful for any such thing for another year.
(BC and AC are obviously Before and After ChatGPT.)
This showed me that taint analysis is kind of slept on. Maybe we should invest in better tooling that allows us to reverse engineer with taint analysis easier. Do we think it is a UI problem? Of course over tainting is a thing, but maybe we can make it work with better UI.
"Provenance" tracks where data came from, but "taint" tracks where unsafe data can flow. It's a uhh not so sexy term, but it is the industry standard term.
The main meaning of taint is contamination. Not really a revolting term to me. Maybe you associate it with some of its other meanings and that’s why you find it revolting.
Taint analysis won't help against memory safety vulnerabilities that directly scribble over untainted data to make it malicious. You need something like a secure enclave to prevent that kind of tampering.
Is this article saying that this AI tool has found 944 NEW unpatched exploits in nginx, and a comparable number in other commonly used server software?
First, there is no "AI", the author showed even a simple static analysis as proposed by them was sufficient to find data-only attacks, in contrary to the common belief that you need heavily customized exploits per application for this kind of attacks. This is the whole point of the research.
Next, I believe they found 944 available "data-only gadgets" usable by a pre-existing memory corruption bug. You still need to find a memory corruption bug first to use them, in the same sense that you need to hunt for ROP gadgets to get arbitrary code execution on a W^X system.
Well, with all the enshitified complex file formats out there, I bet this is a "piece of cake" (for the people of the field with good tooling) to find exploits in the state management of code dealing with them. And very high computer languages/specialized script languages won't protect anything as those are "logic errors". I think "closing" state machines on such complex file formats is just pointless, better design new file formats from the ground up, but this time with a much stronger focus on robustness (keep that state machine as small and simple as possible and super rigorously well defined, and I mean like maths maniacs). And it is hard: because the purpose of the file format must come AFTER the robustness, namely tough compromise on the purpose itself may have to be done.
Data-only attacks are somewhat low-hanging fruit. Classical static analysis could already find them before AI got this strong, and LLMs make identification even easier. But the real threat is risk buried in business logic, especially abuse of normal business logic. Take e-commerce refund abuse. Bug hunters would not even call it a risk, yet fraud rings have arbitraged millions off this kind of logic. And because the logic is legitimate business logic, it is very hard to detect.
Going on a bit of a tangent:
'Classic' non-AI fuzzers like AFL are still insanely useful and powerful, as are static analysis tools.
LLMs make all of these much, much easier to use. The other night, before I went to bed I told Kimi to go and fuzz filesystem code in the latest Linux kernel. I woke up to 26 crashes with reproducers and fixes. I'm still busy reviewing and upstreaming them. (Some have already landed.)
Similar use case here! Combining AI with fuzzers is so powerful, especially for creating a special fuzzing harness, or generating seeds for hard to reach code. That was taking hours/days and was frustratingly boring work before. Unfortunately the Codex models refuse a lot for me, I’m mostly using the cheapest models because they refuse the least, have you found Kimi to be a good alternative? Any other you tested that you can recommend? Thinking of switching.
1 reply →
At work, I am arguing with tech leads that they should sanitize user inputs for a public facing chatbot.
I want to cry.
1 reply →
> (Some have already landed.)
Do you have links to some of these?
1 reply →
It is more than LLMs are faster using those tools, than they are finding more errors by themselves analysing the code.
1 reply →
I still remember me and my friends on club live finding that the games you could just submit the scores for and get free xbox stuff, and then doing some research online years later we found the entire thing was setup by employees to abuse themselves with plausible deniability.
Club live lost msft millions of dollars by itself.
Lol, heh, how long before we find AI is setting up hidden doors like this to extract money from software they make via external methods.
> And because the logic is legitimate business logic, it is very hard to detect.
Hard to detect at n=1, yes. But larger scale - are you assuming that no Accounting or Sales managers are watching the returns ratios, nobody in Shipping is minding carrier delivery failure metrics, and nobody in Returns is raising alarms about the bricks they're receiving?
> Data-only attacks, those that do not affect a program’s control flow, have long been considered too sophisticated and niche to pose a practical threat.
Leveraging user data to get malicious behavior is the basis of interpreter eval injection (php, js, perl, shell calls, SQL ...). These attacks are like 50 years old. What do I miss?
I think of “data-only” as weird shorthand for “app-specific exploit primitives.”
An example I’ve seen somewhere is if a buffer overflow lets you change the value of another variable, but not directly control the instruction pointer. The exploit developer then has to figure out a way to turn their very constrained primitive into something useful, versus having access to a more powerful and generic primitive (stack return pointer, write-what-where, etc).
The example in the post is, basically, command injection, but it requires you to manipulate the app-specific state into a vulnerable state.
Computing is a huge subject, and there are many branches. For context, one branch, which I call "memory corruption studies", ever since the 1990s, is almost an entire research discipline with its own standing in infosec. As a result, entire OS concepts were invented to mitigate them (there even exists entire operating systems nearly dedicated to this this, such as OpenBSD and HardenedBSD), CPU hardware was literally modified to help mitigating them, new complier code generators were invented to mitigate them. Thousands of PhD degrees were awarded based studies on them. The research arm of every major tech firm has teams dedicated to them. There are infosec companies dedicated to this single subject.
In this context, practical data-only attacks as reported by this paper are the holy grail in this field.
As an outsider critic, you may say it's a side-effect of C/C++. If you use dynamic programming languages, this issue doesn't exist in your universe because there's no fundamental difference between code and data. But in another parallel universe of systems programming, it's a huge subject. Both the Morris worm and the publication of the article Smashing the Stack for Fun and Profit in Phrack were regarded by many as the milestones of hacker culture and canonical models of hacking, all the exploitation and mitigation studies that followed it were partially motivated by new hackers who wanted to "advance the field of hacking", so any advancement would be considered significant by a hacker. This is Hacker News, and I thought most people would understand this background. But apparently many developers are application-focused nowadays and hack in different universes, and it's not the case.
Most languages do not provide direct eval.
> The attack effectively modifies only the arguments of the execve syscall
I feel this checklist of shell-tools [0] is relevant, although the focus is more on how setuid is dangerous because you might not know the fancier arguments someone could supply.
> GTFOBins is a curated list of Unix-like executables that can be used to bypass local security restrictions in misconfigured systems.
[0] https://gtfobins.org/
> Data-only attacks ... have long been considered too sophisticated and niche to pose a practical threat.
I thought the whole point of fuzzing was an example of finding data-only attacks.
Corrupting program memory via malicious input data is known as a code-execution attack, not a data-only attack. The fuzzed program usually crashes because its executable code or the control flow got overwritten directly by the input, or indirectly by the program code itself when it tries to process bad data. An exploit involves injecting external code, or overwriting memory addresses (like a virtual table or a stack return address) to override the original logic flow to do something else.
A data-only attack would be an attack that reuses the original logic by only corrupting data inputs (such as a flag or a file path), without overwriting code or overriding the logic. W^X, stack canary, or CFI won't work in these cases since no code is tampered by the attacker. In almost ever talk about compiler mitigations, you always hear a passing-by mention of data-only attacks - before the speaker immediately dismisses them as an academic curiosity when the software industry is still facing a flood of stack smashing and ROP attacks.
> The fuzzed program usually crashes because its executable code or the control flow got overwritten directly by the input, or indirectly by the program code itself when it tries to process bad data.
Add assertions to your code. Voila, your run-of-the-mill fuzzer can now hunt for arbitrary problems with your program by turning them into crashes.
When fuzzing C programs, I usually also add undefined-behaviour sanitizers and friends, in the mode where they crash when you run into the kinds of UB they can detect.
3 replies →
TFA says memory corruption is a data-only attack, because it just overwrites data in memory.
2 replies →
I'm missing a critical part of the explanation.
If "the server has a memory safety bug that allows a malicious client to overflow some buffer and overwrite, for instance, the contents of the cgi_bin_path variable", then why is this a data-only attack? Instill need to overflow a buffer the "traditional" way.
The boundaries are a _little bit_ vague of course. Traditional attacks transition from "data" (the buffer) to a reality where all control is handed to the attacker (ROP, traditional executable stack pivots, etc.)
The attacker controls what's executed, and stitches side-effects together to achieve the desired attack. This also starts with "data" but ends with "instructions of the attacker's choosing doing what the attacker wants".
In a data-flow attack (which _still_ has some characteristics of the former), the attacker does _not_ have the ability to pick-and-choose what gets executed, even in the final stages of their attack. They _still_ need to overflow the buffer, and put some other data in some other registers, but their attack _never_ gets to the point where they pick the next instruction.
Likely, at the end of the exploit, arbitrary code execution or privilege escalation has been unlocked. But the exploit chain itself doesn't go through a stage of arbitrary execution (either through ROP, or executable stack). That's how I like to distinguish anyway.
Yes, the author uses specialized terminology which is easy to misunderstand. They automate DEP compliant transition from buffer overflow to RCE.
I'm positively surprised that their tool is not yet another LLM wrapper.
The quality of research (and by extent HN submissions) has really plummeted since LLMs have become marginally useful
(2024) in title/article date already spoils that, this article comes from BC/AC era boundary, so the tool itself must've been written in BC era. LLMs wouldn't be useful for any such thing for another year.
(BC and AC are obviously Before and After ChatGPT.)
The last 2 years has been a very long eon.
This showed me that taint analysis is kind of slept on. Maybe we should invest in better tooling that allows us to reverse engineer with taint analysis easier. Do we think it is a UI problem? Of course over tainting is a thing, but maybe we can make it work with better UI.
Is that really the name for it? It sounds revolting
Can’t we just use prim and proper terms like provenance
"Provenance" tracks where data came from, but "taint" tracks where unsafe data can flow. It's a uhh not so sexy term, but it is the industry standard term.
1 reply →
The main meaning of taint is contamination. Not really a revolting term to me. Maybe you associate it with some of its other meanings and that’s why you find it revolting.
1 reply →
Taint analysis won't help against memory safety vulnerabilities that directly scribble over untainted data to make it malicious. You need something like a secure enclave to prevent that kind of tampering.
[dead]
Is this article saying that this AI tool has found 944 NEW unpatched exploits in nginx, and a comparable number in other commonly used server software?
First, there is no "AI", the author showed even a simple static analysis as proposed by them was sufficient to find data-only attacks, in contrary to the common belief that you need heavily customized exploits per application for this kind of attacks. This is the whole point of the research.
Next, I believe they found 944 available "data-only gadgets" usable by a pre-existing memory corruption bug. You still need to find a memory corruption bug first to use them, in the same sense that you need to hunt for ROP gadgets to get arbitrary code execution on a W^X system.
correction: s/static analysis/data-flow analysis/
Well, with all the enshitified complex file formats out there, I bet this is a "piece of cake" (for the people of the field with good tooling) to find exploits in the state management of code dealing with them. And very high computer languages/specialized script languages won't protect anything as those are "logic errors". I think "closing" state machines on such complex file formats is just pointless, better design new file formats from the ground up, but this time with a much stronger focus on robustness (keep that state machine as small and simple as possible and super rigorously well defined, and I mean like maths maniacs). And it is hard: because the purpose of the file format must come AFTER the robustness, namely tough compromise on the purpose itself may have to be done.