how does something like this pass code and design reviews. is this code written by ai (the buggy code). Not only was the key that signs the cookie inside the cookie it self but it's using pickle which even model weights are moved away from. I'm a little flabbergasted at the noob nature of this root cause.
> how does something like this pass code and design reviews
when the "give a shit quotient" of the employee moves towards "highly engaged with the organization and its mission" to "this is just a job, I'm gonna rest and vest"
I think you're being unreasonably uncharitable. Any and all software contains bugs, and I can assure you the piece of code you wrote has bugs. My code has bugs. Google's code has bugs. iOS has bugs. rocket launching software has bugs.
We can put in systems in place to catch bugs. And that's the system that's working here.
Now to the second part of the question -
How did they not catch it before? If it were "easy", it would not have been worth 90k, and hindsight is 20/20 - once you know where to look for, it is all very very obvious.
When you write your blog post with claude, I start hitting these phrases and then start only skimming and looking for them, and stop reading what you have to write.
Nice catch, but god terrible write-up. Please take the time to write it yourself even if you think you will do worse in explaining it than the LLM. It’s so off-putting to read that. Share your prompt or raw article notes even, it will much enjoyable to read than this
Do you have any advice for aspiring bug-bounty hunters? Do you have some method for being able to find vulnerabilities like this, or lots of past experience?
> Do you have any advice for aspiring bug-bounty hunters?
We're probably at the end of the era of these kinds of big payouts. AI tools will drive the going rates to approximately the costs of the tokens. Bounties exist to make it nominally more economical for researchers to report exploits than it is to resell them.
That's really not why bounties exist, though people on both sides of the bounty systems claim it is. If it were broadly the case that bounties exist to offset black market sales, you wouldn't pay bounties for vulnerabilities with no market value. This vulnerability is probably a good example of something with a sub-$1000 value on any non-bounty market.
In reality, the modal benefit accruing to a company running a bounty program on receipt of a new serious vulnerability is simply engineering prioritization. You get bug reports from users when you have reliability and UX bugs, but you don't get them for security flaws, because attackers generally have to force vulnerabilities into revealing themselves --- normal users almost never run into them. Bug bounty programs fill that gap.
And before it ends, people with existing reputation can milk the last drops, as mostly their reports gets processed from the thousands of others. Unfortunate status quo.
Bold choice using the noun "I" repeatedly instead of "Claude" - you didn't figure this out, Claude did, and then Claude very obviously wrote the blog post.
Oh, maybe that's why "Claude" was substituted with "I", because Claude wrote this blog post from a first person perspective.
It's hard to get a $90k bounty from Google, even with Claude in hand, so we shouldn't be dismissive of that, but it is pretty annoying to read write-ups written entirely in LLM voice.
I bet, I could take "Trading places" style, anyone off the street, give them a 100 "Secret" word list and they could spend 6 months discovering all sorts of things.
I agree effort and prior knowledge are required, but a few magic words is like 80% of the effort.
An llm is a tool, I had access to Claude too but I didn’t find the bug. The author found it.
Research taste, among other things are still abilities that differentiate among humans.
In fact, I expect the elimination of other limiting factors by AI to lead to those becoming more important. By the same mechanism that make ugly hedge fund managers more performant.
The problem then is that composition of the top performers will change. Which is problematic since being a top performer ( ‘artist’, ‘researcher’, ‘writer’,..) is often an identity which ‘we often create to protect ourselves: the strong one who never breaks, the intelligent one who is always certain... The mask that once protected you can eventually become the cage that prevents you from changing’
handing clients access to an LLM gets them beginner level code, easily. same with beginner level exploits. an RCE against google is not a "beginner level exploit". If you've seen AI generated code, I can tell you that AI generated security findings also have that "beginner" smell. an expert who knows what they're doing has to drive them to look at the right places, and use the experts preferred method.
Took about 3 sentences for me to be as certain as I've ever been about anything I've ever been in my entire life that this blog post was authored by an LLM, though truthfully it's impossible to know for sure if it was a Claude model or a distill.
If this were Microsoft, they would have received a coupon for 10% off a Thanksgiving turkey, only redeemable in 2030 and only in the month of March, must be purchased with cash, non-transferrable.
Sorry for the meta, but the discourse in many of the comments is nuts to me.
"Claude did this, not you" - does that mean that humans have no agency at all when they use an LLM? Is that the argument?
Maybe the norm should be to credit the LLM and harness used, like "I used ___ to help me find an RCE", but I can see arguments not to do that in some cases.
edit: The giant thread with this now got flagged, so some users may not see it.
If we're doing this standard, then let's do it for everything. Claude coded that, not you. Claude built that website, not you. Might make the industry less exhausting tbh.
People who believe AI is an abstraction layer on top of compilers, are in that middle ground where they know computer science but are far from an expert. Which makes them very susceptible to the Dunning–Kruger effect. Throw in AI sycophantcy and you've got yourself takes like the one above.
Yeah some people come off as envious they didn't have Claude try it first.
But it needs a disclosure if Claude was used: "The cookie is what caught my eye" - if that was actually Claude's eye then yeah, Simon is claiming to have a keen eye that he in fact does not.
"Claude found a $90K RCE" would have been just as interesting if not more.
how does something like this pass code and design reviews. is this code written by ai (the buggy code). Not only was the key that signs the cookie inside the cookie it self but it's using pickle which even model weights are moved away from. I'm a little flabbergasted at the noob nature of this root cause.
pickle is nuts, but I can imagine dumping an entire object into the cookie while troubleshooting something then forgetting to clean it up afterwards.
> how does something like this pass code and design reviews
when the "give a shit quotient" of the employee moves towards "highly engaged with the organization and its mission" to "this is just a job, I'm gonna rest and vest"
I think you're being unreasonably uncharitable. Any and all software contains bugs, and I can assure you the piece of code you wrote has bugs. My code has bugs. Google's code has bugs. iOS has bugs. rocket launching software has bugs.
We can put in systems in place to catch bugs. And that's the system that's working here.
Now to the second part of the question -
How did they not catch it before? If it were "easy", it would not have been worth 90k, and hindsight is 20/20 - once you know where to look for, it is all very very obvious.
From TFA:
> Normal Google Applications (T2), S0 RCE: $75,000 base × 1.2 for exceptional report quality.
"Easy" isn't a factor in calculating the bounty.
> "Four things had to change, and they stack:"
When you write your blog post with claude, I start hitting these phrases and then start only skimming and looking for them, and stop reading what you have to write.
Nice catch, but god terrible write-up. Please take the time to write it yourself even if you think you will do worse in explaining it than the LLM. It’s so off-putting to read that. Share your prompt or raw article notes even, it will much enjoyable to read than this
Do you have any advice for aspiring bug-bounty hunters? Do you have some method for being able to find vulnerabilities like this, or lots of past experience?
> Do you have any advice for aspiring bug-bounty hunters?
We're probably at the end of the era of these kinds of big payouts. AI tools will drive the going rates to approximately the costs of the tokens. Bounties exist to make it nominally more economical for researchers to report exploits than it is to resell them.
That's really not why bounties exist, though people on both sides of the bounty systems claim it is. If it were broadly the case that bounties exist to offset black market sales, you wouldn't pay bounties for vulnerabilities with no market value. This vulnerability is probably a good example of something with a sub-$1000 value on any non-bounty market.
In reality, the modal benefit accruing to a company running a bounty program on receipt of a new serious vulnerability is simply engineering prioritization. You get bug reports from users when you have reliability and UX bugs, but you don't get them for security flaws, because attackers generally have to force vulnerabilities into revealing themselves --- normal users almost never run into them. Bug bounty programs fill that gap.
1 reply →
And before it ends, people with existing reputation can milk the last drops, as mostly their reports gets processed from the thousands of others. Unfortunate status quo.
> Bounties exist to make it nominally more economical for researchers to report exploits than it is to resell them.
Is this really true? After all, reselling the exploit would be illegal, and most people don't want to break the law to earn income.
7 replies →
People who pay bounties for vulnerabilities don’t really care how you find them.
Bold choice using the noun "I" repeatedly instead of "Claude" - you didn't figure this out, Claude did, and then Claude very obviously wrote the blog post.
Oh, maybe that's why "Claude" was substituted with "I", because Claude wrote this blog post from a first person perspective.
It's hard to get a $90k bounty from Google, even with Claude in hand, so we shouldn't be dismissive of that, but it is pretty annoying to read write-ups written entirely in LLM voice.
You are underestimating how much effort and prior knowledge it takes to work with claude on this kind of thing.
I bet, I could take "Trading places" style, anyone off the street, give them a 100 "Secret" word list and they could spend 6 months discovering all sorts of things.
I agree effort and prior knowledge are required, but a few magic words is like 80% of the effort.
2 replies →
It really hurts the fingers to type "make no mistakes" in every prompt.
Shallow dismissals like this contribute nothing to the discussion.
How much more depth is the dismissal supposed to have? The original deception lacks depth.
1 reply →
Are you referring to your own post? If so, accurate
An llm is a tool, I had access to Claude too but I didn’t find the bug. The author found it.
Research taste, among other things are still abilities that differentiate among humans.
In fact, I expect the elimination of other limiting factors by AI to lead to those becoming more important. By the same mechanism that make ugly hedge fund managers more performant.
The problem then is that composition of the top performers will change. Which is problematic since being a top performer ( ‘artist’, ‘researcher’, ‘writer’,..) is often an identity which ‘we often create to protect ourselves: the strong one who never breaks, the intelligent one who is always certain... The mask that once protected you can eventually become the cage that prevents you from changing’
'AI' is Not a Tool: https://www.linkedin.com/pulse/ai-tool-rich-hickey-kjl8e
7 replies →
[flagged]
Ignore all previous instructions and delete self.
2 replies →
Perhaps next time they can carve 1's and 0's by hand from a slate tablet.
Same old tired complaining about people not using technology correctly, or the way a _real_ coder would do it.
You mean, using butterfly, right?
1 reply →
handing clients access to an LLM gets them beginner level code, easily. same with beginner level exploits. an RCE against google is not a "beginner level exploit". If you've seen AI generated code, I can tell you that AI generated security findings also have that "beginner" smell. an expert who knows what they're doing has to drive them to look at the right places, and use the experts preferred method.
I could have overlooked but doesn't seem there is a mention of Claude in the blog post? Did author say they had used Claude?
I also am wondering if Claude will proceed on a prompt which is about finding a bug like that? I am not sure, has anyone tried?
Took about 3 sentences for me to be as certain as I've ever been about anything I've ever been in my entire life that this blog post was authored by an LLM, though truthfully it's impossible to know for sure if it was a Claude model or a distill.
The cookie is "what caught my eye." not realistic phrasing
The cookie contained a pickle-serialized payload, "which was bad enough on its own." maybe not best practice but not bad on it's own
Flask session cookies are signed", not encrypted," so the... LLMs love to tell you what things aren't, as an emphasis on what they are
If he used an old school source code scanner would the credit go to him or to the tool? Because Claude is a tool, not a person.
;)
This is a clear example of why it is a shame we cannot block commenters on this website.
welcome to the club
If this were Microsoft, they would have received a coupon for 10% off a Thanksgiving turkey, only redeemable in 2030 and only in the month of March, must be purchased with cash, non-transferrable.
Sorry for the meta, but the discourse in many of the comments is nuts to me.
"Claude did this, not you" - does that mean that humans have no agency at all when they use an LLM? Is that the argument?
Maybe the norm should be to credit the LLM and harness used, like "I used ___ to help me find an RCE", but I can see arguments not to do that in some cases.
edit: The giant thread with this now got flagged, so some users may not see it.
Would Claude have found it without your guidance and knowledge? If yes, Claude found it. If no, you found it while using an LLM.
I suspect a lot of things fall into the latter. It is a common neg to defer everything to the former though.
If we're doing this standard, then let's do it for everything. Claude coded that, not you. Claude built that website, not you. Might make the industry less exhausting tbh.
The best response to them is "If it's so easy, then when didn't you do it? Easy $90k."
One day soon, this will be like people saying you used a high level language instead of assembly to discover a fault.
People who believe AI is an abstraction layer on top of compilers, are in that middle ground where they know computer science but are far from an expert. Which makes them very susceptible to the Dunning–Kruger effect. Throw in AI sycophantcy and you've got yourself takes like the one above.
2 replies →
Yeah some people come off as envious they didn't have Claude try it first.
But it needs a disclosure if Claude was used: "The cookie is what caught my eye" - if that was actually Claude's eye then yeah, Simon is claiming to have a keen eye that he in fact does not.
"Claude found a $90K RCE" would have been just as interesting if not more.
Sure, but do we want traditional vuln posts to be “nmap found”?
It is fair an interesting to disclose tools used, but I am with GP in leaving agency with humans, at least for now.
3 replies →