I handle reports for a one million dollar bug bounty program.
AI spam is bad. We've also never had a valid report from an by an LLM (that we could tell).
People using them will take any being told why a bug report is not valid, questions, or asks for clarification and run them back through the same confused LLM. The second pass through generates even deeper nonsense.
It's making even responding with anything but "closed as spam" not worth the time.
I believe that one day there will be great code examining security tools. But people believe in their hearts that that day is today, and that they are riding the backs of fire breathing hack dragons. It's the people that concern me. They cannot tell the difference between truth and garbage.
This has been going for years before AI - they say we live in a "post-truth society". The generation and non-immediate-rejection of AI slop reports could be another manifestation of post-truth rather than a cause of it.
> I believe that one day there will be great code examining security tools.
As for programming, I think that we will simply continue to have incrementally better tools based on sane and appropriate technologies, as we have had forever.
What I'm sure about is that no such tool can come out of anything based on natural language, because it's simply the worst possible interface to interact with a computer.
people have been trying various iterations of "natural language programming" since programming languages were a thing. Even COBOL was supposed to be more natural than other languages of the era.
This sounds more like an influx of scammers than security researchers leaning too hard on AI tools. The main problem is the bounty structure. And I don’t think these influx of low quality reports will go away, or even get any less aggressive as long as there is money to attract the scammers. Perhaps these bug bounty programs need to develop an automatic pass/fail tester of all submitted bug code, to ensure the reporter really found a bug, before the report is submitted to the vendor.
It's unfortunately widespread. We don't offer bug bounties, but we still get obviously LLM-generated "security reports" which are just nonsense and waste our time. I think the motivation may be trying to get credit for contributing to open source projects.
Simply charge a fee to submit a report. At 1% of the payment for low bounties it's perfectly valid. Maybe progressively scale that down a bit as the bounty goes up. But still for a $50k bounty you know is correct it's only $500.
The improvement history of tools beside LLMs, I suspect. First we had syntax highlighting, and we were wondered. Now we have fuzzers and sandbox malware analysis, who knows what the future will bring?
This is interesting because they've apparently made a couple thousand dollars reporting things to other companies. Is it just a case of a broken clock being right twice a day? Seems like a terrible use of everyone's time and money. I find it hard to believe a random person on the internet using ChatGPT is worth $1000.
There are places that will pay bounties on even very flimsy reports to avoid the press / perception that they aren't responding to researchers. But that's only going to remain as long as a very small number of people are doing this.
It's easy for reputational damage to exceed $1'000, but if 1000 people do this...
$1000 is cheap... The real question is when will companies become wise to this scam?
Most companies make you fill in expense reports for every trivial purchase. It would be cheaper to just let employees take the cash - and most employees are honest enough. However the dishonest employee isn't why they do expense reports (there are other ways to catch dishonest employees). There used to be a scam where someone would just send a bill for "services" and those got paid often enough until companies realized the costs and started making everyone do the expense reports so they could track the little expenses.
Can someone explain the ip address in the hackerone profile[0]? I can't tell if 139.224.130.174 is a reference to something real or just hallucinated by the LLM to look "cool". Wikipedia says that this /8 is controlled by "MIX"[1] but my google-fu is failing me atm.
You can tell it's ChatGPT from the stupid icon.
In one of the iterations they started using thses emojis which are disturbing for me.
The answer to the first question has obvious ChatGPT writing style.
Good god did they hallucinate the segmentation fault and the resulting GDB trace too? Given that the diffs don’t even apply and the functions don’t even exist, I guess the answer is yes - in which case, this is truly a new low for AI slop bug reports.
An real report would have a GDB trace that looks like that, so it isn't hard to create such a trace. Many of us could create a real looking GDB trace just as well by hand - it would be tedious, boring, and pointless but we could.
If I wanted to slip a vulnerability into a major open source project with a lot of eyes on it, using AI to DDOS their vulnerability reports so they're less likely to find a real report from someone who caught me seems like an obvious (and easy) step.
Looking at one of the bogus reports, it doesn't even seem like a real person. Why do this if you're not trying to gain recognition?
> Why do this if you're not trying to gain recognition?
They're doing it for money, a handful of their reports did result in payouts. Those reports aren't public though, so there's no way to know if they actually found real bugs or the reviewer rubber-stamped them without doing their due diligence.
Reading the straw that broke the camel's back commit illustrates the problem really well: https://hackerone.com/reports/3125832 . This shit must be infuriating to dig through.
I wonder if reputation systems might work here - you could give anyone who id's with an AML/KYC provider some reputation, enough for two or three reports, let people earn reputation digging through zero rep submissions and give someone like 10,000 reputation for each accurate vulnerability found, and 100s for any accurate promoted vulnerabilities. This would let people interact anonymously if they want to edit, quickly if they found something important and are willing to AML/KYC, and privilege quality people.
Either way, AI is definitely changing economics of this stuff, in this case enshittifying first.
The vast majority of developers are 10-100x more likely to find a security hole in a random tool than spend time improving their reputation on a bug bounty site that pays < 10% their salary.
That makes it extremely hard to build a reputation system for a site like that. Almost all the accounts are going to be spam, and the highest quality accounts are going to freshly created and take ~ 1 action on the platform.
Or a deposit system: pay 2€ for a human to read this message, you'll get it back if it's not spam
What if the human marks it as spam but you're actually legit? Deposit another 2€ to have the platform (like Hackerone or whichever you're reporting via) give a second opinion, you'll get the 4€ back if you weren't spamming. What to do with the proceeds from spammers? The first X euros of spam reports go to upkeep of the platform, the rest to a good cause defined by the projects to whom the reports were submitted because they were the ones who had to deal with reading the slop so they get at least this much out of it
Raise deposit cost so long as slop volume remains unmanageable
This doesn't discriminate against people who aren't already established, but it may be a problem if you live in a low-income country and can't easily afford 20€ (assuming it ever gets to that deposit level). Perhaps it wouldn't work, but it can first be trialed at a normal cost level. Another concern is anonymity and payment. We hackers are often a paranoid lot. One can always support cash in the mail though, the sender can choose whether their privacy is worth a postage stamp
Reputation systems for this kind of thing sounds like rubbing some anti-itch cream on bullet wound. I feel like the problem seems to me to be behavior, not a technology issue.
Personally I can't imagine how miserable it would be for my hard-earned expertise to be relegated to sifting through SLOP where maybe 1 in hundreds or even thousands of inquiries is worth any time at all. But it also doesn't seem prudent to just ignore them.
I don't think better ML/AI technology or better information systems will make a significant difference on this issue. It's fundamentally about trust in people.
I consider myself a left leaning soyboy, but this could be the outcome of too "nice" of a discourse. I won't advocate for toxicity, but I am considering if we bolster the self-image of idiots when we refuse to call them idiots. Because you're right, this is fundamentally a people problem, specifically we need people to filter this themselves.
> I feel like the problem seems to me to be behavior, not a technology issue.
To be honest, this has been a grimly satisfying outcome of the AI slop debacle. For decades, the general stance of tech has been, “there is no such thing as a behavioral/social problem, we can always fix it with smarter technology”, and AI is taking that opinion and drowning it in a bathtub. You can’t fix AI slop with technology because anything you do to detect it will be incorporated into better models until they evade your tests.
We now have no choice but to acknowledge the social element of these problems, although considering what a shitshow all of Silicon Valley’s efforts at social technology have been up to now, I’m not optimistic this acknowledgement will actually lead anywhere good.
> I feel like the problem seems to me to be behavior, not a technology issue.
Yes, it's a behavior issue, but that doesn't mean it can't be solved or at least minimized by technology, particularly as a technology is what's exacerbating the issue?
IMO, this AI crap is just the next step of the "let's block criminal behavior with engineering" path we followed for decades. That might very well be the last straw, as it is very unlikely we can block this one efficiently and reliably.
It's due time we ramp-up our justice systems to make people truly responsible and punished for their bad behavior online, including all kind of spams, scams, fishing and disinformation.
That might involve the end of anonymity on internet, and lately I feel that the downsides of that are getting smaller and smaller compared to it's upsides.
Didn't even have to click through to the report in question to know it would be all hallucinations -- both the original patchfile and the segfault ("ngtcp2_http3_handle_priority_frame".. "There is no function named like this in current ngtcp2 or nghttp3.") I guess these guys don't bother to verify, they just blast out AI slop and hope one of them hits?
Reminds me of when some LLM (might have been Deepseek) told me I could add wasm_mode=True in my FastHTML python code which would allow me to compile it to WebAssembly, when of course there is no such feature in FastHTML. This was even when I had provided it full llms-ctx.txt
I had Google's in-search "AI" invent a command line switch that would have been very helpful... if it existed. Complete with usage caveats and warnings!
> I guess these guys don't bother to verify, they just blast out AI slop and hope one of them hits?
Yes. Unfortunately, some companies seem to pay out the bug bounty without even verifying that the report is actually valid. This can be seen on the "reporter"'s profile: https://hackerone.com/evilginx
A prominent project in which people have a stake in seeing bugs fixed can afford to charge a refundable deposit against reporters.
Say, $100.
If your report is true, or even if it is incorrect but honestly mistaken, you get your $100 back.
If it is time-wasting slop with hallucinated gdb crash traces, then you don't get your money back (and so you don't pay the deposit in the first place, and don't send such a report, unless you're completely stupid, or too rich to care about $100).
If AI slopsters have to pay to play, with bad odds and no upside, they will go elsewhere.
Well the reporter in the report that stated it that they are open for employment https://hackerone.com/reports/3125832 Anyone want to hire them? They can play with ChatGPT all day and spam random projects with the AI slop.
I can imagine that most LLMs, if you ask it to find a security vulnerability in a given piece of code, will make something up completely out of the air. I've (mistakenly) sent valid code with an unrelated error and to this day I get nonsense "fixes" for these errors.
This alignment problem between responding with what the user wants (e.g. a security report, flattering responses) and going against the user seems a major problem limiting the effectiveness of such systems.
Counterpoint we have a CVE attributable to ours and I suspect the difference is my co-founder was an offensive kernel researcher so our system is tuned for this in a way your average...ambulance chaser is unable to do.
The amount of bad reports curl in particular has gotten is staggering and it's all from people who have no background just latching onto a tool that won't elevate them.
Edit: Also shoutout to one of our old professors Brendan Dolan-Gavitt who now works on offensive security agents who has a highly ranked vulnerability agent XBOW.
So these tools are there and doing real work its just there are so many people looking for a quick buck that you really have to tease the noise from the bs.
Except if you read the blog post we helped a very confused maintainer when they had this dropped on them with no explanation on hacker news except "oooh potential scary heap vuln"
Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!
I recently had this happen from a senior engineer. What's really frustrating is I TOLD them the issues and how to fix it. Instead of listening to what I told them, they plugged it into GPT and responded with "Oh, interesting this is what GPT says" (Which, spoiler, was similar but lacking from what I'd said).
Meaning, instead of listening to a real-life expert in the company telling them how to handle the problem they ignored my advice and instead dumped the garbage from GPT.
I really fear that a number of engineers are going to us GPT to avoid thinking. They view it as a shortcut to problem solve and it isn't.
>They view it as a shortcut to problem solve and it isn't
Oh but it is, used wisely.
One: it's a replacement for googling a problem and much faster. Instead of spending half an hour or half a day digging through bug reports, forum posts, and stack overflow for the solution to a problem. LLMs are a lot faster, occasionally correct, and very often at least rather close.
Two: it's a replacement for learning how to do something I don't want to learn how to do. Case Study: I have to create a decent-enough looking static error page for a website. I could do an awful job with my existing knowledge, I could spend half a day relearning and tweaking CSS, elements, etc. etc. or I could ask an LLM to do it and then tweak the results. Five minutes for "good enough" and it really is.
LLMs are not a replacement for real understanding, for digging into a codebase to really get to the core of a problem, or for becoming an expert in something, but in many cases I do not want to, and moreover it is a poor use of my time. Plenty of things are not my core competence or anywhere near the goals I'm trying to achieve. I just need a quick solution for a topic I'm not interested in.
I often do this - ask a LLM for an answer when I already have it from an expert. I do it to evaluate the ability of the LLM. Usually not in the presence of said expert tho.
Is it possible that what happened was an impedance mismatch between you and the engineer such that they couldn’t grok what you told them but ChatGPT was able to describe it in a manner they could understand? Real-life experts (myself included, though I don’t claim to be an expert in much) sometimes have difficulty explaining domain-specific concepts to other folks; it’s not a flaw in anyone, folks just have different ways of assembling mental models.
You should ask yourself why this organization wants engineering advice from a chatbot more than from you.
I doubt the reason has to do with your qualities as an engineer, which must be basically sound. Otherwise why bother to launder the product of your judgment, as you described here someone doing?
> I really fear that a number of engineers are going to us GPT to avoid thinking. They view it as a shortcut to problem solve and it isn't.
How is this sentiment not different from my grandfather’s sentiment that calculators and computers (and probably his grandfather’s view of industrialization) are a shortcut to avoid work? From my perspective most tools are used as a shortcut to avoid work; that’s kinda the while point—to give us room to think about/work on other stuff.
It is supremely annoying when i ask in a group if someone has experience with a tool or system and some idiot copies my question into some LLM and paste the answer. I can use the LLM just like anyone, if i'm asking for EXPERIENCE it is because I want the opinion of a human who actually had to deal with stuff like corner cases.
I work in a corporate environment as I’m sure many others do. Many executives have it in their head that LLMs are this brand new efficiency gain they can pad profit margins with, so you should be using it for efficiency. There’s a lot of push for that, everywhere where I work.
I see email blasts suggesting I should be using it, I get peers saying I should be using it, I get management suggesting I should use it to cut costs… and there is some truth there but as usual, it depends.
I, like many others, can’t be asked to take on inefficiency in the name of efficiency ontop of currently most efficient ways to do my work. So I too say “ChatGPT said: …” because I dump lots of things into it now. Some things I can’t quickly verify, some things are off, and in general it can produce far more information than I have time to check. Saying “ChatGPT said…” is the current CYA caveat statement around the world of: use this thing but also take liability for it. No, if you practically mandate I use something, the liability falls on you or that thing. If it’s a quick verify I’ll integrate it into knowledge. A lot of things aren’t.
> I see email blasts suggesting I should be using it, I get peers saying I should be using it, I get management suggesting I should use it to cut costs
The ideal scenario: you write a few bulletpoints and ask Copilot to turn it into a long-form email to send out. Your receiving coworker then asks Copliot to distill it back into a few bullet points they can skim.
You saved 5 minutes but one of your points was ignored entirely and 20% of your output is nonsensical.
Your coworker saved 2 minutes but one of their bulletpoints was hallucinated and important context is missing from the others.
Microsoft collects a fee from both of you and is the only winner here.
It just feels to me like a boss walking into a car mechanic's shop holding some random tool, walking up to a mechanic, and:
"Hey, whatcha doin?"
"Oh hi, yea, this car has a slight misfire on cyl 4, so I was just pulling one of the coilpacks to-"
"Yea alright, that's great. So hey! You _really_ need to use this tool. Trust me, it's gonna make your life so much easier"
"umm... that's a 3d printer. I don't really think-"
"Trust me! It's gonna 10x your work!"
...
I love the tech. It's the evangelists that don't seem to bother researching the tech beyond making an account and asking it to write a couple scripts that bug me. And then they proclaim it can replace a bunch of other stuff they don't/haven't ever bothered to research or understand.
Seriously. Being able to look up stuff using AI is not unique. I can do that too.
This is kind of the same with any AI gen art. Like I can go generate a bunch of cool images with AI too, why should I give a shit about your random Midjourney output.
Comfyui workflows, fine-tuning models, keeping up with the latest arxiv papers, patching academic code to work with generative stacks, this stuff is grueling.
I mean… I have a fancy phone camera in my pocket too, but there are photographers who, with the same model of fancy phone camera, do things that awe and move me.
It took a solid hundred years to legitimate photography as an artistic medium, right? To the extent that the controversy still isn’t entirely dead?
Any cool images I ask AI for are going to involve a lot less patience and refinement than some of these things the kids are using AI to turn out…
For that matter, I’ve watched friends try to ask for factual information from LLMs and found myself screaming inwardly at how vague and counterproductive their style of questioning was. They can’t figure out why I get results I find useful while they get back a wall of hedging and waffling.
In general those point to the person's understanding being shallow. So far when someone says "GPT said..." it is a new low in understanding, and there is no more to the article they googled or second stackOverflow answer with a different take on it, it is the end of the conversation.
The complaint isn't about stating the source. The complaint is about asking for advice, then ignoring that advice. If one asks how to do something, get a reply, then reply to that reply 'but Google says', that's just as rude.
It's a "source" that cannot be reproduced or actually referenced in any way.
And all the other examples will have a chain of "upstream" references, data and discussion.
I suppose you can use those same phrases to reference things without that, random "summaries" without references or research, "expert opinion" from someone without any experience in that sector, opinion pieces from similarly reputation-less people etc. but I'd say they're equally worthless as references as "According to GPT...", and should be treated similarly.
It depends on if they are just repeating things without understanding, or if they have understanding. My issue is that people that say "I asked gpt" is that they often do not have any understanding themselves.
Copy and pasting from ChatGPT has the same consequences as copying and pasting from StackOverflow, which is to say you're now on the hook supporting code in production that you don't understand.
the first 2 bullet points give you an array of answers/comments helping you cross check (also I'm a freak, and even on SO, I generally click on the posted documentation links).
"I asked X and it said..." is an appeal to authority and suspect on its face whether or not X is an LLM. But when it's an LLM, then it's even worse. Presumably, the reason for the appeal is because the person using it considers the LLM to be an authoritative or meaningful source. That makes me question the competence of the person saying it.
> Something that really frustrates me about interacting with
Something that frustrates me with LLMs is that they are optimized such that errors are as silent as possible.
It is just bad design. You want errors to be as loud as possible. So they can be traced and resolved. On the other hand, LLMs optimize human preference (or some proxy of this). While humans prefer accuracy, it would be naive to ignore all the other things that optimize this objective. Specifically, humans prefer answers that they don't know are wrong over those that they do know are wrong.
This doesn't make LLMs useless but certainly it should strongly inform how we use them. Frankly, you cannot trust outputs, so you have to verify. I think this is where there's a big divergence between LLM users (and non-users). Those that blindly trust and those that don't (extreme case is non-users). If you need to constantly verify AND recognize that verification is extra hard (because it is optimized to be invisible to you), it can create extra work, not less.
It really is two camps and I think it says a lot:
- "Blindly" trust
- "Trust" but verify
Wide range of opinions in these two camps, but I think it comes down to some threshold of default trust or default suspicion.
This happens to me all the time at work. People have turned into frontends for LLM, even when it's their job to know the answer to these types of questions. We're talking technical leads.
Seems like if all you do is forward questions to LLMs, maybe you CAN be replaced by a LLM.
There was a brief period of time in the first couple weeks of ChatGPT existing where people did this all the time on Hacker News and were upvoted for it. I take pride in the fact that I thought it was cringeworthy from the start.
I find that only acceptable (only little annoying) when this is some lead in case we're we have no idea what could be the issue, it might help to brainstorm and note that this is not verified information is important.
most annoying is when people trust chatgpt more that experts they pay. we had case when our client asked us for some specific optimization, and we told him that it makes no sense, then he asked the other company that we cooperate with and got similar response, then he asked chatgpt and it told him it's great idea. And guess what, he bought $20k subscription to implement it.
I do this occasionally when it's time sensitive, and I cannot find a reasonable source to read. e.g., "ChatGPT says cut the blue wire, not the red one. I found the bomb schematics it claims say this, but they're paywalled."
If that's all the available information and you're out of time, you may as well cut the blue wire. But, pretty much any other source is automatically more trustworthy.
I had someone at work lead me down a wild goose chase because claude told them to do something which was outright wrong to solve some performance issues they were having in their app. I helped them do this migration and it turned put that claude’s suggestions made performance worse! I know for sure the time wasted on this task was not debited from the so called company productivity stats that come from AI usage.
I do this, but it’s because I am evangelizing proper use of the tool to developers who don’t always understand what it can and can’t do.
Recently I used o3 to plan a refactoring related to upgrading the version of C++ we are using in our product. It pointed out that we could use a tool built in to VS 2022 to make a particular change automatically based on compilation output. I was not familiar with this tool and neither were the other developers on the team.
I did confirm its accuracy myself, but also made sure to credit the model as the source of information about the tool.
Wow that's a wildly cynical interpretation of what someone is saying. Maybe it's right, but I think it's equally likely that people are saying that to give you the right context.
If they're saying it to you, why wouldn't you assume they understand and trust what they came up with?
Do you need people to start with "I understand and believe and trust what I'm about to show you ..."?
I do not need people to lead on that. That’s precisely why leading on “I asked ChatGPT and it said…” makes me trust something less — the speaker is actively assigning responsibility for what’s to come to some other agent, because for one reason or another, they won’t take it on themselves.
I can see why this would be frustrating, but it's probably a good thing to have people be curious and consult an expert system.
Current systems are definitely flawed (incomplete, biased, or imagined information), but I'd pick the answers provided by Gemini over a random social post, blog page, or influencer every time.
The solution is simple. Before submitting a security report, the reporter must escrow $10 which is awarded to the reviewer if the submission turns out to be AI slop.
Of course you can say that when people are obviously copy and pasting AI slop, but if someone used AI in order to find a valid security issue, and reported it effectively, how would they know that AI was even involved?
There is or at various times was, nitter for twitter, Invidious for youtube, Imginn for instagram, and even many variations of ones for hackernews like hckrnews.com & ones that are lighter, work better in terminals, etc.
Anything for linkedin, a light interface that doesn't required logging in?
I pretty much stopped going to linkedin years ago because they started aggressively directing a person to login. I was shocked this post works without login. I don't know if that is how it has always been, or if that is a recent change, or what. It would be nice to have alternative interfaces.
In case some people are getting gated here is their post:
===
Daniel Stenberg
curl CEO. Code Emitting Organism
That's it. I've had it. I'm putting my foot down on this craziness.
1. Every reporter submitting security reports on #Hackerone for #curl now needs to answer this question:
"Did you use an AI to find the problem or generate this submission?"
(and if they do select it, they can expect a stream of proof of actual intelligence follow-up questions)
2. We now ban every reporter INSTANTLY who submits reports we deem AI slop. A threshold has been reached. We are effectively being DDoSed. If we could, we would charge them for this waste of our time.
We still have not seen a single valid security report done with AI help.
I don’t think there exists any alternative frontend for LinkedIn.
LinkedIn actually just lack week started demanding I upload ID to be able to log in…... which I’m not going to do, so LinkedIn content is effectively inaccessible to me even with an account.
Don’t you see? The boy who cried wolf was just using the wrong terminology, he was exhaustively probing the village’s disaster recovery methods. I wonder what other fables can receive such a helpful reframing.
I am more interested about the why than on bashing AI-based code analyzers. Without checking, I am sure that AI will be able to find all sorts of vulnerabilities in my untested, unpublished week-end projects.
What in curl makes AI-based analysis completely ineffective?
The more positive take, and I think the biggest reason is that curl is just well made. But along the way, it most likely uses plenty of code analysis tools: static analysis, testing, coverage, fuzzing,... the classic. And I am sure these tools catch bugs before they are published. Is there an overlap between one of these tools and AI, can one substitute for the other?
Another possibility is that curl is "weird" enough to throw off AI-based code analysis. We won't change curl for that reason, but it may be good to know.
And yeah, it may just be that AI just sucks but only looking at one side of the equation is not very productive I think.
The article mentions spam and AI slop, it is a problem for sure, but the claim here is much stronger than "stop spamming me", it is "AI never worked". And I find it a bit surprising, because when I introduce an new category of tool on some code base I work with, AI or not, I almost always find at least a problem or two.
I'm pretty sure it's your "more positive take". It's just a mature project which many, many competent eyeballs analyzing and securing it, and correspondingly many, many more incompetent eyeballs looking to make a quick bug bounty.
> Is there an overlap between one of these tools and AI, can one substitute for the other?
AI is a crude facsimile of any tool, which is both why it's useful and why it's ineffective. In the case linked from the post, it's hallucinating function names and likely hallucinating the entire patch. This hallucination would be an annoyance for the submitter using an AI tool to discover potential security vulnerabilities, and is both an annoyance and waste of time for the maintainer who was given the hallucination in bad faith.
It's probably a net positive that ChatGPT isn't going around detecting zero day vulnerabilities. We should really be saving those for the state actors to find.
Shame they need to put up with that spam. However, every big open source project has by now had good contributions with "AI help". Many millions of developers are using AI a little as a tool, like Google.
And that increase in LLM usage has resulted in an enormous increase of code duplications and code churn in said open source projects. Any benefit from new features implemented by LLMs is being offset by the tech debt caused by duplication and the maintenance burden of constantly reverting bad code (i.e. churn).
Yes. The internet has also created a ton of email spam but I wouldn't say "we've never seen a single valid contribution to our project that had internet help". Many millions of developers are using AI. Sometimes in a good way. When that results in a good MR, they likely don't even mention they used Google, or stackoverflow, or AI, they just submit.
I unironically can't remember a single case where AI managed to find a vulnerability in an open source project.
And most contributions with 'AI help' tend to not follow the code practices of the code base itself, while also in general generating worse code.
Also, just like in HTTP stuff 'if curl does it its probably right', I'm also tend to think that 'if the curl team says something its bullshit its probably bullshit'.
You wouldn't say "the Google search engine contributed to an open source project". Similarly, many millions of developers are using AI. Sometimes in a good way. When that results in a good MR, they likely don't even mention they used Google, or stackoverflow, or AI, they just submit.
I handle reports for a one million dollar bug bounty program.
AI spam is bad. We've also never had a valid report from an by an LLM (that we could tell).
People using them will take any being told why a bug report is not valid, questions, or asks for clarification and run them back through the same confused LLM. The second pass through generates even deeper nonsense.
It's making even responding with anything but "closed as spam" not worth the time.
I believe that one day there will be great code examining security tools. But people believe in their hearts that that day is today, and that they are riding the backs of fire breathing hack dragons. It's the people that concern me. They cannot tell the difference between truth and garbage.
>It's the people that concern me. They cannot tell the difference between truth and garbage.
Suffice to say, this statement is an accurate assessment of the current state of many more domains than merely software security.
This has been going for years before AI - they say we live in a "post-truth society". The generation and non-immediate-rejection of AI slop reports could be another manifestation of post-truth rather than a cause of it.
> I believe that one day there will be great code examining security tools.
As for programming, I think that we will simply continue to have incrementally better tools based on sane and appropriate technologies, as we have had forever.
What I'm sure about is that no such tool can come out of anything based on natural language, because it's simply the worst possible interface to interact with a computer.
people have been trying various iterations of "natural language programming" since programming languages were a thing. Even COBOL was supposed to be more natural than other languages of the era.
https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...
This sounds more like an influx of scammers than security researchers leaning too hard on AI tools. The main problem is the bounty structure. And I don’t think these influx of low quality reports will go away, or even get any less aggressive as long as there is money to attract the scammers. Perhaps these bug bounty programs need to develop an automatic pass/fail tester of all submitted bug code, to ensure the reporter really found a bug, before the report is submitted to the vendor.
It's unfortunately widespread. We don't offer bug bounties, but we still get obviously LLM-generated "security reports" which are just nonsense and waste our time. I think the motivation may be trying to get credit for contributing to open source projects.
Simply charge a fee to submit a report. At 1% of the payment for low bounties it's perfectly valid. Maybe progressively scale that down a bit as the bounty goes up. But still for a $50k bounty you know is correct it's only $500.
12 replies →
> I believe that one day there will be great code examining security tools.
Based on current state, what makes you think this is given?
The improvement history of tools beside LLMs, I suspect. First we had syntax highlighting, and we were wondered. Now we have fuzzers and sandbox malware analysis, who knows what the future will bring?
> They cannot tell the difference between truth and garbage.
I honestly think that in this context, they don't care - they put in essentially zero effort on the minuscule chance that you'll pay out something.
It's the same reason we have spam. The return rates are near zero, but so is the effort.
For those of you who don't want to click into linked in, https://hackerone.com/reports/3125832 is the latest example of a invalid curl report
This is interesting because they've apparently made a couple thousand dollars reporting things to other companies. Is it just a case of a broken clock being right twice a day? Seems like a terrible use of everyone's time and money. I find it hard to believe a random person on the internet using ChatGPT is worth $1000.
There are places that will pay bounties on even very flimsy reports to avoid the press / perception that they aren't responding to researchers. But that's only going to remain as long as a very small number of people are doing this.
It's easy for reputational damage to exceed $1'000, but if 1000 people do this...
1 reply →
$1000 is cheap... The real question is when will companies become wise to this scam?
Most companies make you fill in expense reports for every trivial purchase. It would be cheaper to just let employees take the cash - and most employees are honest enough. However the dishonest employee isn't why they do expense reports (there are other ways to catch dishonest employees). There used to be a scam where someone would just send a bill for "services" and those got paid often enough until companies realized the costs and started making everyone do the expense reports so they could track the little expenses.
Can someone explain the ip address in the hackerone profile[0]? I can't tell if 139.224.130.174 is a reference to something real or just hallucinated by the LLM to look "cool". Wikipedia says that this /8 is controlled by "MIX"[1] but my google-fu is failing me atm.
[0] https://hackerone.com/evilginx?type=user [1] https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addre...
Per WHOIS, it's assigned to Alibaba Cloud (could be a VM there):
You can tell it's ChatGPT from the stupid icon. In one of the iterations they started using thses emojis which are disturbing for me. The answer to the first question has obvious ChatGPT writing style.
Daniel posting about the LinkedIn post: https://mastodon.social/@bagder/114455578286549482
Recent toots on account has the news as well
Good god did they hallucinate the segmentation fault and the resulting GDB trace too? Given that the diffs don’t even apply and the functions don’t even exist, I guess the answer is yes - in which case, this is truly a new low for AI slop bug reports.
The git commit hashes in the diff are interesting: 1a2b3c4..d4e5f6a
I think my wetware pattern-matching brain spots a pattern there.
5 replies →
An real report would have a GDB trace that looks like that, so it isn't hard to create such a trace. Many of us could create a real looking GDB trace just as well by hand - it would be tedious, boring, and pointless but we could.
2 replies →
Not sure what timeline this is anymore where a tech website loads up a completely blank page on my mobile device.
Welcome to the web in 2025, where it takes 5MB of JS and everything else to load a blog post containing 640B of text.
If I wanted to slip a vulnerability into a major open source project with a lot of eyes on it, using AI to DDOS their vulnerability reports so they're less likely to find a real report from someone who caught me seems like an obvious (and easy) step.
Looking at one of the bogus reports, it doesn't even seem like a real person. Why do this if you're not trying to gain recognition?
> Why do this if you're not trying to gain recognition?
They're doing it for money, a handful of their reports did result in payouts. Those reports aren't public though, so there's no way to know if they actually found real bugs or the reviewer rubber-stamped them without doing their due diligence.
It should be called "Denial of Attention" attack!
Reading the straw that broke the camel's back commit illustrates the problem really well: https://hackerone.com/reports/3125832 . This shit must be infuriating to dig through.
I wonder if reputation systems might work here - you could give anyone who id's with an AML/KYC provider some reputation, enough for two or three reports, let people earn reputation digging through zero rep submissions and give someone like 10,000 reputation for each accurate vulnerability found, and 100s for any accurate promoted vulnerabilities. This would let people interact anonymously if they want to edit, quickly if they found something important and are willing to AML/KYC, and privilege quality people.
Either way, AI is definitely changing economics of this stuff, in this case enshittifying first.
there is a reputation system already. according to hackerone reputation system, it is a credible reporter. it's really bad
The vast majority of developers are 10-100x more likely to find a security hole in a random tool than spend time improving their reputation on a bug bounty site that pays < 10% their salary.
That makes it extremely hard to build a reputation system for a site like that. Almost all the accounts are going to be spam, and the highest quality accounts are going to freshly created and take ~ 1 action on the platform.
Or a deposit system: pay 2€ for a human to read this message, you'll get it back if it's not spam
What if the human marks it as spam but you're actually legit? Deposit another 2€ to have the platform (like Hackerone or whichever you're reporting via) give a second opinion, you'll get the 4€ back if you weren't spamming. What to do with the proceeds from spammers? The first X euros of spam reports go to upkeep of the platform, the rest to a good cause defined by the projects to whom the reports were submitted because they were the ones who had to deal with reading the slop so they get at least this much out of it
Raise deposit cost so long as slop volume remains unmanageable
This doesn't discriminate against people who aren't already established, but it may be a problem if you live in a low-income country and can't easily afford 20€ (assuming it ever gets to that deposit level). Perhaps it wouldn't work, but it can first be trialed at a normal cost level. Another concern is anonymity and payment. We hackers are often a paranoid lot. One can always support cash in the mail though, the sender can choose whether their privacy is worth a postage stamp
Reputation systems for this kind of thing sounds like rubbing some anti-itch cream on bullet wound. I feel like the problem seems to me to be behavior, not a technology issue.
Personally I can't imagine how miserable it would be for my hard-earned expertise to be relegated to sifting through SLOP where maybe 1 in hundreds or even thousands of inquiries is worth any time at all. But it also doesn't seem prudent to just ignore them.
I don't think better ML/AI technology or better information systems will make a significant difference on this issue. It's fundamentally about trust in people.
I consider myself a left leaning soyboy, but this could be the outcome of too "nice" of a discourse. I won't advocate for toxicity, but I am considering if we bolster the self-image of idiots when we refuse to call them idiots. Because you're right, this is fundamentally a people problem, specifically we need people to filter this themselves.
I don't know where the limit would go.
4 replies →
> I feel like the problem seems to me to be behavior, not a technology issue.
To be honest, this has been a grimly satisfying outcome of the AI slop debacle. For decades, the general stance of tech has been, “there is no such thing as a behavioral/social problem, we can always fix it with smarter technology”, and AI is taking that opinion and drowning it in a bathtub. You can’t fix AI slop with technology because anything you do to detect it will be incorporated into better models until they evade your tests.
We now have no choice but to acknowledge the social element of these problems, although considering what a shitshow all of Silicon Valley’s efforts at social technology have been up to now, I’m not optimistic this acknowledgement will actually lead anywhere good.
4 replies →
I guess I'm confused by your position here.
> I feel like the problem seems to me to be behavior, not a technology issue.
Yes, it's a behavior issue, but that doesn't mean it can't be solved or at least minimized by technology, particularly as a technology is what's exacerbating the issue?
> It's fundamentally about trust in people.
Who is lacking trust in who here?
1 reply →
IMO, this AI crap is just the next step of the "let's block criminal behavior with engineering" path we followed for decades. That might very well be the last straw, as it is very unlikely we can block this one efficiently and reliably.
It's due time we ramp-up our justice systems to make people truly responsible and punished for their bad behavior online, including all kind of spams, scams, fishing and disinformation.
That might involve the end of anonymity on internet, and lately I feel that the downsides of that are getting smaller and smaller compared to it's upsides.
Didn't even have to click through to the report in question to know it would be all hallucinations -- both the original patchfile and the segfault ("ngtcp2_http3_handle_priority_frame".. "There is no function named like this in current ngtcp2 or nghttp3.") I guess these guys don't bother to verify, they just blast out AI slop and hope one of them hits?
Reminds me of when some LLM (might have been Deepseek) told me I could add wasm_mode=True in my FastHTML python code which would allow me to compile it to WebAssembly, when of course there is no such feature in FastHTML. This was even when I had provided it full llms-ctx.txt
I had Google's in-search "AI" invent a command line switch that would have been very helpful... if it existed. Complete with usage caveats and warnings!
This was like two weeks ago. These things suck.
5 replies →
>"ngtcp2_http3_handle_priority_frame"
I wonder if you could use AI to classify the probability factor that something is AI bullshit and deprioritize it?
AI red tape.
> I guess these guys don't bother to verify, they just blast out AI slop and hope one of them hits?
Yes. Unfortunately, some companies seem to pay out the bug bounty without even verifying that the report is actually valid. This can be seen on the "reporter"'s profile: https://hackerone.com/evilginx
Considering that even the reporter responded to requests for clarification with yet another AI slop, they likely lack the technical background.
"they likely lack the ethical background."
FTFY
A prominent project in which people have a stake in seeing bugs fixed can afford to charge a refundable deposit against reporters.
Say, $100.
If your report is true, or even if it is incorrect but honestly mistaken, you get your $100 back.
If it is time-wasting slop with hallucinated gdb crash traces, then you don't get your money back (and so you don't pay the deposit in the first place, and don't send such a report, unless you're completely stupid, or too rich to care about $100).
If AI slopsters have to pay to play, with bad odds and no upside, they will go elsewhere.
> evilginx updated the severity from none to high
Well the reporter in the report that stated it that they are open for employment https://hackerone.com/reports/3125832 Anyone want to hire them? They can play with ChatGPT all day and spam random projects with the AI slop.
Growth hack: hire this person to find vulnerabilities in competitors' products.
Effective altruism: hire this guy to manipulate software company's stock prices with highly publicized "vulnerabilities" in their products...
I can imagine that most LLMs, if you ask it to find a security vulnerability in a given piece of code, will make something up completely out of the air. I've (mistakenly) sent valid code with an unrelated error and to this day I get nonsense "fixes" for these errors.
This alignment problem between responding with what the user wants (e.g. a security report, flattering responses) and going against the user seems a major problem limiting the effectiveness of such systems.
Counterpoint we have a CVE attributable to ours and I suspect the difference is my co-founder was an offensive kernel researcher so our system is tuned for this in a way your average...ambulance chaser is unable to do.
https://blog.bismuth.sh/blog/bismuth-found-the-atop-bug
https://www.cve.org/CVERecord?id=CVE-2025-31160
The amount of bad reports curl in particular has gotten is staggering and it's all from people who have no background just latching onto a tool that won't elevate them.
Edit: Also shoutout to one of our old professors Brendan Dolan-Gavitt who now works on offensive security agents who has a highly ranked vulnerability agent XBOW.
https://hackerone.com/xbow?type=user
So these tools are there and doing real work its just there are so many people looking for a quick buck that you really have to tease the noise from the bs.
I would try to find a better example than CVE-2025-31160. If you ask me, this kind of 'vulnerability' is CVE spam.
Except if you read the blog post we helped a very confused maintainer when they had this dropped on them with no explanation on hacker news except "oooh potential scary heap vuln"
Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!
I recently had this happen from a senior engineer. What's really frustrating is I TOLD them the issues and how to fix it. Instead of listening to what I told them, they plugged it into GPT and responded with "Oh, interesting this is what GPT says" (Which, spoiler, was similar but lacking from what I'd said).
Meaning, instead of listening to a real-life expert in the company telling them how to handle the problem they ignored my advice and instead dumped the garbage from GPT.
I really fear that a number of engineers are going to us GPT to avoid thinking. They view it as a shortcut to problem solve and it isn't.
>They view it as a shortcut to problem solve and it isn't
Oh but it is, used wisely.
One: it's a replacement for googling a problem and much faster. Instead of spending half an hour or half a day digging through bug reports, forum posts, and stack overflow for the solution to a problem. LLMs are a lot faster, occasionally correct, and very often at least rather close.
Two: it's a replacement for learning how to do something I don't want to learn how to do. Case Study: I have to create a decent-enough looking static error page for a website. I could do an awful job with my existing knowledge, I could spend half a day relearning and tweaking CSS, elements, etc. etc. or I could ask an LLM to do it and then tweak the results. Five minutes for "good enough" and it really is.
LLMs are not a replacement for real understanding, for digging into a codebase to really get to the core of a problem, or for becoming an expert in something, but in many cases I do not want to, and moreover it is a poor use of my time. Plenty of things are not my core competence or anywhere near the goals I'm trying to achieve. I just need a quick solution for a topic I'm not interested in.
2 replies →
I wonder if this is an indication that they didn't really understand what you said to begin with.
If I had a dollar for every time I told someone how to fix something and they did something else...
Let's just say not listening to someone and then complaining that doing something else didn't work isn't exactly new.
I often do this - ask a LLM for an answer when I already have it from an expert. I do it to evaluate the ability of the LLM. Usually not in the presence of said expert tho.
1 reply →
Is it possible that what happened was an impedance mismatch between you and the engineer such that they couldn’t grok what you told them but ChatGPT was able to describe it in a manner they could understand? Real-life experts (myself included, though I don’t claim to be an expert in much) sometimes have difficulty explaining domain-specific concepts to other folks; it’s not a flaw in anyone, folks just have different ways of assembling mental models.
7 replies →
Those people weren't engineers to start with.
2 replies →
You should ask yourself why this organization wants engineering advice from a chatbot more than from you.
I doubt the reason has to do with your qualities as an engineer, which must be basically sound. Otherwise why bother to launder the product of your judgment, as you described here someone doing?
> I really fear that a number of engineers are going to us GPT to avoid thinking. They view it as a shortcut to problem solve and it isn't.
How is this sentiment not different from my grandfather’s sentiment that calculators and computers (and probably his grandfather’s view of industrialization) are a shortcut to avoid work? From my perspective most tools are used as a shortcut to avoid work; that’s kinda the while point—to give us room to think about/work on other stuff.
9 replies →
It is supremely annoying when i ask in a group if someone has experience with a tool or system and some idiot copies my question into some LLM and paste the answer. I can use the LLM just like anyone, if i'm asking for EXPERIENCE it is because I want the opinion of a human who actually had to deal with stuff like corner cases.
It's the 2025 version of lmgtfy.
9 replies →
If it's not worth writing, it's not worth reading.
15 replies →
I work in a corporate environment as I’m sure many others do. Many executives have it in their head that LLMs are this brand new efficiency gain they can pad profit margins with, so you should be using it for efficiency. There’s a lot of push for that, everywhere where I work.
I see email blasts suggesting I should be using it, I get peers saying I should be using it, I get management suggesting I should use it to cut costs… and there is some truth there but as usual, it depends.
I, like many others, can’t be asked to take on inefficiency in the name of efficiency ontop of currently most efficient ways to do my work. So I too say “ChatGPT said: …” because I dump lots of things into it now. Some things I can’t quickly verify, some things are off, and in general it can produce far more information than I have time to check. Saying “ChatGPT said…” is the current CYA caveat statement around the world of: use this thing but also take liability for it. No, if you practically mandate I use something, the liability falls on you or that thing. If it’s a quick verify I’ll integrate it into knowledge. A lot of things aren’t.
> I see email blasts suggesting I should be using it, I get peers saying I should be using it, I get management suggesting I should use it to cut costs
The ideal scenario: you write a few bulletpoints and ask Copilot to turn it into a long-form email to send out. Your receiving coworker then asks Copliot to distill it back into a few bullet points they can skim.
You saved 5 minutes but one of your points was ignored entirely and 20% of your output is nonsensical.
Your coworker saved 2 minutes but one of their bulletpoints was hallucinated and important context is missing from the others.
Microsoft collects a fee from both of you and is the only winner here.
It just feels to me like a boss walking into a car mechanic's shop holding some random tool, walking up to a mechanic, and:
"Hey, whatcha doin?"
"Oh hi, yea, this car has a slight misfire on cyl 4, so I was just pulling one of the coilpacks to-"
"Yea alright, that's great. So hey! You _really_ need to use this tool. Trust me, it's gonna make your life so much easier"
"umm... that's a 3d printer. I don't really think-"
"Trust me! It's gonna 10x your work!"
...
I love the tech. It's the evangelists that don't seem to bother researching the tech beyond making an account and asking it to write a couple scripts that bug me. And then they proclaim it can replace a bunch of other stuff they don't/haven't ever bothered to research or understand.
Seriously. Being able to look up stuff using AI is not unique. I can do that too.
This is kind of the same with any AI gen art. Like I can go generate a bunch of cool images with AI too, why should I give a shit about your random Midjourney output.
Comfyui workflows, fine-tuning models, keeping up with the latest arxiv papers, patching academic code to work with generative stacks, this stuff is grueling.
Here's an example https://files.meiobit.com/wp-content/uploads/2024/11/22l0nqm...
Being dismissive of AI art is like those people who dismiss electronic music because there's a drum machine.
Doing things well still requires an immense amount of skill and exhaustive amount of effort. It's wildly complicated
5 replies →
I mean… I have a fancy phone camera in my pocket too, but there are photographers who, with the same model of fancy phone camera, do things that awe and move me.
It took a solid hundred years to legitimate photography as an artistic medium, right? To the extent that the controversy still isn’t entirely dead?
Any cool images I ask AI for are going to involve a lot less patience and refinement than some of these things the kids are using AI to turn out…
For that matter, I’ve watched friends try to ask for factual information from LLMs and found myself screaming inwardly at how vague and counterproductive their style of questioning was. They can’t figure out why I get results I find useful while they get back a wall of hedging and waffling.
2 replies →
How can you be so harsh on all the new kids with Senior Prompt Engineer in their job titles?
They have to prove to someone that they're worth their money. /s
I had to deal with someone who tried to check in hallucinated code with the defense "I checked it with chatGPT!"
If you're just parroting what you read, what is it that you do here?!
I hope you dealt with them by firing them.
1 reply →
Manage people?
2 replies →
As much as I'm also annoyed by that phrase, is it really any different from:
- I had to Google it...
- According to a StackOverflow answer...
- Person X told me about this nice trick...
- etc.
Stating your sources should surely not be a bad thing, no?
It is not about stating a source, the bad thing is treating chatGPT as an authoritative source like it is a subject matter expert.
6 replies →
In general those point to the person's understanding being shallow. So far when someone says "GPT said..." it is a new low in understanding, and there is no more to the article they googled or second stackOverflow answer with a different take on it, it is the end of the conversation.
Well, it is not, but the three "sources" you mention are not worth much either, much like ChatGPT.
8 replies →
All three of those should be followed by "...and I checked it to see if it was a sufficient solution to X..." or words to that effect.
The complaint isn't about stating the source. The complaint is about asking for advice, then ignoring that advice. If one asks how to do something, get a reply, then reply to that reply 'but Google says', that's just as rude.
It's a "source" that cannot be reproduced or actually referenced in any way.
And all the other examples will have a chain of "upstream" references, data and discussion.
I suppose you can use those same phrases to reference things without that, random "summaries" without references or research, "expert opinion" from someone without any experience in that sector, opinion pieces from similarly reputation-less people etc. but I'd say they're equally worthless as references as "According to GPT...", and should be treated similarly.
It depends on if they are just repeating things without understanding, or if they have understanding. My issue is that people that say "I asked gpt" is that they often do not have any understanding themselves.
Copy and pasting from ChatGPT has the same consequences as copying and pasting from StackOverflow, which is to say you're now on the hook supporting code in production that you don't understand.
14 replies →
the first 2 bullet points give you an array of answers/comments helping you cross check (also I'm a freak, and even on SO, I generally click on the posted documentation links).
I agree wholeheartedly.
"I asked X and it said..." is an appeal to authority and suspect on its face whether or not X is an LLM. But when it's an LLM, then it's even worse. Presumably, the reason for the appeal is because the person using it considers the LLM to be an authoritative or meaningful source. That makes me question the competence of the person saying it.
Something that frustrates me with LLMs is that they are optimized such that errors are as silent as possible.
It is just bad design. You want errors to be as loud as possible. So they can be traced and resolved. On the other hand, LLMs optimize human preference (or some proxy of this). While humans prefer accuracy, it would be naive to ignore all the other things that optimize this objective. Specifically, humans prefer answers that they don't know are wrong over those that they do know are wrong.
This doesn't make LLMs useless but certainly it should strongly inform how we use them. Frankly, you cannot trust outputs, so you have to verify. I think this is where there's a big divergence between LLM users (and non-users). Those that blindly trust and those that don't (extreme case is non-users). If you need to constantly verify AND recognize that verification is extra hard (because it is optimized to be invisible to you), it can create extra work, not less.
It really is two camps and I think it says a lot:
Wide range of opinions in these two camps, but I think it comes down to some threshold of default trust or default suspicion.
This happens to me all the time at work. People have turned into frontends for LLM, even when it's their job to know the answer to these types of questions. We're talking technical leads.
Seems like if all you do is forward questions to LLMs, maybe you CAN be replaced by a LLM.
There was a brief period of time in the first couple weeks of ChatGPT existing where people did this all the time on Hacker News and were upvoted for it. I take pride in the fact that I thought it was cringeworthy from the start.
I find that only acceptable (only little annoying) when this is some lead in case we're we have no idea what could be the issue, it might help to brainstorm and note that this is not verified information is important.
most annoying is when people trust chatgpt more that experts they pay. we had case when our client asked us for some specific optimization, and we told him that it makes no sense, then he asked the other company that we cooperate with and got similar response, then he asked chatgpt and it told him it's great idea. And guess what, he bought $20k subscription to implement it.
I do this occasionally when it's time sensitive, and I cannot find a reasonable source to read. e.g., "ChatGPT says cut the blue wire, not the red one. I found the bomb schematics it claims say this, but they're paywalled."
If that's all the available information and you're out of time, you may as well cut the blue wire. But, pretty much any other source is automatically more trustworthy.
> when this is some lead in case we're we have no idea what could be the issue
English please
2 replies →
I had someone at work lead me down a wild goose chase because claude told them to do something which was outright wrong to solve some performance issues they were having in their app. I helped them do this migration and it turned put that claude’s suggestions made performance worse! I know for sure the time wasted on this task was not debited from the so called company productivity stats that come from AI usage.
I do this, but it’s because I am evangelizing proper use of the tool to developers who don’t always understand what it can and can’t do.
Recently I used o3 to plan a refactoring related to upgrading the version of C++ we are using in our product. It pointed out that we could use a tool built in to VS 2022 to make a particular change automatically based on compilation output. I was not familiar with this tool and neither were the other developers on the team.
I did confirm its accuracy myself, but also made sure to credit the model as the source of information about the tool.
Wow that's a wildly cynical interpretation of what someone is saying. Maybe it's right, but I think it's equally likely that people are saying that to give you the right context.
If they're saying it to you, why wouldn't you assume they understand and trust what they came up with?
Do you need people to start with "I understand and believe and trust what I'm about to show you ..."?
I do not need people to lead on that. That’s precisely why leading on “I asked ChatGPT and it said…” makes me trust something less — the speaker is actively assigning responsibility for what’s to come to some other agent, because for one reason or another, they won’t take it on themselves.
Thanks for this. It's a great response I intend to use going forward.
the problem is that when you ask a ChatBot something, it always gives you an answer...
I can see why this would be frustrating, but it's probably a good thing to have people be curious and consult an expert system.
Current systems are definitely flawed (incomplete, biased, or imagined information), but I'd pick the answers provided by Gemini over a random social post, blog page, or influencer every time.
The solution is simple. Before submitting a security report, the reporter must escrow $10 which is awarded to the reviewer if the submission turns out to be AI slop.
Of course you can say that when people are obviously copy and pasting AI slop, but if someone used AI in order to find a valid security issue, and reported it effectively, how would they know that AI was even involved?
There is or at various times was, nitter for twitter, Invidious for youtube, Imginn for instagram, and even many variations of ones for hackernews like hckrnews.com & ones that are lighter, work better in terminals, etc.
Anything for linkedin, a light interface that doesn't required logging in?
I pretty much stopped going to linkedin years ago because they started aggressively directing a person to login. I was shocked this post works without login. I don't know if that is how it has always been, or if that is a recent change, or what. It would be nice to have alternative interfaces.
In case some people are getting gated here is their post:
===
Daniel Stenberg curl CEO. Code Emitting Organism
That's it. I've had it. I'm putting my foot down on this craziness.
1. Every reporter submitting security reports on #Hackerone for #curl now needs to answer this question:
"Did you use an AI to find the problem or generate this submission?"
(and if they do select it, they can expect a stream of proof of actual intelligence follow-up questions)
2. We now ban every reporter INSTANTLY who submits reports we deem AI slop. A threshold has been reached. We are effectively being DDoSed. If we could, we would charge them for this waste of our time.
We still have not seen a single valid security report done with AI help.
---
This is the latest one that really pushed me over the limit: https://hackerone.com/reports/3125832
===
> Anything for linkedin, a light interface that doesn't required logging in?
I just opened the site with JS off on mobile. No issues.
I don’t think there exists any alternative frontend for LinkedIn.
LinkedIn actually just lack week started demanding I upload ID to be able to log in…... which I’m not going to do, so LinkedIn content is effectively inaccessible to me even with an account.
[flagged]
Fortunately, they can beat an automated idiot agent, which is much closer to what we have today.
[flagged]
3 replies →
Your statement is proven wrong by the article you are replying to.
The word "intelligent" is doing a lot of heavy lifting here. Theoretically yes, but maybe we're not there yet.
Did you actually read the objection? They are basically getting DDoS by spurious vuln reports. How do you "get over" that.
Don’t you see? The boy who cried wolf was just using the wrong terminology, he was exhaustively probing the village’s disaster recovery methods. I wonder what other fables can receive such a helpful reframing.
1 reply →
[flagged]
6 replies →
I am more interested about the why than on bashing AI-based code analyzers. Without checking, I am sure that AI will be able to find all sorts of vulnerabilities in my untested, unpublished week-end projects.
What in curl makes AI-based analysis completely ineffective?
The more positive take, and I think the biggest reason is that curl is just well made. But along the way, it most likely uses plenty of code analysis tools: static analysis, testing, coverage, fuzzing,... the classic. And I am sure these tools catch bugs before they are published. Is there an overlap between one of these tools and AI, can one substitute for the other?
Another possibility is that curl is "weird" enough to throw off AI-based code analysis. We won't change curl for that reason, but it may be good to know.
And yeah, it may just be that AI just sucks but only looking at one side of the equation is not very productive I think.
The article mentions spam and AI slop, it is a problem for sure, but the claim here is much stronger than "stop spamming me", it is "AI never worked". And I find it a bit surprising, because when I introduce an new category of tool on some code base I work with, AI or not, I almost always find at least a problem or two.
I'm pretty sure it's your "more positive take". It's just a mature project which many, many competent eyeballs analyzing and securing it, and correspondingly many, many more incompetent eyeballs looking to make a quick bug bounty.
> Is there an overlap between one of these tools and AI, can one substitute for the other?
AI is a crude facsimile of any tool, which is both why it's useful and why it's ineffective. In the case linked from the post, it's hallucinating function names and likely hallucinating the entire patch. This hallucination would be an annoyance for the submitter using an AI tool to discover potential security vulnerabilities, and is both an annoyance and waste of time for the maintainer who was given the hallucination in bad faith.
It's probably a net positive that ChatGPT isn't going around detecting zero day vulnerabilities. We should really be saving those for the state actors to find.
Shame they need to put up with that spam. However, every big open source project has by now had good contributions with "AI help". Many millions of developers are using AI a little as a tool, like Google.
And that increase in LLM usage has resulted in an enormous increase of code duplications and code churn in said open source projects. Any benefit from new features implemented by LLMs is being offset by the tech debt caused by duplication and the maintenance burden of constantly reverting bad code (i.e. churn).
https://arc.dev/talent-blog/impact-of-ai-on-code/
Yes. The internet has also created a ton of email spam but I wouldn't say "we've never seen a single valid contribution to our project that had internet help". Many millions of developers are using AI. Sometimes in a good way. When that results in a good MR, they likely don't even mention they used Google, or stackoverflow, or AI, they just submit.
2 replies →
I unironically can't remember a single case where AI managed to find a vulnerability in an open source project.
And most contributions with 'AI help' tend to not follow the code practices of the code base itself, while also in general generating worse code.
Also, just like in HTTP stuff 'if curl does it its probably right', I'm also tend to think that 'if the curl team says something its bullshit its probably bullshit'.
You wouldn't say "the Google search engine contributed to an open source project". Similarly, many millions of developers are using AI. Sometimes in a good way. When that results in a good MR, they likely don't even mention they used Google, or stackoverflow, or AI, they just submit.
2 replies →