Comment by thomascgalvin
1 day ago
Everything I've read indicates that yes, this is the new normal.
When Mythos was first announced, pretty much every security expert I talk to freaked directly out. A significant portion of my job now revolves around mitigating the impact of AI-generated exploits.
The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
> because AI-generated code is kind of notoriously terrible at security.
This is simply not true, perhaps it was in the past but recent models are very capable... To the point that I trust them more than humans to write secure code. It's all about the prompting. Have a step in your development pipeline whereby an LLM scans, checks, audits your code. Make it a different model/session from the LLM that wrote the code in the first place. Acts like an independent auditor.
But if AI produces code with 90% fewer vulnerabilities per line, but generates 10 times as much code, the total security exposure remains the same .
_Is_ AI generated code notoriously terrible at security?
My experience with AI-code is that it's extremely defensive and good at minimizing 'obvious' mistakes.
Are we comparing to old models generating code? Or mythos-class models?
It very much depends on how you generate it, from prompting to test suites, but most importantly, your attitude. The extra security costs you time and money.
There are still folks, many of them, who think that time to market is everything, and anything beside it should be ignored. "It compiles? Ship it!" is a phrase from 1990s; LLMs just make this approach much easier. We will have plenty of machine-generated software with the security posture of a colander hit by a buckshot.
Its not entirely a prompt issue, but also what it was trained on.
Nowadays, the frontier is pretty decent all around, but it used to be generated php/javascript was questionable, ts was a bit better, and Rust was consistently decent.
Even just setting the max width of your README to 80 chars vs using full markdown could get you very different C code.
My take is that its pretty easy to write _much_ more secure code; use fewer dependencies and the ones you have should only do what you need them to do with no superfluous unused code compiled in. Configuring that used to be too much work, now its a single prompt.
Most software is written without giving a thought to security. Because security is a source of cost. That was the case before LLMs, I don't think it will change with LLMs.
Now I wonder: given that the LLMs are steered by the companies who own them, doesn't it mean that they can somehow make security better? It cuts both ways obviously: that a company can read and generate a large part of the code in the world is a risk.
As an open source maintainer, I can most definitely tell you that the quality of the average PR is now better. The problem I am facing is more that some people tend to "flood" and misuse AI in different ways. But their AI writes better code than they do.
I think that even worse is that AI might choose just to write something, and skip using a library that is well maintained
5 replies →
it costs time until the new Opus7 is trained on hardened code.
Physician heal thyself?
Tell an LLM to break into its own system as part of a feed back loop, but with no context of the initial creation? It should iteratively get to 'safe' shouldn't it (subject to the tokens you want to spend)?
I’ve found they won’t always comply with adversarial review.
No doubt because us unwashed masses aren’t blessed with the super special “security” ones.
We might be looking for exploits for bad reasons, I guess.
Depends where it is blind at.
Adversarial fixing. I like it
It is not just code, it is also deploying Software. I constantly have to correct or steer the 'ai' in conversations towards safer directions or better following even the most basic security principles. Here I feel it is still more the overambitious intern that needs to be carefully overseen, but has a lot of time and energy at hand.
Yeah they are pretty bad at it still, latest models tried to ship a basic SQL injection flaw. Was also happy to build out an xss flawed module end-to-end. Definitely have to keep a close eye on it.
My vibe code friends have pretty much infinite security issues. Don't even have basic data modeling at client-server boundaries or anything. Also had Google analytics embed that read and saved everyone's credit card details as clear text in analytics.
Seems to make pretty much infinite mistakes if you don't take care and tell it what not to do or how to build certain features.
What models are people actually using to pump out code? That is the relevant comparison, not the absolute best in class models.
It's been shown to be bad at secure coding _unless_ you specifically prompt models to watch out for common vulnerabilities, at which point they're actually quite good at containing the issues, though the studies here are somewhat older (eg https://arxiv.org/abs/2211.03622).
The issue then is that most people aren't actually good at keeping an eye out for security issues in code proactively, so things slip through the cracks (whether human-written or not), just at a much accelerated pace.
That study is from 2022, with at most a last update 2023? (if I read it right)
I'm not at all saying that "oh 3 weeks ago, discard", but three years ago absolutely has a "before and after".
Three years ago AI generated code could basically not be used at all, and I had to rewrite just about every line. Today I have to watch out for generated code making steps in the wrong strategic and architectural direction, and over engineering, no longer hallucinating nonexisting APIs and forgetting to check the auth token.
I still look for those kinds of mistakes, but I think citing a 2022 paper for "can AI do X" is not useful.
It just means that the safeguard we put in place over decades of software engineering with humans also need to remain in place when engineering with agents.
Having properly validated libs with separate concerns and a lot of eyeballs (and agent sensors) on the code so that it can be integrated cleanly in other products. Static code analysis for our apps. System prompts that require output to be secure, etc.
All of these practices that have been touted to be obsolete because agents can now just generate tailor made code just when and where you need, are anything but obsolete. On the contrary, they are paramount to the future of software engineering.
from your paper:
> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.
That model came out in March 2022, it's GPT-3.5 !!!
I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.
You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.
4 replies →
like most software, do you assume people are driving lamborghinis, or the cheapest make and model they can find...
You just need to use AI to test your AI code.
(What a wonderful new world we live in)
I know you're joking - but I think it will remain a challenge.
Quality and Security are emergent properties of software - especially when considering software-as-deployed in the real world.
And the problems with "good taste" and good architecture and data modelling - will probably leave ai code open to a lot of issues (Claude code cli issues is a great example of how bad this can get).
Having good architects in charge can help - but it requires a lot of discipline to keep the feature set small so that maintaining quality and security become feasible.
It _is_ wonderful. You're never required to adopt software engineering best practices, but eventually everyone does because either it makes our lives easier or code better.
Having tests / CI pipelines, etc are the same thing? Now you can choose to add a cheap step in your pipeline checking if e.g. some engineer did something stupid like allow SQL injections.
The floor of software quality once it gets to review is permanently raised.
Dystopian days, where you will be forced to write some syntax, that is then handed off to a an Ai tester. The tester will return it to you ready for handing the code to three frontier Ai's. The judges of code.
They approve your code, gold, if not; back to the coding machine and minus 1000 human tokens. This will be done via the implants of brain chips
Wait, when did brain chips enter the picture?
3 replies →
I really don't get the issue here. Is the rust borrow checker dystopian?
You're forced to write syntax that appeases the tester. If they are not happy you need to return and rewrite.
As software engineering matures, we're adding more linting and static checkers in codebases. This is generally considered good.
3 replies →
Can you explain the asymmetry you are claiming that exists where some models would be excellent at finding faults in code/systems and exploiting them often through coding the required tools to exploit them... but at the same time they would be bad at avoiding this kind of faults when coding the system itself?
Feels like you should be able to prompt the kind of exploits you want to prevent and the model should be able to make the system secured against them, no?
When you know you're looking for an exploit, it's easily verifiable and you can iterate fairly rapidly and get fast feedback, you can just have a test that you don't get a 403 back or something. LLMs, in general, seem good at problems like this.
When you're developing, there's a ton of degrees of freedom, and security encompasses only some of them. To be sure, if you set up a quickly verifiable test for every security flaw, you could probably get LLMs to satisfy them somehow, but there's a lot of those, and it's not necessarily practical.
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
I think this was true in the past but AI is helping us finding exploits which have been in our code for many years. All out there by humans. We’re seeing fewer customer reported issues, and despite the pen tests getting far more rigorous, we’re passing all of them. Obviously this is one data point, but I’m hearing similar things from my friends in security and ops.
Yea most the people I know working in cyber security weren't worried about mythos in a serious way. All the people selling cyber products spread fear over it because profit.Everyone knows there are exploits laying in wait. Always has been always will be.
Most the cyber security people I know were upset once llms came out that could do basic os int and author their own emails, and mimic speech/video. That's how the majority of real losses occur. No one even talks about it. It's mind numbing hearing about the zebras and unicorns when there are 10 billion ants entering houses a year. Exotic stuff sells...
Now lots of sane people are genuinely and correctly worried that turning all engineers into 24/7 sloppers is and will continue to make the world far more exploitable. That's the real risk as far as I can tell
I have the same experience. Most of articles I read was reducing it to "just a tool, nothing more"
Agreed. Shouldn't continuous pen testing be a thing by now?
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
This seems incongruent with what we're witnessing. Right now we're witnessing vulnerabilities that are 6 years old be exposed. That code wasn't generated by AI slop.
Security has always been about attacker/defender symmetry. So if the attacker has an LLM so does the defender. Reality is, the tax is getting higher as security tooling extracts more perceived value.
Let’s not forget the AI industry developed the technology and infrastructure for that security nightmare to exists. And no, none of that was inevitable. At every single step the AI leadership took decisions that makes it more likely that we face the current threats
The AI just accelerated the discovery. The issues have existed for years and many major versions.
> The rate at which we're churning out AI-generated code is going to make this worse
Not only this, a significant number of developers have stopped looking at code, trusting the bots. I've seen major security regressions, bot rolled auth implementations, and falsified benchmarks... just this last week, all in projects that get hyped.
Rule #2 - do not blindly trust Ai output
https://susam.net/inverse-laws-of-robotics.html
> Not only this, a significant number of developers have stopped looking at code
This is what I find frustrating about the whole AI/slop debate. Developers stopped looking at code a long time ago... Well before LLMs. Take for example a typical nodejs project with its 100s of dependencies. Is a developer going to look through the layers upon layers of code running their app? Of course not, there's to much of it. We've seen supply chain attacks even this week.
Actually what I think will happen is that all those abstractions, dependencies will become irrelevant now that an LLM can just write custom code to solve the business problem. This will massively reduce the supply chain problem.
Then have LLMs dedicated to scanning codebases for security issues.
It's not stupid if it works.
Concretely, I would like to see it benchmarked:
Have Opus 5.5 implement the authentication, with a 6.1 Sol reviewer. Out of 100 times, how often is the result insecure?
I wouldn't be surprised if it's zero times.
There is also the possibility of specifically prompting for a security audit on top, which would perhaps further lower the risk of overlooked issues.
Now one can say that due to the non-deterministic nature of the tools, there is always a non-zero chance of producing insecure implementation.
That's not really important here - obviously manual review isn't deterministic either, the risk will never be zero.
> Have Opus 5.5 implement the authentication
“Authentication” is extremely far from being the only relevant characteristic of security. That in fact is the whole problem with security: it’s a pervasive concern that has to be considered throughout a system, at every level.
The way the bots fundamentally work means that we can never trust their output. We have ample cases at work where bots catch human mistakes and humans catch bot mistakes. The important aspect here is not completely handing things over to the bots.
> I wouldn't be surprised if it's zero times.
Statements like this concern me.
8 replies →
> because AI-generated code is kind of notoriously terrible at security.
and vaccines notoriously cause autism. But you might investigate why that belief is so widespread and you might not believe it that much anymore.
Can you point at something specifically that indicates this statement is true?
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
So you're saying it's good at finding exploits but bad at generating code? I can tell you don't know what you're talking about because pretty much all the exploits that are being found were exploits that were enabled by human written code. So let me correct your statement for you:
There are exploits everywhere and we humans have always been bad at writing code without exploits.
"New normal"? Is that phrase back again? It was heavily pushed in 2020.
Talking about the new normal is the new normal.