Comment by Buttons840
7 hours ago
I'd feel a lot better if I knew some red team security researchers were going hard trying to break into critical national infrastructure, just to ensure it's not too easy, but probably this isn't happening because it would be a felony to even try. The bad guys are free to try though.
All this will tend to get blamed on AI, but AI didn't put these vulnerabilities into place. AI is just a tool that can be used to discover vulnerabilities, and the only question we have to answer is whether or not that tool should be available to regular people.
Some will try to frame the question as being about whether we should allow AI aid in cyber-security, but that's not the question. This technology isn't going away. The bad guys will have it. Nations will have it. Again, the question is, can regular people also continue to have it?
As someone who has helped to secure critical national infrastructure several years ago, I can say that it was a mess to disclose the vulnerabilities I identified, and in some cases it took years to patch them. Some of these vulnerabilities were RCEs and mass PII leaks. It's simply not worth the risk for most people - who aren't in the USA - to bother doing this sort of security research. However, if you are in the USA, there are provisions in the CFAA that enable good-faith security research to take place (https://www.justice.gov/archives/opa/pr/department-justice-a...), so to the best of my understanding, you're able to do good-faith research without being charged or prosecuted, and you should take advantage of this if you have the skills or models to do so. I am not based in the USA, and there are no laws in my country that guarantee me zero jail time for simply trying to do the right thing, so I am not incentivized to continue finding vulnerabilities in my country's infrastructure.
Edit: If other countries don't catch up and include similar provisions in their laws, they'll simply fall behind and put their citizens at risk. I'm sure every country has individuals willing to help that currently feel unsafe - look at the large amount of bug bounty hunters worldwide who, I'm sure, wouldn't mind practicing their skills by securing their country's infrastructure.
It is absolutely happening, paid by your taxpayer dollars (if you’re American).
Most major countries have this kind of programs. Those could probably be bigger and they will always miss obvious stuff but they exist, at least for CNI.
> I'd feel a lot better if I knew some red team security researchers were going hard trying to break into critical national infrastructure
They are. Most critical infrastructure providers worldwide have in house or contract red teams. Every western government has an agency dedicated to cyber defense and they all have red team programs.
AI is actually a big reason why there are so many vulnerabilities over the past year and a half. People don't even know what their code bases look like anymore. It's really bad so sayeth my friends doing consulting.
A lot of companies cyber security posture is asking Claude to "find all security issues make no mistakes".
That's certainly one reason, but frontier LLMs are also really good at finding existing vulnerabilities in completely human-authored codebases.
That’s not backed by data at all. We’re discovering more existing vulnerabilities.
It's both.
I would expect code written by the latest frontier models with a "find all security issues make no mistakes" pass to be more secure than average human code written in the pre-AI era (not a high bar).
I would expect it depends on the language. Less unchecked memcpy, more forgetting to check the user is an admin.
> I would expect code written by the latest frontier models with a "find all security issues make no mistakes" pass to be more secure than average human code written in the pre-AI era (not a high bar).
"If you write code as cleverly as you can, you are, by definition, too stupid to debug it."
Do you disagree with this? Because if you don't, then the rate of bugs/RCEs in a completely AI-written-software world will be exactly the same as the rate of bugs in a completely human-written-software world.
A combination of human-authorship + AI-review, with absolutely no AI-written code is going to have the lowest rate of bugs, because there is always one entity who understands the code (because they wrote it) and one entity who is smarter than the author who can find bugs.
The AI-generated code I see from SOTA models is absolute spaghetti, and they start running into problems spotting bugs after even a small number of refinements/changes to the software (say, 50).
The problem with all this spaghetti is that they are quick to write something that even they can't easily read. Even mediocre but experienced humans are not so eager to write something that they can't easily debug.
IOW, for any non-trivial problem, with AI-generated code you are guaranteed to get spaghetti that it cannot understand. With humans, it's only a probability that that will happen.
A lot of people expect that. Apparently it's not working out so well in reality. My buddy who is big in this said something to the effect of, when a human codebase has human understanding it may have issues but it's strengthened by ai. When an ai codebase has little human understanding it's not strengthened by people very much. So his take was the most secure code bases he is seeing right now are human ones which use ai tools to help read and review rather then write. My guess is that's also cultural and industry specific. He is niched into businesses that actually care and usually doesn't take work from cowboy outfits.
imagine if we just turned off 90% of the non-critical internet for two weeks, hardened up in an absolute mad dash where everyone has access to top tier cyber models, and then afterwards we go back to business as usual. it’d be a disaster short-term but long-term would it be worse than the years of slow drip we’re about to go through? idk! I kinda think it’d be interesting to find out!
Imagine if everyone had just stayed at home for a week in 2020.
It'd take more than a week, but even now, if the whole world coordinated a simultaneous quarantine, we'd eliminate covid, the flu, the common cold, and a thousand nameless respiratory cruds.
2 replies →
Everyone would be very difficult. You'd still need some people to make sure essential services are still functioning, and that still involves quite a large number of people.
That was my first thought as well. With COVID, we recognised that WFH was viable for white collar work. Perhaps a global 2 week cleanse will allow everyone to re-evaluate life priorities outside of the internet.
I see hope in what happened during the blackout in Spain/Portugal.
6 replies →
This poem was written for people like you https://micaelnussbaumer.com/posts/the-pig-god-its-progeny-a...
Regular people are often bad guys too, you know. If the reality was "bad guys will have it. Should regular people too?" The entire core dilemma would be entirely eliminated lol.
Don't worry - Siri is on it - that's actually why they've been so lacklustre. Just too busy hacking for the good guys