Comment by lionkor
7 hours ago
Software made for one, made by LLMs which regurgitate the average of existing tools, are going to have more security issues, not less.
7 hours ago
Software made for one, made by LLMs which regurgitate the average of existing tools, are going to have more security issues, not less.
But how would you exploit them when every one of them is subtly different?
With software that's deployed to millions of computers you have an abundance of targets, but trying to target some random LLM average todo list at scale is hard, isn't it?
Yes, but it should be fairly easy to "simply" attack the common technologies that LLMs keep parroting. NextJS, or some Rust tools, or whatever other tools LLMs "love" using, are all great targets.
Once millions of completely unskilled developers have "workflows" that consist of asking an LLM to make a thing, followed by those LLMs pulling in the same 100 (often outdated versions of) dependencies, you have a beautiful attack vector.
Yes, it's "easy" to attack something like Obsidian. It's probably easier to attack a couple hundred dependencies LLMs like to use, or to test what LLMs commonly do to implement things from scratch, and attack those weaknesses.
We are just lucky that enough real, smart, people engineered things that actually work, are well understood, and keep us safe, like firewalls.
I am imagining some poor sod working for NSA TAO trying to hack a bespoke web microservice stack. He spends dozens of hours slaving away at the keyboard, skipping sleep and eating terrible meals at his desk, desperate to get RCE as quickly as possible, because he needs to traverse all the way to the DB layer and exfil data or his boss will pass him over for his next promotion.
At day 9, right as he is getting ready to deploy his beautifully crafted shell code, the clock hits midnight UTC. The website shuts down for maintenance.
"This is it" he thinks. "As soon as the backups finish I'm getting in. No problem."
Minutes tick by. He gets up, stretches, sits back down, watches the clock impatiently. Then, as he prepares to start refreshing the site he recollects, "I'm glad I begged so hard to get authorization to use this PHP 0day."
His partially obscured terminal window has the script ready to launch, all arguments pre-populated, waiting for the link and session token to be pasted in and executed.
The site comes back up. But the url of his launch point returns 404. Undaunted, he returns to a previous url. It is also 404. He curses aloud. Beginning to perspire, he goes to the homepage and prepares to navigate back to the launch point.
The link isn't there. Well, it's there, but it has changed.
"What the....!" The link is no longer a PHP url. He mouses over other links. NO links say PHP anymore. Starting to panic, he clicks on links at random. Not a single one appears to be PHP.
The following morning he schedules an urgent meeting with his supervisor.
"How's that project coming along. Got anything yet?"
"No. I, uh...I'm going to need a bit more time."
"Oh?"
"Yeah. Uh. The site. It got..." He mutes his microphone and, for the 22nd time since midnight, he screams in frustration. Unmuting, he continues:
"It got rewritten. Completely. In Nim."
"What??"
"Yeah. It's some esoteric language that just got a new web framework. I guess somebody decided they wanted to mess around with it. So they vibe coded a complete translation. The whole front end is nimlang now. None of the PHP attacks are going to work on it."
His supervisor expresses his disgust and ends the call.
11 days later the process repeats itself, this time with Rust.
The TAO engineer submits an application to change jobs to the DoD's procurement division, then requests an appointment with a mental health counselor.