Most people will hardly read what the LLM spits out after 3 hours of use and execute the code. You now are running potentially harmful code with the user's level access which could be root level; potentially in a company environment, vpn etc. It's really scary, because at first glance it will look 100% legitimate.
Your neural network (LLM or otherwise) could be undetectably backdoored in a way that makes it provide malicious outputs for specific inputs.
Right now nobody really trusts LLM output anyway, so the immediate harm is small. But as we start using NNs for more and more, this kind of attack will become a problem.
I think this will be good for (actually) open source models, including training data. Because that will be the only way to confirm the model isn't hijacked
It's even better if you have anything automated executing your tests and whatnot (like popular VSCode plugins showing a nice graphical view of which errors arise from where through your local repo). You could own a developer's machine before they had the time to vet the offending code.
Most people will hardly read what the LLM spits out after 3 hours of use and execute the code. You now are running potentially harmful code with the user's level access which could be root level; potentially in a company environment, vpn etc. It's really scary, because at first glance it will look 100% legitimate.
Your neural network (LLM or otherwise) could be undetectably backdoored in a way that makes it provide malicious outputs for specific inputs.
Right now nobody really trusts LLM output anyway, so the immediate harm is small. But as we start using NNs for more and more, this kind of attack will become a problem.
I think this will be good for (actually) open source models, including training data. Because that will be the only way to confirm the model isn't hijacked
But how would you confirm it if there’s no ‚reproducible build‘ and you don’t have the hardware to reproduce?
6 replies →
Supply chain attacks, I'd reckon.
Get malicious code stuffed into Cursor (or similar)-built applications -- doesn't even have to fail static scanning, just got to open the door.
Sort of like the xz debacle.
It's even better if you have anything automated executing your tests and whatnot (like popular VSCode plugins showing a nice graphical view of which errors arise from where through your local repo). You could own a developer's machine before they had the time to vet the offending code.
Yeah esp Cursor YOLO mode (auto write code and run commands) is getting very popular
https://forum.cursor.com/t/yolo-mode-is-amazing/36262
5 replies →
Yeah that would be the most obvious "real" exploit (on the code generation side)