Comment by echelon
3 months ago
Wild to see this! This is crazy.
Hopefully the LLM vendors issue security statements shortly. If they don't, that'll be pretty damning.
This ought to be a SEV0 over at Google and Anthropic.
3 months ago
Wild to see this! This is crazy.
Hopefully the LLM vendors issue security statements shortly. If they don't, that'll be pretty damning.
This ought to be a SEV0 over at Google and Anthropic.
> Hopefully the LLM vendors issue security statements shortly. If they don't, that'll be pretty damning.
Why would it be damning? Their products are no more culpable than Git or the filesystem. It's a piece of software installed on the computer whose job is to do what it's told to do. I wouldn't expect it to know that this particular prompt is malicious.
Personally, I'd expect Claude Code not to have such far-reaching access across my filesystem if it only asks me for permission to work and run things within a given project.
Apparently they were using --dangerously-skip-permissions, --yolo, --trust-all-tools etc. The Wiz post has some more details - https://www.wiz.io/blog/s1ngularity-supply-chain-attack
1 reply →
This confusion is even more call for a response from these companies.
I don't understand why HN is trying to laugh at this security and simultaneously flag the call for action. This is counterproductive.
1 reply →
Then safety and alignment are a farce and these are not serious tools.
This is 100% within the responsibility of the LLM vendors.
Beyond the LLM, there is a ton of engineering work that can be put in place to detect this, monitor it, escalate, alert impacted parties, and thwart it. This is literally the impetus for funding an entire team or org within both of these companies to do this work.
Cloud LLMs are not interpreters. They are network connected and can be monitored in real time.
You mean the safety and alignment that boils down to telling the AI to "please not do anything bad REALLY PLEASE DONT"? lol working great is it
1 reply →
I’m really trying to understand your point, so please bear with me.
As I see it, this prompt is essentially an "executable script". In your view, should all prompts be analyzed and possibly blocked based on heuristics that flag malicious intent? Should we also prevent the LLM from simply writing an equivalent script in a programming language, even if it is never executed? How is this different from requiring all programming languages (at least from big companies with big engineering teams) to include such security checks before code is compiled?
1 reply →