Comment by no_multitudes
12 hours ago
I don't understand the distinction you're drawing between this and a real prompt injection attack. Would true prompt injection require that the LLM "intentionally" launches the malware, rather than unintentionally?
Yes. The point with this attack is that you can't really avoid it by 'aligning' the LLM any better, nor really by filtering the actions it takes better.
Part of the attack happens via the readme in the zip file, which is something the agent reads and follows (or better said in this attack, it does explicitly not follow those instructions for safety reasons, but decides to do something else).
Plausibly the auto mode classifier could catch the potential module shadowing attack and deny execution of Python from the untrusted directory.