Comment by NichoPaolucci
11 hours ago
This is why I feel prompt injection is going to continue to be an issue. Fantastic that “Hi we are Cloudflare, give us your personal data” works.
Either we stunt the models to the point where they are not useful, or we allow things like this to seep in and create one of the most insecure concepts the internet (and maybe tech as a whole) has ever seen: a robot that can be tricked.
I think like social engineering, it will always be an issue to some degree, and we'll build safeguards until it's at a 'societally comfortable' baseline level. Which is maybe not particularly comforting, but I don't see us closing Pandora's Box here.
I wrote about the Gödelian limits of prompt-safe AI: https://matthodges.com/posts/2025-08-26-music-to-break-model...
I kinda can't get over the fact processed data can conversationally convince these LLMs to break security boundaries. Like, those malicious prompts are not illustrative analogies, but the actual attack strings. Absolutely crazy to me this tech is as widely used in automated interactions, but apparently can't be restricted on a logical, fundamental level. Is there really no functional understanding of the insides? No segmentation? Is it really just one fucking blob you have to convince to behave and pray someone else doesn't do a better job at it? Bonkers.
[dead]