Comment by pixl97
5 hours ago
The issue is with the nature of agentic systems and how often they run, a .001 failure is still huge. Attackers monitor their attack chances and use the most successful attacks so the actual success on the attackers side is much closer to 1 than .001.
This is the problem, this isn't a 'failure' mode where something randomly goes wrong, like a person accidentally sticking their hand in a machine. This is war where you are under active attack and the attackers adapt quickly.
Because of the breadth of LLM capabilities they have a nearly unbound attack surface. You spend so much of your effort making sure your prompt is secure that you either massively increase costs, or you partially lobotomize your agent making it less capable.
Solutions for these problems will have to come inside the model itself because both the attackers and models are far more capable than you on doing dumb/evil shit. Until the model itself can go "hey, wait a darn minute" a new fun way of encoding the attack can always be found.
> This is war where you are under active attack and the attackers adapt quickly.
It all depends on what the use case is.
For example, consider a system which takes English questions from business users, translates them to SQL, then runs it (as the business user) against a reporting database
How does the attacker get access to it? And even if they somehow do, what harm can they actually do to it? The only tool it has is one to run SELECT against a DB, and it can only SELECT data the user already has access to, and we have timeouts/etc to protect against overly expensive queries
If your design involves lots of general purpose do-anything agents with very generic tools, the risk profile is very different from narrowly scoped purpose-specific AI services
I mean, yeah, consider a system that takes English questions and does nothing with them, it’s very secure. What’s your point? It doesn’t really refute what the commenter above you was saying.