Comment by lcnPylGDnU4H9OF
2 days ago
That’s not really a reason not to treat the agent like it’s “rogue”. The point is, if it accepts any untrusted inputs then, from a security perspective, it is possible for any given (untrusted) input to contain a prompt injection payload that jailbreaks the model and tells it to do things it shouldn’t do.
As such, it can be told to do bad stuff in a way that can’t be prevented and therefore should not be given read access to anything you don’t want others to know about, nor write access to any data of which you care about the integrity.
That is out of scope of the service. What kind of user agent the actual user deputizes to interact with a service, is the user's own choice and responsibility. In general, it's not something a service can solve on their end.
Services can certainly make this safer by providing means to get more restricted credentials, so that users can deputize semi-trusted delegates, such as agents vulnerable to injection.
The important point being made in this discussion is that this is already a common thing with OAuth, but mostly unheard of with web sessions and cookies.