Comment by friendzis

What a serious security model for a meatbag agent looks like? No, but seriously, an admin in a small org is a huge key-person risk in that they (or their stolen creds) can wipe enough and quick enough to effectively disable the business altogether.

More security conscious admins will at least segment their creds and implement four eyes principles somewhere, but were are back at square one of "asking user for confirmation".

Larger orgs, even if by necessity, segment their human agents, their creds and plaster four eyes principle liberally. But this relies on safeguards against agents colluding and ignoring some inputs, which sounds a bit scary for artificial agents.

Say you implement some swarm of agents, where access-enabled sub-agents are extremely restricted with system prompts and some access filtering. Then none of the agents in the swarm should be able to spawn themselves, otherwise a rogue agent can overwrite any safeguards. That, again, leaves the user with manually approving/denying network requests / hosts / sessions.

While I don't like anthropomorphising LLMs, the problem domain seems quite damn close to that of a key person going rogue within an org. The general solution seems to be liberal amounts of trust and ~~sweet compensation~~ gaslighting about replaceability.