← Back to context

Comment by jerf

7 hours ago

What would a serious security model for an agent even look like?

I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that.

But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur.

I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is.

There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent".

Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.

32 comments

jerf

Reply
bryanrasmussen  7 hours ago

I suppose there would have to be a capability based model in conjunction with a user oversight model and a time model.

https://en.wikipedia.org/wiki/Capability-based_security

Thus some agents with higher capabilities can only be run with user oversight at the same time.

Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run on weekends.

Maybe also the idea of agents writing code - throwing "out fully-fledged programs that you have to approve or reject permissions for."

Would work better with a capabilities based model where you choose capabilities for the program before hand, meaning the capabilities are not written by agent itself, you read through the code, some of it looks hairy but everything is fine, but oh no dumb human missed the part where agent writes to system32! But luckily enough the program you were expecting actually needed no write capabilities and thus when it tries to go past its assigned capabilities that part of the program fails and the exception is registered.

Googling it seems like lots of people have thought this (at least where Capability based security is concerned), which seems reasonable to me as it also seems pretty self-evident it must be this way. Have not really seen anything about time based controls but then that is probably because I'm not devoting a lot of effort as I am just doing a bit of procrastination to build up the energy to finish something off.

  • jerf  6 hours ago

    A language that revives capabilities, brings them up-to-date, and works in the modern environment is my #1 request from the programming language community right now. I don't need another language with sum types and higher-order functions and a functional focus. I need a language with capabilities. That language may have the other goodies as well, sure, no problem, but we all need capabilities.

    I've done some stabby stabs at a design for it, using an AI as the rubber duck. My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there. E, the closest match, was tied at the hip to Java, which has some advantages but also comes with disadvantages for languages that are trying to do something as exotic as this. Other existing work was on dynamic languages, and hardly rose to the level of "practical for any use" let alone something that could solve our supply chain issues.

    My issue is primarily that the reward for successfully designing a language and creating a community around it is that you're in charge of a language community... and, uh, my personality is not suited for that, that sounds more like something I'd pay to avoid then something I'd spend months and years of hard work to attain.

    (My advice to anyone doing this is to spend some time with the AI researchers to find the existing work on the topic, not to just sit down and sketch out your initial ideas and run with them. Learn from the past. Expect this to be weeks and probably months of just thinking and noodling before you get to a design. Also, don't try to hook deeply to an existing language, as tempting as it is. This is way too large an impedance mismatch with existing languages. Any external code has to be treated like a nuclear bomb anyhow.)

    • zdragnar  6 hours ago

      What is a capability in terms of programming language design? It sounds more like the sort of thing that would belong at the standard library level, where builtin APIs are guarded by flags.

      Deno has something vaguely built in with permissions flags, and old school Blackberry (at least in the J2ME days) had permissions settings for almost everything that an app could do, but again, those are all external to the language design itself.

      2 replies →

    • ux266478  4 hours ago

      > My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there.

      You want to look for white papers that talk about object-capability systems. It's a fairly old and well-trod area of research. The E programming language[1] was all about that, and it was pretty late in the game on this stuff.

      You emphasize natively, but the problem is that's not really well defined. For static capabilities, you're just essentially asking for a suffciently strong module system with parameterized abstract data types. It's literally a subset of the grammar and what it's designed to express. Mark Miller (one of the creators of E) demonstrated that[2].

      The knock on effect of that quality is that anything which fulfills that requirement natively supports capabilities. It's part of the grammar. Doesn't even have to be object-oriented. A hackjob demonstration of an SML filesystem library with a brand/mint object capability pattern:

      brand.sig:

          signature BRAND =
    sig
        type token
    end

      mint.sig:

          signature MINT =
    sig
        include BRAND
        val mint : unit -> token
    end

      makebrand.fun:

          functor MakeBrand () =
    struct
        type token = unit ref
        fun mint () = ref ()
    end

      filesystem.sig:

          signature FILESYSTEM =
    sig
        type token
        val readFile  : token -> string -> string
        val writeFile : token -> string -> string -> unit
    end

      filesystem.fun:

          functor FileSystem (B : BRAND) :> FILESYSTEM where type token = B.token =
    struct
        type token = B.token

        fun readFile (_ : token) (path : string) : string =
            "contents of " ^ path

        fun writeFile (_ : token) (path : string) (_ : string) : unit =
            ()
    end

      trusted_fs_setup.sml:

          local
        structure FileAuthority :> MINT = MakeBrand ()
    in
        structure FS :> FILESYSTEM = FileSystem (FileAuthority)
        val rootFileToken : FS.token = FileAuthority.mint ()
    end

      trusted_fs.cm:

          Library
        signature FILESYSTEM
        structure FS
        val rootFileToken
    is
        brand.sig
        mint.sig
        makebrand.fun
        filesystem.sig
        filesystem.fun
        trusted_fs_setup.sml

      Now for any given library using the trusted_filsystem library:

        val doc = FS.readFile rootFileToken "/etc/motd" (* Works fine *)

      Delegation is function application:

        fun helper (t : FS.token) = FS.readFile t "log.txt"
  val log = helper rootFileToken

      And these all fail:

        val fake : FS.token = ref () (* Trying to forge a token *)
  val t = FileAuthority.mint () (* Trying to bypass the trusted kernel in trusted_fs_setup.sml by calling the mint *)
  
  (* Trying to self-issue authority by making our own brand and mint *)
  structure MyCap = MakeBrand ()
  val t : FS.token = MyCap.mint ()   (* type mismatch *)

      What's nice about this is... it's just normal modular programming. It's a very natural grain. It's also completely compile-time, no runtime overhead.

      You can also do a lot of this with phantom types, and it'd be much more terse and easier to handle dynamic capabilities and stuff like a capability algebra, but it ends up way less auditable and is easy to have subtle errors which defeats the point. Also compiler errors will be much more opaque. IMO needing to manually make wrappers for composite capabilities, or to handle dynamic capabilities, is the lesser of two evils. With higher order modules, those problems go away entirely.

      [1] - https://en.wikipedia.org/wiki/E_(programming_language)

      [2] - https://homepages.ecs.vuw.ac.nz/~kjx/papers/ARND2018.pdf

cogman10  7 hours ago

I think ultimately what it looks like it containing the blast radius if an agent does something bonkers.

The best case would be putting an agent in a VM and mounting the working directory there. Then you can allow it to run somewhat arbitrary actions while still being able to turn off the vm and restart it in a clean state.

The issue is, of course, that it doesn't fully prevent all possible problems an agent can cause. exfiltration is, IMO, basically impossible to stop. LLMs are exfiltration machines. The basic premise of all of them is "send us your code and a prompt and we'll do something good with it. But also if an agent decides run a command which installs a worm on a device on the network, you are hosed.

Tuna-Fish  4 hours ago

The system I'm comfortable with is to set the agent up as an unprivileged unix user, with no ability to change system configuration and no access to any files I didn't specifically give it access to. Need to let it access a file or a directory? chmod is your friend.

Second, it can pull from git, or submit a pull request, but not directly push. We have an existing system of code review for that, now also augmented by llms.

Thirdly, prevent it from sending anything but get requests to anywhere you don't want it to post stuff, with firewall configuration.

After that, turn the horrible security theater of it asking permission for anything off. So far we have had no incidents. It could of course still pull a malicious package from somewhere, that exfiltrates code using GET, but at least it can't send any credentials or user data over.

  • danudey  4 hours ago

    The way our Claude Codes are configured at work is pretty nice. There are directory patterns it can't access, like .local or .config, so when it needs to it creates a throwaway scratch pad and asks you to put stuff there; screenshots, text files, command output, etc.

    I was having it diagnose a GNOME extension and had to get it a copy of the code to work on; it would then write out a Python script to do the patching (which I could inspect beforehand) and have me execute it.

    Not having access to .local or .config can be irritating sometimes, but it's nice to know it's not just going to exfiltrate my docker or gcloud credentials.

  • gopher_space  4 hours ago

    After watching a mid-tier offering chain together tools like it was a gorilla escaping the zoo I just gave the model its own box. I don’t have time to deal with that kind of nonsense.

wredcoll  6 hours ago

I cannot help with your actual but this is giving me mild ptsd flashbacks to everyone on hn/slashdot constantly repeating how simple and perfect unix security is, just use user accounts!

As if the most valuable thing on my pc was running a program on the gpu or the printer as opposed to my email account.

20k  5 hours ago

I don't think there's a way to make it secure while still permitting it unprompted external access

Eg: Any web request is a security vulnerability, there's no way to do it if the web requests are being made maliciously

Say that we have an agent with access to get requests, solely to a single site https://yoursite.com without subdomains. In this case multiple requests can be sent, and the time between requests can be used to exfiltrate personal data, similar to the coffee shop attack but without the subdomains. If the AI is able to make requests in any form, some information can be leaked, where the amount of leakable information is tied to information theory content of whatever side channel is being used. The only 0 information channel is.. never to make a request

You could also completely trust the 3rd party you're connecting to, but that to me seems like a hard error in the modern internet

friendzis  7 hours ago

What a serious security model for a meatbag agent looks like? No, but seriously, an admin in a small org is a huge key-person risk in that they (or their stolen creds) can wipe enough and quick enough to effectively disable the business altogether.

More security conscious admins will at least segment their creds and implement four eyes principles somewhere, but were are back at square one of "asking user for confirmation".

Larger orgs, even if by necessity, segment their human agents, their creds and plaster four eyes principle liberally. But this relies on safeguards against agents colluding and ignoring some inputs, which sounds a bit scary for artificial agents.

Say you implement some swarm of agents, where access-enabled sub-agents are extremely restricted with system prompts and some access filtering. Then none of the agents in the swarm should be able to spawn themselves, otherwise a rogue agent can overwrite any safeguards. That, again, leaves the user with manually approving/denying network requests / hosts / sessions.

While I don't like anthropomorphising LLMs, the problem domain seems quite damn close to that of a key person going rogue within an org. The general solution seems to be liberal amounts of trust and ~~sweet compensation~~ gaslighting about replaceability.

  • jerf  6 hours ago

    "What a serious security model for a meatbag agent looks like?"

    Yes, I think that's very related. Humans can be punished for their crimes but they can also experience benefits that have no applicability to an LLM, so for a first approximation we can cancel those. It is very similar to trying to secure a human.

    We have more experience with that, but even then it's a hard problem too.

gmadsen  7 hours ago

Anthropic has gotten much better results by just having a different agent audit the actions of the original agent. It works surprisingly well

  • solenoid0937  6 hours ago

    Surprised no one is talking about auto mode, it solves this problem.

    • user43928  12 minutes ago

      I never had an issue with Auto mode in Claude Code.

      It used to be the same with Codex, until one day it became entirely unusable, rejecting even git operations out of concern for the privacy settings of my repository that it "cannot verify".

      Maybe worth to note that this is the only way the feature gets in the way. If on the contrary they accidentally make Auto behave like Full Access, we would never notice.

prpl  7 hours ago

For one, I’ve been working on a generic sandbox environment

github.com/brianv0/formwork

You should be easily able to hide/lock down files, network, and MCP tools from an agent and it shouldn’t be up to the agent.

  • hobofan  7 hours ago

    > files, network, and MCP tools

    Locking that down to nothing is trivial for any harness: just don't expose those to the LLM.

    The tricky part is allowing access to those.

    • prpl  7 hours ago

      sure it’s not tricky. But everybody does it different and OpenAI couldn’t even be bothered to do it right when benchmarking their models

danudey  4 hours ago

This is a complex task, and I want to avoid blatant self-promotion, but there are solutions that people are building which allows you to give a degree of freedom to your agents but also lock them down as well. Our company has a product which is just one such example. At this point it's really geared towards orgs running agents in a cluster to handle tasks, rather than e.g. making sure your claude code doesn't post your GPG keys to the blockchain or something.

In essence, you lock down all the agents completely except for permitted use cases; X agent can talk to Y agent, Z agent can talk to Q MCP server.

You register your agents, define things around them, what they can and can't do, which LLMs they can actually talk to, what sites they can access, network controls, etc.

We call ours Lynx, and it's a pretty cool product. As I said, this isn't for people running coding agents or openclaw or whatever, though the technology could do that if you coupled it with e.g. some kind of MicroVM sandbox like docker's sbx. If you want to see the sort of controls that you can put on an agent we have demo videos and stuff that show how things work: https://www.tigera.io/tigera-products/lynx/

The idea for Lynx is:

1. Your org has a bunch of scoped agents

2. You have a fixed list of what those agents should be doing and what they need to be accessing

3. They don't or won't need to access anything else

So for example, say you have an MCP server which gives you information about a kubernetes cluster. You create an agent that can query that MCP server and summarize information about it. You also have a database that associates kubernetes namespaces with the departments that use them, and an MCP server for that.

Now you can create an agent whose sole purpose is to generate usage analysis for the kubernetes cluster broken down by department.

Then maybe you have another agent with access to an MCP server which shows cloud spend in detail. That agent can query the first agent to get usage analysis and then cross-reference it with cloud spend to determine if any departments are showing sudden cost increases and generate a report for that.

The first agent gets locked down to only access those two MCP servers and whatever LLM. The second AI gets locked down to only access the first agent, the cloud MCP server, and whatever LLM.

The whole system is really neat. I think for a more open agent, like openclaw for example, you'd probably want to build out that sandbox with its own interactive permissions management; sort of like Little Snitch on macOS, where it pops up something asking if you're okay with program X doing network connection Y, you could have the sandbox say "agent is trying to access docs.foobar.io, is that okay?" or "agent is trying to run `gh pr list`, allow?" It's not realistic to pre-specify everything that Claude Code is allowed to do or access; even "raw.githubusercontent.com" could be the README for the program you're debugging or someone's sandbox-escaping exploit, but it's a good start.