Comment by mnahkies
21 hours ago
Damn, I've been building something very similar the past few weeks - haven't quite gotten to a public release yet.
Mines based on libvirt and uses cloud-init for creating the images, with shared golden images across projects.
Skimming your docs it wasn't totally clear to me if your images are built per project, or COW from shared toolchains.
Looks like you have a few features in there that I might take some inspiration from though, eg the proxying stuff.
Funny how many of us are trying to solve the same problem at the moment.
One thing I'm struggling with at the moment which I may as well ask here.
Many of my projects use pnpm, which ideally uses hard links from a shared store of dependencies.
One of my goals is to reduce the disk overhead of VM based isolation, so I'd like a shared pnpm store across projects (and the same for maven, etc as well)
With https://virtio-fs.gitlab.io/ mounts, each mount inside the VM is a separate FS, and I can't hard link across them. I've tried playing around with ideas like bind mounting the directory structure from the host in a folder to have a single virtio-fs mount inside the VM - but the bind mounts still can't be hard linked across it seems.
I'm basically wondering if I'm missing something or just trying to do something that isn't possible.
All the host folders are on the same FS - but I don't want to organise the host system around the VM mounting needs, and I also don't want to mount extraneous directories into the VM (eg: simple solution could be just mount ~/projects but kinda self defeating)
(Fedora workstation host)
Use a host fs that supports CoW.
add me as well. I learned about incus from HN ~a month ago, and was poking an idea of having a "development base image". The advantage compared to containers, is that you can hot-plug devices to the running instance: mounts, secrets, etc. incus also supports --vm and container mode with minimal difference in cli commands.
https://linuxcontainers.org/incus/
my main bottleneck was exactly in usability: how to make it least effort to spin up per use case and keep the set of operating scripts small - i didn't want to build an app for this, just shortcuts to use existing app.
Add me to the list, although I use podman containers (https://github.com/evertheylen/probox). I find that the hard part is not restricting the container/VM, but to make it "easy" to do things in the container that require something protected. How will you commit? Where does your agent run, and can it access github? How will you deploy? How will you SSH to a server?
Happy to hear what other people are thinking. It seems like putting an intercepting HTTP proxy is getting popular in other projects.
I usually commit outside the sandbox. Agents will see the git history and will emulate that but they can't sign the commits. If I authorize a series of commits I can just `git rebase -i HEAD~N` and adjust the messages and re-sign at the same time. With git hooks as a threat vector I'm considering just giving them read-only access to `.git` otherwise I can't safely go back in there without the sandbox.
I don't give it access to GH - it can read most things via API or web scraping fast enough and I manually run the commands I need outside the sandbox.
No need for SSH - it's running locally.
---
Given how many holes you're poking into the container I think you'd be better served with a real VM.
It feels like a fundamental problem. The more you restrict the agents, the less useful they become. The less you restrict them, the more dangerous they become.
It makes sense, but a good system where you can say "give this VM access to repo X, branch Y, and server Z for the next 4 hours" without leaking any secrets into the VM would be a major improvement already.
Human coders and sys admins routinely make catastrophic mistakes, too, so it's not like we're facing a novel problem.
On the git access part, I don't let my agent VMs have write access to GitHub currently. I do give them dedicated users on a self hosted gitea instance, and basically manage the blast radius through branch protection etc.
I think at a high level I'm trying to treat them as junior/intern employees where I give them enough permission to do their role, but not enough rope to hang themselves.
I see. But then I have to debug some failing CI pipeline and the agent is quite good at going through the logs, so I give in and `gh auth login` anyway...
1 reply →
You can have a separate GitHub account for your agent (GitHub ToS actually allows that since April) and that account can fork your repo and open PRs. This way it cannot do any real damage but can still push.
See this user for example: https://github.com/filippo-agent
Do you have more background on this? My team actually ran into some issues with GH when we tried this earlier this year (felt like it was around that same time).
1 reply →
Somebody shared this in another thread: https://pleasedonotescape.com/
I guess they can add all the sibling comments to the list, as well as mine: https://github.com/AdrianVollmer/Agent-Sandbox
(Python, Podman, Squid for network filtering)
It's a neat snapshot at the time it was creates but it's getting out of date.
Also been working on something similar, with qemu/kvm; the idea is to sandbox agents and limit resource usage per project. Its not production-ready yet, but the repo is public - https://github.com/blackshieldpt/catchpen
> Mines based on libvirt and uses cloud-init for creating the images, with shared golden images across projects.
This sounds good. Cloud-init is still worth the time, I think (as I just said in another comment)
Add me to the list lmao. It seems like this is pretty popular among folks
Ha, add me to the list. I built one with libvirt and cloud-init too, mostly focused on network containment (private/host networks are blocked) and VM hardening. There's also a check command that runs nmap from inside the VM to verify nothing private is reachable.
https://github.com/ertug/kvm-sandbox
Heh, yeah also very similar. I don't strip devices like you are - I actually want the memory balloon functionality for example.
My working threat model for my projects is less about "malicious agent trys to escape VM" and more "well meaning agent got confused and screwed up" - the added protection of a VM is desirable, and it's useful for stuff like nested virtualization (agent working on improving the VM orchestration), makes containers inside the sandbox straightforward
I hadn't seen the stuff about filtering spoofed packets so I'll probably add that.
[dead]