Comment by kouteiheika
4 hours ago
> And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test?
Well, everyone and their dog has a custom harness now (including myself!), and more often than not they're just... pretty much the same, just with a slightly tweaked UI and different defaults? The harness needs to either be one of the popular ones, or have some sort of a unique feature which actually differentiates how it runs, not just how it looks/controls like.
For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session. This means that any files the agent modifies aren't actually modified on disk until the user uses `/apply`, and rewinding the session also rewinds the state of the disk, and this works for all I/O and all tools (including anything spawned by the `bash` tool), and also supports at-will forking of the session including the filesystem state. This means that the agent can run fully YOLO and won't be able to nuke your disk, and it also means that there's no need to use git worktrees and such because multiple agents can just run in the same directory at the same time without an issue.
Essentially, what I'm trying to say: I'd love to see more experimentation in the harness space which doesn't just involve "a different GUI".
> For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session...
What are the mechanics of this? You mentioned bash so I asssume you don't "control" all the tooling. Is it running in a VM with an overlay filesystem that takes snapshots at every file interaction? Or perhaps outside a VM but containerized with a FUSE-based overlay filesystem?
It's not a VM; it's containerized with a FUSE-based custom overlay, and the agent never touches a real mount (with two exceptions: passthrough read-only mounts for things like /bin, and ephemeral tmpfs mount at /tmp).
What I've found difficult with harness design is deciding which features are actually going to be useful, and which ones are cool-sounding ideas that get people talking, but nobody actually uses in practice. It should be easy to tell the difference, but it isn't!
I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
> What I've found difficult with harness design is deciding which features are actually going to be useful, and which ones are cool-sounding ideas that get people talking, but nobody actually uses in practice. It should be easy to tell the difference, but it isn't!
Same. But there's a trick to it, in a way: vibe code it first, test it out yourself, and then decide whether it should be reverted or kept and deslopped/properly shipped.
> I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
For me personally it's a game-changing workflow, because it means I don't have to sandbox (e.g. with a Docker) my agents while still being able to run full YOLO. Essentially have my cake and eat it too. If the agent does something stupid I can just rewind. And I don't need to spend as much effort continuously reviewing what it does, because I can just do it right at the end on the `/apply` screen.
(But this only applies if the filesystem is actually fully virtualized, as opposed to the git-based checkpointing that so many harnesses have, which to me is pretty much useless because it doesn't actually give me any guarantees. That is: the agent running `rm -Rf /*` should be a non-event that doesn't do anything to my filesystem, and can only nuke the agent's view of the filesystem.)
Of course, this could just be me, and other people may not really care. But then, is your objective to make the best possible harness for you and people like you, or for everyone? In my case it's the former.