Still exploring different options but at the moment I run a daemon on each node that connects via iroh to link the nodes (and NAT traversal). Once the link is established the nodes share data via a Raft-based datastore. I can launch a TUI client on any machine. This presents a chat interface like a typical LLM harness. From here I can configure which model I want to use, how the model is executed and on which node (for example I can use Ollama on machine A to serve a model that’s running on machine B, but executing commands via this system on machine C). Each daemon then acts like a typical LLM harness, but headless. So the commands and tool calls are streamed into the shared store and finally end up in my TUI. Since all the state is in the distributed store I can pick up the progress wherever I want, as long as there’s a node running there.
Still a lot of big problems like latency so I’m not too happy with the current design. It’s a LOT of state to replicate, and obviously things get problematic when an agent was working on a project on machine A but then ends up on machine B which doesn’t have that project directory around at all. So the agent needs to re-orient itself to figure out what happened. Still pretty fun to play with.
Still exploring different options but at the moment I run a daemon on each node that connects via iroh to link the nodes (and NAT traversal). Once the link is established the nodes share data via a Raft-based datastore. I can launch a TUI client on any machine. This presents a chat interface like a typical LLM harness. From here I can configure which model I want to use, how the model is executed and on which node (for example I can use Ollama on machine A to serve a model that’s running on machine B, but executing commands via this system on machine C). Each daemon then acts like a typical LLM harness, but headless. So the commands and tool calls are streamed into the shared store and finally end up in my TUI. Since all the state is in the distributed store I can pick up the progress wherever I want, as long as there’s a node running there.
Still a lot of big problems like latency so I’m not too happy with the current design. It’s a LOT of state to replicate, and obviously things get problematic when an agent was working on a project on machine A but then ends up on machine B which doesn’t have that project directory around at all. So the agent needs to re-orient itself to figure out what happened. Still pretty fun to play with.