I think we need a new area of study around UI/Agent connection. It can kinda be done with tools, but I'd we need much deeper primitives to allow the UI to inform the Agent and vice-versa. Right now we've just given up, replacing the UI with an Agent message view, but I think that's just because nobody is thinking about how the two can compliment each other
I've been playing around with using a hidden markov model informed by a UI state event stream, with the end state fed into the Agent as a hint on each message turn. Then the Agent can make a tool call to add events to the HMM. This has been really interesting, but I haven't struck the right balance to make it actually feel good for the user yet
Isn't this the direction Claude on the web is going towards? It's a mix of message view on the left and specific interactive UI on the right panel. There is also MCP Apps spec now
I think we need a new area of study around UI/Agent connection. It can kinda be done with tools, but I'd we need much deeper primitives to allow the UI to inform the Agent and vice-versa. Right now we've just given up, replacing the UI with an Agent message view, but I think that's just because nobody is thinking about how the two can compliment each other
I've been playing around with using a hidden markov model informed by a UI state event stream, with the end state fed into the Agent as a hint on each message turn. Then the Agent can make a tool call to add events to the HMM. This has been really interesting, but I haven't struck the right balance to make it actually feel good for the user yet
Isn't this the direction Claude on the web is going towards? It's a mix of message view on the left and specific interactive UI on the right panel. There is also MCP Apps spec now
It's hard to say what is needed now, vs needed in a few years.
For example, many rumors say SSI has solved learning and retaining state. That would significantly change AI requirements.