Comment by pradn

A middle ground would be an LLM taking structured inputs ("mouse clicked at X,Y") and making "draw" calls at whatever level: pixels, SVG, HTML/CSS. While the LLM's output tokens could be mapped to just these calls, it could also just do a bunch of tool calls that write to these formats as well -- it's a matter of efficiency.

We have Jev to prove LLMs that output constrained decisions are useful. Why not make LLMs just output UI natively?

At a fast enough inference speed, you could just connect the network to it.

So an LLM would then just take the network, user input, and output a UI, with tool calls to deal with memory and other resources. In theory, you don't need an OS any more.