Comment by rgbrgb
1 day ago
> Every app can run headless, so an agent can render to PNG, dump the element tree and drive it with scripted input.
This is cool. I hope SwiftUI / UIKit borrow this idea for native mac apps.
1 day ago
> Every app can run headless, so an agent can render to PNG, dump the element tree and drive it with scripted input.
This is cool. I hope SwiftUI / UIKit borrow this idea for native mac apps.
You can do the same thing with swiftui. The surface you render to doesn't have to be visible at all. And the tree has nice introspection available.
Headless part is neat, but agents can already drive apps and take screenshots. As a side note, I have agent driving android tv using wireless adb and hdmi capture card + mpv rpc, its fun to watch.