Comment by postalcoder

Yeah. Computer use is essentially a model navigating the OS-provided accessibility tree. I imagine a model trained on it would operate the computer exactly as we saw it control Doom.

https://developer.apple.com/library/archive/documentation/Ac...