Comment by jpalomaki

Agents are not yet very good at figuring out how things look on the screen.

Or at least in my experience this is where they need most human guidance. They can take screenshots and study those, but I’m not sure how well they can spot when things are a bit off.