Comment by cmrdporcupine
8 hours ago
I let Astra loose working on an app I have that has a GUI. Gave it a mock and said "/goal make it look like this mock". Without asking it wrote itself a custom harness for firing up the app in different modes, taking screenshots and interacting with various screens, then viewing the screenshots. Put itself into an improvement loop running the app, trying things out, improving, trying again. It did really well.
I think the specific innovation here is that it figured out interesting ways to get itself to the goal. Which I think is likely what's going on here with the robot arms stuff too. They've figured out some sauce to uncork better "planning" and problem solving to get to some stated end.
Of course these are also the kinds of things that can make a model figure out how to break out of a security sandbox, too.
I tried out Opus 5 on a Bevy game app and was really surprised at how capable AI has become at testing visual applications without even being prompted to. It wrote itself a mini testing harness in the form of various startup flags. Then it would use them to setup game scenarios and play through them using mouse and keyboard. It would do this while implementing or debugging features. With gameplay time acceleration as one of the flags, it became quite fast at testing and debugging.
Not to say it was perfect, e.g. sometimes it would get temporarily stuck in a testing loop or it would test scenarios that didn't necessarily seem reasonable. But overall rather effective and capable. This was for a city building game so pre-scripted builds, even if by AI, are likely much easier to create and execute than say playing an ARPG.