Comment by SillyUsername
6 hours ago
For the last 2 days I've been trying to create a skill for Openclaw that would allow traditional control of the Adeept Tank's robot arm (a small open source toy tank that looks like a bomb disposal robot).
ASTRA HAS BEEN UTTER SHIT.
It is much more expensive than Sol 5.6 Medium / High and did nothing but write unit tests and junk code, despite having access to the vendor original source, an API, and the full tank specs.
Failure Examples:
* In two instances had the direction of the servos wrong.
* Calculated the maximum extent of the gripper wrong, and the closure, so it didn't grip.
* Code failed to take into account the gripper requires continuous torque when lifting a pair of socks, so couldn't lift.
* Failed to actually start physical testing more than opening and closing the gripper, and that was when I asked about progress.
* Code failed quite spectacularly to calculate camera gimbal extent range correctly.
* Code failed to use the ultrasonic in range to target until I pointed it out, the skill also didn't advise gimbal angle adjustment to correct range overshoot to the wall behind a small object.
The test environment has both an onboard ultrasonic for distance, onboard camera, and a bird eyes view camera (birds eyes only while training).
I've stopped using Astra Low (default) and gone back to Sol 5.6 low/medium/high for the training, it's cheaper and now I'm back to fine tuning, after it had to redo large chunk of the gripper/arm code and prevent unnecessary hard stop code kicking in based on the wrong profiling.
It's cost me around 1000 to 1250 credits (£50), burnt in around 2 hours, looking mostly at recorded video, and photos, and writing bad code based on bad assumptions. I've also burnt through regular Plus 5 hour quota in about 30-45 minutes with it.
I'm not sure why this has been voted down, it's a counterpoint to the hype with factual anecdotes to back up the claim of it's performance Vs the article itself. I've got the source and video to prove it too.
You used Astra with Low effort only?
There is something deeply wrong with Astra. I can’t quite put my finger on it. On the one hand it is a lot more knowledgeable, which makes sense since it’s a larger model. On the other hand that knowledge doesn’t reliably translate to intelligence or insight. Certainly tasks like 3D modeling it does extremely well. Other stuff like complex coding problems in an existing codebase it stumbles more often than not. This morning it ran around in circles. It implemented a feature, then convinced itself that it should have followed “proper TDD”, deleted all the code it had written and wrote 8,500 LoC of unit tests. At that point I was down to 35% of quota so I stopped it and gave the task to Opus 5.
Really weird model. No idea how it did so well on all the benchmarks.
Which benchmarks? Only the ones OpenAI cherry-picked.
It debuted as ~same score as Sol on Artificial Analysis. People couldn't accept it so they had to change the formula.
The model is a big step forward only in desktop use and 3D. That's impressive, but for software engineering, Fable is still in a league of its own.
That's exactly the kind of behaviour I've seen, unbelievable amount of unit tests, and revisiting and revising the same code over and over again. If I was cynical, I'd say almost like it was deliberately trying to burn quota, even after I told it quota was getting low and to move onto actual physical testing.
was it maybe over-quantised to reduce costs?