Comment by enraged_camel

>> Yeah, sorry, skill issue. If you let AI run wild (if one session is $680 yeah it ran pretty wild) don't complain how it messed up your codebase.

Not the OP, but I consider myself a skilled and heavy AI user, with multiple subscriptions in both platforms, plus OpenRouter.

A couple of weeks ago I gave 5.6 Sol a small/medium sized ticket to simplify part of the auth system. The ticket had a lot of details another Sol agent had collected during an exploratory session, and it was all vetted by Opus 5. I thought to myself that the implementation agent should have everything it needs. I still had it write a plan just in case, read the plan, made sure it matched the ticket, then clicked Approve and walked away.

I came back later that afternoon to a horror show. The agent had written 25,000+ LoC in the worktree. After 15 minutes of skimming through it, I realized that it had made the specced change, then convinced itself that it needed stronger verification, and over a series of compaction cycles ended up writing a static analysis harness so that it could prove that the change would be safe. Total bonkers.

Except, according to another Sol agent I showed the worktree to, the harness didn't actually do what the original agent claimed. The review agent said 98% of the worktree's code should be thrown away, and only the fix and its relevant unit and integration tests should be retained. I also asked Opus 5, and it theorized that 5.6 Sol must have gone through too many compaction cycles and lost track of its original goal.

This never happens to me with Claude models. Yes they write a lot of code and verbose comments, but I've never had a situation where a ticket that should take several hundred LoCs ended up with tens of thousands. When Claude overengineers something, I catch it during the planning phase, and it implements plans faithfully.

5.6 Sol is simply unreliable. It's too relentless and doesn't know when to stop. That's probably what caused the OP's $680 incident. I find it fascinating that people like it so much.