Comment by jph
7 hours ago
Lots of errors. Opus 5 is also giving me many more hallucinations, including things that aren't even in the right territory. It's also telling me that it's making many mistakes, and the language feels off-kilter as if it's not using typical clear phrases.
Opus 5 / Fable - instead of implementing things one-by-one in a branch - decided to branch off and do everything from the starting point - meaning that they re-did work over and over, in different ways, and that almost nothing could merge back at the end.
Looking through the logs, it kept saying, "Wait I'm confused. I already did this." But I guess it decided, oh well, I'll just do it again instead of take a second to figure out what I'm doing...
Quality of output of both Fable and Opus seems to have gone down quite a bit over the past week. Many mistakes.
Yes, it’s way off field in many things, gets into weird minutia without seeing a way out, and it’s often seeing a clear sequence of work but then halts on a statement like “ok I’m going to start now.” Then after expiring the cache when I notice and ask why didn’t it the response is “no reason starting now!”
I see this behavior constantly in 5 - the quality of opus and fable have degraded constantly since 4.6 was such a riotous success
It is aggressively proactive in ways that make it very hard to use. I had to turn down the effort level to “low” to stop it from going off in random directions every couple of turns.
I saw this even on the weekend when it wasn't under heavy load. We agreed a problem was due to one choice of word in a prompt and then it went off and wrote 600 lines of code to "fix" the issue.
Yeah I noticed informing me of mistakes it made during sessions. It felt really off when it informed me of a mistake it almost made but caught it before it landed.