Comment by surgical_fire
8 hours ago
I still have to correct Claude on very basic misconceptions whenever I get it to code shit.
Sometimes it gets wrong things that I had spelled out already.
It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake.
"You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"
https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
> I still have to correct Claude on very basic misconceptions whenever I get it to code shit.
Can you give a simple example?
I would have agreed 2 years ago, but it's extremely rare I see a frontier model making a silly mistake these days.
At least for me it’s quite easy to see them go into endless loops where no meaningful work is done and it just keeps going until I stop the process and tell it what to try instead.
To be fair it is no where near what we had just one year ago and the rate of change only seems to be increasing.
Also, I don’t have 30 million dollars to spare spawning tens of thousand sub agents like what they did with Navier-Stokes so I’m clearly not testing the full capabilities of these models.
Yes. Yesterday, Opus 5 on Claude Code with high effort.
It was to build an extremely simple job using an internal framework to walk through a table and log the ids os some records that have a certain scenario.
There's a ton of jobs exactly like this in the codebase, and the framework code is in the codebase as well.
It was so silly I even thought of writing it myself, probably took me longer to steer claude to do it for me.
Anyway, it refused to use a method from the framework to retrieve the parameter as a list, it wanted to retrieve it as a string and parse the commas. I had spelled out in the initial prompt what method it should use.
I really don't like Claude much. Frontier my ass.