Comment by dangom
12 hours ago
The choice happens implicitly rather than explicitly. If Claude tries an approach and hits a wall, it'll try a different approach. If an API call keeps not working, it'll choose a different API. It a tool is broken, it'll use something else. If it can't find docs nor read the code, it'll try to implement functionality from scratch. If you give it messy tools with confusing docs, you'll notice Claude not calling them as you'd expect, and instead trying something simpler instead.
"something simpler/simpler approach" are terms I search for in evals because they almost always indicate the model going off the rails (assuming the input prompt was decent).
Yesterday I had it using an internal library without documentation or source code. LSP integration wasn't working. It didn't have decompilation tools or the ability to download them.
I came back to my terminal to find it had written its own tool to decompile the assembly, and successfully completed the task using that info.
That's hilarious