Comment by reasonableklout
1 day ago
Does the model get things wrong? Or does it just give up in trying to completely understand some very messy codebases?
1 day ago
Does the model get things wrong? Or does it just give up in trying to completely understand some very messy codebases?
All models still gets things wrong. Even Fable 5.1 I have to regularly correct when working on sufficiently complex code. So, I still want models to get smarter!
Fable is good, especially at conceptual stuff, don’t get me wrong, but it has limits. Opus 5.5 is a machine at getting correct code out once the concepts are in place. But I can’t trust either to not make mistakes.
As an aside, getting things wrong isn’t simply about not being smart enough, it’s also about the model insufficiently assessing whether it actually knows enough to conclude that some hypothesis is correct. As a human I have inner feelings of certainty and uncertainty regarding the things I am thinking about, which guide me in what I need to ascertain or verify more closely. My impression is that LLMs are lacking in that area, and it’s why they are most effective when there are external verification mechanisms.