Comment by lifeisloving

I dont think the quality has gotten that much better. Take claude 4.5, the probability of getting usable outputs was less likely than with opus 5.5, and the usuable outputs weren't great.

I think newer models are more likely to produce the usuable output, almost always, but its still not great. The frequency in which you get usable results leads one to believe that quality is improving.. No usability is improving but the result is still meh.

Also put an llm in an environment without an lsp or compiler to check against and let me know how capable you think they are on their own. The "loop" is doing a lot of heavy lifting here.