Comment by utopiah
9 hours ago
Indeed, that's what I kind of hinted at in https://news.ycombinator.com/item?id=46437688 briefly after, namely that OK, one can "generate" a "solution", that's much easier than before... but until we can verify somehow that it actually does what it say it does (and we know of hallucinations and have no reason to believe this changed) then testing itself, especially of well know "problems" is more and more important.
That being said, it doesn't answer the "why" in the first place, an even more important question. At least though it does help somehow to compare with existing alternatives.
Isn’t this how all software development works? Folks commit code, it’s tested, and reviewed, and then deployed.
Why would this be any different?
That's not how software development works.
Folks think, they write code, they do their own localized evaluation and testing, then they commit and then the rest of the (down|up)stream process begins.
LLM's skip over the "actually verify that the code I just wrote does what I intended it to" step. Granted, most humans don't do this step as thoroughly and carefully as would be desirable (sometimes through laziness, sometimes because of a belief in (down|up)stream testing processes). But LLM's don't do it at all.
They absolutely can do that if you give them the tools. Seeing Claude (I use it with opencode agents) run curl and playwright to verify and then fix it's implementation was a real 'wow' moment for me.
6 replies →
> LLM's skip over the "actually verify that the code I just wrote does what I intended it to" step.
I'm not sure where this idea comes from. Just instruct it to write and run unit tests and document as it goes. All of the ones I've used will happily do so.
You still have to verify that the unit tests are valid, but that's still far less work than skipping them or writing the code/tests yourself.
2 replies →
> actually verify that the code I just wrote does what I intended it to
That's what the author did when they ran it.
Claude Opus 4.5 will routinely test its own code before handing it off to you, even with zero instruction to do so.
4 replies →