Comment by gsnedders

There’s plenty of things you can very deterministically verify — if the LLM writes a test case which causes an ASan violation, you probably have a bug; if the LLM writes a test case which run the various corruption checking tools and they find corruption then you either have corruption or you have a bug in the corruption checking tool; etc.

There’s plenty of ways in which you can guide LLMs, and sandbox them, to get a much higher hit rate.

And that’s before the increase of validity of bugs compared with a year ago — we’re past the peak of slopageddon, and maybe even at a point where a higher percentage of security bugs reported are valid than was the case pre-LLMs.