Comment by lelanthran
1 hour ago
> You're asking basically to ignore bugs and correctness.
Not ignore correctness, just bugs that will be caught by tooling.
> Can it be a useful comparison?
IME, yes. LLMs in an agent-loop are trivially able to write spaghetti code that will never do an off-by-one error or something else that is easily caught by tooling, which is not something humans can do.
Judging code on whether it has bugs easily caught by tooling is pointless - LLMs are running the tooling in a loop anyway, so no matter how bad or poor their code actually is, it never exhibits bugs that are caught by tooling.
> just bugs that will be caught by tooling
Bugs are bugs, if the instruction is "ignore bugs except for those that can be caught by you" then the instruction is basically "ignore bugs".
And it implies "ignore correctness" because when program is incorrect we usually refer to it as a... you guessed it, "bug".