Comment by mike_hearn
2 days ago
But, will that problem remain or is it simply that the models haven't been RLVRd on QA tasks yet? QA feels like a very RL suitable problem. You can take a codebase that has a known set of bugs in it, and proceed to train the model until it identifies those bugs. You can even generate such programs automatically by having adversarial training going on (one model deliberately writes a buggy program, another searches for the bugs). This works even with GUI bugs, drifts from the specs and so on.
I don't see any theoretical reason that this weakness must remain for longer than six months or so. I mean, both Claude and GPT are very good at doing code reviews and that's a form of QA.
The problem will remain due to information theoretic concerns. It is not possible for an AI to read minds. You can say to an AI "build me an accounting system for a small tax firm" and we can, for the sake of argument, even stipulate that it will produce a functional system based strictly off that prompt. However, no matter how smart the AI, the result can not be any more detailed than that. Did you want it for MacOS? It may have built it for windows. Did you want it integrated into an already-existing database server? It can't have done that, and since you didn't specify the database you want it could have guessed wrong, and certainly can't guess about what tables you may already have you want to integrate into. It probably didn't guess your jurisdiction correctly (unless it was fed that in some other contextual prompt). It can't guess your most common work flows and optimize them. It can't pick out specialized features you want just for your special use case.
None of that is because AIs aren't good. It's just plain information.
An AI could sit there and interrogate the user for this information. In that alone, it would already have exceeded what a lot of people are going to want from a system like this. People, normal people, want AIs to magic stuff into existence, not spend hours interrogating them for a design requirements elicitation. That's a barrier a lot of people will not cross, and the AI companies, who fully realize that, aren't building that way.
Normal people do not want to even do the work it takes to fully specify a problem, even if an AI walks them right through it, let alone the verification that the work is correct, and actually what they intended rather than what they asked for.
And of course, in practice, AIs are not emitting bug-free code, miss entire requirements at a shot, and have a variety of other characteristic failure modes in this space that I expect are fundamental to the LLM architecture and aren't ever going to leave. If they continue to be able to make bigger systems they will simply continue to make correspondingly big errors and omissions in the process. No comment as to what the next generation AI architecture will do, though.