Comment by jerf

The problem will remain due to information theoretic concerns. It is not possible for an AI to read minds. You can say to an AI "build me an accounting system for a small tax firm" and we can, for the sake of argument, even stipulate that it will produce a functional system based strictly off that prompt. However, no matter how smart the AI, the result can not be any more detailed than that. Did you want it for MacOS? It may have built it for windows. Did you want it integrated into an already-existing database server? It can't have done that, and since you didn't specify the database you want it could have guessed wrong, and certainly can't guess about what tables you may already have you want to integrate into. It probably didn't guess your jurisdiction correctly (unless it was fed that in some other contextual prompt). It can't guess your most common work flows and optimize them. It can't pick out specialized features you want just for your special use case.

None of that is because AIs aren't good. It's just plain information.

An AI could sit there and interrogate the user for this information. In that alone, it would already have exceeded what a lot of people are going to want from a system like this. People, normal people, want AIs to magic stuff into existence, not spend hours interrogating them for a design requirements elicitation. That's a barrier a lot of people will not cross, and the AI companies, who fully realize that, aren't building that way.

Normal people do not want to even do the work it takes to fully specify a problem, even if an AI walks them right through it, let alone the verification that the work is correct, and actually what they intended rather than what they asked for.

And of course, in practice, AIs are not emitting bug-free code, miss entire requirements at a shot, and have a variety of other characteristic failure modes in this space that I expect are fundamental to the LLM architecture and aren't ever going to leave. If they continue to be able to make bigger systems they will simply continue to make correspondingly big errors and omissions in the process. No comment as to what the next generation AI architecture will do, though.