Comment by jmathai
2 years ago
> it rarely happens that it just finds stuff that's flat out wrong
"Flat out wrong" implies determinism. For answers which are deterministic such as "syntax checking" and "correctness of code" - this already happens.
ChatGPT, for example, will write and execute code. If the code has an error or returns the wrong result it will try a different approach. This is in production today (I use the paid version).
Dollars to doughnuts says they are using GPT3.5.
I'm currently working with some relatively obscure but open source stuff (JupyterLite and Pyodide) and ChatGPT 4 confidently hallucinates APIs and config options when I ask it for help.
With more mainstream libraries it's pretty good though