Comment by rcarr
1 year ago
Has anyone tried asking it to generate the libraries/functions that it's hallucinating and seeing if it can do so correctly? And then seeing if it can continue solving the original problem with the new libraries? It'd be absolutely fascinating if it turns out it could do this.
Not for libraries, but functions will sometimes get created if you work with an agent coding loop. If the tests are in the verification step, the code will typically be correct.
I sometimes give it snippets of code and omit helper functions if they seem obvious enough, and it adds its own implementation into the output.