Comment by cameronh90
2 hours ago
In terms of getting a the consensus answer to a specific question, they're likely better than a median software engineer at most questions.
Where they still fall massively short, even compared to a straight out of college junior developer, is figuring out what the correct question to ask is.
This has led to situations like Fable 5 spending six days debugging a deadlock because it didn't think to add any observability to detect when the job stalls, and would wait until the job timed out each time. Any junior developer would have fixed it in a few mins, but I left Fable running out of curiosity to see how long it'd take.
This may be true now but isn’t the trend clear? GPT-2 couldn’t write a coherent function. Now we have the SWE oracle that can autonomously navigate your codebase, write code at a superhuman level, and give the correct answer to most engineering questions.
If anything I feel I am understating the capability of current models here. What is stopping the next generation of models from closing the feedback loop further?