Comment by ryeights
4 hours ago
Outside of the most niche domains like frontier AI development, are there software engineering questions that Sol 5.6/Fable 5 can't answer better than the median software engineer if explicitly asked? It strikes me that models already have superhuman software architecture capabilities, they are simply underelicited.
Furthermore the AI companies are spending gobs of money and compute to address this—harness development, orchestration/agent coordination, computer use are all designed to lower the skill level required to get the best results out of a given model.
Soon the models themselves will be more persistent, more observant, and more skilled than any of us at driving models to obtain the "right" solution in a given SWE context.
In terms of getting a the consensus answer to a specific question, they're likely better than a median software engineer at most questions.
Where they still fall massively short, even compared to a straight out of college junior developer, is figuring out what the correct question to ask is.
This has led to situations like Fable 5 spending six days debugging a deadlock because it didn't think to add any observability to detect when the job stalls, and would wait until the job timed out each time. Any junior developer would have fixed it in a few mins, but I left Fable running out of curiosity to see how long it'd take.
This may be true now but isn’t the trend clear? GPT-2 couldn’t write a coherent function. Now we have the SWE oracle that can autonomously navigate your codebase, write code at a superhuman level, and give the correct answer to most engineering questions.
If anything I feel I am understating the capability of current models here. What is stopping the next generation of models from closing the feedback loop further?
Google can also answer it better than most if explicitly asked.
An explicit query returns a specific answer, but means nothing about whether it can use that knowledge to create better software.