Comment by chasing0entropy
5 months ago
The models were restricted from accessing the internet and forced to develop their own solutions internally.
I think researchers will find that human coders are unable to solve most coding problems without access to the internet.
What kinds of problems are you talking about? There are problems that require you to learn new libraries and services constantly, accessing documentation, and there are actual software problems when you have to reflect on your own large code base. I work on both kinds of problems, and in the first case, the models are actually well versed in say, all of CloudFormation syntax, that I would have to look up. On the opposite end, I have written many features on trips, unable to be distracted by the internet, just me and the code, and being able to read library source code.
The fact is, programming requires abstract modeling that language models aren’t demonstrating the capability of fully replicating. At least, not that we can see, yet.
A decent human programmer with experience in a particular domain may rely on internet access to look up API documentation and other generic references, but if you read the paper, you'll see that the AI systems tested suffered from more basic deficiencies in approach and reasoning ('3.6. Discussion', starting on page 7).
If you're requiring human coders to write valid, compilable code, maybe. But if you're doing that, you're doing coding interviews wrong.
Any interviews I've run or been a part of have required the interviewee to demonstrate their problem-solving skills using pseudo-code.
And you created the internet then?
Damn. How did they write code before 1992?
We did have usenet in the 80s and gopher in the 90s. But yes, in those days it was that mythical "paper" stuff (or were we still using papyrus? I forget)
Seriously? I think I've been most productive pre-internet days when stuck on a trans-atlantic flight with a java (shudder, talk about PTSD) reference manual and laptop that could barely last 2 hours on battery, with emphasis on the measure-twice, cut once mentality.
It's painful to watch junior coders copy-n-paste from SO or W3schools (including code samples clearly labelled not-for-production) with little effort to understanding what they are doing.