Comment by godelski

  > The thing is, AI researchers have continually underestimated the pace of AI progress

What's your argument?

That because experts aren't good at making predictions that non-experts must be BETTER at making predictions?

Let me ask you this: who do you think is going to make a less accurate prediction?

Assuming no one is accurate here, everybody is wrong. So the question is who is more or less accurate. Because there is a thing as "more accurate" right?

  >> In 2022, they thought AI wouldn’t be able to write simple Python code until around 2027.

Go look at the referenced paper[0]. It is on page 3, last item in Figure 1, labeled "Simple Python code given spec and examples". That line is just after 2023 and goes to just after 2028. There's a dot representing the median opinion that's left of the vertical line half way between 2023 and 2028. Last I checked, 8-3 = 5, and 2025 < 2027.

And just look at the line that follows

  > In 2023, they reduced that to 2025, but AI could maybe already meet that condition in 2023

Something doesn't add up here... My guess, as someone who literally took that survey, is what's being referred to as "a simple program" has a different threshold.

Here's the actual question from the survey

  Write concise, efficient, human-readable Python code to implement simple algorithms like quicksort. That is, the system should write code that sorts a list, rather than just being able to sort lists.
  
  Suppose the system is given only:
    A specification of what counts as a sorted list
    Several examples of lists undergoing sorting by quicksort

Is the answer to this question clear? Place your bets now!

Here, I asked ChatGPT the question[1], it got it wrong. Yeah, I know it isn't very wrong, but it is still wrong. Here's an example of a correct solution[2] which shows the (at least) two missing lines. Can we get there with another iteration? Sure! But that's not what the question was asking.

I'm sure some people will say that GPT gave the right solution. So what that it ignored the case of a singular array and assumed all inputs are arrays. I didn't give it an example of a singular array or non-array inputs, but it did just assume. I mean leetcode questions pull out way more edge cases than I'm griping on here.

So maybe you're just cherry-picking. Maybe the author is just cherry-picking. Because their assertion that "AI could maybe already meet that condition in 2023" is not unobjectively true. It's not clear that this is true in 2025!

[0] https://arxiv.org/abs/2401.02843

[1] https://chatgpt.com/share/688ea18e-d51c-8013-afb5-fbc85db0da...

[2] https://www.geeksforgeeks.org/python/python-program-for-inse...