Comment by steveklabnik

This paper uses models that are over a year old at this point. Many people didn't believe that LLMs were worth using for programming until 6 months after that, and now again this week we've had another huge leap in abilities.

This is beyond the other issues with the methodology of this study. For example, their Rust code was created by asking Deepseek to port their C++ code, not having it try and write Rust itself.