Comment by eudamoniac

1 day ago

I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services as well.

6 comments

eudamoniac

Reply
LeafItAlone  21 hours ago

We’ve been tracking performance and bugs for years. Including commits those bugs were introduced in. So when LLM-generated code started working its way into our codebases, we have the before and after. And even comparing human generated code today with LLM-generated code today.

bigstrat2003  21 hours ago

I find it unbelievable. I've seen the code LLMs write and it sucks compared to what a typical human produces. The only way an LLM is doing better than human programmers is if your human programmers were producing really terrible work.

  • LeafItAlone  21 hours ago

    I personally find it unbelievable that you have access to all of the public GitHub projects available and still think the typical developer writes good code.

  • nharziro  7 hours ago

    What exactly do you mean when you say llm generated code? Are people prompting llms for changes and features without reviewing the code or iterating on it and then comparing that to what human writes? Because if so it's not surprising that you're getting worse results. Humans also write code through iteration. You can definitely get llms to write good code by enforcing guardrails and constraints through tooling and agent.md, and iterative reviews to nudge towards what you want. The first pass will look nothing like the committed code. I don't expect the llm to one shot anything.

  • tarun_anand  21 hours ago

    Believe it or not, most programmers by definition are average. Hence, producing code better than them is not a hard feat to achieve for today's models.