I'm building an AI agent for Godot, and in paid user testing we found the median speed up time to complete a variety of tasks[0] was 2x. This number was closer to 10x for less experienced engineers
[0] tasks included making games from scratch and resolving bugs we put into template projects. There's no perfect tasks to test on, but this seemed sufficient
Have you evaluated the maintainability of the generated code? Becuause that could of course start to count in the negative direction over time.
Some of the AI generated I've seen has been decent quality, but almost all of it is much more verbose or just greater in quantity than hand written code is/would be. And that's almost always what you don't want for maintenance...
That sounds reasonable to me. AI is best at generating super basic and common code, it will have plenty of training on game templates and simple games.
Obviously you cannot generalize that to all software development though.
As you get deeper beyond the starter and bootstrap code it definitely takes a different approach to get value.
This is in part because context limits of large code bases and because the knowledge becomes more specialized and the LLM has no training on that kind of code.
But people are making it work, it just isn't as black and white.
> That sounds reasonable to me. AI is best at generating super basic and common code
I'm currently using AI (Claude Code) to write a new Lojban parser in Haskell from scratch, which is hardly something "super basic and common". It works pretty well in practice, so I don't think that assertion is valid anymore. There are certainly differences between different tasks in terms of what works better with coding agents, but it's not as simple as "super basic".
One concern is those less experienced engineers might never become experienced if they’re using AI from the start. Not that everyone needs to be good at coding. But I wonder what new grads are like these days. I suspect few people can fight the temptation to make their lives a little easier and skip learning some lessons.
I estimated that i was 1.2x when we only had tab completion models. 1.5x would be too modest. I've done plenty of ~6-8 hour tasks in ~1-2 hours using llms.
I recently used AI to help build the majority of a small project (database-driven website with search and admin capabilities) and I'd confidently say I was able to build it 3 to 5 times faster with AI. For context, I'm an experienced developer and know how to tweak the AI code when it's wonky and the AI can't be coerced into fixing its mistakes.
I'm building an AI agent for Godot, and in paid user testing we found the median speed up time to complete a variety of tasks[0] was 2x. This number was closer to 10x for less experienced engineers
[0] tasks included making games from scratch and resolving bugs we put into template projects. There's no perfect tasks to test on, but this seemed sufficient
Have you evaluated the maintainability of the generated code? Becuause that could of course start to count in the negative direction over time.
Some of the AI generated I've seen has been decent quality, but almost all of it is much more verbose or just greater in quantity than hand written code is/would be. And that's almost always what you don't want for maintenance...
That sounds reasonable to me. AI is best at generating super basic and common code, it will have plenty of training on game templates and simple games.
Obviously you cannot generalize that to all software development though.
As you get deeper beyond the starter and bootstrap code it definitely takes a different approach to get value.
This is in part because context limits of large code bases and because the knowledge becomes more specialized and the LLM has no training on that kind of code.
But people are making it work, it just isn't as black and white.
1 reply →
> That sounds reasonable to me. AI is best at generating super basic and common code
I'm currently using AI (Claude Code) to write a new Lojban parser in Haskell from scratch, which is hardly something "super basic and common". It works pretty well in practice, so I don't think that assertion is valid anymore. There are certainly differences between different tasks in terms of what works better with coding agents, but it's not as simple as "super basic".
2 replies →
One concern is those less experienced engineers might never become experienced if they’re using AI from the start. Not that everyone needs to be good at coding. But I wonder what new grads are like these days. I suspect few people can fight the temptation to make their lives a little easier and skip learning some lessons.
I estimated that i was 1.2x when we only had tab completion models. 1.5x would be too modest. I've done plenty of ~6-8 hour tasks in ~1-2 hours using llms.
Indeed. I just did a 4-6 month refactor + migration project in less than 3 weeks.
I recently used AI to help build the majority of a small project (database-driven website with search and admin capabilities) and I'd confidently say I was able to build it 3 to 5 times faster with AI. For context, I'm an experienced developer and know how to tweak the AI code when it's wonky and the AI can't be coerced into fixing its mistakes.
What's the link?
The site is password protected because it's intended for scholarly researchers, and ironically the client doesn't want LLMs scraping it.
Downvoted for...confidently saying how successful I was using an AI? I don't get it.