Comment by piva00

AI is not doing 30-50% of our work in a company of ~10k employees.

It's helping with a lot of toil work that used to be annoying to do, PMs can do their own data analysis without having to pull me out of deliverable tasks to craft a SQL query for something and put it up on a dashboard; I don't need to go copy-paste-adapt test cases to cover a change in some feature, I don't need, most times, to open many different sections of documentation to figure out how a library/framework/language feature should be used.

It's a boost to many boring tasks but anything more complex takes as much work to setup and maintain the environment for a LLM to understand the context, the codebase, the services' relationships, the internal knowledge, the pieces of infrastructure, as it does for me to just do the work.

I've been hybridising as much as I can, when I feel there's something a LLM would be good at I do the foundational work to set it up, and prompt it incrementally to work on the task so I can review each step before it goes haywire (which it usually does), it takes effort to read what's been generated, explain what it did wrong so it can correct course, and iteratively build 80% of the solution, most times it's not able to completely finish it since there's a lot of domain knowledge that isn't documented (and there's no point in documenting since it changes often enough). Otherwise it's been more productive to just do the work myself: get pen and paper to think through the task, break it down after I have a potential solution, and use LLMs to just do the very boring scaffolding for the task.

Does it help me to get unstuck when there's some boring but straightforward thing to do? Absolutely. Has it ever managed to finish a complex task even after being given all the context, setup the Markdown documentation, explain the dependencies, the project's purpose, etc.? No, it hasn't, not even close, in many cases it gave me more work to actually massage the code it wrote into something useful than if I had done it myself. I'm tired of trying the many approaches people seem to praise about and see it crumble, I spent a whole week in 2 of our services writing all the Markdown files, iterating through them to fix any missing context it could need, and every single time it broke down at some point while trying to execute a task so, for now, I just decided to use it as a nice tool and stopped getting anxious about "missing out".