Comment by sidhsjsjhz
1 day ago
> "I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed" 15 minutes later the developer ask me if I want it fixed today or Tuesday.
Did you find out if it actually fixed the issue? I was on the other end of this last week. Reported an issue to another team, they asked Claude to diagnose, and when they gave me the response back I pushed back cause it didn’t make sense given the behavior we were seeing. Turns out Claude had hallucinated reading a log (it said it did but it didn’t). Sent it down a hole rabbit hole from there.
I’ve had the one shots you describe and they’re great, but they’re the happy path and require almost 0 skill to find. I’ve yet to see a case where developing an expert level knowledge of your domain isn’t the best way to get good at LLMs. Knowledge of how to interface with these tools is helpful but changes (and depreciates) rapidly as the models get smarter and/or other people commoditize it. Deep subject knowledge is still the best way to get a lot out of these tools. This applies to development and other areas I have a deep knowledge of.
“LLM training” is kinda snake oil. Learn your craft deeply and you’ll be able to catch up on LLM training in a few days, but you’ll be light years ahead of a person who’s only expertise is different ways to prompt.
To be clear, that one shot prompt was supported by nearly a years worth of work around developing an enviroment that allows me that level of effortless automation.
The issues was really simple, I knew the application, I knew what the bug was and already had a workaround in my head. I simply stopped at going down that rabbit hole and told claude to deal.
It identified the issue quickly, found the workaround and made the developer aware. All those things I would have done I just watched it do for me, plus, the app is vibe coded anyways, I just need it fixed and regardless of the AI path it was faster shooting out the bug report. Which, was simply poor processing of a spreadsheet.
> The issues was really simple, I knew the application, I knew what the bug was and already had a workaround in my head.
If that's the case, Claude isn't gaining you anything. The hard (and time consuming) part of programming is finding the correct solution, not typing the characters into an editor. When I know the code and have a good idea of the solution, I can write the code to fix it in mere minutes.
If only it was that easy, we are gated behind controls and processes that consume a significant amount of time between finding the bug, and a production push. AI quite literally automates the stuff I don't want to do, like create a fill out a jira ticket in the exact way that team wants it.
I don't have time for that, and neither do the teams that consume these request. As time has gone on me and the other teams I work with are no longer touching code, the AI will follow processes and procedures that pre date AI just fine with some human reviewing and approving.
AI has turned what was weeks at minimum to get a bug fix out, to just doing it all with with some human gating. It's not been a smooth process but watching it play out in real time it's getting faster, more efficient, and less prone to problems.
And that the crux off the situation, patching and securing configurations need to be done much faster than today, project glasswing has shown us that AI can scale to identify and solve those problems faster than a human.