Comment by nekoashide
1 day ago
In my experience at a large bank with unlimited AI, my spend is in the top 5% and I'm leveraging AI just like you.
I was in a meeting with someone who had a bug in an application that I don't own myself on Friday so I told claude,
"I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed"
15 minutes later the developer ask me if I want it fixed today or Tuesday.
That user could have done the same thing as me, had access to all the same systems and tools as I have, and also received the same AI training I had. The difference is that some users are just not, for lack of better words, AI native.
> "I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed" 15 minutes later the developer ask me if I want it fixed today or Tuesday.
Did you find out if it actually fixed the issue? I was on the other end of this last week. Reported an issue to another team, they asked Claude to diagnose, and when they gave me the response back I pushed back cause it didn’t make sense given the behavior we were seeing. Turns out Claude had hallucinated reading a log (it said it did but it didn’t). Sent it down a hole rabbit hole from there.
I’ve had the one shots you describe and they’re great, but they’re the happy path and require almost 0 skill to find. I’ve yet to see a case where developing an expert level knowledge of your domain isn’t the best way to get good at LLMs. Knowledge of how to interface with these tools is helpful but changes (and depreciates) rapidly as the models get smarter and/or other people commoditize it. Deep subject knowledge is still the best way to get a lot out of these tools. This applies to development and other areas I have a deep knowledge of.
“LLM training” is kinda snake oil. Learn your craft deeply and you’ll be able to catch up on LLM training in a few days, but you’ll be light years ahead of a person who’s only expertise is different ways to prompt.
To be clear, that one shot prompt was supported by nearly a years worth of work around developing an enviroment that allows me that level of effortless automation.
The issues was really simple, I knew the application, I knew what the bug was and already had a workaround in my head. I simply stopped at going down that rabbit hole and told claude to deal.
It identified the issue quickly, found the workaround and made the developer aware. All those things I would have done I just watched it do for me, plus, the app is vibe coded anyways, I just need it fixed and regardless of the AI path it was faster shooting out the bug report. Which, was simply poor processing of a spreadsheet.
> The issues was really simple, I knew the application, I knew what the bug was and already had a workaround in my head.
If that's the case, Claude isn't gaining you anything. The hard (and time consuming) part of programming is finding the correct solution, not typing the characters into an editor. When I know the code and have a good idea of the solution, I can write the code to fix it in mere minutes.
1 reply →
Often though working through a bug teaches the developer something about the relevant abstraction. That knowledge can get lost in this new process.
While you're right, and I'm constantly thinking/worrying about my own processes giving me AI apathy, you can use AI in a way that develops your understanding of a bug. It really depends on your workflow.
When I use AI to debug a problem, I'm constantly questioning the specifics because I want to learn it, and often I'm better than the AI at quickly recognising patterns that point to a specific issue, where the AI will spin for some time trying to work out where to start - even if it has full context.
So I guess I'm half with you, but an LLM used right can still teach you effectively.
Yes. I agree with that as well. If you can intuit that it's (for example) an off-by-one error somewhere in your logic, asking an LLM to pinpoint that issue can save you tons of time that is better spent thinking about your abstractions. Also, I tend to ask high-level questions to the frontier models about best practices with common abstractions. But "the screen flickers here, please fix" is a bad pattern, IMO. Understanding why the screen is flickering is likely to educate you about where your abstractions are leaking.
Claude will happily add a null guard without checking why the null is there in the first place...
I find this story awkward. When Claude "..tell the developer what needs to be fixed" was it impersonating you?
Regardless, why not help the developer use the tools effectively instead?
I explained in another thread, but yeah, it acts on my behalf using my access to take actions with my approval. It has knowledge of all my work and leverages various data sources, (Jira, Confluence various mcp's) to gather information and take actions. If I ever have to correct it, or if the actions it takes are going down the wrong path I make sure and understand why and correct that.
It just got really good over time, especially as new processes get brought it and things change. I can't keep up with that like that AI can, just have it review and go down the path of ensuring that it fit's in my workflow as efficiently as possible and move on.
As far as the developer? I gave him ideas but he's on another team and I got my own guys to feed.