Comment by JauntyHatAngle
1 day ago
When I hear this I do wonder to myself how they're using AI.
For me, as long as I'm properly RPI looping it and not blindly pressing "yes" then it will nearly always reach the solution, usually a fair bit quicker, because it effectively becomes an ideation machine that can keep more thoughts and knowledge in it than my brain can.
However, I'm using it through giving it the context, it has access to the repos, access to the sytems, I'm telling it where the logs are, I have docs to show it what each part of the system is for.
Along with that, it does depend on my own instincts/knowledge for me to read its response and for me to say "no, you've over-thought/engineered this and this is actually the better solution", but its very are nowadays for me that it can't find the 1 liner, as long as I've fed in the right context, mostly pre-done because I've spent a bit of time building out the context tree for the repos/systems in a way that allows it to know what it needs to know.
I'm not doubting that people spin their wheels and couldn't find the one liner, but if its as simple as you say, that does seem like people who aren't great at LLMs along with a lack of instinct/experience.
I guess in essence, I think you can use LLMs in an old school troubleshooting way, and I find it still speeds that up the majority of the time. Its basically how I use it most of the time. And like old-school troubleshooting, if you build out the LLMs context over time, it also grows in capability, as long as its being used as a tool and not blindly trusted.
Should I not assume that most senior developers who are serious about LLMs do this?
In my experience at a large bank with unlimited AI, my spend is in the top 5% and I'm leveraging AI just like you.
I was in a meeting with someone who had a bug in an application that I don't own myself on Friday so I told claude,
"I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed"
15 minutes later the developer ask me if I want it fixed today or Tuesday.
That user could have done the same thing as me, had access to all the same systems and tools as I have, and also received the same AI training I had. The difference is that some users are just not, for lack of better words, AI native.
> "I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed" 15 minutes later the developer ask me if I want it fixed today or Tuesday.
Did you find out if it actually fixed the issue? I was on the other end of this last week. Reported an issue to another team, they asked Claude to diagnose, and when they gave me the response back I pushed back cause it didn’t make sense given the behavior we were seeing. Turns out Claude had hallucinated reading a log (it said it did but it didn’t). Sent it down a hole rabbit hole from there.
I’ve had the one shots you describe and they’re great, but they’re the happy path and require almost 0 skill to find. I’ve yet to see a case where developing an expert level knowledge of your domain isn’t the best way to get good at LLMs. Knowledge of how to interface with these tools is helpful but changes (and depreciates) rapidly as the models get smarter and/or other people commoditize it. Deep subject knowledge is still the best way to get a lot out of these tools. This applies to development and other areas I have a deep knowledge of.
“LLM training” is kinda snake oil. Learn your craft deeply and you’ll be able to catch up on LLM training in a few days, but you’ll be light years ahead of a person who’s only expertise is different ways to prompt.
To be clear, that one shot prompt was supported by nearly a years worth of work around developing an enviroment that allows me that level of effortless automation.
The issues was really simple, I knew the application, I knew what the bug was and already had a workaround in my head. I simply stopped at going down that rabbit hole and told claude to deal.
It identified the issue quickly, found the workaround and made the developer aware. All those things I would have done I just watched it do for me, plus, the app is vibe coded anyways, I just need it fixed and regardless of the AI path it was faster shooting out the bug report. Which, was simply poor processing of a spreadsheet.
2 replies →
Often though working through a bug teaches the developer something about the relevant abstraction. That knowledge can get lost in this new process.
While you're right, and I'm constantly thinking/worrying about my own processes giving me AI apathy, you can use AI in a way that develops your understanding of a bug. It really depends on your workflow.
When I use AI to debug a problem, I'm constantly questioning the specifics because I want to learn it, and often I'm better than the AI at quickly recognising patterns that point to a specific issue, where the AI will spin for some time trying to work out where to start - even if it has full context.
So I guess I'm half with you, but an LLM used right can still teach you effectively.
1 reply →
Claude will happily add a null guard without checking why the null is there in the first place...
I find this story awkward. When Claude "..tell the developer what needs to be fixed" was it impersonating you?
Regardless, why not help the developer use the tools effectively instead?
I explained in another thread, but yeah, it acts on my behalf using my access to take actions with my approval. It has knowledge of all my work and leverages various data sources, (Jira, Confluence various mcp's) to gather information and take actions. If I ever have to correct it, or if the actions it takes are going down the wrong path I make sure and understand why and correct that.
It just got really good over time, especially as new processes get brought it and things change. I can't keep up with that like that AI can, just have it review and go down the path of ensuring that it fit's in my workflow as efficiently as possible and move on.
As far as the developer? I gave him ideas but he's on another team and I got my own guys to feed.
> RPI loop
What's that?
Research Plan Implement.
At the risk of you already knowing this or similar - in essence for a slice of a piece of work, you get the LLM to research the information required into (usually) a research file, which you go back and forth with in-line or via conversation until you're happy.
You then build the plan for the piece of work, again going back and forth with the LLM until you agree with the plan, with the key here being you have a specific set of concrete steps with very limited vagueness. Again this is quite often a plan document.
You then implement the plan with your LLM, following the concrete steps and monitor/evaluate the result.
For me its the 101 mandatory first way to use LLM that turns it from guesswork to genuinely useful. (that or some other similar method, SDD or whatever)
Because without it or something similar you're fully dependent on AI's weaknesses not coming to the fore (context/scoping failures that are silent unless you demand visibility up front and ruthlessly chop/clarify), and you often ascribe bad AI slop as model limitations instead of context/scoping failures.
Ah, that makes sense. I'm doing the same thing but I knew it as Spec Driven Development (SDD, I guess).
I have no idea why you're being downvoted. Your comment makes a lot of sense and I've upvoted it...