Comment by stack_framer
17 hours ago
My employer pays for Claude, and my approach is to use it as a better Google search. It's often not better.
Just today it made three glaring mistakes in one session:
1. It read a file in the wrong directory, because that file had the same name as the file in the right directory. It apologized when I challenged it, promising me that it would remember to "read import statements" in the future.
2. It miscounted the number of times a function was called in my repo. It said 20, while my built-in IDE search accurately showed 17. Again, it apologized when I corrected it.
3. It referred to a variable by name that does not exist anywhere in my code. It apologized, and said it was referring to a variable used internally by one of the third-party packages installed in my repo.
So many apologies.
It's the little things like this that remind me on a regular basis just how little I can trust artificial "intelligence."
I have had a ton of success in exposing AST-based tools to agents when working in large, old codebases.
Without them, not only do agents get simple things like function call counts wrong, they tend to return different results. I use this as an example when showing people how the tooling works.
Grep is fine for simple use cases. A step up from that is ast-grep and I need to explore this tool more. But I had the most success building a small pipeline that reads the old code base, parses it file by file using tree sitter, and then loads it into a SQLite database for querying. For example, I have it capture construct definitions and usages and represent those as directed edges and nodes in a single table depending on the node type. The agent is instructed on how to query it and perform interesting queries like build call graphs, or determine dependencies between domains (modularity is not great in this codebase) which is helpful for us to extract around capability lines.
I also calculate fitness statistics, and have some code to capture specific details and knowledge about this very old framework that short circuits agent work in the future. We have some “interesting” magical libraries and functions that block static analyzers from going beyond the call site. This is mitigated, and means agents don’t have to “guess”.
Making all of this available to the different team members at my work has been pretty helpful. It’s faster (fewer tool calls), cheaper (fewer tokens), and accurate.
I wish LSP servers would fill that gap, but they tend to work based on a cursor position.
Otherwise, that’s exactly the tool to help those kind of queries IMO
Is your employer forcing you to use Haiku to save on costs?
It's a small tangent but I am constantly taken aback by just how much the discourse here has slid from nerdom to dorkeyness. All the talk here used to be pendantic and technical and overly complex nerd speak, always stuck in the process itself, feeling above being a 'user', etc.
Now everything is just like above, dorkspeak. Where it reminds me so much more of kids arguing in the playground about "who would win in a fight Darth Vader or Batman"; or console-vs-PC debates. Everything is about the genuine complexity of navigating certain products, of being first and foremost a consumer of something and putting all your energy into comparing various things you are free to choose from.
Its not even like its less techincal, or more mean now, or anything like that. It's just very different and I know its been a while but it feels like it happened overnight.
I call it the redditification of HN. The trend has been massively increased with Corona, and now post AI it's essentially the norm.
People can't even discuss about any politics anymore.
Back in Obama's days there were always interesting discussions in the political threads and they were rarely insta flagged. And it was mostly nuances wrt business needs and what that means for our societies. The PRISM news also frequently got heated, and still stayed somewhat technical.
Extremely noticable compared to the platform it is today.
Tbf though, HN always had a few topics it was extremely irrational about.
Eg Apple since the start and Musk post 2012...
2 replies →
>It's a small tangent but I am constantly taken aback
I'm amused by your surprise.
>Its not even like its less techincal
I'd wager that, actually, it is less technical. We constantly see otherwise very techinical people here poking their heads up and admitting that they haven't been writing any code for 6 months or more, and the ones that brag about it are seemingly unaware that they've been reduced to being a technical PM (the ones that contest this probably havent worked with capable technical PMs). No one wants to engage with vibe-coded Show HN entries because the poster may not grasp the implementation. No one appears to be doing (or sharing) anything super novel with all this coding superpower they have suddenly attained so there's nothing technically interesting to talk about. The last time I found an AI application submission interesting was the one where the person was trying to turn their pet's random keyboard typing into a language (if I remember correctly).
If the guidelines banned any AI comments that didn't preface their comment with the model and version of AI they used, their application domain, and the programming language being used I'm sure there would be less contentious dorky debate and more technical/practical discussion.
And since many here don't see the endgame with AI resulting in anything good for their career or society, like the endless remote work debates, they comment because from a strategic point of view they do not want to cede the narrative to the other side on such an important topic. understandably. hence more dorky debates.
1 reply →
Or maybe not everyone's codebases are another TODO web app.
lol ik right?
i read some of these posts and it feels like the experience with boomers i had to help with their computers at my college job.
they did the least and expected the most.
I am skeptical of letting AI do everything as well but this does seem like your using a less capable model. Fable doesnt really do this. In my experience it does just "get" what to do given a clearly defined and measurable outcome
Are you using the best models? I feel like my experience is completely different. I am an expert in a small part of a huge monolithic codebase that I’ve worked in for years. When customers report issues that would take me days or weeks to debug, AI can figure it out on the first try.
…as far as you know, at the first glance.
Could you work out that it was wrong, given weeks to go and check its work? If so, your trust is misplaced.
You ARE taking days or weeks to go and check, yes?
>You ARE taking days or weeks to go and check, yes?
I've never taken weeks to go and check bugfixes in the before times, I don't see why I'd expect it now. Once we know what the cause of the bug is, validating the fix and writing a test for it is usually trivial.
Yes, but it’s usually only hours to validate. I understand the codebase so I have the intuition to know almost immediately if the theory makes sense. Validating and reproducing the root cause is often an order of magnitude easier than identifying a production-reported issue.
I'm amused at how many hacker news accounts saw this and immediately jumped to the conclusion that you either were using the model wrong or using the wrong model.
ive been regularly seeing this exact reaction online since November 2025, sadly :/
It's just a little baffling to see someone describe a level of performance I haven't experienced since 2025, despite frequently using the tech, as being a frequent concern.
Might be the type of projects you are working on and how much you care about performance and code quality.
Working on more complex, logic heavy projects with strong performance needs I find the models to be useful but certainly not 'one shot' on pretty much anything. And often incredibly frustrating and genuinely bad code that collapses performance and bloats systems - like what a really bad junior might write.
When I'm working on large standard crud web projects with already decent architecture and a good harness and skills, honestly they work pretty well a lot of the times, the code looks good, does what it needs and fits in with the architectural style.
I really think a lot of hn people just write simple repetitive software, and a small portion works on complex, weird, dense, and novel'ish logic projects. The two obviously don't have the same experiences.
> I'm amused at how many hacker news accounts saw this and immediately jumped to the conclusion that you either were using the model wrong or using the wrong model.
Yes, so I'll disclose it: I was using Opus 5.5, on medium effort, in Claude desktop, which has full access to my entire repo.
Now everyone can officially lambast me for "using the model wrong or using the wrong model," exactly as you say. But I find it quite interesting that one of the commenters here assumed I was using Opus 4.6, because these mistakes sound like that old version! I'm using the version released just four freaking days ago!
I expect some commenters will now say, "Oh, you should have been using Fable, you old boomer." To them I say: "Yeah, well my employer doesn't allow me to use Fable." And, in jest: "Now get off my lawn."
Exactly this. If you were using the exact model today that 6-12 months ago people here were telling you "absolutely does not make this kind of mistake anymore" they'd tell you the same thing again, just replacing $OLDER_MODEL with $NEWER_MODEL. Why did $OLDER_MODEL not make this mistake 6-12 months ago, but it does now? The answer to that question is obvious, but AI-bros cannot understand that.
It's basically impossible at this point to take these people seriously anymore.
If you're tired of shenanigans pulled by these AI companies then you should host your models locally.
My model weights don't change unless I change them.
1 reply →
I still do upfront planning and then do careful review of all AI content. The behavior the parent described above hasn't happened for me since around Opus 4.6.
The most telling one is counting function invocations wrong, because that's simply not how models work anymore. They use terminal commands and Python scripts for research like that (if not an LSP, if one took the time to set up their tools most effectively).
Combined with their attitude, I have little doubt that the parent has disabled tool calls, is working in some janky Harness like chat/Duo/Juno, or is using a severely reduced or outdated model.
What model are you using? The early stopping after hitting a wrong directory instead of continuing on, and the hallucinations suggests you are using a old model.
See my comment here:
https://news.ycombinator.com/item?id=49858166