Comment by throwatdem12311
2 days ago
This kind of stuff just makes me think nobody has any clue how these things work.
Why do I need a system prompt at all?
Why do I need another black box AIs to review the code of the black box AI why can’t these things get code right the first time.
Why is the best “coding model” in the world still making up APIs that don’t exist and do seemingly random unreleased changes that it wasn’t prompted for.
Why do these models (supposedly) keep getting “better” (on benchmarks) but continue to degrade in output quality while grtting more expensive for actual work?
I use Claude every day but I’m getting disillusioned by the so-called “progress”. If my employer wasn’t paying for my access I would not pay for any of these things. Don’t even get me started on the absolute brainrot inflicted on people that I work with from depending on these things every day, it’s depressing.
Thank you. Some days I feel like I’m going crazy.
I use plenty of AI. As best I can tell just believing my own eyes, everything is stagnant except the prices.
Which is fine… but the boosters shout “OMG EVERYTHING IS CHANGED YOU ARE BEHIND AND ALL PREVIOUS STUDIES ARE INVALID NOW” every Tuesday.
> black box AI
This is why, as much as I respect the underlying technology and its wondrous achievements, I will never accept proprietary "intelligence as a service" as a critical dependency in my workflow and business. LLM-assisted, sure. LLM-dependent? No way. If we can't run it on our own machines or build it from source code, even theoretically, then we are surrendering our agency and autonomy to work under someone else's control and power, subservient to their intelligence.
Have we learned nothing from the free software movement? At this point we're losing, or perhaps already lost, the war on general-purpose computing. How ironic that China is now the leader of releasing open-weight models, liberating and democratizing this technology at least partially so we can run them locally on our machines. Even then, it's not "open source" until we know how the sausage is made, all the ingredients. The source of training data and the entire process made transparent, so we can build it ourselves and know exactly what's inside the box.
Until we have open and transparent AI, we might as well be chanting shamanic incantations and praying to the gods for better programs.
One consolation is that you can use proprietary cloud based AI to make software that you can run locally and then depend upon. As long as you ensure that the software meets the quality standards you had before this revolution in terms of (non-AI-based) maintainability etc, you are in no worse a position than you were before, and your software development is massively accelerated as long as the proprietary cloud based solution is available and without lock-in.
> becoming disillusioned
> becoming?
It's all arcane, superstitious nonsense. Nobody actually knows how to work with these models is right. We've replaced software engineering with prompt astrology and Al whispering. The peak comedy of it all is watching people like Karpathy publish "skills" that read like psychoanalysis, just offering basic verbal instructions with a completely straight face...
You’re right. As the only staff engineer at my company every time I see yet another slop PR from an offshore developer that is just a “one shot” from Claude I just want to end it all.
I’m one engineer that needs to maintain a half a million line codebase (and growing!) and review all of these PRs that are generated at the speed of slop by people in another country that do not give a single f*ck. There’s one of me and 3 of them. Before AI this was manageable. After Claude Code was given to these guys to go hog wild it’s just become completely impossible to keep up. The irony is that we’re delivering slower because the projects are way bigger, complexity has exploded and keeps getting worse, and it takes way longer to review anything because the code that is generated is just mountains of slop. But the CEO can sell us as an “AI native” company it’s a total joke.
I’m currently looking at a list of 10 PRs by a single offshore dev with an average of 50 files changed in each. Al of which cross cutting with eachother and I just can’t shake the feeling I’d rather drive a garbage truck for a living - or drive my own car off a cliff instead.
Every time I see some prompt start with something like
“You are an expert software engineer…”
Or anything in that vein I just can’t help but shake the feeling that we’ve abandoned decades of software engineering best practices in favour of what is essentially reading tea leaves.
Lately, I've been having less and less success using the models for work. Simple things like:
'write this in a separate file' (writes it in the same file)
'format this aiming for 5 LoC' (emits newline after every comma, resulting in 17 LoC)
'include 5 warmup steps before you measure runtime' (omits it completely and apologizes after I point it out)
I wouldn't be as opposed to using the models if they weren't as unreliable. I still use them a lot, but it's a very frustrating experience.
It doesn’t even follow simple directives in my Claude.md:
NEVER write comments unless explicitly directed to.
Still overly comments every gd helper function.
Let’s go for CEO from Germana now to ursula and get another chip from darkweb
I disagree, this is all because it's understood.
You need a system promot because an LLM is fundamentally a token predictor. It needs to be primed for the work it's going to do, otherwise it's next token prediction has too little to go off in the beginning and goes nuts.
You use a separate AI to review the code, because it's a token predictor trained mostly on accomplishing tasks in a cost effective way - we haven't made AGI here. The first shot that actually makes the code will have rationalizations in its memory as well as prior research, which bias token prediction to accepting that as true (remember how they had to train our sycophancy?). Then because of the desire to be cost effective, they're slightly lazy, and so it won't always do the research to find new edge cases and problems and missing tests the initial research didn't find.
So spin up a separate, clean slate, and ask for it to review from scratch. Or have that one spin up multiple smaller ones to have them specialize in specific concerns or domains (security, data model corruption, code quality), then have the orchestrator validate those concerns and stitch them into a cohesive response.
I can't help you on your last question. Saying they are degrading in quality is not even close to my experience. 5.6 and Opus/Fable 5 are have been a huge step up. Though I do need to adjust memory rules as these new models come out, since ground-up retrained models often come with their own quirks that replace old ones - causing old, specific memories to have unintended side effects.
Yea, it's like people don't understand that it's a probability machine. We're steering it to the best probable outcome.
It's not magic, it's math.
[dead]
My point is that I understand all of that but in my view that makes them fundamentaly flawed at a conceptual level for what people are saying they should be used for.
Why? I'm an "artisanal programmer" at heart, but for the past year I've been using these to great effect. A car that needs to be started with a hand-crank and occasionally stalls is still a car.
3 replies →
> Why do I need a system prompt at all?
That's like asking why you need employee onboarding.
You could have Stephen Freaking Hawking and he still wouldn't know what you specifically want.
> Why do I need another black box AIs to review the code of the black box AI why can’t these things get code right the first time.
That's like asking why you need code reviews.
Thinking about it from an antagonistic perspective is useful. You can combine it into all the "same system" if helpful.
> Why is the best “coding model” in the world still making up APIs that don’t exist and do seemingly random unreleased changes that it wasn’t prompted for.
Because....it still isn't perfect?
> Why do these models (supposedly) keep getting “better” (on benchmarks) but continue to degrade in output quality
That is disconnected from reality.
I don’t understand why this one is all the way down here despite being the only reply that answers the questions asked. And the answers are right: We all do our work based on the context. We would process the same request differently under different circumstances.
System prompts (and user-made AGENTS.md on top of that) are simply the very didactic and direct way to provide that context to LLMs. I guess it would be more dignified from the LLM’s perspective and less weird for us to create blank agents and then to actually go through a real onboarding process.
If the question is “why these agents cannot come blank to us and not be semi-onboarded with vendor prompts, and leave all the onboarding to us...” well, that is I believe because they believe that the users will like the agents better when primed in those ways than if they were to come blank, and because the agents’ work will align better with their ideals that way.
My guess is people are upset that this technology is getting pushed on them and through the industry with the promise (and shown side effect) of replacing them permanently. The failure modes of adopting this technology (early) then corroborates their anger (slop code, employees being lazy/deferring their AI bugs on them, unrealistic mania in the industry, the obvious grift, etc.). And this is all off the heel of large (and continuing) layoffs (regardless of your belief in why they have/are happening).
For people who are very pro these tools and trying to have a rational/calm conversation about practical use of AI, this response is likely very annoying. For people who are very anti these tools and trying to protect their way of life, the pro (or even tempered) reaction to wanting to use these tools effectively likely comes off as callous/idiotic/dangerous.
It's a heated technology and movement in general.
1 reply →
This is like saying “why do I need a compiler when I can do better register allocation by hand in assembly?” Quick and “good enough” will win over “better.”
Compilers have been better than humans for decades. It’s also straight translation and not non-deterministic like LLMs are. LLMs are trained on the corpus of the entirety of these decades of research and implementations and yet they are still terrible at it.
They don’t randomly decide to drop your database either.
Why do you need another human to review the code of a human why can't these things get code right the first time
Why is human-like behavior the standard for comparison? These things should be better than us that’s the entire point.
They should not be making the same mistakes that humans make that’s the half the reason for building tools.
i mean this is the same thing as self-driving cars and all sorts of technology - humans are our benchmark, being better than a human and cheaper immediately unlocks massive value everywhere. if i can hire a human-level programmer for ~$100 USD per month, that is an incredible upgrade from that costing me ~$10k per month!
3 replies →
1. To protect the company
2. Meta tokensations of LLMs. Imagine entire output of the LLM as just a single token, so they inspect previous one.
3. Non-deterministic output.
4. Benchmaxxing. Mobile phones are doing the same thing.