Comment by bob1029
1 day ago
A natural evolution of engineers losing touch with the customers and users.
I'm noticing some of the concern play out regarding AI weakening the capabilities of software people.
I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.
I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.
> 30 minutes of patient, old school troubleshooting
This is also the best way to understand a codebase, and it's quite enjoyable.
I think the models are getting too egotistical. They're so confident of their fixes they won't bother suggesting basic techniques like isolation of the problem by disabling code paths, for example. They'll keep shotgunning less-and-less likely fixes with undiminishing confidence until the code is full of fixes that do nothing. Then they'll double down on why these should be kept.
> and it's quite enjoyable
Used to be somewhat enjoyable. Nothing pleasant about digging around codebase that was heavily affected by the last 12-18 months of AI-ing.
We have a tool at work that was entirely spun out of a prototype that itself was entirely driven by a single guy with an unlimited Claude budget. I only touch it on the edges, where I need to integrate it every so slightly into one of my systems. It's a crucial piece of software for the team, though. I've looked over the source, the thing that strikes me the most is the verbosity. Human software engineers would've been much more terse, so the thing has a fucking lot of code that's doing very little. They're trying to deal with that now, while also driving feature requests forward, and from the distance it looks like an unmanageable shit show.
6 replies →
Depends on the prior state.
For one codebase I work in, LLM written code is noticeably and measurably (we have literally measured it for bugs, performance, etc.) better than what existed before. So the LLM code is a welcome change.
16 replies →
The field is already exploding in SLOC. If LLMs can generate 10x but only produce a 2x productivity boost in reasoning or understanding - each year -then neither humans or AI will be able to keep up with the mountains of slop that power everything from websites, telecom, banking, automotive, aerospace, industrial, medical, and renewables.
Guess we’ll need even more human software engineers to fix it.
4 replies →
Also if your entire stack is on the cloud, mess of lambdas and other proprietary services, difficult as hell to follow logs, can't really run locally.
[dead]
[dead]
I have no evidence at all, but could it be that repeating the same thing over and over again in the context makes models latch on to it wrongly and gives them more confidence despite it being wrong? Kind of like a sunk cost fallacy…
Although I do acknowledge sometimes they too point out something I thought was quite right but turns out of be wrong…
A session is a screenplay between someone called User, and someone called Assistant who is a helpful and competent assistant to the user. Harnesses bring in tools to the screenplay.
A screenplay where a character has done a certain thing is one where they're more likely to do it again.
It's the uncanny valley of AI. It's still not quite good enough yet that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
EDIT: don’t get me wrong. I still think AI is incredibly useful for a lot of tasks! But when implementing an architecturally hairy thing, I find it less stressful and equally quick to jump down to the editor level and use AI just for code completion.
Pretty much, The one thing I use it for is as a sanity check, pretty much "Look at <SomeFile>, point out issues you see, summarise them tersely" and it'll spot stuff a code review by a human might have spotted (in the mythical land where people actually do code reviews properly and don't just flag a spelling mistake to "show they looked at it").
Beyond that I don't trust it at all and I still write all my code the meat sack way.
Trust is earned not given and it hasn't earned it yet.
If anything, I think this hype cycle is fast exposing just how many people, teams, and companies just don't care about what is correct. They just wanna feel good about themselves and get paid.
I for one welcome the fact that this whole thing has driven me back to books and deeper into the fundamentals. I have never read so much on math, hardware, and history as in the past 3 years or so.
7 replies →
My advice is to try letting the agent fill in the gaps. You can probably architect better than it can. Write your class outlines, explicitly define the public facing bits and what you want APIs to look like. Write the key integration tests that you know ought to pass.
The real advantage is that agents routinely write code without any silly copy/paste mistakes like accidentally accessing x twice on a coordinate operation instead of x and y. You can add some comments for what the function should do, throw in some real/pseudo code and let the LLM figure it out.
> It's the uncanny valley of AI. It's still not quite good enough yet that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
You might as well have left it with: "It's still not quite good enough yet that you can trust it". That's the core of the issue. It doesn't matter what you ask it to do, it can't be trusted. Some things are just easier to verify and correct than others.
> That's the core of the issue. It doesn't matter what you ask it to do, it can't be trusted.
I dunno about that - whenever I ask it if I'm any good, I remain confident that it will assure me that I am!
Nah, It's not even good for small code changes.
Try using it with Ansible. It spits back complete buffoonery.
I was investigating an ansible playbook yesterday that had a 45 line comment to explain a single apt install command, completely and utterly useless. I am updating my neovim to just collapse all comments, the noise is unbearable.
6 replies →
I agree, except for the use of the word "yet" .
I think what's missing is fundamental. I think the reason it sucks so much to work with LLM-generated code is that LLMs will never "know" what it's like to be human. They don't "understand" our frustrations and motivations, and they're missing the vast array of useful mental tactics we've evolved to cope with corporal existence.
At this point I think progress towards a good colleague bot would require a new architecture which allows continuous leaning, and for the LLM to be raised as a human child (maybe in a simulation at 1000x speed or something).
I'd say it's more about learning how to organize your work more efficiently.
If you think about a product like marble: it's something that most be chiseled out of time.
Some people can chisel better products: the AI is just a better chisel.
Sometime still has to guide the chisel and judge the art/product.
In our cases, the market judges products.
I think my point is that its sometimes (!) easier to use the manual chisel rather than go for the automatic chisel and then fix its mistakes.
3 replies →
> In our cases, the market judges products.
It happens, but it's rare. When last did a product take over a market without 100s of millions, sometimes even billions, of VC dollars?
There is no motivation to build a better mousetrap today, because the drooling idiot with a Claude account will look at how quick you signup users, clone it in a week (hey, it only needs to be superficially the same), and get VC money to dump until you go out of business.
This seems to misunderstand the parent comment. The sibling comment plays along with the analogy and mentions having to fix the mistakes of an automatic chisel, but the original comment is alluding to the fact that it is definitely not an automatic chisel. There is no uncanny valley in an automatic chisel.
Programming up until this point was done by using deterministic tools to build products. LLMs appear to be nondeterministic tools in their current incarnation, at least to humans.
If an automatic chisel had a feature that could switch from chiseling from sedimentary rock to chiseling marble but would randomly and nondeterministically switch to the other mode during its use it would be considered defective. But with LLMs the industry has collectively decided that the nondeterministic automatic chisel builds so fast that the current defect rate from the nondeterminism is acceptable.
It would make sense to isolate the last line of defense from LLMs, ie the tests, but this rarely seems to happen any more. Once the tests are contaminated with LLMs all bets are off.
And people forget that along with atrophying skills and reasoning due to less coding, the skill degradation is hastened because the programmer is faced with the reality that they would have to constantly figure out, review or edit someone else's code (ie, the LLM's) if they truly wanted to maintain a last line of defense. But as this type of task is literally the least liked task in programming, the programmer passes it off to the LLM as well to avoid burnout...
3 replies →
I think the speed/context size of the large models is a threshold. I've been using a local model and watching it do killer stuff, and also shit out useless things; all in real time, requiring active steering.
Your assumption is that LLMs will ever leave this uncanny valley.
Maybe unforeseen breakthroughs and different architectures are achieved. Given LLM fundamental shortcomings grounded in mathematics and information theory, I highly doubt they will and we will always need to deal with these issues in some capacity.
I think that bigger context windows help, but I feel that for AI to cross this chasm, it needs to be able to encode more abstract context knowledge. I think the holy grail here is online learning.
Agreed. For AI to be something you can trust to operate autonomously, it needs to actually be able to understand the things it is working with and reason about them. LLMs cannot, by their very nature, do that. There can be no reliability with such a tool.
[dead]
I'm seeing this happen in the security space right now. Someone on my team I was helping train and bring along is all of sudden regressing in their understanding of the issues we're working on, and instead focusing on AI tool outputs to do their job for them.
I can share a weird story:
Usually, I take my time to understand each keyword of the code I'm looking at, especially if it is new to me, like terraform.
I work in a team/with one architect, who only did the DevOps/Infra stuff for the past years and I had the expectation he knows what he is doing and talking about.
At around 2 weeks, I noticed how his knowledge has severe gaps and how he takes things at face value or uses terminology interchangably, which confuses me. It sounds plausible, but it does not actually translate into a working system or shared understanding.
Then one day I did some pair programming with him and whenever there was an error or a resource missing, he would type it into the LLM, copy paste it out of it and then brute force error messages. He did not even wait a second to think or reconcile whats happening on the screen or what the exact requirement is. Never taking one step back and questioning any assumption.
Now that the timeline is shifting and everyone starts to be stressed, he continues to vibe code through me and it is so tiring, there is no higher level planning or architecture, its just a reactive type of trial and error to be faster. It feels like these people are so used to talking to bots, that they treat you like an agent they can chat to or talk through monologs with.
It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...
So right now I'm considering quitting IT for a couple of years until people calm down, but I think its pretty futile
I don't think it's generalizable. The kind of person who copy and pastes from the AI is the kind who did the same from StackOverflow before. It's more compelling, and we probably see more of them because of it, but it's the same general thing.
The kind of person who insists on understanding things and working through the problem has always been rarer. It's not "humble", it's "inquisitive" and "persistent".
13 replies →
Great comment.
At my place, this is what they want. They want people to smash through things as fast as possible. They don’t want people to sit and craft a solution which takes in to account the whole. They are choosing tools which are low code, and use llm’s to produce what they need. as they say “this is the way things are going”.
22 replies →
> It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...
That was never humility. That was fear. This technology is just enabling people to show their through colors. The unearned confidence is a hell of a drug.
You captured this phenomenon very well in this comment. Appreciate you sharing it because it’s hard to describe exactly what makes this sort of behavior so bizarre.
3 replies →
I don't think this is going to change because it helps automate code production. As in, once we reach a point where this method produces acceptable results, we can fire all engineers and automate the process.
>> he continues to vibe code through me
So quit pair programming. I never did, never will do that, nor worked at a place that remotely encouraged that. Each to their own, that's how it should be.
1 reply →
In my mind, this, not copyright or water use, is the best reason to boycott AI. It'll make you incompetent.
I saw this the past year - new employees would put problems into Claude first instead of debugging. A year back I was debug manually first, now I do the same.
The speed AI debugs at is incredible and yes, we lose touch the more we use it like any manager feet up barking orders to their underlings to get things done.
At least at my company the OKRs are quite clear and demand heavy AI utilization above all else
To be fair the cyclical nature of funding and defunding security teams which causes scaling up and scaling down, has always left a race to the bottom in security.
Any CISO or head of security loves the ideal of completely using AI to handle incidents, tune detections, implement mitigations, track vulnerabilities, pen testing, etc. This feels like it driving security teams to have less critical thinking.
What is your relationship to this team? Their manager? A senior colleague? Something about this sounds like a failure of communication or leadership. Why are you stand offish from them? Why are you treating them like children? If they don't know what they're doing why did you hire them? Now that you did why are you treating them like a burden and not an opportunity to mentor, teach, and uplift?
You can't force someone to improve themselves. AI-fueled ego trips are really difficult to sublimate. The only viable solution in this case is to overwhelm the pupil with mastery. Demonstrate that even without AI assistance that you can run circles around their solutions. If someone actually came to me and asked for advice on something, I cannot imagine I would turn it down. Many developers are absolute monsters. Pretending like that is not the case only makes the whole thing worse.
I also read that comment as an adversarial situation at work.
It used to be that when someone else at your company was asking for something that wasn't a priority, you would erect bureaucratic roadblocks to protect your time. Now, the new normal is to just forward their questions to AI and sling the slop back over to them.
Someone outside the team "giving you a solution" is seldom a great thing in my experience
1 reply →
When I hear this I do wonder to myself how they're using AI.
For me, as long as I'm properly RPI looping it and not blindly pressing "yes" then it will nearly always reach the solution, usually a fair bit quicker, because it effectively becomes an ideation machine that can keep more thoughts and knowledge in it than my brain can.
However, I'm using it through giving it the context, it has access to the repos, access to the sytems, I'm telling it where the logs are, I have docs to show it what each part of the system is for.
Along with that, it does depend on my own instincts/knowledge for me to read its response and for me to say "no, you've over-thought/engineered this and this is actually the better solution", but its very are nowadays for me that it can't find the 1 liner, as long as I've fed in the right context, mostly pre-done because I've spent a bit of time building out the context tree for the repos/systems in a way that allows it to know what it needs to know.
I'm not doubting that people spin their wheels and couldn't find the one liner, but if its as simple as you say, that does seem like people who aren't great at LLMs along with a lack of instinct/experience.
I guess in essence, I think you can use LLMs in an old school troubleshooting way, and I find it still speeds that up the majority of the time. Its basically how I use it most of the time. And like old-school troubleshooting, if you build out the LLMs context over time, it also grows in capability, as long as its being used as a tool and not blindly trusted.
Should I not assume that most senior developers who are serious about LLMs do this?
In my experience at a large bank with unlimited AI, my spend is in the top 5% and I'm leveraging AI just like you.
I was in a meeting with someone who had a bug in an application that I don't own myself on Friday so I told claude,
"I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed"
15 minutes later the developer ask me if I want it fixed today or Tuesday.
That user could have done the same thing as me, had access to all the same systems and tools as I have, and also received the same AI training I had. The difference is that some users are just not, for lack of better words, AI native.
> "I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed" 15 minutes later the developer ask me if I want it fixed today or Tuesday.
Did you find out if it actually fixed the issue? I was on the other end of this last week. Reported an issue to another team, they asked Claude to diagnose, and when they gave me the response back I pushed back cause it didn’t make sense given the behavior we were seeing. Turns out Claude had hallucinated reading a log (it said it did but it didn’t). Sent it down a hole rabbit hole from there.
I’ve had the one shots you describe and they’re great, but they’re the happy path and require almost 0 skill to find. I’ve yet to see a case where developing an expert level knowledge of your domain isn’t the best way to get good at LLMs. Knowledge of how to interface with these tools is helpful but changes (and depreciates) rapidly as the models get smarter and/or other people commoditize it. Deep subject knowledge is still the best way to get a lot out of these tools. This applies to development and other areas I have a deep knowledge of.
“LLM training” is kinda snake oil. Learn your craft deeply and you’ll be able to catch up on LLM training in a few days, but you’ll be light years ahead of a person who’s only expertise is different ways to prompt.
3 replies →
Often though working through a bug teaches the developer something about the relevant abstraction. That knowledge can get lost in this new process.
3 replies →
I find this story awkward. When Claude "..tell the developer what needs to be fixed" was it impersonating you?
Regardless, why not help the developer use the tools effectively instead?
1 reply →
> RPI loop
What's that?
Research Plan Implement.
At the risk of you already knowing this or similar - in essence for a slice of a piece of work, you get the LLM to research the information required into (usually) a research file, which you go back and forth with in-line or via conversation until you're happy.
You then build the plan for the piece of work, again going back and forth with the LLM until you agree with the plan, with the key here being you have a specific set of concrete steps with very limited vagueness. Again this is quite often a plan document.
You then implement the plan with your LLM, following the concrete steps and monitor/evaluate the result.
For me its the 101 mandatory first way to use LLM that turns it from guesswork to genuinely useful. (that or some other similar method, SDD or whatever)
Because without it or something similar you're fully dependent on AI's weaknesses not coming to the fore (context/scoping failures that are silent unless you demand visibility up front and ruthlessly chop/clarify), and you often ascribe bad AI slop as model limitations instead of context/scoping failures.
2 replies →
I gave the team an exact solution on a silver platter and they still failed to identify how to go about it
I think what's happening is ... poorly aligned and led engineers [in] this ego inflation feedback loop where they are completely detached from reality
A story about a team of humans with some very human problems.
I've felt that AI can figure out and fix 90% issues, but it rarely does minimal, non invasive fixes. That still requires manual effort. But going from a broad to minimal fix is still a different skillset from actual debugging, so in the the end it does lead to skill atrophy.
I it usually doesn't get me in this weird state of mind, but I once spent 6 months (all-in) building a thing that I, once finished, just left alone completely (on disk gathering dust). Weird experience. So I'd say AI physchosis is real.
But was this team capable of patient, old school troubleshooting before they used AI? I’ve seen someone paid as a senior engineer define the root cause of an outage as “that code was written before our team started here”. (I spent half an hour arguing with him and then ended the RCA).
> A natural evolution of engineers losing touch with the customers and users.
I disagree. I think this happens as soon as the MBAs come on board, where everything becomes a metric, and you work towards OKRs.
> The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.
This critique (if it can be termed such) admits that the code itself has little value especially compared to the work of “30 minutes of patient, old school troubleshooting”.
In my past experience, developers who critique more-junior colleagues in this way work better in isolation than as part of a team because working as part of a team would require mentoring colleagues through the often difficult process of troubleshooting.
Giving “the team an exact solution on a silver platter” may not be helpful if implementation depends on knowledge withheld because one does not have the resources, ability, or motivation to mentor.
Something about giving a fish rather than teaching to fish comes to mind.
It's okay, they'll fix it by asking AI to design their training courses for them. That'll fix everything, right?
Are your silver platters producing something faster, better, or cheaper?
Do your silver platters give you some competitive edge? If not, then is the ego problem yours, or your coworkers?
I think LLMs have some of the same risks and benefits of stimulant drugs. They can make you more productive if used effectively as a tool, but they can also delude you into thinking you are better than you are and create a dependence such that you aren't just less productive without the LLM/drug, you fail to be productive at all because you don't know how to function without it.
That sounds somewhat applicable to many tools. Like Vim/Emacs, for example. Or computers and smart phones in general.
Yeah, like caffeine is to meth
3 replies →
Computers - mostly no.
Smartphones - mostly yes.
Smartphones are fully portable which makes them always available and super addictive.
Computers have 10% of that addictive power.
Sincerely , I think you're blaming the AI incorrectly there. You just got incompetents on your payroll.
So how do you build competence in a world where AI is preached to be the most reasonable way to solve problems because it's supposed to be faster than humans?
The same way people have been doing so for years before ai. You may or may not remember, but the parent comment is basically a 1:1 copy of people complaining about how stack overflow provided young devs all the answers.
Some people took the answers and learned from them, incompetents just copy pasted them and called it a day.
The share of incompetence just went up significantly over the years, so its a lot more noticeable at this point.
Have a single human AI chef. Everyone else has to write an engineering statement and submit it to the AI chef. All that interaction is outside the codebase. Engineers will take turns - perhaps 1 month stints - being the AI chef.
I guarantee you'll spend less on tokens, have better documentation, better code, and most importantly more competent engineers.
1 reply →
That's like asking "how do you build competence in plowing a field with a horse in a world of tractors"
4 replies →
Incompetence is implied by slop usage.
incompetent people, surely?
Are your silver platters producing something faster, better, or cheaper?
Do your silver platters give you some competitive edge? If not, then is the ego problem yours, or your coworkers?
Having said all that, I'm aware of the intoxicating effects of feeling empowered from knowledge. There's an old saying: a little learning can be dangerous....
> “A natural evolution of engineers losing touch with the customers and users”
I’m a neophyte to software teams. I work at an accounting consultancy and deal with lots of SaaS platforms. I’m pissing in the wind about this or that product quality issue through the support queue on a weekly basis. There’s prolly an XKCD comic about it.
It’s a rare company that responds to the issues raised through support. Anything that could make engineering more responsive without degrading the product quality would be welcome to us.
That happens all the damn time... Some otherwise competent developer decides to use AI, and falls into the pattern where AI will pretend to solve their problem for a week, while if they would have to think about it for an entire long hour to discover how to solve it by themselves.
As a bonus, that person will spend half of the week asking their peers for crazy delusional stuff.
I think I've heard late-career engineers from the past century say similar things, although I can't remember exactly quotes.
I wonder if every generation of engineers ends up with such hand-wringing about the next.
I'm also genuinely curious what fraction of the time they were right to worry.
I don't think it's engineers, it's the rest of the org insulating the tech workers from every side of the business
I think there are many cases where it was the tech workers themselves who argued for isolation from the customer so that they may focus harder on whatever tasks. I used to be one of these workers. I argued very hard for it. I regret that today.
On the surface it seems rational, but it quickly turns into a system of perverse incentives because now the development team must maintain an illusion that they are constantly overwhelmed with tasks and could never hope to spare a microsecond to assist the customer. This misalignment is how you wind up building your own web frameworks and databases from scratch. It turns into a self serving monster that eventually dominates the entire business. From the perspective of the business, many of these development teams look like they're behind some modern day iron curtain.
If the solution is so simple, why claude did not found it? At this point we can assume, it is better than 90% of engineers (including me).
After three decades of outsourcing to lowest bidder, I do not buy that humans are somehow better!
> patient, old school troubleshooting
I usually see similar arguments around systems with major red flags (no docs, poor CI, decade ago no CVS...). And engineers with private stash of workarounds for job security!
Claude does not do anything special.
Or perhaps claude was misconfigured, it had no access to relevant part of system, and it tryied to work within its limitation. Often it means decompiling binaries in desperate loop...
Claude regular spits out six helper functions instead of... A twenty line for loop. It overengineers most things.
Overabstracting, deduplicating things that don't need to be. Building metaclasses because it saw a single orchestrator in the whole codebase.
If it is a better engineer than you... You need practice.
The funny thing is that if you never understand the codebase then you will keep thinking Claude is doing a great work delivering all this incredible software, when all it has done is created unnecessary tech debt.
3 replies →
> A twenty line for loop. It overengineers most things.
Anecdote I like to tell.. I was working on a financial planning software, intentionally purely vibe coded as an experiment.
I eventually discovered AI had implemented seven duplicate copies of tax calculation functions. All of them different. All of them wrong. All of them giving different answers for same input.
Not even the most junior of newbie junior engineers would do something this crazy. But AI was happy to do it. It will solve the immediate problem, efficiently. Even if the most efficient solution is something ridiculous like this.
9 replies →
Oh and how is it any different than most software engineers?
How many times I heard ORM are bad only to recreate the same shit?
How many times I heard ORM had bad performance and see 1+n stuff everywhere?
How many times I have seen tight coupling in the name of DRY?
7 replies →
I am former java enterprise dev, so yes I often code this way. Unit testing, decomposition... Some projects CI refuse to merge commits with 20 line loop and duplicated code...
But that is not a point. Claude can code tight compact loops, it just needs to be instructed to do so! If it does "enterprise code", it means it had no instructions about code style.
If your documentation, spec, agent.md does not have proper guidance on coding style... yet another red flag!
1 reply →
100%
It is my pet peeve with Claude and why I don't prefer it for most stuff
(also the comment spam - but that's a all of them in a way or another)
So after 30 years of outsourcing to the bottom 10%, you think Claude is better than the bottom 90% even though it’s so stupid that it doesn’t even know it should ask for advice or more information when it’s stuck?
It just follows instructions you give it.
Some asian devs will go for weeks without asking for help, all while giving amazing fake status reports. Loosing face etc...
2 replies →
Claude asks me all the time when it's stuck, what are your prompts?
I think some of these anti-claude comments have some face, then I get a peek into how people use them, and it seems all wrong.
The tooling is incredibly configurable, but comes out of the box with a big auto button, because thats what people want. Doesn't mean advanced users have to use it.
Because simplicity is hard and often the result of careful thought. Anybody can keep piling pile of shit on top of pile of shit which is why that sort of code is so common in our industry.
While I actually agree with you (though, outsourcing to lowest bidder would account for much of what you're seeing with humans), I just saw Bug Hunt Bench scores that gave me some pause:
https://x.com/PawelHuryn/status/2095982259761475945
https://bughunt.productcompass.pm/?preset=all
Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.
(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)
Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.
Still, basic debuging and trouble shooting is where LLM generally shine. Any model can bisect git history and isolate newly introduced bug.
If model can not automatically reproduce bug, while human manually can... you got a problem in CI.
> Luna Max performing better than Fable 5.1 High
Perhaps you are reading too many benchmarks.
Edit for answer :
I agree Luna is great cheap model.
But if Fable was hitting security limits, yet was still included in benchmarks...
What flies better? Elephant or paper plane. You can make objective benchmark about that. But not much value for logistics company
1 reply →
> At this point we can assume, it is better than 90% of engineers (including me).
Hard disagree. We absolutely cannot assume that. You can posit it, and we can have an informed debate about it. This is what irks me the most about LLM fans: they constantly try to reframe the debate to have their worldview as the agreed-upon starting point.
That final 10% is the hard part. 90% is easy.
I think people on this website have a way, way to high opinion of themselves. Barring a few exceptions, we are middling developers, or way worse than this. Just read the comments from a year or twoon how chatgpt4.5 is the best at coding or how Claude sonnet 4 is better than a lot of Devs... At least I don't have imposter syndrome anymore.