Comment by juancn
15 hours ago
Not a single mention of value or revenue.
Only three metrics went up: LOC, token spend and presumably release speed/frequency.
What's the markup (profit/COGS)? is it up?
Customer satisfaction? Is NPS moving one way or another?
I don't get the obsession with more code. Whatever happened to good development practices? Many of which, if followed allow LLMs to work more efficiently ironically enough. I would rather hear that you've fixed bugs that have stumped your dev team for years, and have had next to no outages, to me that's more valuable than just slopping features out the door.
While I agree with your sentiment, very strongly actually, there has always been a tension between what I'll call "engineering teams" and "product teams". The engineering teams want the long-term platform stability, performance, etc. The product teams want to push that new feature out, quick! The overall business needs both (types of thinking and doing) to survive, of course, but not all leadership sees the world through the lens engineering team lens. I'd argue most actually lean the other way, which is understandable though unfortunate. I'd also posit that since these product teams are often "closer to the business", those people typically win favor with the real decision makers.
These new tools only exacerbate this issue.
This tension is why I refer to myself as a product engineer. Teams where non-technical product managers (ie. your "product team") become feature factories that eventually drown in their own tech debt. Teams where navel-gazing engineering managers (ie. your "engineering team") have too much power end up with premature platformization, and beautiful architecture diagrams while user needs go unmet.
The key is to understand that computers exist to serve human needs, and the less code / UI / hoops to solve a problem the better, and to think end-to-end. If you are able to do that, most systems built by committee have a lot of low-hanging fruit that can be improved without much sacrifice. In practice, political realities and operational concerns may come in the way, but I firmly believe this is the only way to successfully operate as an engineer, and it will only become more important in the age of AI.
2 replies →
If you build your platform well, then you can execute fast, but also safe.
I push "money and data are sacred"; if we lose either of those, someone cries. Build the platform slow and fast. Governance, scaling etc sits here.
Provide reference APIs and let the product team one shot whatever they want and need, build fast and experimental, but your platform enforces constraints.
Best of both worlds (though feels like we're back in backend vs frontend development)
3 replies →
I'm lucky I guess, I feel like the product team is on "my side" with respect to building a resilient system. They understand that bad architecture shows up as a fragmented and friction-full UX.
The demand for new features come from sales and executive people who want to check more boxes on a customer's checklist.
1 reply →
Features don’t usually come from product teams. They come from above: some Head Of, VP Of, and the like. Those people only ask, they know very little about execution. They want their wishes come true in a timely manner
2 replies →
Well support teams are even closer to the customer than product and I think they would advocate for perf/stability over more features.
2 replies →
I've spent most of the past two weeks trying to get my AI to stop writing so goddamned much code. It's great that I can, say, completely switch between two databases, but I wish the AIs would develop a concept of energy spend or something and spend maybe a moment checking whether something is already implemented in the code base before blasting out 2000 new lines defining a new database access layer, and also rewriting random code it shouldn't be touching for some task hundreds of lines at a time just because it can. I don't care how easy it is to write some tests that sort of kind of cover it hopefully maybe... no new code is as tested as the stuff that has been doing it in the field for years already.
Just because it has the power doesn't mean it should be used.
I'm not asking for everything to be some sort of beautiful code whose concision would lead Knuth to tears. I'm just asking that you not slather dozens of existing functions with three new parameters and dozens of lines of code each that basically do nothing, to cover a use case that I accidentally vaguely alluded to, if you read my prompt a certain way, but isn't really what I was interested in.
Totally agree but I suspect that our desires are in contrast with ai companies profits
1 reply →
One thing I've spotted several times throughout this year: the larger volume of code leads to exactly the sort of problems it always has. It just happens real fast now, with fewer people. Problems that used to take years to erupt now manifest before an MVP ships, but people just sweep it all under the rug in the name of moving fast.
The rate at which software is produced has never been higher, nor has the rate at which it is thrown away for being broken and useless. It’s not at all obvious to me that if you control for both of those rate changes that the amount of useful software with a lifespan measured in years created per developer (ie, the software that actually has long term value) has actually changed all that much.
As we all know, LOC as a metric has been around for ages, usually pushed by managers who don't understand development wanting some indicator that you are working hard.
It's no surprise that the same people are now using the same metric against LLMs except this time the LLM isn't telling them it's a daft measurement.
I drastically improved the reliability of an internal IoT device so it can tolerate all kinds of abuse (abrupt power loss during update, unreliable SD card, bootstrapping over network when the SD card is missing/damaged, fail-safe updates, recovery in a network isolated environment). This is the first release that I really leaned into AI, having been pretty skeptical before.
I think AI has been a net positive and has been used responsibly at my work. It can be used to make existing products better and more secure. You just won't naturally hear about that. There's some challenges like emerging social mores but overall good.
> I think AI has been a net positive and has been used responsibly at my work. It can be used to make existing products better and more secure. You just won't naturally hear about that. There's some challenges like emerging social mores but overall good.
It always works best when layering on top of good human code. The thing is, you start to run out of that as you lean in. Tricky.
2 replies →
> It can be used to make existing products better and more secure. You just won't naturally hear about that.
Why? I keep asking for examples of LLM being used to make code more simple and elegant (which I would say is adjacent to making software better, if not more secure), and never get a reply, not a single one. I'm not claiming it doesn't exist, I know it must exist. If only via a lots of careful massaging of system prompts and harnesses, even if it takes too much effort to be worth it, I would like to see it.
Of course people can't share internal examples, but this is all over the world for many months now, where are the open source examples? Not "something new someone couldn't have done before", but someone making their precious even neater (while also adding things to it, but not primarily). Put differently, can LLM delete code, can they curate it well, can they not write weird comments nobody wants or needs, that sound like echoes of the user prompts and responses to them, but just explain tricky parts of the code, like a human would?
Also, IMO a VERY useful thing to learn about a tool or a chemical or whatever is, "what happens if I keep applying it to the same small thing over and over?" -- because that's the level of scientist I never evolved past, the kind of stuff you do as a kid -- , whereas a lot of this seems to be more about how it feels to apply it here and there until it looks good, and then move on and never look back. Which really isn't much at all, especially given we're such a short time in. I mean, how useful were mugs made of lead? Are people aware of the (in hindsight) insane things that were done with x-rays as those were invented, before the potential for harm was known? I bet they had a great time playing with that stuff, for a while.
What if in 5 years it turns out that a few years ago, LLM got into this habit of implementing certain things in a certain way because it seemed the best to "them" (the training, whatever), but there's a security issue with that -- what do you do? You can only let more agents loose, and hope they don't miss anything, because you cannot even do something like "patch PHP" (which is not my thought, Casey Muratori mentioned that in on of those podcast things).
Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though. It just does, like suggesting we go to Mars, because everything is better there, and to do that, we have to blow up Earth. Or saying to magically be able to fly from now on, we need to saw off the branch we're sitting on. Even if I thought life on Mars was the future or that we could fly, if we just thought of it the right way, this isn't a way forward IMO.
5 replies →
We used to understand that code was harder to read than it is to write. It follows, directly, that better engineering discipline means shipping LESS code. Less to support, less to understand, means a product that is easier to maintain and extend.
Now, token calculators spew out code at a pace no human can match, which completely removes the barrier of effort - which was one of the largest natural counters to shipping massive amounts of slop.
I'll be impressed by a coding model when it writes less code than a human, and spends more time refactoring than it does just adding new layers. Until then, this vibecoding is just madness.
You have to get the agent to refactor it doesnt spontaneously do it.
Idk imo code is and has always been a liability. Like the toilet paper bear says "less is more"
1. The issue is that most people and organizations do not care about maintenance. Maintenance doesn’t get you the promotion, accolades, or immediate quarterly revenue because great maintenance tends to be invisible.
2. This is exacerbated by the lack of loyalty people have towards their organizations because their orgs don’t value or reward loyalty. That’s because leadership doesnt care about the long term view because of item #1. Hence, everyone is only doing short 1 - 4 year stints in order to get meaningful raises
3. The exceptions in the private sector are businesses that not publicly traded.
4. Even then, very few people care about quality that no one else can see
I think you are overcorrecting. LOC is highly correlated with productive output. I’m willing to bet on it.
agree 100%
it really isn't a flex ppl think it is when they're burning tokens churning agents that are working for "hours" doing work
we don't need to read or write code anymore, but we do need to write (as humans) the instructions for agents.
having ADRs, product specs, acceptance criteria, etc. that are properly maintained leads to agents working for just a few minutes and pumping out reliable code.
the more vague the work for agents, the more randomness and inefficiency there will be. that leads to bugs, constantly re-writing code, etc.
imo ofc, i'm not yet running a billion dollar company so maybe i'm wrong.
Can you please point me to any project based around spec-driven development with agents which has worked well long-term, ideally one that isn't another agentic devleopment tool?
1 reply →
I think the HN crowd overestimates the quality of software. There is a (very) long tail of truly abysmal software that people are forced to use all the time. The reason it gets to be abysmal is because it’s often back-office admin software that helps people file forms for, say, logistics paperwork or school district expense tracking. AI-native engineering teams can create dramatic improvements for that long tail because the people running that software are either incompetent or checked out, and a pull towards average quality by AIs would yield great improvement.
One of the most uncomfortable truths about our profession is that there is literally no floor to how shitty software can be and yet still make people billions of dollars.
Reminds me of an incident where I had to go pick up a missed delivery (an expensive photo printer) from the dispatch warehouse because the shipping company couldn’t get a signature after three attempts.
Whatever, I drove down to this set of huge industrial warehouses with trucks coming and going by the dozen and found the customer service office.
“That’ll be half an hour, I recommend waiting at the cafe.”
“Half an hour for what? Aren’t all the packages right here?”
“They are, but looking up the package ID in the system takes at least 30 minutes, maybe 33.”
“33!? Wait… don’t tell me, let me guess: it has been getting slowly and steadily worse over years? Predictably taking longer and longer, right?”
“Yeah, six months ago it was 27 minutes! How did you know?”
“Umm… isn’t this a bit of drag on your operations?”
“Absolutely! Everything takes forever, every shipment is delayed, … <long rant> …”
“I… bet. Why hasn’t anyone fixed it!? It’s a one line fix!”
“No budget!”
This was one of the biggest logistics companies in the country about to undergo a multi-billion-dollar merger.
Their core business software was missing database indexes even on primary key columns!
Madness.
1 reply →
> There is a (very) long tail of truly abysmal software that people are forced to use all the time.
We generally call it “enterprise software,” and it’s usually abysmal because the people buying/funding it are VP-level executives who care about compliance/reporting/visibility and who never have to actually use the software.
This actually comes up on HN pretty regularly. Or used to; like many topics it’s taken a back seat to AI.
...and now with AI much more of that dreadful software can be built with all the bullshit featuritis that said managers who don't use it think they want.
It's similar with AI-generated writing: AI can't transcend mediocrity, but it can elevate terrible writing or terrible code to be merely mediocre.
none of these types of pieces of content focus on what was created for users, its always about how sky high their "shipping" metrics are (PRs merged, LoC, etc). it makes it extremely hard for me to care, as they all boil down to "im a dev who made some loops on loops on loops that writes a lot of code and spends a lot of tokens."
i will say, at least this article says "we merge 5x more and our revert rate has stayed the same." so thats at least a metric i can start to respect
Any good product leader will tell you, sometimes not building something is actually good for the customer. Slower dev cycles have you the ability to take time with decisions and make adjustments. The obsession with shipping more code is making things worse for users and I would argue, damages the company in the long run.
Eveyone has a backlog. What happens when you clear it in a month? Do you suddenly have better ideas about what you want to build? Do you suddenly have better intuitive designs? Do you have more users or the same users using the product twice as much? In most cases, the answer is no.
AI gives you the speed to execute when you need to, but knowing what to build and how will still remain a bottleneck.
Now the real question is "is the revert rate the same because you still have the same process, or do you fix-forward more often now, so your revert-rate is because of higher-severity incidents?"
The way we do this is to have period "post project" metric checks, looking for
- are the users using the new feature (repeatedly)
- what sentiment is associated with it on customer calls (extracted via call transcripts, but ideally the relevant engineer joins the call)
I mean we (huge tech company with ~2000 devs) literally never actually revert anything, in my 2 years here I have quite literally never reverted a commit. We have a staging/release cycle that means nothing ever needs a revert, you just fix issues in followup commits on trunk
This is another one of the infinite meaningless metrics that AI cash burners love to use without context
Don't you ever merge commits that wreck CI and are too complex to fix immediately? That happens here all the time.
1 reply →
Ehhhh:
1) You can merge a "new change" to fix a merge, instead of reverting
2) It's unclear whether the substance of their individual PRs are the same. Never thought I'd want an LOC metric.
3) Really, I'm going to double down on the idea that merges could be fixing merges. Also happens to increase the merge metric.
Is your company making 5x more? Are your customers 5x happier? Are you getting paid 5x more?
What’s the point of doing 5x more?
The “more features != better” is still a valid principle IMO, and AI doesn’t help with that. Probably is making it worse.
Perhaps it shows that it does not matter how good your engineering is, or if you have sped up feature development 10x, if the shop is not providing value then who cares?
You're thinking about this at the wrong level. This org has made the decision to be AI-native. They aren't A/B testing being AI-native vs not being AI-native. That's the kind of thing "the market" does; if a company's markup is higher than another's it can attract more investment or its equity can trade for higher pricing on public markets.
I don't think it makes sense to A/B test "AI-native" vs other regimes of development unless you are a large, decentralized company like an F5 company or a Big Tech. How will a team of 25 (the size of the company if you read the thread) have enough output to even A/B test the difference?
It was the article that used metrics related to engineering velocity as evidence of success. It makes sense to ask how this is impacting their business metrics. The company is not in the business of shipping more code. I don’t see how asking how this ties back to core business results is thinking about it at the wrong level.
> It was the article that used metrics related to engineering velocity as evidence of success
Success for what? The article opened with: "Last week Will Larson (@Lethain ) asked how leaders are evaluating engineering managers for AI skills. My reply , a list of questions to think about, got a lot of bookmarks. More broadly, I've been thinking about how to build an AI-native engineering org for a while. Sharing some lessons from the last nine months of building at FurtherAI."
The opener is "how to build an AI-native engineering org" and the context is a company of roughly 22 engineers (from a reply.) Nothing here says "evidence of success"
> I don’t see how asking how this ties back to core business results is thinking about it at the wrong level.
In an company of 22 engineers, how would you make these measurements? Would you split your company into a velocity focused group and a quality focused group and make them compete? How? What would a customer motion look like? What would these groups share? This makes no sense. This is like asking a small company to A/B test teams that used a computer vs didn't use a computer.
A larger, more mature organization has the luxury of splitting business units into AI-native vs not-AI-native workflows and comparing COGS/business objectives against each other. I know of very large tech consulting services who did exactly this around the beginning of the cloud era to figure out which aspects of their business they wanted to place on clouds and which aspects they kept on-prem, and even then that was messy because part of the value of on-prem is the scale benefit you get by owning and operating your own DC. But no 22 person engineering team I saw A/B tested this out. Companies were founded either with a cloud-first philosophy or an on-prem philosophy.
5 replies →
They probably did not even think about asking customers.
why talk to the customer when a chatbot can do it for you? your obviously not thinking ai-native... /s
since when does an engineering org care about revenue or customer satisfaction? Most software devs have no idea how the business they work for operates and think their direct deposit every two weeks is a fundamental law of nature. Also, many software devs are anti-customer and think customers are wrong even when the customer is talking to them one-on-one describing their needs in plain english.
I'm sorry you had to work at (or with) a crappy engineering org. Good ones really do care about the business.
Funny you should say this… there another post on HN about engineers using AI are going to make PMs redundant!
LOC has no impact on revenue whatsoever. We've known about that for decades.
I've gone through a few recruiter screens lately and the number of people who mention "number of deploys per day" as a metric they are proud of causes my eyes to roll.
What can you possibly be doing as a product that justifies you releasing 60 times in a working day?
It's certainly better than the old models of releasing on a quarterly cadence with the test team and all that, but it feels like the metric has become the goal when you are releasing that often.
This was the case at a previous job. Someone read some "state of devops report" or similar BS where they wrote that good teams ship often. They took from that it was shipping often that made good teams... And as a result we had some contrived deploy process on Slack with buttons and emojis and what not to deploy. Not a single customer saw any value out of this of course, because shipping something is not the same as shipping something valuable.
So I guess releasing 60 times in a working day proves that you're part of a good team /s
That's why I'm sceptical about the value of ai in software development. Much more can now be done but I don't see costumers paying more for it. The only way it can work is by laying off software engineers and saving money on their salaries.
Tend to agree. AI has failed to produce more business outcome. The only thing left is cost efficiency by reducing headcount. Thus 5x more productivity means needing just 1/5th your current staffing.
and IME the #1 rule for us is work in a profit center, not a cost center. When you're cutting there's a law of diminishing returns, but the floor is so incredibly low you don't want to be near it just to clear the blade.
[dead]