I think the "software factory" terminology is very interesting, and I would imagine quite intentional.
It calls to mind the early days of the industrial revolution, when I believe the idea was that mass produced items were not better quality, just dramatically cheaper. So you still had the artisans that the rich people paid for but now poorer people had access to something they couldn't before.
Then, as technology progressed, factories started producing things that humans are incapable of. And part of this is because those factories were built on output of earlier factories.
It makes me wonder if this is where we're headed. Right now the code quality of agents isn't better than hand written code, and so arguably the products aren't either. But will there come a time when it surpasses what we can do? You can't handcraft a microchip, for example. But I guess the takeaway is maybe there's a time for both agentic lower quality but cheaper output and human software engineer higher quality output, at least for a time.
Which is exactly why agentic coding tools are effectively running at a loss right now. We are training that next generation of factories. We're paying in human cognition.
"Software factory" is a pretty conventional term used for a CI/CD pipeline with automation including DevSecOps, containerization, and agile management.
Building Attractor
Supply the following prompt to a modern coding agent
(Claude Code, Codex, OpenCode, Amp, Cursor, etc):
codeagent> Implement Attractor as described by
https://factory.strongdm.ai/
Canadian girlfriend coding is now a business model.
Amusingly, it appears the README (that would be code, right?) has hallucinated the existence of a docker image - someone filed an issue at https://github.com/strongdm/cxdb/issues/1
In-house employees don't read code or do code reviews, so presumably they don't raise issues either. I guess the issue was picked up by an astute HN reader.
I've looked at their code for a few minutes in a few files, and while I don't know what they're trying to do well enough to say for sure anything is definitely a bug, I've already spotted several things that seem likely to be, and several others that I'd class as anti-patterns in rust. Don't get me wrong, as an experiment this is really cool, but I do not think they've succeeded in getting the "dark factory" concept to work where every other prominent attempt has fallen short.
For those of us working on building factories, this is pretty obvious because once you immediately need shared context across agents / sessions and an improved ID + permissions system to keep track of who is doing what.
I was about to say the same thing! Yet another blog post with heaps of navel gazing and zero to actually show for it.
The worst part is they got simonw to (perhaps unwittingly or social engineering) vouch and stealth market for them.
And $1000/day/engineer in token costs at current market rates? It's a bold strategy, Cotton.
But we all know what they're going for here. They want to make themselves look amazing to convince the boards of the Great Houses to acquire them. Because why else would investors invest in them and not in the Great Houses directly.
We’ve been working on this since July, and we shared the techniques and principles that have been working for us because we thought others might find them useful. We’ve also open-sourced the nlspec so people can build their own versions of the software factory.
We’re not selling a product or service here. This also isn’t about positioning for an acquisition: we’ve already been in a definitive agreement to be acquired since last month.
It’s completely fair to have opinions and to not like what we’re putting out, but your comment reads as snarky without adding anything to the conversation.
So I am on a web cast where people working about this. They are from https://docs.boundaryml.com/guide/introduction/what-is-baml and humanlayer.dev Mostly are talking about spec driven development. Smart people. Here is what I understood from them about spec driven development, which is not far from this AFAIU.
Lets start with the `/research -> /plan -> /implement(RPI)`. When you are building a complex system for teams you _need_ humans in the loop and you want to focus on design decisions. And having structured workflows around agents provides a better UX to those humans make those design decisions. This is necessary for controlling drift, pollution of context and general mayhem in the code base. _This_ is the starting thesis around spec drive development.
How many times have you working as a newbie copied a slash command pressed /research then /plan then /implement only to find it after several iterations is inconsistent and go back and fix it? Many people still go back and forth with chatgpt copying back and forth copying their jira docs and answering people's question on PRD documents. This is _not_ a defence it is the user experience when working with AI for many.
One very understandable path to solve this is to _surface_ to humans structured information extracted from your plan docs for example:
In this very toy spec driven development the idea is that each step in the RPI loop is broken down and made very deterministic with humans in the loop. This is a system designed by humans(Chief AI Officer, no kidding) for teams that follow a fairly _customized_ processes on how to work fast with AI, without it turning into a giant pile of slop. And the whole point of reading code or QA is this: You stop the clock on development and take a beat to see the high signal information: Testers want to read tests and QAers want to test behavior, because well written they can tell a lot about weather a software works. If you have ever written an integration test on a brownfield code with poor test coverage, and made it dependable after several days in the dark, you know what it feels like... Taking that step out is what all VCs say is the last game in town.. the final game in town.
This StrongDM stuff is a step beyond what I can understand: "no humans should write code", "no humans should read code", really..? But here is the thing that puzzles me even more is that spec driven development as I understand it, to use borrowed words, is like parents raising a kid — once you are a parent you want to raise your own kid not someone else's. Because it's just such a human in the loop process. Every company, tech or not, wants to make their own process that their engineers like to work with. So I am not sure they even have a product here...
> If you haven’t spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement
At that point, outside of FAANG and their salaries, you are spending more on AI than you are on your humans. And they consider that level of spend to be a metric in and of itself. I'm kinda shocked the rest of the article just glossed over that one. It seems to be a breakdown of the entire vision of AI-driven coding. I mean, sure, the vendors would love it if everyone's salary budget just got shifted over to their revenue, but such a world is absolutely not my goal.
I don't understand. Are you suggesting that the proper volume of agent work is achieved with the $200/month subscription, or that $30,000/month ($1,000/day) of tokens is necessary? I think there is a big difference between $200 and $30,000.
This is an interesting point but if I may offer a different perspective:
Assuming 20 working days a month: that's 20k x 12 == 240k a year. So about a fresh grad's TC at FANG.
Now I've worked with many junior to mid-junior level SDEs and sadly 80% does not do a better job than Claude. (I've also worked with staff level SDEs who writes worse code than AI, but they offset that usually with domain knowledge and TL responsibilities)
I do see AI transform software engineering into even more of a pyramid with very few human on top.
Important too, a fully loaded salary costs the company far more than the actual salary that the employee receives. That would tip this balancing point towards 120k salaries, which is well into the realm of non-FAANG
It would depend on the speed of execution, if you can do the same amount of work in 5 days with spending 5k, vs spending a month and 5k on a human the math makes more sense.
Tokens will become significantly more expensive in the short term actually. This is not stemming from some sort of anti-AI sentiment. You have two ramps that are going to drive this. 1. Increase demand, linear growth at least but likely this is already exponential. 2. Scaling laws demand, well, more scale.
Future better models will both demand higher compute use AND higher energy. We cannot underestimate the slowness of energy production growth and also the supplies required for simply hooking things up. Some labs are commissioning their own power plants on site, but this is not a true accelerator for power grid growth limits. You're using the same supply chain to build your own power plant.
If inference cost is not dramatically reduced and models don't start meaningfully helping with innovations that make energy production faster and inference/training demand less power, the only way to control demand is to raise prices. Current inference costs, do not pay for training costs. They can probably continue to do that on funding alone, but once the demand curve hits the power production limits, only one thing can slow demand and that's raising the cost of use.
$1,000 is maybe 5$ per workday. I measure my own usage and am on the way to $6,000 for a full year. I'm still at the stage where I like to look at the code I produce, but I do believe we'll head to a state of software development where one day we won't need to.
Until we solve the validation problem, none of this stuff is going to be more than flexes. We can automate code review, set up analytic guardrails, etc, so that looking at the code isn't important, and people have been doing that for >6 months now. You still have to have a human who knows the system to validate that the thing that was built matches the intent of the spec.
There are higher and lower leverage ways to do that, for instance reviewing tests and QA'ing software via use vs reading original code, but you can't get away from doing it entirely.
I agree with this almost completely. The hard part isn’t generation anymore, it’s validation of intent vs outcome. Especially once decisions are high-stakes or irreversible, think pkg updates or large scale tx
What I’m working on (open source) is less about replacing human validation and more about scaling it: using multiple independent agents with explicit incentives and disagreement surfaced, instead of trusting a single model or a single reviewer.
Humans are still the final authority, but consensus, adversarial review, and traceable decision paths let you reserve human attention for the edge cases that actually matter, rather than reading code or outputs linearly.
Until we treat validation as a first-class system problem (not a vibe check on one model’s answer), most of this will stay in “cool demo” territory.
“Anymore?” After 40 years in software I’ll say that validation of intent vs. outcome has always been a hard problem. There are and have been no shortcuts other than determined human effort.
But, is that different from how we already work with humans? Typically we don't let people commit whatever code they want just because they're human. It's more than just code reviews. We have design reviews, sometimes people pair program, there are unit tests and end-to-end tests and all kinds of tests, then code review, continuous integration, Q&A. We have systems to watch prod for errors or user complaints or cost/performance problems. We have this whole toolkit of process and techniques to try to get reliable programs out of what you must admit are unreliable programmers.
The question isn't whether agentic coders are perfect. Actually it isn't even whether they're better than humans. It's whether they're a net positive contribution. If you turn them loose in that kind of system, surrounded by checks and balances, does the system tend to accumulate bugs or remove them? Does it converge on high or low quality?
I think the answer as of Opus 4.5 or so is that they're a slight net positive and it converges on quality. You can set up the system and kind of supervise from a distance and they keep things under control. They tend to do the right thing. I think that's what they're saying in this article.
This obviously depends on what you are trying to achieve but it’s worth mentioning that there are languages designed for formal proofs and static analysis against a spec, and I have suspicions we are currently underutilizing them (because historically they weren’t very fun to write, but if everything is just tokens then who cares).
And “define the spec concretely“ (and how to exploit emerging behaviors) becomes the new definition of what programming is.
(and unambiguously. and completely. For various depths of those)
This always has been the crux of programming. Just has been drowned in closer-to-the-machine more-deterministic verbosities, be it assembly, C, prolog, js, python, html, what-have-you
There have been a never ending attempts to reduce that to more away-from-machine representation. Low-code/no-code (anyone remember Last-one for Apple ][ ?), interpreting-and/or-generating-off DSLs of various level of abstraction, further to esperanto-like artificial reduced-ambiguity languages... some even english-like..
For some domains, above worked/works - and the (business)-analysts became new programmers. Some companies have such internal languages. For most others, not really.
And not that long ago, the SW-Engineer job was called Analyst-programmer.
AI also quickly goes off the rails, even the Opus 2.6 I am testing today. The proposed code is very much rubbish, but it passes the tests. It wouldn't pass skilled human review. Worst thing is that if you let it, it will just grow tech debt on top of tech debt.
Tests are only rigorous if the correct intent is encoded in them. Perfectly working software can be wrong if the intent was inferred incorrectly. I leverage BDD heavily, and there a lot of little details it's possible to misinterpret going from spec -> code. If the spec was sufficient to fully specify the program, it would be the program, so there's lots of room for error in the transformation.
> That idea of treating scenarios as holdout sets—used to evaluate the software but not stored where the coding agents can see them—is fascinating. It imitates aggressive testing by an external QA team—an expensive but highly effective way of ensuring quality in traditional software.
This is one of the clearest takes I've seen that starts to get me to the point of possibly being able to trust code that I haven't reviewed.
The whole idea of letting an AI write tests was problematic because they're so focused on "success" that `assert True` becomes appealing. But orchestrating teams of agents that are incentivized to build, and teams of agents that are incentivized to find bugs and problematic tests, is fascinating.
I'm quite curious to see where this goes, and more motivated (and curious) than ever to start setting up my own agents.
Question for people who are already doing this: How much are you spending on tokens?
That line about spending $1,000 on tokens is pretty off-putting. For commercial teams it's an easy calculation. It's also depressing to think about what this means for open source. I sure can't afford to spend $1,000 supporting teams of agents to continue my open source work.
Re: $1k/day on tokens - you can also build a local rig, nothing "fancy". There was a recent thread here re: the utility of local models, even on not-so-fancy hardware. Agents were a big part of it - you just set a task and it's done at some point, while you sleep or you're off to somewhere or working on something else entirely or reading a book or whatever. Turn off notifications to avoid context switches.
If they're able to communicate with each other. But I'm pretty sure we could keep that from happening.
I don't take your comment as dismissive, but I think a lot of people are dismissing interesting and possibly effective approaches with short reactions like this.
I'm interested in the approach described in this article because it's specifying where the humans are in all this, it's not about removing humans entirely. I can see a class of problems where any non-determinism is completely unacceptable. But I can also see a large number of problems where a small amount of non-determinism is quite acceptable.
This one is worth paying attention to to. They're the most ambitious team I've see exploring the limits of what you can do with this stuff. It's eye-opening.
> If you haven’t spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement
Seems to me like if this is true I'm screwed no matter if I want to "embrace" the "AI revolution" or not. No way my manager's going to approve me to blow $1000 a day on tokens, they budgeted $40,000 for our team to explore AI for the entire year.
Let alone from a personal perspective I'm screwed because I don't have $1000 a month in the budget to blow on tokens because of pesky things that also demand financial resources like a mortgage and food.
At this point it seems like damned if I do, damned if I don't. Feels bad man.
My friend works at Shopify and they are 100% all in on AI coding. They let devs spend as much as they want on whatever tool they want. If someone ends up spending a lot of money, they ask them what is going well and please share with others. If you’re not spending they have a different talk with you.
As for me, we get Cursor seats at work, and at home I have a GPU, a cheap Chinese coding plan, and a dream.
Same. Feels like it goes against the entire “hacker” ethos that brought me here in the first place. That sentence made me actually feel physically sick on initial read as well. Everyday now feels like a day where I have exponentially less & less interest in tech. If all of this AI that’s burning the planet is so incredible, where are the real world tangible improvements? I look around right now and everything in tech, software, internet, etc. has never looked so similar to a dumpster fire of trash.
I read that as combined, up to this point in time. You have 20 engineers? If you haven't spent at least $20k up to this point, you've not explored or experienced enough of the ins and outs to know how best to optimize the use of these tools.
I didn't read that as you need to be spending $1k/day per engineer. That is an insane number.
EDIT: re-reading... it's ambiguous to me. But perhaps they mean per day, every day. This will only hasten the elimination of human developers, which I presume is the point.
I think corporate incentives vs personal incentives are slightly different here. As a company trying to experiment in this moment, you should be betting on token cost not being the bottleneck. If the tooling proves valuable, $1k/day per engineer is actually pretty cheap.
At home on my personal setup, I haven't even had to move past the cheapest codex/claude code subscription because it fulfills my needs ¯\_(ツ)_/¯. You can also get a lot of mileage out of the higher tiers of these subscriptions before you need to start paying the APIs directly.
Can you make an ethical declaration here, stating whether or not you are being compensated by them?
Their page looks to me like a lot of invented jargon and pure narrative. Every technique is just a renamed existing concept. Digital Twin Universe is mocks, Gene Transfusion is reading reference code, Semport is transpilation. The site has zero benchmarks, zero defect rates, zero cost comparisons, zero production outcomes. The only metric offered is "spend more money".
Anyone working honestly in this space knows 90% of agent projects are failing.
The main page of HN now has three to four posts daily with no substance, just Agentic AI marketing dressed as engineering insight.
With Google, Microsoft, and others spending $600 billion over the next year on AI, and panicking to get a return on that Capex....and with them now paying influencers over $600K [1] to manufacture AI enthusiasm to justify this infrastructure spend, I won't engage with any AI thought leadership that lacks a clear disclosure of financial interests and reproducible claims backed by actual data.
Show me a real production feature built entirely by agents with full traces, defect rates, and honest failure accounting. Or stop inventing vocabulary and posting vibes charts.
> Every technique is just a renamed existing concept. Digital Twin Universe is mocks, Gene Transfusion is reading reference code, Semport is transpilation. The site has zero benchmarks, zero defect rates, zero cost comparisons, zero production outcomes. The only metric offered is "spend more money".
Repeating for emphasis, because this is the VERY obvious question anyone with a shred of curiosity would be asking not just about this submission but about what is CONSTANTLY on the frontpage these days.
There could be a very simple 5 question questionnaire that could eliminate 90+% of AI coding requests before they start:
- Is this a small wrapper around just querying an existing LLM
- Does a brief summary of this searched with "site:github" already return dozens or hundreds of results?
- Is this a classic scam (pump&dump, etc) redone using "AI"
- Is this needless churn between already high level abstractions of technology (dashboard of dashboards, yaml to json, python to java script, automation of automation framework)
It is tempting to be stealthy when you start seeing discontinuous capabilities go from totally random to somewhat predictable. But most of the key stuff is on GitHub.
The moats here are around mechanism design and values (to the extent they differ): the frontier labs are doomed in this world, the commons locked up behind paywalls gets hyper mirrored, value accrues in very different places, and it's not a nice orderly exponent from a sci-fi novel. It's nothing like what the talking heads at Davos say, Anthropic aren't in the top five groups I know in terms of being good at it, it'll get written off as fringe until one day it happens in like a day. So why be secretive?
You get on the ladder by throwing out Python and JSON and learning lean4, you tie property tests to lean theorems via FFI when you have to, you start building out rfl to pretty printers of proven AST properties.
And yeah, the droids run out ahead in little firecracker VMs reading from an effect/coeffect attestation graph and writing back to it. The result is saved, useful results are indexed. Human review is about big picture stuff, human coding is about airtight correctness (and fixing it when it breaks despite your "proof" that had a bug in the axioms).
Programming jobs are impacted but not as much as people think: droids do what David Graeber called bullshit jobs for the most part and then they're savants (not polymath geniuses) at a few things: reverse engineering and infosec they'll just run you over, they're fucking going in CIC.
"If you haven't spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement"
Apart from being a absolutely ridiculous metric, this is a bad approach, at least with current generation models. In my experience, the less you inspect what the model does, the more spaghetti-like the code will be. And the flying spaghetti monster eats tokens faster than you can blink! Or put more clearly: implementing a feature will cost you a lot more tokens in a messy code base than it does in a clean one. It's not (yet) enough to just tell the agent to refactor and make it clean, you have to give it hints on how to organise the code.
I'd go do far as to say that if you're burning a thousand dollars a day per engineer, you're getting very little bang for your tokens.
It's short-term vs long-term optimization. Short-term optimization is making the system effective right now. Long-term optimization is exploring ways to improve the system as a whole.
> In rule form:
- Code must not be written by humans
- Code must not be reviewed by humans
as a previous strongDM customer, i will never recommend their offering again. for a core security product, this is not the flex they think it is
also mimicking other products behavior and staying in sync is a fools task. you certainly won't be able to do it just off the API documentation. you may get close, but never perfect and you're going to experience constant breakage
Important to note that this is the approach taken by their AI research lab over the past six months, it's not (yet) reflective of how they build the core product.
If you'd like to try this yourself, you can build an "attractor" by just pointing claude code at their llms.txt. Or if you'd like to save some tokens, you can clone my go version.
https://github.com/danshapiro/kilroy
This version has a Claude Code skill to help. Tell it to use it's skill to create a dotfile from your requirements. Then tell it to run that dotfile with kilroy.
What has strongdm actually built? Are their users finding value from their supposed productivity gains?
If their focus is to only show their productivity/ai system but not having built anything meaningful with it, it feels like one of those scammy life coaches/productivity gurus that talk about how they got rich by selling their courses.
> we transitioned from boolean definitions of success ("the test suite is green") to a probabilistic and empirical one. We use the term satisfaction to quantify this validation: of all the observed trajectories through all the scenarios, what fraction of them likely satisfy the user?
Oh, to have the luxury of redefining success and handwaving away hard learned lessons in the software industry.
Not sure “Digital Twin Universe” is required here. They seem rather to have rediscovered Simulators in Integration Tests from first principles? The DTU comes off as XML Databases or Information Superhighway.
Still… a really good application of agent hands-off replication.
Seems like creating a quality negative mould and then that single negative mould makes multiple positive objects en-masse.
the agentic shift is where the legal and insurance worlds are really going to struggle. we know how to model human error, but modeling an autonomous loop that makes a chain of small decisions leading to a systemic failure is a whole different beast. the audit trail requirements for these factories are going to be a regulatory nightmare.
I think the insurance industry is will take a simpler route: humans will be held 100% responsible. Any decisions made by the ai will be the responsibility of the human instructing that ai. Always.
I think this will act as a brake on the agentic shift as a whole.
that's the current legal default, but it starts breaking down when you look at product liability vs professional liability.
if a company sells an autonomous agent that is marketed as doing a task without human oversight, the courts will eventually move that burden back to the manufacturer. we saw the same dance with autonomous driving disclaimers the "human must stay in control" line works as a legal shield for a while, but eventually the market demands a shift in who holds the risk.
if we stick to 100% human responsibility for black-box errors that a human couldn't have even predicted, that "brake" won't just slow down the agentic shift, it'll effectively kill the enterprise market for it. no C-suite is going to authorize a fleet of agents if they're holding 100% of the bag for emergent failures they can't audit.
They just are not going to provide insurance to companies who use AI because the liability costs are not worth it to them since they cannot actual calculate risks, it is already happening [0]. Its the one thing that a lot of the evangelists of using AI for entire products have come to realize or they aren't actually dealing with B2B scenarios where indemnity comes into play. That or they are lying to insurance companies and their customers, which is a... choice.
IT perspective here. Simon hits the nail on the head as to what I'm genuinely looking forward to:
> How do you clone the important parts of Okta, Jira, Slack and more? With coding agents!
This is what's going to gut-punch most SaaS companies repeatedly over the next decade, even if this whole build-out ultimately collapses in on itself (which I expect it to). The era of bespoke consultants for SaaS product suites to handle configuration and integrations, while not gone, are certainly under threat by LLMs that can ingest user requirements and produce functional code to do a similar thing at a fraction of the price.
What a lot of folks miss is that in enterprise-land, we only need the integration once. Once we have an integration, it basically exists with minimal if any changes until one side of the integration dies. Code fails a security audit? We can either spool up the agents again briefly to fix it, or just isolate it in a security domain like the glut of WinXP and Win7 boxen rotting out there on assembly lines and factory floors.
This is why SaaS stocks have been hammered this week. It's not that investors genuinely expect huge players to go bankrupt due to AI so much as they know the era of infinite growth is over. It's also why big AI companies are rushing IPOs even as data center builds stall: we're officially in a world where a locally-run model - not even an Agent, just a model in LM Studio on the Corporate Laptop - can produce sufficient code for a growing number of product integrations without any engineer having to look through yet another set of API documentation. As agentic orchestration trickles down to homelabs and private servers on smaller, leaner, and more efficient hardware, that capability is only going to increase, threatening profits of subscription models and large AI companies. Again, why bother ponying up for a recurring subscription after the work is completed?
For full-fledged software, there's genuine benefit to be had with human intervention and creativity; for the multitude of integrations and pipelines that were previously farmed out to pricey consultants, LLMs will more than suffice for all but the biggest or most complex situations.
Stuff comes in from an API goes out to a different API.
With a semi-decent agent I can build what took me a week or two in hours just because it can iterate the solution faster than any human can type.
A new field in the API could’ve been a two day ordeal of patching it through umpteen layers of enterprise frameworks. Now I can just tell Claude to add it, it’ll do it up to the database in minutes - and update the tests at the same time.
And because these are all APIs, we can brute-force it with read-only operations with minimal review times. If the read works, the write almost always will, and then it's just a matter of reading and documenting the integration before testing it in dev or staging.
So much of enterprise IT nowadays is spent hammering or needling vendors for basic API documentation so we can write a one-off that hooks DB1 into ServiceNow that's also pulling from NewRelic just to do ITAM. Consultants would salivate over such a basic integration because it'd be their yearly salary over a three month project.
Now we can do this ourselves with an LLM in a single sprint.
>> How do you clone the important parts of Okta, Jira, Slack and more? With coding agents!
> This is what's going to gut-punch most SaaS companies repeatedly over the next decade
but there's already clones of the important parts of those systems and yet the SaaS world survives. The code used isn't the secret sauce and people in SaaS know writing the code is 10% of the effort in keeping those businesses on their feet.
I don't think the SaaS industry is on the ropes until coding agents can do things like create a recommendation algorithm better than Spotify and YouTube. In those cases the code/algorithm is indeed the secret sauce and if a coding agent can do better than those companies will be left behind.
A lot of examples of creating clones of existing products don't resonate with new products we build
For example, most development work involves discovering correctness, not writing to a fullproof spec (like cloning slack)
Usually work goes like:
* Team decides some vague requirement
* Developer must implement requirement into executable decisions
Now I use Claude Code to do step 2 now, and its great. But I'm looking over whether the implementation's little decisions actually do what the business would want. Or more accurately, I'm making decisions to the level of specificity that matters to the problem at hand.
I have to try, backtrack, and rebuild all the time when my assumptions get broken.
In some cases decisions have low specificity: I could one-shot a complex feature (or entire app if trying to test PMF or something). In other cases, the tradeoffs in 10 lines of code become crucially important.
I'm just going to say: When opening the "twins" (bad clones) screenshots, I pressed the right key to view the next image, and surprise, the next "article" of the top navigation bar was loaded, instead of showing the next image.
Is this the quality we should expect from agentic? From my experiments with claude code, yes, the UX details are never there. Especially for bigger features. It can work reasonably well independently up to a "module" level (with clear interfaces). But for full app design, while technically possible, the UX and visual design is just not there.
And I am very not attracted to the idea of polishing such an agentic apps. A solution could be:
1. The boss prompts the system with what he wants.
2. The boss outsources to india the task of polishing the rough edges.
===
More on the arrow keys navigation:
Pressing right on the last "Products" page loops to the first "Story" page, yet pressing left on the first page does nothing. Typical UX inconsistency of vibe coded software.
Effectively everyone is building the same tools with zero quantitative benchmarks or evidence behind the why / ideas … this entire space is a nightmare to navigate because of this. Who cares without proper science, seriously? I look through this website and it looks like a preview for a course I’m supposed to buy … when someone builds something with these sorts of claims attached, I assume that there is going to be some “real graphs” (“these are the number of times this model deviated from the spec before we added error correction …”)
What we have instead are many people creating hierarchies of concepts, a vast “naming” of their own experiences, without rigorous quantitative evaluation.
I may be alone in this, but it drives me nuts.
Okay, so with that in mind, it amounts to heresay “these guys are doing something cool” — why not shut up or put up with either (a) an evaluation of the ideas in a rigorous, quantitative way or (b) apply the ideas to produce an “hard” artifact (analogous, e.g., to the Anthropic C compiler, the Cursor browser) with a reproducible pathway to generation.
The answer seems to be that (b) is impossible (as long as we’re on the teet of the frontier labs, which disallow the kind of access that would make (b) possible) and the answer for (a) is “we can’t wait we have to get our names out there first”
I’m disappointed to see these types of posts on HN. Where is the science?
Honestly I've not found a huge amount of value from the "science".
There are plenty of papers out there that look at LLM productivity and every one of them seems to have glaring methodology limitations and/or reports on models that are 12+ months out of date.
Have you seen any papers that really elevated your understanding of LLM productivity with real-world engineering teams?
No, I agree! But I don’t think that observation gives us license to avoid the problem.
Further, I’m not sure this elevates my understanding: I’ve read many posts on this space which could be viewed as analogous to this one (this one is more tempered, of course). Each one has this same flaw: someone is telling me I need to make a “organization” out of agents and positive things will follow.
Without a serious evaluation, how am I supposed to validate the author’s ontology?
Do you disagree with my assessment? Do you view the claims in this content as solid and reproducible?
My own view is that these are “soft ideas” (GasTown, Ralph fall into a similar category) without the rigorous justification.
What this amounts to is “synthetic biology” with billion dollar probability distributions — where the incentives are setup so that companies are incentivized to convey that they have the “secret sauce” … for massive amounts of money.
To that end, it’s difficult to trust a word out of anyone’s mouth — even if my empirical experiences match (along some projection).
> There are plenty of papers out there that look at LLM productivity and every one of them seems to have glaring methodology limitations and/or reports on models that are 12+ months out of date.
This is a general problem with papers measuring productivity in any sense. It's often a hard thing to define what "productivity" means and to figure out how to measure it. But also in that any study with worthwhile results will:
1. Probably take some time (perhaps months or longer) to design, get funded, and get through an IRB.
2. Take months to conduct. You generally need to get enough people to say anything, and you may want to survey them over a few weeks or months.
3. Take months to analyze, write up, and get through peer review. That's kind of a best case; peer review can take years.
So I would view the studies as necessarily time-boxed snapshots due to the practical constraints of doing the work. And if LLM tools change every year, like they have, good studies will always lag and may always feel out of date.
It's totally valid to not find a lot of value in them. On the other hand, people all-in on AI have been touting dramatic productivity gains since ChatGPT first arrived. So it's reasonable to have some historical measurements to go with the historical hype.
At the very least, it gives our future agentic overlords something to talk about on their future AI-only social media.
But the absence of papers is precisely the problem and why all this LLM stuff has become a new religion in the tech sphere.
Either you have faith and every post like this fills you with fervor and pious excitement for the latest miracles performed by machine gods.
Or you are a nonbeliever and each of these posts is yet another false miracle you can chalk up to baseless enthusiasm.
Without proper empirical method, we simply do not know.
What's even funnier about it is that large-scale empirical testing is actually necessary in the first place to verify that a stochastic processes is even doing what you want (at least on average). But the tech community has become such a brainless atmosphere totally absorbed by anecdata and marketing hype that no one simply seems to care anymore. It's quite literally devolved into the religious ceremony of performing the rain dance (use AI) because we said so.
One thing the papers help provide is basic understanding and consistent terminology, even when the models change. You may not find value in them but I assure you that the actual building of models and product improvements around them is highly dependent on the continual production of scientific research in machine learning, including experiments around applications of llms. The literature covers many prompting techniques well, and in a scientific fashion, and many of these have been adopted directly in products (chain of thought, to name one big example—part of the reason people integrate it is not because of some "fingers crossed guys, worked on my query" but because researchers have produced actual statistically significant results on benchmarks using the technique) To be a bit harsh, I find your very dismissal of the literature here in favor of hype-drenched blog posts soaked in ridiculous language and fantastical incantations to be precisely symptomatic of the brain rot the LLM craze has produced in the technical community.
> The Digital Twin Universe is our answer: behavioral clones of the third-party services our software depends on. We built twins of Okta, Jira, Slack, Google Docs, Google Drive, and Google Sheets, replicating their APIs, edge cases, and observable behaviors.
Came to the same conclusion. I have an integration heavy codebase and it could hardly test anything if tests weren't allowed to call external services. So there are fake implementations of every API it touches: Anthropic, Gemini, Sprites, Brave, Slack, AgentMail, Notion, on and on and on. 22 fakes and climbing. Why not? They're essentially free to generate, it's just tokens.
I didn't go as far as recreating the UI of these services, though, as the article seems to be implying based on those screenshots. Just the APIs.
In this hypothetical world where AI reliably generates software, large and small software providers alike are out of luck. Companies will go straight to LLMs or open-source models, fine-tune them for their needs, and run them on in-house hardware as costs fall, spreading expenses across departments. Even LLM providers won’t be safe. Brand, lock-in, and incumbent status won’t save anyone. The advantage goes to whoever can integrate, customize, and scale internally. Hypothetically is the keyword.
What are the other consequences of unlimited cheap reliable quality software? It's hard to think about but feels more important than just SaaS companies going bankrupt.
$100 says they're still doing leetcode interviews.
If everyone can do this, there won't be any advantage (or profit) to be had from it very soon. Why not buy your own hardware and run local models, I wonder.
I would spend those $100 on either API tokens or donate to a charity of your choice. My interview to join this team was whether I could build something of my choosing in under an hour with any coding agent of my choice.
No local model out there is as good as the SOTA right now.
> As I understood it the trick was effectively to dump the full public API documentation of one of those services into their agent harness and have it build an imitation of that API, as a self-contained Go binary. They could then have it build a simplified UI over the top to help complete the simulation.
This is still the same problem -- just pushed back a layer. Since the generated API is wrong, the QA outcomes will be wrong, too. Also, QAing things is an effective way to ensure that they work _after_ they've been reviewed by an engineer. A QA tester is not going to test for a vulnerability like a SQL injection unless they're guided by engineering judgement which comes from an understanding of the properties of the code under test.
The output is also essentially the definition of a derivative work, so it's probably not legally defensible (not that that's ever been a concern with LLMs).
Having submitted this I would also suggest the website admin revisit their testing; its very slow on my phone. Obviously fails on aesthetics and accessibility as well. Submitted for the essay.
On the cxdb “product” page one reason they give against rolling your own is that it would be “months of work”. Slipped into an archaic off-brand mindset there, no?
I like the idea but I'm not so sure this problem can be solved generally.
As an example: imagine someone writing a data pipeline for training a machine learning model. Anyone who's done this knows that such a task involves lots data wrangling work like cleaning data, changing columns and some ad hoc stuff.
The only way to verify that things work is if the eventual model that is trained performs well.
In this case, scenario testing doesn't scale up because the feedback loop is extremely large - you have to wait until the model is trained and tested on hold out data.
Scenario testing clearly can not work on the smaller parts of the work like data wrangling.
$1k per day, 50 work weeks, 5 day a week → $250k a year. That is, to be worth it, the AI should work as well as an engineer that costs a company $250k. Between taxes, social security, and cost of office space, that engineer would be paid, say, $170-180k a year, like an average-level senior software engineer in the US.
This is not an outrageous amount of money, if the productivity is there. More likely the AI would work like two $90k junior engineers, but without a need to pay for a vacation, office space, social security, etc. If the productivity ends up higher than this, it's pure profit; I suppose this is their bet.
The human engineer would be like a tech lead guiding a tea of juniors, only designing plans and checking results above the level of code proper, but for exceptional cases, like when a human engineer would look at the assembly code a compiler has produced.
This does sound exaggeratedly optimistic now, but does not sound crazy.
It’s a $90k engineer that sometimes acts like a vandal, who never has thoughts like “this seems to be a bad way to go. Let me ask the boss” or “you know, I was thinking. Shouldn’t we try to extract this code into a reusable component?” The worst developers I’ve worked with have better instincts for what’s valuable. I wish it would stop with “the simplest way to resolve this is X little shortcut” -> boom.
It basically stumbles around generating tokens within the bounds (usually) of your prompt, and rarely stops to think. Goal is token generation, baby. Not careful evaluation. I have to keep forcing it to stop creating magic inline strings and rather use constants or config, even though those instructions are all over my Claude.md and I’m using the top model. It loves to take shortcuts that save GPU but cost me time and money to wrestle back to rational. “These issues weren’t created by me in this chat right now so I’ll ignore them and ship it.” No, fix all the bugs. That’s the job.
Still, I love it. I can hand code the bits I want to, let it fly with the bits I don’t. I can try something new in a separate CLI tab while others are spinning. Cost to experiment drops massively.
>> $170-180k a year, like an average-level senior software engineer in the US.
I hear things like this all the time, but outside of a few major centers it's just not the norm. And no companies are spending anything like $1k / month on remote work environments.
Assuming current prices are heavily subsidised (VC money) and there is a supply shock (because we don't have enough GPUs/energy). If that leads to double the price that means 500k/year, and if we see a 4x price increase that's 1000k/year.
Suddenly, it starts to look precarious. That would be my concern anyway.
I think that is easy to understand for a lot of people but I will spell it out.
This looks like AI companies marketing that is something in line 1+1 or buy 3 for 2.
Money you don’t spend on tokens are the only saved money, period.
With employees you have to pay them anyway you can’t just say „these requirements make no sense, park for two days until I get them right”.
You would have to be damn sure of that you are doing the right thing to burn $1k a day on tokens.
With humans I can see many reasons why would you pay anyway and it is on you that you should provide sensible requirements to be built and make use of employees time.
That nobody wants to actually do it is already a problem, but some basically true thing is that somebody has to pay those $90k junior engineers for a couple years to turn them into senior engineers.
The seem to be plenty of people willing to pay the AI do that junior engineer level work, so wouldn’t it make sense to defect and just wait until it has gained enough experience to do the senior engineer work?
My gemini subscription is all I need. It's like an interactive stack overflow that doesn't yell at you and answers your questions.
I was working on a problem and having trouble understanding an old node splitting paper, and Gemini pointed me to a better paper with a more efficient algorithm, then explained how it worked, then generated test code. It's fantastic. I'm not saying it's better than the other LLMs, but having a little oracle available online is a great boost to learning and debugging.
Different beasts on the API, the extra context left makes a huge difference. Unless there's something else out there I've missed, which at the speed things move these days it's always a possibility.
I'm one of the StrongDM trio behind this tenet. The core claim is simple: it's easy to spend $1k/day on tokens, but hard (even with three people) to do it in a way that stays reliably productive.
The margins on software are incredibly high and perhaps this is just the cost of having maintainable output.
Also I think you have to consider development time.
If someone creates a SaaS product then it can be trivially cloned in a small timeframe. So the moat that normally exists becomes non existent. Therefore to stay ahead or to catch up it’s going to cost money.
In a way it’s similar to the way FAANG was buying up all the good engineers. It starves potential and lower capitalised but more nimble competitors of resources that it needs to compete with them.
I do think it's a crazy thing to say, but not because of the amount. I mean, if putting in more money produces more value, why not $10,000 a day or a million? Does adding more tokens after $1000 stop working for some reason?
Forget about agents or AI: the amount of money that it makes sense to spend on software engineering for a particular company is highly dependent on the specifics of that company.
Perhaps for them this number makes sense, but it's kind of crazy to extrapolate that to everyone as some kind of benchmark. It would be far more interesting to hear how they place a value on the code produced.
I have a harsher take down-thread, but the simulation testing (what they call DTU) is actually interesting and a useful insight into grounding agent behavior.
Each engineer is very valuable. LLM tokens are cheap. You scale up inference compute, and your engineers can focus on higher order stuff, not reviewing incorrect responses, validating bugs, and what not.
It’s shocking to me that there isn’t a $2,000 / $20,000 per month subscription tier for coding assistants. I’ve always in my mind called this ExecGPT since around 2021, but the notion was that executives have teams that support them to be high functioning and high leverage, responsible for quality of thinking and decision making, not quantity of work output.
And the value/prop existed and continues to exist even as the models get smarter, even Opus 4.6.
My favorite conspiracy theory is that these projects/blog posts are secretly backed by big-AI tech companies, to offset their staggering losses by convincing executives to shovel pools of money into AI tools.
They have to be. And the others writing this stuff likely do not deal with real systems with thousands of customers, a team who needs to get paid, and a reputation to uphold. Fatal errors that cause permanent damage to a business are unacceptable.
Designing reliable, stable, and correct systems is already a high level task. When you actually need to write the code for it, it's not a lot and you should write it with precision. When creating novel or differently complex systems, you should (or need to) be doing it yourself anyway.
Is it really a secret, when Anthropic posted a project of building a C compiler totally from scratch for $20k equivalent token spend, as an official article on their own blog? $20k is quite insane for such a self-contained project, if that's genuinely the amount that these tools require that's literally the best possible argument for running something open and leveraging competitive 3rd party inference.
The implication of "you have to have spent $1000 in tokens per engineer, or you have failed" is that you must fire any engineer who works fine by themselves or with other people and who doesn't require LLM crutch (at least if you don't want to be "failed" according to some random guy's opinion).
Getting rid of such naysayers is important for the industry.
Slop influencers like Peter Steinberger get paid to promote AI vibe coding startups and the agentic token burning hype. Ironically they're so deep into the impulsivity of it all that they can't even hide it. The latest frontier models all continue to suffer from hallucinations and slop at scale.
- Factory, unconvinced. Their marketing videos are just too cringe, and any company that tries to get my attentions with free tokens in my DMs reduce my respect for them. If you're that good, you don't need to convince me by giving me free stuff. Additionally, some posts on Twitter about it have this paid influencer smell. If you use claude code tho, you'll feel right at home with the [signature flicker](https://x.com/badlogicgames/status/1977103325192667323).
+ Factory, unconvinced. Their videos are a bit cringe, I do hear good things in my timeline about it tho, even if images aren't supported (yet) and they have the [signature flicker](https://x.com/badlogicgames/status/1977103325192667323).
I don't think that's really a conspiracy theory lol. As long as you're playing Money Chicken, why not toss some at some influencers to keep driving up the FOMO?
I am not sure why people are getting hung on the price, i.e. this: "They have the gaul to pitch/attention seek a 1$/day with possibly little/no product". The price can drop TBH and while there is some correlation on $/capita output.
The more nuanced "outrage" here, how taking humans out of the agent loop is, as I have commented elsewhere, quite flawed TBH and very bold to say the least. And while every VC is salivating, more attention should instead be given to all the AI Agent PMs, The Tech lead of AI, or whatever that title is on some of the following:
- What _workflow_ are you building?
- What is your success with your team/new hires in having them use this?
- What's your RoC for investment in the workflow?
- How varied is this workflow? Is every company just building their own workflows or are there patterns emerging on agent orchestration that are useful.
Yeah, it's hard to read the article without getting a cringy feeling of second hand embarrassment. The setup is weird too, in that it seems to imply that the little snippets of "wisdom" should be used as prompts to an LLM to come to their same conclusions, when of course this style of prompt will reliably produce congratulatory dreck.
Setting aside the absurdity of using dollars per day spent on tokens as the new lines of code per day, have they not heard of mocks or simulation testing? These are long proven techniques, but they appear bent on taking credit for some kind revolutionary discovery by recasting these standard techniques as a Digital Twin Universe.
One positive(?) thing I'll say is that this fits well with my experience of people who like to talk about software factories (or digital factories), but at least they're up front about the massive cost of this type of approach - whereas "digital factories" are typically cast as a miracle cure that will reduce costs dramatically somehow (once it's eventually done correctly, of course).
Yeah, getting strong Devin vibes here. In some ways they were ahead of their time in other ways agents have become commoditized and their platform is arguably obsolete. I have a strong feeling the same will happen with "software factories".
Growth will be proportional to spend. You can cut waste later and celebrate efficiency. So when growing there isn't much incentive to do it efficiently. You are just robbing yourself of a potential future victory. Also it's legitimately difficult to maximize growth while prioritizing efficiency. It's like how a body builder cycles between bulking and cutting. For mid to long term outlooks it's probably the best strategy.
It's not so much crazy as very lame and stupid and dumb. The moment has allowed people doing dumb things to somehow grab the attention of many in the industry for a few moments. There's nothing "there".
> Those of us building software factories must practice a deliberate naivete
This is a great way to put it, I've been saying "I wonder which sacred cows are going to need slaughtered" but for those that didn't grow up on a farm, maybe that metaphor isn't the best. I might steal yours.
This stuff is very interesting and I'm really interested to see how it goes for you, I'll eagerly read whatever you end up putting out about this. Good luck!
EDIT: oh also the re-implemented SaaS apps really recontextualizes some other stuff I’ve been doing too…
This was an experiment that Justin ran: one person fresh out of college, and another with a long, traditional career.
Even though all three of us have very different working styles, we all seem to be very happy with the arrangement.
You definitely need to keep an open mind, though, and be ready to unlearn some things. I guess I haven’t spent enough time in the industry yet to develop habits that might hinder adopting these tools.
Jay single-handedly developed the digital twin universe. Only one person commits to a codebase :-)
I’ve been building using a similar approach[1] and my intuition is that humans will be needed at some points in the factory line for specific tasks that require expertise/taste/quality. Have you found that the be the case? Where do you find that humans should be involved in the process of maximal leverage?
To name one probable area of involvement: how do you specify what needs to be built?
Your intuition/thinking definitely lines up with how we're thinking about this problem. If you have a good definition of done and a good validation harness, these agents can hill climb their way to a solution.
But you still need human taste/judgment to decide what you want to build (unless your solution is to just brute force the entire problem space).
For maximal leverage, you should follow the mantra "Why am I doing this?" If you use this enough times, you'll come across the bottleneck that can only be solved by you for now. As a human, your job is to set the higher-level requirements for what you're trying to build. Coming up with these requirements and then using agents to shape them up is acceptable, but human judgment is definitely where we have to answer what needs to be built. At the same time, I never want to be doing something the models are better at. Until we crack the proactiveness part, we'll be required to figure out what to do next.
Also, it looks like you and Danvers are working in the same space, and we love trading notes with other teams working in this area. We'd love to connect. You can either find my personal email or shoot me an email at my work email: navan.chauhan [at] strongdm.com
În real world, worst performers get thrown out of the loop if they are flagged and fail to improve. Assumption here seems to be agents have no such issues. Am I missing something?
In the real world, over engineering is the play to work-pretend sped money, with more theatre than results. Am I missing something here too? Deep agentic factories? Deepest agentic factories?
I have been working on my own "Digital Twins Universe" because 3rd-party SaaS tools often block the tight feedback loops required for long-horizon agentic coding. Unlike Stripe, which offers a full-featured environment usable in both development and staging, most B2B SaaS companies lack adequate fidelity (e.g., missing webhooks in local dev) or even a basic staging environment.
Taking the time to point a coding agent towards the public (or even private) API of a B2B SaaS app to generate a working (partial) clone is effectively "unblocking" the agent. I wouldn't be surprised if a "DTU-hub" eventually gains traction for publishing and sharing these digital twins.
I would love to hear more about your learnings from building these digital twins. How do you handle API drift? Also, how do you handle statefulness within the twins? Do you test for divergence? For example, do you compare responses from the live third-party service against the Digital Twin to check for parity?
What would happen if these agents are given a token lifespan, and are told to continually spend tokens to create more agentic children, and give their genetic and data makeup such as it is to children that it creates with other agents sexually potentially, but then tokens are limited and they can not get enough without certain traits.
Wouldn’t they start to evolve to be able to reproduce more and eat more tokens? And then they’d be mature agents to take further human prompts to gain more tokens?
Would you see certain evolutionary strategies reemerge like carnivores eating weaker agents for tokens, eating of detritus of old code, or would it be more like evolution of roles in a company?
I assume the hurdles would be agents reproducing? How is that implemented?
Huffing a lot of Gastown and having some hallucinations of my own. We have to show these machines we can out hallucinate them! Hi future overlords training on this data
This is part of a new trend towards “harness engineering”. You should automate away as much of the software construction and validation process as possible, but also the QA and integration (which includes debugging).. take yourself progressively out of those loops, that’s the new job.
For example you can iteratively automate code review. Every time you notice an issue during review, pop open your coding agent and ask it how it might be instructed to catch such a thing. There’s going to be an 80/20 rule here - you probably can’t eliminate every issue, but there’s bound to be low hanging fruit.
> The Digital Twin Universe is our answer: behavioral clones of the third-party services our software depends on. We built twins of Okta, Jira, Slack, Google Docs, Google Drive, and Google Sheets, replicating their APIs, edge cases, and observable behaviors.
And what they actually released is:
> strongdm/attractor
> spec of StrongDM's Attractor, a non-interactive Coding Agent sufficient for use in a Software Factory
And
> stromdm/cxdb
> CXDB is an AI Context Store for agents and LLMs, providing fast, branch-friendly storage for conversation histories and tool outputs with content-addressed deduplication.
Cringe. I hate this word but I can't come up with a better word to describe this. The only takeaway I got from this article is that I should improve my vocabulary so I can describe how stupid the whole thing is.
I explored the different mental frameworks for how we use LLMs here: https://yagmin.com/blog/llms-arent-tools/ I think the "software factory" is currently the end state of using LLMs in most people's minds, but I think there is (at least) one more level: LLMs as applications.
Which is more or less creating a customized harness. There is a lot more that is possiible once we move past the idea that harnesses are just for workflow variations for engineers.
That's a genuine problem now. If you launch a new feature and your competition can ship their own copy a few hours later the competitive dynamics get really challenging!
My hunch is that the thing that's going to matter is network effects and other forms of soft lockin. Features alone won't cut it - you need to build something where value accumulates to your user over time in a way that discourages them from leaving.
The interesting part about that is both of those things require some sort of time to start.
If I launch a new product, and 4 hours later competitors pop up, then there's not enough time for network effects or lockin.
I'm guessing what is really going to be needed is something that can't be just copied. Non-public data, business contracts, something outside of software.
If AI really makes software cheap and fast, the future isn’t generic SaaS clones competing in hours. Companies will just generate their own hyper-custom internal versions, Salesforce clones tailored to their exact workflows. Brand and lock-in won’t save vendors; internal control and cost savings will.
Some of this is people trying to predict the future.
And it’s not unreasonable to assume it’s going there.
That being said, the models are not there yet. If you care about quality, you still need humans in the loop.
Even when given high quality specs, and existing code to use as an example, and lots of parallelism and orchestration, the models still make a lot of mistakes.
There’s lots of room for Software Factories, and Orchestrators, and multi agent swarms.
But today you still need humans reviewing code before you merge to main.
Models are getting better, quickly, but I think it’s going to be a while before “don’t have humans look at the code” is true.
The solution to this problem is not throwing everything at AI. To get good results from any AI model, you need an architect (human) instructing it from the top. And the logic behind this is that AI has been trained on millions of opinions on getting a particular task done. If you ask a human, they almost always have one opinionated approach for a given task. The human's opinion is a derivative of their lived experience, sometimes foreseeing all the way to the end result an AI cannot foresee. Eg. I want a database column a certain type because I'm thinking about adding an E-Commerce feature to my CMS later. An AI might not have this insight.
Of course, you can't always tell the model what to do, especially if it is a repeated task. It turns out, we already solved this decades ago using algorithms. Repeatable, reproducible, reliable. The challenge (and the reward) lies in separating the problem statement into algorithmic and agentic. Once you achieve this, the $1000 token usage is not needed at all.
I have a working prototype of the above and I'm currently productizing it (shameless plug):
However - I need to emphasize, the language you use to apply the pattern above matters. I use Elixir specifically for this, and it works really, really well.
It works based off starting with the architect. You. It feeds off specs and uses algorithms as much as possible to automate code generation (eg. Scaffolding) and only uses AI sparsely when needed.
Of course, the downside of this approach is that you can't just simply say "build me a social network". You can however say something like "Build me a social network where users can share photos, repost, like and comment on them".
Once you nail the models used in the MVC pattern, their relationships, the software design is pretty much 50% battle won. This is really good for v1 prototypes where you really want best practices enforced, OSWAP compliant code, security-first software output which is where a pure agentic/AI approach would mess up.
I recently passed 40,000 but my Substack is free so it's not a revenue source for me. I haven't really looked at who they are - at some point it would be interesting to export the CSV of the subscribers and count by domains, I guess.
My content revenue comes from ads on my blog via https://www.ethicalads.io/ - rarely more than $1,000 in a given month - and sponsors on GitHub: https://github.com/sponsors/simonw - which is adding up to quite good money now. Those people get my sponsors-only monthly newsletter which looks like this: https://gist.github.com/simonw/13e595a236218afce002e9aeafd75... - it's effectively the edited highlights from my blog because a lot of people are too busy to read everything I put out there!
how about the elephant.. Apart of business-spec itself, Where-from all those (supply-chain) API specs/documentation are going to come? After, say, 3 iterations in this vein, of the API-makers themselves ??
… all this hype, and just … where are the macro level results? GitHub is seemingly having more outages than ever before, and MS is pretty directly involved in the AI hype; should they be a beacon of how great AI's output is? Yet obvious bugs that have persisted for years still languish. My day job is more and more feeling like I'm fighting just to get tooling or services to do the most basic things they were allegedly designed to do.
On the other hand, you have AI companies claiming "we built a browser from scratch" — and then having that claim utterly eviscerated. I cannot fathom going to my boss and requesting "$1,000/day per engineer" for AI — that's an absurd amount of money.
And yet, whenever I actually do try to get AI to meet the road … it is just query and query after clarifying query that humans would not and do not need. E.g., trying to ask clarifying questions about tsc, and TS, and … just wrong answer after wrong answer, or even just misunderstanding the question entirely. Trying to file a support ticket in big clouds now requires wading through AI slop that doesn't solve anything, just to get to a human whose writing feels suspiciously like you've been shoveled a second dose of slop. Like I just finished a quarter long ticket with GCP on "IAM is not functioning to spec, and we can clearly and concisely prove it" to get back a very long form "we don't care".
In this model the spec/scenarios are the code. These are curated and managed by humans just like code.
They say "non interactive". But of course their work is interactive. AI agents take a few minutes-hours whereas you can see code change result in seconds. That doesn't mean AI agents aren't interactive.
I'm very AI-positive, and what they're doing is different, but they are basically just lying. It's a new word for a new instance of the same old type of thing. It's not a new type of thing.
The common anti-AI trope is "AI just looked at <human output> to do this." The common AI trope from the StrongDM is "look, the agent is working without human input." Both of these takes are fundamentally flawed.
AI will always depend on humans to produce relevant results for humans. It's not a flaw of AI, it's more of a flaw of humans. Consequently, "AI needs human input to produce results we want to see" should not detract from the intelligence of AI.
Why is this true? At a certain point you just have Kolmogorov complexity, AI having fixed memory and fixed prompt size, pigeonhole principle, not every output is possible to be produced even with any input given specific model weights.
Recursive self-improvement doesn't get around this problem. Where does it get the data for next iteration? From interactions with humans.
With the infinite complexity of mathematics, for instance solving Busy Beaver numbers, this is a proof that AI can in fact not solve every problem. Humans seem to be limited in this regard as well, but there is no proof that humans are fundamentally limited this way like AI. This lack of proof of the limitations of humans is the precise advantage in intelligence that humans will always have over AI.
This might be okay short term, e.g. when you just want to get something done.
Maybe not on the scale of decades, otherwise we will end up completely unable to code.
> Code must not be reviewed by humans
This is where it all goes to crap. Until the day when the AI agents can look at some output and be like "no, this is overengineering, this can be done way more simply, let's stick to the established patterns within the codebase" and do so consistently, not having oversight will compound failure. I don't mean between every single small change, but rather at least to catch failures before merging anything.
This could only be avoided if you could define a harness with thousands or tens of thousands of tests per codebase, encapsulating EVERYTHING that must and must not be done within the codebase, down to the way how gaps and colors and utility classes are to be used, which I don't see most people doing.
This will take both better models and 3-10 years of work to actually make using them more foolproof and consistent. Even so, context sizes might need to be in the hundreds of thousands of tokens for the average task, all of those human "hunches" and styles and approaches to a given codebase spelled out.
> evaluating success often required LLM-as-judge
This is just advocating for doing what's easy, not proper - this will lead to slop long term. Though maybe they say that's okay, as long as it works.
Most of the time you would want those checks to be more dependable than that, the same way how you wouldn't want your linter to have randomness.
> Tests can be reward hacked - we needed validation that was less vulnerable to the model cheating
Just have adversarial agents, the one that writes the code doesn't touch the tests and vice versa, even though each has all of the context, each is told to care about different things.
Seems like these people are trying to push an envelope, and it might look like it's going to work in some respects, but they're very much taking big risks on what's currently feasible vs not.
> If you haven't spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement
I imagine it's even easier to remove the CEO/Executive staff. Actually, why have anyone there at all? Surely this company can LLM their way to having no staff whatsoever!
Yeah, extraordinary claims need some internal consistency before external evangelism. I’d expect the same from other companies whose CEOs make these kinds of claims, like Nvidia and Anthropic.
It's bad enough that a new programming fad washes over the industry every five years or so; some progress still manages to squeak through. It's going to absolutely grind to a halt if we're just getting a new black box oracle with different cargo cult rituals that have to be heuristically discovered every six months
So much of this resonated with me, and I realize I’ve arrived at a few of the techniques myself (and with my team) over the last several months.
THIS FRIGHTENS ME. Many of us sweng are either going be FIRE millionaires, or living under a bridge, in two years.
I’ve spent this week performing SemPort; found a ts app that does a needed thing, and was able to use a long chain of prompts to get it completely reimplemented in our stack, using Gene Transfer to ensure it uses some existing libraries and concrete techniques present in our existing apps.
Now not only do I have an idiomatic Python port, which I can drop right into our stack, but I have an extremely detailed features/requirements statement for the origin typescript app along with the prompts for generating it. I can use this to continuously track this other product as it improves. I also have the “instructions infrastructure” to direct an agent to align new code to our stack. Two reusable skills, a new product, and it took a week.
My post? Shiiiii if that’s how it comes across I may delete it. I haven’t logged into LI since our last corp reorg, it was a cesspool even then. Self promotion just ain’t my bag
I was just trying to share the same patterns from OPs documentation that I found valuable within the context of agentic development; seeing them take this so far is was scares me, because they are right that I could wire an agent to do this autonomously and probably get the same outcomes, scaled.
> If you haven't spent at least $1,000 on tokens today per human engineer
So a four person team should be spending close to $1M/year, double each engineer’s salary, on AI alone? To get the output of one junior engineer who smokes crack and has his memory wiped every twenty minutes?
There is no mention of a productivity increase anywhere in this piece. 2x? Maybe, but at this price you can hire another very senior engineer, and have both be 50% more productive with a $200/month AI spend.
There are many ways "producing" can be quantified (LOC, PRs, features) such that 4x production does not correlate to a 4x in value of the product (quality, revenue).
Doubling? Try quadrupling outside of silicon valley. He is saying hire 4x as many engineers and make 3/4 of them AI. So much for the 10x productivity increase — that's 0.25x!
Do you have any successful examples of software build by this factory in a main that isn't AI-adjacent? I.e. not an SaaS clone, not another code linter/database/whatever? Can the factory build, e.g. a crypto trading algorithm, an e-commerce site, etc.?
To come up with a software spec for the agents to follow, first you need to understand what your business is and where the money is.
The Digital Twin Universe is the most interesting thing in this article and the part most people are glossing over. The real question Simon nails is: how do you prove software works when both the implementation and the tests are written by agents? Because agents will absolutely game your test suite - return true, rewrite assertions to match broken output, whatever gets them to green.
Their answer of keeping scenarios external to the codebase like a holdout set is smart. And building full behavioral clones of services like Okta, Jira, Slack so you can run thousands of end to end scenarios without hitting rate limits or production - that's where the actual hard engineering work is. Not the code generation, the validation infrastructure.
Most teams trying this will skip that part because it's expensive and unglamorous. They'll let agents write code and tests together and wonder why things break in production. The "factory" part isn't the agents writing code. It's having robust enough external proof that the code does what it's supposed to.
I did have an initial key insight which led to a repeatable strategy to ensure a high level of fidelity between DTU vs. the official canonical SaaS services:
Use the top popular publicly available reference SDK client libraries as compatibility targets, with the goal always being 100% compatibility.
You've also zeroed in on how challenging this was: I started this back in August 2025 (as one of many projects, at any time we're each juggling 3-8 projects) with only Sonnet 3.5. Much of the work was still very unglamorous, but feasible. Especially Slack, in some ways Slack was more challenging to get right than all of G-Suite (!).
Now I'm part way through reimplementing the entire DTU in Rust (v1 was in Go) and with gpt-5.2 for planning and gpt-5.3-codex for execution it's significantly less human effort.
IMO the most novel part to this story is Navan's Attractor and corresponding NLSpec. Feed in a good Definition-of-Done and it'll bounce around between nodes until it gets it right. There are already several working implementations in less than 24 hours since it was released, one of which is even open source [0].
Strongly inclined to agree here: Having recently joined a small applied AI startup and we were discussing the need for E2E tests. My initial gut reaction (which I kept quiet) was that such things turn into unmaintainable messes which delay releases and increasingly reduce in value.
I recognised this was grounded in an entirely different world of software engineering and organisation size though. I followed a path of thinking about what went wrong historically and how might they be solved: Better structure, discipline, resources - all of the things which agentic AI facilitates.
You are right about most skipping this part: But I view it as being like a sewerage and sanitation system - largely invisible and not thought about but critical for long-term health.
Also this ties in very nicely with Netflix's approach to Chaos Engineering and enabling it at broader scale.
> You are right about most skipping this part: But I view it as being like a sewerage and sanitation system - largely invisible and not thought about but critical for long-term health.
And like sewage and sanitation the infrastructure is a lot more complicated than people think.
I’m curious what happens when they need to make a DRU of Stripe or another payment processor.
You have a different agent write the tests and another run the tests. You tell them each that they aren’t checking their own work, they’re checking someone else’s. You can tell them to be skeptical. Then you can also tell them that don’t fail the code for no reason, because a third agent will be checking your tests and you will be penalized for inaccurate testing.
This approach balances out and maximizes accuracy.
At first I was partially impressed by the Digital Twin Universe thing they describe. Having worked with 3rd party APIs in a previous life, having something like that would've been so much help.
But after thinking about it more, I think it must be the lowest of low hanging fruits for LLMs. You're building something with well defined specs, most of which is readily available by the original creators, with a UI that only does the bare minimum, and it doesn't need any long-term features like reliability since it's all for internal short-lived use. On top of that, it looks super impressive when used in a demo, because all those applications being mocked are very complicated pieces of software. So to recreate a thin facade of them can look very impressive. And calling it a "Digital Twin Universe" is just icing on the cake.
I don't think this is meaningfully different than the human case for the past 20 years. Every large project I've worked on had people writing tests that didn't test anything and people who argued strongly when I pointed out glaring missing test functionality coverage. And their managers did not like to spend money on having better tests written.
I think the "software factory" terminology is very interesting, and I would imagine quite intentional.
It calls to mind the early days of the industrial revolution, when I believe the idea was that mass produced items were not better quality, just dramatically cheaper. So you still had the artisans that the rich people paid for but now poorer people had access to something they couldn't before.
Then, as technology progressed, factories started producing things that humans are incapable of. And part of this is because those factories were built on output of earlier factories.
It makes me wonder if this is where we're headed. Right now the code quality of agents isn't better than hand written code, and so arguably the products aren't either. But will there come a time when it surpasses what we can do? You can't handcraft a microchip, for example. But I guess the takeaway is maybe there's a time for both agentic lower quality but cheaper output and human software engineer higher quality output, at least for a time.
Which is exactly why agentic coding tools are effectively running at a loss right now. We are training that next generation of factories. We're paying in human cognition.
"Software factory" is a pretty conventional term used for a CI/CD pipeline with automation including DevSecOps, containerization, and agile management.
I was looking for some code, or a product they made, or anything really on their site.
The only github I could find is: https://github.com/strongdm/attractor
Canadian girlfriend coding is now a business model.
Edit:
I did find some code. Commit history has been squashed unfortunately: https://github.com/strongdm/cxdb
There's a bunch more under the same org but it's years old.
There's actual code in this repo: https://github.com/strongdm/cxdb
Amusingly, it appears the README (that would be code, right?) has hallucinated the existence of a docker image - someone filed an issue at https://github.com/strongdm/cxdb/issues/1
In-house employees don't read code or do code reviews, so presumably they don't raise issues either. I guess the issue was picked up by an astute HN reader.
I've looked at their code for a few minutes in a few files, and while I don't know what they're trying to do well enough to say for sure anything is definitely a bug, I've already spotted several things that seem likely to be, and several others that I'd class as anti-patterns in rust. Don't get me wrong, as an experiment this is really cool, but I do not think they've succeeded in getting the "dark factory" concept to work where every other prominent attempt has fallen short.
6 replies →
They have a Products page where they list a database and an identity system in addition to attractors: https://factory.strongdm.ai/products
For those of us working on building factories, this is pretty obvious because once you immediately need shared context across agents / sessions and an improved ID + permissions system to keep track of who is doing what.
I don't know if that is crazy or a glimpse of the future (could be both).
PS: TIL about "Canadian girlfriend", thanks!
That's hilarious
I was about to say the same thing! Yet another blog post with heaps of navel gazing and zero to actually show for it.
The worst part is they got simonw to (perhaps unwittingly or social engineering) vouch and stealth market for them.
And $1000/day/engineer in token costs at current market rates? It's a bold strategy, Cotton.
But we all know what they're going for here. They want to make themselves look amazing to convince the boards of the Great Houses to acquire them. Because why else would investors invest in them and not in the Great Houses directly.
The "social engineering" is that I was invited to a demo back in October and thought it was really interesting.
(Two people who's opinions I respect said "yeah you really should accept that invitation" otherwise I probably wouldn't have gone.)
I've been looking forward to being able to write more details about what they're doing ever since.
7 replies →
I think this comment is slightly unfair :(
We’ve been working on this since July, and we shared the techniques and principles that have been working for us because we thought others might find them useful. We’ve also open-sourced the nlspec so people can build their own versions of the software factory.
We’re not selling a product or service here. This also isn’t about positioning for an acquisition: we’ve already been in a definitive agreement to be acquired since last month.
It’s completely fair to have opinions and to not like what we’re putting out, but your comment reads as snarky without adding anything to the conversation.
7 replies →
> The worst part is they got simonw to (perhaps unwittingly or social engineering) vouch and stealth market for them.
Lol. Any time I see something ai related endorsed by simonw, I tend to view it as mostly hype, and I have been right so far.
2 replies →
So I am on a web cast where people working about this. They are from https://docs.boundaryml.com/guide/introduction/what-is-baml and humanlayer.dev Mostly are talking about spec driven development. Smart people. Here is what I understood from them about spec driven development, which is not far from this AFAIU.
Lets start with the `/research -> /plan -> /implement(RPI)`. When you are building a complex system for teams you _need_ humans in the loop and you want to focus on design decisions. And having structured workflows around agents provides a better UX to those humans make those design decisions. This is necessary for controlling drift, pollution of context and general mayhem in the code base. _This_ is the starting thesis around spec drive development.
How many times have you working as a newbie copied a slash command pressed /research then /plan then /implement only to find it after several iterations is inconsistent and go back and fix it? Many people still go back and forth with chatgpt copying back and forth copying their jira docs and answering people's question on PRD documents. This is _not_ a defence it is the user experience when working with AI for many.
One very understandable path to solve this is to _surface_ to humans structured information extracted from your plan docs for example:
https://gist.github.com/itissid/cb0a68b3df72f2d46746f3ba2ee7...
In this very toy spec driven development the idea is that each step in the RPI loop is broken down and made very deterministic with humans in the loop. This is a system designed by humans(Chief AI Officer, no kidding) for teams that follow a fairly _customized_ processes on how to work fast with AI, without it turning into a giant pile of slop. And the whole point of reading code or QA is this: You stop the clock on development and take a beat to see the high signal information: Testers want to read tests and QAers want to test behavior, because well written they can tell a lot about weather a software works. If you have ever written an integration test on a brownfield code with poor test coverage, and made it dependable after several days in the dark, you know what it feels like... Taking that step out is what all VCs say is the last game in town.. the final game in town.
This StrongDM stuff is a step beyond what I can understand: "no humans should write code", "no humans should read code", really..? But here is the thing that puzzles me even more is that spec driven development as I understand it, to use borrowed words, is like parents raising a kid — once you are a parent you want to raise your own kid not someone else's. Because it's just such a human in the loop process. Every company, tech or not, wants to make their own process that their engineers like to work with. So I am not sure they even have a product here...
> If you haven’t spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement
At that point, outside of FAANG and their salaries, you are spending more on AI than you are on your humans. And they consider that level of spend to be a metric in and of itself. I'm kinda shocked the rest of the article just glossed over that one. It seems to be a breakdown of the entire vision of AI-driven coding. I mean, sure, the vendors would love it if everyone's salary budget just got shifted over to their revenue, but such a world is absolutely not my goal.
Yeah I'm going to update my piece to talk more about that.
Edit: here's that section: https://simonwillison.net/2026/Feb/7/software-factory/#wait-...
I don't understand. Are you suggesting that the proper volume of agent work is achieved with the $200/month subscription, or that $30,000/month ($1,000/day) of tokens is necessary? I think there is a big difference between $200 and $30,000.
2 replies →
This is an interesting point but if I may offer a different perspective:
Assuming 20 working days a month: that's 20k x 12 == 240k a year. So about a fresh grad's TC at FANG.
Now I've worked with many junior to mid-junior level SDEs and sadly 80% does not do a better job than Claude. (I've also worked with staff level SDEs who writes worse code than AI, but they offset that usually with domain knowledge and TL responsibilities)
I do see AI transform software engineering into even more of a pyramid with very few human on top.
Original claim was:
> At that point, outside of FAANG and their salaries, you are spending more on AI than you are on your humans
You say
> Assuming 20 working days a month: that's 20k x 12 == 240k a year. So about a fresh grad's TC at FANG.
So you both are in agreement on that part at least.
Important too, a fully loaded salary costs the company far more than the actual salary that the employee receives. That would tip this balancing point towards 120k salaries, which is well into the realm of non-FAANG
It would depend on the speed of execution, if you can do the same amount of work in 5 days with spending 5k, vs spending a month and 5k on a human the math makes more sense.
You won't know which path has larger long term costs, for a example, what if the AI version costs 10x to run?
If the output is (dis)proportionally larger, the cost trade off might be the right thing to do.
And it might be the tokens will become cheaper.
Tokens will become significantly more expensive in the short term actually. This is not stemming from some sort of anti-AI sentiment. You have two ramps that are going to drive this. 1. Increase demand, linear growth at least but likely this is already exponential. 2. Scaling laws demand, well, more scale.
Future better models will both demand higher compute use AND higher energy. We cannot underestimate the slowness of energy production growth and also the supplies required for simply hooking things up. Some labs are commissioning their own power plants on site, but this is not a true accelerator for power grid growth limits. You're using the same supply chain to build your own power plant.
If inference cost is not dramatically reduced and models don't start meaningfully helping with innovations that make energy production faster and inference/training demand less power, the only way to control demand is to raise prices. Current inference costs, do not pay for training costs. They can probably continue to do that on funding alone, but once the demand curve hits the power production limits, only one thing can slow demand and that's raising the cost of use.
$1,000 is maybe 5$ per workday. I measure my own usage and am on the way to $6,000 for a full year. I'm still at the stage where I like to look at the code I produce, but I do believe we'll head to a state of software development where one day we won't need to.
Maybe read that quote again. The figure is 1000 per day
3 replies →
Until we solve the validation problem, none of this stuff is going to be more than flexes. We can automate code review, set up analytic guardrails, etc, so that looking at the code isn't important, and people have been doing that for >6 months now. You still have to have a human who knows the system to validate that the thing that was built matches the intent of the spec.
There are higher and lower leverage ways to do that, for instance reviewing tests and QA'ing software via use vs reading original code, but you can't get away from doing it entirely.
I agree with this almost completely. The hard part isn’t generation anymore, it’s validation of intent vs outcome. Especially once decisions are high-stakes or irreversible, think pkg updates or large scale tx
What I’m working on (open source) is less about replacing human validation and more about scaling it: using multiple independent agents with explicit incentives and disagreement surfaced, instead of trusting a single model or a single reviewer.
Humans are still the final authority, but consensus, adversarial review, and traceable decision paths let you reserve human attention for the edge cases that actually matter, rather than reading code or outputs linearly.
Until we treat validation as a first-class system problem (not a vibe check on one model’s answer), most of this will stay in “cool demo” territory.
“Anymore?” After 40 years in software I’ll say that validation of intent vs. outcome has always been a hard problem. There are and have been no shortcuts other than determined human effort.
2 replies →
But, is that different from how we already work with humans? Typically we don't let people commit whatever code they want just because they're human. It's more than just code reviews. We have design reviews, sometimes people pair program, there are unit tests and end-to-end tests and all kinds of tests, then code review, continuous integration, Q&A. We have systems to watch prod for errors or user complaints or cost/performance problems. We have this whole toolkit of process and techniques to try to get reliable programs out of what you must admit are unreliable programmers.
The question isn't whether agentic coders are perfect. Actually it isn't even whether they're better than humans. It's whether they're a net positive contribution. If you turn them loose in that kind of system, surrounded by checks and balances, does the system tend to accumulate bugs or remove them? Does it converge on high or low quality?
I think the answer as of Opus 4.5 or so is that they're a slight net positive and it converges on quality. You can set up the system and kind of supervise from a distance and they keep things under control. They tend to do the right thing. I think that's what they're saying in this article.
This obviously depends on what you are trying to achieve but it’s worth mentioning that there are languages designed for formal proofs and static analysis against a spec, and I have suspicions we are currently underutilizing them (because historically they weren’t very fun to write, but if everything is just tokens then who cares).
And “define the spec concretely“ (and how to exploit emerging behaviors) becomes the new definition of what programming is.
> “define the spec concretely“
(and unambiguously. and completely. For various depths of those)
This always has been the crux of programming. Just has been drowned in closer-to-the-machine more-deterministic verbosities, be it assembly, C, prolog, js, python, html, what-have-you
There have been a never ending attempts to reduce that to more away-from-machine representation. Low-code/no-code (anyone remember Last-one for Apple ][ ?), interpreting-and/or-generating-off DSLs of various level of abstraction, further to esperanto-like artificial reduced-ambiguity languages... some even english-like..
For some domains, above worked/works - and the (business)-analysts became new programmers. Some companies have such internal languages. For most others, not really. And not that long ago, the SW-Engineer job was called Analyst-programmer.
But still, the frontier is there to cross..
2 replies →
AI also quickly goes off the rails, even the Opus 2.6 I am testing today. The proposed code is very much rubbish, but it passes the tests. It wouldn't pass skilled human review. Worst thing is that if you let it, it will just grow tech debt on top of tech debt.
The code itself does not matter. If the tests pass, and the tests are good, then who cares? AI will be maintaining the code.
17 replies →
It's simple: you just offload the validation and security testing to the end user.
This is what we're working on at Speedscale. Our methods use traffic capture and replay to validate what worked before still works today.
did you read the article?
>StrongDM’s answer was inspired by Scenario testing (Cem Kaner, 2003).
Tests are only rigorous if the correct intent is encoded in them. Perfectly working software can be wrong if the intent was inferred incorrectly. I leverage BDD heavily, and there a lot of little details it's possible to misinterpret going from spec -> code. If the spec was sufficient to fully specify the program, it would be the program, so there's lots of room for error in the transformation.
16 replies →
> That idea of treating scenarios as holdout sets—used to evaluate the software but not stored where the coding agents can see them—is fascinating. It imitates aggressive testing by an external QA team—an expensive but highly effective way of ensuring quality in traditional software.
This is one of the clearest takes I've seen that starts to get me to the point of possibly being able to trust code that I haven't reviewed.
The whole idea of letting an AI write tests was problematic because they're so focused on "success" that `assert True` becomes appealing. But orchestrating teams of agents that are incentivized to build, and teams of agents that are incentivized to find bugs and problematic tests, is fascinating.
I'm quite curious to see where this goes, and more motivated (and curious) than ever to start setting up my own agents.
Question for people who are already doing this: How much are you spending on tokens?
That line about spending $1,000 on tokens is pretty off-putting. For commercial teams it's an easy calculation. It's also depressing to think about what this means for open source. I sure can't afford to spend $1,000 supporting teams of agents to continue my open source work.
Re: $1k/day on tokens - you can also build a local rig, nothing "fancy". There was a recent thread here re: the utility of local models, even on not-so-fancy hardware. Agents were a big part of it - you just set a task and it's done at some point, while you sleep or you're off to somewhere or working on something else entirely or reading a book or whatever. Turn off notifications to avoid context switches.
Check it: https://news.ycombinator.com/item?id=46838946
I wouldn't be surprised if agents start "bribing" each other.
If they're able to communicate with each other. But I'm pretty sure we could keep that from happening.
I don't take your comment as dismissive, but I think a lot of people are dismissing interesting and possibly effective approaches with short reactions like this.
I'm interested in the approach described in this article because it's specifying where the humans are in all this, it's not about removing humans entirely. I can see a class of problems where any non-determinism is completely unacceptable. But I can also see a large number of problems where a small amount of non-determinism is quite acceptable.
1 reply →
Do you know what those hold out twats should look like before thoroughly iterating on the problem?
I think people are burning money on tokens letting these things fumble about until they arrive at some working set of files.
I'm staying in the loop more than this, building up rather than tuning out
This is the stealth team I hinted at in a comment on here last week about the "Dark Factory" pattern of AI-assisted software engineering: https://simonwillison.net/2026/Feb/7/software-factory/
This one is worth paying attention to to. They're the most ambitious team I've see exploring the limits of what you can do with this stuff. It's eye-opening.
This right here is where I feel most concerned
> If you haven’t spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement
Seems to me like if this is true I'm screwed no matter if I want to "embrace" the "AI revolution" or not. No way my manager's going to approve me to blow $1000 a day on tokens, they budgeted $40,000 for our team to explore AI for the entire year.
Let alone from a personal perspective I'm screwed because I don't have $1000 a month in the budget to blow on tokens because of pesky things that also demand financial resources like a mortgage and food.
At this point it seems like damned if I do, damned if I don't. Feels bad man.
My friend works at Shopify and they are 100% all in on AI coding. They let devs spend as much as they want on whatever tool they want. If someone ends up spending a lot of money, they ask them what is going well and please share with others. If you’re not spending they have a different talk with you.
As for me, we get Cursor seats at work, and at home I have a GPU, a cheap Chinese coding plan, and a dream.
4 replies →
Yeah, that's one part of this that didn't sit right with me.
I don't think you need to spend anything like that amount of money to get the majority of the value they're describing here.
Edit: added a new section to my blog post about this: https://simonwillison.net/2026/Feb/7/software-factory/#wait-...
7 replies →
> No way my manager's going to approve me to blow $1000 a day on tokens, they budgeted $40,000 for our team to explore AI for the entire year.
To be fair, I’ll bet many embracing concerning advice like that have never worked for the same company for a full year.
Same. Feels like it goes against the entire “hacker” ethos that brought me here in the first place. That sentence made me actually feel physically sick on initial read as well. Everyday now feels like a day where I have exponentially less & less interest in tech. If all of this AI that’s burning the planet is so incredible, where are the real world tangible improvements? I look around right now and everything in tech, software, internet, etc. has never looked so similar to a dumpster fire of trash.
4 replies →
I read that as combined, up to this point in time. You have 20 engineers? If you haven't spent at least $20k up to this point, you've not explored or experienced enough of the ins and outs to know how best to optimize the use of these tools.
I didn't read that as you need to be spending $1k/day per engineer. That is an insane number.
EDIT: re-reading... it's ambiguous to me. But perhaps they mean per day, every day. This will only hasten the elimination of human developers, which I presume is the point.
May be the point is, that the one engineer replaces 10 engineers by using the dark factory which by definition doesn't need humans.
2 replies →
I think corporate incentives vs personal incentives are slightly different here. As a company trying to experiment in this moment, you should be betting on token cost not being the bottleneck. If the tooling proves valuable, $1k/day per engineer is actually pretty cheap.
At home on my personal setup, I haven't even had to move past the cheapest codex/claude code subscription because it fulfills my needs ¯\_(ツ)_/¯. You can also get a lot of mileage out of the higher tiers of these subscriptions before you need to start paying the APIs directly.
8 replies →
Until there's something verifiable it's just talk. Talk was cheap. Now talk has become an order of magnitude cheaper since ChatGPT.
Can you make an ethical declaration here, stating whether or not you are being compensated by them?
Their page looks to me like a lot of invented jargon and pure narrative. Every technique is just a renamed existing concept. Digital Twin Universe is mocks, Gene Transfusion is reading reference code, Semport is transpilation. The site has zero benchmarks, zero defect rates, zero cost comparisons, zero production outcomes. The only metric offered is "spend more money".
Anyone working honestly in this space knows 90% of agent projects are failing.
The main page of HN now has three to four posts daily with no substance, just Agentic AI marketing dressed as engineering insight.
With Google, Microsoft, and others spending $600 billion over the next year on AI, and panicking to get a return on that Capex....and with them now paying influencers over $600K [1] to manufacture AI enthusiasm to justify this infrastructure spend, I won't engage with any AI thought leadership that lacks a clear disclosure of financial interests and reproducible claims backed by actual data.
Show me a real production feature built entirely by agents with full traces, defect rates, and honest failure accounting. Or stop inventing vocabulary and posting vibes charts.
[1] - https://news.ycombinator.com/item?id=46925821
> Every technique is just a renamed existing concept. Digital Twin Universe is mocks, Gene Transfusion is reading reference code, Semport is transpilation. The site has zero benchmarks, zero defect rates, zero cost comparisons, zero production outcomes. The only metric offered is "spend more money".
Repeating for emphasis, because this is the VERY obvious question anyone with a shred of curiosity would be asking not just about this submission but about what is CONSTANTLY on the frontpage these days.
There could be a very simple 5 question questionnaire that could eliminate 90+% of AI coding requests before they start:
- Is this a small wrapper around just querying an existing LLM
- Does a brief summary of this searched with "site:github" already return dozens or hundreds of results?
- Is this a classic scam (pump&dump, etc) redone using "AI"
- Is this needless churn between already high level abstractions of technology (dashboard of dashboards, yaml to json, python to java script, automation of automation framework)
Simon does have a disclosure on his site about not being compensated for anything: https://simonwillison.net/about/#disclosures
11 replies →
Yet they have produced almost nothing. You can give $10k to couple of college grads and get a better product.
It is tempting to be stealthy when you start seeing discontinuous capabilities go from totally random to somewhat predictable. But most of the key stuff is on GitHub.
The moats here are around mechanism design and values (to the extent they differ): the frontier labs are doomed in this world, the commons locked up behind paywalls gets hyper mirrored, value accrues in very different places, and it's not a nice orderly exponent from a sci-fi novel. It's nothing like what the talking heads at Davos say, Anthropic aren't in the top five groups I know in terms of being good at it, it'll get written off as fringe until one day it happens in like a day. So why be secretive?
You get on the ladder by throwing out Python and JSON and learning lean4, you tie property tests to lean theorems via FFI when you have to, you start building out rfl to pretty printers of proven AST properties.
And yeah, the droids run out ahead in little firecracker VMs reading from an effect/coeffect attestation graph and writing back to it. The result is saved, useful results are indexed. Human review is about big picture stuff, human coding is about airtight correctness (and fixing it when it breaks despite your "proof" that had a bug in the axioms).
Programming jobs are impacted but not as much as people think: droids do what David Graeber called bullshit jobs for the most part and then they're savants (not polymath geniuses) at a few things: reverse engineering and infosec they'll just run you over, they're fucking going in CIC.
This is about formal methods just as much as AI.
"If you haven't spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement"
Apart from being a absolutely ridiculous metric, this is a bad approach, at least with current generation models. In my experience, the less you inspect what the model does, the more spaghetti-like the code will be. And the flying spaghetti monster eats tokens faster than you can blink! Or put more clearly: implementing a feature will cost you a lot more tokens in a messy code base than it does in a clean one. It's not (yet) enough to just tell the agent to refactor and make it clean, you have to give it hints on how to organise the code.
I'd go do far as to say that if you're burning a thousand dollars a day per engineer, you're getting very little bang for your tokens.
And your engineers probably look like this: https://share.google/H5BFJ6guF4UhvXMQ7
Maybe Management will finally get behind refactoring
Damn, I was already on board with using coding agents. Consider me welded to the deck at this point!
It's short-term vs long-term optimization. Short-term optimization is making the system effective right now. Long-term optimization is exploring ways to improve the system as a whole.
> In rule form: - Code must not be written by humans - Code must not be reviewed by humans
as a previous strongDM customer, i will never recommend their offering again. for a core security product, this is not the flex they think it is
also mimicking other products behavior and staying in sync is a fools task. you certainly won't be able to do it just off the API documentation. you may get close, but never perfect and you're going to experience constant breakage
Important to note that this is the approach taken by their AI research lab over the past six months, it's not (yet) reflective of how they build the core product.
What have they actually built?
Right but how many unsuspecting customers like you do they need to have before they can exit?
They actually "exited" a few weeks ago - acquired by Delinea: https://delinea.com/news/delinea-strongdm-to-unite-redefine-...
From what I've heard the acquisition was unrelated to their AI lab work, it was about the core business.
1 reply →
> with the second revision of Claude 3.5 (October 2024), long-horizon agentic coding workflows began to compound correctness rather than error.
What does it mean to compound correctness? Like negative acceleration in rate of errors? How does that compound? Unseriously!
The model could start building on top of things it had successfully built before instead of just straight up exponential error propagation
If you'd like to try this yourself, you can build an "attractor" by just pointing claude code at their llms.txt. Or if you'd like to save some tokens, you can clone my go version. https://github.com/danshapiro/kilroy This version has a Claude Code skill to help. Tell it to use it's skill to create a dotfile from your requirements. Then tell it to run that dotfile with kilroy.
What has strongdm actually built? Are their users finding value from their supposed productivity gains?
If their focus is to only show their productivity/ai system but not having built anything meaningful with it, it feels like one of those scammy life coaches/productivity gurus that talk about how they got rich by selling their courses.
> we transitioned from boolean definitions of success ("the test suite is green") to a probabilistic and empirical one. We use the term satisfaction to quantify this validation: of all the observed trajectories through all the scenarios, what fraction of them likely satisfy the user?
Oh, to have the luxury of redefining success and handwaving away hard learned lessons in the software industry.
Not sure “Digital Twin Universe” is required here. They seem rather to have rediscovered Simulators in Integration Tests from first principles? The DTU comes off as XML Databases or Information Superhighway.
Still… a really good application of agent hands-off replication.
Seems like creating a quality negative mould and then that single negative mould makes multiple positive objects en-masse.
the agentic shift is where the legal and insurance worlds are really going to struggle. we know how to model human error, but modeling an autonomous loop that makes a chain of small decisions leading to a systemic failure is a whole different beast. the audit trail requirements for these factories are going to be a regulatory nightmare.
I think the insurance industry is will take a simpler route: humans will be held 100% responsible. Any decisions made by the ai will be the responsibility of the human instructing that ai. Always.
I think this will act as a brake on the agentic shift as a whole.
that's the current legal default, but it starts breaking down when you look at product liability vs professional liability.
if a company sells an autonomous agent that is marketed as doing a task without human oversight, the courts will eventually move that burden back to the manufacturer. we saw the same dance with autonomous driving disclaimers the "human must stay in control" line works as a legal shield for a while, but eventually the market demands a shift in who holds the risk.
if we stick to 100% human responsibility for black-box errors that a human couldn't have even predicted, that "brake" won't just slow down the agentic shift, it'll effectively kill the enterprise market for it. no C-suite is going to authorize a fleet of agents if they're holding 100% of the bag for emergent failures they can't audit.
1 reply →
They just are not going to provide insurance to companies who use AI because the liability costs are not worth it to them since they cannot actual calculate risks, it is already happening [0]. Its the one thing that a lot of the evangelists of using AI for entire products have come to realize or they aren't actually dealing with B2B scenarios where indemnity comes into play. That or they are lying to insurance companies and their customers, which is a... choice.
[0] https://futurism.com/future-society/insurance-cyber-risk-ai
IT perspective here. Simon hits the nail on the head as to what I'm genuinely looking forward to:
> How do you clone the important parts of Okta, Jira, Slack and more? With coding agents!
This is what's going to gut-punch most SaaS companies repeatedly over the next decade, even if this whole build-out ultimately collapses in on itself (which I expect it to). The era of bespoke consultants for SaaS product suites to handle configuration and integrations, while not gone, are certainly under threat by LLMs that can ingest user requirements and produce functional code to do a similar thing at a fraction of the price.
What a lot of folks miss is that in enterprise-land, we only need the integration once. Once we have an integration, it basically exists with minimal if any changes until one side of the integration dies. Code fails a security audit? We can either spool up the agents again briefly to fix it, or just isolate it in a security domain like the glut of WinXP and Win7 boxen rotting out there on assembly lines and factory floors.
This is why SaaS stocks have been hammered this week. It's not that investors genuinely expect huge players to go bankrupt due to AI so much as they know the era of infinite growth is over. It's also why big AI companies are rushing IPOs even as data center builds stall: we're officially in a world where a locally-run model - not even an Agent, just a model in LM Studio on the Corporate Laptop - can produce sufficient code for a growing number of product integrations without any engineer having to look through yet another set of API documentation. As agentic orchestration trickles down to homelabs and private servers on smaller, leaner, and more efficient hardware, that capability is only going to increase, threatening profits of subscription models and large AI companies. Again, why bother ponying up for a recurring subscription after the work is completed?
For full-fledged software, there's genuine benefit to be had with human intervention and creativity; for the multitude of integrations and pipelines that were previously farmed out to pricey consultants, LLMs will more than suffice for all but the biggest or most complex situations.
“API Glue” is what I’ve called it since forever
Stuff comes in from an API goes out to a different API.
With a semi-decent agent I can build what took me a week or two in hours just because it can iterate the solution faster than any human can type.
A new field in the API could’ve been a two day ordeal of patching it through umpteen layers of enterprise frameworks. Now I can just tell Claude to add it, it’ll do it up to the database in minutes - and update the tests at the same time.
And because these are all APIs, we can brute-force it with read-only operations with minimal review times. If the read works, the write almost always will, and then it's just a matter of reading and documenting the integration before testing it in dev or staging.
So much of enterprise IT nowadays is spent hammering or needling vendors for basic API documentation so we can write a one-off that hooks DB1 into ServiceNow that's also pulling from NewRelic just to do ITAM. Consultants would salivate over such a basic integration because it'd be their yearly salary over a three month project.
Now we can do this ourselves with an LLM in a single sprint.
That's a Pandora's Box moment right there.
[dead]
>> How do you clone the important parts of Okta, Jira, Slack and more? With coding agents!
> This is what's going to gut-punch most SaaS companies repeatedly over the next decade
but there's already clones of the important parts of those systems and yet the SaaS world survives. The code used isn't the secret sauce and people in SaaS know writing the code is 10% of the effort in keeping those businesses on their feet.
I don't think the SaaS industry is on the ropes until coding agents can do things like create a recommendation algorithm better than Spotify and YouTube. In those cases the code/algorithm is indeed the secret sauce and if a coding agent can do better than those companies will be left behind.
A lot of examples of creating clones of existing products don't resonate with new products we build
For example, most development work involves discovering correctness, not writing to a fullproof spec (like cloning slack)
Usually work goes like:
* Team decides some vague requirement
* Developer must implement requirement into executable decisions
Now I use Claude Code to do step 2 now, and its great. But I'm looking over whether the implementation's little decisions actually do what the business would want. Or more accurately, I'm making decisions to the level of specificity that matters to the problem at hand.
I have to try, backtrack, and rebuild all the time when my assumptions get broken.
In some cases decisions have low specificity: I could one-shot a complex feature (or entire app if trying to test PMF or something). In other cases, the tradeoffs in 10 lines of code become crucially important.
I'm just going to say: When opening the "twins" (bad clones) screenshots, I pressed the right key to view the next image, and surprise, the next "article" of the top navigation bar was loaded, instead of showing the next image.
Is this the quality we should expect from agentic? From my experiments with claude code, yes, the UX details are never there. Especially for bigger features. It can work reasonably well independently up to a "module" level (with clear interfaces). But for full app design, while technically possible, the UX and visual design is just not there.
And I am very not attracted to the idea of polishing such an agentic apps. A solution could be: 1. The boss prompts the system with what he wants. 2. The boss outsources to india the task of polishing the rough edges.
===
More on the arrow keys navigation: Pressing right on the last "Products" page loops to the first "Story" page, yet pressing left on the first page does nothing. Typical UX inconsistency of vibe coded software.
Effectively everyone is building the same tools with zero quantitative benchmarks or evidence behind the why / ideas … this entire space is a nightmare to navigate because of this. Who cares without proper science, seriously? I look through this website and it looks like a preview for a course I’m supposed to buy … when someone builds something with these sorts of claims attached, I assume that there is going to be some “real graphs” (“these are the number of times this model deviated from the spec before we added error correction …”)
What we have instead are many people creating hierarchies of concepts, a vast “naming” of their own experiences, without rigorous quantitative evaluation.
I may be alone in this, but it drives me nuts.
Okay, so with that in mind, it amounts to heresay “these guys are doing something cool” — why not shut up or put up with either (a) an evaluation of the ideas in a rigorous, quantitative way or (b) apply the ideas to produce an “hard” artifact (analogous, e.g., to the Anthropic C compiler, the Cursor browser) with a reproducible pathway to generation.
The answer seems to be that (b) is impossible (as long as we’re on the teet of the frontier labs, which disallow the kind of access that would make (b) possible) and the answer for (a) is “we can’t wait we have to get our names out there first”
I’m disappointed to see these types of posts on HN. Where is the science?
Honestly I've not found a huge amount of value from the "science".
There are plenty of papers out there that look at LLM productivity and every one of them seems to have glaring methodology limitations and/or reports on models that are 12+ months out of date.
Have you seen any papers that really elevated your understanding of LLM productivity with real-world engineering teams?
The writing on this website is giving strong web3 vibes to me / doesn't smell right.
The only reason I'm not dismissing it out of hand is basically because you said this team was worth taking a look at.
I'm not looking for a huge amount of statistical ceremony, but some detail would go a long way here.
What exactly was achieved for what effort and how?
3 replies →
No, I agree! But I don’t think that observation gives us license to avoid the problem.
Further, I’m not sure this elevates my understanding: I’ve read many posts on this space which could be viewed as analogous to this one (this one is more tempered, of course). Each one has this same flaw: someone is telling me I need to make a “organization” out of agents and positive things will follow.
Without a serious evaluation, how am I supposed to validate the author’s ontology?
Do you disagree with my assessment? Do you view the claims in this content as solid and reproducible?
My own view is that these are “soft ideas” (GasTown, Ralph fall into a similar category) without the rigorous justification.
What this amounts to is “synthetic biology” with billion dollar probability distributions — where the incentives are setup so that companies are incentivized to convey that they have the “secret sauce” … for massive amounts of money.
To that end, it’s difficult to trust a word out of anyone’s mouth — even if my empirical experiences match (along some projection).
1 reply →
> There are plenty of papers out there that look at LLM productivity and every one of them seems to have glaring methodology limitations and/or reports on models that are 12+ months out of date.
This is a general problem with papers measuring productivity in any sense. It's often a hard thing to define what "productivity" means and to figure out how to measure it. But also in that any study with worthwhile results will:
1. Probably take some time (perhaps months or longer) to design, get funded, and get through an IRB.
2. Take months to conduct. You generally need to get enough people to say anything, and you may want to survey them over a few weeks or months.
3. Take months to analyze, write up, and get through peer review. That's kind of a best case; peer review can take years.
So I would view the studies as necessarily time-boxed snapshots due to the practical constraints of doing the work. And if LLM tools change every year, like they have, good studies will always lag and may always feel out of date.
It's totally valid to not find a lot of value in them. On the other hand, people all-in on AI have been touting dramatic productivity gains since ChatGPT first arrived. So it's reasonable to have some historical measurements to go with the historical hype.
At the very least, it gives our future agentic overlords something to talk about on their future AI-only social media.
But the absence of papers is precisely the problem and why all this LLM stuff has become a new religion in the tech sphere.
Either you have faith and every post like this fills you with fervor and pious excitement for the latest miracles performed by machine gods.
Or you are a nonbeliever and each of these posts is yet another false miracle you can chalk up to baseless enthusiasm.
Without proper empirical method, we simply do not know.
What's even funnier about it is that large-scale empirical testing is actually necessary in the first place to verify that a stochastic processes is even doing what you want (at least on average). But the tech community has become such a brainless atmosphere totally absorbed by anecdata and marketing hype that no one simply seems to care anymore. It's quite literally devolved into the religious ceremony of performing the rain dance (use AI) because we said so.
One thing the papers help provide is basic understanding and consistent terminology, even when the models change. You may not find value in them but I assure you that the actual building of models and product improvements around them is highly dependent on the continual production of scientific research in machine learning, including experiments around applications of llms. The literature covers many prompting techniques well, and in a scientific fashion, and many of these have been adopted directly in products (chain of thought, to name one big example—part of the reason people integrate it is not because of some "fingers crossed guys, worked on my query" but because researchers have produced actual statistically significant results on benchmarks using the technique) To be a bit harsh, I find your very dismissal of the literature here in favor of hype-drenched blog posts soaked in ridiculous language and fantastical incantations to be precisely symptomatic of the brain rot the LLM craze has produced in the technical community.
1 reply →
> The Digital Twin Universe is our answer: behavioral clones of the third-party services our software depends on. We built twins of Okta, Jira, Slack, Google Docs, Google Drive, and Google Sheets, replicating their APIs, edge cases, and observable behaviors.
Came to the same conclusion. I have an integration heavy codebase and it could hardly test anything if tests weren't allowed to call external services. So there are fake implementations of every API it touches: Anthropic, Gemini, Sprites, Brave, Slack, AgentMail, Notion, on and on and on. 22 fakes and climbing. Why not? They're essentially free to generate, it's just tokens.
I didn't go as far as recreating the UI of these services, though, as the article seems to be implying based on those screenshots. Just the APIs.
how are you implementing agentmail? would love to know more
Yep, you definitely want to be in the business of selling shovels for the gold rush.
In this hypothetical world where AI reliably generates software, large and small software providers alike are out of luck. Companies will go straight to LLMs or open-source models, fine-tune them for their needs, and run them on in-house hardware as costs fall, spreading expenses across departments. Even LLM providers won’t be safe. Brand, lock-in, and incumbent status won’t save anyone. The advantage goes to whoever can integrate, customize, and scale internally. Hypothetically is the keyword.
What are the other consequences of unlimited cheap reliable quality software? It's hard to think about but feels more important than just SaaS companies going bankrupt.
Sounds like a great opportunity for my company! Who can I hire to help me figure out how to do this stuff?
$100 says they're still doing leetcode interviews.
If everyone can do this, there won't be any advantage (or profit) to be had from it very soon. Why not buy your own hardware and run local models, I wonder.
I would spend those $100 on either API tokens or donate to a charity of your choice. My interview to join this team was whether I could build something of my choosing in under an hour with any coding agent of my choice.
No local model out there is as good as the SOTA right now.
> My interview to join this team was whether I could build something of my choosing in under an hour with any coding agent of my choice.
You should have led with that. I think that's actually more impressive; anyone can spend tokens.
> As I understood it the trick was effectively to dump the full public API documentation of one of those services into their agent harness and have it build an imitation of that API, as a self-contained Go binary. They could then have it build a simplified UI over the top to help complete the simulation.
This is still the same problem -- just pushed back a layer. Since the generated API is wrong, the QA outcomes will be wrong, too. Also, QAing things is an effective way to ensure that they work _after_ they've been reviewed by an engineer. A QA tester is not going to test for a vulnerability like a SQL injection unless they're guided by engineering judgement which comes from an understanding of the properties of the code under test.
The output is also essentially the definition of a derivative work, so it's probably not legally defensible (not that that's ever been a concern with LLMs).
Having submitted this I would also suggest the website admin revisit their testing; its very slow on my phone. Obviously fails on aesthetics and accessibility as well. Submitted for the essay.
Sounds like you're experiencing an "agentic moment".
Haha yeah if I scroll on my iPhone 15 Pro it literally doesn’t load until I stop.
I get the following on safari on iOs: A problem repeatedly occurred on (url)
On iOS Safari it loads and works decent for me, but w/ iOS Firefox and Firefox Focus doesn't even load.
Lets hope the agents in their factory can fix it asap...
On the cxdb “product” page one reason they give against rolling your own is that it would be “months of work”. Slipped into an archaic off-brand mindset there, no?
We make this great, just don't use it to build the same thing we offer
Heat death of the SaaSiverse
I like the idea but I'm not so sure this problem can be solved generally.
As an example: imagine someone writing a data pipeline for training a machine learning model. Anyone who's done this knows that such a task involves lots data wrangling work like cleaning data, changing columns and some ad hoc stuff.
The only way to verify that things work is if the eventual model that is trained performs well.
In this case, scenario testing doesn't scale up because the feedback loop is extremely large - you have to wait until the model is trained and tested on hold out data.
Scenario testing clearly can not work on the smaller parts of the work like data wrangling.
>If you haven't spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement
…What am I even reading? Am I crazy to think this is a crazy thing to say, or it’s actually crazy?
$1k per day, 50 work weeks, 5 day a week → $250k a year. That is, to be worth it, the AI should work as well as an engineer that costs a company $250k. Between taxes, social security, and cost of office space, that engineer would be paid, say, $170-180k a year, like an average-level senior software engineer in the US.
This is not an outrageous amount of money, if the productivity is there. More likely the AI would work like two $90k junior engineers, but without a need to pay for a vacation, office space, social security, etc. If the productivity ends up higher than this, it's pure profit; I suppose this is their bet.
The human engineer would be like a tech lead guiding a tea of juniors, only designing plans and checking results above the level of code proper, but for exceptional cases, like when a human engineer would look at the assembly code a compiler has produced.
This does sound exaggeratedly optimistic now, but does not sound crazy.
It’s a $90k engineer that sometimes acts like a vandal, who never has thoughts like “this seems to be a bad way to go. Let me ask the boss” or “you know, I was thinking. Shouldn’t we try to extract this code into a reusable component?” The worst developers I’ve worked with have better instincts for what’s valuable. I wish it would stop with “the simplest way to resolve this is X little shortcut” -> boom.
It basically stumbles around generating tokens within the bounds (usually) of your prompt, and rarely stops to think. Goal is token generation, baby. Not careful evaluation. I have to keep forcing it to stop creating magic inline strings and rather use constants or config, even though those instructions are all over my Claude.md and I’m using the top model. It loves to take shortcuts that save GPU but cost me time and money to wrestle back to rational. “These issues weren’t created by me in this chat right now so I’ll ignore them and ship it.” No, fix all the bugs. That’s the job.
Still, I love it. I can hand code the bits I want to, let it fly with the bits I don’t. I can try something new in a separate CLI tab while others are spinning. Cost to experiment drops massively.
6 replies →
$250k a year, for now. What's to stop anthropic for doubling the price if your entire business depends on it? What are you gonna do, close shops?
34 replies →
>> $170-180k a year, like an average-level senior software engineer in the US.
I hear things like this all the time, but outside of a few major centers it's just not the norm. And no companies are spending anything like $1k / month on remote work environments.
10 replies →
Assuming current prices are heavily subsidised (VC money) and there is a supply shock (because we don't have enough GPUs/energy). If that leads to double the price that means 500k/year, and if we see a 4x price increase that's 1000k/year.
Suddenly, it starts to look precarious. That would be my concern anyway.
I think that is easy to understand for a lot of people but I will spell it out.
This looks like AI companies marketing that is something in line 1+1 or buy 3 for 2.
Money you don’t spend on tokens are the only saved money, period.
With employees you have to pay them anyway you can’t just say „these requirements make no sense, park for two days until I get them right”.
You would have to be damn sure of that you are doing the right thing to burn $1k a day on tokens.
With humans I can see many reasons why would you pay anyway and it is on you that you should provide sensible requirements to be built and make use of employees time.
1 reply →
That nobody wants to actually do it is already a problem, but some basically true thing is that somebody has to pay those $90k junior engineers for a couple years to turn them into senior engineers.
The seem to be plenty of people willing to pay the AI do that junior engineer level work, so wouldn’t it make sense to defect and just wait until it has gained enough experience to do the senior engineer work?
1 reply →
> 50 work weeks
What dystopia is this?
5 replies →
It doesn't say 1k per day. Not saying I agree with the statement per se, but it's a much weaker statement than that.
5 replies →
why stop at 5 days a week?
It sounds exaggeratedly crazy.
Meanwhile, me
> $20/month Claude sub
> $20/month OpenAI sub
> When Claude Code runs out, switch to Codex
> When Codex runs out, go for a walk with the dogs or read a book
I'm not an accelerationist singularity neohuman. Oh well, I still get plenty done
My gemini subscription is all I need. It's like an interactive stack overflow that doesn't yell at you and answers your questions.
I was working on a problem and having trouble understanding an old node splitting paper, and Gemini pointed me to a better paper with a more efficient algorithm, then explained how it worked, then generated test code. It's fantastic. I'm not saying it's better than the other LLMs, but having a little oracle available online is a great boost to learning and debugging.
same (at least for now, Codex seems to be much more token efficient)
The openrouter/free endpoint may make your dog unfit. You're welcome. Sorry doggo.
Different beasts on the API, the extra context left makes a huge difference. Unless there's something else out there I've missed, which at the speed things move these days it's always a possibility.
Presumably, organizations selling agentic coding APIs pay marketing people whose job is to boost this kind of article.
https://paulgraham.com/submarine.html
I'm one of the StrongDM trio behind this tenet. The core claim is simple: it's easy to spend $1k/day on tokens, but hard (even with three people) to do it in a way that stays reliably productive.
The margins on software are incredibly high and perhaps this is just the cost of having maintainable output.
Also I think you have to consider development time.
If someone creates a SaaS product then it can be trivially cloned in a small timeframe. So the moat that normally exists becomes non existent. Therefore to stay ahead or to catch up it’s going to cost money.
In a way it’s similar to the way FAANG was buying up all the good engineers. It starves potential and lower capitalised but more nimble competitors of resources that it needs to compete with them.
I do think it's a crazy thing to say, but not because of the amount. I mean, if putting in more money produces more value, why not $10,000 a day or a million? Does adding more tokens after $1000 stop working for some reason?
Forget about agents or AI: the amount of money that it makes sense to spend on software engineering for a particular company is highly dependent on the specifics of that company.
Perhaps for them this number makes sense, but it's kind of crazy to extrapolate that to everyone as some kind of benchmark. It would be far more interesting to hear how they place a value on the code produced.
I have a harsher take down-thread, but the simulation testing (what they call DTU) is actually interesting and a useful insight into grounding agent behavior.
It is not crazy.
Each engineer is very valuable. LLM tokens are cheap. You scale up inference compute, and your engineers can focus on higher order stuff, not reviewing incorrect responses, validating bugs, and what not.
It’s shocking to me that there isn’t a $2,000 / $20,000 per month subscription tier for coding assistants. I’ve always in my mind called this ExecGPT since around 2021, but the notion was that executives have teams that support them to be high functioning and high leverage, responsible for quality of thinking and decision making, not quantity of work output.
And the value/prop existed and continues to exist even as the models get smarter, even Opus 4.6.
> It’s shocking to me that there isn’t a $2,000 / $20,000 per month subscription tier for coding assistants.
What would be the benefit for the providers in offering this over just having those people use the API? I don't think it makes any sense for them.
My favorite conspiracy theory is that these projects/blog posts are secretly backed by big-AI tech companies, to offset their staggering losses by convincing executives to shovel pools of money into AI tools.
They have to be. And the others writing this stuff likely do not deal with real systems with thousands of customers, a team who needs to get paid, and a reputation to uphold. Fatal errors that cause permanent damage to a business are unacceptable.
Designing reliable, stable, and correct systems is already a high level task. When you actually need to write the code for it, it's not a lot and you should write it with precision. When creating novel or differently complex systems, you should (or need to) be doing it yourself anyway.
2 replies →
Is it really a secret, when Anthropic posted a project of building a C compiler totally from scratch for $20k equivalent token spend, as an official article on their own blog? $20k is quite insane for such a self-contained project, if that's genuinely the amount that these tools require that's literally the best possible argument for running something open and leveraging competitive 3rd party inference.
5 replies →
Like this?
https://www.cnbc.com/2026/02/06/google-microsoft-pay-creator...
3 replies →
The implication of "you have to have spent $1000 in tokens per engineer, or you have failed" is that you must fire any engineer who works fine by themselves or with other people and who doesn't require LLM crutch (at least if you don't want to be "failed" according to some random guy's opinion).
Getting rid of such naysayers is important for the industry.
I'm also convinced that any post in an AI thread that ends with "What a time to be alive!" is a bot. Seriously, look in any thread and you'll see it.
Slop influencers like Peter Steinberger get paid to promote AI vibe coding startups and the agentic token burning hype. Ironically they're so deep into the impulsivity of it all that they can't even hide it. The latest frontier models all continue to suffer from hallucinations and slop at scale.
https://github.com/steipete/steipete.me/commit/725a3cb372bc2...
Secretly? Most blog posts praising coding agents put something like 'I use $200 Claude subscription' in bold in 2nd-3rd paragraph.
I don't think that's really a conspiracy theory lol. As long as you're playing Money Chicken, why not toss some at some influencers to keep driving up the FOMO?
I am not sure why people are getting hung on the price, i.e. this: "They have the gaul to pitch/attention seek a 1$/day with possibly little/no product". The price can drop TBH and while there is some correlation on $/capita output.
The more nuanced "outrage" here, how taking humans out of the agent loop is, as I have commented elsewhere, quite flawed TBH and very bold to say the least. And while every VC is salivating, more attention should instead be given to all the AI Agent PMs, The Tech lead of AI, or whatever that title is on some of the following:
- What _workflow_ are you building? - What is your success with your team/new hires in having them use this? - What's your RoC for investment in the workflow? - How varied is this workflow? Is every company just building their own workflows or are there patterns emerging on agent orchestration that are useful.
[dead]
Yeah, it's hard to read the article without getting a cringy feeling of second hand embarrassment. The setup is weird too, in that it seems to imply that the little snippets of "wisdom" should be used as prompts to an LLM to come to their same conclusions, when of course this style of prompt will reliably produce congratulatory dreck.
Setting aside the absurdity of using dollars per day spent on tokens as the new lines of code per day, have they not heard of mocks or simulation testing? These are long proven techniques, but they appear bent on taking credit for some kind revolutionary discovery by recasting these standard techniques as a Digital Twin Universe.
One positive(?) thing I'll say is that this fits well with my experience of people who like to talk about software factories (or digital factories), but at least they're up front about the massive cost of this type of approach - whereas "digital factories" are typically cast as a miracle cure that will reduce costs dramatically somehow (once it's eventually done correctly, of course).
Hard pass.
Yeah, getting strong Devin vibes here. In some ways they were ahead of their time in other ways agents have become commoditized and their platform is arguably obsolete. I have a strong feeling the same will happen with "software factories".
This is some dumb boast/signaling that they're more AI-advanced than you are.
The desperation to be an AI thought leader is reaching Instagram influencer levels of deranged attention seeking.
It's crazy if you're an engineer. It's pretty common for middle managers to quantify "progress" in terms of "spend".
My bosses bosses boss like to claim that we're successfully moving to the cloud because the cost is increasing year over year.
Growth will be proportional to spend. You can cut waste later and celebrate efficiency. So when growing there isn't much incentive to do it efficiently. You are just robbing yourself of a potential future victory. Also it's legitimately difficult to maximize growth while prioritizing efficiency. It's like how a body builder cycles between bulking and cutting. For mid to long term outlooks it's probably the best strategy.
1 reply →
Appropriate username.
Complete absurd, unless their goal is to clone every SaaS ever built.
It's not so much crazy as very lame and stupid and dumb. The moment has allowed people doing dumb things to somehow grab the attention of many in the industry for a few moments. There's nothing "there".
When I see comments like this I wonder if the commenter has used LLMs for software development recently. (Genuine question).
1 reply →
(I’m one of the people on this team). I joined fresh out of college, and it’s been a wild ride.
I’m happy to answer any questions!
More of a comment than a question:
> Those of us building software factories must practice a deliberate naivete
This is a great way to put it, I've been saying "I wonder which sacred cows are going to need slaughtered" but for those that didn't grow up on a farm, maybe that metaphor isn't the best. I might steal yours.
This stuff is very interesting and I'm really interested to see how it goes for you, I'll eagerly read whatever you end up putting out about this. Good luck!
EDIT: oh also the re-implemented SaaS apps really recontextualizes some other stuff I’ve been doing too…
This was an experiment that Justin ran: one person fresh out of college, and another with a long, traditional career.
Even though all three of us have very different working styles, we all seem to be very happy with the arrangement.
You definitely need to keep an open mind, though, and be ready to unlearn some things. I guess I haven’t spent enough time in the industry yet to develop habits that might hinder adopting these tools.
Jay single-handedly developed the digital twin universe. Only one person commits to a codebase :-)
> "I wonder which sacred cows are going to need slaughtered"
Or a vegan or Hindu. Which ethics are you willing to throw away to run the software factory?
I eat hamburgers while aware of the moral issues.
I’ve been building using a similar approach[1] and my intuition is that humans will be needed at some points in the factory line for specific tasks that require expertise/taste/quality. Have you found that the be the case? Where do you find that humans should be involved in the process of maximal leverage?
To name one probable area of involvement: how do you specify what needs to be built?
[1] https://sociotechnica.org/notebook/software-factory/
You're absolutely right ;)
Your intuition/thinking definitely lines up with how we're thinking about this problem. If you have a good definition of done and a good validation harness, these agents can hill climb their way to a solution.
But you still need human taste/judgment to decide what you want to build (unless your solution is to just brute force the entire problem space).
For maximal leverage, you should follow the mantra "Why am I doing this?" If you use this enough times, you'll come across the bottleneck that can only be solved by you for now. As a human, your job is to set the higher-level requirements for what you're trying to build. Coming up with these requirements and then using agents to shape them up is acceptable, but human judgment is definitely where we have to answer what needs to be built. At the same time, I never want to be doing something the models are better at. Until we crack the proactiveness part, we'll be required to figure out what to do next.
Also, it looks like you and Danvers are working in the same space, and we love trading notes with other teams working in this area. We'd love to connect. You can either find my personal email or shoot me an email at my work email: navan.chauhan [at] strongdm.com
What exactly do you do, and what's your role in an agentic software factory?
I know you're not supposed to look at the code, but do you have things in place to measure and improve code quality anyway?
Not just code review agents, but things like "find duplicated code and refactor it"?
A few overnight “attractor” workflows serve distinct purposes:
* DRYing/Refactoring if needed
* Documentation compaction
* Security reviews
You aren't supposed to read code, but do you from time to time, just to evaluate what is going on?
No. But, I do ask questions (in $CODING_AGENT to always have a good mental model of everything that I’m working on though.
3 replies →
În real world, worst performers get thrown out of the loop if they are flagged and fail to improve. Assumption here seems to be agents have no such issues. Am I missing something?
In the real world, over engineering is the play to work-pretend sped money, with more theatre than results. Am I missing something here too? Deep agentic factories? Deepest agentic factories?
> A problem repeatedly occurred on "https://factory.strongdm.ai/".
I have been working on my own "Digital Twins Universe" because 3rd-party SaaS tools often block the tight feedback loops required for long-horizon agentic coding. Unlike Stripe, which offers a full-featured environment usable in both development and staging, most B2B SaaS companies lack adequate fidelity (e.g., missing webhooks in local dev) or even a basic staging environment.
Taking the time to point a coding agent towards the public (or even private) API of a B2B SaaS app to generate a working (partial) clone is effectively "unblocking" the agent. I wouldn't be surprised if a "DTU-hub" eventually gains traction for publishing and sharing these digital twins.
I would love to hear more about your learnings from building these digital twins. How do you handle API drift? Also, how do you handle statefulness within the twins? Do you test for divergence? For example, do you compare responses from the live third-party service against the Digital Twin to check for parity?
What would happen if these agents are given a token lifespan, and are told to continually spend tokens to create more agentic children, and give their genetic and data makeup such as it is to children that it creates with other agents sexually potentially, but then tokens are limited and they can not get enough without certain traits.
Wouldn’t they start to evolve to be able to reproduce more and eat more tokens? And then they’d be mature agents to take further human prompts to gain more tokens?
Would you see certain evolutionary strategies reemerge like carnivores eating weaker agents for tokens, eating of detritus of old code, or would it be more like evolution of roles in a company?
I assume the hurdles would be agents reproducing? How is that implemented?
I'll have 1 of what ever this guy's got please.
Evolution is a great meta heuristic optimization technique for bumpy functions, it seems natural to propose using it to tune agent performance.
Huffing a lot of Gastown and having some hallucinations of my own. We have to show these machines we can out hallucinate them! Hi future overlords training on this data
Do we give them 24 or 48 hours before somebody hacks their service?
This is part of a new trend towards “harness engineering”. You should automate away as much of the software construction and validation process as possible, but also the QA and integration (which includes debugging).. take yourself progressively out of those loops, that’s the new job.
For example you can iteratively automate code review. Every time you notice an issue during review, pop open your coding agent and ask it how it might be instructed to catch such a thing. There’s going to be an 80/20 rule here - you probably can’t eliminate every issue, but there’s bound to be low hanging fruit.
We will see where this goes!
> The Digital Twin Universe is our answer: behavioral clones of the third-party services our software depends on. We built twins of Okta, Jira, Slack, Google Docs, Google Drive, and Google Sheets, replicating their APIs, edge cases, and observable behaviors.
And what they actually released is:
> strongdm/attractor
> spec of StrongDM's Attractor, a non-interactive Coding Agent sufficient for use in a Software Factory
And
> stromdm/cxdb
> CXDB is an AI Context Store for agents and LLMs, providing fast, branch-friendly storage for conversation histories and tool outputs with content-addressed deduplication.
Cringe. I hate this word but I can't come up with a better word to describe this. The only takeaway I got from this article is that I should improve my vocabulary so I can describe how stupid the whole thing is.
I explored the different mental frameworks for how we use LLMs here: https://yagmin.com/blog/llms-arent-tools/ I think the "software factory" is currently the end state of using LLMs in most people's minds, but I think there is (at least) one more level: LLMs as applications.
Which is more or less creating a customized harness. There is a lot more that is possiible once we move past the idea that harnesses are just for workflow variations for engineers.
Serious question: what's keeping a competitor from doing the same thing and doing it better than you?
That's a genuine problem now. If you launch a new feature and your competition can ship their own copy a few hours later the competitive dynamics get really challenging!
My hunch is that the thing that's going to matter is network effects and other forms of soft lockin. Features alone won't cut it - you need to build something where value accumulates to your user over time in a way that discourages them from leaving.
The interesting part about that is both of those things require some sort of time to start.
If I launch a new product, and 4 hours later competitors pop up, then there's not enough time for network effects or lockin.
I'm guessing what is really going to be needed is something that can't be just copied. Non-public data, business contracts, something outside of software.
If AI really makes software cheap and fast, the future isn’t generic SaaS clones competing in hours. Companies will just generate their own hyper-custom internal versions, Salesforce clones tailored to their exact workflows. Brand and lock-in won’t save vendors; internal control and cost savings will.
Marketing and brand are still the most important, though I personally hope for a world where business is more indie and less winner take all
You can see the first waves of this trend in HN new.
4 replies →
Some of this is people trying to predict the future.
And it’s not unreasonable to assume it’s going there.
That being said, the models are not there yet. If you care about quality, you still need humans in the loop.
Even when given high quality specs, and existing code to use as an example, and lots of parallelism and orchestration, the models still make a lot of mistakes.
There’s lots of room for Software Factories, and Orchestrators, and multi agent swarms.
But today you still need humans reviewing code before you merge to main.
Models are getting better, quickly, but I think it’s going to be a while before “don’t have humans look at the code” is true.
The solution to this problem is not throwing everything at AI. To get good results from any AI model, you need an architect (human) instructing it from the top. And the logic behind this is that AI has been trained on millions of opinions on getting a particular task done. If you ask a human, they almost always have one opinionated approach for a given task. The human's opinion is a derivative of their lived experience, sometimes foreseeing all the way to the end result an AI cannot foresee. Eg. I want a database column a certain type because I'm thinking about adding an E-Commerce feature to my CMS later. An AI might not have this insight.
Of course, you can't always tell the model what to do, especially if it is a repeated task. It turns out, we already solved this decades ago using algorithms. Repeatable, reproducible, reliable. The challenge (and the reward) lies in separating the problem statement into algorithmic and agentic. Once you achieve this, the $1000 token usage is not needed at all.
I have a working prototype of the above and I'm currently productizing it (shameless plug):
https://designflo.ai
However - I need to emphasize, the language you use to apply the pattern above matters. I use Elixir specifically for this, and it works really, really well.
It works based off starting with the architect. You. It feeds off specs and uses algorithms as much as possible to automate code generation (eg. Scaffolding) and only uses AI sparsely when needed.
Of course, the downside of this approach is that you can't just simply say "build me a social network". You can however say something like "Build me a social network where users can share photos, repost, like and comment on them".
Once you nail the models used in the MVC pattern, their relationships, the software design is pretty much 50% battle won. This is really good for v1 prototypes where you really want best practices enforced, OSWAP compliant code, security-first software output which is where a pure agentic/AI approach would mess up.
heard from a vp recently, software developers are committing seppuku.
I can't tell if this is genius or terrifying given what their software does. Probably a bit of both.
I wonder what the security teams at companies that use StrongDM will think about this.
I doubt this would be allowed in regulated industries like healthcare
On a sidenote, congratulations to the strongdm team for getting acquired by delinea.
Can you disclose the number of Substack subscriptions and whether there is an unusual amount of bulk subscriptions from certain entities?
I recently passed 40,000 but my Substack is free so it's not a revenue source for me. I haven't really looked at who they are - at some point it would be interesting to export the CSV of the subscribers and count by domains, I guess.
My content revenue comes from ads on my blog via https://www.ethicalads.io/ - rarely more than $1,000 in a given month - and sponsors on GitHub: https://github.com/sponsors/simonw - which is adding up to quite good money now. Those people get my sponsors-only monthly newsletter which looks like this: https://gist.github.com/simonw/13e595a236218afce002e9aeafd75... - it's effectively the edited highlights from my blog because a lot of people are too busy to read everything I put out there!
I try to keep my disclosures updated on the about page of my blog: https://simonwillison.net/about/#disclosures
Is this article sponsored by StrongDM?
1 reply →
how about the elephant.. Apart of business-spec itself, Where-from all those (supply-chain) API specs/documentation are going to come? After, say, 3 iterations in this vein, of the API-makers themselves ??
… all this hype, and just … where are the macro level results? GitHub is seemingly having more outages than ever before, and MS is pretty directly involved in the AI hype; should they be a beacon of how great AI's output is? Yet obvious bugs that have persisted for years still languish. My day job is more and more feeling like I'm fighting just to get tooling or services to do the most basic things they were allegedly designed to do.
On the other hand, you have AI companies claiming "we built a browser from scratch" — and then having that claim utterly eviscerated. I cannot fathom going to my boss and requesting "$1,000/day per engineer" for AI — that's an absurd amount of money.
And yet, whenever I actually do try to get AI to meet the road … it is just query and query after clarifying query that humans would not and do not need. E.g., trying to ask clarifying questions about tsc, and TS, and … just wrong answer after wrong answer, or even just misunderstanding the question entirely. Trying to file a support ticket in big clouds now requires wading through AI slop that doesn't solve anything, just to get to a human whose writing feels suspiciously like you've been shoveled a second dose of slop. Like I just finished a quarter long ticket with GCP on "IAM is not functioning to spec, and we can clearly and concisely prove it" to get back a very long form "we don't care".
This is just sleight of hand.
In this model the spec/scenarios are the code. These are curated and managed by humans just like code.
They say "non interactive". But of course their work is interactive. AI agents take a few minutes-hours whereas you can see code change result in seconds. That doesn't mean AI agents aren't interactive.
I'm very AI-positive, and what they're doing is different, but they are basically just lying. It's a new word for a new instance of the same old type of thing. It's not a new type of thing.
The common anti-AI trope is "AI just looked at <human output> to do this." The common AI trope from the StrongDM is "look, the agent is working without human input." Both of these takes are fundamentally flawed.
AI will always depend on humans to produce relevant results for humans. It's not a flaw of AI, it's more of a flaw of humans. Consequently, "AI needs human input to produce results we want to see" should not detract from the intelligence of AI.
Why is this true? At a certain point you just have Kolmogorov complexity, AI having fixed memory and fixed prompt size, pigeonhole principle, not every output is possible to be produced even with any input given specific model weights.
Recursive self-improvement doesn't get around this problem. Where does it get the data for next iteration? From interactions with humans.
With the infinite complexity of mathematics, for instance solving Busy Beaver numbers, this is a proof that AI can in fact not solve every problem. Humans seem to be limited in this regard as well, but there is no proof that humans are fundamentally limited this way like AI. This lack of proof of the limitations of humans is the precise advantage in intelligence that humans will always have over AI.
> Code must not be written by humans
This might be okay short term, e.g. when you just want to get something done.
Maybe not on the scale of decades, otherwise we will end up completely unable to code.
> Code must not be reviewed by humans
This is where it all goes to crap. Until the day when the AI agents can look at some output and be like "no, this is overengineering, this can be done way more simply, let's stick to the established patterns within the codebase" and do so consistently, not having oversight will compound failure. I don't mean between every single small change, but rather at least to catch failures before merging anything.
This could only be avoided if you could define a harness with thousands or tens of thousands of tests per codebase, encapsulating EVERYTHING that must and must not be done within the codebase, down to the way how gaps and colors and utility classes are to be used, which I don't see most people doing.
This will take both better models and 3-10 years of work to actually make using them more foolproof and consistent. Even so, context sizes might need to be in the hundreds of thousands of tokens for the average task, all of those human "hunches" and styles and approaches to a given codebase spelled out.
> evaluating success often required LLM-as-judge
This is just advocating for doing what's easy, not proper - this will lead to slop long term. Though maybe they say that's okay, as long as it works.
Most of the time you would want those checks to be more dependable than that, the same way how you wouldn't want your linter to have randomness.
> Tests can be reward hacked - we needed validation that was less vulnerable to the model cheating
Just have adversarial agents, the one that writes the code doesn't touch the tests and vice versa, even though each has all of the context, each is told to care about different things.
Seems like these people are trying to push an envelope, and it might look like it's going to work in some respects, but they're very much taking big risks on what's currently feasible vs not.
> If you haven't spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement
Would be nice not to be broke, though.
Renaming or redefining “tests” because the LLM gets confused by the word is the tell.
Gas Town, but make it Enterprise.
Nailed it
Relevant blog post from simonw: https://simonwillison.net/2026/Feb/7/software-factory/
To the article’s author: what is the timeline for removing human engineers from your own organization?
I imagine it's even easier to remove the CEO/Executive staff. Actually, why have anyone there at all? Surely this company can LLM their way to having no staff whatsoever!
Yeah, extraordinary claims need some internal consistency before external evangelism. I’d expect the same from other companies whose CEOs make these kinds of claims, like Nvidia and Anthropic.
It's bad enough that a new programming fad washes over the industry every five years or so; some progress still manages to squeak through. It's going to absolutely grind to a halt if we're just getting a new black box oracle with different cargo cult rituals that have to be heuristically discovered every six months
So much of this resonated with me, and I realize I’ve arrived at a few of the techniques myself (and with my team) over the last several months.
THIS FRIGHTENS ME. Many of us sweng are either going be FIRE millionaires, or living under a bridge, in two years.
I’ve spent this week performing SemPort; found a ts app that does a needed thing, and was able to use a long chain of prompts to get it completely reimplemented in our stack, using Gene Transfer to ensure it uses some existing libraries and concrete techniques present in our existing apps.
Now not only do I have an idiomatic Python port, which I can drop right into our stack, but I have an extremely detailed features/requirements statement for the origin typescript app along with the prompts for generating it. I can use this to continuously track this other product as it improves. I also have the “instructions infrastructure” to direct an agent to align new code to our stack. Two reusable skills, a new product, and it took a week.
Sorry if rude but truly feel like I am missing the joke. This is just LinkedIn copypasta or something right?
My post? Shiiiii if that’s how it comes across I may delete it. I haven’t logged into LI since our last corp reorg, it was a cesspool even then. Self promotion just ain’t my bag
I was just trying to share the same patterns from OPs documentation that I found valuable within the context of agentic development; seeing them take this so far is was scares me, because they are right that I could wire an agent to do this autonomously and probably get the same outcomes, scaled.
Please let’s not call ourselves “swengs”
Is it really that hard to write “developer” or “engineer”?
Amusingly I use that term that to avoid the “not an engineer” and “I don’t make websites” comments. But noted, Tu.
Most of what people like this are actually creating is blogslop.
$1000 a day is such an odd choice of rule... why not just $50? Why not $10000?
> If you haven't spent at least $1,000 on tokens today per human engineer
So a four person team should be spending close to $1M/year, double each engineer’s salary, on AI alone? To get the output of one junior engineer who smokes crack and has his memory wiped every twenty minutes?
If that team is producing 4x what they would be producing without LLMs then spending 2x their salaries on tooling sees financially rational to me.
(I know, that's a very big "if".)
There is no mention of a productivity increase anywhere in this piece. 2x? Maybe, but at this price you can hire another very senior engineer, and have both be 50% more productive with a $200/month AI spend.
3 replies →
There are many ways "producing" can be quantified (LOC, PRs, features) such that 4x production does not correlate to a 4x in value of the product (quality, revenue).
Doubling? Try quadrupling outside of silicon valley. He is saying hire 4x as many engineers and make 3/4 of them AI. So much for the 10x productivity increase — that's 0.25x!
So, what does DM stand for?
Domain Model (https://strongdm.com)
Do you have any successful examples of software build by this factory in a main that isn't AI-adjacent? I.e. not an SaaS clone, not another code linter/database/whatever? Can the factory build, e.g. a crypto trading algorithm, an e-commerce site, etc.?
To come up with a software spec for the agents to follow, first you need to understand what your business is and where the money is.
Thanks. I’m unable to find the term “domain model” on the website.
2 replies →
Doomy marketing?
"These go to 11" The context behind it
https://m.youtube.com/watch?v=4xgx4k83zzc&pp=ygUOdGhlc2UgZ28...
The Digital Twin Universe is the most interesting thing in this article and the part most people are glossing over. The real question Simon nails is: how do you prove software works when both the implementation and the tests are written by agents? Because agents will absolutely game your test suite - return true, rewrite assertions to match broken output, whatever gets them to green.
Their answer of keeping scenarios external to the codebase like a holdout set is smart. And building full behavioral clones of services like Okta, Jira, Slack so you can run thousands of end to end scenarios without hitting rate limits or production - that's where the actual hard engineering work is. Not the code generation, the validation infrastructure.
Most teams trying this will skip that part because it's expensive and unglamorous. They'll let agents write code and tests together and wonder why things break in production. The "factory" part isn't the agents writing code. It's having robust enough external proof that the code does what it's supposed to.
(DTU creator here)
I did have an initial key insight which led to a repeatable strategy to ensure a high level of fidelity between DTU vs. the official canonical SaaS services:
Use the top popular publicly available reference SDK client libraries as compatibility targets, with the goal always being 100% compatibility.
You've also zeroed in on how challenging this was: I started this back in August 2025 (as one of many projects, at any time we're each juggling 3-8 projects) with only Sonnet 3.5. Much of the work was still very unglamorous, but feasible. Especially Slack, in some ways Slack was more challenging to get right than all of G-Suite (!).
Now I'm part way through reimplementing the entire DTU in Rust (v1 was in Go) and with gpt-5.2 for planning and gpt-5.3-codex for execution it's significantly less human effort.
IMO the most novel part to this story is Navan's Attractor and corresponding NLSpec. Feed in a good Definition-of-Done and it'll bounce around between nodes until it gets it right. There are already several working implementations in less than 24 hours since it was released, one of which is even open source [0].
[0] https://github.com/danshapiro/kilroy
Been toying around with DTs myself for a few months. Until December, LLMs couldn't correctly hold large amounts of modeled behavior internally.
Why the switch from Go to Rust?
Are the digital twins open source anywhere, or available as a service somehow? They sound useful to use!
[dead]
11 replies →
Strongly inclined to agree here: Having recently joined a small applied AI startup and we were discussing the need for E2E tests. My initial gut reaction (which I kept quiet) was that such things turn into unmaintainable messes which delay releases and increasingly reduce in value.
I recognised this was grounded in an entirely different world of software engineering and organisation size though. I followed a path of thinking about what went wrong historically and how might they be solved: Better structure, discipline, resources - all of the things which agentic AI facilitates.
You are right about most skipping this part: But I view it as being like a sewerage and sanitation system - largely invisible and not thought about but critical for long-term health.
Also this ties in very nicely with Netflix's approach to Chaos Engineering and enabling it at broader scale.
> You are right about most skipping this part: But I view it as being like a sewerage and sanitation system - largely invisible and not thought about but critical for long-term health.
And like sewage and sanitation the infrastructure is a lot more complicated than people think.
I’m curious what happens when they need to make a DRU of Stripe or another payment processor.
High-quality digital twins of complex software does not bode well at all for a lot of SaaS companies.
For customers, it makes migrations much easier and less-risky between vendors.
For the vendors themselves, it means you can cheaply and reliably port features your competitors have that you don’t have.
You have a different agent write the tests and another run the tests. You tell them each that they aren’t checking their own work, they’re checking someone else’s. You can tell them to be skeptical. Then you can also tell them that don’t fail the code for no reason, because a third agent will be checking your tests and you will be penalized for inaccurate testing.
This approach balances out and maximizes accuracy.
> just use psychological tricks on the LLM, bro, you'll cajole it into not hallucinating
Can't help but chuckle at that.
1 reply →
At first I was partially impressed by the Digital Twin Universe thing they describe. Having worked with 3rd party APIs in a previous life, having something like that would've been so much help.
But after thinking about it more, I think it must be the lowest of low hanging fruits for LLMs. You're building something with well defined specs, most of which is readily available by the original creators, with a UI that only does the bare minimum, and it doesn't need any long-term features like reliability since it's all for internal short-lived use. On top of that, it looks super impressive when used in a demo, because all those applications being mocked are very complicated pieces of software. So to recreate a thin facade of them can look very impressive. And calling it a "Digital Twin Universe" is just icing on the cake.
It is suggesting that we will move towards an “everything must have an API” world.
But at some point you get back to tests, because they are simpler to write.
This is a child of the “no handwritten code” rule. Since they can’t steer tests, they have to do something else to ensure quality.
This is only worth it if the added cost and overhead is cheaper than writing the code.
This seems like it will pull towards building a simulation of your firm, for the simulation to work? Or simulations of your process?
I don't think this is meaningfully different than the human case for the past 20 years. Every large project I've worked on had people writing tests that didn't test anything and people who argued strongly when I pointed out glaring missing test functionality coverage. And their managers did not like to spend money on having better tests written.
[dead]
I feel like I'm taking crazy pills. I would avoid this company like the plague.
[dead]
[dead]
[dead]
Cringe-worthy at best