Comment by jerf
3 days ago
QA is becoming the blocker for projects in an AI era, and the real source of value. AIs can lay down a good base of testing but I see even frontier models still need help with testing because they are basically as bad as humans in terms of testing only what comes to mind, if not even a touch worse. Though they are really good at blasting through problems once they are found.
Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services may be happy to instantiate some front office system for themselves via AI prompt, but they aren't going to want to QA that post-AI any more than they want to now. Once we all settle down, SaaS will manage more code than before, and do more QA, and Joe Bob's etc. will be even less interested in trying to replicate the work with AI because it will represent an even larger QA load than it does today, relative to the size of their business.
Or, to put it in the context of this article, even if you can prompt a database into being today, which you can, you can't prompt something into existence that has been tested and QA'd like Postgres, and you would be well advised to ponder that gap between your prompt and billions of computer hours of field testing very carefully before you decide where to store your company's most important data.
> Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services
OK, so this is slightly different since it's more about how runtime behavior than design-time behavior, but... I'm in a SaaS workplace, and I've been trying to seed the following idea:
1. The very cheapest/fastest end of the market will be eaten up by Job Bob's Bait Shack stuff. Cheap wrappers around a prompt that goes "Pretend you're a computer that obeys the following rules" jank. A race to the bottom we can't win.
2. Most of our (desirable) customers will (eventually) realize they can't tolerate systems which casually or unexpectedly violate known business rules. That leads to angry consumers, junk data, broken workflows, and lawsuits.
3. When our product can capture and enforce those rules with confidence, that's both an important differentiator and also a kind of moat/stickiness.
4. Therefore there is a strategic business justification for Engineering to fence-off where we use LLMs in production. When we use them for design, it's to create better deterministic systems than before.
The last point is one I make often. AI cannot be trusted to be deterministic, but they can produce deterministic tools that are verifiable as such. Those tools I can trust as much as any other complex tool from the pre-AI era.
> pre-AI
I'd say pre-LLM, just because I don't want to surrender the term "AI" to one particular subtype without a fight.
Some of those complex tools would be "Good Old-Fashioned AI", which generally means stuff doing symbolic logic rather than mystery-weights.
I agree that it makes little sense for all but the largest customers to write their own software. Isn't there more of a threat from another software cloning the SaaS and being happy with smaller margin?
I don’t think I’ve worked at a place that wasn’t trying to automate away QA. Perhaps Delivery Hero pre-acquisition, since we had 2,5 QAs for 4 devs in my team.
So… are we gonna use the extra productivity (and money) from coding agents to hire QA people? Oh boy, is that highly unlikely.
But I like your take. I wish it were so, perhaps we wouldn’t have had Tahoe.
But, will that problem remain or is it simply that the models haven't been RLVRd on QA tasks yet? QA feels like a very RL suitable problem. You can take a codebase that has a known set of bugs in it, and proceed to train the model until it identifies those bugs. You can even generate such programs automatically by having adversarial training going on (one model deliberately writes a buggy program, another searches for the bugs). This works even with GUI bugs, drifts from the specs and so on.
I don't see any theoretical reason that this weakness must remain for longer than six months or so. I mean, both Claude and GPT are very good at doing code reviews and that's a form of QA.
The problem will remain due to information theoretic concerns. It is not possible for an AI to read minds. You can say to an AI "build me an accounting system for a small tax firm" and we can, for the sake of argument, even stipulate that it will produce a functional system based strictly off that prompt. However, no matter how smart the AI, the result can not be any more detailed than that. Did you want it for MacOS? It may have built it for windows. Did you want it integrated into an already-existing database server? It can't have done that, and since you didn't specify the database you want it could have guessed wrong, and certainly can't guess about what tables you may already have you want to integrate into. It probably didn't guess your jurisdiction correctly (unless it was fed that in some other contextual prompt). It can't guess your most common work flows and optimize them. It can't pick out specialized features you want just for your special use case.
None of that is because AIs aren't good. It's just plain information.
An AI could sit there and interrogate the user for this information. In that alone, it would already have exceeded what a lot of people are going to want from a system like this. People, normal people, want AIs to magic stuff into existence, not spend hours interrogating them for a design requirements elicitation. That's a barrier a lot of people will not cross, and the AI companies, who fully realize that, aren't building that way.
Normal people do not want to even do the work it takes to fully specify a problem, even if an AI walks them right through it, let alone the verification that the work is correct, and actually what they intended rather than what they asked for.
And of course, in practice, AIs are not emitting bug-free code, miss entire requirements at a shot, and have a variety of other characteristic failure modes in this space that I expect are fundamental to the LLM architecture and aren't ever going to leave. If they continue to be able to make bigger systems they will simply continue to make correspondingly big errors and omissions in the process. No comment as to what the next generation AI architecture will do, though.
Not to mention the blast radius of any small mistake in payment processing, tax filing, or any other financial or regulatory issue, is just so massive compared to making some ai slop bar special flyer. Joe's bait shack is still just going to pay for square.
There will be a ton of smaller Joe’s bait shacks, all competing against each other in a race to the bottom - and with relatively limited blast radius per shop.