Comment by nemo1618
2 days ago
I am quite tired of this take, frankly. The implication is that if we continue iterating on prompt optimization, we're going to reinvent what, JavaScript? BASIC? Lisp?
English is not a programming language. Yet English is sufficient to communicate requirements to the degree that we actually care about. A programmer's job is to translate English into lower-level machine language. Necessary to this process is "filling in the gaps" -- that is, extrapolating the expressed intent to cover all the little details that were left unspecified. This system works because humans are at least minimally competent at predicting the preferences of other humans. If your prediction turns out to be wrong, you get feedback and iterate.
Well, guess what. LLMs are also competent at predicting the preferences of humans. LLMs can "fill in the gaps" like no one's business. LLMs can iterate on requirements like no one's business.
Product managers do not speak to programmers in a language that encodes exact requirements, and yet working software somehow gets shipped anyway. LLMs do not need exact requirements either.
Maybe I’m behind the curve here because I work in an SRE/DevOps context as of late - but LLMs routinely shit the bed and fail to solve basic issue for me when I try to use them (Thanks, Management)
I don’t need a model to shit out a REST endpoint. I need it to figure out esoteric errors that take hours or days of debugging. They just don’t do well here. Of course, if a diligent engineer refined considerations from a PM and Engineering Manager I wouldn’t have the job I have.
Further evidence that there is some kind of weird parallel universe thing going on with LLMs. "Figuring out esoteric errors" is one of the things I would cite as a particular strength of agents. I am repeatedly amazed at their ability to root-cause weird behavior on my systems. Here is one example: https://xcancel.com/lukechampine/status/2047032091053859138
I had an experience where we were getting some unusual behavior in a new release of a data store (for various reasons main hadn’t been released in quite a while). We gave Claude code the weird symptom details and the git repo and the tags for the good version and the bad version and it found the subtle bug. This was six months ago.
Now unlike the SRE case, we were the dev team and understood exactly what the logging meant as far as a problem goes, so our prompt started with the correct 0.1% of the system to look at. SRE typically has to start by finding that 0.1% slice from rather more generic metrics. And their interventions have higher risk than a controlled rollout of new code with a specific fix.
This is my experience as well. The two areas I get the most leverage out of AI are root cause analysis and vibe coding green field prototypes.
I dunno, Claude announces it found “the smoking gun” every single time.
Yes, it IS confident of its weights & biases... but, if you keep at it, Claude WILL find the smoking gun, eventually. Even a broken watch is correct twice a day ( unless it's a digital watch, without a battery, in which case, it's just broken... )
But seriously -- newer Claude (and OpenAI and Google and ???) models DO find the smoking gun, if you let them keep going until they reveal the weird chain of events that leads to a bug. I was seeing the most obscure UART driver bug, where it would work at 1,500,000 baud (!) but fail by only outputting the 1st char at 230.4k and 460.8k -- and it was due to a very narrow race that would check the buffer, if not full, insert a character, and return BUT sometimes the TX Complete interrupt would happen between the check and the insert, and something else would insert, and then - buf overflow. At 1,500,000 the other process didn't have time to do that phantom insert. ANYWAY, Claude found this and proposed a fix -- simpler: spins on IRQ-protected buffer empty checks.
I'd hate to think how long it would have taken me to find that.
And THAT's the problem -- of course a human CAN find it, with sufficient focus and time; I'm sure you've found a complicated bug pretty easily sometimes, by sheer luck or good engineering instinct.
BUT, it seems to me, as human, we are capable of creating potential execution paths that EXCEED our ability to EVER figure it out -- due to not being smart enough, not enough time on the problem, or something makes it economically unfeasible.
THIS is where LLMs shine -- let 'em bang at the code for as long as it takes.
The recent Mythos bug-finding explosion I think is proof of this conjecture. I think of it like a chessboard, where a machine really can look at all possible execution paths, and locate obscure bugs; a human programer (akin to a chess program) is doing 'alpha-beta pruning' of what's likely, and only after that list is exhausted are the really weird possibilities examined.
LLMs are our friends. And, as for "WTF did the LLM just do" when it generates code? I always include the instruction "For this code you just wrote, use Best Practices to document this code, function by function and class by class, and when necessary, line-by-line, so that a junior SW developer can completely understand how this code works, using the documentation standard we use (e.g. Doxygen)."
I have also used this technique to learn new languages, or explore ones I only know a little -- it has been a godsend for leveling me up on common lisp, for example. "Give detailed comments explaining what the code is doing, assuming the code reader is fluent in C and Python, and use analogs when possible." Stuff like that.
I'm going to assume you are a product manager, or someone who does not do the actual software development. The place where you draw the line of abstraction, is what I'm basing that assumption on.
The reason why a PO can explain something in English, and you get something useful out at the other end (of the developer), is because of a myriad of other decisions you don't see. The reason why some software systems end up being efficient in maintenance and further development, is because of these myriad of other decisions.
The many decisions are the "devil in the details" that LLMs don't get right. Or, let's not anthropomorphize unnecessarily -- LLMs don't know right from wrong, and don't reason or reflect. They could only get this right by sheer luck. In a big numbers game, they'll always get it wrong. If you want to be a PO (or vibe coder, etc) and use English language on one end, and get these details right, there is only one possible approach:
A tight loop with expert knowledge reviewer. The programmer that knows pretty much what they want, in a small section. A LLM can draft it out so that the programmer saves time typing. This isn't really useful for the PO. (PS: The same general advice applies for any other use of LLMs. Tight loop. Expert reviewer)
You'd need a language that can express important details otherwise lost to the English language. And you'd need this to be deterministic. The "myriad of tiny decisions" are the true basis for the code implementation. If they're not expressible in the English language, and they're not achievable by LLMs, there really isn't any other way to achieve them.
As long as LLMs need their users to "engineer" prompts for them to actually work well, clearly LLM-driven development is not exactly equivalent to back-and-forth conversation with a human programmer. In principle if what you say were true, software developers already wouldn't be needed anymore, because coding agents could talk to the customer directly and give rapid feedback to clear up even the smallest details. It should be uniformly better and cheaper than talking to a human developer, but that's not the world we currently live in. Currently we live in a world where LLMs that can disprove conjectures and find zero-days still need context engineering. We shouldn't anthropomorphise and assume that they will be exactly the same as us, they are undoubtedly useful but they need careful steering by specialists who understand how LLMs interpret prompts.