Comment by sarchertech
19 hours ago
An LLM isn’t a compiler because there’s no formal method for translating prompts to code. It’s chaotic and small changes in prompts will result in vastly different code.
Come back to me when your app has users and adding new features subtly (or not so subtly) breaks every work flow that you haven’t explicitly tested.
You can’t commit the prompt and regenerate the source code each time because the whole reason that an LLM is useful is that it makes thousands of decisions for you. And those decisions are different each time you regenerate.
The only way to enforce that those decisions are the same each time you regenerate is to encode all of them in tests. But at even moderate complexity that leads to an overconstraint problem that will halt development.
We see this when using LLMs on large apps. Anthropic gave up on their C compiler. Even with an unlimited budget they stopped being able to make progress on it.
I see this in some games I made for my 4 year old. I had them one shot some “juice” when he gets an addition problem right. Combination of screen shake, sounds, flashing light, explosions etc…
It looks pretty cool, but when I tried to tweak the animations with prompts it was always worse. I eventually went in and edited the code myself and I could see why it was so hard for the LLM to change anything because it was a horrific mess of interwoven animations.
I was able to pull everything apart and manually adjust what I wanted.
What helps is to ask the model to make design docs and to note down features and how they work in md files. It sort of provides another layer of persistence separate from code.
But yeah, overall you have to be ok with the app being approximate too. Maybe after an update a button is a different size, or in a different place, or it suddenly has an animation to it. Those smaller things are a bit harder to control when making changes at scale, and if not clearly documented.
For me this is not necessarily a big drawback, for things like games, the core game loop, performance and game feel are a lot more important than any small UI tweaks.
Hopefully, the better the models get, those side-effects will only be improvements, not degradations.
> But yeah, overall you have to be ok with the app being approximate too. Maybe after an update a button is a different size, or in a different place, or it suddenly has an animation to it. Those smaller things are a bit harder to control when making changes at scale,
Now come back to me when you have paying users.
Better yet come back when you have paying users who depend on your app to do their job. And in addition to buttons changing location, you are constantly breaking their work flows because they are using the app in ways you didn’t anticipate.
I do have paying users for https://uxwizz.com
I only started using AI for development for this product (13 years developed without AI) a few months ago, and customers are really happy with the changes.
I managed to implement feature requests that were pending for years. It took probably 1 month to implement what would have taken 1 year without AI.
3 replies →
> overall you have to be ok with the app being approximate too
And with epsilon big enough any output it good output...