Comment by galaxyLogic
6 days ago
That is a great insight, AI code is like legacy code, which is "legacy" because it was written long time ago and hasn't been worked on by anybody, so nobody knows and it is hard to learn how the code does its thing.
But so why couldn't AI-generated code be maintained by AI? Why couldn't AI in general be used to fix and evolve legacy code?
My two insights are:
Legacy code is difficult to work with because the reasons "why" something has to be some way are missing. The "whys" can change over time, be interwoven with business requirements not in the codebase, and compound/nest with other whys that have come and gone. Often only humans in the business know, and you have to figure out who and what to ask.
Second insight is that LLMs have so far been very bad at holding on to the thread of winding, changing requirements, which is something it would have to be good at in order to revisit old codebases. Even one it wrote itself. It would need some kind of historical changes to business requirements database, which I have never seen a business keep, it'sjust the collective human knowledge of the business. Otherwise it'll make changes without accounting for historic requirements as well as your new ones.
I think there are probably some workarounds, like getting it to write lengthy documentation to refer to. Long term perhaps it just gets smart enough. Legacy codebases are something I specialise in working with somewhat, I spent a long time consulting so I was often parachuting into unfamiliar codebases and trying to quickly understand years to a decade or more, of evolving requirements in a business/codebase. I have a lot more to say on the topic but those are two of my insights.
Good point. Whys are lost because whys are often never written down. World around us always changes, so the whys change too.
This may have something to so with the AI "frame-problem".We can't document all "whys" in the current "frame" because they depend on other "frames" like the current legislation. We don't write down "this will work because ..." . We just assume it, and asume many other causally related things.
In other wrods it would very hard or impossible to document "what must not change" for our system to continue to be useful. For instance Sun must still exist and air must be breathable for this system to be useful.
That’s already addressed in the article; it’s like paying off credit card debt with another credit card.
I guess it all depends on your end goal. If you intend to make your vibe coded app into a profitable, maintainable product with any sort of reliability guarantees you better understand its failure modes and how to fix things when they break. It’s really hard to create recovery procedures for a black box.