Comment by tenthirtyam
19 days ago
My expectation is that there'll never be a single bust-up moment, no line-in-the-sand beyond which we'll be able to say "it doesn't work anymore."
Instead agent written code will get more and more complex, requiring more and more tokens (& NPU/GPU/RAM) to create/review/debug/modify, and will rapidly pass beyond any hope of a human understanding even for relatively simple projects (e.g. such as a banking app on your phone).
I wonder, however, whether the complexity will grow slower or faster than Moore's law and our collective ability to feed the AIs.
Maybe software systems will become more like biological organisms. Huge complexity with parts bordering on chaos, but still working reasonably well most of the time, until entropy takes its course.
It's already like that, for a long time. Humans are quite capable of creating complex systems that become unwieldy the bigger they get. No one person can understand all of it. I will offer the AT&T billing system as an example that I'm all too familiar with as a customer, due to the pain it causes me. So many ridiculous problems with that system, it's been around a long time, and it is just so screwball.
Biological systems are vastly more complex, and less brittle, in the sense that killing a single cell doesn't cause system failure like for example removing an object file from a program often would. Just look at the complexity of a single sell, and try to imagine what an analog of similar complexity would be in software.
3 replies →