Comment by blueblazin
7 hours ago
Yes! I absolutely have! And I've been thinking a lot lately about how to prevent them from collapsing into this state. It is getting better with newer and better models though. But yes, there comes a point when the codebase size and complexity crosses a threshold and we enter a churn state with very little progress.
I was motivated to make the codebase visualizer linked above to help me understand how the various pieces of the system that is being vibe coded connect with each other.
Now I'm thinking about how I can build something similar for agents. How can we teach "systems thinking" to a model?
Have you seen this with Opus 5.5?
I've been mostly working on very large modeling projects (think detailed architeture over 1000 acres) or animation because this really was the "new" capability area that Opus 5.5 "unlocked".
I have been surprised how well it has worked at scale with many sub-agents. However the code base is limited to the rendering engine mostly.
Certainly in the older models they hit a complexity limit, but I haven't really hit it with Opus 5.5. I have had it make data flow designs that I didn't like because they were more complicated than it should have been or the perenial problem of trying to make everything backwards compatible when working on greenfield refactors, but I haven't seen it choke on a code base yet.
Yes, I've also hit it with Opus 5.5, but on way fewer projects. My long standing projects that I marked as failed/to-soon with 4.8 have entered the grey area of slow-progress-with-handholding like smaller projects did in the past.
To my experience, it's a simple complexity curve where the first bits go fast and the speed goes down very fast once you reach a certain level of complexity. Size doesn't matter as much as complexity: with the right foundations, it keeps zooming through it. Now I do wonder how much my inexperience in the complicated project is a brake on progress vs. it just being a hard to tackle problem.
My sense is the limit is more tied to the problem than the code base size. I work in some very large code bases, but it's not an issue because the nature of the problem (think what a standard SaaS would need).
On the other hand, I have a TS only project core is sub-20k LoC (NES harness), where it is struggling (for now, see other comments)
I just have it constantly update real-world analogues for keeping track of everything. I have it create guides, note cards, flowcharts, running to-do lists, etc. with every change.
It helps us keep track of things, why not AI, too?
> How can we teach "systems thinking" to a model?
I've been working on process more than model. Similar to having a PR review team, you need multiple teams or processes that run over the code, markdown, assumptions. I still don't think this is enough.
I'm working on an NES harness, and the agent was churning with detecting certain state changes like dialogs, menu navigation, screen change. The bot and I got churning and I put it down for a while. A few days later it occured to me to ask the bot "Does the emulator and game state change with frames? Can we just get the state each frame rather than trying to be clever about it?"
It said yes, but added some nonsense about being performance hindered... (flash model). I haven't worked on it again yet, but my intention was to next have it build a small recorder based on this so it can get frame accurate game state change.
This is just a long way of saying, sometimes you have to step back and help them help themselves. They have a really hard time with recognizing ambiguity or how to determine lack of context that could be filled.