Comment by corv
3 hours ago
I like your constructive outlook and I want to believe that we can now focus on the bigger picture, but I'm not really seeing it around me. Most everyone I know that's "AI-pilled" has spent the newfound capacity on throughput instead of altitude.
Is anyone actually seeing a shift towards improved structure rather than more code, faster? We seem to be living in the Gatling-gun version of the picture John Carmack drew when modernity also ushered into precision marksmanship...
Are those people working on products or platforms? Platforms still have to be rock solid and deliver performance guarantees to the products built on top of them.
>> Is anyone actually seeing a shift towards improved structure rather than more code, faster?
Yes. At work we recently finished a complete rewrite of the platform. The old codebase got abandoned and two new codebases got stood up. Previous stack was Phoenix LiveView and the new one is Phoenix API + Vue /w TypeScript. Zero code shared between the two. We took the opportunity to re-architect a lot of the core functionality and get rid of the tech debt that had been hounding us and killing our velocity for the past six years.
We finished the rewrite in two months. A little over 700k LoC total. 95% of it was planned, orchestrated and written by AI. We could have gotten it done faster, but we were well aware of the potential for slop and thus paid an absurd amount of attention to (and tokens for) code quality. For example, there is a 100% test coverage requirement in both server and client, combined with AI-driven review rules that say all tests must be non-vacuous, plus a ton of invariants enforced via deterministic checks. Everything from standard linting and formatting rules to AI-necessitated stuff like "comments cannot be more than five lines each" is in there. At the end of each week, we get together for ~30 minutes to review each bug that was fixed that week (there haven't been that many) and try to figure out if more deterministic checks can be added (or new ones extended) to prevent that class of bug from re-appearing. We actually have an Architecture epic with a dedicated Fable agent orchestrating all the initiatives and tickets in it and keeping track of improvements (and the occasional regression). So things like accessibility, usage of shared helpers, enforcement of common patterns, etc. are all kept track of.
The difference between the old codebase and the two new ones has been night and day. The new codebases, combined, are ~35% larger than what we had before, and yet there are way fewer bugs (we actually got suspicious at first and had to check to make sure our logging instrumentation was configured correctly). Performance is also stellar, features have been a lot easier to add and extend, and everything is a lot easier to find. When someone asks a question or has an issue we have a pretty good idea of what part of the codebase to look in and we're usually right.
It wasn't all unicorns and rainbows. Not everyone on the team was proficient with AI-driven development, so they had to be brought up to speed quickly and taught the tools and also what works and doesn't. Code reviews were also difficult and time-consuming. We had to deal with our anxieties about working in two new codebases none of us had authored much code in ourselves. What made all of it worse was that we were racing towards a deadline imposed by external factors so we all ended up working a lot of hours. It was essentially between getting it done this summer or waiting until next summer and we bit the bullet and did it. It absolutely would not have been possible without AI.
> For example, there is a 100% test coverage requirement in both server and client, combined with AI-driven review rules that say all tests must be non-vacuous,
look, your application works, right? so it doesn't really matter what you or i think, and this is why AI matters. but this, your "100% test coverage" - that is pure slop. just 20 years ago, all the most popular software shipped with NO tests. are you getting it?
What kind of software are you talking about? Test harnesses were commonly used in 2006. JUnit was created in 1997.
> just 20 years ago, all the most popular software shipped with NO tests
not sure what you're point is here. It sounds similar to "we use to use blood letting and leeches and doctors didn't clean their hands and everything was fine so what are you getting at?"
Good tests have real benefits. The fact that people shipped without them in the past in no way suggests they aren't needed or have no point.
And 80 years ago cars didn't have seatbelts. Your point?
>> but this, your "100% test coverage" - that is pure slop.
Not really, but I can see why some people think that.
We treat 100% test coverage as "required, but by itself not sufficient". It doesn't give us false confidence that everything will be perfect or anything like that. But it provides us with the discipline to make sure no corners are cut, and the bugs that are fixed don't come back.
One refreshing aspect was that during PR reviews we stopped debating whether something needed test coverage. Instead we focused on what was being tested and how.