Comment by jillesvangurp
3 days ago
Most of the examples here are big government IT projects. But it's unfair to single out software projects here. There are a lot of big government projects that fail or face long and expensive delays. A lot of public sector spending is like that. In fact, you'd be hard pressed to find examples where everything worked on time and on budget.
Mostly the issues are non technical and grounded in a lack of accountability and being too big to fail. A lot of these failures are failing top down. Unrealistic expectations, hand wavy leadership, and then that gets translated into action. Once these big projects get going and are burning big budgets and it's obvious that they aren't working, people get very creative at finding ways to tap into these budgets.
Here in Germany, the airport in Berlin was opened only a few years ago after being stuck in limbo a decade after it was supposed to open and the opening was cancelled only 2 weeks before it was supposed to happen. It was hilarious, they had signs all over town announcing how they were going to shut down the highway so the interior of the old airport could be transported to the new one. I kid you not. They were going to move all the check-in counters and other stuff over and then bang on it for a day or two and then open the airport. Politicians, project leadership, etc. kept insisting it was all fine right up until the moment they could not possibly ignore the fact that there was lots wrong with the airport and that it wasn't going to open. It then took a decade to fix all that. There's a railway station in Stuttgart that is at this point very late in opening. Nuclear plant projects tend to be very late and over budget too.
Government IT projects aren't that different than these. It's a very similar dynamic. Big budgets, decision making is highly political, a lack of accountability, lots of top down pretending it's going to be fine, big budgets and companies looking to tap into those, and a lot of wishful thinking. These are all common ingredients in big project failures.
The software methodology is the least of the challenges these projects face.
It is not just government. Private companies also have the same problem.
One reason why aws got so big is because it took months to get infrastructure to provision a virtual machine.
There's an obvious imminent selection bias with government projects because they're by nature subject to public scrutiny, plus the stakeholders are literally everyone. Private companies can waste billions internally and it'll never make it into the news.
In my first big job in a big legacy company, 30% of ongoing effort was "how to implement this feature which needs a database without a database".
We also paid some security company to use it as a proxy in front of our server to implement some server redirects because it was simpler than configuring our own servers. Simple one-liner conf changes were a week of emails with support staff.
something left out there with government though is incentive misaligned and hence corruption, which is smaller in a private scale (exists nonetheless)
I think if we look at the lack of accountability it's obvious that one major problem is that many of these projects do heavily rely on contract work. No company or gov in the world can supply the perfect brain- and manpower necessary on day one (on a huge and complex project that requires expert knowledge). So there is an prevalent delusion that talent just spawns at project kickoff and those people even care about what they do.
Maybe this is some artifact we carried over from the industrial era. We expect that complex machinery is built by expert companies over night and they just work, with a bit of maintenance and knowledge transfer. But software doesn't work like that.