Comment by busterarm
20 hours ago
This often massively discounts the cost of reversing decisions. People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them.
I worked at a large, publicly-traded multinational where decades prior and they were still just a 4 man startup they decided the database server and all timestamps should be in the local timezone.
They are still using EST today even when they have global sharding of their customers/databases between US, EU, LATAM, SEA...
--- you're also assuming that the product roadmap will afford your engineers any time to build it the second, third, fourth, etc. time.
There seems to be some general pattern here that you can find pretty often in "dev war stories" contexts:
(1) We're a small startup/new product team/etc, let's just build the MVP and keep everything simple!
(2) Now we're not small anymore and suddenly have all kinds of nonfunctional requirements we never imagined before! But our simple architecture from before is making everything a pain now!
The natural instinct is then to compromise on the "simpleness" of the first prototype and already try to anticipate all the scaling and nonfunctional requirements that might come later - but that rarely seems to work, as you can't really how (and if) the project will grow.
Seems to me, the real question here is why those teams are still using the "MVP" code even after being well inside the "scale up" phase. Shouldn't this be the point where you gradually migrate to a codebase that is more manageable at scale?
> The natural instinct is then to compromise on the "simpleness" of the first prototype and already try to anticipate all the scaling and nonfunctional requirements that might come later
This is a form of Second System Effect Brooks wrote about in 1975.
https://en.wikipedia.org/wiki/Second-system_effect?wprov=sft...
>> People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them
Because for maintenance there is a different team and budget. Same goes about the meetings. Recent example: it's okay to blow $1000 in man-hours to discuss and reject an increase of the compute capacity of my virtual workstation. I bet the cost associated with request was ~$25 of increase to the monthly bill. But it is from the other budget, so it was refused.
I sort of agree. But I kind of think it misses the point.
Bike shedding means no decision is made. That is the worst possible outcome as it provides absolutely zero value. Even a wrong decision provides at least some value even if it fails horribly. Maybe we paint the shed black and it burns down due to excessive heat, but it’s a bike shed so who actually cares if we need to rebuild the entire thing and paint it white this time.
And when it comes to bike shedding, getting it “wrong” is by definition already low stakes. It just doesn’t matter a whole lot.
Even the timezone example isn’t really that material IMO. Sure, it’s an extra annoyance to normalize timestamps to locale in every app/debugging query. But in the end it’s just not that big of a deal.
Worse would have been that early team arguing about it for 6mo without implementing anything. Then no one would be employed currently to curse those early developers to begin with.
Often times the only wrong decision is not making a decision. Very rarely are there decisions to be made that are not overtly obvious which are actually material. These exceedingly rare situations are where the winners from losers get sorted out - but that sort of decision in my experience so far happen a handful of times during an entire career.
As I moved up the ladder in my life I’ve often found my job is to basically flip a coin when a decision choice is presented to me. Very rarely is the topic worth me spending any material amount of mental energy on.
> Bike shedding means no decision is made. That is the worst possible outcome as it provides absolutely zero value.
And "bike shedding" is now a term that people hyperbolically apply to any discussion of something the labelling-participant doesn't really care about. A lot of times doing nothing is actually the best business outcome. Waiting until you have more, better information is great.
Lots of stakeholders ask for things that they think matter and then realize two weeks later doesn't and a ton of man hours have already been spent trying to build those skateboards as fast as possible.
> Even the timezone example isn’t really that material IMO. Sure, it’s an extra annoyance to normalize timestamps to locale in every app/debugging query. But in the end it’s just not that big of a deal.
Oh god are you underestimating the long term impact of data not being stored in UTC. For one, it will be a perfectly reasonable thing that your engineers forget this fact and build new systems in UTC, because that's the reasonable thing to do. Also as the years go on your company will make acquisitions of other companies and whatever timezones their data is stored in (hopefully UTC but obviously not always). To properly integrate this data you have to determine for every timestamp whether there was Daylight Savings Time happening at the time on every single object. You also have to figure out all kinds of edge cases around leap seconds.
Otherwise you have a ton of data that's out of chronology. Pity you if that's a thing that matters to you (it almost certainly does).
Also maybe you're lucky enough to be in a position to modify timestamps of all of your objects in all your datastores without significant engineering effort and coordination...but probably not. Obviously the reason the company I mentioned hasn't done it is because they're in that position...
But say you don't want to "fix it" and just want to do this "on every query"...now this process above is something you have to do across maybe thousands of services for potentially every request...
No big deal, he says...
> This often massively discounts the cost of reversing decisions. People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them.
This is often because no thought was put into making decisions reversible. Version control, CI/CD, virtualization for example can make it easier to reverse decisions and make it faster and easier to replace bad designs.
When you first introduce the idea you'll find a bunch of people think decisions are Reversible which are not, as you say. And the flip side of Reversible decisions is the Last Responsible Moment, which runs afoul of Hofstadter's Law, and people wait until halfway past the last responsible moment.
The key to reversing a decision is getting over Sunk Cost, to start thinking of some code as scaffolding. Scaffolding allows you to get on to other work and then remove it after, because it's either not needed or the 'real' solution has been installed. People get defensive when you propose to rip out their code. Hey that code made us $250k at a time when we were about to miss payroll. Yes. It did. Thank you for your service. But now it's costing us $30k a month and that shit needs to go.
Getting people to figure out that if a decision is important, making it later is actually the sane thing to do, is a challenge. Because many people's intuition is that we should put energy into this now while it's fresh and we have abundant energy. We can 'solve' it and not have it dangling over our heads. But we don't know the right answer yet. We don't know the strength of our tools or the expertise of our coworkers.
The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes.
(And everything should be in GMT unless you can literally point me to a several hundred page treatise on why another time zone is the correct one. Yeah I've worked west coast places that got bought by NY or Chicago companies and it's a clusterfuck if you both didn't use GMT)
Don't get me wrong, I'm a fan of the idea. I pretty much follow a lot of what you're saying without giving it names. I'm just used to people giving lip service to certain ideas as an excuse to move with less friction in their org and end up doing long-term organizational damage.
Thank you for the added detail.
> The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes.
Also agree, but teams use sprints and "the roadmap" as a way to say no to fixing bottlenecks they've created for other teams and don't want to take the time and effort to resolve.