Comment by lpapez

The fact that it took them 4 months to roll out does not mean this is the hard part: it's simply a coordinated rollout with incremental, staggered deployments and rate-limited migrations.

Changing data structures with that approach just takes its time as you avoid lock-step updates between components. Sure, by definition this type of development and deployment complexity is hard.

However, what I found the hardest is pushing engineers to adopt this evolutionary data structures mindset, and unless you do that right for the full team, someone will sneak in a backwards-incompatible change that blows the entire effort up.

So it is hard, but primarily for different-mindset-needed, and only then for technical complexity.

“changing data once its running in production is hard, changing the data structure is even harder”

100% agreement on this. There are a class of optimizations that can happen transparently. Those can happen at any time, and are fine to defer. Not all profiling and scalability improvements fall into this bucket. Some are very expensive to roll out, and ignoring these concerns can cause huge headaches down the line. Not fun to hear, but it’s definitely true. Even with LLMs, this can still be a huge challenge.

> This is a misconception

Pretty sure they were joking.

First version was a three layer VM monster, with the main entrance done as HTTP proxy written in PHP.

The way I usually prevent having to scale out to a bazillion systems is never getting more than 10 users.

You're never going to get promoted with that attitude!

I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.

Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.

I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.

Remember that everything has an opportunity cost. Running a lot of servers might cost $10 million annually, but if the product team had to choose between a project that would recoup $5 million of that vs. an opportunity to earn $50 million ARR for the same amount of work, the logical answer would be obvious.

It was already reasonably lean. If they had 10 bazillion systems, they now need somewhere between 6 and 8 bazillion systems.

You can build foundations that aren't extermely optimal but have future optimisations in mind.

Exactly, and you need to start turning a profit before the end of that runway. Even if that means running code that is suboptimal.

This reasoning is largely centered around the runway being finite. You obviously can't have costs so high you are making a huge loss, but also there's little value in improving margins past profitability until you actually have a stable segment of the market.

we are all perfectly smooth, round, and filled with an incompressible liquid

Every startup is one bet in a Martingale strategy played by the class of people who remain solvent when you bust.

So obviously you start at optimization