Comment by sandeepkd
8 hours ago
Its a yes if you do not know the domain space, query patterns well enough and also if the cost of optimization or time for optimization may have detrimental impact to business. In this case it most likely means that the crowd in the room did not anticipate much on this in early phases and no one in the room pointed these things out. The irony is that these performance and disk numbers are heavily discussed as a part of system design interviews.
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia
This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
“changing data once its running in production is hard, changing the data structure is even harder”
100% agreement on this. There are a class of optimizations that can happen transparently. Those can happen at any time, and are fine to defer. Not all profiling and scalability improvements fall into this bucket. Some are very expensive to roll out, and ignoring these concerns can cause huge headaches down the line. Not fun to hear, but it’s definitely true. Even with LLMs, this can still be a huge challenge.
The fact that it took them 4 months to roll out does not mean this is the hard part: it's simply a coordinated rollout with incremental, staggered deployments and rate-limited migrations.
Changing data structures with that approach just takes its time as you avoid lock-step updates between components. Sure, by definition this type of development and deployment complexity is hard.
However, what I found the hardest is pushing engineers to adopt this evolutionary data structures mindset, and unless you do that right for the full team, someone will sneak in a backwards-incompatible change that blows the entire effort up.
So it is hard, but primarily for different-mindset-needed, and only then for technical complexity.
> This is a misconception
Pretty sure they were joking.