← Back to context

Comment by jjice

1 day ago

> They instead make completely unfounded assertions that are wrong more than 80% of the time and when they are wrong they tend to be wrong by several orders of magnitude.

I completely agree. It blows my mind how fifteen minutes of testing something gets replaced with a guess. The most common situation I see this in (over and over again) is with DB indexes.

The query is slow? Add a bunch of random indexes. Let's not look at the EXPLAIN and make sure the index improves the situation.

I just recently worked with a really strong engineer that kept saying we were going to need to shard our DB soon, but we're way to small of a company for that to be justified. Our DB shouldn't be working that hard (it was all CPU load), there had to be a bad query in there. He even started drafting plans for sharding because he was adamant that it was needed. Then we checked RDS Performance Insights and saw it was one rogue query (as one should expect). It was about about a 45 minute fix and after downsizing one notch on RDS, we're sitting at about 4% most of the time on the DB.

But this is a common thing. Some engineers will _think_ there's going to be an issue, or when there is one, completely guess what it is without getting any data.

Another anecdote from a past company was them upsizing their RDS instance way more than they should need for their load because they dealt with really high connection counts. There was no way this number of connections should be going on based on request frequency. After a very small amount of digging, I found that they would open a new DB connection per object they created (this was PHP). Sometimes they'd create 20 objects in a loop. All the code was synchronous. You ended up with some very simple HTTP requests that would cause 30 DB connections to be established and then dropped.