Comment by zxcvbn4038

With some planning we could have accommodated the 120K rps rate and more, but just out of the blue it caused a lot of issues, the database shards for historic information tended to be configured for infrequent access to large amounts of historic data, their access completely thrashed our caches, etc. We did want Google to index us, if there had been an open dialog we could have created a separate path for their traffic that bypassed the cache and we could have brought additional database servers into production to handle the increased load, we even had a real time events feed that updated whenever content was created or updated that we would have given Google free access to that so they could just crawl the changes instead of having to scan the site for updates, but since they would not talk to anyone none of that happened.