Comment by navanchauhan

I think there’s a difference between crawling websites at a reasonable pace instead of just hammering the server to the point it’s unusable.

Nobody has problems with the Google Search indexer trying to crawl websites in a responsible way