Comment by jeroenhd

One of my issues had to do with pure bandwidth, dozens of bots were requesting the same image over and over again adding random crap to the query. My server was being slow for me during crawling peaks.

> An actual proper static site would likely 10x that.

A lot of the stuff I host isn't static. Any non-static content also gets overloaded pretty quickly. Hosting something like a Git server can turn pretty resource intensive if shitty bots start diffing every commit with every other commit.

I'm generally lazy when it comes to blocking IPs, but when I do run into an issue, I just blanket block IP ranges. When that doesn't work, ASNs get blocked. So far I haven't had to take most measures mentioned here but I probably will once the broken bots start coming from botnets using residential IP addresses.