Comment by jacquesm

> Nobody malicious brings down crawlers.

You're wrong about that. I've more than once brought down crawlers on purpose, especially the ones that didn't respect robots.txt.