Comment by matt_wulfeck
8 years ago
This is why web crawlers are built with upper boundaries on everything!
Nobody malicious brings down crawlers. It's just unexpected things you find out on the internet.
8 years ago
This is why web crawlers are built with upper boundaries on everything!
Nobody malicious brings down crawlers. It's just unexpected things you find out on the internet.
> Nobody malicious brings down crawlers.
You're wrong about that. I've more than once brought down crawlers on purpose, especially the ones that didn't respect robots.txt.