Comment by kolinko

Well, 10k individual anonymous crawlers accessing a site once a minute are as bad - traffic wise - as one accessing the site 10k a minute. Even worse bc you can't block them.

Having said that I agree crawling should be legal in general. If an agent is acting in my name, it is not much different than a browser.