Comment by echoangle

I think some people use hosting that is paid per request/load, so having crawlers make unwanted requests costs them money.