Comment by dawnerd

Exactly, why should sites give free bandwidth to the google bots hammering them for nothing in return? Outside of retail, there's no point in allowing google to crawl if you're not getting anything in return.