Comment by bonestamp2

Maybe they want sites to block them that don't want to be crawled since it probably saves them a lawsuit down the road.