Comment by bonestamp2

4 months ago

Maybe they want sites to block them that don't want to be crawled since it probably saves them a lawsuit down the road.

0 comments