Comment by bonestamp2

6 months ago

Maybe they want sites to block them that don't want to be crawled since it probably saves them a lawsuit down the road.

0 comments