Comment by troyvit

5 months ago

It looks like the tool lets anybody that robots.txt allows through. IOW it doesn't stop all public scraping, just the scraping you want to stop.

0 comments