Comment by j16sdiz

25 days ago

if you are crawling the entire web, you should respect robots.txt and don't fetch anything disallowed. full stop.

0 comments