Comment by j16sdiz

1 day ago

if you are crawling the entire web, you should respect robots.txt and don't fetch anything disallowed. full stop.

0 comments