Comment by xyzal 18 hours ago One would assume legit spiders obey robots.txt. 3 comments xyzal Reply lolc 17 hours ago This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt. xyzal 2 hours ago Exactly. You disobey robots file => we'll make your crawl gain a net negative. heyethan 8 hours ago The real problem is most of them just pretend to be normal users now robots.txt only works if they choose to care.
lolc 17 hours ago This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt. xyzal 2 hours ago Exactly. You disobey robots file => we'll make your crawl gain a net negative.
heyethan 8 hours ago The real problem is most of them just pretend to be normal users now robots.txt only works if they choose to care.
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
Exactly. You disobey robots file => we'll make your crawl gain a net negative.
The real problem is most of them just pretend to be normal users now robots.txt only works if they choose to care.