← Back to context Comment by xyzal 19 hours ago One would assume legit spiders obey robots.txt. 3 comments xyzal Reply lolc 19 hours ago This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt. xyzal 4 hours ago Exactly. You disobey robots file => we'll make your crawl gain a net negative. heyethan 9 hours ago The real problem is most of them just pretend to be normal users now robots.txt only works if they choose to care.
lolc 19 hours ago This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt. xyzal 4 hours ago Exactly. You disobey robots file => we'll make your crawl gain a net negative.
heyethan 9 hours ago The real problem is most of them just pretend to be normal users now robots.txt only works if they choose to care.
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
Exactly. You disobey robots file => we'll make your crawl gain a net negative.
The real problem is most of them just pretend to be normal users now robots.txt only works if they choose to care.