← Back to context Comment by xyzal 5 months ago One would assume legit spiders obey robots.txt. 3 comments xyzal Reply lolc 5 months ago This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt. xyzal 5 months ago Exactly. You disobey robots file => we'll make your crawl gain a net negative. heyethan 5 months ago [flagged]
lolc 5 months ago This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt. xyzal 5 months ago Exactly. You disobey robots file => we'll make your crawl gain a net negative.
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
Exactly. You disobey robots file => we'll make your crawl gain a net negative.
[flagged]