Comment by lolc
19 hours ago
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
19 hours ago
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
Exactly. You disobey robots file => we'll make your crawl gain a net negative.