Comment by lolc
21 hours ago
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
21 hours ago
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
Exactly. You disobey robots file => we'll make your crawl gain a net negative.