Comment by vlian2088
1 day ago
in my own words: if something is publicly visible, it will get scrapped. you could rate limit your website to 1 request per minute with a custom Turing test upon each request, and it would still, still get fully scrapped. the Internet is getting scrapped by hundreds of players, and at least some of them will bother to investigate and bypass the countermeasures even for a tiny obscure blog.
Ah. I’m not talking about a perfect and total block like Reddit wants. I just want the vast bulk of scrapers (whether AI or not) to hit a 403 auth-required wall and give up, which is what they’ll mostly do. Good enough for me :)
If I did want a near-perfect block I’d post to a telnet BBS and have forums accessible only by QWKmail. Which would of course drive some jerk to synthesize a republish just to troll me. Isn’t humanity wonderful?