Comment by trollbridge
9 days ago
And whilst the IA will honour requests not to archive/index, more aggressive scrapers won't, and will disguise their traffic as normal human browser traffic.
So we're basically decided we only want bad actors to be able to scrape, archive, and index.
> If you find yourself wondering, or just feeling, "Why is everyone I wind up dealing with an asshole?" you might want to consider the possibility that you have set up an asshole filter.
https://siderea.dreamwidth.org/1209794.html
It’s hard to filter out legit looking traffic tho.
> we're basically decided we only want bad actors to be able to scrape, archive, and index
AI training will be hard to police. But a lot of these sites inject ads in exchange for paywall circumvention. Just scanning Reddit for the newest archive.is or whatever should cut off most of the traffic.