Comment by WD-42
8 days ago
This is sad, but predictable. At the end of the day if I can follow a link to an Anubis protected site and view it on my phone, the crawlers will be able to as well.
I see a lot more private networks in our future, unfortunately.
The private network is only as good as the weakest link which has to offer a reason for people to go through the trouble of accessing either by paying money or other means (acquiring special equipment).
And by putting a wall up you end up losing a large portion of the market to those that will now simply arbitrage and fill the space you leave behind.
There is simply no way to stop crawlers/scrapers, period, unless you put a meter on it or go offline.
The people that are already using Anubis don’t care about “losing a large portion of the market” we just want to work on our FOSS projects without paying unnecessary bills due to out of control crawling.