Comment by GuB-42
2 hours ago
Anubis is not free either, it is a matter of how much it costs to run Anubis vs to let bots in.
When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.
It’s done out of necessity, you can read about why here: https://people.kernel.org/monsieuricon/creepy-crawlies
As a user/reader/viewer I absolutely hate Anubis and usually turn around when I see it pop up (at least on my phone where it takes ages to compute), but with stats like that, I get why a site operator would resort to using it.