Comment by kstrauser

And yet, in practice, it's true. I was getting about 600K requests to my Forgejo server a day from crawlers making idiotic queries: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h...

I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing.

I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.