Comment by duttish
1 month ago
Today it appears you have two options 1) Anubis or similar 2) Accept that 90-95% of your traffic is LLM scrapers fetching every file for every commit for every repo etc etc. There's some comments above with specific lists.
I built bot detection into the web framework I wrote. It's blocking well over >90% of the abuse.
The secret 3rd option is write some code.