Comment by vunderba
4 months ago
There's also popular repository that maintains a comprehensive list of LLM and AI related bots to aid in blocking these abusive strip miners.
4 months ago
There's also popular repository that maintains a comprehensive list of LLM and AI related bots to aid in blocking these abusive strip miners.
I didn't know about this. Thank you!
After some digging, I also found a great way to surprise bots that don't respect robots.txt[1] :)
[1]: https://melkat.blog/p/unsafe-pricing