Comment by tarpitt
14 hours ago
Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.
it's hard to separate spambot that only accesses 3-5 links per IP and a legit user. Changing content for legit user can be devastating
Put a cookie wall in front. The bot will either load the cookie and have a persistent identifier, or not load the cookie and not get in
why is this not the answer? then you can also rate limit each cookie as well.
3 replies →