Comment by inigyou
12 hours ago
Put a cookie wall in front. The bot will either load the cookie and have a persistent identifier, or not load the cookie and not get in
12 hours ago
Put a cookie wall in front. The bot will either load the cookie and have a persistent identifier, or not load the cookie and not get in
why is this not the answer? then you can also rate limit each cookie as well.
it'll load the cookie, make one request, move to a different ip, load the cookie, make one request, move to a different ip, ...
Why are you serving new cookies to someone who already has a cookie?
If you are discovering urls you have to wait for a previous request to finish. The rate limit should work. Requests without a cookie wait 2 seconds. Request with cookies can only make human scale number of requests per second? (1?)
If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess.
The next more advanced version is that URLs are unique to your cookie.
Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user.
You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request. (a cookie url mismatch gets the rate limited version of the page)