Comment by saghm 1 month ago Couldn't they just rate limit them? Are they literally using a new IP for every request? 3 comments saghm Reply anematode 1 month ago There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution. sroussey 1 month ago We used to call this snow shoeing. kstrauser 1 month ago In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.
anematode 1 month ago There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution. sroussey 1 month ago We used to call this snow shoeing.
kstrauser 1 month ago In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.
There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.
We used to call this snow shoeing.
In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.