Comment by mlmonkey
18 hours ago
Couldn't the original website owner just look at the logs and see where most of the requests are coming from? And also the referrers would be a dead giveaway, right? That's assuming someone is looking at the `access.log` file occasionally.
They could use residential proxies
And residential proxies are becoming increasingly legitimate because of over-zealous AI blocks.
Blocking IP ranges and user agents has nothing to do with "AI". It's a bad practice by lazy sysadmins who'd rather discriminate users than fix their infra. Writing this from a non-profit ISP (fdn.fr) whose IP range is blocked by Google and Reddit and noone still knows why or how to unblock it [0].
Residential proxies on the other hand i don't think have any legitime usecase. I would support people working around Youtube to export interesting videos to Bittorrent. But what's happening is malicious actors going around rate limiting to run bad-behaving scrapers on smaller sites. That, and much worse is the residential proxy economy.
[0] https://www.fdn.fr/the-fdn-and-gitoyen-networks-are-blocked-...