Comment by awsanswers
17 hours ago
Unacceptable, sorry this is happening. Do you know about fail2ban? You can have it automatically filter IPs that violate certain rules. One rule could be matching on the bot trying certain URLs. You might be able to get some kind of honeypot going with that idea. Good luck
They said that it is coming from different ip addresses every time, so fail2ban wouldn't help.
Amazon does publish every IP address range used by AWS, so there is the nuclear option of blocking them all pre-emptively.
https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-rang...
I'd do that, but my DNS is via route 53. Blocking AWS would block my ability to manage DNS automatically as well as certificate issuance via DNS-01.
2 replies →
It’ll most likely eventually help, as long as they don’t have an infinite address pool.
Do these bots use some client software (browser plugin, desktop app) that’s consuming unsuspecting users bandwidth for distributed crawling?
Monitor access logs for links that only crawlers can find.
Edit: oh, I got your point now.