Comment by jandrese 4 months ago If a bot ignores robots.txt that's a paddlin'. Right to the blacklist. 5 comments jandrese Reply nabla9 4 months ago The linked article explains what happens when you block their IP. gs17 4 months ago For reference:> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).It's really absurd that they seem to think this is acceptable. viraptor 4 months ago Block the whole ASN in that case. therealdrag0 4 months ago What about adding fake sleeps?
nabla9 4 months ago The linked article explains what happens when you block their IP. gs17 4 months ago For reference:> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).It's really absurd that they seem to think this is acceptable. viraptor 4 months ago Block the whole ASN in that case. therealdrag0 4 months ago What about adding fake sleeps?
gs17 4 months ago For reference:> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).It's really absurd that they seem to think this is acceptable. viraptor 4 months ago Block the whole ASN in that case. therealdrag0 4 months ago What about adding fake sleeps?
The linked article explains what happens when you block their IP.
For reference:
> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).
It's really absurd that they seem to think this is acceptable.
Block the whole ASN in that case.
What about adding fake sleeps?