Comment by eloisius
4 hours ago
How do you reliably detect the bots? Instead of slowban, I would feed it something abhorrent like an archive of 4chan posts.
4 hours ago
How do you reliably detect the bots? Instead of slowban, I would feed it something abhorrent like an archive of 4chan posts.
It's mostly about luring them into URLs they're explicitly told in robots.txt that they shouldn't index. I do some identification via reverse DNS of known crawlers I actually want like Googlebot, though they respect robots.txt, in case something goes wrong and they accidentally get flagged.