Comment by raxxorraxor

Everytime we need to deploy such mechanisms, you reward those that already crawled the data and you penalize newcomers and other honest crawlers.

For some sites Anubis might be fitting, but it should be mindfully deployed.