Comment by thayne

I'm curious how the author identifies the crawlers that use random User Agents and and distinct ip addresses per request. Is there some other indicator that can be used to identify them?

On a different note, if the goal is to waste resources for the bot, on potential improvement could be to uses very large images with repeating structure that compress extremely well as jpegs for the templates, so that it takes more ram and cpu to decode them with relatively little cpu and ram required to generate them and bandwidth to transfer them.