Comment by Nemi

I run an ecommerce auction website on wordpress that is highly dynamic and can't be cached. Bots are a waste of cpu. I even turned off "prefetch" requests as I found that (android in particular) browsers will prefetch every auction item on the page as the user scrolls down the summary page looking at items.