Comment by hinkley

Again, that delay is mostly about me, and my employer, not the rest of the world.

However if you are running a SaaS or hosting service with thousands of domain names routing to your servers, then this dynamic becomes a little more important, because now the spider can be hitting you for fifty different domain names at the same time.