Comment by dspillett

> My experience with a crawler is that people don’t actually provide a robots.txt far more often.

I think many don't see the point of robots.txt as it will just be ignored by most bots anyway, so instead concentrate on more complicated measures.