Comment by shermantanktop

> robots.txt is a service to crawlers to help them avoid wasting time

I doubt anyone edits robots.txt with a warm fuzzy feeling about how much they are helping crawlers be efficient.