Comment by arjie

Makes sense. Somewhat strange (but not too strange since it's not standardized) they don't have a Crawl-Delay except for Slurp. Why not on User-Agent: *

Confusing. Would even prefer a commented crawl delay that specifies in human language since then I could special-case it.