Comment by magicmicah85
9 hours ago
I have to imagine that's not available to avoid training by commercial entities. https://www.transformativeworks.org/ai-and-data-scraping-on-...
9 hours ago
I have to imagine that's not available to avoid training by commercial entities. https://www.transformativeworks.org/ai-and-data-scraping-on-...
Makes sense. Somewhat strange (but not too strange since it's not standardized) they don't have a Crawl-Delay except for Slurp. Why not on User-Agent: *
Confusing. Would even prefer a commented crawl delay that specifies in human language since then I could special-case it.