Comment by arjie
8 hours ago
There's no way to just download all of it, is there? You have to scrape it slowly? I don't mind doing it that way, but flat downloads would be pretty neat. How do mirrors work? Do they run read-through caches?
8 hours ago
There's no way to just download all of it, is there? You have to scrape it slowly? I don't mind doing it that way, but flat downloads would be pretty neat. How do mirrors work? Do they run read-through caches?
I have to imagine that's not available to avoid training by commercial entities. https://www.transformativeworks.org/ai-and-data-scraping-on-...
Makes sense. Somewhat strange (but not too strange since it's not standardized) they don't have a Crawl-Delay except for Slurp. Why not on User-Agent: *
Confusing. Would even prefer a commented crawl delay that specifies in human language since then I could special-case it.
I feel like I am coming in to the middle of a discussion. Towards what end?
Just data hoarding tendencies, faster and better search, etc.
there's some dumps on archive.org
i worked on a project a long time ago and downloaded a bunch of it
Thank you. Do you still have your archive? I'd email you if you had address on profile.
I think I uploaded the only archive I had