Comment by dylan604
2 months ago
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
2 months ago
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
Common Crawl's archive has metadata that says when each record (html file) was crawled.
But who stores the metadata for the last date the site updated so you know if it needs to be refetched or not.
We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.
1 reply →