Comment by eduction

2 months ago

Can BitTorrent’s architecture contribute anything useful here?

I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.

7 comments

eduction

Reply
fragmede  2 months ago

Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.

  • ccgreg  2 months ago

    Common Crawl's dataset was downloaded in full 100 times in 2025.

    We agree that it would be great if it was even more widely used.

  • dylan604  2 months ago

    as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?