Comment by binarymax

2 months ago

It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.

0 comments