Comment by darkwater
2 years ago
Oh, TBH I was expecting some kind of opensource, local-first scraper + search engine where you put a list of websites/info source "bubble" and it will scrape them over time and present them to you locally.
2 years ago
Oh, TBH I was expecting some kind of opensource, local-first scraper + search engine where you put a list of websites/info source "bubble" and it will scrape them over time and present them to you locally.
Yes, except we want a shared resource for an index so we aren't all crawling sites in excess.
Eg, here's the latest central or distributed DB, the site in your personal list is on it so we don't need to send the crawler there because it has already been indexed and is crawled by someone else.