Comment by showerst
4 days ago
How would that scale to 10TB+ of plain text though? Presumably the indexes would be many gigabytes, especially with full text search.
4 days ago
How would that scale to 10TB+ of plain text though? Presumably the indexes would be many gigabytes, especially with full text search.
The client only needs to get indexes for the specific search; if the index is just a list of TF-IDF term scores per document (which gets you a very reasonable start on search relevance) some extremely back-of-the-envelope math leads me to guess at an upper bound in the low tens of megabytes per (non-stopword) term, which seems doable for a client to download on demand.