Comment by manca
1 year ago
I love projects like this. It shows the true potential of what LLMs and RAG can unlock. Imagine applying the same method on the actual content within the threads and extract the sentiment, as well as summarize the key points of a particular thread -- the options are limitless.
My only piece of advice, though: try to do the reranking using some other rerankers instead of an LLM -- you'll save both on the latency AND the cost.
Other than that, good job.
Thanks! I tried a few other approaches and found the LLM results were overall better (latency and cost aside). Maybe that should be an option made available to users though...
Cohere has a very cheap, fast and effective reranking API!
https://cohere.com/rerank
i think not, better results >>> better latency + cost
Maybe a combined approach beats either? Let some non-LLM reranker quickly spit out two results, and fill in the rest with the LLM.