Comment by v1sea
1 year ago
Can you go into more detail on how it works or provide some references for articles/papers on implementing a system like this? Is it just RAG?
Testing it out, I'd say the results for "graph visualization" are focused if a bit incomplete. So to me it has high precision, but lower recall.
I don't see this searching comments. That could be a nice extension. Thanks for sharing.
It is mostly RAG, although I suppose that doesn't say much about the system: one thing I've found is that the way you clean and process the data substantially changes the quality of the results. I'll write a little blog post sharing some of the learnings!
If you feel up for it, you should share your email in the righthand "Unhappy with your results?" widget. My plan is to manually look into the disappointing searches and follow-up with better results for folks, in addition to fixing whatever can be fixed.
Agreed re: searching comments (which it indeed currently doesn't do).
i am not surprised that 'the way you clean and process the data substantially changes the quality of the results.' can you share anything about your approach here?
I'll write up a little blog post once the traffic dies down a bit!
In the meantime, one thing that comes to mind is that simply embedding the whole contents of the webpages after scraping them didn't yield very good search results. As an example, an article about Python might only mention Python by name once. I found that trimming extraneous strings (e.g. menus, share links), and then extracting key themes + embedding those directly yielded much, much better results.
3 replies →