Comment by mapontosevenths
15 hours ago
I think that if I were to build this I might skip the LLM and just setup a nice interface to the source files using qmd style search. [0]
That way the tiny model isn't acting as a potentially incorrect interpreter but you still get semantic search to help you find the right documents quickly.
I suppose there is something to be said for a good summarization, I just worry that these tiny models aren't really GOOD summarizers.
Awesome, I didn't know qmd style search, thank you!
I'm all up for using the better/simpler task for the job. That's been a thing lately also with the Jev/decision models hype, most of those things can either already be done by a pretrained classifier[0], or you can start with a bigger model and then train your smaller/adhoc one when you have some data (it doesn't even take much for most applications, you can get about 95% accuracy on email classification with 50-100 examples)
0: http://jeffyclassify.com/blog/when-llms-fail-at-classificati...