Comment by HenryNdubuaku
17 hours ago
Hey, thanks a lot for this feedback, very useful and actionable for us! Quite a few of these came down to our tool definitions in the playground as well as out triggers. We updated them just now and these should be more reliable. Really this goes to show that needle shines through after putting in the work to make the tool list around it good for your use case. As for the reasoning, yes its main function is really to provide more words/keywords that the model can latch onto when generating the tool call response, since this is a SAN model it needs more grounding in existing context.
I can’t help but wonder how well more traditional approaches would do with this. Something like a map of statements to actions, with fuzzy search - then remove what used to be the labour intensive part of this by handing it to a decent llm to generate the sentences.
That's a really good point and I think it's not yet clear how well, say, 8-30MB worth of regexs with accompanying algorithmic structure would do on these tasks. I would imagine they do quite well on a well defined task, but it would be much harder to then adapt this set to a new domain. A big part of Needle's promise is how easy it is to finetune. Ultimately I think the two approaches can be more complimentary to each other, rather than choosing only one (see triggers!).
I think a good "traditional" approach would look like a BM25 algorithm over an index of trigger phrases for each category, sitting behind a majority-vote classifier. The "fine tuning" would be done by reindexing the data, generating different/new phrases, and tuning the classification threshold.
1 reply →