Comment by OutOfHere
2 days ago
How would you use tool-calling to filter through millions of documents? You need some search functionality, whether old-school search or embedding search. If you have only thousands of documents, then sure, you don't need search, as you can feed them all to the LLM.
I haven’t built either system but it seems clear that tool calling will be ‘O(num_targets * O(search tool))’, while RAG will be ‘O(embed_query * num_targets)’.
RAG looks linear (constant per lookup) while tools look polynomial. And tools will possibly fill up the limited LLM context too.
You give the LLM search tools.
That's missing the point. You are hiding the search behind the tool, but it's still search. Whether you use a tool or a hardcoded workflow is irrelevant.