Comment by ericrallen
12 hours ago
Maybe there is some optimization logic that only appends tool details that are required for the user’s query?
I’m sure they are trying to slash tokens where they can, and removing potentially irrelevant tool descriptors seems like low-hanging fruit to reduce token consumption.
I definitely see different prompts based on what I'm doing in the app. As we mentioned there are different prompts for if you're asking questions, doing Cmd-K edits, working in the shell, etc. I'd also imagine that they customize the prompt by model (unobserved here, but we can also customize per-model using TensorZero and A/B test).
Yes this is one of the techniques apps can use. You vectorize the tool description and then do a lookup based on the users query to select the most relevant tools, this is called pre-computed semantic profiles. You can even hash queries themselves and cache tools that were used and then do similarity lookups by query.