Comment by ijk

Well, you can also batch your own queries. Not much use for a chatbot but for an agentic system or offline batch processing it becomes more reasonable.

Consider a system were running a dozen queries at once is only marginally more expensive than running one query. What would you build?