I'm always confused by LLM proxies that claim to support tool calling. Even for Bifrost that claims to be doing it, at least when I was checking it out, I found out that while it injects the list of MCP tools that's available on the proxy-side, it doesn't actually make the call on client's behalf, and clients get confused by it (response returns MCP call request whose tool doesn't exist on the client-side).
I'm always confused by LLM proxies that claim to support tool calling. Even for Bifrost that claims to be doing it, at least when I was checking it out, I found out that while it injects the list of MCP tools that's available on the proxy-side, it doesn't actually make the call on client's behalf, and clients get confused by it (response returns MCP call request whose tool doesn't exist on the client-side).
LiteLLM is basically Bifrost in the Python ecosystem.
Yeah, but bifrost seems tighter and claims to use way fewer resources
hey - litellm will migrate its core to Rust soon. please follow this issue if you're interested! https://github.com/BerriAI/litellm/issues/31263
also if you had resource issues with your production deployment, keen to hear more details
3 replies →