Comment by popPopBoom
6 hours ago
Interesting approach. The multi-model progress monitoring idea is clever, most routing I've seen is either static rules or a cheap classifier that picks once upfront. Querying in parallel and deciding mid-turn feels different.
One thing I'm curious about: how do you handle the latency hit from spinning up multiple models on the harder turns? Does the user-facing latency still feel competitive with just going straight to Claude, or is there a noticeable pause while the router decides?
Also, any plans to expose the routing decisions (or at least the model chosen per turn) so people can debug when it picks poorly? That seems useful for the feedback loop you're asking for.
Since we fan out in parallel, by definition the latency is only at worst the latency of the slowest model. The model in between is fairly light, so it seems to not materially affect things. Deciding mid-turn also lets us resolve earlier, often saving on the latency question. We hope to publish a detailed study about this soon.
We are exposing the routing decision information already, scroll over the response in the Dashboard and it'll tell you what models it considered and what it chose.