Comment by dweez
3 hours ago
I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
I think routing should be pushed 'down the stack' so to speak.
What I mean is that most tasks can be recursively fragmented into smaller tasks, and once you've hit suitable leaf nodes -- where the task is very granular -- you can begin to deterministically show which models perform better or worse for that specific task. Then, when your agent is running a workflow, you may use various models for different steps in a workflow. For example, some models may excel at exploration, some at determining a good architectural fit for an implementation, some at actually writing the implementation, and so on.
But you don't know until you define your 'work' taxonomy, and still further, you won't know until you have a statistically significant number of runs on a given chunk of work. Once you have that, though, you can hone in on models that excel at one specific task or another and prefer those the majority of the time (say ~80%) and hold back the remainder work as a 'test corpus' just in case a different or new model does even better.
This is something I've kept in the back of my head as I've been working through my agent harness primitives -- specifically enabling different models per chunk of work.
You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to.
What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't think LLMs can really bring their uncertainty to the surface in that way yet? Their internal confidence can be measured and returned, so you could probably front a more powerful model with a knowledgeable assistant, but they can't consciously mark their own homework?
> Their internal confidence can be measured and returned
It can? I was under the impression that confidence was either self-reported by the LLM or assessed by having another model interpret the output response. If there's a confidence score at the level of the actual model math, that's news to me.
You certainly can't ask an LLM for its confidence level because it will generate that answer as it sees fit.
My understanding was that models can be post-trained to assess their own confidence on short prompts within a level of accuracy (there was an article about this a few weeks ago that I can't find), but can't use it to reason, and that research has shown that internally they effectively have a measurable sense of truth but can't surface it:
https://arxiv.org/abs/2410.02707
I mangled what I was trying to say but the point I guess is, it is effectively in there, and researchers can see it and therefore score it, but it is not something the LLM can use.
The problem seems to me (layman's understanding at best) that the LLM is inherently confident within the words of its answer, because of what the model is trained to do and how it is trained. So you can never get an accurate "I don't know this" from a model while it is answering; it is bullshitting.
(One of the things that is most interesting to me at the moment is asking a small local LLM what it knows about a topic and then testing it. It can have no idea of what it wasn't trained on, but any question about what it knows is seemingly going to trigger it to work through all the summary word associations it did find in training, so it can sort of summarise its knowledge that way, with some likelihood of success, without any sense of introspection)
At least one model can today. https://github.com/cactus-compute/cactus-hybrid
1 reply →
I mean can't you just directly test some variations of the query against many models at once and pick the cheapest model that hits your accuracy goal? If it's a one time run then ofc this is all pointless, but for ongoing tasks it makes sense. This seems more logical to me than making another AI model of some sorts intuit the right LLM for the job.