Comment by crazytweek
6 hours ago
Our approach queries multiple models at once and uses their progress to make decisions (this technique is novel AFAIK, let us know if you know anyone else doing this)
Sounds like DigitalOceans Model synthesis (https://www.digitalocean.com/blog/model-synthesis) or AilinOne (https://ailin.guide/) ?
These approaches, along with OpenRouter Fusion (https://openrouter.ai/blog/announcements/fusion-beats-fronti...) are different from ours in that:
1. we trained custom models to predict LLM performance (this performs much better than asking LLMs directly to predict performance)
2. we cut off model responses early if they are not performing well, saving on costs versus running them to completion
3. finally we serve the unmodified output of the chosen model, instead of synthesizing the outputs into one, which makes us a model router
Can you tell if this site is doing two and three?
https://news.ycombinator.com/item?id=49028740:
Sounds like this would be expensive on the input token side with some saving on the decode side. For workloads with large documents / context that could be an issue.
AFAIK they solve this through their RAG based knowledgebases to only search and use the most relevant Information.