Comment by mrinterweb
7 hours ago
There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
> Claude code does some of this by handing off the "explore" agent work to haiku.
That is not handing off to a specialized model, its just handing off to a lighter and interior model (compared to the parent model). That by itself can create issues like the lighter model not capturing all the data that the parent needs.
The idea is that we get specialized models that are better then general purpose models. But its rare for a specialized model to beat a strong general model.
There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good.
And if the tasks is repetitive to the point that specialization is useful, you can get into a situation that your better off having a program written for that reputative nature, then delegating to other models. And then have the main strong model, deal with the (semi)cleaned up data.
> There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good.
Smaller models are cheaper, sometimes faster. I agree that the “we’re an LLM fine-tuned for X” hasn’t worked out because you can just train Claude to do X (and Anthropic will), but not burning Opus/Fable tokens on dumb-but-token-heavy tasks is good sense.
As we move from “integrate AI into Y” to “optimize the ROI on Y”, we’ll see more of this.
castform founder here. the roi optimization makes sense. i think there are lots of usecases for which even a 2% gain in accuracy can be quite useful. off the top of my head
- high volume customer support. higher accuracy means fewer escalation, reducing labor costs - fraud detection. catching even one extra fraud attempt could mean a lot in savings - and ofc the classic ads use-case where at scale bps in improvement could mean millions in revenue :)
castform founder here. while it's "rare for a specialized model to beat a strong general model" today, i think the tech/knowhow on how to do so is getting there. we see some early signs of this in industry e.g.
https://shopify.engineering/sidekicks-continual-learning-loo... https://thinkingmachines.ai/news/learning-to-replicate-exper... https://castform.com/blog/elsa/
at the end of the day, models are only as good as the data they're trained on. and if one has access to proprietary data, it should yield specialized models that do better than larger general models
You can register models with mcp. I think it’s an expensive solution, but it is available in the framework. I use a light weight bus protocol that lets agents interact and pass short messages with pointers. It’s very efficient.
There are! Chroma has Context1, SID has SID-1, and you'd actually be surprised at how easy it is to post-train your own with pretty good pass@ recall@ ndcg@ etc.
There's also Hornet who have shared some interesting talks & blogs lately. I don't know that I'd exclusively use agents for retrieval the way Neon outlines here as well. I think distillation similar to what ZeroEntropy has done for bespoke retrieval & reranking with _some_ agent manipulation on top-k results works better (IME).
> Claude code does some of this by handing off the "explore" agent work to haiku
This is no longer necessarily true. As of 2.1.198 [0] (released July 1st): "The built-in Explore agent now inherits the main session’s model (capped at opus) instead of running on haiku"
[0] https://code.claude.com/docs/en/changelog#2-1-198
I feel like the future is people building applications with tightly integrated LLMs that work hand in hand with the application's own lifecycle and code.
I also didn't realize that people were using agentic harnesses for search, it's an interesting idea. If the context length is short enough it should be fairly cheap compared to running "normal" agentic coding workloads where you have O(100k) context length for doing almost anything.
castform founder here. that's a future we are really excited about too :) ideally, you can post-train the llm within the application itself, as it's being used. both interesting infrastructure & algorithmic challenges here
There has been an over-obsession with frontier models and benchmarks. Most of the work will be done by task specific models. You don't put Phds on the factory floor.
Models keep on improving though, so doesn't fine tuning become an ongoing task with ongoing maintenance burden?
(one of the blog post authors here) -> once you set up a finetuning pipeline, it's often trivial to rerun it on top of a new open weights model. so, it's orthogonal to base model improvements
this is exactly what leopold talks about in situational awareness
yes, and this is why we need model routing
castform founder here: totally! we also think model routing is also a post-training problem i.e. getting a model to predict the difficulty of a task and match it to the right model -> we're gonna be sharing more on that soon :)
> There is so much opportunity for purpose built models like this.
OpenAI etc could themselves do this, and maybe they already do? Where the public-facing interface delegates to multiple little goblins behinds the scenes
castform founder here. openai actually deprecated their finetuning apis a few months back weirdly.
Exactly. There could be a lot of value for inference companies to do this. Could save a lot of money being able to hand off highly repetitive known tasks to far smaller specialized models.