Comment by ClassAndBurn
2 hours ago
Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required.
Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.
Codex and Claude Code already have the workflow feature which basically does this on demand, so it's getting there.
Indeed. And you can make this case about any tooling at all that is model adjacent.
Including by extension all programs, operating systems, or eventually hardware, I suppose.
> Sol Ultra style is the path forward
That’s called brute force so not really.
I seriously doubt this, especially in a world in which there's not just one model.
This makes sense if the models some how become unified.
I guess this is kind of an information theory thing. As a model converges AGI would a generalized harness ability appear?
Except in this case, it isn't yet smart enough. But I agree, building this capability in is coming, and will be really awesome.
It's likely smart enough. It just needs to be told to do it and provided the ability to introspect it. How close could foundational models get to building this harness if explicitly prompted to?
We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.
Let's maybe say not experienced enough / insufficiently RL'ed then -- 5.6 Sol did not reach for a harness solution like this when it got only 13% or son AGI-3 recently. I agree it's interesting to find the point in the prompting when it could 'tip' and do this. I have no instinct for where that point is, except that it must be somewhere, because I bet the Schema Harness was not hardcoded.
Letting the provider decide for the harness is a terrible idea in my eyes. Outsourcing harnessing is giving up control over the AI and equivalent to abandoning your sovereignty. It is a regression to a pre-enlightenment era.