Comment by ahmadyan
4 days ago
There are a few benchmark, where they run different harness with the same model, e.g. [artficial analysis](https://artificialanalysis.ai/agents/coding-agents) but honestly they are easily game-able.
For example, consider something like SWEBench, where a test should pass. Harness or a skill can just tell the agent keep trying until the test pass. So same model can have a lot of behavior changes between the harnesses.
In my experience (i own a multi-agent IDE with 50+ supported harness) folks stick to first party, usually either openai or anthropic, grok, agy; if they are willing to pay for the subscription the value is unbeatable. And they don't risk trying different harnesses.
I made a harness too, and recently added eval with opencode and pi and I sat between them. No gaming other than trying to figure out why I was really bad on some evals (I was not returning thinking tokens which broke cache, for example, and different model families do things way different here). I have yet to try my harness for coding but I do have a few terabytes for it to read. And I’d rather use a local model that’s fast and good at retention, but that run could take a year, lol.