Comment by hi_hi
4 days ago
Took me a little while to figure out this was a llm harness for coding (also called an agent, I suppose, but that term seems overloaded and ambiguous to me)
There are many of these available, alongside the provider ones.
Are there any benchmarks which help me decide which ones I should try over the standard codex/claude?
Codex and Claude should not be seen as ‘the standard’
They are vendor offerings but you don’t have to take them.
You kinda do have to take Claude if you want to take advantage of their subscription pricing, sadly.
this is no longer the case. If your harness handles login through the Agents SDK, you can use your sub now. That's why your Claude sub can be used in Amp and T3 Code for free now, but not in Pi (at least not without workarounds)
4 replies →
Anthropic and OpenAI models are available via a number of different channels eg AWS, GitHub copilot, Google Cloud… using those kinds of platforms behind a vendor-neutral tool gives you the ability to swap models out while maintaining the same workflow.
1 reply →
Seems foolish to use a single-vendor harness to be honest. With how fast things change, and the price/performance you get from open weight models, building a workflow around a single llm provider seems super limiting.
So is OpenCode. They are quite a vendor.
I use "harness" for the tool which calls the llm, has the tools, interacts with the computer (filesys, bash, etc), does the session bookeeping
and "agent" as the instance which is running... this is the more overloaded term because there are "agent" definitions, this is where a combinatorial exists, because the running "agent" is a combination of agent definitions, context, tool choice where you can modify each independently to create a new "agent"
I think Scion (Google's) does a decent job defining these concepts: https://googlecloudplatform.github.io/scion/concepts/
https://harnesstax.github.io/
From what I gathered, a simple harness like the one from Pi is sufficient.
Similarly, I would advise against using tons of skills and keep the ones you genuienly find useful (they are just .md files so you can copy them). The only skill I use is /strategic-compact from everything claude code (ecc) because I like to read the summary before a manual compaction.
There are a few benchmark, where they run different harness with the same model, e.g. [artficial analysis](https://artificialanalysis.ai/agents/coding-agents) but honestly they are easily game-able.
For example, consider something like SWEBench, where a test should pass. Harness or a skill can just tell the agent keep trying until the test pass. So same model can have a lot of behavior changes between the harnesses.
In my experience (i own a multi-agent IDE with 50+ supported harness) folks stick to first party, usually either openai or anthropic, grok, agy; if they are willing to pay for the subscription the value is unbeatable. And they don't risk trying different harnesses.
I made a harness too, and recently added eval with opencode and pi and I sat between them. No gaming other than trying to figure out why I was really bad on some evals (I was not returning thinking tokens which broke cache, for example, and different model families do things way different here). I have yet to try my harness for coding but I do have a few terabytes for it to read. And I’d rather use a local model that’s fast and good at retention, but that run could take a year, lol.
The main four seem to be Pi, Opencode, Claude Code, abd Codex.
I do often see benchmarks comparing them and usually Pi comes out on top, but I think the benchmarks are generally testing one moderately sized clear task. Pi is minimal and great in that situation.
For more general usage, where tasks are not clear, you might interrupt at any moment to steer it, etc. benchmarks are harder I guess. I personally love pi because anytime there's something I don't like, I spin up a separate session and fix it.
E.g. I've found the "reply in simple technical English" hack amazing but I have to keep saying it over and over. So I had an agent write an extension in about ten minutes that appends a short hidden message saying all user and dev facing text must be in STE, apply that without talking about STE. Works amazingly well.
I would state the same, but looking at commercial offerings for agents, such as MCP servers or skill instructions in the vender's own docs, opencode is often not mentioned at all.
Here is an example I encountered recently: The Figma MCP Server documents usage with Claude (Code), Codex, Cursor, Grok and Antigravity [], but notably not OpenCode. More so, their oauth registration seems to use some kind of whitelist, meaning just adding the MCP by url does not work either. To make it work, my agent edited the config entry to use "Codex" as the oauth name, which worked.
That's just one example, but something you see quite frequently.
> The Figma MCP Server documents usage with Claude (Code), Codex, Cursor, Grok and Antigravity [], but notably not OpenCode
In other words, big industry supporting bigger industry. I don't think that's surprising or says much about the real landscape.
Could opencode do the same? Thinking of trying pi because of these extensions
The harnesses all do the same thing in a general sense. They loop over chat with defined tagging/syntax that issues commands and run sub-agents that do the same.
Pi is the most basic with the ability to add features.
opencode also supports more features and was build as an open alternative to claude.
AGENTS.md
Here is a benchmark where they actually evaluated harnesses across same models and measured task completion rates: https://frontierharness.org/
You might look at Block's open-sourced Goose (https://goose-docs.ai) and view it as something of a neutral opinion for a harness application.
the person said that there were too many samey harnesses and they couldn't see why or which to choose and you thought you'd fix that by adding another to the pile?