Comment by bluesnowmonkey
12 hours ago
I think we’re still figuring out the right abstraction for offering agents as a product.
- LLMs are a great foundation but building your own harness is a huge undertaking, a deep rabbit hole.
- There are harnesses available as open source libraries but that’s still coupled to an environment. Where does the state persist? Like maybe I’m a Cloudflare worker and don’t even have a file system.
Agent as a service like this lets you plug in the tools it needs to be whatever kind of agent you want. But they still get to encapsulate and continue to iterate on the really deep parts of the harness that all agents need like memory and context management.
That said, my money right now is not on the offerings from OpenAI and Anthropic because they’re stuck using their own proprietary frontier models and those aren’t actually the best choice for most agents right now. A competitor who is not an LLM lab gets their pick of the market at any given moment. Like you’d want to be using GLM 5.3 Flash right now for most things agentic.
> building your own harness is a huge undertaking, a deep rabbit hole.
I eventually gave up on this task. It's not possible to fight OpenAI or Anthropic's engineering teams. Their reasoning models have all kinds of undocumented back door access to the base models that you'd never be able to replicate from the outside. Even if you had full access you would not have the engineering man hours or experience to keep up.
I think this Agents API thing is a step too far, but Chat Completion is too cold now. Something approximating Responses API seems like the happy medium. You still get most of the control with the only blackbox part being the reasoning loop / tokens. Building agents using the GPT5.6 family w/ Responses API feels pretty close to Star Trek computer shit to me. I thought I was being clever with my DIY contraption on top of chat completion, but it wasn't even close. I have embraced the reality that I will need to use opaque reasoning tokens to give my clients the experiences they are paying me to provide.
I built my harness in pi within herdr, I cloned (zipped and downloaded) 0xRichardH/pi-herdr-subagents and went from there, and used pi to build itself, adding gate workflow state control, provider fallbacks (I use many token plans), subagent skill injection, etc.
It is highly custom to my needs and wants, and I think every developer needs to do this. I only talk to my planner which plans, and it subs out to designer, oracle, coders, testers, and reviewers, etc. It is thus highly optimized for correctness. You can TDD or no TDD. You can fast track small changes. I tweak my harness dozens of times as I encounter new edge cases (esp when I switch models and encounter models not as good at following instructions).
As you can start to see, it is better to own the harness because nobody can build something custom that 100% fits your needs or development philosophy.
I cannot agree more. We all have our own needs, and it’s never been easier to make something that meets them exactly.
I have been running my own setup and evolving it since November as I come up with more ideas and needs. The nicest bit is not being tied to one provider or another. Anthropic decides to stop supporting Claude agent sdk with subs? Or astra outstrips other models? My harness now works with Claude/cursor/codex and soon ACP. It didn’t start that way, it was originally all in on Claude, but now all the things I built on top of Claude now work with the others too.
I did a similar thing, took those sub agents primitives and ran with it: https://github.com/durdn/herdr-interactive-subagents I do still use more than one harness though.
Correct me if I'm wrong, but the harness will always be dependent on the underlying model, and useless without it. All custom harnesses are being built, could be obsolete in the next big-generation-jump of the models.
I might be absolutely wrong, but "harnesses" / cc-derivatives became "good enough" only maybe a year ago max. Before that, people were pushing for gigantic folder structures with custom documents and "pretend you're X" stuff.
My point is, depended on what you're trying to achieve, testing out current-gen harnesses, and nudging your workflows towards them might be better RoI, rather than chasing something that might be throwaway code a quarter later.
Obviously, this really depends on whether you believe model development will speed up or slow down in the upcoming future.
6 replies →
I also built a custom harness tailored to my needs using pi.dev underneath. From time to time I use the 'official' harnesses of openai/anthropic but can't confirm that they are much better adapted to that harness.
100% every professional developer at some point should build its own harness as daily driver
I'm very interested in this but I am confused on what Pi provides you if you are building the harness? What does Pi get you that writing from scratch doesn't?
Any good starting points or tutorials you recommend?
1 reply →
After being burned by the rug-pull of OpenAI retiring the Assistants API in favor of Responses last year, I swore off using heavily stateful APIs for language model access. I could be totally wrong, but at this point I'm more willing to use a proprietary harness headless than to abstract it into an API.
[dead]
I feel like you could use an open-source harness like Pi and get 100+% of what these closed APIs offer without getting locked to OpenAI. What do you think is missing from them?
Lack of risk to the business. If it was just me and my tinker toys, the open path makes a lot of sense.
I am in the business of provisioning custom AI robots for paying customers. I need to be able to provide certain assurances and offer operational simplicity or the whole thing falls apart. These systems need to survive in their environments. Accounting and compliance teams feel a lot more comfortable with the landscape around the frontier model companies.
If I went to my favorite client on Monday and tried to sell them what HN seems to think is the most ideologically pure AI solution, I would probably be fired as a vendor.
I've been working on a custom managed agent (see my other top-level comment), I find it is actually a manageable undertaking. It does feel herculean, but somehow doable. I do not find their hidden reasoning tokens to be insurmountable as long as you match the behavior of codex or CC (which takes work, but, again, is doable). My managed agent harness currently matches Codex on several benchmarks like Terminal Bench.
> > building your own harness is a huge undertaking, a deep rabbit hole. > I eventually gave up on this task.
It's not trivial, but cmon, i did during weekends from my phone and FOR ME it's so much better than the codex or claude, it has every i need and want :D
I'm using my own harness for work and hobby, has github integration, review mode, interactive voice mode, overlayed worktree, browser integration, mcp and much more.
Using claude and codex feels like picking up a club, in-line with the caveman skill...
Yet i find claude code worse than opencode.
I don't think anyone who's tried something else can like claude code. The only reason to use it is the subsidized plan and nothing else.
> It's not possible to fight OpenAI or Anthropic's engineering teams
And yet, I find Pi so much better to work with than Claude Code.
I don’t find it better than Claude Code but much better than OpenCode. So I am curious why less is more in that context.
I think the abstraction is only part of the problem. The other part is that all these companies offering ai products are deeply untrustworthy, and I don’t want to let them any further into my stack than I have to. Claude code and codex are great because they are lightweight, and operate on top of the rest of my tools with little to no change needed, so they can be eliminated or migrated away from with zero cost. They’re not a dependency of anything. And that’s as much as I’m willing to trust OpenAI or Claude.
Just letting you know, this comment inspired me to finally just say "screw it" and launch what I've been building for the past year. https://www.cadenya.com
Interesting web design. It's rare to see a mobile website with actual personality.
(In fact, "need to support mobile" is the main reason given for the loss of personality from the web as a whole!)
Interesting concept, but blue ocean as to your target customer. Would be good to see other case studies, use cases for this. Also, thank you for not another “bring your keys” product. Just abstract it away for me.
Congrats on the launch! The readme post in the blog was helpful to understand what you’re selling. Maybe you could convey those ideas more in the homepage?
Truly love the website.
1 reply →
> you’d want to be using GLM 5.3 Flash right now for most things agentic
That was yesterday. I think the crown currently belongs to DeepSeek Flash v4.1 for the next few days or weeks.
I can't wait for next week
There might not be a good abstraction. I've built a few harnesses for different types of workflows, and the details are so different I struggle to see a good abstraction. It's also not clear there should be - if you look at most complex software systems, it's a collection of smaller abstractions/tools/systems pulled together to achieve X.
> LLMs are a great foundation but building your own harness is a huge undertaking, a deep rabbit hole
I’ve been doing this for the past few months. I started with a server where I ran pi in tmux and then used that to build an LLM gateway and agent session manager, then built deterministic workflows using bash scripts and a skill/script distribution system. The app works on desktop, mobile and web and it works great. Non technical colleagues are using it to build and ship real software and it’s cheap AF even using API pricing because it works well with Luna or deepseek.
I also use Luna and DeepSeek in a custom harness. (Mine is very minimal.) GLM also works great.
I had issues with some other models but it seems to do with the system prompt and tool calling format. Some models seem to only work well with some harnesses.
Ha yeah GLM and Kimi seem to always biff the tool calls. I recently added in a raw stream log switch to diagnose but since DS and Luna are so good and cheap I haven’t been that motivated to debug it …
This is the "i made a voice controlled agent" thread all over again. lol, I too have made a stable of harnesses and tools to run them and have different levels of them monitoring each other and different spends to code/review/triage.
IMO building a harness is not wildly difficult (customize pi?) but the offerings from openai and anthropic are wildly subsidized in the subscriptions so they win by default if you want frontier capabilities. Glm 5.3 flash is great but it's not cheaper than a codex or Claude code 200 dollar sub and it does not have astra or fable level capabilities.
Try 50 lines!
https://minimal-agent.com/
I made my own harness based on this, which I jerry rigged to a Codex sub.
I just have a slack bot running on a VM that sees a message and invokes pi.
It would be trivial for every request to clone a full lxd container and have all the tools and repos required if I wanted to allow it to do even more.
Not sure why anyone prefers to choose locked in options
> Not sure why anyone prefers to choose locked in options
Convenience. And OPEX vs CAPEX something something.
Agree, as long as models are interchangeable, it doesn't make sense to be locked into a single lab's managed agent platform. You probably want to swap between models and own the agent state.
https://github.com/omnara-ai/omnara - this is a self hostable agent API that I'm working on. It stores the state of all agents in a postgres db you can easily query, rather than a local json file or sqlite file per agent.
Been using bedrock agent core and seems to work fine for me. Although there might be a better abstraction.
The best answer I’ve come to thus far is the model we (estuary.dev) are building out now: offering mcp.estuary.dev with tools for creating a sandbox with our CLI pre-installed, a tool for requesting that a tightly scoped access token be injected into a named sandbox file (this is the approval gate), and a tool for executing arbitrary commands in the sandbox (presumably our flowctl CLI, but let the model rip).
The intent is that anybody can drive it from Claude/ChatGPT/Pi on their phone after MCP sign-in (oauth), the model has full computer use capability, but we can also leverage it to build guided agent workflows in our own dashboard.
Is GLM 5.3 Flash that good? I'm using it through atlascloud for my current project and testing performance against opus and gemini models. I think I'm mostly concerned about speed because they are mostly doing tool calls.
Also yes to an open runtime.
Libraries such as agent development kit (https://adk.dev/) provides abstraction over multiple LLM vendors, long-term memory (persistance + compaction) and allow us to manage subagents & their lifecycles. Vendor neutral memory & context management is a challenge as default long-term memory uses vertext AI (gemini) in ADK.
> Where does the state persist?
Spider men meme of developers pointing at each other thinking "Not it".
I built several harnesses in different products over the last two years. Fully agree with you that doing it right is a rabbit hole. Certain system properties that you almost always want in a harness used within a SaaS (for example) are non-obvious at the start and require certain architectural choices. It's easy to start down a path and then find a gap a couple days before launch.
Async tool calls, having the agent wait indefinitely for a human response, and showing a form or questions to the user via a tool call are a few common capabilities that come up that a product manager might miss at first.
This is why I've been building Nvoken. LLM agnostic, ergonomic SDKs, flexible tool call patterns, tenant and user-aware budget enforcement, etc.
I'd really appreciate any and all feedback on this! It gives you some free tokens on signup and it's super quick to try.
https://nvoken.com
> Async tool calls, having the agent wait indefinitely for a human response, and showing a form or questions to the user via a tool call are a few common capabilities that come up that a product manager might miss at first.
All of this is specified in the ACP spec, so if you build your agents from that - you don't end up skipping features.
Also vital is proper prompt caching, tool design and some connection retry mechanism.
> All of this is specified in the ACP spec
Oh good reminder. I need to do that.
> Also vital is proper prompt caching, tool design and some connection retry mechanism.
prompt caching is weirder than i originally thought, and so variable across providers. Retry is easy, but can you explain more on tool design?
Agents are the wrong paradigm entirely and have limited places where they actually belong.
Going off and searching the web isn't really it.
You need to create 'new worlds' where they can operate best - and even then constrain what it does.
I think things like onecli are the direction we will take. The secrets and state will be proxied api calls.
[dead]
[dead]