Comment by seizethecheese
5 hours ago
We're working on a battle system for local coding agents [1] and Argon is really bad. Literally the worst model in our initial ranking, and Deepseek Flash, Muse Wonder, Grok are all in there.
Top is Opus 5.5, followed by Fable.
I asked Gemini to review the repo. Apparent issues include:
The full report is here, if you're interested: https://docs.google.com/document/d/1-HqujXuDRJSWJIqEvmctcINr...
Prompt was:
> I read on hacker news the following, which does not match my expectations. Check if something in the harness or bench is hurting gemini perf.
followed by your post.
Wow, cool, thanks!
> anthropic only prompt caching
Anthropic has custom prompt cache requirements, but we've verified all providers are working well.
> Blinded model.api.id = "contestant" bypasses gemini.txt and serves Claude's prompt (default.txt)
This is wrong. We're serving default.txt which is opencode's default prompt. We are not serving a custom prompt for any model, including Gemini.
> Corrupted/dropped Gemini thought_signature (reasoning_details) in tool loops
We blind the reasoning details in the frontend because this would leak information. Some models summarize thinking while others provide raw thinking. I'll check if we're corrupting the thought signature that's actually sent to the backend.
So far this sort of looks like an example of Gemini being bad. Was this Argon?
That doesn't seem to match the arena.ai rankings, who I'm assuming has a far larger dataset of rankings than you currently hold.
https://arena.ai/leaderboard
You're right Arena.ai has far more data. Our data is actual local coding agent head-to-head battles, so I think our quality is far higher.
Their agents leaderboard is literally just randomizing the agent and looking at metadata like tool call errors. In my opinion, it's poor quality data in high volume.
Their web dev leaderboard is in cloud containers, most people don't run agents like that.
I ran a statistical test on our data. Deepseek vs. Argon. Posterior probability DeepSeek 4.1 Flash is stronger: 99.1%.
Have you tested older Gemini models? I wonder if their anti-distillation security measures are silently downgrading you to a less capable model.
1 reply →
I've used argon via `agy` cli tool and was not impressed at all. However, I decided to burn up the remainder of my ultra credits on a visual reasoning pet project.
Argon in low and medium mode had higher quality results than it did when running in high mode, almost every time. My tasks involved the agent finding some details in a raster image and then mark it up with predictions about what it found.
Argon on high would do the work, and then a whole bunch of other stuff i did not ask for. All that extra broke everything else downstream from the analysis.
However, doing the same work with Argon in low effort was on par with results from Haiku 5.5 (xhigh) and GPT-6.1-Terra (medium). So, Argon low works better than Argon high, but for my tasks -- a very weak model relative to how google positions it.
I'm ready for Geimini Unobtanium!
Okay, this is interesting. We are hardcoding the effort (to improve data concentration), and Argon is hardcoded to "high".
Have you considered that the harness might be the issue?
Well, my perspective is somewhat narrow (I use it primarily for software engineering and pretty technical data engineering work), so it's possible that Argon is well-suited for the work I do but not for other tasks. I suppose it's just more data points to consider.
Our data is all software engineering run via local coding agents, but yeah your specific use case could be one where Argon is stronger. (Or your personal preference.)
This does not mesh with other battles like Arena where Argon is nearer to the top.
Yeah, we are doing something different and actually running parallel local coding agents head to head, something Arena doesn't do.
> local coding agents
what does "local" coding agents mean? Surely you're not hosting it..
Yeah, this is a new project so help me out here. I mean that the harness is running locally. I'm trying to distinguish this from arena.ai webdev arena, where they run the agent in a cloud container.
my anecdotal experience is that argon and muse wonder full are both a full tier below opus/sol.
So far in our data, there's no difference between Wonder and Sol 6.1. This is with wide error bars, as we're just getting started. Note this is Wonder, not Full as it's not on openrouter yet.
But yes, Opus is a tier above Wonder and Argon with very high confidence.