Comment by seizethecheese
4 hours ago
You're right Arena.ai has far more data. Our data is actual local coding agent head-to-head battles, so I think our quality is far higher.
Their agents leaderboard is literally just randomizing the agent and looking at metadata like tool call errors. In my opinion, it's poor quality data in high volume.
Their web dev leaderboard is in cloud containers, most people don't run agents like that.
I ran a statistical test on our data. Deepseek vs. Argon. Posterior probability DeepSeek 4.1 Flash is stronger: 99.1%.
Have you tested older Gemini models? I wonder if their anti-distillation security measures are silently downgrading you to a less capable model.
Only 3.8 Flash which also didn't do particularly well.
It's possible, but we are getting real prompts from people on real projects, and the instructions about the battle are pretty minimal.