Comment by softwaredoug
12 hours ago
I'm seeing reporting it gets 98.6% on ARC-AGI3[1] (previously like 30% with Fable)
https://venturebeat.com/technology/welcome-to-the-agi-era-op...
12 hours ago
I'm seeing reporting it gets 98.6% on ARC-AGI3[1] (previously like 30% with Fable)
https://venturebeat.com/technology/welcome-to-the-agi-era-op...
This is with the caveat that OpenAI uses their own harness for this:
> On ARC-AGI-3, GPT-6 Astra was run with our responses API harness , which changes two settings to better match real-world performance. The changes do not specifically target ARC-AGI-3.
Its 62 percent when using a neutral harness. https://arcprize.org/blog/astra
Yet it is an impressive number. But yeah when you see a number 99 you have doubts. Thanks for the link
Interesting both this and Sol got approximately a 37% boost with the custom harness.
This should be normalised and expected - the responses API harness allows it to use the custom compaction that is not allowed otherwise. It is entirely fair to allow OpenAI to use their own compaction algorithm..
"This should be allowed, let me explain the reason they cheated and state again that they should be allowed to cheat."
5 replies →
"On the current ARC-AGI-3 leaderboard, conventional frontier-model runs sit dramatically below Astra's reported 98.6% result.
But the comparison isn't straightforward.
OpenAI's own evaluation notes say Astra uses the company's Responses API harness, while comparison models can operate under different configurations."
The blog post says 99.9%. Oddly, it does better on ARC-AGI-3 than it does on version 1 or 2 of the same benchmark (though gets 95+ on all three)
I strongly suspect that is way above the human average anyway, esp. ARC 2 and 3 are really tough unless you happen to be great at those spacial puzzles or video games.
Scoring for ARC-AGI-3 is constructed so that the median(-ish) human score is 100%, so this is not a superhuman result. However, the scaling is weird, since it's built from terms that look like (AI turns taken / median human turns) ^ 2, and it weights later levels higher than early levels. So it's not at all clear that 100% is twice as good as 50%.
1 reply →
Really though? I would believe something like this if a model could one shot every solution in the set. I don't pay much attention to these things and maybe this stuff is available but I would bet the session/reasoning transcript is absolutely horrendous from an intelligence standpoint.
At this point the only valid ARC-AGI benchmark left is to make up the next series of ARC-AGI benchmark puzzles that current models presumably can't handle.
1 reply →
100%, some say.-