A dataset being as popular as their's is will contaminate the data just by people discussing it and creating their own public test sets of similar problems.
Still, probably not that much compared to employees targeting it.
ARC's harness is just straight up broken. No serious harness removes reasoning context between each step. Not only does this significantly lower performance over all reasoning LLMs, but it also increase cost as you destroy the cache on every turn. Tossing the oldest entry when context fills up instead of using compaction is equally bad with the same issues.
https://mvakde.github.io/blog/44-on-arc-1/ makes a good case that all the performance on the arc agi tests is overfitting, based on the fact that v1 performance did not translate directly to v2 performance
I think it could indicate that "semi-private" dataset likely leaked to their training data.
A dataset being as popular as their's is will contaminate the data just by people discussing it and creating their own public test sets of similar problems.
Still, probably not that much compared to employees targeting it.
It says "Provider Adapter" so presumably they put some manual work in to make this work.
AGI means that the answers to these benchmarks are now accessible, and the defense capabilities of the testing organizations are now negligible.
ARC has their own writeup on the result, which offers some nuance. https://arcprize.org/blog/astra
tl;dr it's 62% when apples-to-apples to other models, which is still notable.
ARC's harness is just straight up broken. No serious harness removes reasoning context between each step. Not only does this significantly lower performance over all reasoning LLMs, but it also increase cost as you destroy the cache on every turn. Tossing the oldest entry when context fills up instead of using compaction is equally bad with the same issues.
Woah, that is a crazy interesting read!
The no-reasoning version scores 35% while the low reasoning one scores 17%? What?
I suspect this is "no reasoning set" which might be "default: medium" or perhaps some smart routing. I don't think it's literally "no reasoning".
It says (none), which usually means reasoning disabled.
I would be surprised if none = use default reasoning
It's simulating the Dunning-Kruger effect.
Look at those costs!
Right?
Between $18k-40k to run a benchmark.
But scored less on V2 and V1 ... too much overfitting?
https://mvakde.github.io/blog/44-on-arc-1/ makes a good case that all the performance on the arc agi tests is overfitting, based on the fact that v1 performance did not translate directly to v2 performance
saturated before (higher degree) AGI-2