Comment by aliljet

12 hours ago

The ARC-AGI-3 score is ridiculously high. Is this benchmaxxing or something way different? It's really hard to discern how we're approaching breakthroughs...

6 comments

aliljet

Reply
Legend2440  12 hours ago

They explain why here: https://openai.com/index/how-two-settings-tripled-our-arc-ag...

TL;DR all the other models are being crippled by limitations of their harness.

>First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.

>Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.

  • janalsncm  12 hours ago

    Ok so the correct comparison would be to fix the harness on the old model and re-compare. Now they are comparing a new model to an old crippled one.

  • _superposition_  12 hours ago

    Exactly what I suspected. Of course a machine can just iterate relentlessly the way a human can't.

    I guess token counts are somewhat of a metric.

    IMO intelligence has peaked and all future gains will come from faster tps and more iteration.