Comment by NitpickLawyer
3 days ago
> people built harnesses that solves arc agi 3,
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
For frontier models, not local.
https://schema-harness.github.io/
Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
No but the session traces are available. It passes the sniff test considering how AGI-3 is scored and how this wrapper works.
For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human.
Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model.
https://huggingface.co/schema-harness
2 replies →
If you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.
1 reply →