Comment by daytonellwanger
4 hours ago
Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?
4 hours ago
Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?
This is not actually running the Arc-AGI-3 anymore. To summarize TFA:
1. The AI plays the game and records outputs.
2. The AI does TDD using those outputs to create its own copy of the game.
3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
To make an apt comparison... You go to get your chess ELO. You don't know chess at all and you're really bad at it, so you pull out your laptop and write a chess engine. Then when you go to get ranked, you just copy the moves from the software. Now you're a grand master.
> model's ability to intuit game rules naturally, like a human
This line of thinking terrifies me for the future of humanity...
If you look at pretty much every animal other than the intelligent 5, they cannot use tools. Tool use is a human superpower. The model writing a tool that does the hard work for them, at least to me, is a sign of intelligence (laziness is the mother of all invention).
Where this becomes really interesting is when the LLM can write novel tools to solve novel problems.
Trying to say that AI has to act like a human to be AGI is something that will end up with us humans going extinct from a digital intelligence we don't understand at all.