Comment by foota
5 days ago
> Ban offline training/pretraining. Models must train from scratch after submission Previously this was considered impossible so rule. My model shows this is possible Guarantees no synthetic data can be used It makes the comparison fair across differet models. Otherwise some models like LLMs can benchmaxx ARC by using ungodly amounts of offline training. (Since the benchmark has been around a long time, many ARC-like datasets have been created)
I'm not an ML researcher, so YMMV, but... how could a model learn to answer these ARC-AGI questions without training beforehand?
train from scratch only during the 12 hours allowed on Kaggle
Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun
[dead]
how would you as a human know the answer to the arc-agi questions?
I spent years learning logic and doing puzzles. I don't think a baby or even average kid could solve these.
Predicate logic is trivially realized by linear transformations (I.e., matrices), and these matrices are easily discovered via gradient descent with appropriate reward functions.
> I don't think a baby or even average kid could solve these
The reward functions of a typical baby or kid is not 'get a huge dopamine boost when you solve a logic puzzle' (or whatever neurotransmitter, I don't know).