Comment by evilmathkid
5 days ago
2 different things are being called leakage here
1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training
2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser compute
--
also re LLMs: they do not follow this 3 split since (a) Incredibly hard to keep a pretrain dataset clean, (b) common in labs to benchmaxx during postraining (and known to do so on ARC)
> weight update during eval: this is a form of test time training and not really cheating.
Possibly "not really cheating", but it does make benchmark comparisons unfair - especially as the other models are unlikely to have their weights updated during the eval.
no, most models on kaggle are finetuning during test time, (including LLM based approaches)
Pure frontier LLMs dont, but thats because nobody knows how to make it work cleanly and at scale. Once someone makes it work, it will be deployed