Comment by JBits

They look at each problem individually.

As the ARC AGI guidelines [1] state, 'a core design principle of ARC-AGI is that the test taker must not know what the test will be.'

[1] https://arcprize.org/policy#dataset-security