Comment by tintor 13 hours ago Those people haven't verified their results against the private set: https://arcprize.org/leaderboard 5 comments tintor Reply andriy_koval 13 hours ago Astra also not verified using private set, but on "semi-private" set andrewchambers 10 hours ago if that is true then why is astra on the official ARC leaderboard now ? andriy_koval 10 hours ago ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.It is described in their methodology: https://arcprize.org/policyIt makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI. 2 replies →
andriy_koval 13 hours ago Astra also not verified using private set, but on "semi-private" set andrewchambers 10 hours ago if that is true then why is astra on the official ARC leaderboard now ? andriy_koval 10 hours ago ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.It is described in their methodology: https://arcprize.org/policyIt makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI. 2 replies →
andrewchambers 10 hours ago if that is true then why is astra on the official ARC leaderboard now ? andriy_koval 10 hours ago ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.It is described in their methodology: https://arcprize.org/policyIt makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI. 2 replies →
andriy_koval 10 hours ago ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.It is described in their methodology: https://arcprize.org/policyIt makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI. 2 replies →
Astra also not verified using private set, but on "semi-private" set
if that is true then why is astra on the official ARC leaderboard now ?
ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.
It is described in their methodology: https://arcprize.org/policy
It makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI.
2 replies →