← Back to context Comment by andriy_koval 14 hours ago Astra also not verified using private set, but on "semi-private" set 4 comments andriy_koval Reply andrewchambers 12 hours ago if that is true then why is astra on the official ARC leaderboard now ? andriy_koval 12 hours ago ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.It is described in their methodology: https://arcprize.org/policyIt makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI. tintor 8 hours ago Where are results for private data?Which LLMs participate on private set? Open weight LLMs only? 1 reply →
andrewchambers 12 hours ago if that is true then why is astra on the official ARC leaderboard now ? andriy_koval 12 hours ago ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.It is described in their methodology: https://arcprize.org/policyIt makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI. tintor 8 hours ago Where are results for private data?Which LLMs participate on private set? Open weight LLMs only? 1 reply →
andriy_koval 12 hours ago ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.It is described in their methodology: https://arcprize.org/policyIt makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI. tintor 8 hours ago Where are results for private data?Which LLMs participate on private set? Open weight LLMs only? 1 reply →
tintor 8 hours ago Where are results for private data?Which LLMs participate on private set? Open weight LLMs only? 1 reply →
if that is true then why is astra on the official ARC leaderboard now ?
ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.
It is described in their methodology: https://arcprize.org/policy
It makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI.
Where are results for private data?
Which LLMs participate on private set? Open weight LLMs only?
1 reply →