{"id":49555559,"user":"woah","time":1788464223,"time_ago":"14 hours ago","type":"comment","content":"

Haven't people demonstrated all kinds of weak LLMs getting good ARC-AGI-3 scores with special harnesses?","comments":[{"id":49555733,"user":"tintor","time":1788464879,"time_ago":"14 hours ago","type":"comment","content":"

Those people haven't verified their results against the private set: https://arcprize.org/leaderboard","comments":[{"id":49555898,"user":"andriy_koval","time":1788465346,"time_ago":"14 hours ago","type":"comment","content":"

Astra also not verified using private set, but on "semi-private" set","comments":[{"id":49558045,"user":"andrewchambers","time":1788474758,"time_ago":"11 hours ago","type":"comment","content":"

if that is true then why is astra on the official ARC leaderboard now ?","comments":[{"id":49558085,"user":"andriy_koval","time":1788475047,"time_ago":"11 hours ago","type":"comment","content":"

ARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.

It is described in their methodology: https://arcprize.org/policy

It makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI.","comments":[{"id":49559850,"user":"tintor","time":1788489191,"time_ago":"7 hours ago","type":"comment","content":"

Where are results for private data?

Which LLMs participate on private set? Open weight LLMs only?","comments":[{"id":49559868,"user":"andriy_koval","time":1788489322,"time_ago":"7 hours ago","type":"comment","content":"

Yes, they run competitions once a year amongst open weight models","comments":[],"comments_count":0,"level":5,"url":"item?id=49559868"}],"comments_count":1,"level":4,"url":"item?id=49559850"}],"comments_count":2,"level":3,"url":"item?id=49558085"}],"comments_count":3,"level":2,"url":"item?id=49558045"}],"comments_count":4,"level":1,"url":"item?id=49555898"}],"comments_count":5,"level":0,"url":"item?id=49555733"}],"comments_count":6,"url":"item?id=49555559"}

Comment by woah