Comment by montroser
5 hours ago
Yeah, it is very suspicious. Each time they update their ranking algorithm, the closed models from US labs jump up and any previously high ranking open model jumps down. It happened with MiMo, Kimi, GLM. All were right there amongst the very top, and each time that happened, AA followed quickly with an index update that shunted them dramatically lower.
Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.
I also look at the ECI from Epoch.ai as well, but they haven’t evaluated Carbon yet. Probably because the model isn’t widely available yet.
This could be another point informing your prior as to how much benchmaxxing and distilling of closed models is being performed by the open weights publishers as well.