Comment by highfrequency
3 hours ago
Not sure how to interpret the Artificial Analysis leaderboard anymore. I checked a few days ago and Muse was rated higher than the best OpenAI models; it seems that shortly afterward they changed the methodology and now Muse is way lower and Gemini is #1.
This likely isn't AA putting its finger on the scale for specific interests, rather in the case of Muse/Astra, it's to avoid looking really bad because it scores things way out of wack with people's anecdotal vibes on model capabilities. It is possible that labs are getting preliminary AA results back and raising concerns about scoring that then get folded into updated benchmarks, but I strongly doubt it's nefarious.
They announced their new Index v5 yesterday, they frequently add new benchmarks and rebalance old ones.