Comment by ActionHank
5 hours ago
yeah, IMO AA should be excluding unreleased models from their benches or risk becoming an irrelevant marketing platform that no one uses anymore.
5 hours ago
yeah, IMO AA should be excluding unreleased models from their benches or risk becoming an irrelevant marketing platform that no one uses anymore.
It has already become irrelevant to anyone paying attention. Raw scores at published benchmarks mean next to nothing in modern use. They are either gamed to hell or unsolved to the point where differences can be explained by randomness. Speed, cost per token, tokens per task and hallucination levels are the useful axes you can compare along these days if you're not doing scientific frontier level research.
They probably prefer being used by the deep-pocketed organizations that have access to Gemini 4 Carbon even if it means losing the entire rest of the market.
They've only recently emerged as something people reference and now they're already dying.