Comment by Topfi
6 hours ago
> Carbon ranks #1 on the recently updated v5.0 of Artificial Analysis Intelligence Index [0] at 59.
If you are going to hyperlink, maybe make sure that your model is actually on the page.
Also, the screen reader has a bug ([[duration]] minutes instead of the actual value) and you have once again announced a model we cannot use.
Edit: Both have been fixed after posting this. Maybe a Gemini 4 Carbon Agent has read and fixed the screen reader bug xD
I cannot fathom what motivates a lab, whose releases for the last year haven't been competitive, to shoot themselves in the foot like this. All this does is make it easier for competitors to snipe your actual launch, even if there was potential with this model (doubtful given 4 Argon, 3.1 Pro and 3 Pro), now everyone knows what exactly they need to release eval results wise to beat Carbon. There is this old idea of "Your competitor always is the first to have an idea what your next product will do", but in this case, Anthropic and co don't even need to guess, they got specific numbers to target.
Announce your models the moment they are available on API and in the subscription for all paying subscription users (cough Argon cough), stop with the cryptic tweets to drum up hype you know full well you won't live up to and make models that tool call, task adhere, task infer and handle long context tasks well beyond simple back-and-forth, chat style prompting. Reminder, during Ox Alpha, actual Google employees implied that was their model. How is this something anyone would believe is a good idea and how are they still allowed to tweet anything after? Please, stop that, it isn't helpful.
If frontier isn't your thing, stop wasting money on moonshot pre-train runs, fund research and open weight via Gemma and sell your inference hardware to other labs, that's far more lucrative anyway, as seen with Colossus 1.
[0] https://artificialanalysis.ai/evaluations/artificial-analysi...
Oh come now, this rolled release is now the standard for frontier models. Also, the fact that AA hasn't updated their page is not a scandal, just a launch day hiccup.
> Oh come now, this rolled release is now the standard for frontier models.
Sort of? Yes, OpenAI has been bad about slow rolling but Anthropic's last few models have been available essentially at the same time as they announced it. In most cases I updated Claude Code and was using the new model right away.
Compare that to OpenAI who sometimes says "Available to all Plus/Pro/etc plans" without saying anything about "rolling out" and then it's a crap-shoot on when access will actually roll out. That said, they have gotten better, it used to be 1-3 weeks sometimes to get a new model/feature they announced was available for your plan.
And then there is Google... Who loves to talk about models like this and then give no timeline for release. Even when they do release the models it's miserable to find out how to even use it. Is it in the API? Which API? Google Cloud/Vertex/Gemini Cloud? Whatever it's called this week? Coding plan?
No, what Google does (or rather fails to do) in its rollouts does not compare to OpenAI/Anthropic.
As for the AA page, it's just embarrassing. Any smaller company doing that would get roasted, Google doesn't get a pass, learn how to coordinate or wait for something to actually be true/live before posting.
There was none for Fable 5.1, Opus 5.5, Sonnet 5.5, GPT-6.1 Sol or GPT-6.1 Terra and while Astra had a period between announcement and availability, it was a single day, so not sure that applies either.
Gemini 4 Argon (at best Sonnet/Terra level) meanwhile took weeks to “come out” after their public announcement and when it finally did, paying Google AI plan users couldn't access Argon on Gemini or Antigravity even a week after. Three weeks from announcement to accessible for paying customers is not standard outside Google and this has been a problem with every Gemini release for the last year, so even using governmental requirements as an excuse isn’t possible here.
I will say that I dislike how OpenAI releases model only for Work and Codex but not via Chat, Anthropics "everywhere all at once" strategy is I feel the most user friendly.
AA has also lost all credibility to me with all of their index changes being unannounced and following an open weight model getting close to the top spot. I gave them the benefit of the doubt for v4.2, but to do this again immediately after Kimi K3.1 got into the top 5 is a little too suspicious for me. For Opus and Astra to stay at about the same score while everything else drops feels like they are adjusting weights specifically in their favor.
Even in the optimistic case that this isn't some conspiracy, it highlights how arbitrary their scores are. I had trusted their previous ratings and their new index shows that even AA doesn't agree with them. What's to say this new index is any better and won't just change drastically again?
I imagine there is immense pressure to announce taking the #1 spot while they still have it. It's probably better than releasing a model that hasn't been properly vetted, but I agree it's annoying.
Fair point, though if they are still vetting, that'd likely also entail some post training or at least the addition of a filter, should they find something they do not like. As we've seen with Fable 5 and Opus 4.8 Fallback, that can significantly affect user experience and benchmark scores.
So if they are still vetting and changes can thus happen, that'd mean the score could change for better or worse. Not sure they'd want the embarrassment of a #1 score changing after announcement.
There is an earnings call today for Alphabet.
We're reaching the point where models are only going to hold the number one spot for hours.
I honestly still need at least a week before I have a decent picture of GPT-6.1 Sol performance. The announcement pace is truly insane.
> If you are going to hyperlink, maybe make sure that your model is actually on the page.
It's on my screen. Try asking your agent to refresh the page.
As written, both issues were fixed ten minutes after I posted the comment.
gemini 4 carbon has now come on the artificial analysis page. I think that I had checked it 5 minutes ago and it wasn't there.
For historians, The blog post is 31 minutes ago so we can say that it was true for 20-30 minutes.
Slightly embarrasing but it seems to be the case that the model is actually worth what they were saying on AA. It's just that it is not available to people and I think that Anthropic and OAI will release a better model to public before google might.
AA has tons of its flaws too and the recent shift to their new v5 has been nothing but controversial for many.