Comment by CobrastanJorji
20 hours ago
As an aside, I do not like unnecessarily misleading graphs. Own your strengths and weaknesses.
In the "Results" section, they choose to compare Ultramarine only to models that it's better than on every metric. They ignore models like Fable 5.1, GPT-6 Astra, Claude Opus 5.5, etc.
Then, in the "Score against cost per task" graph, which is focused on price, suddenly all of those models that perform better than Ultramarine come roaring back because they are also more expensive.
Just list yourselves as the second best sometimes. Your thing is impressive. Don't let the idiot marketing guy hide that behind stupid tricks.
Yes, for me when they do this I immediately dismiss the model, don't even bother testing it. If you need to lie with statistics your model isn't good enough.
Yes, I'm far more likely to purchase if I can understand the full picture. It's not quite as good as frontier models, but it's 60 times cheaper.
Honesty and transparency are attractive.
Maybe you are more likely to buy things when benchmarks are honest, but openai and anthropic have been cheating at those for years. They get purchased a lot. Most purchases are emotional things, even for technical products. That's why they sell llms like it's crack, a weapon, or if you don't you will starve to death in the future. One thing I like about the Chinese and euro labs is they don't market their products in harmful ways, but that's a tangent.
Sounds like there's a gap in the market for an honest player to fill.
3 replies →
I really would like to understand how it can be 60 times cheaper. Is that a like for like comparison? Is there a more equivalent frontier model that is closer in price?
I think they just decided to use the formatting different than the others.
Normally, the background highlight color indicates the leader for every benchmark, but they chose to highlight their column even though there are better models.
APEX Agents => Claude Fable 5 and Gemini.
> I think they just decided to use the formatting different than the others.
500px of the landing page is dedicated to share how they're more "token efficient" than the base model they used. Apparently this -6.5% difference is worth highlighting as the very top feature of this fine-tune they've made.
How much is 6.5% percent concretely though? Assuming the six benchmarks have ~30 tasks each (180 in total), and we're doing 120 tok/s lets say, time/task for GLM-5.3 Flash would be ~25 minutes and this "token efficient fine-tune" would arrive at ~23 minutes.
Again, a 6.5% difference is highlighted as the most impactful change for this fine-tune, but it's a marginal difference. They're definitely trying to mislead people into thinking this is something more than it is, otherwise this landing page would have been much more realistic and less self-patted.
> Ultramarine 1 Flash is Fable-class on knowledge work spanning finance, law, consulting, and operations. It is also 60x cheaper than Fable.
This is not a general purpose model, it makes sense to only bench it on tasks it's supposed to be good at, which is to help out finance-related knowledge work.
That looks like extreme cherrypicking and doing this speaks against the Ultramarine team. Virtue is at a premium in these times.
Not going to be trying it just based on this observed behaviour.
The benchmarks are not ideal.
When are they ever?
I can’t remember a time when I found a benchmark to be genuinely useful even for deterministic software
I like them for computer games and hardware. But yeah, generally functional capabilities are much more important.
This is not an aside at all. It is on point. It seems like whatever statistical trick they used also inflated gemini 3.8 to absurd scores as well.
in my startup, our publicist used to make buzz feed style listicles, consistently listing us as the second or third solution
but it was way deeper than that, because it wasn't actually about people looking up some search string, it was actually about predictably pinging the google alerts for the top company and their founders, and then my same publicist messaging them as a rando on linkedin or email suggesting that they join us for a fireside chat at a conference they had their hands in
with the fireside chat in front of everyone, and the record of it, boosting us to the collective conscious riding off of the more established brand's presence
but yeah our personal google results also looked amazing for several years too
everything serves a parallel purpose, but yeah there's almost no utility in self aggrandizing on your own homepage