Comment by elxr
14 hours ago
The 4-tier naming is honestly as clear as you can get. I have no idea what people are so confused about.
Terra and Luna are dirt cheap and good enough for 50% (or more) of the typical junior dev's workload, and great for routine work. Sol and Astra have so much written about them at this point that it should be obvious what those are good at. The large/gigantic models for things that need full intelligence and slightly more branching thinking.
The obnoxious/confusing thing is that models are released individually rather than as a family.
Luna < Terra < Sol < Astra, fine. But does Terra 6.1 beat Sol 6? Does Fable 5.1 beat Opus 5.5? If I build something around this Terra 6.1 release, then when Luna 6.2 is released in a month, should I swap?
I understand the confusion, but isn't this kind of similar to asking does the iPhone 14 Max beat the iPhone 16 Mini? At least with phones there are hard specs we can use to compare, but in a lot of cases the answer will be "it depends"
At least there is a simple system - the words 'Mini' and 'Max' are intended to refer to the size of the item (that one is miniature, and the other is their maximum size).
If they were called iPhone 14 Luna and iPhone 16 Sol then this is less clear.
Or on the flipside, if the models were called GPT 6.1 Mini and GPT 6.1 Max, nobody would be arguing about the names being hard to grok. The words 'mini' or 'small' are more descriptive than 'Luna' or 'Terra'.
7 replies →
> does the iPhone 14 Max beat the iPhone 16 Mini?
Sure, in a hypothetical world where the latest Max phone is two generations older than the latest Mini phone. In which case now you have to evaluate not only screen size but ALSO specs and component age and software support etc. How badly do you need the bigger screen? Etc
If you are comparing the iPhone 16 Max to the iPhone 16 Mini, everything is a lot easier.
> iPhone 16 Mini
I wish
Yeah, but shouldn't then all 4 tiers be released at once? So we get 6.1 for all Luna/Terra/Sol/Astra?
It's more like if Apple called the iPhone 16 the "iPhone 14 16 edition"
2 replies →
I mean yes, phone manufacturers have the same problem.
No. It’s not similar at all.
Run your own evals then if you have this much of a need to "correctly" choose. That'll make it clear.
I'll just stick to mostly Sol on low/medium/high and start experimenting with Terra tomorrow onward to see what it does noticeably better than Luna/Deepseek flash.
Any pareto frontier model, especially from the leading labs is guarnateed to be good enough for whatever your expectations are at this point. A quick few day test should be enough for anyone to make their own conclusions.
Right, having products so poorly described that a customer needs to run their own evals to choose well is exactly what is being complained about.
Or I'm going to give up and not use any. Consumers aren't perfectly rational. This isn't like toothpaste, where, despite being overwhelmed at the store, I still need to brush my teeth so ahhhhhh fuckit and just pick a toothpaste out of the 30 available. The alternative is I keep using Google search and think AI is stupid. Or just use Facebook's thing.
2 replies →
> > But does Terra 6.1 beat Sol 6?
> Run your own evals then if you have this much of a need to "correctly" choose.
No. OpenAI want my money, it's on them to communicate properly.
5 replies →
> Luna < Terra < Sol < Astra, fine. But does Terra 6.1 beat Sol 6? Does Fable 5.1 beat Opus 5.5? If I build something around this Terra 6.1 release, then when Luna 6.2 is released in a month, should I swap?
I understand how it can be confusing, but I don't think that this is exactly a problem.
Luna through Astra offer qualitative descriptions of the model's size. Presumably OpenAI trains its biggest model (Astra-ish, maybe even larger) first, then distills it to smaller sizes.
Verison numbers reflect something about the architecture or training dataset. Again, they keep these details close to their chest, but we should expect that weight-for-weight higher versions should beat lower versions.
This offers us two axes of 'better', but these axes are also not directly comparable. An earlier but bigger model may or may not beat a newer but smaller model. This is where 'spiky' intelligence comes in.
This is also where calls to 'run your own evaluation' really do start to matter. If you care enough about this performance[1], then you also have specific problems that you care about. No generic benchmark can fully answer that question, hence the need for a specific evaluation.
[1] — And you don't have to! A casual user is probably served reasonably well by anything put out in the past six months. I think Microsoft is still serving GPT-4o under some Copilot branding.
I think it is a problem, for OpenAI. If their answer to "which of your models are better" is to throw their hands in the air and say "fuck if we know, go figure it out yourself" then that's going to go poorly as soon as they try to actually sell to some of the market they're predicting they can sell to.
Whether they train the biggest model first or not is not relevant; what matters is what they release. Right now we've got 6.1 Sol and 6.0 Astra. Over at Anthropic they've got Opus 5.5 and Fable 5.1. The existence of a hypothetical Fable 5.5 that I'm not allowed to use doesn't help me.
> The 4-tier naming is honestly as clear as you can get. I have no idea what people are so confused about.
I wouldn't necessarily say that. Atmospheric mass may be obvious to us, but I suspect it's not nearly as universal knowledge as one may assume.
And I actually assumed at first that they were based on distance from us (terra < luna < sol), so I kept confusing terra and luna.
It’s totally clear. About as clear as mud.