Comment by A_D_E_P_T
15 hours ago
Wow. If those charts are true, this effectively matches GPT-6-Astra at a fraction of the cost, and is neck-and-neck with 6.1-Sol.
15 hours ago
Wow. If those charts are true, this effectively matches GPT-6-Astra at a fraction of the cost, and is neck-and-neck with 6.1-Sol.
Looks great on the first chart, but terrible on the Terminal Bench 4.0 where it needs xhigh to come close to 6.1 Sol, and costs more for a slightly worse result.
I would guess only on the nitpicked benchmarks?
The pelicans place it nicely between Sol and Luna in terms of both pricing and results.
pelican is also nitpicked (and probably the most useless)
Cherry-picked is the term, not nitpicked.
Yes, it may be dumb, but it isn't useless because it is both a) is a benchmark and b) is not picked by openai
1 reply →
Not necessarily, they may have selected benchmarks where Terra performs relatively well compared to those models. Pretty much every model release these days just cherry-picks & highlights the benchmarks where their latest model performs best. We'll have to wait for the complete benchmark suite to have a better picture.