Comment by hadlock
3 days ago
I agree, however It's too easy to game arbitrary limits like "medium". There is no olympics for "just rolled out of bed and slightly hungover"
3 days ago
I agree, however It's too easy to game arbitrary limits like "medium". There is no olympics for "just rolled out of bed and slightly hungover"
We don't even need a standard for reasoning levels.
Just accept whatever the provider names as "medium" and compare the accuracy between models vs the time it takes to complete a task.
The fashion industry has been doing this for decades. Medium means different things for different brands.
You don't need to directly compare the same reasoning levels. Today you can compare Sol on Low vs Haiku on Max, for example, but you can't look at GLM 5.3 High, only Max.
For some tasks I genuinely feel Opus 5.5 does better on low than medium or high. The trouble, as always, is figuring out which.
Edit: I realize after commenting that you're not suggesting medium is useless, but just arbitrary and differs between models. Personally I don't want to directly compare mediums between models, I just want to be able to compare across effort levels.
i run opus on medium, sonet on high, and haiku on high for implementation based how complicated is for planing and specing and review opus is on high usually, except if if the task is contained.
did not try to run it on low, will have to try it.
Opus 5.5 on low is good if you are relatively sure a given task is straightforward and time spent gathering background information is purely wasted or possibly even detrimental. Opus 5.5 low seems great at doing the most obvious thing based on the prompt without thinking much about it.
1 reply →
That's the beauty of the line charts; the labels are arbitrary points on the line. You can easily compare the series visually.
A graph of time (or cost) per task against success rate does it. Just include all the different levels.
This exactly. The three interesting parameters are total time to answer, total cost, and output quality (for any given purpose).
[flagged]