Well, if you have a clearly defined task it's easy. When using them for my pipeline of writing explanations for Chinese words I have clear ranking, for example - opus 5.5 clearly better than opus 5 at writing and knowing details, annoying nit picker when it comes to finding errors (high accuracy, low usefulness) all in repeatable numbers on different datasets. The problem is that these models are most useful when you are facing a new task that you haven't encountered before. And yup then it's astrology.
Well, if you have a clearly defined task it's easy. When using them for my pipeline of writing explanations for Chinese words I have clear ranking, for example - opus 5.5 clearly better than opus 5 at writing and knowing details, annoying nit picker when it comes to finding errors (high accuracy, low usefulness) all in repeatable numbers on different datasets. The problem is that these models are most useful when you are facing a new task that you haven't encountered before. And yup then it's astrology.