Comment by gertlabs

This is something we omit for a few reasons but it's probably the biggest blind spot in our evaluations; we opt-in to auto-reasoning/adaptive reasoning or max thinking token budgets where supported (supported by most models now), but when an explicit reasoning level is required, we fall back to High reasoning. In practice, we've found most models scale High-><whatever marketing term is max reasoning> pretty consistently, but if one vendor started throwing 10x the resources into max reasoning and they didn't support auto-reasoning, they would be unfairly penalized in our evaluations.