Comment by fastball
3 months ago
Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).
3 months ago
Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).
Medium or low supposedly prevents Opus 5 from overthinking:
https://xcancel.com/danshipper/status/2080700057892815114
https://cognition.com/frontiercode
Quality vs cost - medium is the sweet (perhaps better too!) spot.
That is just a single benchmark tho
my issue with frontier code is that it uses a model judge for quality whereas slop code bench forces a model to grapple with its own garbage code in order to receive a functionality reward
If I need something smarter I use Fable. Medium works well and is quick. Opus 5 medium feels much better to me than Opus 4.8 medium.
yeah someone will have to re-run this bench on various effort levels. unfortunately it is not cheap