Comment by vikramkr

How does it do on a more modern benchmark? Was harsh towards this in my other comment but that's a combination of this benchmark is useless, and also these open source models are pretty good and there might actually be real utility to good custom harnesses tuned for them that's getting ignored because the parity with the frontier on a saturated and contaminated benchmark makes for better headlines. I'd be curious if there's anything with these types of harnesses that make them work for realistic/relevant workflows