Comment by shelfchair
12 hours ago
I don't think this is relevant on AA's FrontierCode score. AA runs FrontierCode 2.0 benchmark using their own Stirrup agent harness [1].
[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0
12 hours ago
I don't think this is relevant on AA's FrontierCode score. AA runs FrontierCode 2.0 benchmark using their own Stirrup agent harness [1].
[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0
My mistake, thank you for catching.
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?