Comment by Roark66

When the weights are closed I don't believe any benchmark.

I just got Qwen3.8-27B to score extra 10% on SWE Pro by adding a proxy in front of it that has few simple "harness like features": - when the model gets stuck it tells it to "go on" - when it sends no output, malformed json, slips to wrong tool use format, etc it asks it to "try again better" - detects repetition and tells the model. - injects a prompt about "planning tool use" when it seems to be using same tools repeatedly. - injects a reminder it can use tools if there are no tool uses for over X messages.

10% - with just that.

I have more to test. My point is, open weights models get tested on naked model quality. "Frontier" models get tested as a model + whatever secret sauce they choose to put in front.