Comment by hedgehog

Thanks, that's helpful. Do you still have the full data of pass/fail for each test and model? It would be interesting retrospectively to see if there is such a subset.

I built some tools for personal research projects that could potentially help success rates on long dev tasks, empirically it seems to but I've never actually done a quantitative evaluation. SlopCodeBench looks promising for all the reasons you point out in your post.