Comment by dhorthy
8 hours ago
Yeah and I started with subsets although I did not find them representative, eg I picked 6 challenges for a subset and fable scored 33%, while on the full challenge it landed closer to 14-15% range
The part II post linked in my top comment has a list of the challenges I used for that.
To run them, just point your model at the paper, it can find the repo, etc and with a key or sub, get it running
Thanks, that's helpful. Do you still have the full data of pass/fail for each test and model? It would be interesting retrospectively to see if there is such a subset.
I built some tools for personal research projects that could potentially help success rates on long dev tasks, empirically it seems to but I've never actually done a quantitative evaluation. SlopCodeBench looks promising for all the reasons you point out in your post.