Comment by weiliddat

For most of these benchmarks, I feel like a p50 and p95 (using SWE salary as a proxy?) human benchmark as reference would be interesting.

Edit: FWIW the paper the post quoted has repositories as slop baseline https://arxiv.org/html/2603.24755v1#S4.SS2