Comment by kingstnap

The first implies generalization. It's not a test of generalization.

It's actually close to the second. "Here are 200 software questions, will drill you on *other stuff* until you can pass exactly these 200. If the other stuff isn't improving your scores we will change ratios of it till it does."

The reason it benchmaxes is that *other stuff* ends up looking more and more like SWE Bench without you realizing it.