Comment by btilly

And of course, study participants are way too often 18-24 year olds who happen to go to college. Such a tiny slice of the general population.

It gets worse. Typically 18-24 year olds who happen to go to the same college as the researcher is working at. So, for example, if this is a large state school then it is a population selected for having SAT scores in a range. Namely above the cutoff to get into the school, but below the cutoff for more desirable schools.

Now suppose that you're doing ability testing. You should expect that any pair of unrelated abilities that help you on SATs will be inversely correlated, because being good at the one thing but landing in that range means you have to be worse at something else. And sometimes that will be the other thing you're looking at.

Several years ago I remember running into a bunch of popular science articles that I found dubious. I tracked down the paper and decided that their analysis suffered from exactly that flaw.