← Back to context

Comment by jltsiren

2 days ago

In the almost 20 years I've done academic research, I've met thousands of scientists. Some of them have been involved in various scandals, but as far as I know, none of the scandals were about scientific integrity. When it comes to academic scandals, those involving scientific integrity seem to be rare.

The reproducibility crisis seems to be mostly about applying the scientific method naively. You study a black box nobody really understands. You formulate a hypothesis, design and perform an experiment, collect data, and analyze the data under a simple statistical model. Often that's the best thing you can do, but you don't get reliable results that way. If you need reliability, you have to build models that explain and predict the behavior of the former black box. You need experiments that build on a large number of earlier experiments and are likely to fail in obvious ways if the foundations are not fundamentally correct.

I'm pretty bad at getting grants myself, but I've known some people who are really good at it. And they are not "playing the game", or at least that's not the important part. What sets them apart is the ability to see the big picture, the attention to details, the willingness to approach the topic from whatever angle necessary, and vision of where the field should be going. They are good at identifying the problems that need to be solved and the approaches that will likely solve them. And then finding the right people to solve them.

I guess at a very high level, the question is, do you think the current system and what it incentives is fine/optimal (and are sanguine presumably then about things like the the Lesné Aβ*56 fraud or the OP article [the failure of over half the biomedical experiments tested to repro]), or do you think it can be improved?

To me it clearly seems like there is room for improvement!

Even granting that most scientific researchers are pure of heart and noble of purpose, the kinds of science we get (and how quickly we uncover spurious results) are still going to depend on the systemic incentives of funding, publishing, & prestige -- so it's worth trying to structure those systems in a way that rewards good science as much as possible.