Comment by michaelt
2 days ago
> And we must ensure that explicit suggestions to modify one’s science in the service of one’s career – “you need to do X to be published”, “you need to publish Y to graduate”, “you need to avoid criticizing Z to get hired” – carry social penalties as severe as a suggestion of plagiarism or fraud.
One of the pernicious things in this area is that, even as we teach young researchers how to avoid making mistakes and engage sceptically with the work of others and that scientific fraud is a nontrivial issue, we also tell them how to commit fraud themselves and that their competition is doing it.
"Watch out for P-hacking, that's where the researcher uses a form of analysis that has a small chance of a false positive, and analyses loads of subsets of your dataset until a false positive arises and just publishes that one"
"Watch out for over-fitting to benchmarks, like a car taking the speed crown by sacrificing the ability to corner"
"Watch out for incomplete descriptions of test setups, like testing on a 'continent-scale map' but not mentioning how detailed a map it was"
"Watch out for citations where the cited paper doesn't say what is claimed, some people will copy-and-paste citations without reading the source paper"
"Watch out for papers using complicated notation, fancy equations and jargon to make you feel this looks like a 'proper' paper"
"Watch out for deceptive choice of accurate numbers, like a study with a 25% completion rate including the drop-outs in the number of participants"
"Watch out for simulations with inaccurate noise models, if the noise is gaussian in the simulation but a random walk in reality, great simulated results won't transfer to reality"
I've made no suggestion at all that you should modify your science or commit fraud - but I've also just trained you in how to do it.
It's really not that hard to come up with ways to commit fraud if you want to. On the other hand, it's very easy to make such mistakes if you don't know to avoid them. This characterization is very misguided IMO.
Ah, perhaps I wasn't clear about what I'm trying to say. I don't think we should stop training researchers in common mistakes and fraudulent methods to watch out for.
I'm just saying: I don't believe anyone actually tells budding researchers that they should commit fraud. Instead I think the process is probably more like this:
Year 1: Statistics/research training. Here are a load of subtle mistakes to watch out for and avoid. Scientific fraud happens sometimes. Don't do it, it's very dishonest.
Year 2: Starting research. Gee a lot of these papers I'm reading are hard to reproduce, or unclear. Maybe fraud is widespread - or maybe they're just smarter or better equipped than me.
Year 3: "You really ought to have published some papers by now, the average student in your position has 3 papers. If you don't want to flunk out you really need to start showing some progress"
I still disagree. It's more like "omg I should have published at least a few papers by now, what am I doing" and then you start frantically looking for provable things in the dataset. You find one that you can also support with a nice story. Now either a) You were not tought about how or why this is wrong and you publish the paper b) You were, and know that you should collect a separate dataset to test the hypothesis. But also, there is a huge existential pressure to just close your eyes and roll with it.
It's not that you need to be tought how to cheat, it's that you need to be tought how to avoid unintentionally cheating.
3 replies →