Comment by godelski
1 day ago
Switch back and forth between trying and reviewing. Often it can be good to just try before reviewing, to get your feet wet. Don't spend too much time. Then when reviewing you're going to understand it more. Repeat this process.
But there's some things to remember that are incredibly important
- a paper doesn't *prove* something, it suggests it is *probably* right
- under the conditions of the paper's settings, which aren't yours
- just because someone had X outcome before doesn't mean you won't get Y outcome
- those small details usually dominate success
- sometimes a one liner seemingly throw away sentence is what you're missing
- sometimes the authors don't know and the answer is 5 papers back that they've been building on
- DO NOT TREAT PAPERS AS *ABSOLUTE* TRUTH
- no one is *absolutely* right, everyone is *some* degree of wrong
- other researchers are just like you, writing papers just like you
- they also look back at their old papers and say "I'm glad I'm not that bad anymore"
- a paper demonstrating your idea is a positive signal, you're thinking in the right direction
As soon as you start treating papers as "this is fact" you tend to overly generalize the results. But the details dominate so you just kill your own creativity. You kill your own ideas before you know they're right or wrong. More importantly you don't know how right or how wrong.
Your bullet points explain most of the replication crisis, from my perspective.
They're definitely deeply related. For example, a lot of works get rejected over "novelty" issues. Well, if success and/or failure depend on something seemingly small then it will almost never get through review because it seems like low novelty. Though it'll get through review if authors are convincing enough, which often leads to some minor exaggerations.
Combine that with the publish-or-perish paradigm and I think we got significant coverage. People don't even consider diving deeper into things and are encouraged to take the route of "assume paper is correct" because that's the fastest way to push out research. But if the foundation is shaky, then everything built on it is shaky too.
Which, that's a distinction in the hard and more formal fields like math and physics. They have no issues pushing out papers that may have errors in them because the process is to attack works as hard as possible. Then whatever is left is where you build again. You definitely have people take advantage of this, like Avi Loeb publishing about aliens, but it is realistically a small price to pay. And hey, even Loeb's work still contributes. If at some point it actually is aliens, then there's work existing that can be built upon. And when it continues to not be aliens, there's existing work to build on since really his problem is more that the papers just end up concluding "and this is why we can't rule out aliens!" (-__-)
Anyways, long story short, my advice is to just remember that you, and everybody else, is a blubbering idiot and it is a absolute fucking miracle a bunch of mostly hairless apes can even communicate, let alone postulate about the cosmos. At the end of the day we're all on the same team, seeking truth. Truth matters more than our egos and if we start to forget how dumb we are then we'll only hinder our pursuit of truth.