Comment by gus_massa
3 years ago
> These policies, like all forms of gatekeeping, are potentially terrific solutions for weak-link problems because they can stamp out the worst research. But they’re terrible solutions for strong-link problems because they can stamp out the best research, too. Reviewers are less likely to greenlight papers and grants if they’re novel, risky, or interdisciplinary. When you’re trying to solve a strong-link problem, this is like swallowing a big lump of kryptonite.
Scientist can publish whatever we like. Good research, bad research, whatever. Just open a blog like Tao https://terrytao.wordpress.com/ There are no "gatekeepers".
One problem is how a committee can evaluate that. The bad solution we reached is to just count the number of papers in serious peer review journals. But now there are a los of predatory journals where you can just publish whatever you like if you pay, and other kinds of bad journals. So there are more complicated rules to define what is "serious" and what is a "journal".
I don't understand what the author propose. If all journals publish whatever people submit, they just become a clone of WordPress, but we already have WordPress and a few alternatives.
Other problem is how non-scientist can consume that. Which groundbreaking results should be copied to newspapers. Now the problem is that journalist in the science section has little scientific formation, so it's difficult to evaluate every single post in WordPress and decide which are good. So they use papers in serious peer review journals as a good approximation, but many times they just copy the bad press release of the university.
Another problem is which results should get applied to public policies. We expect that the person in charge is an expert and have experts advisors in each area. But it will be too much work for them to be reading all day WordPress to find promising results, and evaluate all of them. Also, when people in other areas can just copy-and-paste a result to use it instead of measuring it again.
My guess is that in that case there will develop a net of relialable curators, something like "awesome-volcanoes" or "awesome-cetacean". Each one is too much work for a single person, so they may ask for help from trusted friends. They may even give advice about stile and clarity, before the blog post is included in the awesome list. And now we have reinvented the peer review journal system, and the only difference is that the papers are never printed in dead trees.
Maybe we are bound for the future not unlike the XVIII century where people who do original research have to be independently wealthy (or have a patron), whereas universities do something more akin to scholastics/theology (reiteration of existing knowledge with only occassional accumulation of new one).
We just need an up/down voting mechanism for scientific papers with voting restricted to people knowledgeable in the field. Don't know how to implement it but it should be designed with incentives to vote and guards against it being gamed.
Moderation is hard.
Also, there is the homeopathy problem. Do only homeopaths vote if a homeopathic paper is good or bad, or any medical doctor can vote?
It will be very difficult to distinguish voting ring from serious research, because some communities are very small and each one cite each other paper, and have some papers together from time to time.
Now the idea is that the jornal has a magic public vote, and if the paper is bad the reputation of the journal is stained, and after enough bad papers the hivemind decides it's a bad journal.
Another form of votes are the citations. So instead of karma, there is the total number of citations, the h-index, and other form of counting them. They also can be gamed, because people can add self citations or citations to friends, specially in the introduction. And sometimes the reviewers ask for a cites, like in https://news.ycombinator.com/item?id=35738717 but most of the times it's more subtle.
There are a few new journals that claim to be "peer review", but are actually just a forum with upvotes and commets. I'm not sure if any of them are "serious" because they are not in my area, but moderations is very hard. If votes are secret, there is no incentive not to voter crap. If votes are public, you can be coerced to vote.
Note that now it's also very hard to know if a journal that is not in your area is serious or not. There are some metrics like impact index (average number of citations ¿during the first year?), but they are easy to game, different communities have a different number of average citations in each paper, and different community have different time to publish. An impact index of 1 is good en math and bad in physics.