Comment by downWidOutaFite
3 years ago
We just need an up/down voting mechanism for scientific papers with voting restricted to people knowledgeable in the field. Don't know how to implement it but it should be designed with incentives to vote and guards against it being gamed.
Moderation is hard.
Also, there is the homeopathy problem. Do only homeopaths vote if a homeopathic paper is good or bad, or any medical doctor can vote?
It will be very difficult to distinguish voting ring from serious research, because some communities are very small and each one cite each other paper, and have some papers together from time to time.
Now the idea is that the jornal has a magic public vote, and if the paper is bad the reputation of the journal is stained, and after enough bad papers the hivemind decides it's a bad journal.
Another form of votes are the citations. So instead of karma, there is the total number of citations, the h-index, and other form of counting them. They also can be gamed, because people can add self citations or citations to friends, specially in the introduction. And sometimes the reviewers ask for a cites, like in https://news.ycombinator.com/item?id=35738717 but most of the times it's more subtle.
There are a few new journals that claim to be "peer review", but are actually just a forum with upvotes and commets. I'm not sure if any of them are "serious" because they are not in my area, but moderations is very hard. If votes are secret, there is no incentive not to voter crap. If votes are public, you can be coerced to vote.
Note that now it's also very hard to know if a journal that is not in your area is serious or not. There are some metrics like impact index (average number of citations ¿during the first year?), but they are easy to game, different communities have a different number of average citations in each paper, and different community have different time to publish. An impact index of 1 is good en math and bad in physics.