Comment by logicallee
6 hours ago
I answered requests to be a peer reviewer. (I'm not sure why I was selected, I don't have many publications or credentials.) I saw a lot of papers with hallucinated references. I also remember one paper that described a methodology that I don't think the authors really performed, I think it was just academic fraud where they pretended to have performed an experiment. At the time that I answered the journal requests, AI could hallucinate fake reports, but agents weren't powerful enough to run the experiments yet.
These days agents are able to really perform genuine experiments and write up the results. A prompt like this: "You'll work autonomously end to end to select a research task that meaningfully advances the state of the art in AI, is clearly defined and worth performing, that people would be interested in reading, and that you can perform on this hardware" (insert details) " in a week. Carefully log your steps so that your results can be replicated. Then, do a research review and write your paper about it up with correct, cited references. You must check all of your citations. Look up current lists of "Claudisms", (such as use of the word "genuinely", or "load-bearing"), and remove them from your writeup. After writing your writeup, edit it and pare it down, remove anything unnecessary, keep it fast paced and interesting. Also, try to tell a story, be engaging in your writeup. Don't use violent metaphors, remove references to killing, strangulation, etc. Your writeup should be ready to publish and accurately reflect a real experiment with a meaningful result that advances the state of the art and contributes to understanding. Be concise and focus on why it matters."
Okay, so there's the prompt. You can give it to any AI and have a journal-ready publication in a week. I guess you can ask it to add charts and stuff, if you want to be fancy.
If I gave my agent the above prompt, would I be one of the authors? Maybe it's fair to say I guided, facilitated, elicited, or advised it. But it's clear that the AI would be the one that is actually selecting and running the experiment and writing up the results.
Someone could probably get a publication without even reading the paper they wrote their name on. Their only contribution might be editing their name into the PDF.
It can be turned around as well, for validating existing research, creating a bot that checks papers against all the well-known logical fallacies, issues with statistical methods, checks the images etc.
I hate to say this, but a researcher in biology ain't a statistician or logician. Many math paper hand waves over many parts of a proof.. yet both of those can be and have been useful in practice.
That may well be the case. My suspicion is that many research papers would not pass such a quality review. I am not saying anything about their usefulness, though.
You are not suggesting it, but as some might, I want to emphasize that I think it is, really, really, really bad idea to argue for limiting exposure of the bad practices because the papers may be useful even when not following good practices.
The science can work only when we can trust that the foundation it has been built on, including previous research, is solid.
The trust is partly based on the fact that science is self-correcting. As we know from charge of electron measurements, the existing narrative can work against the self-correction, even when everybody is trying to be as truthful as possible.
My impression is that to a large extend, on many fields, publication numbers (and money) have become so important that self-correction process may have become broken.