Comment by pwinnski
17 hours ago
The entire point of an academic paper is to add to the sum of human knowledge. How can an LLM trained on a subset of human knowledge possibly even begin to accurate evaluate such a paper?
I trust an LLM to review that the language used in the paper is grammatically correct, but not to evaluate new information for accuracy.
Schmidthuber has an answer: https://arxiv.org/abs/0812.4360
For a more practical approach you need to use proxies: https://zby.github.io/commonplace/articles/what-an-automated...
This is very bad logic
1) Humans also are trained on a subset of human knowledge. 2)A lot of papers are just about experimenting something, and then applying simple stats. Eg empirical studies, around 1/3rd of published papers. Like, we tried this drug or did this experiment, from a sample size X here are the results. An expert is needed to maybe comment on the conclusion/hypothesis of the underlying suspected mechanism, but LLMs are still very useful on catching bad statistics or p hacking (so so common)