Comment by jbdamask

Thanks. I can guess who this is but not 100% sure.

Totally agree with what you're saying. This tool ignores supplemental materials right now. There are a few reasons - some demographic, some technical. Anything that smells like data science would need more rigor.

Have you looked into DocETl (https://www.docetl.org/)? I could imagine a paper pipeline that was tuned to extract conclusions, methods, and supplemental data into separate streams that tried to recapitulate results. Then an LLM would act as the judge.