Comment by dag100
6 days ago
Unless reviewing becomes more profitable than publishing, anything that makes both easier will drive one up far more than the other. And it is difficult to conceive of something that would make reviewing much easier without making publishing much easier.
Just as a counterpoint ML and AI research has become much more reproducible over time. I feel like this is relevant because ML / AI researchers are huge power users of AI tools.
Between 2016 and 2021 the share of ML/ robotics/ AI researchers being reproducible (ie contianing code and similar instructions to reproduce) doubled [1].
The major US labs have gone largely closed source (I.e. they no longer publish frontier research) but the Chinese ecosystem has incredibly reproducible code.
This is field dependent obviously but I think it atleast gives reason to be optimistic.
Yes people will churn out fake slop research, but it feels like that can be categorized and then ignored.
[1] https://arxiv.org/pdf/2308.10008
That's good to hear about ML and AI research, but most research is not based on computers and so would require laboratory setups to reproduce. Not only is trying to reproduce such findings (beyond what is effectively a sanity check) through simulations a lost cause, if AI can reproduce such research it would be capable of doing such research itself... in which case it would be far more fruitful to use AI to do further research.
This thread is about a product based on fully reproducible research though so I feel like we should stay grounded. Claude science is meant to be used in the context of reproducible science research, there is a decent reason to not be cynical on future research being reproducible.
> if AI can reproduce such research it would be capable of doing such research itself
Well there is a big distinction between research validation and research generation, it is generally much easier to verify that a math proof is true or false than to find a truly novel proof.
But yes in the long run I’d think AI will be doing tons of research and it will by default reproducible. So maybe we’re aligned after all?