Comment by D-Machine
4 hours ago
Yup, strongly agree with all of this, especially the RCT stuff.
This has all been profoundly obvious for at least well over a decade or even two now. A consequence has been that too many serious people are driven away from academia and research, to the detriment of science generally.
I've no idea what to do about all this, because people have voiced obvious and easy solutions for decades, but they are all routinely ignored.
I think political lobbying for legal changes might be the only realistic pathway, in that current legislation (eg. GDPR in the EU) is extremely punitive even for minor violations.
I personally am trying to float using local LLMs to create anonymized case files and auto-suggest publishing cases in my hospital, which knowing how things work will probably never amount to anything.
Or if you want to float truly insane ideas I guess you can shop around with blackhat groups and see if anyone has stolen some juicy records/data during all the ransomware attacks and databreaches over the years, and do some rogue scientific publishing. Obviously that's crazy, but I have to admit that the notion of pirate scientists plundering and publishing data is hilarious to me.
Which are the obvious and easy solutions?
Make analysis code available. Make anonymized data available for download without people having to jump through hoops to get it. If you have highly sensitive data, release only the variables or other statistics needed to reproduce core analyses. Don't only do garbage null-hypothesis significance testing or statistical analyses on the full data, also do ML approaches were you have to actually show your analyses replicate on held-out subsets, and report this. Make reviews open (anonymizing as needed) so we can see when biased or incompetent reviewers are blocking good publications. Allow public review (or at least broader academic open review, in some form), since it is no longer defensible to delegate review and decisions to one or two random people that just happen to be emailed and have the time / are on some editorial / review board. Also allow public post-publication review. Publish null findings / results, if only in minimal forms so we don't waste time and money trying to reproduce garbage. Make articles available and don't charge insane article processing fees or open access fees of thousands of USD (especially since hosting fees are not that crazy, and also because journals don't do any of the formatting work half the time anyway, and make academics or RAs or students do all the typesetting and formatting, even though now this could all be automated with template files, mostly).
Most of these things are easy to do for the majority of papers, especially in the past 20 years with the internet and modern tech and software. Plenty of frameworks exist already that have done most and/or at least some of these things, but, collectively, academia is decades behind overall.