Comment by godelski

This is what I wished academia would be. I'm finishing my PhD and despite loving teaching and research (I've been told I'd make a good professor, including from students) I just don't see the system doing what it should. Truthfully, I'm not aware of any such environment other than maybe a handful of small groups (both in academia and industry).

I think we've become overly metricized. In an effort to reduce waste we created more. Some things are incredibly hard to measure and I'm not sure why anyone would be surprised that one of those things is research. Especially low level research. You're pushing the bounds of human knowledge. Creating things that did not previously exist! Not only are there lots of "failures", but how do you measure something that doesn't exist?

I write "failure" in quotes because I don't see it that way, and feel like the common framing of failure is even anti scientific. In science we don't often (or ever) directly prove some result but instead disprove other things and narrow down our options. In the same way every unsuccessful result decreases your search space for understanding where the truth is. But the problem is that the solution space is so large and in such a high dimension that you can't effectively measure this. You're exactly right, it looks like waste. But in an effort to "save money" we created a publish or perish paradigm, which has obviously led to many perverse incentives.

I think the biggest crime is that it severely limits creativity. You can't take on risky or even unpopular ideas because you need to publish and that means passing "peer review". This process is relatively new to science though. It didn't exist in the days of old scientists you reference[0]. The peer review process has always been the open conversation about publications, not the publications themselves nor a few random people reading it who have no interest and every reason to dismiss. Those are just a means to communicate, something that is trivial with today's technologies. We should obviously reject works with plagiarism and obvious factual errors, but there's no reason to not publish the rest. Theres no reason we shouldn't be more open than ever[1]. But we can't do this in a world where we're in competition with another. It only works in a world where we're united by the shared pursuit of more knowledge. Otherwise you "lose credit" or some "edge".

And we're really bad at figuring out what's impactful. Critically, the system makes it hard to make paradigm shifts. A paradigm shift requires a significant rethinking of the current process. It's hard to challenge what we know. It's even harder to convince others. Every major shift we've seen first receives major pushback and that makes it extremely difficult to publish in the current environment. I've heard many times "good luck publishing, even if you can prove it". I've also seen many ideas be put on the infinite back burner because despite being confident in the idea and confident in impact it's known that in the time it'd take to get the necessary results you could have several other works published, which matters far more to your career.

Ironically, I think removing these systems will save more money and create more efficient work (you're exactly right!). We have people dedicating their lives to studying certain topics in depth. The truth is that their curiosity highly aligns with what are critical problems. Sometimes you just know and can't articulate it well until you get a bit more into the problem. I'm sure this is something a lot of people here have experienced when writing programs or elsewhere. There's many things that no one gets why you'd do until after it's done, and frequently many will say it's so obvious after seeing it.

I can tell you that I (and a large number of people) would take massive pay cuts if I could just be paid to do unconditional research. I don't care about money, I care about learning more and solving these hard puzzles.

I'd also make a large wager that this would generate a lot of wealth for a company big enough to do such a program and a lot of value to the world if academia supported this.

(I also do not think the core ideas here are unique to academia. I think we've done similar things in industry. But given the specific topic it makes more sense to discuss the academic side)

[0] I know someone is going to google oldest journal and find an example. The thing is that this was not the normal procedure. Many journals, even in the 20th century, would publish anything void of obvious error.

[1] put on open review. Include code, data, and anything else. Make comments public. Show revisions. Don't let those that plagiarize just silently get rejected and try their luck elsewhere (a surprisingly common problem)