Comment by torben-friis
16 hours ago
Humanity is very biased for the culmination of work, considering everything that comes before and after busywork for the lower masses.
Replicating a paper is just as valuable scientifically as publishing it, but how many careers advance through replication?
If we move the goal from "find the solution" to "clear up the LLMs work" that doesn't bode well neither for the attractiveness of the problem nor for the career of the professional that takes the challenge.
> Replicating a paper is just as valuable scientifically as publishing it, but how many careers advance through replication?
A lot. In fields where knowledge is incrementally building on previous work the reason the whole field hasn't collapsed from the replication crisis is that usually the results that are really high impact are replicated in as an initial step in new research building on it. It's almost never the focus of the paper but you'll often find a quick mention in methods/supplemental of some previous work that was verified to be valid by a replication of a key technique etc. you'll have crisis where old tools are found to be problematic and findings end up revisited etc. Plus fields like clinical research where there's an awful lot of focus on replicating findings using staged clinical trials with increasing statistical power to determine if new interventions work - that's driven by regulatory requirements grounded in good science and a lot of people make careers in just that.
In mathematics, finding novel proofs of a given result is often valuable; it may be a shorter proof (demonstrating better/expanded understanding of the problem) or a translation of the problem into a new domain, setting up more cross-domain advances.
>Replicating a paper is just as valuable scientifically as publishing it, but how many careers advance through replication?
I don’t think this is true, especially for novel or unexpected results. I suppose it depends on what you mean by scientifically, and there is a debate in the philosophy of science about what the value of research even is, but a successful replication does not result in substantial updates to one’s beliefs in the way new research does. And if the goal of science is to change our beliefs and bring them closer to what is “real”, successful replications can’t be as valuable as the initial research almost by definition.
From a pure statistical perspective the first scientific paper shouldn’t update your beliefs as much as the independent replication study.
People don’t behave this way, but a high percentage of all papers have known flaws and that goes up even higher when you consider unknown flaws. Replication doesn’t own its own solve the underlying issue, but independent replication removes a huge range of potential issues on top of providing more information.
I agree, with the caveat that it probably matters how novel the paper is, the effect size, and the confidence interval. A reputed new psychological phenomenon I'd be more skeptical of than, idk, a newly detected exoplanet.
That alleged superconductor from a few years ago - everybody kind of held their breath and waited for the reproduction.
Hello.
You have created a fraud machine. Why? With no answer checking then why not make up the most fraudulent crap you can get away with?
Examples: A huge portion of recent non-reproducable science papers.
---
Your thinking, along with everybody that's doing this rat race is causing the pumping out of papers with questionable data, but very little to ensure we are actually making correct science.
I think successful replications are as valuable as the original research because they're not unsuccessful replications
It is true for mathematics certainly. I would guess it is less true for science per se.
I find that to be an issue of maturity (focusing only on the climax and not the process). In Japan, where I live, the culture has a greater appreciation for the context & process, not just the moment of victory.
If you examine the consequences of the inversion of the peak, you realise the need for a balanced perspective.