Replication, the ability of another lab to reproduce a finding, is the gold standard of science, reassurance that you have discovered something true. But….
… colleagues could not replicate 47 of 53 landmark papers about cancer
Fears [of] questionable findings began to emerge in 2005, when Dr. John P. A. Ioannidis, a kind of meta-scientist who researches research, wrote a paper pointedly titled “Why Most Published Research Findings Are False.”
…. examining more than a decade’s worth of highly regarded papers — the effect of a daily aspirin on cardiac disease, for example, or the risks of hormone replacement therapy for older women. Ioannidis found that a large proportion of the conclusions were undermined or contradicted by later studies.
His work was just the beginning. Concern about the problem has reached the point that the journal Nature has assembled an archive, filled with reports and analyses, called Challenges in Irreproducible Research.
Among them is a paper in which C. Glenn Begley, who is chief scientific officer at TetraLogic Pharmaceuticals, described an experience he had while at Amgen, another drug company. He and his colleagues could not replicate 47 of 53 landmark papers about cancer. Some of the results could not be reproduced even with the help of the original scientists working in their own labs.
Given what is at stake, it seems like a moral failing that the titles of the papers were not revealed. That was forbidden, we’re told, by confidentiality agreements imposed by the labs.
Maybe the researchers deeply believed that their findings were true. But that is the problem. The more passionate scientists are about their work, the more susceptible they are to bias.
The fear that much published research is tainted has led to proposals to make replication easier by providing more detailed documentation, including videos of difficult procedures. A call for the establishment of independent agencies to replicate experiments has led to a backlash, a fear that perfectly good results will be thrown out.
Scientists talk about “tacit knowledge,” the years of mastery it can take to perform a technique. The image they convey is of an experiment as unique as a Rembrandt.