Research Requires Reproduction of Results

source

Independent verification of a scientific result greatly increases our confidence in that result.

Although we never prove anything in scientific research, as more and more experiments achieve the same answer, confidence grows that we understand some ground truth. That could be as simple as measuring the speed of a falling object to confirm the theory of gravity, or as complex as tracking the way that neurons traffic cargo along their axons to a synapse (axonal transport). In these examples, the latter relies on strong fundamental understanding achieved by reproducible measurement of the former.

Scientists have Serious Problems with Reproducibility

Many other articles cover the issue of reproducibility, such as this one, or this one.

One obvious and easy-to-fix problem is the poor descriptions of experimental methods that are given in scientific publications. Some journals are dealing with experimental methods head-on, such as the Cell family of journals that introduced STAR (Structured, Reproducible, Accessible Reporting) methods in 2016.

Bias Hinders Reproducibility

Bias in scientific research is actually the rule rather than the exception.

The reasons for this are interesting and require discussion, such as the pressure to publish for promotion or tenure, but that is beyond the scope of this article. It is accepted that bias is usually unconscious or unintentional, so methods to reduce bias should be embraced.

'Blinding' Improves Reproducibility

source

You've probably heard the term 'double-blind' study, especially in reference to clinical trials.

A 'Blind' study means that the participant is not informed ('blinded') about the test. 'Double-blind' means that both the scientist and the participant don't know the treatments they are giving and receiving, respectively.

The classic example of a single-blind study is the Pepsi challenge, where a person would be allowed to taste two visually identical drinks, and chose the one they like more. One might question the results of this test as the person giving the two glasses might treat the Pepsi differently, e.g. with more ice, and therefore influence the study. The scientist could also influence the study subconsciously. Thus, more control is needed to get unbiased results.

To make the Pepsi challenge double-blind, a third party (call them the administrator) prepares the two glasses of soda and then hands them to the scientist with random labels. The scientist would then ask the participant to taste each soda and report their preference. In this case, the scientist cannot influence the outcome overtly or subconsciously. The scientist would also then be unable to determine whether the person preferred Pepsi or Coca Cola without reporting the data to the adminstrator, who knows the identity of the random labels.

Double-blind studies are required to assess drug safety and efficacy because studies without blinding, called open label, are not reproducible. Although it should be encouraging that we've required this scientific rigor in studies where the outcome could harm people, the prevalence of blinding in research is very low.
Many lines of evidence support the need for blinding. For a nice overview of the evidence and a perspective see this article in PloS Biology. For example, one study looked at over 2,000 conference proceedings and found that studies without blinding or randomization were over 3 times more likely to achieve their desired statistical outcome than those that used blinding and/or randomization. That is unacceptable and we need to hold ourselves as scientists to a higher bar.

It is Easy to Blind Experiments

In most cases, we can easily avoid this problem of bias by introducing blinding.

For example, one researcher could collect the samples, (e.g. mice, etc.) and give them random labels before handing them to a technician for experimentation and data collection.

Although it does make our experiments a little harder to complete, we cannot be lazy while searching for the truth. Incorrect conclusions resulting from bias waste millions of taxpayer dollars, and more importantly waste too much of the few precious hours and days we all have remaining.