Go back

Negative results are important

Curt Rice

“When you have eliminated the impossible, whatever remains, however improbable, must be the truth.” So said the famous detective Sherlock Holmes in Arthur Conan Doyle’s novel The Sign of the Four.

Research fails; almost always. Scientists discard hypotheses like so many untried drones, cast out before even getting a chance. This is the nature of research. It’s the approach that gives us the best results—the breakthroughs, the insights, the improvements in our quality of life. Failure in research is not a problem; in fact, it can push us forward.

Thomas Edison’s associate, the story goes, was frustrated with nearly a thousand unsuccessful experiments for a project. He was ready to throw in the towel, but Edison talked him out of it. “I cheerily assured him that we had learned something,” he is reported to have said in a 1921 interview. “We had learned for a certainty that the thing couldn’t be done that way, and that we would have to try some other way.”

But if failure in research is itself not a problem, the communication of such failures is. If we learn from negative results, why is it so hard to publish them? Is there a systemic challenge for the distribution of such research? If so, what changes might help?

Journals don’t have policies against publishing negative results. The World Association of Medical Editors states, on the contrary, that “studies with negative results…should receive equal consideration”. Even so, statistically significant results increase the chance of publication, lowering the odds that negative results get into print.

It may be difficult to publish negative results simply because reviewers find such studies less interesting. It could also be that editors are under pressure from publishers to increase the impact of their journals, and they know that studies with negative results are less cited than those with positive results.

Perhaps the attitudes of researchers are relevant, too. It may be more interesting to move onto a new hypothesis than to write up the results of a failed one. Maybe it’s embarrassing to have a hypothesis that didn’t pan out.

Some offer more sinister speculation: researchers don’t publish their own negative results because they don’t want to help the competition. This highlights a peculiar feature of the organisation of universities.

University departments have two core activities: teaching and research. Each applies different pressures on hiring decisions. Teaching coverage requires breadth, while research success requires depth. Hence, research communities tend to span institutions.

Those institutions may collaborate, but they actually compete with one another for funding. Could this be an incentive to leave negative results unpublished? Might a team consider its negative results a competitive advantage, insofar as such results contribute to the context for subsequent positive results?

So the skew in favour of positive results follows from a lack of incentives to publish negative results. If researchers themselves find the results uninteresting, or anticipate that reviewers will be less enthusiastic, or know they will be cited less, or even believe they surrender a competitive advantage by publishing such results, why should they bother?

What changes could counter these pressures? Can we imagine incentives to publish negative results? I think at least two developments could play a role.

First, funding models that give economic rewards for publication could lead to the appearance of more negative results. Norway has recently adopted an elaborate national scheme of this type. Publication records are now directly tied to funding for travel, research assistance, equipment and so on—publication itself generates funds. Tracking the developments triggered by this system may reveal an increase in the publishing of negative results.

Second, there is increased political pressure to connect research with innovation. The European Commission’s next funding programme is the proposed Horizon 2020—The Framework Programme for Research and Innovation. Pressure to demonstrate innovation may yield increased publication of negative results. Negative results can winnow the possible directions for innovative applications, and thereby demonstrate the usefulness of research—in turn increasing the chances of EU funding. Examples range from medicine and evolutionary biology to the social sciences.

So, there is broad—though not unanimous—consensus that publishing negative results is important for science. In many ways, today’s system discourages this. We should keep an eye on developments to see if they lead to change.

And if they do, then we will join Sherlock Holmes in a more open pursuit of truth—not only by eliminating hypotheses, but by telling each other when we do.

More to say? Email comment@ResearchResearch.com

Curt Rice is the pro-rector for R&D at the University of Tromsø, and chairman of the board for the Current Research Information System in Norway. He blogs on topics related to university leadership.